Offline-First Database Options for Web Applications in 2020

Intro

For most people building web applications, the general idea is that the user is connected to your service a fair amount of the time; there might be periods of time where they go offline, but the goal is for a state of being “connected”. This assumption means that the most common methods for storing and retrieving user data for the UI are things like:

Standard AJAX requests
- Database agnostic; the client-side code does not even necessarily know what type of DB is on the other end
- Could be REST, GraphQL, etc.
Websocket session
- Similar to AJAX; database agnostic, requires active connection
Managed database services (such as Google Firestore and AWS DynamoDB)
- These typically support an “offline” mode, but the main functionality is for sync / online operations
- You don’t need to have a server

However, there is a gap left by all of the above approaches. What if we are developing something that is meant to be used offline, after the initial page load? Is there a way we can persist and query user data locally, without ever needing a server?

Apps that are designed to be fully functional without an internet connection are often categorized as “offline-first”. There is even a community around the topic.

Options

In 2020, the best overall option for local data persistence is going to be the IndexedDB browser API. The explanation for why this is the superior option (over localStorage, cookies, etc.) is long and technical, so I’ll defer to the experts (this is a good summary, from web.dev). However, using IndexedDB is a little tricky (it is a low-level API) and most developers (including myself) are going to prefer a wrapper library around it, which will make writing code to use it easier.

My research into the libraries supporting IndexedDB-backed local persistance and querying is the reason for this post. I was also interested in which libraries supported things like manual import / export shortcuts (so I don’t have to write a huge amount of code just so a user can download their own data), sync, NodeJS support, and legacy fallbacks.

Comparison Table

I’ve been using Google Sheets to record my findings – you can access it here (no login required). I’ve also copied and pasted the main table below:

Library	IndexedDB	localstorage (fallback)	NodeJS	Sync?	Native Export / Import (aka dump imports)?	Notes
Dexie.js	Yes	No, not really.	/ = You can polyfill	/ = adapter required	Yes (adapter)	Although legacy support might not be as nice as localForage, and sync not as nice as PouchDB, overall has great performance and is well-liked. Also in active production use with some major players.
PouchDB	Yes	Yes (adapter)	Yes (adapter)	Yes!	/ = maybe with some other libs	CouchDB interoperability and sync features are the main selling points, but is also a very well thought out product, that is well-liked and maintained. If you are not used to NoSQL / CouchDB, the syntax can be a little daunting.
RxDB	Yes	No	Yes	Yes	Yes	RxDB is built on top of PouchDB, and you can think of it as providing a reactive layer between your data and your UI, whereas normally you might need to manually code timed synchronization events. Since it is built on PouchDB, syntax is still NoSQL / documents. It comes with a large feature-set, beyond just the reactivity; schema migrations, ORM capabilities, and field encryption are all features not present in PouchDB, but come out-of-the-box with RxDB.
localForage	Yes	Yes	No	No	No	localForage has been around for a long time, and is stable and well-liked, but it also remains focused on a simple localStorage like API. For some, this might not be enough and you might want to evaluate some of the alternatives with a larger feature set.
nanoSQL	Yes	Yes	Yes	No	Yes	Seems like a very ambitious, but well thought out project, with good results. Extensibility and portability seem to be the key here, which is evident in the number of adapters, plugins, and customization options.
alaSQL	Yes	Yes	Yes	No	No	Looks really neat for large in-memory queries and offers some unique compatibilities (supports Excel `.xls` and `.xlsx`), so might be a good pick for BI applications. For lots of constant persistent syncing (e.g. IndexedDB read / writes), might not be the best pick since the IndexedDB connection is sort of manual.
LokiJS	/ = adapter required	Yes (default on Web)	Yes	No	It’s complicated / Depends	Focus is on in-memory manipulation.
lowdb	No	Yes	Yes	No	Kind of (since it is just JSON, you should be able to dump and import)	Focus is on fast, easy-to-use, and small footprint. Supports simple `.json` files as persisted database stores. Good for fast prototyping, but might be under-powered for larger projects.
WatermelonDB	/ = Using LokiJS Adapter	No	Yes	Yes.	No	Similar to RxDB, WatermelonDB is focused on data reactivity, tied to the React UI layer. Syntax uses a decent amount of ES6 decorators.
TypeORM	/ = Using sql.js + localForage adapters	Yes	Yes	Not really	No	ORM, not DB Layer
Orbit	Yes (std adapter)	Yes (std adapter)	Yes	Yes	No	This came from the Ember JS team and seems designed to solve a lot of complex problems. A lot of it is kind of over my head…
ydn-db	Yes	Yes	No	Yes	No	This does not seem in-use or maintained.
JsStore	Yes	No	No	No	No	Main focus seems to be on providing SQL syntax, but with IndexedDB backing.
turtleDB	Yes	No	No	Yes	No	Abandoned?
kinto.js	Yes	No	No	Yes	Yes	Looks like this was built by Mozilla employees and used for several large-scale Firefox products / features (see “Kinto at Mozilla”). kinto.js is also a part of a much larger eco-system – “Kinto-Storage”. For sync, it should be noted that this specifically requires a server running the Kinto server (which runs on Python + PostgreSQL)

Side Note – Managed Database SDKs

You might have noticed that some (very popular) managed / serverless databases were left out of the above table. Notably, Google Firestore and AWS DynamoDB. I’m always a little wary of recommending solutions like these, because they typically come with vendor lock-in, proprietary components, and cryptic pricing structures.

However… let’s be honest, many developers prefer these frameworks because they are well-supported, thoroughly documented, and easy to use. Some examples of these types of services and how they might (or might not) fit our needs:

AWS Amplify DataStore
- This might be the closest to what we are looking for, in a large-scale commercial offering, for several reasons:
  - Designed to align with the offline-first paradigm; local on-device database, that takes advantage of modern standards (e.g. IndexedDB on web), while also supporting syncing once online
  - Amplify DataStore appears to support a completely offline mode – no AWS account required
  - Supports sync (DataStore <-> AWS AppSync <-> Amazon DynamoDB)
  - Built-in conflict resolution
Google Cloud Firestore
- My opinion: This does not look like a good fit for “offline-first” applications, for several reasons:
  - Offline data with Firestore is meant to be used for caching or temporary network outages, but not as an offline-first local database
  - They openly admit that too much offline data slows down queries
  - It sounds like Google is not going to make efforts towards better offline support
Realm
- Realm is an entire platform for data usage in mobile apps. A lot of it seems pretty proprietary. They also do not (yet) have a web version.
- If you are looking for real-time sync with advanced conflict resolution strategies, this might be what you are looking for, but seems a little niche and outside of what I’m looking for

Side Note – NodeJS and Other Environments

Web is a very special environment, especially when it comes to databases. Part of the reason why things like IndexedDB are so important with web is because we don’t have access to the local filesystem with JavaScript; we can’t simply create a SQLite file and read/write to it. However, in NodeJS, as well as native environments like Android, iOS, and desktop applications, we are not subjected to the same restrictions.

However, several of the approaches and solutions above also work in non-web environments, such as NodeJS. For example, RxDB works with the web, as well as NodeJS, Electron, or even mobile (React-Native).

Wrap Up

What Should I Use?

You might have noticed that I did not declare a winner out of the options I came across in my research. That is because I don’t really think there is one clear best pick for all types of web apps. The general conclusion reached is:

No matter what, use something backed by the strongest standardized low-level API (that is IndexedDB for the forseeable future)
You should decide how much you care about the relationship between the UI and the data
- Do you want bi-directional data binding? Optimistic UI updates?
- If you care a lot about reactivity, you might want RxDB or WatermelonDB
Native export / import might not be worth that much as a feature, since you can always code a reusable method anyways, which can also handle versioning conflicts unique to your app

Donate

Follow Me Elsewhere

Share This Post

2 thoughts on “Offline-First Database Options for Web Applications in 2020”

Leave a Reply Cancel reply