Choosing the browser-side data storage API

Kamishibai has limited offline support at present, and we plan to bring it to full support within 2013. By full support, we mean that despite Ponzu being a web-page, we want to make the vast majority of pages available even when offline. The reason we take this very seriously is because WiFi at conferences tends to be very sketchy and temperamental.

Choosing the data storage API

Although HTML5 has support for offline applications, it is still very much in its infancy.

The Application Cache API is relatively well supported on all new browsers (> IE9). However, this API, while easy to use for the most basic cases, tends to be difficult for more advanced uses. Most programmers reserve it for static assets.

To store each presentation, session and user profile, we need a cache that is more flexible. Hence, in these cases, we use a data storage API that allows us to create, update and remove records programmatically and at will (the Application Cache API can only update all entries or zero entries). The APIs for this have not been well developed. We have localStorage, webSQL and indexedDB APIs.

localStorage is the simplest to use but also has the least features. The storage capacity is also rather small and no mainstream browser allows more than 5MBytes of storage.

webSQL is basically a sqllite3 database wrapped in Javascript. Since it can use SQL statements, there is a lot that you can do with it. Unfortunately, it has been abandoned by the W3C working-group because it is too tied to the sqlite3 implementation. webSQL can be expanded to 50MBytes of storage even on mobile devices, and is well supported on both iOS and Android. Safari and Chrome desktop browsers also support this API.

indexedDB is the replacement for webSQL. It is currently supported on IE10, Chrome and Firefox. Safari does not support it on both desktop and iOS versions. Also, the Android stock browser does not support this. It is the newest standard, and will most likely become the standard in the future. However, it will take several years before it becomes mainstream enough.

We ended up combining localStorage and webSQL for Kamishibai. Our rationale is discussed below. In the future, we plan to add indexedDB support.

webSQL looks great on features, but falls short on some important details

Having the full power of an SQL database and also broad support on mobile devices, webSQL looked like the prime candidate for Kamishibai data storage. Furthermore, since the 5MByte storage limit for localStorage would most likely be insufficient for large conferences, we initially planned to go 100% webSQL. However, we hit some snags.

First of all, when we use the openDatabase command to create a new webSQL database on Safari, and databases with an initial size exceeding 5MBytes brings up a modal dialog to confirm with the user. This is perfectly OK if the user is already familiar with the Ponzu system. However, a large number of people would be afraid and alarmed. They might opt to not allow access. More seriously, they might be deterred from visiting the site.

Memcached like LRU (least recently used) management

Since localStorage can only store 5MBytes and WebSQL puts up a modal dialog to confirm usage of more than 5MB, we have to consider how we are going to manage with storage that is smaller than the conference scientific program (it will easily exceed 10MBytes).

One idea is to purge entries that are infrequently accessed. A good scheme is the LRU management in memcached. The idea is to detect when the storage quota has been exceeded, and then to delete old entries to make space. Old entries are those that have not been recently accessed.

Implementing an LRU-type scheme is possible in localStorage. localStorage fires a QUOTA_EXCEEDED_ERR when we try to store more than the quota (5MB). We can catch this error and delete old entries, then retry.

However, webSQL at least on Safari is different. A quota exceeded error fires, but only after the modal dialog has shown up. This means that we cannot transparently delete old entries. This makes it very difficult to use webSQL as a LRU cache. Hence webSQL is good if we can get the user to agree to using a large quota (like 50 MB). However, using webSQL within the 5MB limit is very difficult if the user rejects the request.

Final scheme

Our final scheme is to initially use localStorage, using an LRU-based storage scheme. Since we will not be able to store the whole scientific program, this is basically a history-like cache.

We will also provide a store data locally button that the user can click to download the whole scientific program. This will use 50MB of webSQL storage. A modal dialog will pop up, but the user is expected to approve it. If they don’t, then we will fall back to localStorage. If the user uses webSQL, then we have no further need for localStorage, and we work 100% in webSQL.

Leave a Reply

Your email address will not be published. Required fields are marked *