Tag Archives: Kamishibai

For us, the largest benefit of Javascript templating is reduced size

There are quite a few Javascript templates. In my projects however, there are very few cases where I would prefer using any of them in place of regular HTML being pushed out from the server (running Ruby-on-Rails). The same can be said of the Ponzu conference system.

As far as I understand, the benefits of using Javascript templates are 1) reducing the load on the server (generating JSON is less load than generating full HTML), 2) speed if used in combination with a single page design.

The downside is the additional work that browsers have to do, which can be a problem on mobile where the devices are not as powerful as their desktop counterparts.

I’ve touched this subject before in these two posts [1](https://code.castle104.com/?p=289), [2](https://code.castle104.com/?p=291).

As discussed by David Heinemeier Hansson, the same benefits can be achieved without Javascript templates by using a PJAX/Turbolinks/Kamishibai like system that eliminates reloading Javascript and CSS on each page transition, and the use of aggressive caching on the server side to reduce the load of HTML generation.

There is a real case however, where I feel a strong need for a Javascript tempting language.

That is when I try to cache responses in browser side cache. The issue is that HTML is extremely verbose, and is a real killer in terms of storage consumption when you are working with repetitive content.

For example, the following is a “social box” that we use in Ponzu for the like button and a voting button. It takes about 2,000 bytes. Each social box is associated with a presentation so we have hundreds to thousands of social boxes for each conference. This can easily fill up the limited browser side cache.

<div class="" id="presentation_326_social_box"><div class='like_box'>
<div class='like' id='like_button_326'>
<span class='social_controls'>
<!-- To invalidate Like related paths, we need a like object -->
<a href="/likes?like%5Bpresentation_id%5D=326" class="button icon like" rel="nofollow">like</a>
<div class='prompt_message'>
(
To add to your schedule, please &quot;like&quot; it first.
)
</div>
</span>
<div class='social_stats'>
<img alt="Like" src="/assets/like-c3719a03fc7b33c23fda846c3ccfb175.png" title="いいね!を押すと、応援メッセージになります。またあなたのスケジュールに登録されます。" />
<a href="/presentations/326/likes?user=1">15 people</a>
liked this.
</div>
<div class='likes_list' id='likes_presentation_326'></div>
</div>

</div>
<div class='vote_box'>
<div class='like' id='vote_button_326'>
<span class='social_controls'>
<form accept-charset="UTF-8" action="/likes/vote" class="new_like" id="presentation_326_new_like" method="post"><div style="margin:0;padding:0"></div>


<span>

<label for="presentation_326_like_score_1">Excellent!</label>
</span>
<span>

<label for="presentation_326_like_score_2">Unique!</label>
</span>
<span>

<label for="presentation_326_like_score_0">No Vote</label>
</span>
</form>
</span>
</div>

</div>
</div><div id="session_details_presentation_326"></div>

Most of this content is repetitive and will be identical for each “social_box”. In fact, the content that is unique to each individual social box can be summarized in the following JSON.

[{ 
    presentation_id:326,
    user_id:1,
    score: 0,
    liked: 0,
    scheduled: 0
}]

If we could use Javascript templating to generate the 2,000 byte HTML from this small set of JSON, local storage savings would be huge.

This is one feature that we will be adding to Kamishibai and Ponzu in the near future, to enable the ultimate goal of complete offline viewing.

On HTTP caching

Kamishibai provides support for storing Ajax responses on the client using either localStorage or WebSQL (and Indexed DB is planned in the future). This enables us to dramatically speed up page loads by not sending the request out to the server, but retrieving the response from internal storage. It also allows us to provide offline access to pages.

HTTP itself provides the HTTP cache protocol which allows the server to control browser cache through the “Cache-Control”, “Expires”, “Last-Modified”, “If-Modified-Since”, “ETag” and “If-None-Match” HTTP headers. Ruby-on-Rails also provides methods that allow us to easily manage these headers.

The question is, why did we create our own caching mechanism for Kamishibai, instead of using HTTP cache. In the following, I hope to provide the answer.

Many Ajax requests are Private Content

The following is an excerpt from the above link describing use-cases from HTTP cache.

Private content (ie. that which can be considered sensitive and subject to security measures) requires even more assessment. Not only do you as the developer need to determine the cacheability of a particular resource, but you also need to consider the impact of having intermediary caches (such as web proxies) caching the files which may be outside of the users control. If in doubt, it is a safe option to not cache these items at all.

Should end-client caching still be desirable you can ask for resources to only be cached privately (i.e only within the end-user’s browser cache):

In Ponzu, our scientific conference information system with social network features, a lot of the content is “Private content”. For example, we generally only show the abstract text to conference participants (non-participants can only view the presentation titles and authors). Hence Ajax requests for presentation detail pages cannot be handled with HTTP cache.

URLs alone are not sufficient as the cache key

HTTP caching uses the URL only as the cache key. If the content changes depending on values in cookies, then HTTP caching doesn’t work.

However with Ponzu, we use cookies to store the current user id and we also store the locale. We display slightly different content depending on the privileges of each user and we also provide different translations. We do all this while keeping the URL the same. Keeping the URL the same is important to maximize social sharing.

Hence in Ponzu, URLs alone are not sufficient as the cache key.

Flexible purging of HTTP cache is not possible

HTTP cache does not allow flexible purging of the cache. For example, if you set “max-age” to a large value (e.g. for a few days), then you cannot touch the cache on the browser until the cache has expired. If you suddenly have an emergency notification that you need to put up, you can’t do it. You have to wait until the cache expires, whatever happens.

With Ponzu, we want to set very long cache expiry dates to maximize fast browsing. On the other hand, we want to be able to flush the cache when an emergency arises. An emergency might be an urgent notification, but it also may be a bug.

Hence HTTP cache is not particularly suitable, and we would not want to set long expiry times with it unless we were extra sure that the content would not change.

Summary

As we can see, HTTP cache is not suitable for the majority of Ajax requests (HTML requests) in Ponzu. Although we use it to serve content in the Ruby-on-Rails asset pipeline, we don’t use it for dynamically created content at all. Instead, we use Kamishibai caching which provides more flexibility.

On JavaScript MVC (part 2)

In Kamishibai and Ponzu, speed is a huge concern. This is particularly true for smartphones. In desktop web sites, it is possible to cram a lot of information and navigation elements into a single page to make up for slow page loading. You can see this in news websites like Asahi.com where 80% of the top page consists of navigation and shortcuts. The idea is that instead of asking the user to click a link and reload a new page (which is slow), the user simply can scroll down to see more content. With smartphones, cramming all this information is simply not a good idea and we have to reload pages.

One way to reduce the load time for pages and to update only the parts that you want is to use Javascript MVC or client-side MVC. With client side MVC, the pages are not reloaded as the content is switched. Instead of sending HTML pages, the server sends JSON data to the browser and browser-side javascript is used to construct the DOM from the JSON data. The advantage is that the client does not have to reload the whole page, and that it can intelligently update only the portions of the DOM that need to be redrawn.

In Kamishibai and Ponzu, we seriously contemplated using these Javascript MVC frameworks. However, we decided not to do so. Instead, our approach is similar to how GitHub handles updates with PJAX, and with how the new Basecamp uses Turbolinks. David Heinemeier Hansson gave a presentation describing why they did not use Javascript MVC extensively and a video is on YouTube.

There are other highly respected programmers who question the use of Javascript MVC. Thomas Fuchs, the author of script.aculo.us and Zepto.js has this to say in his blog “Client-side MVC is not a silver bullet” and his comments on a post about one of his projects, Charm.

I’ve come to the realization that this much client-side processing and decoupling is detrimental to both the speed of development, and application performance (a ton of JavaScript has to be loaded and evaluated each time you fire up the app). It’s better to let the server handle HTML rendering and minimize the use of JavaScript on the client. You can still have fast and highly interactive applications, as the new Basecamp shows—letting the server handle most stuff doesn’t mean that you have to cut back on cool front-end features and user friendliness.

We’ve spend a lot of time getting Backbone to work properly, as the easy-of-use quickly deteriorates when your models get more complex. It’s a great choice for simple stuff, but email is far from simple. We also had to add yet an other extra layer of processing to generate “ViewModels” on the server because the normal Rails serialization of objects wouldn’t cut it.

If you do any non-trivial resources, you’ll quickly end up with JSON objects that are just too large, especially for lists. For emails, imagine that you have nested threads, user avatar images, nested assigned cases, etc.
Because of this, you’ll need specialized JSON objects/arrays for different use cases (search, list view, detail view, and others). It follows that you’ll end up with this with more or less any front-end framework (if you care about performance!). Doing this adds complexity, which can be avoided by rendering HTML on the server where access to arbitrarily deeply nested data is relatively cheap (and can be highly optimized by keeping snippets of HTML around in memcache, etc).

In Ponzu and Kamishibai, we ended up following the traditional Rails route, enhanced with techniques that are seen in PJAX and Turbolinks.

What to we want to achieve

For us, the reasons for contemplating the use of a Javascript MVC framework were;

  1. Speed
  2. Use without network connection (offline apps)

A common benefit for Javascript MVC is interactivity, but this was not a concern for the types of web applications we had in mind which tend to be very read-heavy.

David Heinemeier Hansson has written a detailed post on how the new Basecamp dramatically increased performance through PJAX-like techniques and extensive server-side caching.

This technique basically nullifies the first benefit, because it makes it very easy to increase performance. Furthermore, the code will be almost the same as traditional Rails and hence very simple.

The second issue, offline usage, is something that even most Javascript MVC frameworks do not support very well. This is even more so when the data gets complex. Hence this in itself is not a compelling reason to go Javascript MVC; we would still have to figure out how to do offline usage effectively ourselves.

Why do people use Javascript MVC?

I came across this blog post describing why they used a Javascript MVC framework (Ember.js) instead of simple jQuery.

The author, Robin Ward, gives a specific example.

For example, on the bottom of every discourse post there is a button a user can click to like a post. When clicked, it vanishes and adds a footer below the post saying you liked it. If you implementing this in jQuery, you might add a data-post-id to the post. Then you’d bind a click event on your button element to a function that would make the AJAX call to the server. However, the click function passes a reference to the button, not the post. So you then have to traverse the DOM upwards to find the post the button belongs to and grab the id from there. Once you have it, you can make your XHR request. If the XHR succeeds, you then have to traverse the DOM downward from the post to the footer, and add in the text. At this point it works, but you’ve tied your implementation of the button click to a particular DOM structure. If you ever want to change your HTML around, you might have to adjust all the jQuery methods that accessed it. If this example seems simple – consider that in the footer we offer you a link to undo your like. When clicked, the footer text vanishes and the button appears again. Now you’re implementing the opposite operation against the DOM, only in reverse of what you did before. Discourse even takes it a step further – we know that 99% of the time when you click the like button the request is going to work, so we hide the button and show the footer text right away, even before waiting for the server to reply. In the infrequent event that request fails, we’ll show an error message and pop the UI back to the state it was in before. If we were doing that in jQuery, we’d have to have a callback on our AJAX request that knew how to put the UI back into the state it was in before. A prudent programmer might say, okay, well I’ll have a render function that can rebuild the DOM to the current UI state of whether the post is liked or not. Then both ‘undo’ and ‘like’ can call it. If ‘like’ fails it can call it again. Oh, and we have to store the current state of whether the post is liked somewhere. So maybe we add another data-liked=”true” attribute. ACK! Just typing this all out is giving me a headache!. Congratulations, your code is now spaghetti, your data is strewn out in the DOM and your logic is tied to a particular layout of HTML elements.

Although I understand Robin’s point and I have also experienced frustration when we want to update multiple elements that are in separate locations, I tend to think that using a full-blown Javascript MVC framework is an overkill. No doubt, DHH and Thomas Fuchs would send Javascript in the response to do the complex updates.

In fact, it is pretty difficult to find an Ember.js example on the web that does stuff that is complex enough that simple Javascript would not cut it.

A more intelligent RJS

Given that most of the use-cases of Ember.js and Javascript MVC frameworks seems to be stuff that could be done with regular Javascript, but might be a bit complex, the more pragmatic approach in my opinion, is to create a small library that would make updating the DOM through RJS-like methods simpler. It could also manage caching, expiration and updating of the response on the server side, so that subsequent requests do not have to go out to the network.

This is the approach that Kamishibai intends to persue.

Furthermore Kamishibai even handles the two-pane interface which is demonstrated in this Ember.js guide. This is possible because we have included a lot of “intelligence” in the library, whereas with the simple RJS approach, there is not library where complicated logic is stored for reuse.

More on this later when we open up the code.

Deciding which smartphones to support

Deciding which platforms to support and to what extent we should support them is a complex decision.

On the desktop, the vast majority of problems reside in how we should support Internet Explorer (IE) prior to version 10 (which is a very standards-compliant browser). Both IE8 and IE9 are still widely used but have numerous deviations from the HTML and CSS standards. Many features are not supported, and those that are are often buggy. However, since the problem has persisted for such a long time on browsers that have historically been the most popular, the open-source community has constructed many libraries that fill in the holes. Furthermore, the issues and workaround have been extensively documented on the web. Hence the problems are for the most part contained and controllable.

There are also differences and bugs among the “standards-compliant” browsers. However, in recent years, as HTML5 and CSS3 have stabilized, most of the issues have been fixed. In addition, users tend to use the most recent versions of these “standards-compliant” browsers, meaning that we don’t have to deal with the old buggy versions.

On the smartphone and mobile platforms, the situation is very different. Although most mobile browsers are based on the standards-compliant WebKit rendering engine and aspire to be standards-compliant in their Javascript implementations as well, the reality is that there are still a lot of differences.

On the smartphone side, the issues are almost exclusively on the Android side rather than iOS. By far the major reason why these issues persist is because the Android OS is very often not updated by the manufacturers of the phones. As with the desktop browsers, early smartphone browsers contained numerous bugs or lacked many features. Although these bugs were resolved in future iterations and were included in subsequent Android OS releases, a huge number of Android devices did not receive the fixes. The manufacturers simply did not bother to adapt the new Android version to their devices, or, due to the fact that they skimped on RAM to develop “budget” phones, they could not update them due to insufficient hardware resources.

As a result, there remain a huge number of Android phones on old versions of the OS, and hence on old versions of the browser; versions which contains a lot of old bugs.

Bugs in the Android browser tend not to be as severe as the bugs in Internet Explorer. They mostly reside in the HTML5 and CSS3 implementations, both of which were not stable in the webkit code at the time the old Android browser was forked. Other bugs are in “touch interface” implementation, which also was not in the original webkit code. However, since web development in the HTML5/CSS3 era has evolved to emphasize animations, feedback and interactivity, these bugs are very significant.

A table of Android OS versions per phone

To decide which Android versions to support, we use this table (Android端末一覧) on Wikipedia that lists every smartphone model sold in Japan and which Android version it is upgradable to.

We can observe that almost all smartphones that were introduced up till September, 2011 have only been updated to 2.3.4. Up till April, 2012, many new phones ended up being stuck at 2.3.4-6. Only after April 2012, just a year ago, do we see the majority of phones being upgradable to Android 4.0. Hence support for Android 2.3 is inevitable.

Just for comparison, the current version of iOS, iOS6.1), is installable on even the iPhone 3GS, a model that was first sold in Japan on June 2009. June 2009 is a month before the very first Android phone was sold in Japan and that could only be updated to Android 1.6.

Assessing the current and future popularity of Chrome on Android

In developing websites, especially those like Ponzu/Kamishibai which make heavy use of Javascript and CSS3, it is extremely important to decide which browsers to support. Older browsers will tend to not support the features required to make advanced features on the website run, and so the decision has to be make whether to support that old browser at all.

In Ponzu/Kamishibai, we currently support the following platforms;

  1. Newer versions of Safari, Firefox and Chrome on desktop platforms. The decision to not support older versions is based on statistics that show that users of these browsers tend to update quickly to the newest version.
  2. Internet Explorer 10 on windows.
  3. Internet Explorer 8 and 9 are supported through browser-specific code modifications. More technically, we have separate CSS and Javascript files that are uses only on these platforms to make up for deficiencies. Hence testing tends to be less thorough compared to the more fully supported platforms. Supporting older versions of Internet Explorer is a necessary vice, due to the fact that Windows XP (which only supports up to Internet Explorer 8) is still prevalent, and that IT departments within corporations usually restrict updates.
  4. On iOS, we support the latest version only, with brief testing on older versions. This is due to statistics that show rapid adoption of newer versions of iOS. For example, iOS6 was found installed on 85% of devices after only 5 months. Furthermore, iOS6 can be installed on even the iPhone 3GS, a device released almost 4 years ago.
  5. On Android, we support the Android stock browser on Android version 2.3.6 and Android version 4.0. We also support the latest version of Chrome on Android.

Android fragmentation due to slow adoption of new OS versions

Android support is complicated due to two issues.

One is the fragmentation of the Android platform itself. This is well documented and data can be found on Google’s website. As of May 1, 2013, close to 40% of users are on “Gingerbread” (version 2.3.*), an OS version that was first introduced on December, 2010, and superseded by “Ice Cream Sandwich” (version 4.0) on October, 2011.

The Android stock browser is an integral part of Android and is updated together with the OS itself. Hence Android OS fragmentation directly corresponds to browser version fragmentation.

スクリーンショット 2013 05 13 9 35 30

Android fragmentation due to two different Google browsers

Android browser support is further complicated due to the fact that there exists two separate brands of browsers, both of which are developed by Google and both of which may be found as the default browser on even the newest versions of Android.

Up till Android version 4.0, the default browser on Android was always what is commonly referred to as the “stock Android browser”. However in June, 2012, Google released Chrome for Android. Since then, some but not all devices (e.g. Nexus 7) have Chrome as the default browser and do not have the “stock browser” installed.

Will Chrome become the new default browser for Android?

Because Chrome is developed by Google, the question is whether or not Chrome will be the default browser for Android. More practically, the real question is whether Chrome will become a significant proportion of the web audience.

Unfortunately, current statistics point to that not being the case in near future.

The below graph is taken from NetMarketShare which tracks global website usage. The graph is a breakdown of mobile browser usage on April 2013. Chrome usage (which is a combination of Chrome on Android and iOS) is 2.63% whereas Android Browser usage is 22.89%. If we make the assumption that Chrome usage is 100% Android, this calculates to 10.3% of Android users using Chrome. This compares to 28.4% of Android users on “Jelly Bean” (version 4.1.) and 27.5% on “Ice Cream Sandwich” (version 4.0.).

Chrome for Android cannot be run on versions lower that 4.0.*. The above numbers mean that 10.3 / (28.4 + 27.5) = 18.4% of “Chrome-capable” Android devices are running Chrome. If we restrict to “Jelly Bean”, the Android version from which Google removed the Android stock browser on Nexus devices, 36.2% are using Chrome at maximum (assuming that all Chrome users are on Jelly Bean, a rather unrealistic assumption).

These numbers suggest that Chrome for Android adoption is not particularly high, even among newer devices.

It does not look very likely that Chrome will become the number 1 browser on Android for at least a few more years.

スクリーンショット 2013 05 13 10 18 19

Non-Google branded smartphones do not use Chrome for the default

Whereas Google “Nexus” branded devices like the Nexus 7 come with Chrome as the default browser, and without the stock Android browser installed, the same is not true for devices from Samsung.

In this review for the recently introduced Galaxy S4, which comes with Android version 4.2, the default browser is mentioned to have Samsung-specific features which are not available for Chrome. Hence the stock Android browser is not only installed as the default, it has unique features which Samsung intends to use to differentiate from the competition.

It is therefore likely that Samsung has no plans to switch to Chrome as the default browser. Instead, Samsung will most likely add features to the stock Android browser to further differentiate itself. If Samsung were to switch to Chrome, that would mean the browsing experience would be almost identical to other smartphones. Poor differentiation means commoditization, and Samsung will fight hard to prevent that.

In fact, Chrome on Android is pretty badly implemented at this point, and using Chrome as the default browser is likely to worsen the user experience. As Google improves the Chrome code, this is likely to be less of an issue. However, there is a larger issue that will still stop Samsung from using Chrome aggressively.

Android is open-source, which Chrome is not

The comments in this article about Chrome for Android are mostly detailed and provide a lot of insight. Particularly interesting is the fact that while Android (including the browser) is open-source, Chrome is not. This is discussed in more detail here.

Hence for a manufacturer like Samsung, which has resources to modify Android to create a unique user experience, the Android stock browser is an obvious choice. They simply cannot customize Chrome. It is possible that Samsung will switch from the Android stock browser to their own browser based on webkit, but it is unlikely that they will move to Chrome. The same can be said of HTC and other major players.

For less capable manufacturers, they may chose Chrome, but they also might chose a third party which has a more capable browser or which will agree to customize.

Either way, the current situation is such that the first choice for the default browser has disappeared without anything to fill that gap. Chrome in the current licensing status will not fill in that gap, as Samsung will most likely not chose it. As a result, we might see extreme fragmentation in Android browsers.

Conclusion

Chrome is unlikely to become the dominant browser for Android in the near future. This is true even if we only consider new high-end phones since manufacturers seem reluctant to give up their own customized stock browser in favor of Chrome.

The result is likely to be that we will see even more fragmentation of Android browsers. The players will be customized stock Android browsers (from Samsung, HTC and larger manufacturers), Chrome, Amazon Silk and third party browsers (which will be endorsed by smaller manufacturers which cannot customize the stock browser themselves).

The situation is pretty grim for web developers which want to take advantage of cutting edge HTML5/CSS3 features on Android.

Using local data storage space efficiently

As described in my previous post, “Choosing the browser-side data storage API”, storing data locally is tricky. Despite the work that is being done for data storage in HTML5, there still isn’t a good storage option that allows us to easily scale from a small data set to a large one. Storage limitations differ with each browser and we cannot rely on them to provide us with sufficient space.

Therefore, it is important that we use the space that we have as efficiently as possible.

In Kamishibai, we store HTML fragments as is in local storage. The size of the HTML fragments range from a whole page to a small “like” box. Breaking up a page into multiple HTML fragments is advantageous for effective cache management (each fragment might require a different expiry date) and would promote code reuse. However, multiple HTML fragments mean multiple HTTP requests. Downloading the page as a single fragment is the most efficient in terms of HTTP requests.

This means that we have to strike a compromise. Either we chose to optimize cache management by breaking up into small HTML fragments, or we optimize for network speed with larger HTML fragments.

The choice depends on how many pages we want to store locally, and the level of redundancy in each page. If we want to store a large number of pages, then it will be more efficient to use small fragments. Especially if the level of redundancy is high, we will be able to reuse small fragments effectively. Hence the choice will tend towards smaller fragments in conference systems.

Let’s look at the level of redundancy in conference systems.

The main pages in a Ponzu conference system are;

  1. A list of sessions.
  2. A list of presentations within a session.
  3. A presentation page (with a list of related presentations).
  4. A list of presentations in the search results.

A lot of the pages show a list of presentations. It therefore makes a lot of sense to store each element (title, authors and author affiliations of a presentation) separately so that we can construct different lists simply by combining elements.

Furthermore, the title, authors and author affiliations section of each presentation is quite large. In addition to the text, the authors section is composed of links to user profile pages and we additionally have markup of superscript. The markup is significant, and we often see more than 1,000 characters per presentation for the heading alone (sans abstract text).

In MBSJ2012, we did not break up lists into fragments. In addition to the large size, we also observed that rendering took a lot of time when the number of presentations in a list were large. Rendering the author list required a lot of string manipulation and often triggered garbage collection, resulting in long response times (several hundred milliseconds).

In future versions of Ponzu and Kamishibai, we will break up presentation lists. Each presentation will have a long expiry time so that the version cached inside the browser will be used. Additionally, we will use caching on the server. Our current test show that it should improve responsiveness in most cases.

Choosing the browser-side data storage API

Kamishibai has limited offline support at present, and we plan to bring it to full support within 2013. By full support, we mean that despite Ponzu being a web-page, we want to make the vast majority of pages available even when offline. The reason we take this very seriously is because WiFi at conferences tends to be very sketchy and temperamental.

Choosing the data storage API

Although HTML5 has support for offline applications, it is still very much in its infancy.

The Application Cache API is relatively well supported on all new browsers (> IE9). However, this API, while easy to use for the most basic cases, tends to be difficult for more advanced uses. Most programmers reserve it for static assets.

To store each presentation, session and user profile, we need a cache that is more flexible. Hence, in these cases, we use a data storage API that allows us to create, update and remove records programmatically and at will (the Application Cache API can only update all entries or zero entries). The APIs for this have not been well developed. We have localStorage, webSQL and indexedDB APIs.

localStorage is the simplest to use but also has the least features. The storage capacity is also rather small and no mainstream browser allows more than 5MBytes of storage.

webSQL is basically a sqllite3 database wrapped in Javascript. Since it can use SQL statements, there is a lot that you can do with it. Unfortunately, it has been abandoned by the W3C working-group because it is too tied to the sqlite3 implementation. webSQL can be expanded to 50MBytes of storage even on mobile devices, and is well supported on both iOS and Android. Safari and Chrome desktop browsers also support this API.

indexedDB is the replacement for webSQL. It is currently supported on IE10, Chrome and Firefox. Safari does not support it on both desktop and iOS versions. Also, the Android stock browser does not support this. It is the newest standard, and will most likely become the standard in the future. However, it will take several years before it becomes mainstream enough.

We ended up combining localStorage and webSQL for Kamishibai. Our rationale is discussed below. In the future, we plan to add indexedDB support.

webSQL looks great on features, but falls short on some important details

Having the full power of an SQL database and also broad support on mobile devices, webSQL looked like the prime candidate for Kamishibai data storage. Furthermore, since the 5MByte storage limit for localStorage would most likely be insufficient for large conferences, we initially planned to go 100% webSQL. However, we hit some snags.

First of all, when we use the openDatabase command to create a new webSQL database on Safari, and databases with an initial size exceeding 5MBytes brings up a modal dialog to confirm with the user. This is perfectly OK if the user is already familiar with the Ponzu system. However, a large number of people would be afraid and alarmed. They might opt to not allow access. More seriously, they might be deterred from visiting the site.

Memcached like LRU (least recently used) management

Since localStorage can only store 5MBytes and WebSQL puts up a modal dialog to confirm usage of more than 5MB, we have to consider how we are going to manage with storage that is smaller than the conference scientific program (it will easily exceed 10MBytes).

One idea is to purge entries that are infrequently accessed. A good scheme is the LRU management in memcached. The idea is to detect when the storage quota has been exceeded, and then to delete old entries to make space. Old entries are those that have not been recently accessed.

Implementing an LRU-type scheme is possible in localStorage. localStorage fires a QUOTA_EXCEEDED_ERR when we try to store more than the quota (5MB). We can catch this error and delete old entries, then retry.

However, webSQL at least on Safari is different. A quota exceeded error fires, but only after the modal dialog has shown up. This means that we cannot transparently delete old entries. This makes it very difficult to use webSQL as a LRU cache. Hence webSQL is good if we can get the user to agree to using a large quota (like 50 MB). However, using webSQL within the 5MB limit is very difficult if the user rejects the request.

Final scheme

Our final scheme is to initially use localStorage, using an LRU-based storage scheme. Since we will not be able to store the whole scientific program, this is basically a history-like cache.

We will also provide a store data locally button that the user can click to download the whole scientific program. This will use 50MB of webSQL storage. A modal dialog will pop up, but the user is expected to approve it. If they don’t, then we will fall back to localStorage. If the user uses webSQL, then we have no further need for localStorage, and we work 100% in webSQL.

Why Kamishibai uses hash-bangs for navigation

Kamishiba URLs look like

http://mbsj2012.castle104.com/ja#!_/ja/presentations/3366

We are using hash-bangs which are hated by some. I’ll like to outline why we use them despite their unpopularity.

  1. The most important reason is because we want to allow offline viewing. We thought over many alternative ways to do this, but our conclusion was that we needed a boot loader that we could keep in Application Cache. In the above URL, http://mbsj2012.castle104.com/ja is the bootloader. /ja/presentations/3366 is where the content resides.
  2. The stock Android browser for version 4.0 and above do not support popState. Neither does IE9 and below.

Obviously, the first reason is the most important for conference systems because the network connection is generally very unreliable. The only solution to do offline apps without a hash-bang scheme is outlined in Jake Archibald in A List Apart, and it’s a good solution. However, as Jake himself admits

The experience on a temperamental connection isn’t great. The problem with FALLBACK is that the original connection needs to fail before any falling-back can happen, which could take a while if the connection comes and goes. In this case, Gotcha #1 (files always come from the ApplicationCache) is pretty useful.

Unfortunately, a temperamental connection is exactly what you get at conferences. We need a solution that allows us to programmatically define when to FALLBACK using either a timeout or the results of previous requests.

Below our some of our thoughts about the hash-bang and how we are providing solutions to its ill effects.

  1. We think that the hash-bang is necessary to provide a better experience to users. Offline viewing is the most important but there are other smaller things.
  2. The argument against hash-bangs is mostly about the nature of the web and crawlability. Our solution is to provide a mobile website that uses regular URLs. Crawlers don’t have to come to our sophisticated, Javascript-heavy website. We make sure that the mobile website contains all important information and that URLs for the mobile website are automatically converted to our Kamishibai ones.
  3. Although a hash-bang webpage requires two HTTP requests for the HTML, the first request is for the boot loader and will be stored in Application Cache. Hence the first request is local if the user has visited this website before. As a results, only one HTTP request goes over the network.
  4. Although many people state that the hash-bang is temporary and that widespread popState support will make it unnecessary, I disagree. Hash-bang is the only way I know of that will support offline websites.
  5. GMail still uses hash-bangs for most of their stuff. Obviously, GMail doesn’t want bots crawling their website.

Interchangeability of regular URLs and hash-bang URLs

In Kamishibai, we support both regular URLs and hash-bang URLs. Regular URLs will be used if the browser is mobile (iMode) or unsupported or has Javascript disabled. Because we have two URLs, we have to provide mechanisms to reroute, etc.

From regular URL to Kamishibai URL

This will be done within Rails and will be simple rerouting on the server. 302 redirects generally honor the hash fragment, so everything should be OK.

  1. Rails will receive the regular URL.
  2. If the browser that requested it is supported in Kamishibai, then redirect to the corresponding Kamishibai URL via 302. If the browser is not supported, then it gets a regular HTML response. (this may be implemented with a Rails before filter because not all actions can be converted easily).

From Kamishibai URL to regular URL

iMode is Javascript capable, but in reality, a large number may have Javascript disabled. Other carriers do not have Javascript. We cannot rely on Javascript of mobile sites. However, rudimentary Javascript should be provided for our non-supported browsers.

  1. Rails will receive the Kamishibai URL.
  2. Rails will return the top page, not the bootloader. The top page will have a short Javascript snippet that detects whether the URL was actually a Kamishibai URL or not. If it was a Kamishibai URL, then we automatically redirect to the corresponding regular URL. This will work if Javascript is ON but not if Javascript is OFF.
  3. If Javascript is disabled, then there is nothing we can do.

Summary

All clients with Javascript enabled will be able to handle both Kamishibai and regular URL. If Javascript is disabled, they will be able to handle regular URLs but will only show the top page for Kamishibai URLs.

With regard to Googlebot, we try to make it index only the regular URLs. Requests to regular URLs return full iMode pages (if Googlebot). If Googlebot tries to crawl a Kamishibai URL, which might happen if it followed an external link, then it might get a bit confused but because the Javascript is complicated, we expect it to just find the top page. Finding the hash-bang, it might also try to index using the _escaped_fragment_ scheme. We should redirect any of these requests to the regular URL version (using 301 redirects).

What killed the native smartphone app for MBSJ2012?

As of April, 2012, the plan was to create a native application (with “Like” capability) to support smartphones, and to support PCs and feature phones through the website. The website would also provide an API so that the smartphone application could sync “like” information. This decision was changed in July, 2012. Native applications were ditched and the decision was to support all devices through the website.

Below, I describe our perspective as to why a native application was not (and still probably isn’t) suitable.

Smartphone native applications don’t require “native” functionality

Conference program applications are quite simple. They are basically a hierarchical listing of the presentations with a simple search function and a bookmark function. They hardly require any graphics capabilities. Since they are basically viewing-only, complicated editing and gestures are not necessary.

As such, there is very little need for these applications to be written as native applications. A basic web page can easily provide the same set of functions.

The only benefit of a smartphone native application is working offline

Although both Apple and Android smartphones use WebKit for their browsers and are HTML5 capable, storing data offline is still a sore point. The most widely supported API for offline storage is webStorage, but it is limited to 5MB on most browsers. This limits us to a few hundred presentations. By using webSQL and indexed DB, we should be able to use more than 50MB, which will enable us to include thousands of presentations. As such, storing the whole conference program on the browser for offline viewing is still confusing and difficult to program. Furthermore, developing offline web applications is an area technique which is not yet common, and mature libraries have yet to emerge.

Native applications are much easier. Native applications can directly use sqlite3 or keep all the data in the filesystem. The App Stores do not limit the size of applications, so the number of presentations that can be stored is virtually limitless.

The native app developer considered using UIWebView for the poster map

When we were discussing native application development with the software house in May 2012, they brought up the idea of using a UIWebView for displaying the poster map. This is basically the equivalent of providing a link. All the programing for the poster map will be done on the website, and the native application will simply act as a browser to view the page.

I was stunned. The poster map is the only feature where nice graphics would really make a difference, and the only place where I would consider native applications to have a fundamental advantage. Despite that, the developer was saying that they wanted to use web technologies to do that. They were throwing out the opportunity to show that native apps are awesome. It also meant that the only real advantage of a smartphone application, offline viewing, would not be possible for the poster map. This was unacceptable since the room where WiFi would be most fragile was expected to be the poster hall.

I pressed hard on the developer, basically interrogating them on what they really meant, whether they really thought that the UIWebView solution would be appropriate, and whether they would be able to make it work offline. This discussion was rather confusing, and I began to worry about whether they were really going to create something that met up to our expectations.

This is when I started work on a smartphone version of Ponzu to see if a smartphone version might be a better solution than a native application.

Conclusions from the smartphone version

  1. If the smartphone application used Ajax technologies to keep network access to a minimum, 3G networks were sufficient for smartphone clients. We tested in congested areas in Tokyo such as Shibuya station, and found no performance issues.
  2. iPads with only WiFi connections would still be an issue unless we supported offline viewing.

Current thoughts

  1. If we can fully support offline viewing on the mobile website using HTML5 storage, then we can almost completely neutralize the advantages of a native application.
  2. Having two different parties developing for different devices is dangerous. Communication is difficult and the result is almost sure to be inconsistent features and user interfaces.
  3. An outside developer that isn’t committed to excellence will slow down innovation. An outside developer is OK only when innovation is stalled, and you can show them what they need to copy.

Rethinking REST from a Kamishibai perspective

REST is the way the web was designed to work. Thinking of URLs as resources is a guiding principle that aids design of web applications and brings order to URLs.

Browsers traditionally considered URLs to represent pages that would be shown on a single browser window, with the exception of frames. The Kamishibai approach considers URLs to represent fragments of pages. This is similar to frames.

The problem with a one-to-one correspondence between REST URLs and pages is that pages have a lot of redundancy. A single web page shares a lot of common elements with other pages. It is much more efficient to manage web resources at a fragment level.

When we think of URLs as HTML fragment resources, then we can rethink what the root path stands for. In the fragment model, the root path stands for the boot loader, the initialization of the application. It does not stand for any particular page. Therefore, it is actually more natural to use the hash as the navigation indication. The hash based approach is no longer a hack; it is representative of how the web application works.

Another way of seeing it is to rethink what the hash originally stood for. The hash was a way to tell the browser which portion of the web page to show. We can frame the Kamishibai concept in the same way; In Kamishibai, the hash indicates the ID of the HTML element that we want to display in the front. In some cases, the HTML element will already be loaded, which makes it virtually identical to how regular hashes are handled by default. If the HTML element is empty or has not been loaded, Kamishibai loads that page via AJAX. However, this is invisible from the HREF. The HREF simply tells the browser that it wants to view a certain fragment within the root URL resource. It is Kamishibai’s responsibility to prepare that page and display it with animations.

Therefore, we can think of Kamishibai as actually adhering to REST principles. Since it does not alternate the response based on whether it is an AJAX request or not, in some ways it adheres more closely that PJAX.

Looking at the Kamishibai world from a REST perspective, the root URL which holds the whole application can be thought of as a huge single page that you can navigate through with hash links.