Category Archives: Programming

For us, the largest benefit of Javascript templating is reduced size

There are quite a few Javascript templates. In my projects however, there are very few cases where I would prefer using any of them in place of regular HTML being pushed out from the server (running Ruby-on-Rails). The same can be said of the Ponzu conference system.

As far as I understand, the benefits of using Javascript templates are 1) reducing the load on the server (generating JSON is less load than generating full HTML), 2) speed if used in combination with a single page design.

The downside is the additional work that browsers have to do, which can be a problem on mobile where the devices are not as powerful as their desktop counterparts.

I’ve touched this subject before in these two posts [1](https://code.castle104.com/?p=289), [2](https://code.castle104.com/?p=291).

As discussed by David Heinemeier Hansson, the same benefits can be achieved without Javascript templates by using a PJAX/Turbolinks/Kamishibai like system that eliminates reloading Javascript and CSS on each page transition, and the use of aggressive caching on the server side to reduce the load of HTML generation.

There is a real case however, where I feel a strong need for a Javascript tempting language.

That is when I try to cache responses in browser side cache. The issue is that HTML is extremely verbose, and is a real killer in terms of storage consumption when you are working with repetitive content.

For example, the following is a “social box” that we use in Ponzu for the like button and a voting button. It takes about 2,000 bytes. Each social box is associated with a presentation so we have hundreds to thousands of social boxes for each conference. This can easily fill up the limited browser side cache.

<div class="" id="presentation_326_social_box"><div class='like_box'>
<div class='like' id='like_button_326'>
<span class='social_controls'>
<!-- To invalidate Like related paths, we need a like object -->
<a href="/likes?like%5Bpresentation_id%5D=326" class="button icon like" rel="nofollow">like</a>
<div class='prompt_message'>
(
To add to your schedule, please &quot;like&quot; it first.
)
</div>
</span>
<div class='social_stats'>
<img alt="Like" src="/assets/like-c3719a03fc7b33c23fda846c3ccfb175.png" title="いいね!を押すと、応援メッセージになります。またあなたのスケジュールに登録されます。" />
<a href="/presentations/326/likes?user=1">15 people</a>
liked this.
</div>
<div class='likes_list' id='likes_presentation_326'></div>
</div>

</div>
<div class='vote_box'>
<div class='like' id='vote_button_326'>
<span class='social_controls'>
<form accept-charset="UTF-8" action="/likes/vote" class="new_like" id="presentation_326_new_like" method="post"><div style="margin:0;padding:0"></div>


<span>

<label for="presentation_326_like_score_1">Excellent!</label>
</span>
<span>

<label for="presentation_326_like_score_2">Unique!</label>
</span>
<span>

<label for="presentation_326_like_score_0">No Vote</label>
</span>
</form>
</span>
</div>

</div>
</div><div id="session_details_presentation_326"></div>

Most of this content is repetitive and will be identical for each “social_box”. In fact, the content that is unique to each individual social box can be summarized in the following JSON.

[{ 
    presentation_id:326,
    user_id:1,
    score: 0,
    liked: 0,
    scheduled: 0
}]

If we could use Javascript templating to generate the 2,000 byte HTML from this small set of JSON, local storage savings would be huge.

This is one feature that we will be adding to Kamishibai and Ponzu in the near future, to enable the ultimate goal of complete offline viewing.

On HTTP caching

Kamishibai provides support for storing Ajax responses on the client using either localStorage or WebSQL (and Indexed DB is planned in the future). This enables us to dramatically speed up page loads by not sending the request out to the server, but retrieving the response from internal storage. It also allows us to provide offline access to pages.

HTTP itself provides the HTTP cache protocol which allows the server to control browser cache through the “Cache-Control”, “Expires”, “Last-Modified”, “If-Modified-Since”, “ETag” and “If-None-Match” HTTP headers. Ruby-on-Rails also provides methods that allow us to easily manage these headers.

The question is, why did we create our own caching mechanism for Kamishibai, instead of using HTTP cache. In the following, I hope to provide the answer.

Many Ajax requests are Private Content

The following is an excerpt from the above link describing use-cases from HTTP cache.

Private content (ie. that which can be considered sensitive and subject to security measures) requires even more assessment. Not only do you as the developer need to determine the cacheability of a particular resource, but you also need to consider the impact of having intermediary caches (such as web proxies) caching the files which may be outside of the users control. If in doubt, it is a safe option to not cache these items at all.

Should end-client caching still be desirable you can ask for resources to only be cached privately (i.e only within the end-user’s browser cache):

In Ponzu, our scientific conference information system with social network features, a lot of the content is “Private content”. For example, we generally only show the abstract text to conference participants (non-participants can only view the presentation titles and authors). Hence Ajax requests for presentation detail pages cannot be handled with HTTP cache.

URLs alone are not sufficient as the cache key

HTTP caching uses the URL only as the cache key. If the content changes depending on values in cookies, then HTTP caching doesn’t work.

However with Ponzu, we use cookies to store the current user id and we also store the locale. We display slightly different content depending on the privileges of each user and we also provide different translations. We do all this while keeping the URL the same. Keeping the URL the same is important to maximize social sharing.

Hence in Ponzu, URLs alone are not sufficient as the cache key.

Flexible purging of HTTP cache is not possible

HTTP cache does not allow flexible purging of the cache. For example, if you set “max-age” to a large value (e.g. for a few days), then you cannot touch the cache on the browser until the cache has expired. If you suddenly have an emergency notification that you need to put up, you can’t do it. You have to wait until the cache expires, whatever happens.

With Ponzu, we want to set very long cache expiry dates to maximize fast browsing. On the other hand, we want to be able to flush the cache when an emergency arises. An emergency might be an urgent notification, but it also may be a bug.

Hence HTTP cache is not particularly suitable, and we would not want to set long expiry times with it unless we were extra sure that the content would not change.

Summary

As we can see, HTTP cache is not suitable for the majority of Ajax requests (HTML requests) in Ponzu. Although we use it to serve content in the Ruby-on-Rails asset pipeline, we don’t use it for dynamically created content at all. Instead, we use Kamishibai caching which provides more flexibility.

On JavaScript MVC (part 2)

In Kamishibai and Ponzu, speed is a huge concern. This is particularly true for smartphones. In desktop web sites, it is possible to cram a lot of information and navigation elements into a single page to make up for slow page loading. You can see this in news websites like Asahi.com where 80% of the top page consists of navigation and shortcuts. The idea is that instead of asking the user to click a link and reload a new page (which is slow), the user simply can scroll down to see more content. With smartphones, cramming all this information is simply not a good idea and we have to reload pages.

One way to reduce the load time for pages and to update only the parts that you want is to use Javascript MVC or client-side MVC. With client side MVC, the pages are not reloaded as the content is switched. Instead of sending HTML pages, the server sends JSON data to the browser and browser-side javascript is used to construct the DOM from the JSON data. The advantage is that the client does not have to reload the whole page, and that it can intelligently update only the portions of the DOM that need to be redrawn.

In Kamishibai and Ponzu, we seriously contemplated using these Javascript MVC frameworks. However, we decided not to do so. Instead, our approach is similar to how GitHub handles updates with PJAX, and with how the new Basecamp uses Turbolinks. David Heinemeier Hansson gave a presentation describing why they did not use Javascript MVC extensively and a video is on YouTube.

There are other highly respected programmers who question the use of Javascript MVC. Thomas Fuchs, the author of script.aculo.us and Zepto.js has this to say in his blog “Client-side MVC is not a silver bullet” and his comments on a post about one of his projects, Charm.

I’ve come to the realization that this much client-side processing and decoupling is detrimental to both the speed of development, and application performance (a ton of JavaScript has to be loaded and evaluated each time you fire up the app). It’s better to let the server handle HTML rendering and minimize the use of JavaScript on the client. You can still have fast and highly interactive applications, as the new Basecamp shows—letting the server handle most stuff doesn’t mean that you have to cut back on cool front-end features and user friendliness.

We’ve spend a lot of time getting Backbone to work properly, as the easy-of-use quickly deteriorates when your models get more complex. It’s a great choice for simple stuff, but email is far from simple. We also had to add yet an other extra layer of processing to generate “ViewModels” on the server because the normal Rails serialization of objects wouldn’t cut it.

If you do any non-trivial resources, you’ll quickly end up with JSON objects that are just too large, especially for lists. For emails, imagine that you have nested threads, user avatar images, nested assigned cases, etc.
Because of this, you’ll need specialized JSON objects/arrays for different use cases (search, list view, detail view, and others). It follows that you’ll end up with this with more or less any front-end framework (if you care about performance!). Doing this adds complexity, which can be avoided by rendering HTML on the server where access to arbitrarily deeply nested data is relatively cheap (and can be highly optimized by keeping snippets of HTML around in memcache, etc).

In Ponzu and Kamishibai, we ended up following the traditional Rails route, enhanced with techniques that are seen in PJAX and Turbolinks.

What to we want to achieve

For us, the reasons for contemplating the use of a Javascript MVC framework were;

  1. Speed
  2. Use without network connection (offline apps)

A common benefit for Javascript MVC is interactivity, but this was not a concern for the types of web applications we had in mind which tend to be very read-heavy.

David Heinemeier Hansson has written a detailed post on how the new Basecamp dramatically increased performance through PJAX-like techniques and extensive server-side caching.

This technique basically nullifies the first benefit, because it makes it very easy to increase performance. Furthermore, the code will be almost the same as traditional Rails and hence very simple.

The second issue, offline usage, is something that even most Javascript MVC frameworks do not support very well. This is even more so when the data gets complex. Hence this in itself is not a compelling reason to go Javascript MVC; we would still have to figure out how to do offline usage effectively ourselves.

Why do people use Javascript MVC?

I came across this blog post describing why they used a Javascript MVC framework (Ember.js) instead of simple jQuery.

The author, Robin Ward, gives a specific example.

For example, on the bottom of every discourse post there is a button a user can click to like a post. When clicked, it vanishes and adds a footer below the post saying you liked it. If you implementing this in jQuery, you might add a data-post-id to the post. Then you’d bind a click event on your button element to a function that would make the AJAX call to the server. However, the click function passes a reference to the button, not the post. So you then have to traverse the DOM upwards to find the post the button belongs to and grab the id from there. Once you have it, you can make your XHR request. If the XHR succeeds, you then have to traverse the DOM downward from the post to the footer, and add in the text. At this point it works, but you’ve tied your implementation of the button click to a particular DOM structure. If you ever want to change your HTML around, you might have to adjust all the jQuery methods that accessed it. If this example seems simple – consider that in the footer we offer you a link to undo your like. When clicked, the footer text vanishes and the button appears again. Now you’re implementing the opposite operation against the DOM, only in reverse of what you did before. Discourse even takes it a step further – we know that 99% of the time when you click the like button the request is going to work, so we hide the button and show the footer text right away, even before waiting for the server to reply. In the infrequent event that request fails, we’ll show an error message and pop the UI back to the state it was in before. If we were doing that in jQuery, we’d have to have a callback on our AJAX request that knew how to put the UI back into the state it was in before. A prudent programmer might say, okay, well I’ll have a render function that can rebuild the DOM to the current UI state of whether the post is liked or not. Then both ‘undo’ and ‘like’ can call it. If ‘like’ fails it can call it again. Oh, and we have to store the current state of whether the post is liked somewhere. So maybe we add another data-liked=”true” attribute. ACK! Just typing this all out is giving me a headache!. Congratulations, your code is now spaghetti, your data is strewn out in the DOM and your logic is tied to a particular layout of HTML elements.

Although I understand Robin’s point and I have also experienced frustration when we want to update multiple elements that are in separate locations, I tend to think that using a full-blown Javascript MVC framework is an overkill. No doubt, DHH and Thomas Fuchs would send Javascript in the response to do the complex updates.

In fact, it is pretty difficult to find an Ember.js example on the web that does stuff that is complex enough that simple Javascript would not cut it.

A more intelligent RJS

Given that most of the use-cases of Ember.js and Javascript MVC frameworks seems to be stuff that could be done with regular Javascript, but might be a bit complex, the more pragmatic approach in my opinion, is to create a small library that would make updating the DOM through RJS-like methods simpler. It could also manage caching, expiration and updating of the response on the server side, so that subsequent requests do not have to go out to the network.

This is the approach that Kamishibai intends to persue.

Furthermore Kamishibai even handles the two-pane interface which is demonstrated in this Ember.js guide. This is possible because we have included a lot of “intelligence” in the library, whereas with the simple RJS approach, there is not library where complicated logic is stored for reuse.

More on this later when we open up the code.

On Javascript MVC

David Heinemeier Hansson gave a presentation on Backbone.js or rather how they are not using it too much. The video is on YouTube.

He mentions how he is using PJAX (or actually TurboLinks) and extensive caching on the server side to make the new Basecamp just as responsive as a Javascript MVC application would.

This is very similar to the Kamishibai approach.

There are a few things that I’m wondering about, and this post is a memo about these.

Why are Javascript MVC applications responsive?

Possibilities;

  1. Because all the Javascript and CSS files are not being reloaded and re-parsed.
  2. Because the whole screen is not being redrawn. Only a small portion is being rendered.
  3. The server is much faster at returning JSON responses compared to rendering HTML.

The first possibility is addressed by the PJAX-like approach.

The second possibility is not addressed by PJAX itself because it renders the whole page. Kamishibai on the other hand, can render segments of pages so Kamishibai addresses this.

As for the third possibility, I find it incredible that the server might be slower than the Javascript client at rendering HTML (or the DOM). Of course, Rails caching can solve this problem and all is well, but I wonder how a client, especially a mobile client can be that fast. Maybe the issue is that desktop view templates tend to be too complicated.

Analyzing script evaluation speed of jQuery & jQuery Mobile on the iPhone

In a previous post, I discussed how Javascript can slow down website loading speed significantly, and why Kamishibai and Ponzu aim to keep the size of Javascript small.

Today, I used the Safari web inspector to analyze how long it actually takes to evaluate jQuery Mobile, a common framework used for mobile web development.

For evaluation, we used the jQuery Mobile demo page for version 1.2.1.

Below are the results. We evaluated, in order, my MacBook Air with a 1.7GHz Intel Core i5 (Mac OS X 10.8.3), my iPhone 5 (iOS 6.1.3) and my iPhone 4S (iOS 6.1.3).

The numbers that we are interested in are the times that it takes to evaluate Javascript.

MacBook Air 1.7GHz Intel Core i5 2013 04 01 21 56 jQuery mobile MBA

MacBook Air 1.7GHz Core i5: Time taken to evaluate Javascript

  1. jquery-1.7.1.min.js – 10.7ms
  2. jquery.mobile-1.2.1.js – 31.5ms

iPhone 5 2013 04 01 21 56 jQuery Mobile iPhone 5

iPhone 5: Time taken to evaluate Javascript

  1. jquery-1.7.1.min.js – 41.7ms
  2. jquery.mobile-1.2.1.js – 144ms

iPhone 4s 2013 04 01 21 57 jQuery Mobile iPhone 4s

iPhone 4s: Time taken to evaluate Javascript

  1. jquery-1.7.1.min.js – 65.7ms
  2. jquery.mobile-1.2.1.js – 238ms

Conclusion

The total latency of evaluating jQuery Mobile (jquery.min.js + jquery.mobile.js) was 42.2ms for the MBA, 186.1ms for the iPhone 5, and 303.7ms for the iPhone 4s.

Network latencies for a good broadband WiFi connection are about 50ms. For a 3G connection, they are a few hundred ms.

Server latencies can depends a lot on the complexity of the page you wish to display, but can be anywhere between a few ms to a few seconds. We generally try to keep server latency less than 300ms, and for most pages, less than 100ms.

Whereas latencies of 100ms appear to be almost instantaneous to humans, response times of more that 300ms are very noticeable and negatively impact perceived responsiveness.

Given these numbers, and also considering that the iPhone 4s is by no means a slow phone, we conclude that jQuery mobile is still too slow. For the evaluation time of jQuery Mobile to get to the point that it doesn’t matter, we need at least twice the speed of the current iPhone 5. Given the performance increase with each iPhone model, we might see sufficient speed with the 2013 iPhone model. However, for such performance to become mainstream, we will probably have to wait about 3 years, especially considering that Android Javascript performance tends to significantly lag behind mobile Safari.

Why Kamishibai uses hash-bangs for navigation

Kamishiba URLs look like

http://mbsj2012.castle104.com/ja#!_/ja/presentations/3366

We are using hash-bangs which are hated by some. I’ll like to outline why we use them despite their unpopularity.

  1. The most important reason is because we want to allow offline viewing. We thought over many alternative ways to do this, but our conclusion was that we needed a boot loader that we could keep in Application Cache. In the above URL, http://mbsj2012.castle104.com/ja is the bootloader. /ja/presentations/3366 is where the content resides.
  2. The stock Android browser for version 4.0 and above do not support popState. Neither does IE9 and below.

Obviously, the first reason is the most important for conference systems because the network connection is generally very unreliable. The only solution to do offline apps without a hash-bang scheme is outlined in Jake Archibald in A List Apart, and it’s a good solution. However, as Jake himself admits

The experience on a temperamental connection isn’t great. The problem with FALLBACK is that the original connection needs to fail before any falling-back can happen, which could take a while if the connection comes and goes. In this case, Gotcha #1 (files always come from the ApplicationCache) is pretty useful.

Unfortunately, a temperamental connection is exactly what you get at conferences. We need a solution that allows us to programmatically define when to FALLBACK using either a timeout or the results of previous requests.

Below our some of our thoughts about the hash-bang and how we are providing solutions to its ill effects.

  1. We think that the hash-bang is necessary to provide a better experience to users. Offline viewing is the most important but there are other smaller things.
  2. The argument against hash-bangs is mostly about the nature of the web and crawlability. Our solution is to provide a mobile website that uses regular URLs. Crawlers don’t have to come to our sophisticated, Javascript-heavy website. We make sure that the mobile website contains all important information and that URLs for the mobile website are automatically converted to our Kamishibai ones.
  3. Although a hash-bang webpage requires two HTTP requests for the HTML, the first request is for the boot loader and will be stored in Application Cache. Hence the first request is local if the user has visited this website before. As a results, only one HTTP request goes over the network.
  4. Although many people state that the hash-bang is temporary and that widespread popState support will make it unnecessary, I disagree. Hash-bang is the only way I know of that will support offline websites.
  5. GMail still uses hash-bangs for most of their stuff. Obviously, GMail doesn’t want bots crawling their website.

Interchangeability of regular URLs and hash-bang URLs

In Kamishibai, we support both regular URLs and hash-bang URLs. Regular URLs will be used if the browser is mobile (iMode) or unsupported or has Javascript disabled. Because we have two URLs, we have to provide mechanisms to reroute, etc.

From regular URL to Kamishibai URL

This will be done within Rails and will be simple rerouting on the server. 302 redirects generally honor the hash fragment, so everything should be OK.

  1. Rails will receive the regular URL.
  2. If the browser that requested it is supported in Kamishibai, then redirect to the corresponding Kamishibai URL via 302. If the browser is not supported, then it gets a regular HTML response. (this may be implemented with a Rails before filter because not all actions can be converted easily).

From Kamishibai URL to regular URL

iMode is Javascript capable, but in reality, a large number may have Javascript disabled. Other carriers do not have Javascript. We cannot rely on Javascript of mobile sites. However, rudimentary Javascript should be provided for our non-supported browsers.

  1. Rails will receive the Kamishibai URL.
  2. Rails will return the top page, not the bootloader. The top page will have a short Javascript snippet that detects whether the URL was actually a Kamishibai URL or not. If it was a Kamishibai URL, then we automatically redirect to the corresponding regular URL. This will work if Javascript is ON but not if Javascript is OFF.
  3. If Javascript is disabled, then there is nothing we can do.

Summary

All clients with Javascript enabled will be able to handle both Kamishibai and regular URL. If Javascript is disabled, they will be able to handle regular URLs but will only show the top page for Kamishibai URLs.

With regard to Googlebot, we try to make it index only the regular URLs. Requests to regular URLs return full iMode pages (if Googlebot). If Googlebot tries to crawl a Kamishibai URL, which might happen if it followed an external link, then it might get a bit confused but because the Javascript is complicated, we expect it to just find the top page. Finding the hash-bang, it might also try to index using the _escaped_fragment_ scheme. We should redirect any of these requests to the regular URL version (using 301 redirects).

Supporting Android Tablets

In MBSJ2012, we directed all Android devices to the smartphone website, regardless of actual screen size. The rationale was as follows;

  1. The number of 10-inch Android tablets would probably be very low.
  2. 7-inch Android tablets would probably have a hard time displaying the PC website.

In this article, I would like to explore the possibility of doing something more optimal for Android tablets and come to a conclusion.

What do the device manufacturers recommend

Google issued a document on the Android developers’ website regarding the User-Agent string on Chrome for Android. Their recommendation is;

If you are parsing user agent strings using regular expressions, the following can be used to check against Chrome for Android phones and Android tablets:

  • Phone pattern: ‘Android’ + ‘Chrome/[.0-9]* Mobile’
  • Tablet pattern: ‘Android’ + ‘Chrome/[.0-9]* (?!Mobile)’

So basically, if we use the word Mobile, then it’s a smartphone; if we don’t use the word Mobile, then it’s a tablet.

Now let’s see how the device manufacturers set the user agent strings for their default browsers (it may or may not be Chrome, but we’ll see how far the Mobile detection scheme works.

ZYTRAX.COM: Handset Detection.com

Galaxy SII LTE: Mozilla/5.0 (Linux; U; Android 4.0.4; en-ca; SGH-I757M Build/IMM76D) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30

Galaxy Note Android 2.3: Mozilla/5.0 (Linux; U; Android 2.3.5; en-gb; GT-I9220 Build/GINGERBREAD) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1

Galaxy Tab 10.1: Mozilla/5.0 (Linux; U; Android 3.0.1; de-de; GT-P7100 Build/HRI83) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 MobileSafari/534.13

Galaxy Nexus: Mozilla/5.0 (Linux; U; Android 4.0.2; en-us; Galaxy Nexus Build/ICL53F) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30

Google Nexus 7 Android browser: Mozilla/5.0 (Linux; U; Android 4.2; en-us; Nexus 7 Build/JOP40C) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30

Google Nexus 7 Android browser: Mozilla/5.0 (Linux; U; Android 4.2; en-us; Nexus 7 Build/JOP40C) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Safari/534.30

Amazon Kindle Fire: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-us; Silk/1.1.0-84) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16 Silk-Accelerated=true

Amazon Kindle Fire: Mozilla/5.0 (Linux; U; Android 4.0.3; fr-fr; Amazon Kindle Fire Build/MR1) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30

Android browsers can switch their user agent strings from Mobile to desktop (tablet) type, which complicates things.

What we can see here is;

  1. Detecting Mobile is generally sufficient to detect smartphones.
  2. Detecting Mobile in tablets allows users to switch between smartphone and desktop layouts, although most users probably wouldn’t notice that setting.
  3. The Kindle Fire Silk browser mostly uses Mobile whereas the Nexus 7 does not.
  4. The 7-inch tablet category is confusing, and manufacturers seem to feel the same.
  5. The user-agent string treats both 7-inch and 10-inch tablets the same. You cannot distinguish between them.

viewport device-width issues

In Ponzu, we currently use viewport device-width. viewport device-width is a meta tag that Apple introduced with the iPhone that allows fluid websites to adjust best to the screen of the current device.

Without viewport device-width, when an iOS device encountered a fluid website (non-fixed width), it would set the width of the current screen to 980px (body{width:100%} would be the same as body{width:980px}). The display would be zoomed out because neither an iPhone nor an iPad has 980px width. With viewport device-width, the it would be body{width:320px} on the iPhone and body{width:768px} on the iPad, which ensures that the default 16px text size would be legible without zooming.

Therefore, to use the Ponzu same desktop web design on both iPad and PCs, we need to design our fluid layout so that it works with width: 768px. This is possible because 768px, although pretty narrow compared to current day standards, is still wide enough. In 2007, it was still considered a good idea to design web sites for 800px width. Although screens have gotten wider since, 800px is still very workable. 768px isn’t significantly different from 800px.

However, with Nexus 7 and Kindle Fire tablets, viewport device-width is set to 600px. Although this ensures that text is easy to read, it makes it much more difficult to share your website with a desktop design. If we were to use a fluid layout that works well at width:600px, then when people view that same website on a PC with a 1000px width browser window, there will be too much whitespace as it is streched. A single design for width:600px and width:1000px simply isn’t realistic.

Is responsive web a solution?

Responsive design is mainly about using CSS to flexibly adjust to screen sizes. I generally think that responsive design is a bad idea and many of my feelings are shown in this blog article, and this.

In Ponzu, our stance is that creating separate websites is actually easier than complex responsive design techniques. The end result is, of course, better. However, we can’t create too many separate sites, and creating a separate site for 7-inch tablets is out of the question. We have to either fit 7-inch tablets into the smartphone site or the PC site.

Responsive design at or below 600px width

Just to see if responsive design for a Nexus 7 or Kindle Fire is worth it, we looked at responsive design websites at or around 600px. (from 25 Beautiful Responsive Web Design Examples for Inspiration, 11 gorgeous examples of responsive design, Media Queries )

Maryland Craft Beer Festival

Below 700px width, it becomes a single 300px column design.

Daniel Vane

Changes to iPad-optimized at 768px. Images simply shrink until 500px. Single column at 460px.

Formfett

Change to 7-inch tablet version at 670px, which goes down to smartphones.

Humaan

Changes to iPad-optimized at 780px. Single column at 760px down to smartphones.

Andersson-Wise Architects

Changes to iPad-optimized at 780px. Single column at 600px.

Boston Globe

PC is 3-column, iPad is 2-column, 1-column below 640px.

In almost all of the sites that we examined, responsive designs targeted for 7-inch tablets were identical to the smartphone versions (except in image sizes). No layout changes were made specifically for 7-inch tablets and they all adopted the smartphone layout, not the PC layout. On the other hand, iPad layouts were consistently different from smartphone ones. We can safely conclude that even designers versed in responsive techniques simply use the same smartphone layout for 7-inch tablets. To answer the question at the beginning of this section, they don’t think that responsive design for 7-inch tablets is worth it.

As an outlier, FoodSense provides optimized designs for 7-inches.

Our solution

We think that the rational solution is to direct Android 7-inch tablets to the smartphone website. This might not be an ideal user experience for these users, but we think that this is a compromise that has to be made. Redesigning the PC site to accommodate 600px width screens would result in a degradation of user interface for PC users, which constitute 100% of our users. Responsive web techniques are unlikely to provide an effective solution. Since 7-inch tablet users are less than 10% of our users (excluding iPad-mini), and it is absurd to prioritize them above our PC users.

However, Android 7-inch tablets have a user-agent string that is indistinguishable from 10-inch tablets. If we simply use user-agent string detection, then 10-inch tablets will also be directed to the smartphone site. We recognize that this is hardly optimal and that we can technically overcome this using Javascript, cookies, etc. However, the extremely small market share of 10-inch Android tablets means that the technical effort might not be worth it.

Conclusion

Our conclusion is that we should direct all Android devices to the smartphone website, regardless of device screen size. This is the same as our original decision for MBSJ2012. We should also be careful about the Kindle Fire because it does not always report that it is in fact Android.

The essence of our decision is that the Nexus 7 and Kindle Fire tablets cannot accommodate our PC website due to width limitations. Also, redesigning our PC website to fit inside the 600px width for these 7-inch tablets is not worth the effort, and may not be realistic anyway.

As for the iPad mini, it fits 768px width websites, but the font size will be 19% smaller. Since there is no way to distinguish an iPad mini from a regular iPad, the solution is to make sure that regular text is large enough, larger than 16px (the default iPad browser text size). If you are going to have significantly smaller text, make sure that it’s zoomable (which is often not the case when you use mobile javascript frameworks).

Rethinking REST from a Kamishibai perspective

REST is the way the web was designed to work. Thinking of URLs as resources is a guiding principle that aids design of web applications and brings order to URLs.

Browsers traditionally considered URLs to represent pages that would be shown on a single browser window, with the exception of frames. The Kamishibai approach considers URLs to represent fragments of pages. This is similar to frames.

The problem with a one-to-one correspondence between REST URLs and pages is that pages have a lot of redundancy. A single web page shares a lot of common elements with other pages. It is much more efficient to manage web resources at a fragment level.

When we think of URLs as HTML fragment resources, then we can rethink what the root path stands for. In the fragment model, the root path stands for the boot loader, the initialization of the application. It does not stand for any particular page. Therefore, it is actually more natural to use the hash as the navigation indication. The hash based approach is no longer a hack; it is representative of how the web application works.

Another way of seeing it is to rethink what the hash originally stood for. The hash was a way to tell the browser which portion of the web page to show. We can frame the Kamishibai concept in the same way; In Kamishibai, the hash indicates the ID of the HTML element that we want to display in the front. In some cases, the HTML element will already be loaded, which makes it virtually identical to how regular hashes are handled by default. If the HTML element is empty or has not been loaded, Kamishibai loads that page via AJAX. However, this is invisible from the HREF. The HREF simply tells the browser that it wants to view a certain fragment within the root URL resource. It is Kamishibai’s responsibility to prepare that page and display it with animations.

Therefore, we can think of Kamishibai as actually adhering to REST principles. Since it does not alternate the response based on whether it is an AJAX request or not, in some ways it adheres more closely that PJAX.

Looking at the Kamishibai world from a REST perspective, the root URL which holds the whole application can be thought of as a huge single page that you can navigate through with hash links.

Thoughts on PJAX

Before starting work on Kamishibai, I was working with PJAX. PJAX was a nice and clean idea for implementing the HTML5 History API. The promises of PJAX was speeding up of web loading times by virtue of AJAX partial page loading, while at the same time, maintaining standard URLs and being web crawler friendly.

There were several problems with PJAX that made it difficult to work with and suboptimal for a flaky network environment.

For situations where the AJAX container changed between pages, PJAX simply reverted to re-issuing a full page load. This happened often when going backwards. For example, if we were moving forwards towards a page that changed in only a small fragment. Then we would set the AJAX container to the small fragment and this is all that is entered into the History state object. However, when we move back, then that would mean that we have to redraw more than the just the small fragment; we have to redraw the stuff surrounding it. PJAX simply balked in these situations and reverted to a full page load.

It’s great that with PJAX, there is always a way to simply reload the page and get a regular response that will always work. The issue is that it reverts to this too easily.

Also, this means that PJAX will send out two different types of requests depending on the situation it finds itself in. This makes it complicated and wasteful if we want to store the responses in localStorage. For example, if we do a reload of a deep-linked page, we would receive the whole HTML page. The question is, do we want to store all of it in localStorage under the deep-URL? When we request the same URL via PJAX, should we actually use the HTML body in localStorage? What if the case is reversed and we have an HTML fragment in localStorage, but the browser has requested a full reload. What should we show?

I was also unhappy with the way PJAX tended to waste the History memory. PJAX stores a copy of each AJAX body in the History state object. The problem is that this information can only be retrieved when going back. We can’t use this cache to load pages going forward and we have no way to check whether a certain fragment has already been stored. Given the low speed and reliability of mobile networks and the limited RAM of mobile devices, it doesn’t make sense to be so conservative with the use of precious memory.

Another factor is the lack of support of History API in the Android 4.0 and 4.1 stock browsers. Of course there is Chrome on these platforms, but it doesn’t make sense. The lack of History API on Internet Explorer up to 9.0 doesn’t help either, since it means that half of the browsers in the conference room will be gobbling up unnecessary amounts of bandwidth.

Since PJAX takes a one-page-at-a-time approach to the idea of URL resources, it is very compatible with a pure HTML document model, but is not really efficient for cases where bandwidth is limited. The issue that I had is with the use of the application cache manifest. The application cache manifest work with a full HTML document as the smallest unit (fragment). It does not work with Ajax, and requires a full HTML page reload. Therefore it doesn’t work at all with PJAX unless we cache all entry URLs. On the other hand, if we take a hash-based navigation approach, we will only need a single resource URL in the cache manifest. This URL will return the HTML header and body responsible for the “chrome” of all pages. To put it another way, if we wish to use the cache manifest with PJAX, the only way to do it would be to have entries for every single page that could be an entry-page. This would be extremely wasteful, and even if we could do this, we wouldn’t be able to navigate between these pages with AJAX since AJAX wouldn’t access the cached pages.

In summary, we decided against PJAX due to issues in the current implementation. It is also very likely that we will not use PJAX even if the library improves and browser support becomes more ubiquitous. This is because we feel that PJAX cannot use the cache manifest efficiently.

What is Kamishibai?

Below I describe the design goals of Kamishibai.

Kamishibai is a compositior

Kamishibai is a simple compositor, similar to the Quartz Compositor in Mac OS X. What this means can be illustrated by what Kamishibai does, and what it does not.

What Kamishibai does not.

  1. Kamishibai does not generate HTML from data through the use of templates.
  2. Kamishibai does not contain any business logic in the form of “Models”.
  3. Kamishibai is not a Javascript MVC framework.
  4. Kamishibai will never target native application development through phoneGap, etc. That’s not what Kamishibai is for.

What Kamishibai does.

  1. Kamishibai requests HTML fragments from the server and stores them in a local cache.
  2. Kamishibai replaces portions of the current DOM with HTML fragments from the server and displays them.
  3. Kamishibai intelligently places HTML fragments from the server into appropriate locations in the DOM based on attributes on the fragments.
  4. Kamishibai combines multiple fragments to generate a single DOM.
  5. Kamishibai local cache stores HTML fragments and also expires them.
  6. Kamishibai intelligently examines current network stability and uses the cached fragment or requests a new one from the server accordingly.
  7. Kamishibai interprets the URL and makes suitable Ajax requests to generate the page.
  8. Kamishibai intelligently discovers and fragments that are currently missing and requests them from the server.
  9. We think that explicit syncing sucks. Kamishibai will make syncing almost invisible to the end user.
  10. Kamishibai works offline.
  11. Kamishibai does not enforce a specific UI design and does not come with widgets. Designers are expected to be capable of coding their own widgets with HTML and CSS. This makes design extremely flexible so you can easily brand you website.

Why isn’t Kamishibai Javascript MVC?

  1. Ruby and Ruby-on-Rails are superb. There’s a huge amount of functionality that you will miss with current Javascript MVC frameworks.
  2. Javascript MVC frameworks typically require a lot of Javascript on the browser. This will make first load times for mobile devices extremely slow (seconds for Javascript MVC vs tenths of seconds for Kamishibai).
  3. Rendering HTML on mobile devices will always be slower than on powerful server hardware. Furthermore, pages rendered on the server will be cached for later use, making it even faster.
  4. If the server is slow, you can simply add more hardware. You can’t do that with client side libraries. You are always limited by whatever cheap and slow hardware the customer choses.
  5. Javascript MVC requires a steep learning curve. On the other hand, Kamishibai only requires a few extensions onto HTML, CSS. You don’t even need to learn Javascript to get started. You can easily convert preexisting websites, and even WordPress sites.