Category Archives: Kamishibai

For us, the largest benefit of Javascript templating is reduced size

There are quite a few Javascript templates. In my projects however, there are very few cases where I would prefer using any of them in place of regular HTML being pushed out from the server (running Ruby-on-Rails). The same can be said of the Ponzu conference system.

As far as I understand, the benefits of using Javascript templates are 1) reducing the load on the server (generating JSON is less load than generating full HTML), 2) speed if used in combination with a single page design.

The downside is the additional work that browsers have to do, which can be a problem on mobile where the devices are not as powerful as their desktop counterparts.

I’ve touched this subject before in these two posts [1](https://code.castle104.com/?p=289), [2](https://code.castle104.com/?p=291).

As discussed by David Heinemeier Hansson, the same benefits can be achieved without Javascript templates by using a PJAX/Turbolinks/Kamishibai like system that eliminates reloading Javascript and CSS on each page transition, and the use of aggressive caching on the server side to reduce the load of HTML generation.

There is a real case however, where I feel a strong need for a Javascript tempting language.

That is when I try to cache responses in browser side cache. The issue is that HTML is extremely verbose, and is a real killer in terms of storage consumption when you are working with repetitive content.

For example, the following is a “social box” that we use in Ponzu for the like button and a voting button. It takes about 2,000 bytes. Each social box is associated with a presentation so we have hundreds to thousands of social boxes for each conference. This can easily fill up the limited browser side cache.

<div class="" id="presentation_326_social_box"><div class='like_box'>
<div class='like' id='like_button_326'>
<span class='social_controls'>
<!-- To invalidate Like related paths, we need a like object -->
<a href="/likes?like%5Bpresentation_id%5D=326" class="button icon like" rel="nofollow">like</a>
<div class='prompt_message'>
(
To add to your schedule, please &quot;like&quot; it first.
)
</div>
</span>
<div class='social_stats'>
<img alt="Like" src="/assets/like-c3719a03fc7b33c23fda846c3ccfb175.png" title="いいね!を押すと、応援メッセージになります。またあなたのスケジュールに登録されます。" />
<a href="/presentations/326/likes?user=1">15 people</a>
liked this.
</div>
<div class='likes_list' id='likes_presentation_326'></div>
</div>

</div>
<div class='vote_box'>
<div class='like' id='vote_button_326'>
<span class='social_controls'>
<form accept-charset="UTF-8" action="/likes/vote" class="new_like" id="presentation_326_new_like" method="post"><div style="margin:0;padding:0"></div>


<span>

<label for="presentation_326_like_score_1">Excellent!</label>
</span>
<span>

<label for="presentation_326_like_score_2">Unique!</label>
</span>
<span>

<label for="presentation_326_like_score_0">No Vote</label>
</span>
</form>
</span>
</div>

</div>
</div><div id="session_details_presentation_326"></div>

Most of this content is repetitive and will be identical for each “social_box”. In fact, the content that is unique to each individual social box can be summarized in the following JSON.

[{ 
    presentation_id:326,
    user_id:1,
    score: 0,
    liked: 0,
    scheduled: 0
}]

If we could use Javascript templating to generate the 2,000 byte HTML from this small set of JSON, local storage savings would be huge.

This is one feature that we will be adding to Kamishibai and Ponzu in the near future, to enable the ultimate goal of complete offline viewing.

On HTTP caching

Kamishibai provides support for storing Ajax responses on the client using either localStorage or WebSQL (and Indexed DB is planned in the future). This enables us to dramatically speed up page loads by not sending the request out to the server, but retrieving the response from internal storage. It also allows us to provide offline access to pages.

HTTP itself provides the HTTP cache protocol which allows the server to control browser cache through the “Cache-Control”, “Expires”, “Last-Modified”, “If-Modified-Since”, “ETag” and “If-None-Match” HTTP headers. Ruby-on-Rails also provides methods that allow us to easily manage these headers.

The question is, why did we create our own caching mechanism for Kamishibai, instead of using HTTP cache. In the following, I hope to provide the answer.

Many Ajax requests are Private Content

The following is an excerpt from the above link describing use-cases from HTTP cache.

Private content (ie. that which can be considered sensitive and subject to security measures) requires even more assessment. Not only do you as the developer need to determine the cacheability of a particular resource, but you also need to consider the impact of having intermediary caches (such as web proxies) caching the files which may be outside of the users control. If in doubt, it is a safe option to not cache these items at all.

Should end-client caching still be desirable you can ask for resources to only be cached privately (i.e only within the end-user’s browser cache):

In Ponzu, our scientific conference information system with social network features, a lot of the content is “Private content”. For example, we generally only show the abstract text to conference participants (non-participants can only view the presentation titles and authors). Hence Ajax requests for presentation detail pages cannot be handled with HTTP cache.

URLs alone are not sufficient as the cache key

HTTP caching uses the URL only as the cache key. If the content changes depending on values in cookies, then HTTP caching doesn’t work.

However with Ponzu, we use cookies to store the current user id and we also store the locale. We display slightly different content depending on the privileges of each user and we also provide different translations. We do all this while keeping the URL the same. Keeping the URL the same is important to maximize social sharing.

Hence in Ponzu, URLs alone are not sufficient as the cache key.

Flexible purging of HTTP cache is not possible

HTTP cache does not allow flexible purging of the cache. For example, if you set “max-age” to a large value (e.g. for a few days), then you cannot touch the cache on the browser until the cache has expired. If you suddenly have an emergency notification that you need to put up, you can’t do it. You have to wait until the cache expires, whatever happens.

With Ponzu, we want to set very long cache expiry dates to maximize fast browsing. On the other hand, we want to be able to flush the cache when an emergency arises. An emergency might be an urgent notification, but it also may be a bug.

Hence HTTP cache is not particularly suitable, and we would not want to set long expiry times with it unless we were extra sure that the content would not change.

Summary

As we can see, HTTP cache is not suitable for the majority of Ajax requests (HTML requests) in Ponzu. Although we use it to serve content in the Ruby-on-Rails asset pipeline, we don’t use it for dynamically created content at all. Instead, we use Kamishibai caching which provides more flexibility.

Rethinking REST from a Kamishibai perspective

REST is the way the web was designed to work. Thinking of URLs as resources is a guiding principle that aids design of web applications and brings order to URLs.

Browsers traditionally considered URLs to represent pages that would be shown on a single browser window, with the exception of frames. The Kamishibai approach considers URLs to represent fragments of pages. This is similar to frames.

The problem with a one-to-one correspondence between REST URLs and pages is that pages have a lot of redundancy. A single web page shares a lot of common elements with other pages. It is much more efficient to manage web resources at a fragment level.

When we think of URLs as HTML fragment resources, then we can rethink what the root path stands for. In the fragment model, the root path stands for the boot loader, the initialization of the application. It does not stand for any particular page. Therefore, it is actually more natural to use the hash as the navigation indication. The hash based approach is no longer a hack; it is representative of how the web application works.

Another way of seeing it is to rethink what the hash originally stood for. The hash was a way to tell the browser which portion of the web page to show. We can frame the Kamishibai concept in the same way; In Kamishibai, the hash indicates the ID of the HTML element that we want to display in the front. In some cases, the HTML element will already be loaded, which makes it virtually identical to how regular hashes are handled by default. If the HTML element is empty or has not been loaded, Kamishibai loads that page via AJAX. However, this is invisible from the HREF. The HREF simply tells the browser that it wants to view a certain fragment within the root URL resource. It is Kamishibai’s responsibility to prepare that page and display it with animations.

Therefore, we can think of Kamishibai as actually adhering to REST principles. Since it does not alternate the response based on whether it is an AJAX request or not, in some ways it adheres more closely that PJAX.

Looking at the Kamishibai world from a REST perspective, the root URL which holds the whole application can be thought of as a huge single page that you can navigate through with hash links.

What is Kamishibai?

Below I describe the design goals of Kamishibai.

Kamishibai is a compositior

Kamishibai is a simple compositor, similar to the Quartz Compositor in Mac OS X. What this means can be illustrated by what Kamishibai does, and what it does not.

What Kamishibai does not.

  1. Kamishibai does not generate HTML from data through the use of templates.
  2. Kamishibai does not contain any business logic in the form of “Models”.
  3. Kamishibai is not a Javascript MVC framework.
  4. Kamishibai will never target native application development through phoneGap, etc. That’s not what Kamishibai is for.

What Kamishibai does.

  1. Kamishibai requests HTML fragments from the server and stores them in a local cache.
  2. Kamishibai replaces portions of the current DOM with HTML fragments from the server and displays them.
  3. Kamishibai intelligently places HTML fragments from the server into appropriate locations in the DOM based on attributes on the fragments.
  4. Kamishibai combines multiple fragments to generate a single DOM.
  5. Kamishibai local cache stores HTML fragments and also expires them.
  6. Kamishibai intelligently examines current network stability and uses the cached fragment or requests a new one from the server accordingly.
  7. Kamishibai interprets the URL and makes suitable Ajax requests to generate the page.
  8. Kamishibai intelligently discovers and fragments that are currently missing and requests them from the server.
  9. We think that explicit syncing sucks. Kamishibai will make syncing almost invisible to the end user.
  10. Kamishibai works offline.
  11. Kamishibai does not enforce a specific UI design and does not come with widgets. Designers are expected to be capable of coding their own widgets with HTML and CSS. This makes design extremely flexible so you can easily brand you website.

Why isn’t Kamishibai Javascript MVC?

  1. Ruby and Ruby-on-Rails are superb. There’s a huge amount of functionality that you will miss with current Javascript MVC frameworks.
  2. Javascript MVC frameworks typically require a lot of Javascript on the browser. This will make first load times for mobile devices extremely slow (seconds for Javascript MVC vs tenths of seconds for Kamishibai).
  3. Rendering HTML on mobile devices will always be slower than on powerful server hardware. Furthermore, pages rendered on the server will be cached for later use, making it even faster.
  4. If the server is slow, you can simply add more hardware. You can’t do that with client side libraries. You are always limited by whatever cheap and slow hardware the customer choses.
  5. Javascript MVC requires a steep learning curve. On the other hand, Kamishibai only requires a few extensions onto HTML, CSS. You don’t even need to learn Javascript to get started. You can easily convert preexisting websites, and even WordPress sites.