XTech 2005: XML, the Web and beyond.

Markup-based SmartPhone User Interface using the Web Browser Engine

Discuss this paper on the XTech wiki
View XML source for this paper

Keywords

Abstract

ACCESS Co., Ltd. is a leading developer of web browsers for mobile and embedded devices. Its flagship browser, NetFront, is deployed on over 140 million individual devices and 300 separate products. In this paper, we discuss the extension of the NetFront Browser to serve as the root user interface for the mobile handset device. We call this architecture NetFront Dynamic Menu, or NFDM.

The last several years have seen great advancement in the sophistication of web browsing for mobile devices. Now, the typical mobile browser is capable of rendering sophisticated markups such as XHTML, SVG, SMIL, and scripting such as DOM and ECMAScript. It requires only small extensions to use this rendering capability to implement the entire device UI.

This has several advantages. Since the UI is built in markup, it is easily authored and customized. Since it is implemented in the web browser, it is easily updated over-the-air. And last, since it uses the existing web browser code base, little additional code or development effort is necessary.

This paper gives a background of mobile web browsing leading up to the use of the browser for the user interface. It compares this approach with other strategies to implement UI in markup, presents implementation details for NFDM, and presents lessons learned from our implementation.

Introduction

Cellular telephone devices have become amazingly popular over the last few years. Almost 700 million million handsets were sold in 2004 aloneGART. An increasing percentage of these devices now have data capabilities such as messaging, video, and web browsing. This segment for so-called smartphone devices is growing at an incredible pace with 2004 showing year-over-year increases of 86%IDC.

These smartphone devices will perhaps give rise to the next great shift in computing paradigm. Just as mainframes gave way to minicomputers, minicomputers gave way to desktop PCs, soon smartphone devices will drastically outnumber PCs. Smartphones are in every sense a first class computing device.

Memory sizes are routinely at 128 or 256 MB, displays are full color QVGA and greater, and removeable memory cards make possible 4GB of non-volatile storage. Advanced 3G networks such as wCDMA provide data speeds up to 384 kbps that surpass common dial-up access speeds used in the early days of the internet.

Likewise, smartphone OSes now have advanced features similar to desktop PCs. Unlike their cellphone predecessors, these OSes are open for the user to install new and custom software applications. Microsoft Smartphone®, Symbian OS, Palm OS®, BREW®, and even Linux OS are competing to be the OS of choice for tomorrow’s smartphone. These OSes all offer desktop-like capabilities such as multi-processes, dynamic linking, memory protection and MMU support. These capabilities were not generally available in the previous generation cellphone’s embedded operating system.

These devices are truly powerful computers. Yet, used in the manner of a cellphone, they are routinely carried with one individual everywhere he goes. Used primarily for voice communication, the device has become an indispensable part of daily life. Soon, capabilities such as advanced web browsing will enable access to a vast store of information on the internet to make the data services equally indispensable. With this new class of computing device, there is a need for a new class of user interface.

This new type of user interface must be easily customizable to fit the taste of an individual user. Ideally it should take advantage of the rich text, graphics, video, or images users have come to expect on today’s internet. It should be updateable over-the-air to allow for updates of the latest information, or even to entirely change the user interface. This technology should be low cost with minimal difficulty for implementation.

In this paper, we propose the idea for using the web browser to render the entire device user interface. We first discuss the trends in web browsing for mobile devices, next we propose the technology called Dynamic Menu to enable this type of dynamic UI, and lastly we discuss our specific implementation, NetFront Dynamic Menu (NFDM).

Trends in Mobile Web Browsing

In this section, we identify three steps in the evolution of mobile web browsing. As this evolution continues, we propose the fourth step will be implementation of the UI in the web browser.

Unified Web

Initially, the desktop web and mobile web used incompatible technologies. WSP and WML, part of the WAP 1.0 standard, are completely incompatible with HTML and HTTP used in the desktop web. Mobile web content could not be shared with desktop content.

Next, WAP 2.0 adopted XHTML Mobile Profile as the markup standard for mobile content. With this, mobile browsing is based on a subset of markup used for desktop browsing. Much more mobile content becomes available.

Desktop Browsing for Mobile

As mobile handsets increase in computing power and memory, the handset become possible of rendering the same content as the desktop computer. Images, graphics, and the full range of HTML tags can be rendered on the handset. Technologies such as ACCESS’s Smart-FitSMARTFIT re-render desktop content so that it fits better on mobile displays. Today, high-end handsets can display the same content that is viewed on a desktop PC.

Rich Media

Now, as operators strive to differentiate themselves and add more services, rich media targeted for the handset is coming to the fore. SVG, SMIL, PDF, and Flash are all formats used to render rich presentations on the handset. Even voice interactivity is possible with technologies such as XHTML + VoiceXML.

The Compound Document FormatCDF is being developed by the W3C to define a rich environment combining many of these markup languages. Adding ECMAScript and DOM, scripting is possible to create sophisticated applications.

Broswer as UI Engine

With smartphones capable of rendering both rich visual media and sophisticated applications, the next logical step in this evolution is to extend the browser to become the User Interface driver for the smartphone platform. Now, that the typical mobile browser is capable of rendering sophisticated markups such as XHTML, SVG, SMIL, and scripting such as DOM and ECMAScript, it requires only small extensions to use this rendering capability to implement the entire device UI. Examples of what these UIs could look like are shown in figure 1.

Sample User Interface Screens

Benefits

This strategy has several advantages. First, the UI can be easily customized. This allows, for example, the user to customize his device by adding skins or redesigning the entire UI look and feel. A manufacturer can customize the UI and rapidly produce highly customized devices for different customers. Or, in the case of a mobile handset, the operator can personalize the UI for different customer segments. UI customization becomes as simple as authoring a web page. SMIL can be used for sophisticated multimedia UIs or SVG can be used for scalable graphics and icons.

The importance of user customization must not be underestimated. The cellphone handset has become a truly personal computer. As evidenced by the popularity of wallpapers and ringtones, the user strives for the expression to personalize the device. Markup-based customization enables this need.

A second advantage is that the content can be easily updateable. For example, news alerts can be pushed to the root menu of a handset display where the information is most valuable to the user. Or, promotions can be pushed to the user handset where they will be most noticed. This reuses the browser’s existing ability to dynamically update content over the air.

Third, since virtually every smartphone device already has a web browser, this same sophisticated rendering technology is reused to power the UI. Thus, powerful capabilities are achieved with little extra cost. And last, since the UI is developed using traditional web design technologies, the design process is well-understood and existing content creation talent can be leveraged.

Comparison with Other Approaches

NFDM is just one of several possible ways to implement a dynamic menu technology. In this section we discuss other possible approaches: (1) Java-based approach, (2) XML-based approach (3) Proprietary approaches such as Macromedia Flash. We briefly explain the advantages of our browser-based approach against those three approaches.

Compared with the above 3 approaches, NetFront Dynamic Menu has two strong advantages from the viewpoint of integration. One is total memory consumption. Because NFDM is just a small additional module to the browser engine, not much additional memory is required. The second is that the porting effort for NFDM can be shared with the browser application. This means that assuming the browser is already running on the platform, the integration work is less than the other approaches.

NFDM

NetFront Dynamic Menu (NFDM) is ACCESS’s implementation for Dynamic Menu technology. NFDM uses the browser to render UI graphics and built from XHTML, SMIL, SVG, DOM, and ECMAScript. There are two main use cases for NFDM. The first is a handset root menu to where information is aggregated such as application launch icons, news alerts, email alerts, and other dynamic information. The second is an idle screen composed of newsalerts. Of course, since the menu is built from markup, ultimately possibilities are unlimited.

NFDM is built from two components the NFDM agent and the NFDM player. An architecture diagram is shown in figure 2.

NetFront Dynamic Menu Architecture

NFDM Agent

The NFDM Agent is a lightweight process that runs always in the background. Its function is to receive alerts or updates, save data to the handset filesystem if necessary, and to communicate with the NFDM player.

The NFDM Agent uses a modular protocol library to communicate with the server infrastructure and receive update messages. This allows for flexible use of a variety of protocols depending on the particular implementation requirements. SMS push, WAP push, Cell Broadcast, or others can all be used equally effectively.

The message sent to the NFDM agent consists of a URL identifying data to be downloaded and a command for the NFDM agent. For example, the data may be a ringtone and the command to set to device default ringtone. Also, the command may be an ECMAScript expression communicated to the NFDM player. This allows for very powerful dynamic command of the player.

Since the NFDM agent is the single point through which all messages are received, the agent is used as a single gating point for implementing security rules. Depending on the implementation, there are several security techniques that can be employed. Messages can be filtered so that they are allowed only when from an approved list of servers, digital signatures can be used, or a secure protocol such as SSL can be used.

Reliability of delivery is also guaranteed by the NFDM agent. A corrupted message has the potential to crash the entire handset, so standard techniques are used to guarantee reliable delivery.

Once the message is received and any data is downloaded, the Agent sends a message to the NFDM player to take appropriate action. This could be a refresh, a new URL, or execution of an ECMAScript command.

NFDM Player

The NFDM Player is built by constructing a small application layer shell on top of the existing browser engine. Since the existing browser engine is reused, the NFDM player consists of little additional code.

Since the NFDM player may execute ECMAScript commands, standard security techniques are used as when an ECMAScript is executed on a web page. This is a well known problem, so there are several possible solutions. For example, the user may request notification when unknown scripts are executed, scripts may be signed by a trusted authority, or alerts can be launched if the script attempts to access forbidden data.

DirectConnect, a technology similar to LiveConnect, but without the requirement for a JVM, is used for ECMAScript to communicate with the underlying handset functionality. DirectConnect provides objects, methods, and events for this communication. Since ECMAScript communication to the underlying handset functionality must be done through a single DirectConnect object, this single gating point can be used to implement appropriate security measures.

Examples

In order to better clarify what is possible with NFDM technology, below are two examples for how ECMAScript can be used for an operator to dynamically update the information on the user screen.

Example 1: Push an Ad

These functions show how an advertisement can be pushed to a handset. The message to the NFDM agent would be:

http://server.operator.com/ad_image.gif#javascript:push_ad(“ad_image.gif”);

where push_ad is defined as:

function push_ad(ad_source)
{
	document.getElementsByName("ad")[0].src = ad_source;
	document.getElementsByName("ad")[0].style.visibility = "visible";		
}

The first part of the message tells the NFDM agent to download the ad_image file from the designated server. The second part tells the agent to pass the ECMAScript function call to the NFDM player. The player then uses DOM to first, update the ad source, and second, make the ad visible.

Example 2: Update News

This example shows how innerHTML can be used to update a news alert displayed in scrolling marquee. In this example, it is not necessary to download any data, since the updated headline is contained in the function call. For this example, the meesage to the NFDM agent is:

javascript:update_news(“Read this alert!”);

And the update_news function is defined as: 

function update_news(x)
{
	document.getElementsByName("news")[0].innerHTML = x;
}

Conclusions

We implemented pilot versions of NFDM for both Symbian Series 60 and Brew OS. In this section we briefly discuss lessons we learned from these implementation.

Our hypothesis was that users would appreciate the rich graphics and interactivity afforded from menus built in markup, and indeed, this was the case. Users very much liked the improved look and feel. Users also appreciated the convenience of a centralized dashboard for launching applications and receiving alerts for news and messages.

Additionally, our hypothesis was correct that content developers felt they could author interactive menus by learning only some few additional techniques. In general, content developers were comfortable with the new technology.

From our implementation, we learned that performance is critical to a positive user experience. The user is used to instant feedback from a menu, and this must be maintained, both for interactive execution and for download of updated menu content.

Additionally, we learned that tighter OS integration capabilities are desirable for deep integration with the handset UI. This need can be easily solved with cooperation from the device manufacturer, but without this support, it can be difficult for independent software vendors to tightly integrate with the handset OS.

In this paper we present a new technique to expand the functionality of the device web browser to host the device user interface. We compare this technique against other possible implementations, describe our implementation, called NFDM, and lastly present observations from our pilot implementation. In the future, we expect technology of this sort to become increasingly popular for providing personalized, easily updateable user interface for smartphone users.

Bibliography

[GART] http://www.gartner.com/press_releases/asset_121402_11.html
[IDC] http://www.deviceforge.com/news/NS9740643212.html
[SMARTFIT] http://www.access-us-inc.com/Products/client-side/Prod_NetFront_nf.html
[CDF] http://www.w3.org/2004/CDF/
[XUL] http://www.xulplanet.com/
[UIONE] http://brew.qualcomm.com/brew/en/about/brew_uione.html/
[SURF] http://www.surfkitchen.com/

Biography

Daniel F. Zucker

Director of Technology, ACCESS Co., Ltd.

Dr. Daniel F. Zucker is Director of Technology at ACCESS Co. Ltd. Access is a leading provider for embedded browsing technologies running on over 160 million client devices and 500 different embedded products. Dr. Zucker has held engineering positions at Sun Microsystems and Advanced Micro Devices as well as several leadership positions in the mobile computing industry including CTO of ePocrates, the leading supplier of medical applications for the Palm OS Platform, and VP of Engineering for MobileAria, a provider of telematics solutions. Dr. Zucker has authored several academic and professional publications on mobile computing, multimedia, and computer security. He received his BS, MS, and PhD from Stanford University’s Department of Electrical Engineering, Computer Systems Laboratory.

Biography

Michimasa Uematsu

Managing Director of R&D, ACCESS Co., Ltd.

Mr. Michimasa Uematsu is Managing Director of Research and Development at ACCESS Co., Ltd. Mr. Uematsu joined ACCESS in 2000 with primary responsibility for the development of ACCESS’ advanced NetFront browser technologies. Mr. Uematsu now leads all Research and Development projects at ACCESS. From 1992 to 2000, Mr. Uematsu worked for the Information Science Research Institute of SECOM Co., Ltd. where he was engaged in the research of distributed object and component-ware technologies. Mr. Uematsu holds a Master of Science degree in Information Science from Tokyo University.

Biography

Tomihisa Kamada

CTO and VP, ACCESS Co., Ltd.

Dr Tomihisa Kamada is a co-founder of ACCESS CO., LTD. Dr. Kamada received a doctor degree in computer science from University of Tokyo, and now is executive vice-president and chief technology officer of ACCESS. His fields are network protocols, programming languages, visualization, and distributed ubiquitous computing. He is the original developer of "Compact HTML", which is a subset of HTML for mobile phones, and submitted it to W3C (World Wide Web Consortium). His vision is "Net-enabling consumer devices could reduce Digital Divide".