Mobile Application Development

A Usability Study Summary Report

Jan Benway, Jennifer Gove
March 20, 2002

Introduction

In March, 2002, Sun's Forte Tools organization ran a usability study on its closed source module for Mobile Application development. While the majority of the usability problems encountered by the participants during the study were in the closed source module, several were in parts of NetBeans that the participants used while completing the tasks required by the study. This report discusses just the issues related to NetBeans.

Overview

Users were invited to Sun's Usability Lab to work through a number of tasks in the IDE. They were asked to bring some code into the IDE, compile it, and test it. Follow-on tasks included, editing code, packaging an application for deployment to a mobile device, and customizing the packaged application.

Seven experienced developers participated in the study. Six of the participants had 2 or more years of Java experience. One participant was a beginning Java developer who had experience in other languages. All participants were selected for their experience and interest in developing Java applications for mobile devices.

One participant had significant experience with Forte for Java, and a second participant had minimal, but recent experience with the IDE. The other participants had little or no experience with Forte for Java or NetBeans, and primarily used another IDE (2 participants) or primarily used command-line tools (3 participants).

The Forte for Java build used was based on NetBeans 3.3.1.

Results

Usability Successes

It is easy, when reading (or writing) a report like this to forget that for every usability problem encountered by a participant, there were many things that simply did the right thing in the right way (so participants didn't comment on them), as well as things that participants liked enough to comment on. Some of the successes included:

• Participants had few difficulties editing code, or figuring out how compile or execute.
• Four of the seven participants used the Welcome window to help them get started.
• Those who used online help (sometimes prompted by facilitator when they became stuck) tended to find what they needed and followed the instructions to eventual success. (See more about Help use in the Help section, below.)

Usability Problems

This section contains a list of the primary usability problems encountered during the study. There were 7 participants in the study, but not all used the same features of the IDE. Consequently, each issue below lists the number of participants that had the trouble, and the total number of participants that used that feature.

Getting Started: Users had lots of trouble with projects

3 of 7 participants (43%)

The participants were not given any instruction about using projects to reach their goals, and in fact, it was not necessary to work with projects at all to be successful in the usability test tasks. Those new to the IDE tend to assume that projects are a central concept, and persist in trying to use projects even when instructed not to do so by the facilitator. The two experienced Forte for Java users in the study did not attempt to use projects.

• The first things participants needed to do was get some downloaded code into the IDE. All participants who did not have previous experience with FFJ had trouble with this, and tended to look (in vain) to projects as the way to solve the problem.
• One of the tasks participants were asked to do was to package their mobile application into a MIDlet Suite. This needed to be done explicitly by creating a Suite object (similar to a JAR Recipe). Participants did not realize they needed to explicitly create a MIDlet suite because they believed that the project would take care of this for them.
• None of the three participants who used projects during the session ever seemed to understand how they worked. All who began this way needed to be lead away from projects by the facilitator in order to make progress on the tasks.

Recommendations:

• Until NetBeans 4.0 is ready, look for ways to lead new users away from projects as an initial starting point. This will be done as part of the Out of the Box Experience work, which is expected to be implemented for the 3.4 release. A new welcome panel will projects as a convenience for experienced users, but not the way for new users to begin.
• Once a more robust projects system is available (NetBeans 4.0), modules that help users create packaged applications for deployment should create the packaging without an explicit user action.

Getting Started: Bringing Code into the IDE

5 of 7 participants (71%)

The participants were given a zip file containing four Java files enclosed in a package. They needed to bring this code into the IDE, then compile and execute it. The only participants who completed the task without difficulty were the two participants with previous Forte for Java experience. Three participants had so much trouble that they were unable to continue without help from the facilitator.

Three of the users used File->Open File as the way to bring code in, but managed to mount the filesystem at the incorrect level, even with this dialog boxes to guide them:

Recommendation:

Users who are just starting to use the IDE find dialog above baffling, as they have no idea why they are being asked this question. The reason the dialog appears is as a "ramp up" to help explain the concept of mounting to new users. However, it seems that when new users see this dialog box, they are not quite yet ready to learn about these concepts, and the dialog often fails to provide the gentle training it is attempting to provide.

The dialog box provide a second feature, which is to allow users to select a different mounting point than the default one that the IDE has "guessed." However, File->Open File is not the most efficient way to mount filesystems, and it is unlikely that many experienced NetBeans users ever use this action.

Therefore, the best path may be to eliminate this dialog box altogether, and simply mount the filesystem at the mount point that the IDE has "guessed." This has the advantage of giving beginning users what they want without asking them questions they cannot handle, and preventing them from potentially selecting a different, and probably incorrect, mount point. Beginning users would thus be able to have quick success in their first few minutes of using the IDE, and could then learn about mounting at their leisure, through the welcome panel, the online help, or by experimenting with the explorer.

Eliminating the dialog box will not disadvantage more experienced users, because they will mount filesystems using the Mount action directly. This change would have no effect on using the File Open action to open files in already-mounted filesystems.

This recommendation has been filed as enhancement request #23223.

Window Management Difficulties

7 of 7 participants (100%)

In this study, the participants began working with the IDE installed and started up, with all "one time" dialog boxes dismissed for them. They began with the IDE in standard position, with the Welcome window appearing and all default options (including SDI) set for them. All users had some difficulties related to the windowing systems:

• Three users complained specifically about the SDI mode, saying they didn't like dealing with all the different windows. None mentioned that they thought there may be a way to customize the windowing system. Note that there is no way of knowing from this study whether those users who had trouble with SDI would have preferred NetBean's MDI or not.
• Six users had difficulty balancing the online help window with other windows. It tended to pop up unexpectedly.
• Three users had difficulties because wizard were modal -- this prevented them from finding information they needed to help complete the wizard.
• Three users "lost" windows, either because they closed them and couldn't figure out how to open them, or because of the change in workspaces. One user accidentally undocked an explorer window tab into a properties sheet, and was unable to find the tab again.

Recommendation: Continue working on windowing system enhancements.

Difficulties with the Explorer Runtime Tab

4 of 5 (80%)

One of the tasks the participants were asked to do was to make a customization to the mobile emulator runtime environment. Such customizations are done, according to NetBeans precedent, in the runtime tab of the explorer. None of the users found this explorer tab without help. The one user who was successful on this task without help found a shortcut from the execution properties of a node.

Users also had difficulty finding the Runtime tab after reading about it in the help. Instead, they repeatedly clicked the Running workspace tab while saying to themselves, "Runtime tab of the explorer." This happened with three users who used Help to find the emulator registry.

Recommendation: Develop a different paradigm for giving users access to the information traditionally found in the runtime tab, and remove the tab from the explorer. The emulator tools UI was patterned after the Server Registry, which may be found to have similar usability problems. Consideration to this problem should be given when designing the new projects system, to see if there might be a logical, but more discoverable, place to provide access to this functionality. This should be considered as part of the design process for the 4.0 release of NetBeans.

Compiler Output

4 of 7 (57%)

Four users requested more detailed compilation output. For instance, they asked for a timestamp to appear, so they know whether the message "Finished Foo" is from the most recent compilation attempt or an earlier one. Some asked for a more detailed description of what happened. For example, compiling mobile application includes a preverification step, and users requested that this should be mentioned in the output. Some participants just wanted more details, such as specifically what was compiled, and more general status information.

Recommendation: Adding a timestamp may be very useful. Whether more status information on successful compiles would be useful to users once they become more experienced with the IDE is doubtful. Filed enhancement #23221 to capture the idea of the timestamp.

Online Help Findings

Participants in this study tended to use online help. Some interesting findings emerged from observing this:

• The participants almost exclusively used Help from the Help menu (7 of 7). There was one instance of a user clicking the Help button from a dialog box, and there were no instances of using F1. The users primarily turned to Online Help when they had difficulties getting started with a task, and tended not to seek out context-sensitive information while on a given panel or properties sheet.
• Nearly all the help participants sought was specifically for the mobile development module, but most users opened Core Help from the Help menu..
• Once Help was open, the Table of Contents was the primary way users found what the needed (note that this is the default tab). Two participants also used search. None used the index.
• More than half the participants used some of the "See Also" links at the bottom of the help pages.
• Despite the fact that users did not use context-sensitive help buttons, when Help directed them to a wizard, they tended to keep help up and read the help for each wizard page as they progressed through the wizard.
• Five of the participants found the Help window to be poorly sized, and had to resize it manually before they could proceed.

Conclusions

The participants in this study were intelligent, well-qualified Java developers who fit the target user based for the product. Five of them were new to the IDE, and the results presented here thereby focus on issues that users run into in the first two hours of use. These first two hours are critical for "user capture" and it is important that developers become successful enough to feel that the IDE works and is worth spending the effort to learn about its more advanced and powerful features.

Overall, the participants in the study were quite successful, and nearly all finished the rather ambitious list of tasks they were asked to do. However, the participants tended to require some amount of intervention from the facilitator to get over rough spots. If these spots can be smoothed out and improved for beginning users, this will increase the total number of satisfied users of the IDE.

In many cases, improvements for new users can help more advanced users. For example, improving access to the features in the runtime tab can help advanced users who use that functionality only occasionally. If designed well, it can also help advanced to users who use that feature often by reducing the number of clicks required to make frequent adjustments.