In this test, the participants were initially given a questionnaire to fill out and were told about the setup. The moderator then walked them into the lab, and stayed with them for the time that it took to complete the first task. The moderator returned to the Control Room, and for the rest of the time communicated with the participant from there. This way, the participant was less conscious of being observed. The participant worked on the tasks, and thought aloud while doing so. After completing each task, the participant filled out a brief survey. At the end of the session, the participant filled out a final, longer, survey.

Results

Data Scope

What Users Liked

• AutoUpdate - liked that you can get updates from the web.
• Similarity to other IDEs - several users liked that it isn't that different looking from other IDEs.
• Templates - to help get started.
• Clicking on an error in the output window opens the source code to the appropriate line.
• Code Completion, code coloring (all liked this).
• Multiple views of a form - likes form view, component view and editor view.
• New class wizard (several liked this).
• Source control integration, CVS Diff option.
• Explorer navigation of classes.
• Availability of online help (several liked this).
• Source editor automatic indentation, cursor matching (a couple mentioned this).
• Generated code for GUI.
• Welcome screen (a couple mentioned this).
• Feedback in the splash screen at startup.
• Java smarts, like prompting for a main class when its necessary (i.e. for an applet).

Source Control Integration - Users don't understand Mount

• Users (8 of 9) did not realize that they had to use the Mount File System option to access a source control workspace. Every user had to be prompted by the moderator to try this option, after they had spent some time (15 minutes or more) trying to complete the task. After they had been prompted, most of the participants remarked that they would have never found this and most of them went on to say that "mount" meant something akin to an NFS mount to them. In other words, the term mount was loaded with other meanings for most users, and they did not think to use this menu item to reach source control workspaces. Users remarked that they would like a "smarter Open" which would detect that the file they were opening was part of a source control workspace and automatically route them to the mount wizard. 

• Some users also got slightly confused between the IDE workspaces (Editing, Debugging etc.) and source control workspaces. Users did realize that these were two different concepts with the same name.  Some users (4 of 9) mounted multiple versions of the files (say one as a source control workspace and the other as a simple file system). 

Source Control Integration - Where are the commands?

• All users looked for source control options in the Tools menu (when they were trying to mount a file system and when they had completed the mount and were now trying to access specific source control commands such as check in). Some users stated that they were surprised that the source control options were not in the Tools right click menu or the main menu.
• Some users (5 of 9) took time to get used to the fact that the contents of the right click menu changed depending on which item in the Explorer was selected (the object/action linkage was not clear to them).
• Some users found the organization of the items in the right click menu and main menu confusing. While looking for source control commands, users seemed to lose items that they had just used. One user remarked that it was difficult to locate the items in the absence of some organizational principles.
• Users (4 of 5) did not create a working directory. They proceeded using the defaults. One user wanted to create a working directory, but could not figure out how to accomplish this. 
• Users were not sure where to look for the Revisions file. Some CVS users (3 of 5) looked at the properties window. One user looked at the log file.
• One user looked for, and could not find, source control commands in the Explorer's Project tab. 
• One user was frustrated that the Add Tag dialog appeared for each file selected. 
• Users (2 of 3) were not sure that the Branch task had been completed. The IDE did not provide them with any feedback. (Only three users completed the branch task).
• Users (2 of 3) were not sure which branch was active. (Only three users completed the branch task).
• Users (2 of 3) said they would like an option to preview (-n) and an option to get less CVS output (-q).
• One of the five CVS users commented that he would like to see the results in the output window for all the CVS commands.

Feedback - The IDE does not provide enough 

• Users tried to compile files and failed to notice the compilation status displayed in the status bar. They compiled many times before they realized that they had succeeded. Once they did locate the status bar the message that appeared ("<filename> - finished") did not seem very meaningful to them. Users (6 of 9) were never sure when to expect feedback in the output window and when to expect it in the status bar.
• When using CVS users found that the lack of information about the options used was problematic. One user remarked that this is information that he would need and would be inclined to use the command line version of CVS in the absence of such feedback
• When a compiler error is displayed in the output window, users (5 of 9)  select a line other than the one with the line number in it and expect that a double click will open the editor.  (Only the first line works).
• Users (4 of 9) opened more than one IDE because of slow startup and lack of immediate feedback indicating that the IDE was loading.

Creating a Class

• Users (3 of 9) had trouble with naming the class in the New dialog box. The class name entry field did not let the user clear the text "<default>"
• The layout on the Add Methods screens is confusing to some users. They are not sure if they should click Add to start a sequence or Submit an addition. The screen's slow load time contributed to the confusion.
• Users (5 of 9) noticed that there was no way to add import statements using the GUI
• Two users tried to add all the method parameters using the same dialog box
• Two users were confused by the first wizard screen and thought they could add methods there.

Initial Experience and General Issues:

• Four users remarked they did not like the IDE to open with a profusion of windows. 

JAR Packager

• Users  were not able to use the JAR packager effectively due to a window sizing bug.  This bug is currently being addressed. 

Projects - What are they?

• Some users had trouble with the concepts of projects and how they relate to source control. One user looked for source control functionality in the projects tab. 

Final Survey Results

• CVS User Tasks contains the full set of questions.
• Final Survey Results contains a summary of the results.

Bugs Filed