CVS/NetBeans Usability Test

A Summary Report

Maya Venkatraman, Jeff Hoffman, Ann Sunhachawee
November 7, 2000

Goals

The test was designed to detect usability issues that pertained to the integration of two source control suites with the IDE: CVS and TeamWare. This test was also targeted at discovering the usability issues related to the post-installation startup sequence, Java class creation, CVS, TeamWare, and the JAR Packager.

Please note that TeamWare Data are not reported here; all other data are reported in full.

Design

Task List

  • Task 1: Start the IDE and explore for a few minutes. Think aloud.
  • Task 2: Access the code in the repository. View the code. Find its version number.
  • Task 3: Create new classes. Compile the code and execute it.
  • Task 4: Share code. 
  • Task 5: Create a branch.
  • Task 6: Get updates.
  • Task 7: Integrate in new code.
  • Task 8: Create a JAR file.
  • Task 9: Merge the branch. 
Please see the CVS User Tasks for a full description of what the participants were asked to do.

Participants

Nine users participated in the study. The participants were developers or engineers who had at least a year of experience working with Java, and at least three months of experience working with source control. All participants had previous experience with the use of an Integrated Development Environment (IDE) such as NetBeans, Forte for Java, Sun WorkShop, Microsoft VisualStudio, Borland JBuilder, Symantec VisualCafe etc. Five of the participants were familiar with CVS and the other five were familiar with TeamWare. Six of the participants rated themselves as expert Solaris users and 4 rated themselves as average Solaris users. All the participants rated themselves as expert Windows users.

Please see the Participant Profiles for a full description of the participants.

Setup

Participants conducted the user studies using the Boston (Forte for Java - Version 2) Beta version of the IDE running on a Windows NT machine. Studies were conducted in the Usability Labs at Sun Microsystems (Menlo Park). The lab has a one way mirror. The moderator and other viewers could see and hear the participant from the Control Room. The moderator could also talk to the participant from there. The participant was video taped from three angles. The participant's screen was also taped for the duration of the test. 
 
Viewers in the Control Room   View through glass in Control Room

In this test, the participants were initially given a questionnaire to fill out and were told about the setup. The moderator then walked them into the lab, and stayed with them for the time that it took to complete the first task. The moderator returned to the Control Room, and for the rest of the time communicated with the participant from there. This way, the participant was less conscious of being observed. The participant worked on the tasks, and thought aloud while doing so. After completing each task, the participant filled out a brief survey. At the end of the session, the participant filled out a final, longer, survey.

Results

Data Scope

Only CVS and NetBeans usability data are presented; TeamWare data are not presented in this report.

What Users Liked

While this report focuses on areas needing improvement, it is also worthwhile to list things that the participants liked about NetBeans.
  • AutoUpdate - liked that you can get updates from the web.
  • Similarity to other IDEs - several users liked that it isn't that different looking from other IDEs.
  • Templates - to help get started.
  • Clicking on an error in the output window opens the source code to the appropriate line.
  • Code Completion, code coloring (all liked this).
  • Multiple views of a form - likes form view, component view and editor view.
  • New class wizard (several liked this).
  • Source control integration, CVS Diff option.
  • Explorer navigation of classes.
  • Availability of online help (several liked this).
  • Source editor automatic indentation, cursor matching (a couple mentioned this).
  • Generated code for GUI.
  • Welcome screen (a couple mentioned this).
  • Feedback in the splash screen at startup.
  • Java smarts, like prompting for a main class when its necessary (i.e. for an applet).
The remainder of this report discusses those aspects of NetBeans that caused confusion or difficulty.

Source Control Integration - Users don't understand Mount

  • Users (8 of 9) did not realize that they had to use the Mount File System option to access a source control workspace. Every user had to be prompted by the moderator to try this option, after they had spent some time (15 minutes or more) trying to complete the task. After they had been prompted, most of the participants remarked that they would have never found this and most of them went on to say that "mount" meant something akin to an NFS mount to them. In other words, the term mount was loaded with other meanings for most users, and they did not think to use this menu item to reach source control workspaces. Users remarked that they would like a "smarter Open" which would detect that the file they were opening was part of a source control workspace and automatically route them to the mount wizard. 
  • Some users also got slightly confused between the IDE workspaces (Editing, Debugging etc.) and source control workspaces. Users did realize that these were two different concepts with the same name.  Some users (4 of 9) mounted multiple versions of the files (say one as a source control workspace and the other as a simple file system). 

Source Control Integration - Where are the commands?

  • All users looked for source control options in the Tools menu (when they were trying to mount a file system and when they had completed the mount and were now trying to access specific source control commands such as check in). Some users stated that they were surprised that the source control options were not in the Tools right click menu or the main menu.
  • Some users (5 of 9) took time to get used to the fact that the contents of the right click menu changed depending on which item in the Explorer was selected (the object/action linkage was not clear to them).
  • Some users found the organization of the items in the right click menu and main menu confusing. While looking for source control commands, users seemed to lose items that they had just used. One user remarked that it was difficult to locate the items in the absence of some organizational principles.
  • Users (4 of 5) did not create a working directory. They proceeded using the defaults. One user wanted to create a working directory, but could not figure out how to accomplish this. 
  • Users were not sure where to look for the Revisions file. Some CVS users (3 of 5) looked at the properties window. One user looked at the log file.
  • One user looked for, and could not find, source control commands in the Explorer's Project tab. 
  • One user was frustrated that the Add Tag dialog appeared for each file selected. 
  • Users (2 of 3) were not sure that the Branch task had been completed. The IDE did not provide them with any feedback. (Only three users completed the branch task).
  • Users (2 of 3) were not sure which branch was active. (Only three users completed the branch task).
  • Users (2 of 3) said they would like an option to preview (-n) and an option to get less CVS output (-q).
  • One of the five CVS users commented that he would like to see the results in the output window for all the CVS commands.

Feedback - The IDE does not provide enough 

  • Users tried to compile files and failed to notice the compilation status displayed in the status bar. They compiled many times before they realized that they had succeeded. Once they did locate the status bar the message that appeared ("<filename> - finished") did not seem very meaningful to them. Users (6 of 9) were never sure when to expect feedback in the output window and when to expect it in the status bar.
  • When using CVS users found that the lack of information about the options used was problematic. One user remarked that this is information that he would need and would be inclined to use the command line version of CVS in the absence of such feedback
  • When a compiler error is displayed in the output window, users (5 of 9)  select a line other than the one with the line number in it and expect that a double click will open the editor.  (Only the first line works).
  • Users (4 of 9) opened more than one IDE because of slow startup and lack of immediate feedback indicating that the IDE was loading.

Creating a Class

  • Users (3 of 9) had trouble with naming the class in the New dialog box. The class name entry field did not let the user clear the text "<default>"
  • The layout on the Add Methods screens is confusing to some users. They are not sure if they should click Add to start a sequence or Submit an addition. The screen's slow load time contributed to the confusion.
  • Users (5 of 9) noticed that there was no way to add import statements using the GUI
  • Two users tried to add all the method parameters using the same dialog box
  • Two users were confused by the first wizard screen and thought they could add methods there.

Initial Experience and General Issues:

  • Four users remarked they did not like the IDE to open with a profusion of windows. 

JAR Packager

  • Users  were not able to use the JAR packager effectively due to a window sizing bug.  This bug is currently being addressed. 

Projects - What are they?

  • Some users had trouble with the concepts of projects and how they relate to source control. One user looked for source control functionality in the projects tab. 

Final Survey Results

The participants were asked to answer some questions at the end of their session.

Bugs Filed

The results from this test were used to open bugs in the NetBeans Bug repository. The table below lists the Bug numbers and provides a brief description of the bug. 
 
CVS - Usability
Bug No. 
Description 
Suggested Resolution 
Topic 
1. 
8224  Users (4 of 5) did not create a working directory. They proceeded using the defaults. One user wanted to create a working directory, but could not figure out how.  Add a way to create a working directory from CVS wizard. CVS
2. 
8255  Users were not sure where to look for the Revisions file. Some CVS users (3 of 5) looked at the properties window. One user looked at the log file. Provide access to CVS information from within the properties window. CVS
3. 
8223  Users could not find source control commands in the Project tab.  CVS
4. 
8258  Users frustrated that a separate Add Tag dialog appears for each file selected.  CVS
5. 
8259  Users get no feedback if CVS branch has been completed.  CVS
6. 
8260  Users are not sure which branch is currently active.  CVS
7. 
8261  Users cannot set CVS "preview" option.  CVS
8. 
8262  Users would like an option to get less CVS output (-q). Hide all feedback from CVS actions: "QUIET" option. CVS
9. 
8263  Users would like to see output window for results of all CVS commands.  Show all feedback from CVS actions, not just errors. CVS
CVS - Functionality 
Bug No. 
Description 
Resolution 
Topic 
10. 
7874  Opening a java file (with an associated form) caused the java file to appear modified. CVS
11. 
7875  CVS -> Commit from a CVS file system node causes CVS error. CVS
12. 
7876  CVS module windows do not respond to "enter" key to close. CVS

 
Editing 
Bug No. 
Description 
Resolution 
Topic 
13. 
8344  One user said that the highlighting for parenthesis matching feature made it hard to see the text insertion point.  Editing
14. 
8266  No GUI mechanism to add import statements to class. Editing
15. 
7871  New..." dialog has inappropriate ampersand in first panel. Editing
16. 
7882  New Class Wizard > Add Field Panel has blinking cursor in non-active field. . Editing
17. 
8267  One user said that he would like to be able to enter comments for the fields/methods. Editing
18. 
7881  There is no name checking; if the user enters the name of a class that already exists in the package, the wizard just lets them do it and complains after it's too late. New Class Wizard should check if the class name is unique before continuing. Editing

 
Compiling / Building 
Bug No. 
Description 
Resolution 
Topic 
19. 
8341  When a compiler error is displayed in the output window, user selects a line other than the one with the line number in it and expects that a double click will open the editor (only the first line works). Compile
20. 
8342  User's don't realize that compile has completed without errors.  Compiler feedback needs to be more obvious. Compile
21. 
7727  Status bar message after compilation is not meaningful enough. Compile
22. 
8343  It was not clear to one user which object the execute toolbar button will execute. Compile

 
File Manipulation 
Bug No. 
Description
Resolution 
Topic 
23. 
8299  "Which Package" dialog that appears when using File > Open to view a simple text file causes confusion FM
24. 
7727  Two users said they would like to use a "Save as..." menu item to save files with another name FM

 
Documentation 
Bug No. 
Description 
Resolution 
Topic 
25. 
8289
8290
8291
Help is not context sensitive Docs
Human Interface Engineer (HIE) Test team for this test
Jeff Hoffman - HIE Forte Tools (Test Design, Moderator) 
Ann Sunhachawee - HIE Forte Tools and
Maya Venkatraman - HIE NetBeans Open Source. 

Project Features

About this Project

ui was started in November 2009, is owned by Jiří Kovalský, and has 43 members.
By use of this website, you agree to the NetBeans Policies and Terms of Use (revision 20160708.bf2ac18). © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo
 
 
Close
loading
Please Confirm
Close