CVS/NetBeans Usability Test
A Summary Report
Maya Venkatraman,
Jeff
Hoffman, Ann Sunhachawee
November 7, 2000
Goals
The test was designed to detect usability issues that pertained to the
integration of two source control suites with the IDE: CVS and TeamWare.
This test was also targeted at discovering the usability issues related
to the post-installation startup sequence, Java class creation, CVS, TeamWare,
and the JAR Packager.
Please note that TeamWare Data are not reported here; all other data
are reported in full.
Design
Task List
-
Task 1: Start the IDE and explore for a few minutes. Think aloud.
-
Task 2: Access the code in the repository. View the code. Find its
version number.
-
Task 3: Create new classes. Compile the code and execute it.
-
Task 4: Share code.
-
Task 5: Create a branch.
-
Task 6: Get updates.
-
Task 7: Integrate in new code.
-
Task 8: Create a JAR file.
-
Task 9: Merge the branch.
Please see the CVS
User Tasks for a full description of what the participants were asked
to do.
Participants
Nine users participated in the study. The participants were developers
or engineers who had at least a year of experience working with Java, and
at least three months of experience working with source control. All participants
had previous experience with the use of an Integrated Development Environment
(IDE) such as NetBeans, Forte for Java, Sun WorkShop, Microsoft VisualStudio,
Borland JBuilder, Symantec VisualCafe etc. Five of the participants were
familiar with CVS and the other five were familiar with TeamWare. Six of
the participants rated themselves as expert Solaris users and 4 rated themselves
as average Solaris users. All the participants rated themselves as expert
Windows users.
Please see the Participant Profiles
for a full description of the participants.
Setup
Participants conducted the user studies using the
Boston (Forte for Java - Version 2) Beta version of the IDE running on
a Windows NT machine. Studies were conducted in the Usability Labs at Sun
Microsystems (Menlo Park). The lab has a one way mirror. The moderator
and other viewers could see and hear the participant from the Control Room.
The moderator could also talk to the participant from there. The participant
was video taped from three angles. The participant's screen was also taped
for the duration of the test.
 |
 |
Viewers in the Control Room |
View through glass
in Control Room |
In this test, the participants were initially given a questionnaire
to fill out and were told about the setup. The moderator then walked them
into the lab, and stayed with them for the time that it took to complete
the first task. The moderator returned to the Control Room, and for the
rest of the time communicated with the participant from there. This way,
the participant was less conscious of being observed. The participant worked
on the tasks, and thought aloud while doing so. After completing each task,
the participant filled out a brief survey. At the end of the session, the
participant filled out a final, longer, survey.
Results
Data Scope
Only CVS and NetBeans usability data are presented; TeamWare data are not
presented in this report.
What Users Liked
While this report focuses on areas needing improvement, it is also worthwhile
to list things that the participants liked about NetBeans.
-
AutoUpdate - liked that you can get updates from the web.
-
Similarity to other IDEs - several users liked that it isn't that different
looking from other IDEs.
-
Templates - to help get started.
-
Clicking on an error in the output window opens the source code to the
appropriate line.
-
Code Completion, code coloring (all liked this).
-
Multiple views of a form - likes form view, component view and editor view.
-
New class wizard (several liked this).
-
Source control integration, CVS Diff option.
-
Explorer navigation of classes.
-
Availability of online help (several liked this).
-
Source editor automatic indentation, cursor matching (a couple mentioned
this).
-
Generated code for GUI.
-
Welcome screen (a couple mentioned this).
-
Feedback in the splash screen at startup.
-
Java smarts, like prompting for a main class when its necessary (i.e. for
an applet).
The remainder of this report discusses those aspects of NetBeans that caused
confusion or difficulty.
Source Control Integration - Users don't understand Mount
-
Users (8 of 9) did not realize that they had to use the Mount File System
option to access a source control workspace. Every user had to be prompted
by the moderator to try this option, after they had spent some time (15
minutes or more) trying to complete the task. After they had been prompted,
most of the participants remarked that they would have never found this
and most of them went on to say that "mount" meant something akin to an
NFS mount to them. In other words, the term mount was loaded with other
meanings for most users, and they did not think to use this menu item to
reach source control workspaces. Users remarked that they would like a
"smarter Open" which would detect that the file they were opening was part
of a source control workspace and automatically route them to the mount
wizard.
-
Some users also got slightly confused between the
IDE workspaces (Editing, Debugging etc.) and source control workspaces.
Users did realize that these were two different concepts with the same
name. Some users (4 of 9) mounted multiple versions of the files
(say one as a source control workspace and the other as a simple file system).
Source Control Integration - Where are the commands?
-
All users looked for source control options in the Tools menu (when they
were trying to mount a file system and when they had completed the mount
and were now trying to access specific source control commands such as
check in). Some users stated that they were surprised that the source control
options were not in the Tools right click menu or the main menu.
-
Some users (5 of 9) took time to get used to the fact that the contents
of the right click menu changed depending on which item in the Explorer
was selected (the object/action linkage was not clear to them).
-
Some users found the organization of the items in the right click menu
and main menu confusing. While looking for source control commands, users
seemed to lose items that they had just used. One user remarked that it
was difficult to locate the items in the absence of some organizational
principles.
-
Users (4 of 5) did not create a working directory. They proceeded using
the defaults. One user wanted to create a working directory, but could
not figure out how to accomplish this.
-
Users were not sure where to look for the Revisions file. Some CVS users
(3 of 5) looked at the properties window. One user looked at the log file.
-
One user looked for, and could not find, source control commands in the
Explorer's Project tab.
-
One user was frustrated that the Add Tag dialog appeared for each file
selected.
-
Users (2 of 3) were not sure that the Branch task had been completed. The
IDE did not provide them with any feedback. (Only three users completed
the branch task).
-
Users (2 of 3) were not sure which branch was active. (Only three users
completed the branch task).
-
Users (2 of 3) said they would like an option to preview (-n) and an option
to get less CVS output (-q).
-
One of the five CVS users commented that he would like to see the results
in the output window for all the CVS commands.
Feedback - The IDE does not provide enough
-
Users tried to compile files and failed to notice the compilation status
displayed in the status bar. They compiled many times before they realized
that they had succeeded. Once they did locate the status bar the message
that appeared ("<filename> - finished") did not seem very meaningful
to them. Users (6 of 9) were never sure when to expect feedback in the
output window and when to expect it in the status bar.
-
When using CVS users found that the lack of information about the options
used was problematic. One user remarked that this is information that he
would need and would be inclined to use the command line version of CVS
in the absence of such feedback
-
When a compiler error is displayed in the output window, users (5 of 9)
select a line other than the one with the line number in it and expect
that a double click will open the editor. (Only the first line works).
-
Users (4 of 9) opened more than one IDE because of slow startup and lack
of immediate feedback indicating that the IDE was loading.
Creating a Class
-
Users (3 of 9) had trouble with naming the class in the New dialog box.
The class name entry field did not let the user clear the text "<default>"
-
The layout on the Add Methods screens is confusing to some users. They
are not sure if they should click Add to start a sequence or Submit an
addition. The screen's slow load time contributed to the confusion.
-
Users (5 of 9) noticed that there was no way to add
import statements using the GUI
-
Two users tried to add all the method parameters
using the same dialog box
-
Two users were confused by the first wizard screen
and thought they could add methods there.
Initial Experience and General Issues:
-
Four users remarked they did not like the IDE to open with a profusion
of windows.
JAR Packager
-
Users were not able to use the JAR packager
effectively due to a window sizing bug. This bug is currently being
addressed.
Projects - What are they?
-
Some users had trouble with the concepts of projects and how they relate
to source control. One user looked for source control functionality in
the projects tab.
Final Survey Results
The participants were asked to answer some questions at the end of their
session.
Bugs Filed
The results from this test were used to open bugs in the NetBeans
Bug repository. The table below lists the Bug numbers and provides
a brief description of the bug.
|
CVS - Usability
|
|
.
|
Bug No.
|
Description
|
Suggested Resolution
|
Topic
|
1.
|
8224 |
Users (4 of 5) did not create a working directory. They
proceeded using the defaults. One user wanted to create a working directory,
but could not figure out how. |
Add a way to create a working directory from CVS wizard. |
CVS |
2.
|
8255 |
Users were not sure where to look for the Revisions file.
Some CVS users (3 of 5) looked at the properties window. One user looked
at the log file. |
Provide access to CVS information from within the properties
window. |
CVS |
3.
|
8223 |
Users could not find source control commands in the Project
tab. |
. |
CVS |
4.
|
8258 |
Users frustrated that a separate Add Tag dialog appears
for each file selected. |
. |
CVS |
5.
|
8259 |
Users get no feedback if CVS branch has been completed. |
. |
CVS |
6.
|
8260 |
Users are not sure which branch is currently active. |
. |
CVS |
7.
|
8261 |
Users cannot set CVS "preview" option. |
. |
CVS |
8.
|
8262 |
Users would like an option to get less CVS output (-q). |
Hide all feedback from CVS actions: "QUIET" option. |
CVS |
9.
|
8263 |
Users would like to see output window for results of all
CVS commands. |
Show all feedback from CVS actions, not just errors. |
CVS |
CVS - Functionality
|
|
.
|
Bug No.
|
Description
|
Resolution
|
Topic
|
10.
|
7874 |
Opening a java file (with an associated form) caused the
java file to appear modified. |
. |
CVS |
11.
|
7875 |
CVS -> Commit from a CVS file system node causes CVS error. |
. |
CVS |
12.
|
7876 |
CVS module windows do not respond to "enter" key to close. |
. |
CVS |
|
Editing
|
|
.
|
Bug No.
|
Description
|
Resolution
|
Topic
|
13.
|
8344 |
One user said that the highlighting for parenthesis matching
feature made it hard to see the text insertion point. |
. |
Editing |
14.
|
8266 |
No GUI mechanism to add import statements to class. |
. |
Editing |
15.
|
7871 |
New..." dialog has inappropriate ampersand in first panel. |
. |
Editing |
16.
|
7882 |
New Class Wizard > Add Field Panel has blinking cursor
in non-active field. |
. |
Editing |
17.
|
8267 |
One user said that he would like to be able to enter comments
for the fields/methods. |
. |
Editing |
18.
|
7881 |
There is no name checking; if the user enters the name
of a class that already exists in the package, the wizard just lets them
do it and complains after it's too late. |
New Class Wizard should check if the class name is unique
before continuing. |
Editing |
|
Compiling / Building
|
|
.
|
Bug No.
|
Description
|
Resolution
|
Topic
|
19.
|
8341 |
When a compiler error is displayed in the output window,
user selects a line other than the one with the line number in it and expects
that a double click will open the editor (only the first line works). |
. |
Compile |
20.
|
8342 |
User's don't realize that compile has completed without
errors. |
Compiler feedback needs to be more obvious. |
Compile |
21.
|
7727 |
Status bar message after compilation is not meaningful
enough. |
. |
Compile |
22.
|
8343 |
It was not clear to one user which object the execute toolbar
button will execute. |
. |
Compile |
|
File Manipulation
|
|
.
|
Bug No.
|
Description
|
Resolution
|
Topic
|
23.
|
8299 |
"Which Package" dialog that appears when using File > Open
to view a simple text file causes confusion |
. |
FM |
24.
|
7727 |
Two users said they would like to use a "Save as..." menu
item to save files with another name |
. |
FM |
|
Documentation
|
|
.
|
Bug No.
|
Description
|
Resolution
|
Topic
|
25.
|
8289,
8290,
8291 |
Help is not context sensitive |
. |
Docs |
|
|
Human Interface Engineer
(HIE) Test team for this test
-
Jeff Hoffman
- HIE Forte Tools (Test Design, Moderator)
Ann Sunhachawee
- HIE Forte Tools and
Maya
Venkatraman - HIE NetBeans Open Source. |
|
Project Features
About this Project
|