This usability study tested the first prototype of the new
NetBeans projects system, using some developers who were familiar with
NetBeans/Forte and some who were not. By and large, participants
understood and liked the project system's core concepts, e.g., the Java
Sources node, the Resources node and the Output node. Most of their
difficulties were because of prototype limitations (e.g., the inability
to use JFileChooser to open files) or because of known NetBeans problems
(e.g., its extreme context sensitivity).
High Level Findings:
Participants found that the start-up materials on the welcome
screen were useful for orienting to the
IDE.
Some difficulties were experienced in creating,
opening, and closing projects. These usability problems resulted
from having implemented only limited paths to carry out these actions,
from NetBeans extreme context sensitivity, and the current capability of
creating orphaned files not belonging to a project.
When users tried to create,
or add source directories or packages they were confused by some of
the dialogs they encountered. In addition, we found that legacy users
are likely to be tempted to resort to previous common actions, such as
mount filesystem, rather than use the projects implementation, particularly
when experiencing difficulty in proceeding. Such actions will likely
cause further problems. Participants attempted to take more varied
paths than currently implemented for creating / adding source directories
or packages.
While working with source
files, participants attempted drag and drop first in order to move
them about in the project.
Building and compiling
was generally straight-forward for participants with only a few minor issues
encountered. All participants were easily able to find the location of
their compiled file in the logical view presented in the NetBeans explorer.
Participants encountered a few difficulties when working
with resources, for example, they attempted to
add a .jar resource to their project in multiple, currently unsupported,
ways. Once they found the correct path, they were able to easily add the
resource. Participants expressed some doubt about adding the resource,
reqesting more confirmation that the resource was on the classpath.
The functionality to switch from logical
to physical views of files had low discoverability; but once found,
participants understood and liked the distinction.
Introduction
The NetBeans Projects team conducted a usability study in May 2002, in
California. The usability study tested the prototype of the new implementation
of Projects in NetBeans. (Visit the screenshots
page to see what the prototype looks like.) The overriding goal
was to gather data to help guide work on improving the usability of the
organization and UI presentation of Projects. The specific goals for the
usability study was to find out about problem areas in relation to the
completion of common tasks, such as creating a project, opening and closing
projects, pointing to sources and resources, and building, compiling and
running projects. The specific study objectives were as follows.
Test Objectives
To uncover users' understandings and expectations of working with Projects
To discover the paths that users take to access the necessary functionality
of Projects to complete the tasks
To uncover users views of functionality currently being implemented
To discover users views of the current (prototype) implementation of Projects
in NetBeans, as compared with their usual IDE (whether NetBeans / Forte
for Java, or a competitive product.
To gather data that will assist in improved design ideas
To gather usability issues relation to NetBeans (not exclusively relating
to the Projects implementation)
Procedure
During the study participants were asked to complete 12 tasks, and were
asked a number of questions regarding their experiences of completing those
tasks. The scenario used for the tasks was centered around working in a
new job at a firm that produces entertaining and educational Java applications
for kids. The participants had to use the NetBeans Project Manager Prototype
to complete the tasks. Participants worked pseudo-independently with the
facilitator seated at their side. The study was conducted on a 1.4 GHz
Dell PC running Windows 2000. The IDE was installed and open, with
the Welcome window open when the participants came to start the tasks.
Tests ran for up to 2 hours each and included the following tasks:
An informal pretest interview
Opportunity to browse the IDE
Creating a project, and running it
Finding the compiled classes
Opening a project
Creating 2 packages and linking the files in those packages
Running and closing the project
Creating 2 project groupings
Creating a new project inside the project grouping
Pointing to sources for the project just created and running the application
Pointing to a jar file from the project and changing the code so that the
project uses the new resource
Running the application to be sure that the new resources are used
Suggesting potential ways to ensure that a project has the most up-to-date
dependent resources
Closing and opening projects
Switching a project to the physical view from the logical view
Closing projects groupings
Final questions: comparisons with other IDE's, including NetBeans, FFJ
where relevant.
Participants were asked to "think aloud" as they completed the tasks.
Participants
A total of 6 Java programmers, participated in the study.
Screening for the study required all participants to:
have recently used an IDE (for any programming language) for more than
1 year
use their IDE at least twice a week
have used Java for 1 year or longer
Screening also required that half of the participants to:
have had recent experience with NetBeans or Forte for Java for at least
6 months duration
to use Forte for Java or NetBeans at least twice a week
4 participants were external and 2 were internal (Sun employees). 3 participants
(P0, P2, P5) were non NetBeans / Forte for Java users, and 3 participants
(P2, P3, P4) had at least 6 months FFJ or NetBeans experience. See the
table below for details of participant's Java and IDE experience.
Participant Number
Length of Java Experience
IDE's Used & Length
of Experience
P0
4-5 years
Visual Cafe (currently, 1 - 2 years), IBM Visual
Age, JBuilder (a long time ago)
P1
3 - 4 years
FFJ / ant (currently, 6 - 7 months)
Metrowerks CodeWarrior up to 7 months ago.
P2
2.5 - 3 years
JBuilder (1 year). Visual Age for Java (1 year
ago, few months).
P3
3 years
JDeveloper (currently at work), NetBeans (currently, 1
year - personal time) JEdit.
P4
5 years
FFJ (currently, 1 year)
Visual Age for Java (1 year ago, 2 years)
P5
5 years
Visual Studio, Visual Basic. Hasn't used Java since being
at sun: last 3 months!
Materials
The following materials were used during this study:
The Projects
Prototype (Note: due to licensing issues, this version is slightly
older than the one used in the study. However, it is substantially
the same.)
If you would prefer to avoid the trouble of downloading and running the
prototype, please visit the screenshots page
to see what the IDE looked like to study participants.
Main Results
This section contains lists of all the main issues encountered during the
usability test in tabular form below.
Issue: Startup materials are helpful,
particularly the Welcome screen.
Recommendation(s): none needed.
0, 1, 5
CREATING
/ OPENING / CLOSING PROJECTS
Creating a New Project (Under Projects
Node)
Description and Recommendations
Participants
Resolution
Issue: People expect a New in
the Project menu.
Recommendation(s): See below.
1, 3, 4
Issue: NetBeans' extreme context
sensitivity can complicate project creation: unless the user has selected
exactly the right node, she gets the wrong set of templates.
Recommendation(s): Issue 20133
proposes a solution for the context sensitivity problem. It involves making
File|New context-insensitive, so that the user could always find the new
projects templates. In addition, we should consider having a New Projects
menu item, possibly under the Projects menu. Although, this solution may
violate the Java Look and Feel Design Guidelines, and therefore should
be explored as a last resort.
Issue: It
is possible to create an orphaned file that doesn't seem to belong to
a project.
Recommendation(s): This just seems like a bug, and will be filed
as such. Whenever the user creates a new file, it should always be within
the context of an existing project. Although, we might let the user open
an existing file for browsing purposes, without having to attach it to
an existing project. The user would not be able to do anything significant
with that file, however.
0, 3
Creating a Project Group / Comments about
Project Groupings
Description and Recommendations
Participants
Resolution
Issue: Redesign New screen that
still asks for package information when creating a group.
Recommendation(s): This is a known problem, and is being worked on.
1, 5
Issue: Most people felt it was
a reasonable idea.
Recommendation(s): none needed.
0, 2, 3, 4
Issue: Consider augmenting project groups
with more useful functionality. For example, the project group might contain
resources that all projects need.
Proposed Resolution: This is an interesting idea, but one that
might have to wait for a post-4.0 release.
4
Creating a New Project (Under a Project
Group Node)
Description and Recommendations
Participants
Resolution
Issue: Users right-click on a
project group, select New, and get a simple list of available project types.
Unfortunately, some users failed to understand that these were project
templates.
Recommendation(s): This problem will probably be resolved when
we have multiple project types, as each type will have its own folder;
the folder's name (e.g., "Java Projects", "Web Projects") should tell users
that these are projects. In addition, Docs may want to change the name
of the project templates to more clearly indicate that they are projects,
e.g., "Java Application" --> "Simple Java Project".
1,3,5
Issue: If the project group isn't
selected, then the new project gets created outside of that group and there
is no way for the user to rectify this without exiting and restarting the
wizard.
Recommendation(s): The desired project group should be included
as an option in the New Project wizard. If a project group is selected,
then that value should be filled in on the wizard.
0,1,2,3,4,5
Opening the Project into the IDE
Description and Recommendations
Participants
Resolution
Issue: Opening projects from the
File menu is expected, but we need to make sure that it's obvious how to
do this, e.g., File|Open Project.
Recommendation(s): See below.
0,1,2,4,5
Issue: People also expect Project
Open to be under the Project menu.
Recommendation(s): Have an "Open..."
menu that shows a JFileChooser, and an "Open Project..." menu that shows
a dialog for opening (closed) projects. (Note that this dialog has already
been designed; a version is in the current Projects
Infrastructure UI Spec but needs to be revised.) Consider putting the
"Open Project" menu item under the Project menu. Although, this solution
may violate the Java Look and Feel Design Guidelines, and therefore should
be explored as a last resort.
Issue: In general, people did
OK on opening a project from within a group.
Recommendation(s): None needed.
2, 3, 4
Issue: Need to decide whether
a reopened project should appear under its old group or not.
Recommendation(s): This applies to the use of the MRU list to
reopen a project. A project should be re-opened in the last place where
it was closed. Therefore, if project P1 is open in group G1 and G2, and
then the user closes it in G1 and then G2, then reopening P1 from the MRU
should reopen it in G2.
1
Issue: Consider making it more
convenient to reopen a project under a group, e.g., by NOT forcing users
to search through the filesystem to reopen the project.
Recommendation(s): The user should not have to search for project
or group files in order to reopen a project; rather, they should be able
to use the dialog discussed in the Projects
Infrastructure UI Spec.
Closing Projects (in explicit task, plus
in participant-initiated cases)
Description
Participants
Resolution
Issue: Mostly this worked just
fine, but should consider providing Close elsewhere, e.g., under File or
Project since some users expect it there.
Recommendation(s): If the File menu isn't too long, a Close menu
item would be a sensible addition.
0,1,2,3,4,5
CREATING
/ ADDING SOURCE DIRECTORIES AND PACKAGES
Creating a New Source Directory
Description
Participants
Resolution
Issue: The Add Source Directory
dialog is confusing.
Recommendation(s): This dialog needs to be replaced with a JFileChooser;
this is a known issue which is being worked.
0,1,2
Issue: New Sources dialog in the
New Project wizard is confusing.
Recommendation(s): Currently, the IDE has two mechanisms for
getting source directories: the concept of adding an existing directory
and the concept of creating a new directory under the project's home directory.
We should consider unifying these two mechanisms. One way to do this is
to have a single "Add Source Directory" function, which shows a JFileChooser
dialog. The default location for the JFileChooser would be the project's
home directory; the user would have the choice of adding/creating any arbitrary
directory. If we do this, we need to make sure that users know they can
create directories with the JFileChooser, and that the directory creation
process is easy.
0,1
Issue: Not having a default source
directory causes mischief, including orphaned files.
Recommendation(s): Most projects
should come with a pre-created source directory under the project's home
directory. In addition, the process of creating a new file should automatically
add the file's directory to the sources node. As mentioned elsewhere, the
orphaned file problem is a bug.
2
Issue: If a project is selected
and the user does a File|New, the default location presented to her in
the second wizard page is the project's home directory, but it's under
a node called, "Default System".
Recommendation(s): Presumably this is a bug, since it probably
wasn't intentional to expose NetBeans' guts like this.
0,2,4
Issue: The Filesystems tab is
going to be a problem for legacy users.
Recommendation(s): First, the Filesystems tab, as we have discussed,
needs to be substantially reduced in functionality so that users can't
get themselves into trouble. For example, users probably shouldn't be able
to mount subdirectories of a currently mounted filesystem. Second, we need
to consider names for the Filesystems tab that communicate what it does
without making legacy users think that this is their "escape hatch". Perhaps,
"Filesystems Browser"?
4
Adding an Existing Source Directory
Description
Participants
Resolution
Issue: Some users have an expectation
that sources can be added by using Open File.
Recommendation(s): Opening an arbitrary file should add its source
directory to the Java Sources node.
0,1,2,3,4
Issue: The Add Source Directory
dialog is confusing.
Recommendation(s): This dialog needs to be replaced with a JFileChooser;
this is a known issue which is being worked.
0,1,3,5
Issue: Some
users expect to be able to add individual files under the Java Sources
node, rather than adding a whole directory.
Recommendation(s): The easy way to resolve this is to provide
a look for the Java Sources node that simply hides the individual source
directories. The harder way to resolve this is to allow the user to add
both individual files and directories. The latter will require resolving
some UI and development issues.
0,1,2,3,4,5
Issue: IDE doesn't detect the
presence of new directories that have been externally placed into a project
source directory; the IDE must be restarted for the change to be detected.
(Bug)
Recommendation(s): A bug will be filed.
1
Issue: Consider giving users more
paths to create or add source files.
Recommendation(s): File|New and the pop-up menus on the sources
node are probably enough. But, if it doesn't cause too much clutter, it
might be reasonable to let the user create new source files and add directories
from the project node's pop-up menu. Also, as discussed, the Filesystems
Browser should let users easily add a source directory/file to a project.
2,3
Issue: Be able to handle the situation
where the user tries to have source directory = project home directory.
Recommendation(s): When one user did this, the IDE got confused,
threw exceptions, etc. We need to gracefully handle whatever types of source
directories the user might try to add (regardless of whether or not we
let them do that).
4
Issue: Users expect Resources
node to hold non-Java objects, e.g., images, locale data.
Recommendation(s): This requires some careful thought before
we create an artificial split between "real" source files (Java files,
Xml files) and "other" source files (bitmaps, sounds, text files). A better
strategy might be to rethink the name, "Resources", and replace it with
something more accurate. This is probably an issue for docs.
0 (+ see participants responses to questions
after adding dictionary.jar)
Creating packages
Description
Participants
Resolution
Issue: Consider some alternative
ways to create new packages, e.g., by renaming the default package or by
picking up the package from the package statement in the source file (as
in Eclipse).
Recommendation(s): These are more ideas than anything else. They
should be added to the effort to make the IDE more flexible in working
with Java packages.
0,3
Issue: If someone creates a folder
(that is a legitimate Java package) rather than a package, they may be
confused about whether or not they really did create a package.
Recommendation(s): Move the Java Packages menu item up in the
template list, so that it is one of the first things that the user sees.
Also, consider adding some explanatory text to the dialog(s) that create
new folders. The text should say that a folder that has a legal Java package
name will be treated as a package.
0, 4,5
Issue: The user can click on the
project node or the Java Sources node and then use File|New to create a
new package. But if the package is created anywhere but under an existing
source directory, it does not appear in the project.
Recommendation(s): Either constrain package creation to be in
the existing source directories, or automatically add the package's parent
directory when the user creates it.
5
WORKING
WITH SOURCE FILES
New class / Main Wizard
Description
Participants
Resolution
Issue: Package declaration in
new class wizard is wrong. (bug)
Recommendation(s): Will be filed as a bug.
1
Issue: It is sometimes not clear
where package information is kept before the participant has seen the package
node(s).
Recommendation(s): Only one participant had difficulty with this,
but it is something to consider. One idea, besides beefing up the help
materials, is to always show the <default package> under the Java Sources
node.
2
Moving Files into Packages
Description
Participants
Resolution
Issue: Need drag and drop for
moving source files around.
Recommendation(s): A known problem that is being worked on.
0,1,3,4,5
Issue: Consider other ways to
move files into/out of packages, e.g., right-clicking on package or file
or using properties.
Recommendation(s): Being unable to drag and drop, a majority
of the users tried other ways to move the files between packages. Some
of these should be investigated, particularly adding a "Move" pop-up menu
to files.
0,3,4
Viewing Source
Description
Participants
Resolution
Issue: When Form Editor pops up
and obscures the code window, it can confuse users.
Recommendation(s): By default open the code, but have an "Edit
GUI" menu item for objects that have GUIs.
0,1,2,4,5
BUILDING
/ COMPILING / EXECUTING
Building and Compiling
Description
Participants
Resolution
Issue: Orphaned files don't get compiled.
Recommendation(s): As discussed elsewhere, orphaned files should
not be permitted.
0,4
Issue: Not all users could easily set the main class
using the property sheet.
Recommendation(s): The warning dialog that tells the user to
set the project's main class was a stopgap that was put in place for the
prototype. A better approach would be to display a dialog that lets the
user select the main class. Possibly even better would be to detect when
a) the project has no main class, and b) the user has just created a class
with a main method. Then, the user could be asked if the newly created
class should be the project's main class.
4
Issue: Startup materials are important.
Recommendation(s): None. This is just the observation that the
startup materials like the Welcome screen, the tips of the day and the
sample project were used.
0,5
Issue: Execute project needs to be salient.
Recommendation(s): Only one user had problems with this. It will
probably suffice to provide the Execute Project menu item with an icon.
5
Issue: Be able to handle the situation where the
user clicks the class file and does an execute.
Recommendation(s): We should probably
permit the class file to be executed, or at least a good explanation why
we won't let the users do it.
0,5
Finding the Compiled Class
Description
Participants
Resolution
Issue: This worked out well.
Recommendation(s): None needed.
0,1,2,3,4,5
Issue: Need to make the physical location
of the compiled class files discoverable.
Recommendation(s): We could let the
user rely on the physical view. A better solution would be to use tooltips,
or to let the user see the full directory path for the output directories
upon request.
0, 2
RESOURCES
Using an External Library
Description
Participants
Resolution
Issue: In general people had a bit of difficulty
finding this functionality, but once they found it everything worked out
fine.
Recommendation(s): Consider adding more paths to add a resource.
So, for example, the user should be able to add a resource through the
project node's pop-up menu.
0,1,2,3,4,5
Issue: Some users are unsure whether the Jar file
is on the classpath, as there is no feedback indicating that it is, aside
from the existence of the node under Resources.
Recommendation(s): Provide an easy way for the user to view the
entire classpath. This might be done in the project's customizer, or perhaps
on the Resource node's customizer. An alternative would be to have a special
"Current Classpath" menu item on the Resource node's pop-up menu. It would
give a read-only presentation of the current classpath.
0,1,4,5
Issue: At least one person was confused about whether
a resource was being created or added. This might just have been because
of the default name.
Recommendation(s): Consider choosing a different default name
(e.g., the name of the Jar file or directory), or saving the rename function
for later (i.e., after the wizard has been dismissed).
1
Issue: Users expect Resources node to hold non-Java
objects, e.g., images, locale data.
Recommendation(s): This requires some careful thought before
we create an artificial split between "real" source files (.java files,
.xml files) and "other" source files (bitmaps, sounds, text files). A better
strategy might be to rethink the name, "Resources", and replace it with
something more accurate. This is probably an issue for docs.
3,4,5
Resource Dependencies (Discussion)
Description
Participants
Resolution
Issue: Most people guessed that dependencies would
be expressed under the Resources node, or under a similar, parallel node.
However, there was some divergence about exactly how things would be arranged
under Resources.
Recommendation(s): Exactly what to show when another project's
target is used under the Resources node is the focus of ongoing design
work.
1,2,3,5
LOGICAL
AND PHYSICAL GROUPINGS
Description
Participants
Resolution
Issue: Users had some difficulty finding the switch
for logical/physical view. Consider making this more salient.
Recommendation(s): Presumably the work on the Looks
UI will solve this.
Issue: Users understand the difference between physical
and logical views.
Recommendation(s): None needed.
0,1,2,3,4,5
Issue: Users favor the logical view for their work.
Recommendation(s): None needed.
0,1,2,3,4,5
Issue: Users expect the same functionality to be
available in the two different views.
Recommendation(s): None needed.
1,2,3,5
Issue: At least one user wants to filter the files
displayed in the project.
Recommendation(s): None needed. This should be doable with the
IDE's fileset ability.
2
MISCELLANEOUS
ISSUES
Description
Participants
Resolution
Issue: At least two people expressed a wish for
a beefier Projects menu with more functionality.
Recommendation(s): The Projects menu still needs a final round
of design work.
0,3
Issue: One participant commented that presenting
packages individually might cause clutter, since each subpackage will create
an entirely new node.
Recommendation(s): The participant suggested collapsing the package
view, so that subpackages are presented more like subfolders. If package
clutter turns out to be a problem, this solution should be considered as
an alternative view.
2
Issue: Be able to handle the situation where the
user wants to delete the output folder from the Output node.
Recommendation(s): One solution is to just replace the output
node if it's deleted. However, this seems a bit odd, as the user might
have really meant to delete the node. A better solution would be to detect
that the deletion is being done, and to display a dialog asking the user
to select a new output node.
2
Conclusions
This usability study has uncovered some issues that need to be addressed
as the prototype is developed. Fortunately, most of these issues
are limited in scope and in impact; their solutions are relatively clear,
or have been known for some time. With these issues addressed, the
new projects system is poised to become a real asset to the IDE by orienting
new users and by increasing the productivity of experienced users.
