NetBeans Project Manager
A Usability Study of the First Prototype
June 4, 2002
Executive Summary
This usability study tested the first prototype of the new NetBeans projects system, using some developers who were familiar with NetBeans/Forte and some who were not. By and large, participants understood and liked the project system's core concepts, e.g., the Java Sources node, the Resources node and the Output node. Most of their difficulties were because of prototype limitations (e.g., the inability to use JFileChooser to open files) or because of known NetBeans problems (e.g., its extreme context sensitivity).High Level Findings:
- Participants found that the start-up materials on the welcome screen were useful for orienting to the IDE.
- Some difficulties were experienced in creating, opening, and closing projects. These usability problems resulted from having implemented only limited paths to carry out these actions, from NetBeans extreme context sensitivity, and the current capability of creating orphaned files not belonging to a project.
- When users tried to create, or add source directories or packages they were confused by some of the dialogs they encountered. In addition, we found that legacy users are likely to be tempted to resort to previous common actions, such as mount filesystem, rather than use the projects implementation, particularly when experiencing difficulty in proceeding. Such actions will likely cause further problems. Participants attempted to take more varied paths than currently implemented for creating / adding source directories or packages.
- While working with source files, participants attempted drag and drop first in order to move them about in the project.
- Building and compiling was generally straight-forward for participants with only a few minor issues encountered. All participants were easily able to find the location of their compiled file in the logical view presented in the NetBeans explorer.
- Participants encountered a few difficulties when working with resources, for example, they attempted to add a .jar resource to their project in multiple, currently unsupported, ways. Once they found the correct path, they were able to easily add the resource. Participants expressed some doubt about adding the resource, reqesting more confirmation that the resource was on the classpath.
- The functionality to switch from logical to physical views of files had low discoverability; but once found, participants understood and liked the distinction.
Introduction
The NetBeans Projects team conducted a usability study in May 2002, in California. The usability study tested the prototype of the new implementation of Projects in NetBeans. (Visit the screenshots page to see what the prototype looks like.) The overriding goal was to gather data to help guide work on improving the usability of the organization and UI presentation of Projects. The specific goals for the usability study was to find out about problem areas in relation to the completion of common tasks, such as creating a project, opening and closing projects, pointing to sources and resources, and building, compiling and running projects. The specific study objectives were as follows.Test Objectives
- To uncover users' understandings and expectations of working with Projects
- To discover the paths that users take to access the necessary functionality of Projects to complete the tasks
- To uncover users views of functionality currently being implemented
- To discover users views of the current (prototype) implementation of Projects in NetBeans, as compared with their usual IDE (whether NetBeans / Forte for Java, or a competitive product.
- To gather data that will assist in improved design ideas
- To gather usability issues relation to NetBeans (not exclusively relating to the Projects implementation)
Procedure
During the study participants were asked to complete 12 tasks, and were asked a number of questions regarding their experiences of completing those tasks. The scenario used for the tasks was centered around working in a new job at a firm that produces entertaining and educational Java applications for kids. The participants had to use the NetBeans Project Manager Prototype to complete the tasks. Participants worked pseudo-independently with the facilitator seated at their side. The study was conducted on a 1.4 GHz Dell PC running Windows 2000. The IDE was installed and open, with the Welcome window open when the participants came to start the tasks. Tests ran for up to 2 hours each and included the following tasks:- An informal pretest interview
- Opportunity to browse the IDE
- Creating a project, and running it
- Finding the compiled classes
- Opening a project
- Creating 2 packages and linking the files in those packages
- Running and closing the project
- Creating 2 project groupings
- Creating a new project inside the project grouping
- Pointing to sources for the project just created and running the application
- Pointing to a jar file from the project and changing the code so that the project uses the new resource
- Running the application to be sure that the new resources are used
- Suggesting potential ways to ensure that a project has the most up-to-date dependent resources
- Closing and opening projects
- Switching a project to the physical view from the logical view
- Closing projects groupings
- Final questions: comparisons with other IDE's, including NetBeans, FFJ where relevant.
Participants
A total of 6 Java programmers, participated in the study.Screening for the study required all participants to:
- have recently used an IDE (for any programming language) for more than 1 year
- use their IDE at least twice a week
- have used Java for 1 year or longer
- have had recent experience with NetBeans or Forte for Java for at least 6 months duration
- to use Forte for Java or NetBeans at least twice a week
Participant Number | Length of Java Experience | IDE's Used & Length of Experience |
P0 | 4-5 years | Visual Cafe (currently, 1 - 2 years), IBM Visual Age, JBuilder (a long time ago) |
P1 | 3 - 4 years | FFJ / ant (currently, 6 - 7 months)
Metrowerks CodeWarrior up to 7 months ago. |
P2 | 2.5 - 3 years | JBuilder (1 year). Visual Age for Java (1 year ago, few months). |
P3 | 3 years | JDeveloper (currently at work), NetBeans (currently, 1 year - personal time) JEdit. |
P4 | 5 years | FFJ (currently, 1 year)
Visual Age for Java (1 year ago, 2 years) |
P5 | 5 years | Visual Studio, Visual Basic. Hasn't used Java since being at sun: last 3 months! |
Materials
The following materials were used during this study:- Task List
- The Projects Prototype (Note: due to licensing issues, this version is slightly older than the one used in the study. However, it is substantially the same.)
Main Results
This section contains lists of all the main issues encountered during the usability test in tabular form below.For a the data supporting each issue, see the detailed
results.
|
||
Description and Recommendations | Participants | Resolution |
Issue: Startup materials are helpful,
particularly the Welcome screen.
Recommendation(s): none needed.
|
0, 1, 5 | |
|
||
|
||
Description and Recommendations | Participants | Resolution |
Issue: People expect a New in
the Project menu.
Recommendation(s): See below.
|
1, 3, 4 | |
Issue: NetBeans' extreme context
sensitivity can complicate project creation: unless the user has selected
exactly the right node, she gets the wrong set of templates.
Recommendation(s): Issue 20133
proposes a solution for the context sensitivity problem. It involves making
File|New context-insensitive, so that the user could always find the new
projects templates. In addition, we should consider having a New Projects
menu item, possibly under the Projects menu. Although, this solution may
violate the Java Look and Feel Design Guidelines, and therefore should
be explored as a last resort.
|
0, 1, 2, 4, 5 | Issue 20133 |
Issue: It
is possible to create an orphaned file that doesn't seem to belong to
a project.
Recommendation(s): This just seems like a bug, and will be filed
as such. Whenever the user creates a new file, it should always be within
the context of an existing project. Although, we might let the user open
an existing file for browsing purposes, without having to attach it to
an existing project. The user would not be able to do anything significant
with that file, however.
|
0, 3 | |
|
||
Description and Recommendations |
|
|
Issue: Redesign New screen that
still asks for package information when creating a group.
Recommendation(s): This is a known problem, and is being worked on.
|
1, 5 | |
Issue: Most people felt it was
a reasonable idea.
Recommendation(s): none needed.
|
0, 2, 3, 4 | |
Issue: Consider augmenting project groups
with more useful functionality. For example, the project group might contain
resources that all projects need.
Proposed Resolution: This is an interesting idea, but one that
might have to wait for a post-4.0 release.
|
4 | |
|
||
Description and Recommendations |
|
|
Issue: Users right-click on a
project group, select New, and get a simple list of available project types.
Unfortunately, some users failed to understand that these were project
templates.
Recommendation(s): This problem will probably be resolved when
we have multiple project types, as each type will have its own folder;
the folder's name (e.g., "Java Projects", "Web Projects") should tell users
that these are projects. In addition, Docs may want to change the name
of the project templates to more clearly indicate that they are projects,
e.g., "Java Application" --> "Simple Java Project".
|
1,3,5 | |
Issue: If the project group isn't
selected, then the new project gets created outside of that group and there
is no way for the user to rectify this without exiting and restarting the
wizard.
Recommendation(s): The desired project group should be included
as an option in the New Project wizard. If a project group is selected,
then that value should be filled in on the wizard.
|
0,1,2,3,4,5 | |
|
||
Description and Recommendations |
|
|
Issue: Opening projects from the
File menu is expected, but we need to make sure that it's obvious how to
do this, e.g., File|Open Project.
Recommendation(s): See below.
|
0,1,2,4,5 | |
Issue: People also expect Project
Open to be under the Project menu.
Recommendation(s): Have an "Open..."
menu that shows a JFileChooser, and an "Open Project..." menu that shows
a dialog for opening (closed) projects. (Note that this dialog has already
been designed; a version is in the current Projects
Infrastructure UI Spec but needs to be revised.) Consider putting the
"Open Project" menu item under the Project menu. Although, this solution
may violate the Java Look and Feel Design Guidelines, and therefore should
be explored as a last resort.
|
0,1,3 | Projects Infrastructure UI Spec |
|
||
|
|
|
Issue: In general, people did
OK on opening a project from within a group.
Recommendation(s): None needed.
|
2, 3, 4 | |
Issue: Need to decide whether
a reopened project should appear under its old group or not.
Recommendation(s): This applies to the use of the MRU list to
reopen a project. A project should be re-opened in the last place where
it was closed. Therefore, if project P1 is open in group G1 and G2, and
then the user closes it in G1 and then G2, then reopening P1 from the MRU
should reopen it in G2.
|
1 | |
Issue: Consider making it more
convenient to reopen a project under a group, e.g., by NOT forcing users
to search through the filesystem to reopen the project.
Recommendation(s): The user should not have to search for project
or group files in order to reopen a project; rather, they should be able
to use the dialog discussed in the Projects
Infrastructure UI Spec.
|
5 | Projects Infrastructure UI Spec |
|
||
|
|
|
Issue: Mostly this worked just
fine, but should consider providing Close elsewhere, e.g., under File or
Project since some users expect it there.
Recommendation(s): If the File menu isn't too long, a Close menu
item would be a sensible addition.
|
0,1,2,3,4,5 | |
|
||
|
||
|
|
|
Issue: The Add Source Directory
dialog is confusing.
Recommendation(s): This dialog needs to be replaced with a JFileChooser;
this is a known issue which is being worked.
|
0,1,2 | |
Issue: New Sources dialog in the
New Project wizard is confusing.
Recommendation(s): Currently, the IDE has two mechanisms for
getting source directories: the concept of adding an existing directory
and the concept of creating a new directory under the project's home directory.
We should consider unifying these two mechanisms. One way to do this is
to have a single "Add Source Directory" function, which shows a JFileChooser
dialog. The default location for the JFileChooser would be the project's
home directory; the user would have the choice of adding/creating any arbitrary
directory. If we do this, we need to make sure that users know they can
create directories with the JFileChooser, and that the directory creation
process is easy.
|
0,1 | |
Issue: Not having a default source
directory causes mischief, including orphaned files.
Recommendation(s): Most projects
should come with a pre-created source directory under the project's home
directory. In addition, the process of creating a new file should automatically
add the file's directory to the sources node. As mentioned elsewhere, the
orphaned file problem is a bug.
|
2 | |
Issue: If a project is selected
and the user does a File|New, the default location presented to her in
the second wizard page is the project's home directory, but it's under
a node called, "Default System".
Recommendation(s): Presumably this is a bug, since it probably
wasn't intentional to expose NetBeans' guts like this.
|
0,2,4 | |
Issue: The Filesystems tab is
going to be a problem for legacy users.
Recommendation(s): First, the Filesystems tab, as we have discussed,
needs to be substantially reduced in functionality so that users can't
get themselves into trouble. For example, users probably shouldn't be able
to mount subdirectories of a currently mounted filesystem. Second, we need
to consider names for the Filesystems tab that communicate what it does
without making legacy users think that this is their "escape hatch". Perhaps,
"Filesystems Browser"?
|
4 | |
|
||
|
|
|
Issue: Some users have an expectation
that sources can be added by using Open File.
Recommendation(s): Opening an arbitrary file should add its source
directory to the Java Sources node.
|
0,1,2,3,4 | |
Issue: The Add Source Directory
dialog is confusing.
Recommendation(s): This dialog needs to be replaced with a JFileChooser;
this is a known issue which is being worked.
|
0,1,3,5 | |
Issue: Some
users expect to be able to add individual files under the Java Sources
node, rather than adding a whole directory.
Recommendation(s): The easy way to resolve this is to provide
a look for the Java Sources node that simply hides the individual source
directories. The harder way to resolve this is to allow the user to add
both individual files and directories. The latter will require resolving
some UI and development issues.
|
0,1,2,3,4,5 | |
Issue: IDE doesn't detect the
presence of new directories that have been externally placed into a project
source directory; the IDE must be restarted for the change to be detected.
(Bug)
Recommendation(s): A bug will be filed.
|
1 | |
Issue: Consider giving users more
paths to create or add source files.
Recommendation(s): File|New and the pop-up menus on the sources
node are probably enough. But, if it doesn't cause too much clutter, it
might be reasonable to let the user create new source files and add directories
from the project node's pop-up menu. Also, as discussed, the Filesystems
Browser should let users easily add a source directory/file to a project.
|
2,3 | |
Issue: Be able to handle the situation
where the user tries to have source directory = project home directory.
Recommendation(s): When one user did this, the IDE got confused,
threw exceptions, etc. We need to gracefully handle whatever types of source
directories the user might try to add (regardless of whether or not we
let them do that).
|
4 | |
Issue: Users expect Resources
node to hold non-Java objects, e.g., images, locale data.
Recommendation(s): This requires some careful thought before
we create an artificial split between "real" source files (Java files,
Xml files) and "other" source files (bitmaps, sounds, text files). A better
strategy might be to rethink the name, "Resources", and replace it with
something more accurate. This is probably an issue for docs.
|
0 (+ see participants responses to questions after adding dictionary.jar) | |
|
||
|
|
|
Issue: Consider some alternative
ways to create new packages, e.g., by renaming the default package or by
picking up the package from the package statement in the source file (as
in Eclipse).
Recommendation(s): These are more ideas than anything else. They
should be added to the effort to make the IDE more flexible in working
with Java packages.
|
0,3 | |
Issue: If someone creates a folder
(that is a legitimate Java package) rather than a package, they may be
confused about whether or not they really did create a package.
Recommendation(s): Move the Java Packages menu item up in the
template list, so that it is one of the first things that the user sees.
Also, consider adding some explanatory text to the dialog(s) that create
new folders. The text should say that a folder that has a legal Java package
name will be treated as a package.
|
0, 4,5 | |
Issue: The user can click on the
project node or the Java Sources node and then use File|New to create a
new package. But if the package is created anywhere but under an existing
source directory, it does not appear in the project.
Recommendation(s): Either constrain package creation to be in
the existing source directories, or automatically add the package's parent
directory when the user creates it.
|
5 | |
|
||
|
||
|
|
|
Issue: Package declaration in
new class wizard is wrong. (bug)
Recommendation(s): Will be filed as a bug.
|
1 | |
Issue: It is sometimes not clear
where package information is kept before the participant has seen the package
node(s).
Recommendation(s): Only one participant had difficulty with this,
but it is something to consider. One idea, besides beefing up the help
materials, is to always show the <default package> under the Java Sources
node.
|
2 | |
|
||
|
|
|
Issue: Need drag and drop for
moving source files around.
Recommendation(s): A known problem that is being worked on.
|
0,1,3,4,5 | |
Issue: Consider other ways to
move files into/out of packages, e.g., right-clicking on package or file
or using properties.
Recommendation(s): Being unable to drag and drop, a majority of the users tried other ways to move the files between packages. Some of these should be investigated, particularly adding a "Move" pop-up menu to files. |
0,3,4 | |
|
||
|
|
|
Issue: When Form Editor pops up
and obscures the code window, it can confuse users.
Recommendation(s): By default open the code, but have an "Edit
GUI" menu item for objects that have GUIs.
|
0,1,2,4,5 | |
|
||
|
||
|
|
|
Issue: Orphaned files don't get compiled.
Recommendation(s): As discussed elsewhere, orphaned files should
not be permitted.
|
0,4 | |
Issue: Not all users could easily set the main class
using the property sheet.
Recommendation(s): The warning dialog that tells the user to
set the project's main class was a stopgap that was put in place for the
prototype. A better approach would be to display a dialog that lets the
user select the main class. Possibly even better would be to detect when
a) the project has no main class, and b) the user has just created a class
with a main method. Then, the user could be asked if the newly created
class should be the project's main class.
|
4 | |
Issue: Startup materials are important.
Recommendation(s): None. This is just the observation that the
startup materials like the Welcome screen, the tips of the day and the
sample project were used.
|
0,5 | |
Issue: Execute project needs to be salient.
Recommendation(s): Only one user had problems with this. It will
probably suffice to provide the Execute Project menu item with an icon.
|
5 | |
Issue: Be able to handle the situation where the
user clicks the class file and does an execute.
Recommendation(s): We should probably
permit the class file to be executed, or at least a good explanation why
we won't let the users do it.
|
0,5 | |
|
||
|
|
|
Issue: This worked out well.
Recommendation(s): None needed.
|
0,1,2,3,4,5 | |
Issue: Need to make the physical location
of the compiled class files discoverable.
Recommendation(s): We could let the
user rely on the physical view. A better solution would be to use tooltips,
or to let the user see the full directory path for the output directories
upon request.
|
0, 2 | |
|
||
|
||
|
|
|
Issue: In general people had a bit of difficulty
finding this functionality, but once they found it everything worked out
fine.
Recommendation(s): Consider adding more paths to add a resource.
So, for example, the user should be able to add a resource through the
project node's pop-up menu.
|
0,1,2,3,4,5 | |
Issue: Some users are unsure whether the Jar file
is on the classpath, as there is no feedback indicating that it is, aside
from the existence of the node under Resources.
Recommendation(s): Provide an easy way for the user to view the
entire classpath. This might be done in the project's customizer, or perhaps
on the Resource node's customizer. An alternative would be to have a special
"Current Classpath" menu item on the Resource node's pop-up menu. It would
give a read-only presentation of the current classpath.
|
0,1,4,5 | |
Issue: At least one person was confused about whether
a resource was being created or added. This might just have been because
of the default name.
Recommendation(s): Consider choosing a different default name
(e.g., the name of the Jar file or directory), or saving the rename function
for later (i.e., after the wizard has been dismissed).
|
1 | |
Issue: Users expect Resources node to hold non-Java
objects, e.g., images, locale data.
Recommendation(s): This requires some careful thought before
we create an artificial split between "real" source files (.java files,
.xml files) and "other" source files (bitmaps, sounds, text files). A better
strategy might be to rethink the name, "Resources", and replace it with
something more accurate. This is probably an issue for docs.
|
3,4,5 | |
|
||
|
|
|
Issue: Most people guessed that dependencies would
be expressed under the Resources node, or under a similar, parallel node.
However, there was some divergence about exactly how things would be arranged
under Resources.
Recommendation(s): Exactly what to show when another project's
target is used under the Resources node is the focus of ongoing design
work.
|
1,2,3,5 | |
|
||
|
|
|
Issue: Users had some difficulty finding the switch
for logical/physical view. Consider making this more salient.
Recommendation(s): Presumably the work on the Looks
UI will solve this.
|
0,1,2,5 | Looks UI |
Issue: Users understand the difference between physical
and logical views.
Recommendation(s): None needed.
|
0,1,2,3,4,5 | |
Issue: Users favor the logical view for their work.
Recommendation(s): None needed.
|
0,1,2,3,4,5 | |
Issue: Users expect the same functionality to be
available in the two different views.
Recommendation(s): None needed.
|
1,2,3,5 | |
Issue: At least one user wants to filter the files
displayed in the project.
Recommendation(s): None needed. This should be doable with the
IDE's fileset ability.
|
2 | |
|
||
|
|
|
Issue: At least two people expressed a wish for
a beefier Projects menu with more functionality.
Recommendation(s): The Projects menu still needs a final round
of design work.
|
0,3 | |
Issue: One participant commented that presenting
packages individually might cause clutter, since each subpackage will create
an entirely new node.
Recommendation(s): The participant suggested collapsing the package
view, so that subpackages are presented more like subfolders. If package
clutter turns out to be a problem, this solution should be considered as
an alternative view.
|
2 | |
Issue: Be able to handle the situation where the
user wants to delete the output folder from the Output node.
Recommendation(s): One solution is to just replace the output
node if it's deleted. However, this seems a bit odd, as the user might
have really meant to delete the node. A better solution would be to detect
that the deletion is being done, and to display a dialog asking the user
to select a new output node.
|
2 |