NetBeans Project Manager

A Usability Study of the First Prototype

Dirk Ruiz, Jennifer Gove
June 4, 2002

Executive Summary

This usability study tested the first prototype of the new NetBeans projects system, using some developers who were familiar with NetBeans/Forte and some who were not.  By and large, participants understood and liked the project system's core concepts, e.g., the Java Sources node, the Resources node and the Output node.  Most of their difficulties were because of prototype limitations (e.g., the inability to use JFileChooser to open files) or because of known NetBeans problems (e.g., its extreme context sensitivity).

High Level Findings:

  • Participants found that the start-up materials on the welcome screen were useful for orienting to the IDE.
  • Some difficulties were experienced in creating, opening, and closing projects.  These usability problems resulted from having implemented only limited paths to carry out these actions, from NetBeans extreme context sensitivity, and the current capability of creating orphaned files not belonging to a project.
  • When users tried to create, or add source directories or packages they were confused by some of the dialogs they encountered.  In addition, we found that legacy users are likely to be tempted to resort to previous common actions, such as mount filesystem, rather than use the projects implementation, particularly when experiencing difficulty in proceeding.  Such actions will likely cause further problems.  Participants attempted to take more varied paths than currently implemented for creating / adding source directories or packages.
  • While working with source files, participants attempted drag and drop first in order to move them about in the project.
  • Building and compiling was generally straight-forward for participants with only a few minor issues encountered. All participants were easily able to find the location of their compiled file in the logical view presented in the NetBeans explorer.
  • Participants encountered a few difficulties when working with resources, for example, they attempted to add a .jar resource to their project in multiple, currently unsupported, ways. Once they found the correct path, they were able to easily add the resource. Participants expressed some doubt about adding the resource, reqesting more confirmation that the resource was on the classpath.
  • The functionality to switch from logical to physical views of files had low discoverability; but once found, participants understood and liked the distinction.

Introduction

The NetBeans Projects team conducted a usability study in May 2002, in California. The usability study tested the prototype of the new implementation of Projects in NetBeans. (Visit the screenshots page to see what the prototype looks like.)  The overriding goal was to gather data to help guide work on improving the usability of the organization and UI presentation of Projects. The specific goals for the usability study was to find out about problem areas in relation to the completion of common tasks, such as creating a project, opening and closing projects, pointing to sources and resources, and building, compiling and running projects.  The specific study objectives were as follows.

Test Objectives

  • To uncover users' understandings and expectations of working with Projects
  • To discover the paths that users take to access the necessary functionality of Projects to complete the tasks
  • To uncover users views of functionality currently being implemented
  • To discover users views of the current (prototype) implementation of Projects in NetBeans, as compared with their usual IDE (whether NetBeans / Forte for Java, or a competitive product.
  • To gather data that will assist in improved design ideas
  • To gather usability issues relation to NetBeans (not exclusively relating to the Projects implementation)

Procedure

During the study participants were asked to complete 12 tasks, and were asked a number of questions regarding their experiences of completing those tasks. The scenario used for the tasks was centered around working in a new job at a firm that produces entertaining and educational Java applications for kids. The participants had to use the NetBeans Project Manager Prototype to complete the tasks. Participants worked pseudo-independently with the facilitator seated at their side. The study was conducted on a 1.4 GHz Dell PC running Windows 2000.  The IDE was installed and open, with the Welcome window open when the participants came to start the tasks. Tests ran for up to 2 hours each and included the following tasks:
  • An informal pretest interview
  • Opportunity to browse the IDE
  • Creating a project, and running it
  • Finding the compiled classes
  • Opening a project
  • Creating 2 packages and linking the files in those packages
  • Running and closing the project
  • Creating 2 project groupings
  • Creating a new project inside the project grouping
  • Pointing to sources for the project just created and running the application
  • Pointing to a jar file from the project and changing the code so that the project uses the new resource
  • Running the application to be sure that the new resources are used
  • Suggesting potential ways to ensure that a project has the most up-to-date dependent resources
  • Closing and opening projects
  • Switching a project to the physical view from the logical view
  • Closing projects groupings
  • Final questions: comparisons with other IDE's, including NetBeans, FFJ where relevant.
Participants were asked to "think aloud" as they completed the tasks.

Participants

A total of 6 Java programmers, participated in the study.

Screening for the study required all participants to:

  • have recently used an IDE (for any programming language) for more than 1 year
  • use their IDE at least twice a week
  • have used Java for 1 year or longer
Screening also required that half of the participants to:
  • have had recent experience with NetBeans or Forte for Java for at least 6 months duration
  • to use Forte for Java or NetBeans at least twice a week
4 participants were external and 2 were internal (Sun employees). 3 participants (P0, P2, P5) were non NetBeans / Forte for Java users, and 3 participants (P2, P3, P4) had at least 6 months FFJ or NetBeans experience. See the table below for details of participant's Java and IDE experience.
 
Participant Number  Length of Java Experience IDE's Used & Length of Experience
P0 4-5 years Visual Cafe (currently, 1 - 2 years), IBM Visual Age, JBuilder (a long time ago) 
P1 3 - 4 years FFJ / ant (currently, 6 - 7 months)
Metrowerks CodeWarrior up to 7 months ago. 
P2 2.5 - 3 years JBuilder (1 year). Visual Age for Java (1 year ago, few months). 
P3 3 years JDeveloper (currently at work), NetBeans (currently, 1 year - personal time) JEdit. 
P4 5 years FFJ (currently, 1 year)
Visual Age for Java (1 year ago, 2 years)
P5 5 years Visual Studio, Visual Basic. Hasn't used Java since being at sun: last 3 months!

Materials

The following materials were used during this study:
  • Task List
  • The Projects Prototype  (Note: due to licensing issues, this version is slightly older than the one used in the study.  However, it is substantially the same.)
If you would prefer to avoid the trouble of downloading and running the prototype, please visit the screenshots page to see what the IDE looked like to study participants.

Main Results

This section contains lists of all the main issues encountered during the usability test in tabular form below.

For a the data supporting each issue, see the detailed results.
 

ORIENTING TO THE IDE
Description and Recommendations Participants Resolution
Issue: Startup materials are helpful, particularly the Welcome screen.

Recommendation(s): none needed.
 

0, 1, 5  
CREATING / OPENING / CLOSING PROJECTS
Creating a New Project (Under Projects Node) 
Description and Recommendations Participants Resolution
Issue: People expect a New in the Project menu. 

Recommendation(s): See below.
 

1, 3, 4  
Issue: NetBeans' extreme context sensitivity can complicate project creation: unless the user has selected exactly the right node, she gets the wrong set of templates. 

Recommendation(s): Issue 20133 proposes a solution for the context sensitivity problem. It involves making File|New context-insensitive, so that the user could always find the new projects templates. In addition, we should consider having a New Projects menu item, possibly under the Projects menu. Although, this solution may violate the Java Look and Feel Design Guidelines, and therefore should be explored as a last resort.
 

0, 1, 2, 4, 5  Issue 20133
Issue: It is possible to create an “orphaned” file that doesn't seem to belong to a project. 

Recommendation(s): This just seems like a bug, and will be filed as such. Whenever the user creates a new file, it should always be within the context of an existing project. Although, we might let the user open an existing file for browsing purposes, without having to attach it to an existing project. The user would not be able to do anything significant with that file, however.
 

0, 3  
Creating a Project Group / Comments about Project Groupings
Description and Recommendations
Participants
Resolution
Issue: Redesign New screen that still asks for package information when creating a group.

Recommendation(s): This is a known problem, and is being worked on.
 

1, 5  
Issue: Most people felt it was a reasonable idea. 

Recommendation(s): none needed.
 

0, 2, 3, 4  
Issue: Consider augmenting project groups with more useful functionality. For example, the project group might contain resources that all projects need. 

Proposed Resolution: This is an interesting idea, but one that might have to wait for a post-4.0 release.
 

4  
Creating a New Project (Under a Project Group Node)
Description and Recommendations
Participants
Resolution
Issue: Users right-click on a project group, select New, and get a simple list of available project types. Unfortunately, some users failed to understand that these were project templates. 

Recommendation(s): This problem will probably be resolved when we have multiple project types, as each type will have its own folder; the folder's name (e.g., "Java Projects", "Web Projects") should tell users that these are projects. In addition, Docs may want to change the name of the project templates to more clearly indicate that they are projects, e.g., "Java Application" --> "Simple Java Project".
 

1,3,5  
Issue: If the project group isn't selected, then the new project gets created outside of that group and there is no way for the user to rectify this without exiting and restarting the wizard. 

Recommendation(s): The desired project group should be included as an option in the New Project wizard. If a project group is selected, then that value should be filled in on the wizard.
 

0,1,2,3,4,5  
Opening the Project into the IDE
Description and Recommendations
Participants
Resolution
Issue: Opening projects from the File menu is expected, but we need to make sure that it's obvious how to do this, e.g., “File|Open Project”.

Recommendation(s): See below.
 

0,1,2,4,5  
Issue: People also expect Project Open to be under the Project menu. 

Recommendation(s): Have an "Open..." menu that shows a JFileChooser, and an "Open Project..." menu that shows a dialog for opening (closed) projects. (Note that this dialog has already been designed; a version is in the current Projects Infrastructure UI Spec but needs to be revised.) Consider putting the "Open Project" menu item under the Project menu. Although, this solution may violate the Java Look and Feel Design Guidelines, and therefore should be explored as a last resort.
 

0,1,3 Projects Infrastructure UI Spec
Reopening a Project Under a Project Group Node
Description
Participants
Resolution
Issue: In general, people did OK on opening a project from within a group. 

Recommendation(s): None needed.
 

2, 3, 4  
Issue: Need to decide whether a reopened project should appear under its old group or not. 

Recommendation(s): This applies to the use of the MRU list to reopen a project. A project should be re-opened in the last place where it was closed. Therefore, if project P1 is open in group G1 and G2, and then the user closes it in G1 and then G2, then reopening P1 from the MRU should reopen it in G2.
 

1  
Issue: Consider making it more convenient to reopen a project under a group, e.g., by NOT forcing users to search through the filesystem to reopen the project. 

Recommendation(s): The user should not have to search for project or group files in order to reopen a project; rather, they should be able to use the dialog discussed in the Projects Infrastructure UI Spec.
 

5 Projects Infrastructure UI Spec
Closing Projects (in explicit task, plus in participant-initiated cases)
Description
Participants
Resolution
Issue: Mostly this worked just fine, but should consider providing Close elsewhere, e.g., under File or Project since some users expect it there. 

Recommendation(s): If the File menu isn't too long, a Close menu item would be a sensible addition.
 

0,1,2,3,4,5  
CREATING / ADDING SOURCE DIRECTORIES AND PACKAGES
Creating a New Source Directory
Description
Participants
Resolution
Issue: The Add Source Directory dialog is confusing. 

Recommendation(s): This dialog needs to be replaced with a JFileChooser; this is a known issue which is being worked.
 

0,1,2  
Issue: New Sources dialog in the New Project wizard is confusing. 

Recommendation(s): Currently, the IDE has two mechanisms for getting source directories: the concept of adding an existing directory and the concept of creating a new directory under the project's home directory. We should consider unifying these two mechanisms. One way to do this is to have a single "Add Source Directory" function, which shows a JFileChooser dialog. The default location for the JFileChooser would be the project's home directory; the user would have the choice of adding/creating any arbitrary directory. If we do this, we need to make sure that users know they can create directories with the JFileChooser, and that the directory creation process is easy. 
 

0,1  
Issue: Not having a default source directory causes mischief, including orphaned files. 

Recommendation(s): Most projects should come with a pre-created source directory under the project's home directory. In addition, the process of creating a new file should automatically add the file's directory to the sources node. As mentioned elsewhere, the orphaned file problem is a bug.
 

2  
Issue: If a project is selected and the user does a File|New, the default location presented to her in the second wizard page is the project's home directory, but it's under a node called, "Default System".

Recommendation(s): Presumably this is a bug, since it probably wasn't intentional to expose NetBeans' guts like this.
 

0,2,4  
Issue: The Filesystems tab is going to be a problem for legacy users. 

Recommendation(s): First, the Filesystems tab, as we have discussed, needs to be substantially reduced in functionality so that users can't get themselves into trouble. For example, users probably shouldn't be able to mount subdirectories of a currently mounted filesystem. Second, we need to consider names for the Filesystems tab that communicate what it does without making legacy users think that this is their "escape hatch". Perhaps, "Filesystems Browser"?
 

4  
Adding an Existing Source Directory
Description
Participants
Resolution
Issue: Some users have an expectation that sources can be added by using Open File. 

Recommendation(s): Opening an arbitrary file should add its source directory to the Java Sources node.
 

0,1,2,3,4  
Issue: The Add Source Directory dialog is confusing. 

Recommendation(s): This dialog needs to be replaced with a JFileChooser; this is a known issue which is being worked.
 

0,1,3,5  
Issue: Some users expect to be able to add individual files under the Java Sources node, rather than adding a whole directory. 

Recommendation(s): The easy way to resolve this is to provide a look for the Java Sources node that simply hides the individual source directories. The harder way to resolve this is to allow the user to add both individual files and directories. The latter will require resolving some UI and development issues.
 

0,1,2,3,4,5  
Issue: IDE doesn't detect the presence of new directories that have been externally placed into a project source directory; the IDE must be restarted for the change to be detected. (Bug) 

Recommendation(s): A bug will be filed.
 

1  
Issue: Consider giving users more paths to create or add source files. 

Recommendation(s): File|New and the pop-up menus on the sources node are probably enough. But, if it doesn't cause too much clutter, it might be reasonable to let the user create new source files and add directories from the project node's pop-up menu. Also, as discussed, the Filesystems Browser should let users easily add a source directory/file to a project.
 

2,3  
Issue: Be able to handle the situation where the user tries to have source directory = project home directory. 

Recommendation(s): When one user did this, the IDE got confused, threw exceptions, etc. We need to gracefully handle whatever types of source directories the user might try to add (regardless of whether or not we let them do that).
 

4  
Issue: Users expect Resources node to hold non-Java objects, e.g., images, locale data. 

Recommendation(s): This requires some careful thought before we create an artificial split between "real" source files (Java files, Xml files) and "other" source files (bitmaps, sounds, text files). A better strategy might be to rethink the name, "Resources", and replace it with something more accurate. This is probably an issue for docs.
 

0 (+ see participants responses to questions after adding dictionary.jar)  
Creating packages 
Description
Participants
Resolution
Issue: Consider some alternative ways to create new packages, e.g., by renaming the default package or by picking up the package from the package statement in the source file (as in Eclipse). 

Recommendation(s): These are more ideas than anything else. They should be added to the effort to make the IDE more flexible in working with Java packages.
 

0,3  
Issue: If someone creates a folder (that is a legitimate Java package) rather than a package, they may be confused about whether or not they really did create a package. 

Recommendation(s): Move the Java Packages menu item up in the template list, so that it is one of the first things that the user sees. Also, consider adding some explanatory text to the dialog(s) that create new folders. The text should say that a folder that has a legal Java package name will be treated as a package.
 

0, 4,5  
Issue: The user can click on the project node or the Java Sources node and then use File|New to create a new package. But if the package is created anywhere but under an existing source directory, it does not appear in the project. 

Recommendation(s): Either constrain package creation to be in the existing source directories, or automatically add the package's parent directory when the user creates it.
 

5  
WORKING WITH SOURCE FILES
New class / Main Wizard
Description
Participants
Resolution
Issue: Package declaration in new class wizard is wrong. (bug) 

Recommendation(s): Will be filed as a bug.
 

1  
Issue: It is sometimes not clear where package information is kept before the participant has seen the package node(s). 

Recommendation(s): Only one participant had difficulty with this, but it is something to consider. One idea, besides beefing up the help materials, is to always show the <default package> under the Java Sources node.
 

2  
Moving Files into Packages
Description
Participants
Resolution
Issue: Need drag and drop for moving source files around. 

Recommendation(s): A known problem that is being worked on.
 

0,1,3,4,5  
Issue: Consider other ways to move files into/out of packages, e.g., right-clicking on package or file or using properties.

Recommendation(s): Being unable to drag and drop, a majority of the users tried other ways to move the files between packages. Some of these should be investigated, particularly adding a "Move" pop-up menu to files.

0,3,4  
Viewing Source
Description
Participants
Resolution
Issue: When Form Editor pops up and obscures the code window, it can confuse users.

Recommendation(s): By default open the code, but have an "Edit GUI" menu item for objects that have GUIs.
 

0,1,2,4,5  
BUILDING / COMPILING / EXECUTING
Building and Compiling
Description
Participants
Resolution
Issue: Orphaned files don't get compiled. 

Recommendation(s): As discussed elsewhere, orphaned files should not be permitted.
 

0,4  
Issue: Not all users could easily set the main class using the property sheet. 

Recommendation(s): The warning dialog that tells the user to set the project's main class was a stopgap that was put in place for the prototype. A better approach would be to display a dialog that lets the user select the main class. Possibly even better would be to detect when a) the project has no main class, and b) the user has just created a class with a main method. Then, the user could be asked if the newly created class should be the project's main class.
 

4  
Issue: Startup materials are important. 

Recommendation(s): None. This is just the observation that the startup materials like the Welcome screen, the tips of the day and the sample project were used.
 

0,5  
Issue: Execute project needs to be salient. 

Recommendation(s): Only one user had problems with this. It will probably suffice to provide the Execute Project menu item with an icon.
 

5  
Issue: Be able to handle the situation where the user clicks the class file and does an execute. 

Recommendation(s): We should probably permit the class file to be executed, or at least a good explanation why we won't let the users do it.
 

0,5  
Finding the Compiled Class
Description
Participants
Resolution
Issue: This worked out well. 

Recommendation(s): None needed.
 

0,1,2,3,4,5  
Issue: Need to make the physical location of the compiled class files discoverable. 

Recommendation(s): We could let the user rely on the physical view. A better solution would be to use tooltips, or to let the user see the full directory path for the output directories upon request.
 

0, 2   
RESOURCES
Using an External Library
Description
Participants
Resolution
     
Issue: In general people had a bit of difficulty finding this functionality, but once they found it everything worked out fine.

Recommendation(s): Consider adding more paths to add a resource. So, for example, the user should be able to add a resource through the project node's pop-up menu.
 

0,1,2,3,4,5  
Issue: Some users are unsure whether the Jar file is on the classpath, as there is no feedback indicating that it is, aside from the existence of the node under Resources.

Recommendation(s): Provide an easy way for the user to view the entire classpath. This might be done in the project's customizer, or perhaps on the Resource node's customizer. An alternative would be to have a special "Current Classpath" menu item on the Resource node's pop-up menu. It would give a read-only presentation of the current classpath. 
 

0,1,4,5  
Issue: At least one person was confused about whether a resource was being created or added. This might just have been because of the default name. 

Recommendation(s): Consider choosing a different default name (e.g., the name of the Jar file or directory), or saving the rename function for later (i.e., after the wizard has been dismissed).
 

1  
Issue: Users expect Resources node to hold non-Java objects, e.g., images, locale data. 

Recommendation(s): This requires some careful thought before we create an artificial split between "real" source files (.java files, .xml files) and "other" source files (bitmaps, sounds, text files). A better strategy might be to rethink the name, "Resources", and replace it with something more accurate. This is probably an issue for docs.
 

3,4,5  
Resource Dependencies (Discussion)
Description
Participants
Resolution
Issue: Most people guessed that dependencies would be expressed under the Resources node, or under a similar, parallel node. However, there was some divergence about exactly how things would be arranged under Resources.

Recommendation(s): Exactly what to show when another project's target is used under the Resources node is the focus of ongoing design work.
 

1,2,3,5  
LOGICAL AND PHYSICAL GROUPINGS
Description
Participants
Resolution
Issue: Users had some difficulty finding the switch for logical/physical view. Consider making this more salient. 

Recommendation(s): Presumably the work on the Looks UI will solve this. 
 

0,1,2,5 Looks UI
Issue: Users understand the difference between physical and logical views. 

Recommendation(s): None needed.
 

0,1,2,3,4,5  
Issue: Users favor the logical view for their work. 

Recommendation(s): None needed. 
 

0,1,2,3,4,5  
Issue: Users expect the same functionality to be available in the two different views. 

Recommendation(s): None needed.
 

1,2,3,5  
Issue: At least one user wants to filter the files displayed in the project. 

Recommendation(s): None needed. This should be doable with the IDE's fileset ability. 
 

2  
MISCELLANEOUS ISSUES
Description
Participants
Resolution
Issue: At least two people expressed a wish for a beefier Projects menu with more functionality. 

Recommendation(s): The Projects menu still needs a final round of design work.
 

0,3  
Issue: One participant commented that presenting packages individually might cause clutter, since each subpackage will create an entirely new node. 

Recommendation(s): The participant suggested collapsing the package view, so that subpackages are presented more like subfolders. If package clutter turns out to be a problem, this solution should be considered as an alternative view.
 

2  
Issue: Be able to handle the situation where the user wants to delete the output folder from the Output node. 

Recommendation(s): One solution is to just replace the output node if it's deleted. However, this seems a bit odd, as the user might have really meant to delete the node. A better solution would be to detect that the deletion is being done, and to display a dialog asking the user to select a new output node.
 

2  

Conclusions

This usability study has uncovered some issues that need to be addressed as the prototype is developed.  Fortunately, most of these issues are limited in scope and in impact; their solutions are relatively clear, or have been known for some time.  With these issues addressed, the new projects system is poised to become a real asset to the IDE by orienting new users and by increasing the productivity of experienced users.
 

Project Features

About this Project

ui was started in November 2009, is owned by Jiří Kovalský, and has 44 members.
By use of this website, you agree to the NetBeans Policies and Terms of Use (revision 20160708.bf2ac18). © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo
 
 
Close
loading
Please Confirm
Close