NetBeans Java Projects
Usability Study of the Second Prototype
Authors: Jan Rojcek, Chris
LeDantec
Last Updated:
May 20, 2003
Document
History
Revision: 1.10
Executive Summary
The Usability Study was the second such study for the NetBeans Projects System. The study participants were a mix of developers with NetBeans/S1S experience and those with other IDE or tool background. The over arching concepts of the Projects System did well in the study though areas for improvement were apparent from watching the participants use the IDE. Two high level categories of problems were unearthed in the study, communication problems and conceptual problems.
Communication problems encapsulate issues where, for design failings or immature code, the participant was unable to decipher or correctly act on information the IDE presented. These issues can be categorized as times when the participant came across the correct functionality but failed to realize they were there. Most of these issues can be remedied with more attention to dialog text and action names though some functional streamlining will be necessary.
Conceptual problems show failings where the understanding necessary to successfully arrive at functionality is missing. This category of problems will require more thought about why the concepts are necessary and an elimination or better explanation of such concepts.
For more detail on recommendations, please see the Usability Recommendations document.
High Level Findings:
- Active Project - Conceptual
The concept of the Active Project was not understood and contributed to problems for each of the participants. In the best cases it took the participant several minutes to understand that there was an Active Project. Most of the participants never came across the idea Active Project and had to fumble around for a long time before finding context sensitive actions that allowed them to bypass using the concept.
- Running - Conceptual
Running a project is too difficult. The appearance of a Profile dialog after clicking a Run action was confusing -most participants would see the profile dialog and close it immediately because they did not understand why a profile was necessary to run their code.
- Java and Source Root confusion - Conceptual
Java Roots and Source Roots confused all participants. The difference between the two was not clear nor was the reason the participant should worry about such things.
- Working with Project Interdependencies - Conceptual/Communication
Participants had a hard time understanding how to combine two projects together with one used as a resource for the second. This was largely due to labeling and conceptual issues with the different parts that make up a project (Resources, Output)
- Resources - Communication
The connection between the Resources node and the Classpath was not apparent. The node is not as visible under a moderately populated Sources node and the association of Classpath to Resource was not communicated.
- Context Sensitivity - Communication
Some issues in this area were related to the immaturity of the implementation; however, the over context sensitivity of actions like creating a new file or the ambiguous context sensitivity of building and running are issues that caused the majority of the participants problems.
- Other Issues - Communication
Project structure is unclear and hard to decipher with even one or two small projects open.
Consistency between customizers opened from different locations/contexts is poor -the Edit Profiles customizer has version with a 'Done' button, a 'Close' button and an 'Ok' and 'Cancel' button pair.
Settings are too dispersed and context sensitive. Participants kept looking in one or two places for most of the project related settings. In each run the moderator had to help the participant find the right context or pop-up menu in order to complete the task.
Length and order of pop-up menu contents lead to the correct action being overlooked several times.
Wording on actions and dialogs needs to be cleaned up -confusion between Add and New was prevalent as was pointing the project home directory to match a stated source directory for a project with existing sources.
Introduction
During the week of April 22nd through the 24th the team in Prague conducted a usability study on the Projects System slated for a future release of NetBeans. The test was carried out on an early Beta version of the functionality -the IDE that was used for the test was not a final product and much of the functionality planned for the final version was missing as was much of the UI presented in a state consistent with early Beta software.Test Objectives
The objective of this usability study was to test specific design and UI decisions in the new Projects System. The specifics of the test are listed below. While general usability was examined, we were paying close attention to how the participants worked with the logical layout of a project, the flow between different tasks common to rudimentary project work, the understandability of project concepts, UI and task flow they are imbedded in, and how manageable the extensive customizability of the projects system is.
To analyze these aspects of the Project System we used a script of tasks designed to test or touch on as many of topics as possible. The limitations we faced during the test included the early stage of ongoing development of the Projects System encompassing incomplete or missing features and unpolished UI, and testing the depth and breadth of the Projects System in two hours. The tasks chosen were designed to provide specific feedback on the UI that they involved and more abstract feedback on design concepts that have a large impact on user success. The combined affects of these two challenges will be accounted for in the analysis and recommended changes.
Procedure
During the study participants were asked to complete 6 tasks, and were asked a number of questions regarding their experiences of completing those tasks. The scenario used for the tasks was centered around working in a new job at a firm that produces entertaining and educational Java applications for kids. The participants had to use the NetBeans Project Manager Prototype to complete the tasks. Participants worked pseudo-independently with the moderator seated at their side. The study was conducted on a 1.4 GHz Dell PC running Windows 2000. The IDE was installed and participant had to start the IDE. Tests ran for up to 2 hours each and included the following tasks:- Opportunity to browse the IDE
- Creating a "Hello World" project, and running it
- Opening an existing project, and building it
- Creating a text file, package and a new main class
- Running the project with the new main class (setting up a new profile)
- Running the project with a command line parameter
- Creating a new project from existing sources
- Setting up a build dependencies between projects, and running the new project
Participants
A total of 5 Java programmers, participated in the study.
All participants were external. 3 participants (P1, P3, P4) had at least 6
months FFJ or NetBeans experience. See the table below for details of
participant's Java and IDE experience.
| Participant Number | Java Experience | IDE's Used & Length of Experience (most recent listed first) |
| P1 | 5 years | S1S (1 year), VAJ (3 years), JBuilder (1 year) |
| P2 | 2.5 years | VAJ (1.5 years), JBuilder (3 months) |
| P3 | 2 years | S1S (2 years), MS J++ (1 year as a student) |
| P4 | 3 years | FFJ, MS J++, VAJ |
| P5 | 2 years | JDeveloper (1.5 year) |
Materials
The following materials were used during this study:Main Results
This section contains lists of all the main issues encountered during the usability test in tabular form below. Recommendations listed here are only meant as a guide. More specific recommendations are in the Usability Recommendations document.| Description and Recommendations | Participants | Resolution |
| No issues. | ||
| Creating
a New Project |
||
| Description and Recommendations | Participants | Resolution |
| No issues. All of the participants go smoothly through a new project creation. They use either Basic Java App template, or Empty Java App. In case of Empty Java App none of the participants modifies the source directory. Some invoke the new project wizard from main menu, some from the popup menu and one from the welcome screen. | -- | -- |
| Creating
a New Project From Existing Sources |
||
| Description and Recommendations | Participants | Resolution |
| Issue: A participant opens the
NFT wizard, reads template descriptions and cancels the wizard. Invokes
the open project dialog, navigates to the source directory and cancels
the dialog. Then again invokes the wizard and chooses a correct template.
Analysis: This is a communication problem. The text in the wizard does not make it obvious enough that it is the right place to be. Recommendation(s): Go over wizard text and action names. This problem could also be attributed to user haste. See bug 33596. |
5 | |
| Issue: A Participant assumes the
home directory text field is a place where he should specify the location
of existing sources. Probably means he equals the project home with sources
home.
Analysis: Communication problem Recommendation(s): Change text label to more accurately describe what the 'Home Directory' is for. See bug 33596. |
1, 2 | |
| Issue: The Project tree contains
classes twice after finishing the wizard, once in the packages hierarchy,
once in the source hierarchy.
Analysis: When sources are added via the Sources node and then also marked as Java Roots this duplication appears. This problem is a symptom of the ambiguously communicated relationship of Java and Source roots in the project. Recommendation(s): Java and Source roots as user concepts do not work well. The duplication of files in two places is only one symptom connected with confusion about what a Java root is and what a Source root is when adding files to a project. Removing the user concept may help this problem though more thought needs to go into how to mark sources for inclusion on the classpath (which is really what the Java root represents). See bug 33597 |
1, 2, 4, 5 | |
| Issue: Participant tried to add
a classes directly in the source root selector. A classes were shown,
but OK button was disabled when he selected the classes.
Analysis: UI Bug. Recommendation(s): The selector should always show only objects which could be selected. This is not saying whether the sources panel is appropriate in the wizard or not. See bug 33598 |
2 | |
| Issue: A participant use Empty
Java App template to create a project from existing sources. Adds additional
java roots in the java roots panel. But the roots are not correct, they
are "../src/org" and "../src/test".
Analysis: UI Bug Recommendation(s): Wizards should correct such errors when adding Java roots. See bug 33598 |
3 | |
| Issue: A participant set the sources
root in the source roots panel to a incorrect directory ".../src/org".
For some participant it is hard to tell what is the source root, whether
".../src" dir or ".../src/org".
Analysis: UI Bug Recommendation(s): See above. See bug 33599 |
5 | |
| Opening
an Existing Project |
||
| Description and Recommendations | Participants | Resolution |
| No issues. Participants have no problems with opening an existing project. Some users open the wizard from main menu, some from the popup menu. | 1, 2, 3, 4, 5 | |
| Creating
a Java Package |
||
| Description and Recommendations | Participants | Resolution |
| No issues. All of the participants created a new package without problems. | 1, 2, 3, 4, 5 | |
| Creating
a Java Class |
||
| Description and Recommendations | Participants | Resolution |
| Issue: A participant can't find
how to create a new file. He is looking into main menus, completely ignoring
the project structure and contextual menus. Moderator is pointing him
to use the contextual menus.
Analysis: Context sensitivity of New action overdone. Recommendation(s): Make sure common actions like New are accessible in non-context sensitive form from the main menu. See bug 33600 |
5 | |
| Creating
a Text File |
||
| Description and Recommendations | Participants | Resolution |
| Issue: Participant supplies extension
in the name text field in the second NFT wizard panel, but then the third
panel asks for the extension. So, he goes back one step and deletes the
extension from the text field.
Analysis: Old nb problem. Recommendation(s): IDE should resolve this automatically. See bug 33601 |
1, 5 | |
| Issue: A participant creates "worldlibrary.txt.txt"
text file. He doesn't realize that the extension in the third panel is
added to the name supplied in previous panel.
Analysis: See above. Recommendation(s): See above. See bug 33601 |
4 | |
| Issue: Participant creates the
text file outside of the IDE and can't find a way how to add it to the
project.
Analysis: Refresh problem with the current build. For performance reasons automatic refresh has been disabled. In order to update the filesystem the user must manually refresh in the filesystems tab which is not shown be default explorer. Recommendation(s): The refresh issue should be re-examined. Performance implications aside, we cannot guarantee that all our users work will be created from within the IDE. See bug 33602 |
2 | |
| Issue: A participant tries to create
a text file as a new resource through "New Resource..." popup menu item
of the resources node. Then tries to add it as the source under the sources
node. Participant #5 invokes also the new project wizard from the contextual
menu.
Analysis: Wizard wording issues. This is connected to other problems participants had in trying to add resources to a project (see below) the connection of resource to the classpath is ambiguous as it is to other things that may not be strictly considered source. Recommendation(s): Clarify relationship between logical parts of the project and Java specific concepts (sources = class and cohabitating files, resources = classpath and external libs). See bug 33603 |
3, 4, 5 | |
| Building
a Project |
||
| Description and Recommendations | Participants | Resolution |
| Issue: A participant uses "Build
Active Project" toolbar button or main menu item to build a non-active
project.
Analysis: Active project is not understood. Partly because the active project is not highlighted well enough -the little noticed '[active]' text next to the project name is lost in the jumble of text and nodes in the project explorer. This problem is exacerbated because the IDE starts up with an example project that is set active (and new projects do not automatically get the active setting). Recommendation(s): Make the active project title bold and re-order the project nodes to improve discoverability of both Resources and Output node as well as create some visual space between the bolded title and the content of the Sources node. Do not start the IDE with an example Project open. Have a link in the welcome page that will automatically open an example project instead. See bug 33604 |
2, 4, 5 | |
| Issue: A participant doesn't notice
that the build output results belong to a different project than he wants
to build.
Analysis: Symptom of the Active Project issue. Recommendation(s): See above + do a better job of making the compiled project title noticeable in the output window. See bug 33604 |
2, 4 | |
| Issue: A participant doesn't compile
a project, only a class using a shortcut, main menu, or contextual menu
of the class.
Analysis: Work around for Active project issue. Recommendation(s): See above. See bug 33604 |
2, 3 | |
| Issue: A build output says it compiled
a method. Also a menu item says it compiles a method.
Analysis: UI Bug. Recommendation(s): Fix the language. See bug 33596 |
3 | |
| Issue: A build output scrolls to
the last error, instead of the first error. Participant immediately scrolled
to beginning to see first error.
Analysis: UI Bug Recommendation(s): This seems to be an ongoing tug of war -should the output scroll or stay fixed at the top. The results here indicate that for compiler errors the first error is more important than the last (especially since one syntax error can create a string of error following it). See bug 33605 |
2, 3, 4 | |
| Issue: User builds a project with
a project dependency on another project, but is not sure that both projects
are built, when reading the build output.
Analysis: Communication issue. Recommendation(s): Clarify output -if the dependency does not need to be built then the output window should indicate that it is up to date. See bug 33596 |
3 | |
| Running
a Project |
||
| Description and Recommendations | Participants | Resolution |
| Issue: A participant executes a
class using its contextual menu, after closing the profiles dialog open
by the run action. He doesn't set up the profile, just closes the dialog.
Analysis: The barrier to running a project is too high. The concept of profiles is not helpful or necessary at the point and should only be introduced when there is need. Recommendation(s): See below. See bug 33606 |
1, 2, 4, 5 | |
| Issue: A participant tries to execute
a non-active project using "Run Active Project" menu item or shortcut.
Analysis: Active project issue. Recommendation(s): See above. See bug 33604 |
1, 2, 3, 4, 5 | |
| Issue: A participant executed the
active project using main menu, but it runs the old profile pointing to
incorrect main class, which has been deleted. The participant doesn't
know where is the problem. The output just says "No class def found error."
Analysis: UI Bug Recommendation(s): When the class an execution profile depends on is deleted the the profile should be deactivated as well. See bug 33606 |
3 | |
| Issue: No feedback is printed into
the output window about started or finished execution. Especially if user
code doesn't prints anything into the output window.
Analysis: UI Bug Recommendation(s): Improve output window feedback. See bug 33596 |
1, 5 | |
| Issue: A participant executes main
class from its contextual menu. The class is executed even if there are
a build errors.
Analysis: n/a Recommendation(s): A class with build errors shouldn't be run. See bug 33607 |
2, 4 | |
| Setting
up a Profile |
||
| Description and Recommendations | Participants | Resolution |
| Issue: A participant closes the
profiles dialog because he thinks he doesn't need it for execution.
Analysis: Profiles problem - the concept does not have an obvious connection with running the project. Recommendation(s): The profile may not be necessary for many cases where the user wants to run a project -removing it from the direct path would reduce the number of steps to running a project. In simple cases with only one runnable class the IDE should run the class without extra input from the user. When creating a new class that is runnable the IDE should keep track and perhaps offer the a choice when Run project is later selected. More work needs to go into the profile issue. See bug 33606 |
2, 5 | |
| Issue: A participant invokes the
run action from project's popup menu, but it displays the profiles dialog
even if a profile exists. It is because non of the profiles is active(default).
Analysis: UI Bug Recommendation(s): When only one profile exists it should be the default. See bug 33606 |
4 | |
| Issue: User doesn't find where
to set the arguments by looking into menus, so reads the welcome screen.
Analysis: n/a Recommendation(s): None. |
2 | |
| Issue: The main class selector
doesn't display which class is runnable, so a participant has to open
files in the editor to figure out which class is the main class.
Analysis: Show information specifically relevant to the functionality. Recommendation(s): Class selector should display runnable classes in the project. Navigating through the entire source tree does not help the user. See bug 33598 |
4 | |
| Add a
Project as a Resource to Another Project |
||
| Description and Recommendations | Participants | Resolution |
| Issue: A participant can't find
how to set a build dependencies between projects. Most of the participants
look into: Project customizer, Project properties, Configuration editor.
Few of participants are looking for "something named classpath".
Analysis: Two-fold problem: the logical structure of a project gets lost when the sources node is expanded -participants did not notice the Output or Resources node easily; the connection of Java concepts to Project concepts is not obvious (stated above). The problem is not specifically connecting up two projects but understanding what the different parts of a project provide -that Output is the built bits of the code and Resources are the libraries one uses when building a project. Once those concepts were understood or explained by the moderator the rest of the task (opening the wizard and choosing the right template) was trivial. Recommendation(s): Wording needs to be improved and the logical layout to keep things from getting lost when viewed with the expanse of nodes that appear under the Sources node. See bug 33603 |
1, 2, 3, 4, 5 | |
| Issue: A participant invokes help
to read about the project dependencies, but doesn't understand it.
Analysis: n/a Recommendation(s): None. Help was clarified after the first test run. |
1 | |
| Issue: A participant selected a
wrong template in the new resource wizard.
Analysis: Communication issue -the names of things, especially concepts particular to the NetBeans need to be very clear and obvious at a glance. Recommendation(s): Go over naming and other visual cues. See bug 33596 |
2, 3 | |
| Issue: A participant adds a resource
in the runtime classpath panel of profile customizer. But it is not working
for building the project.
Analysis: Bug. Recommendation(s): None. |
4 | |
| Issue: A participant adds a resource
in the runtime classpath panel of profile customizer. Selects "use everywhere"
in the last panel, but after finishing the wizard the resource is not
shown in the list. He does this four times with the same result.
Analysis: Bug. Adding a Resource and specifying that it should be used everywhere should work. Recommendation(s): None. |
4 | |
| Issue: A participant adds the java
class library as a resource to runtime class path of run profile. He points
to ".../anagram-lib/src" directory.
Analysis: The participant may not have completely understood the task and drawn the connection between adding the Output of a different project as the best way to proceed. Instead the participant tried to add the sources which is reasonable given that the class files are currently left in place with the sources. Recommendation(s): Go over wording. See bug 33596 |
5 | |
| Issue: A participant added ".../anagram-lib/src"
directory as a resource. But it added also the "src" dir under the resources
node. This is different from adding packages roots under the sources node.
Analysis: The level at which resources are added is not the same as the level at which sources are added. Recommendation(s): The two need to be the same. See bug 33608 |
5 | |
Conclusions
Several issues have been uncovered -some known, some surprises -and concrete changes need to be made to the UI and interaction to solve them:
- Active Project
The visual representation of the active project need to be improved 100 fold. Bolding the project title is probably the least that should be done to signify that the project is active. Other visual cues might involve changing the explorer background behind non-active projects to a light shade of gray (not enough to look disabled but to draw focus to the active project with the white background).
Project wide actions need to be clearly labeled and have easy to understand rules to avoid the current context ambiguity between build and run (build applying to the current selection and run applying to the active project). Create a small set of duplicated actions in the main menu should be enough -Build, Run, Debug (with the standard shortcuts) apply to the Active project and Build <project name>, Run <project name>, Debug <project name> would apply to the currently selected or context-active project.
- Profiles
The necessity of dealing with Profiles for all cases of running and debugging needs to be removed. Several easy changes can be made that will help stream line the process for simple cases with only one runnable class and for creating new profiles when adding new runnable classes. The dialog should also include more text and better labels to make it more obvious what it is used for so that users are less likely to dismiss it out of hand.
In addition to trimming down the intrusive nature of Profiles the number and complexity should be trimmed as well -run and debug profiles should all be one thing with one dialog -see mail on dev@projects.netbeans.org for more details [find link and drop it here]. A final point to consider is the name Profile -it probably would benefit from an adjective to make it more clear what it is.
While configurations where not specifically tested in this study it would be beneficial to look at how they will be used and draw some comparisons between the success of Profiles and the success of yet another way to manage a group of settings.
- Central Location of Project Options
Settings The study saw the majority of participants returning to one or two places when looking to change settings, add things (Resources), and otherwise manipulate the project. Creating a one-shop for most of the project settings will solve this problem. The context sensitive hooks should remain as they will be useful for users who understand how to manipulate the context sensitive aspects of the IDE.
Looking at the Options spec should provide some reasonable overlap in approach. The all encompassing customizer should be located off the Project node
- Reorder Top Level Project Nodes
Changing the order of the project nodes should help bring more attention to the Resources and Output nodes. When the Sources node is expanded these two nodes get lost in the lower parts of the explorer (even with a small project) and overlooked. The argument at the time was to move them down because they won't be used much -but for just that reason they should be kept at the top. The two are unlikely to be expanded so sources will likely be visible and by having them between the Sources node and the Project node they should be much more discoverable.
- Clarify (de-generalize) Node Names in the Project
The very general names of the nodes in a project -Source, Output and Resources -were not helpful in directing the participant where they needed to go to complete certain tasks or find certain information. Source was straight forward but the distinction between Source and Java Roots was not and needs to be re-examined. For both Resources and Output participants were surprised to look there for their intended purposes. In several cases participants tried to add the text file to the Resources node and none of the participants clearly understood that the Resources node is where the classpath is defined.
Similarly with the Output node, the idea that the Output of a project is the compiled classes was confusing during the process of adding project dependencies -several participants wanted to simply add the source tree of one project to the Resources of the second.
These errors demonstrate that the departure from Java specific terms in the case of Resources and generally vague terms like Output do not aid the user in navigating the IDE. A more appropriate solution would be to more accurately name the nodes for the type of project being worked with -the level of abstraction needed for a one-size fits all label ends up being counterproductive.
These central points are the big fish that need to be addressed. At this point the solutions for each of these issues needs to be worked through with the development teams building and depending on the Projects system -any attempts at solutions listed here are suggestions and for the time being only considered as such.
The wording, in line help, menu layout and other issues lurking in the above
table will be taken care of as a matter of course.
