NetBeans Java Projects

Usability Study of the Second Prototype

Authors: Jan Rojcek, Chris LeDantec
Last Updated: May 20, 2003
Document History
Revision:
1.10

Executive Summary

The Usability Study was the second such study for the NetBeans Projects System. The study participants were a mix of developers with NetBeans/S1S experience and those with other IDE or tool background. The over arching concepts of the Projects System did well in the study though areas for improvement were apparent from watching the participants use the IDE. Two high level categories of problems were unearthed in the study, communication problems and conceptual problems.

Communication problems encapsulate issues where, for design failings or immature code, the participant was unable to decipher or correctly act on information the IDE presented. These issues can be categorized as times when the participant came across the correct functionality but failed to realize they were there. Most of these issues can be remedied with more attention to dialog text and action names though some functional streamlining will be necessary.

Conceptual problems show failings where the understanding necessary to successfully arrive at functionality is missing. This category of problems will require more thought about why the concepts are necessary and an elimination or better explanation of such concepts.

For more detail on recommendations, please see the Usability Recommendations document.

High Level Findings:

  • Active Project - Conceptual

    The concept of the Active Project was not understood and contributed to problems for each of the participants. In the best cases it took the participant several minutes to understand that there was an Active Project. Most of the participants never came across the idea Active Project and had to fumble around for a long time before finding context sensitive actions that allowed them to bypass using the concept.

  • Running - Conceptual

    Running a project is too difficult. The appearance of a Profile dialog after clicking a Run action was confusing -most participants would see the profile dialog and close it immediately because they did not understand why a profile was necessary to run their code.

  • Java and Source Root confusion - Conceptual

    Java Roots and Source Roots confused all participants. The difference between the two was not clear nor was the reason the participant should worry about such things.

  • Working with Project Interdependencies - Conceptual/Communication

    Participants had a hard time understanding how to combine two projects together with one used as a resource for the second. This was largely due to labeling and conceptual issues with the different parts that make up a project (Resources, Output)

  • Resources - Communication

    The connection between the Resources node and the Classpath was not apparent. The node is not as visible under a moderately populated Sources node and the association of Classpath to Resource was not communicated.

  • Context Sensitivity - Communication

    Some issues in this area were related to the immaturity of the implementation; however, the over context sensitivity of actions like creating a new file or the ambiguous context sensitivity of building and running are issues that caused the majority of the participants problems.

  • Other Issues - Communication

    Project structure is unclear and hard to decipher with even one or two small projects open.

    Consistency between customizers opened from different locations/contexts is poor -the Edit Profiles customizer has version with a 'Done' button, a 'Close' button and an 'Ok' and 'Cancel' button pair.

    Settings are too dispersed and context sensitive. Participants kept looking in one or two places for most of the project related settings. In each run the moderator had to help the participant find the right context or pop-up menu in order to complete the task.

    Length and order of pop-up menu contents lead to the correct action being overlooked several times.

    Wording on actions and dialogs needs to be cleaned up -confusion between Add and New was prevalent as was pointing the project home directory to match a stated source directory for a project with existing sources.

Introduction

During the week of April 22nd through the 24th the team in Prague conducted a usability study on the Projects System slated for a future release of NetBeans. The test was carried out on an early Beta version of the functionality -the IDE that was used for the test was not a final product and much of the functionality planned for the final version was missing as was much of the UI presented in a state consistent with early Beta software.

Test Objectives

The objective of this usability study was to test specific design and UI decisions in the new Projects System. The specifics of the test are listed below. While general usability was examined, we were paying close attention to how the participants worked with the logical layout of a project, the flow between different tasks common to rudimentary project work, the understandability of project concepts, UI and task flow they are imbedded in, and how manageable the extensive customizability of the projects system is.

To analyze these aspects of the Project System we used a script of tasks designed to test or touch on as many of topics as possible. The limitations we faced during the test included the early stage of ongoing development of the Projects System encompassing incomplete or missing features and unpolished UI, and testing the depth and breadth of the Projects System in two hours. The tasks chosen were designed to provide specific feedback on the UI that they involved and more abstract feedback on design concepts that have a large impact on user success. The combined affects of these two challenges will be accounted for in the analysis and recommended changes.

Procedure

During the study participants were asked to complete 6 tasks, and were asked a number of questions regarding their experiences of completing those tasks. The scenario used for the tasks was centered around working in a new job at a firm that produces entertaining and educational Java applications for kids. The participants had to use the NetBeans Project Manager Prototype to complete the tasks. Participants worked pseudo-independently with the moderator seated at their side. The study was conducted on a 1.4 GHz Dell PC running Windows 2000. The IDE was installed and participant had to start the IDE. Tests ran for up to 2 hours each and included the following tasks:
  • Opportunity to browse the IDE
  • Creating a "Hello World" project, and running it
  • Opening an existing project, and building it
  • Creating a text file, package and a new main class
  • Running the project with the new main class (setting up a new profile)
  • Running the project with a command line parameter
  • Creating a new project from existing sources
  • Setting up a build dependencies between projects, and running the new project
Participants were asked to "think aloud" as they completed the tasks.

Participants

A total of 5 Java programmers, participated in the study.

All participants were external. 3 participants (P1, P3, P4) had at least 6 months FFJ or NetBeans experience. See the table below for details of participant's Java and IDE experience.

Participant Number Java Experience IDE's Used & Length of Experience (most recent listed first)
P1 5 years S1S (1 year), VAJ (3 years), JBuilder (1 year)
P2 2.5 years VAJ (1.5 years), JBuilder (3 months)
P3 2 years S1S (2 years), MS J++ (1 year as a student)
P4 3 years FFJ, MS J++, VAJ
P5 2 years JDeveloper (1.5 year)

Materials

The following materials were used during this study:

Main Results

This section contains lists of all the main issues encountered during the usability test in tabular form below. Recommendations listed here are only meant as a guide. More specific recommendations are in the Usability Recommendations document.
ORIENTING TO THE IDE
Description and Recommendations Participants Resolution
No issues.
CREATING AND OPENING PROJECTS
Creating a New Project
Description and Recommendations Participants Resolution
No issues. All of the participants go smoothly through a new project creation. They use either Basic Java App template, or Empty Java App. In case of Empty Java App none of the participants modifies the source directory. Some invoke the new project wizard from main menu, some from the popup menu and one from the welcome screen. -- --
Creating a New Project From Existing Sources
Description and Recommendations Participants Resolution
Issue: A participant opens the NFT wizard, reads template descriptions and cancels the wizard. Invokes the open project dialog, navigates to the source directory and cancels the dialog. Then again invokes the wizard and chooses a correct template.

Analysis: This is a communication problem. The text in the wizard does not make it obvious enough that it is the right place to be.

Recommendation(s): Go over wizard text and action names. This problem could also be attributed to user haste. See bug 33596.

5
Issue: A Participant assumes the home directory text field is a place where he should specify the location of existing sources. Probably means he equals the project home with sources home.

Analysis: Communication problem

Recommendation(s): Change text label to more accurately describe what the 'Home Directory' is for. See bug 33596.

1, 2
Issue: The Project tree contains classes twice after finishing the wizard, once in the packages hierarchy, once in the source hierarchy.

Analysis: When sources are added via the Sources node and then also marked as Java Roots this duplication appears. This problem is a symptom of the ambiguously communicated relationship of Java and Source roots in the project.

Recommendation(s): Java and Source roots as user concepts do not work well. The duplication of files in two places is only one symptom connected with confusion about what a Java root is and what a Source root is when adding files to a project.

Removing the user concept may help this problem though more thought needs to go into how to mark sources for inclusion on the classpath (which is really what the Java root represents). See bug 33597

1, 2, 4, 5
Issue: Participant tried to add a classes directly in the source root selector. A classes were shown, but OK button was disabled when he selected the classes.

Analysis: UI Bug.

Recommendation(s): The selector should always show only objects which could be selected. This is not saying whether the sources panel is appropriate in the wizard or not. See bug 33598

2
Issue: A participant use Empty Java App template to create a project from existing sources. Adds additional java roots in the java roots panel. But the roots are not correct, they are "../src/org" and "../src/test".

Analysis: UI Bug

Recommendation(s): Wizards should correct such errors when adding Java roots. See bug 33598

3
Issue: A participant set the sources root in the source roots panel to a incorrect directory ".../src/org". For some participant it is hard to tell what is the source root, whether ".../src" dir or ".../src/org".

Analysis: UI Bug

Recommendation(s): See above. See bug 33599

5
Opening an Existing Project
Description and Recommendations Participants Resolution
No issues. Participants have no problems with opening an existing project. Some users open the wizard from main menu, some from the popup menu. 1, 2, 3, 4, 5
CREATING AND MODIFYING SOURCE FILES/DIRECTORIES/PACKAGES
Creating a Java Package
Description and Recommendations Participants Resolution
No issues. All of the participants created a new package without problems. 1, 2, 3, 4, 5
Creating a Java Class
Description and Recommendations Participants Resolution
Issue: A participant can't find how to create a new file. He is looking into main menus, completely ignoring the project structure and contextual menus. Moderator is pointing him to use the contextual menus.

Analysis: Context sensitivity of New action overdone.

Recommendation(s): Make sure common actions like New are accessible in non-context sensitive form from the main menu. See bug 33600

5
Creating a Text File
Description and Recommendations Participants Resolution
Issue: Participant supplies extension in the name text field in the second NFT wizard panel, but then the third panel asks for the extension. So, he goes back one step and deletes the extension from the text field.

Analysis: Old nb problem.

Recommendation(s): IDE should resolve this automatically. See bug 33601

1, 5
Issue: A participant creates "worldlibrary.txt.txt" text file. He doesn't realize that the extension in the third panel is added to the name supplied in previous panel.

Analysis: See above.

Recommendation(s): See above. See bug 33601

4
Issue: Participant creates the text file outside of the IDE and can't find a way how to add it to the project.

Analysis: Refresh problem with the current build. For performance reasons automatic refresh has been disabled. In order to update the filesystem the user must manually refresh in the filesystems tab which is not shown be default explorer.

Recommendation(s): The refresh issue should be re-examined. Performance implications aside, we cannot guarantee that all our users work will be created from within the IDE. See bug 33602

2
Issue: A participant tries to create a text file as a new resource through "New Resource..." popup menu item of the resources node. Then tries to add it as the source under the sources node. Participant #5 invokes also the new project wizard from the contextual menu.

Analysis: Wizard wording issues. This is connected to other problems participants had in trying to add resources to a project (see below) the connection of resource to the classpath is ambiguous as it is to other things that may not be strictly considered source.

Recommendation(s): Clarify relationship between logical parts of the project and Java specific concepts (sources = class and cohabitating files, resources = classpath and external libs). See bug 33603

3, 4, 5
BUILDING AND RUNNING PROJECTS
Building a Project
Description and Recommendations Participants Resolution
Issue: A participant uses "Build Active Project" toolbar button or main menu item to build a non-active project.

Analysis: Active project is not understood. Partly because the active project is not highlighted well enough -the little noticed '[active]' text next to the project name is lost in the jumble of text and nodes in the project explorer.

This problem is exacerbated because the IDE starts up with an example project that is set active (and new projects do not automatically get the active setting).

Recommendation(s): Make the active project title bold and re-order the project nodes to improve discoverability of both Resources and Output node as well as create some visual space between the bolded title and the content of the Sources node.

Do not start the IDE with an example Project open. Have a link in the welcome page that will automatically open an example project instead. See bug 33604

2, 4, 5
Issue: A participant doesn't notice that the build output results belong to a different project than he wants to build.

Analysis: Symptom of the Active Project issue.

Recommendation(s): See above + do a better job of making the compiled project title noticeable in the output window. See bug 33604

2, 4
Issue: A participant doesn't compile a project, only a class using a shortcut, main menu, or contextual menu of the class.

Analysis: Work around for Active project issue.

Recommendation(s): See above. See bug 33604

2, 3
Issue: A build output says it compiled a method. Also a menu item says it compiles a method.

Analysis: UI Bug.

Recommendation(s): Fix the language. See bug 33596

3
Issue: A build output scrolls to the last error, instead of the first error. Participant immediately scrolled to beginning to see first error.

Analysis: UI Bug

Recommendation(s): This seems to be an ongoing tug of war -should the output scroll or stay fixed at the top. The results here indicate that for compiler errors the first error is more important than the last (especially since one syntax error can create a string of error following it). See bug 33605

2, 3, 4
Issue: User builds a project with a project dependency on another project, but is not sure that both projects are built, when reading the build output.

Analysis: Communication issue.

Recommendation(s): Clarify output -if the dependency does not need to be built then the output window should indicate that it is up to date. See bug 33596

3
Running a Project
Description and Recommendations Participants Resolution
Issue: A participant executes a class using its contextual menu, after closing the profiles dialog open by the run action. He doesn't set up the profile, just closes the dialog.

Analysis: The barrier to running a project is too high. The concept of profiles is not helpful or necessary at the point and should only be introduced when there is need.

Recommendation(s): See below. See bug 33606

1, 2, 4, 5
Issue: A participant tries to execute a non-active project using "Run Active Project" menu item or shortcut.

Analysis: Active project issue.

Recommendation(s): See above. See bug 33604

1, 2, 3, 4, 5
Issue: A participant executed the active project using main menu, but it runs the old profile pointing to incorrect main class, which has been deleted. The participant doesn't know where is the problem. The output just says "No class def found error."

Analysis: UI Bug

Recommendation(s): When the class an execution profile depends on is deleted the the profile should be deactivated as well. See bug 33606

3
Issue: No feedback is printed into the output window about started or finished execution. Especially if user code doesn't prints anything into the output window.

Analysis: UI Bug

Recommendation(s): Improve output window feedback. See bug 33596

1, 5
Issue: A participant executes main class from its contextual menu. The class is executed even if there are a build errors.

Analysis: n/a

Recommendation(s): A class with build errors shouldn't be run. See bug 33607

2, 4
Setting up a Profile
Description and Recommendations Participants Resolution
Issue: A participant closes the profiles dialog because he thinks he doesn't need it for execution.

Analysis: Profiles problem - the concept does not have an obvious connection with running the project.

Recommendation(s): The profile may not be necessary for many cases where the user wants to run a project -removing it from the direct path would reduce the number of steps to running a project.

In simple cases with only one runnable class the IDE should run the class without extra input from the user. When creating a new class that is runnable the IDE should keep track and perhaps offer the a choice when Run project is later selected. More work needs to go into the profile issue. See bug 33606

2, 5
Issue: A participant invokes the run action from project's popup menu, but it displays the profiles dialog even if a profile exists. It is because non of the profiles is active(default).

Analysis: UI Bug

Recommendation(s): When only one profile exists it should be the default. See bug 33606

4
Issue: User doesn't find where to set the arguments by looking into menus, so reads the welcome screen.

Analysis: n/a

Recommendation(s): None.

2
Issue: The main class selector doesn't display which class is runnable, so a participant has to open files in the editor to figure out which class is the main class.

Analysis: Show information specifically relevant to the functionality.

Recommendation(s): Class selector should display runnable classes in the project. Navigating through the entire source tree does not help the user. See bug 33598

4
SETTING UP BUILD DEPENDENCIES
Add a Project as a Resource to Another Project
Description and Recommendations Participants Resolution
Issue: A participant can't find how to set a build dependencies between projects. Most of the participants look into: Project customizer, Project properties, Configuration editor. Few of participants are looking for "something named classpath".

Analysis: Two-fold problem: the logical structure of a project gets lost when the sources node is expanded -participants did not notice the Output or Resources node easily; the connection of Java concepts to Project concepts is not obvious (stated above).

The problem is not specifically connecting up two projects but understanding what the different parts of a project provide -that Output is the built bits of the code and Resources are the libraries one uses when building a project. Once those concepts were understood or explained by the moderator the rest of the task (opening the wizard and choosing the right template) was trivial.

Recommendation(s): Wording needs to be improved and the logical layout to keep things from getting lost when viewed with the expanse of nodes that appear under the Sources node. See bug 33603

1, 2, 3, 4, 5
Issue: A participant invokes help to read about the project dependencies, but doesn't understand it.

Analysis: n/a

Recommendation(s): None. Help was clarified after the first test run.

1
Issue: A participant selected a wrong template in the new resource wizard.

Analysis: Communication issue -the names of things, especially concepts particular to the NetBeans need to be very clear and obvious at a glance.

Recommendation(s): Go over naming and other visual cues. See bug 33596

2, 3
Issue: A participant adds a resource in the runtime classpath panel of profile customizer. But it is not working for building the project.

Analysis: Bug.

Recommendation(s): None.

4
Issue: A participant adds a resource in the runtime classpath panel of profile customizer. Selects "use everywhere" in the last panel, but after finishing the wizard the resource is not shown in the list. He does this four times with the same result.

Analysis: Bug. Adding a Resource and specifying that it should be used everywhere should work.

Recommendation(s): None.

4
Issue: A participant adds the java class library as a resource to runtime class path of run profile. He points to ".../anagram-lib/src" directory.

Analysis: The participant may not have completely understood the task and drawn the connection between adding the Output of a different project as the best way to proceed. Instead the participant tried to add the sources which is reasonable given that the class files are currently left in place with the sources.

Recommendation(s): Go over wording. See bug 33596

5
Issue: A participant added ".../anagram-lib/src" directory as a resource. But it added also the "src" dir under the resources node. This is different from adding packages roots under the sources node.

Analysis: The level at which resources are added is not the same as the level at which sources are added.

Recommendation(s): The two need to be the same. See bug 33608

5

Conclusions

Several issues have been uncovered -some known, some surprises -and concrete changes need to be made to the UI and interaction to solve them:

  • Active Project

    The visual representation of the active project need to be improved 100 fold. Bolding the project title is probably the least that should be done to signify that the project is active. Other visual cues might involve changing the explorer background behind non-active projects to a light shade of gray (not enough to look disabled but to draw focus to the active project with the white background).

    Project wide actions need to be clearly labeled and have easy to understand rules to avoid the current context ambiguity between build and run (build applying to the current selection and run applying to the active project). Create a small set of duplicated actions in the main menu should be enough -Build, Run, Debug (with the standard shortcuts) apply to the Active project and Build <project name>, Run <project name>, Debug <project name> would apply to the currently selected or context-active project.

  • Profiles

    The necessity of dealing with Profiles for all cases of running and debugging needs to be removed. Several easy changes can be made that will help stream line the process for simple cases with only one runnable class and for creating new profiles when adding new runnable classes. The dialog should also include more text and better labels to make it more obvious what it is used for so that users are less likely to dismiss it out of hand.

    In addition to trimming down the intrusive nature of Profiles the number and complexity should be trimmed as well -run and debug profiles should all be one thing with one dialog -see mail on dev@projects.netbeans.org for more details [find link and drop it here]. A final point to consider is the name Profile -it probably would benefit from an adjective to make it more clear what it is.

    While configurations where not specifically tested in this study it would be beneficial to look at how they will be used and draw some comparisons between the success of Profiles and the success of yet another way to manage a group of settings.

  • Central Location of Project Options

    Settings The study saw the majority of participants returning to one or two places when looking to change settings, add things (Resources), and otherwise manipulate the project. Creating a one-shop for most of the project settings will solve this problem. The context sensitive hooks should remain as they will be useful for users who understand how to manipulate the context sensitive aspects of the IDE.

    Looking at the Options spec should provide some reasonable overlap in approach. The all encompassing customizer should be located off the Project node

  • Reorder Top Level Project Nodes

    Changing the order of the project nodes should help bring more attention to the Resources and Output nodes. When the Sources node is expanded these two nodes get lost in the lower parts of the explorer (even with a small project) and overlooked. The argument at the time was to move them down because they won't be used much -but for just that reason they should be kept at the top. The two are unlikely to be expanded so sources will likely be visible and by having them between the Sources node and the Project node they should be much more discoverable.

  • Clarify (de-generalize) Node Names in the Project

    The very general names of the nodes in a project -Source, Output and Resources -were not helpful in directing the participant where they needed to go to complete certain tasks or find certain information. Source was straight forward but the distinction between Source and Java Roots was not and needs to be re-examined. For both Resources and Output participants were surprised to look there for their intended purposes. In several cases participants tried to add the text file to the Resources node and none of the participants clearly understood that the Resources node is where the classpath is defined.

    Similarly with the Output node, the idea that the Output of a project is the compiled classes was confusing during the process of adding project dependencies -several participants wanted to simply add the source tree of one project to the Resources of the second.

    These errors demonstrate that the departure from Java specific terms in the case of Resources and generally vague terms like Output do not aid the user in navigating the IDE. A more appropriate solution would be to more accurately name the nodes for the type of project being worked with -the level of abstraction needed for a one-size fits all label ends up being counterproductive.

These central points are the big fish that need to be addressed. At this point the solutions for each of these issues needs to be worked through with the development teams building and depending on the Projects system -any attempts at solutions listed here are suggestions and for the time being only considered as such.

The wording, in line help, menu layout and other issues lurking in the above table will be taken care of as a matter of course.

Project Features

About this Project

ui was started in November 2009, is owned by Jiří Kovalský, and has 44 members.
By use of this website, you agree to the NetBeans Policies and Terms of Use (revision 20160708.bf2ac18). © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo
 
 
Close
loading
Please Confirm
Close