Build System Usability Study Report

Author: Jan Rojcek

Study Dates: April 7th - 9th, 2004

Table of Contents:

Executive Summary

Usability study of NetBeans IDE based on new build system with J2SE project type showed that after fixing few problems the UI of new build system would be easy to use during the first hour of usage (if performing the same type of tasks as were used during the study). In the study we noted 25 UI problems directly related to UI of new build system and J2SE project type.

The most visible problems included:

Introduction

The study was focused on observing the first hour experience when using NetBeans IDE based on new build system with J2SE project type. Eight participants were asked to try to complete 8 tasks using NetBeans IDE based on new build system. All problems and struggles observed while trying to complete the tasks were noted down.

Participants

We could categorize participants according to programming experience as:

The full participant characteristics can be found here: Participants.

Tasks Summary

Eight tasks were focused on testing following:

The full task's script can be found here: Task Script.

Task Analysis

Task 1: Browse the IDE

All participants had neutral comments about the IDE - at the first look it as they expected.

Task 2: Create and run a Hello World application

Objectives

Goal

Expected Steps

Results

Notes

Task 3: Open and run a project

Objectives

Goal

Expected Steps

Results

Notes

Task 4: Create a project from existing sources and build its JAR

Objectives

Goal

Expected Steps

Alternative:

Note: As "Use Existing Sources" and "Add Existing Sources" were not implemented, facilitator stopped participants, when he clicked "Use Existing Sources" or he tried to use the contextual menu to add existing sources after creating a project.

Results

Notes

Task 5: Set up classpath dependency between projects

Objectives

Goal

Expected Steps

Results

Notes

Task 6: Create resource file, package, main class and run the project

Objectives

Goal

Expected Steps

Results

Notes

Task 7: Set runtime arguments and run the project

Objectives

Goal

Expected Steps

Results

Notes

Task 8: Run a project

Objectives

Goal

Expected Steps

Results

Notes

Findings and Recommendations

[BS-1] Misleading warning about missing main class

Running a project that didn't have a main class set ended up with message written into the Output window. None of the participants found this message helpful, because they couldn't finish the task.

Severity

Because of this problem:

Recommendation

If user runs a project which doesn't have a main class set, following dialog appears.

+--------------------------------------------------------------+
| Warning - Running "My Library" Project                       |
+--------------------------------------------------------------+
|                                                              |
| My Library project doesn't have a main class set.            |
|                                                              |
| Please select the main class:                                |
| +----------------------------------------------------------+ |
| | com.mylibrary.test.Test                                  | |
| | com.mylibrary.test.InteractiveTest                       | |
| |                                                          | |
| |                                                          | |
| +----------------------------------------------------------+ |
|                                                              |
|                                            [ OK ] [ Cancel ] |
+--------------------------------------------------------------+

If a project doesn't have a main class then the same dialog appears with OK button disabled.

+--------------------------------------------------------------+
| Warning - Running "My Library" Project                       |
+--------------------------------------------------------------+
|                                                              |
| My Library project doesn't have a main class set.            |
|                                                              |
| Please select the main class:                                |
| +----------------------------------------------------------+ |
| | <No main class found in the project>                     | |
| |                                                          | |
| |                                                          | |
| +----------------------------------------------------------+ |
|                                                              |
|                                            [ OK ] [ Cancel ] |
+--------------------------------------------------------------+

[BS-2] Automatically created main class wasn't expanded

An automatically created main class wasn't expanded in Projects window. Few participants commented this fact, and one participant didn't notice at all that this class was created so he created another main class.

Severity

Because of this problem all participants were slowed down in completing the task.

Recommendation

Newly created project should always be expanded in Projects window and if a file is automatically created with the project, then it should be selected.

In case of Java Application template a newly created project would be expanded like this:

[ ] My Application
 - [ ] Source Packages
    - [ ] com.myapp
       - [ ] Main.java  - selected
 + [ ] Test Packages

Java Library template would be expanded like this:

[ ] My Library
 - [ ] Source Packages
    - [ ] <default package>  - selected
 + [ ] Test Packages

[BS-3] Automatically created main class wasn't open in the editor

One participant (student with VS.Net past experience) expected a file open in the editor after creating a new project. He created another main class without looking into the Projects window.

Severity

Because of this problem one participant was significantly slowed down.

Recommendation

If a file is automatically created with project, we should consider opening an automatically created file in the editor.

[BS-4] Menu items working with main project were disabled

If participant used J2SE Library template for creating a first project, the Run Main project and Build Main project menu items were disabled. Participant couldn't figure out why they were disabled, because he didn't notice the concept of main project yet. So, he finally used contextual menu items.

Severity

Because of this problem one participant was significantly slowed down.

Recommendation

There are two ways how to solve this problem:

  1. Don't allow the situation when none of the open projects is set as main. So, if user creates any project and there is no project open yet, then make this newly created project as main automatically. This might not work very well for finding [BS-8]. If users closes the main project, it would be incorrect to set one of the open projects as main automatically.
  2. Never disable the main project menu items if there is at least one project open. If the user invokes the menu item and there is no main project set it would ask the user to set the main project (or select the main project directly in some kind of dialog).

I propose to use the second solution and show this dialog:

+--------------------------------------------------------------+
| Warning - Running Main Project                               |
+--------------------------------------------------------------+
|                                                              |
| The main project hasn't been set.                            |
|                                                              |
| Please select the main project:                              |
| +----------------------------------------------------------+ |
| | My Application                                           | |
| | My Library                                               | |
| |                                                          | |
| |                                                          | |
| +----------------------------------------------------------+ |
|                                                              |
|                                            [ OK ] [ Cancel ] |
+--------------------------------------------------------------+

[BS-5] Project Properties didn't propose a list of main classes

Most participants had problems with filling the main class name in Project Properties dialog manually. Few participants didn't know at first weather to fill in a full package name or just a class name.

Severity

Because of this problem all participants were slowed down in completing the task.

Recommendation

Allow the user to select the main class from all main classes in the project.

Alternative 1 (from the UI Spec):

Main Class: |_org.myapp.Main______________| [  Browse... ]

Alternative 2 (with editable combo box):

Main Class: |_org.myapp.Main___________________________|v|

[BS-6] Project didn't have a project file

All participants were looking for a project file when they wanted to open a project. Some of them finally opened the project but they didn't select the project folder directly. They went into the project folder and selected nothing - looks like FileChooser? misbehavior.

Severity

Because of this problem all participants were slowed down in completing the task.

Recommendation

I propose to introduce a project file. It would have the same name as project folder and NB specific extension. Its only purpose would be to serve the role of open project handle from inside and outside of the IDE. It should also have a NB specific icon.

[BS-7] It wasn't possible to open a project using the main toolbar

Few participants used the open file icon in main toolbar as the first choice when they wanted to open a project.

Severity

Because of this problem few participants were slowed down in completing the task.

Recommendation

Three alternatives:

  1. Put only open project action into the toolbar.
  2. Put both - open project and open file actions into the toolbar.
  3. Put only open file action into the toolbar, and allow to open project files using this action. It would most likely not open the required projects.

[BS-8] Menu items working with main project were disabled (2)

After closing the main project and not setting another project as main, the main project menu items were disabled and participant couldn't figure out why.

Severity

Because of this problem 1 participant didn't finish Task 3.

Recommendation

Please, look at the recommendation for finding [BS-4].

[BS-9] Project Folder and Project Location were not explained well.

Few participants tried to put project folder of newly created project at the location of existing sources. They couldn't figure out why it didn't work.

Severity

Because of this problem 4 participants didn't finish Task 4.

Recommendation

???

[BS-10] Create JAR action was hidden.

Few participants couldn't find a way how to create JAR file for project.

Severity

Because of this problem 5 participants didn't finish Task 4.

Recommendation

  1. We need to make the Create JAR action visible in the contextual menu of
project.
  1. We might need to show Create JAR action also in the main menu.

[BS-11] Build output wasn't understood well.

A participant built a different project from the main menu without noticing it by reading the build output.

Severity

One participant was slowed down by this problem.

Recommendation

??? Build output formatting needs to be specified.

[BS-12] Location of project is not shown in IDE.

A participant couldn't find location of newly created project. As a workaround he went to New Project wizard to look at what was the default location of newly created projects.

Severity

Because of this problem 1 participant was slowed down in completing the task.

Recommendation

Show a project folder non-editable field in project customizer:

+----------------------------------------------------------+
|                                                          |
| Project Folder: |_/home/me/work/My Library_____________| |
|                                                          |
| Java Platform: |_JDK 1.4_________________v_| [ Edit... ] |
|                                                          |
| Required Projects:                                       |
| +------------------------------------------------------+ |
| |                                                      | |
| |                                                      | |
| |                                                      | |
| |                                                      | |
| |                                                      | |
| |                                                      | |
| |                                                      | |
| |                                                      | |
| |                                                      | |
| +------------------------------------------------------+ |
| [x] Rebuild Required Projects                            |
|                                                          |
+----------------------------------------------------------+

[BS-13] Location of created JAR wasn't presented well.

A participant couldn't find where was the JAR file created after using Build action.

Severity

Because of this problem one participant was slowed down in completing the task.

Recommendation

It is shown as part of the output messages during the build. If might be shown as a message in Project Properties.

[BS-14] Project wasn't a choice when adding to the classpath.

All participants added JAR not a project to the classpath when defining dependencies.

Severity

Because of this problem all participants defined incorrect dependency between projects.

Recommendation

The results of Task 5 are a bit misleading. All participants added JAR into the classpath because in previous Task 4 they were explicitly asked to locate the library JAR. It means that all participants were informed about existence of the JAR file and therefor the easiest way how to modify the classpath was to choose the first button "Add JAR". Therefor I propose only a small change in ordering of buttons:

Classpath for Compiling Sources:                          
+----------------------------------+                      
| [l] xerces.jar                   | [  Add Project...   ]
| [p] My Lib: my-lib.jar           | [  Add Library...   ]
| [l] XML Advanced Parser          | [ Add JAR/Folder... ]
|                                  |                      
|                                  | [      Remove       ]
|                                  |                     
|                                  | [      Move Up      ]
|                                  | [     Move Down     ]
|                                  |                     
|                                  | [  Edit Library...  ]
|                                  |                      
+----------------------------------+                      

[BS-15] Project Properties dialog was too hard to find.

Two participants couldn't find the project properties dialog. They invoked contextual Properties over the project's node and thought it was the dialog they were looking for. Other participants were confused by having two kinds of properties dialogs.

Severity

Because of this problem 2 participants didn't finish Task 5.

Recommendation

Contextual "Properties" over a project node should open the Project Properties dialog. The special contextual "Project Properties" menu item should be removed.

[BS-16] "No Class Def Found Error" after specifying wrong main class.

A participant couldn't recover from "No Class Def Found Error" written into the output window. He didn't understand that it was a problem of incorrect main class defined in Project Properties.

Severity

Because of this problem 1 participant didn't finish Task 5.

Recommendation

???

[BS-17] Confusion between Runtime and Compilation classpath.

A participant added JAR into the Runtime classpath and couldn't figure out why it wasn't working during compilation.

Severity

Because of this problem 1 participant didn't finish Task 5.

Recommendation

???

[BS-18] File | Project Properties was project-sensitive.

A participant invoked Project Properties from main menu on wrong project. He noticed that after some time.

Severity

Because of this problem 1 participant was slowed down in completing the task.

Recommendation

???

[BS-19] Duplicate JAR on runtime classpath.

A participant wasn't sure about the runtime classpath including also the compiling classpath so he added JAR also to the runtime classpath.

Severity

Because of this problem one participant was confused.

Recommendation

We should expand the compilation classpath in the runtime classpath list. So that all compilation classpath items are directly visible in the runtime classpath list.

Classpath for Running Project:                          
+-------------------------------------------+                      
| [l] xerces.jar [Compiling Sources]        | [  Add Project...   ]
| [p] My Lib: my-lib.jar [Compiling Sources]| [  Add Library...   ]
| [l] classes [Compiled Sources]            | [ Add JAR/Folder... ]
| [l] XML Advanced Parser: xmlparse.jar     |
|                                           | [      Remove       ]
|                                           |                     
|                                           | [      Move Up      ]
|                                           | [     Move Down     ]
|                                           |                     
|                                           | [  Edit Library...  ]
|                                           |                      
+-------------------------------------------+                      

[BS-20] Running incorrect project without main class wasn't well communicated.

Two participants ran a different project which didn't have a main class set and they couldn't find out what was wrong.

Severity

Because of this problem 2 participants didn't finish Task 5.

Recommendation

???

[BS-21] Java Package template was hard to find.

A participant didn't find how to create a package. Another participant used Folder template because he couldn't fine the Java Package template.

Severity

Because of this problem 1 participant didn't finish Task 6.

Recommendation

  1. Implement the New submenu from the UI spec.
  2. Implement the first panel of New File wizard from the UI Spec.
  3. Put "Java Package" template into all java class related categories.

[BS-22] A file created underneath the project folder wasn't shown in Projects window.

A participant couldn't see the file he just created underneath the project folder in Projects window.

Severity

Because of this problem 1 participant was significantly slowed down.

Recommendation

???

[BS-23] "src" folder wasn't expected to be used for resource file

A participant expected different than "src" folder to be used for resource text file. He thought about "rsc" or similar.

Severity

Because of this problem one participant was slowed down in completing the task.

Recommendation

???

[BS-24] It was unclear how to add a resource file on the root of classpath.

A participant copied the resource text file manually into the classes folder to make sure it would show up on the classpath.

Severity

Because of this problem 1 participant didn't really finish Task 6.

Recommendation

???

[BS-25] Copy-paste into the hidden packages doesn't work in Projects window.

1 participant wanted to paste a class into the hidden default package in Projects window but he couldn't, so he achieved that outside of the IDE.

Severity

Because of this problem 1 participant was slowed down in completing the task.

Recommendation

???