Build System Usability Study Report

Author: Jan Rojcek

Study Dates: April 7th - 9th, 2004

Table of Contents:

Executive Summary

Usability study of NetBeans IDE based on new build system with J2SE project type showed that after fixing few problems the UI of new build system would be easy to use during the first hour of usage (if performing the same type of tasks as were used during the study). In the study we noted 25 UI problems directly related to UI of new build system and J2SE project type.

The most visible problems included:

  • Participants didn't understand warning message about main class not being set. [BS-1]
  • Participants couldn't find out why the main project menu items were disabled. [BS-8]
  • Participants confused project folder with project location. [BS-9]
  • Participants couldn't find how to create JAR. [BS-10]
  • Participants couldn't find Project Properties dialog. [BS-15]
  • Participants didn't understand "No Class Def Found Error" - meaning the main class wasn't found. [BS-16]
  • Participants didn't know how to create a package. [BS-21]

Introduction

The study was focused on observing the first hour experience when using NetBeans IDE based on new build system with J2SE project type. Eight participants were asked to try to complete 8 tasks using NetBeans IDE based on new build system. All problems and struggles observed while trying to complete the tasks were noted down.

Participants

We could categorize participants according to programming experience as:

  • 2 students (P3, P6)
  • 4 junior programmers (P1, P2, P7, P8)
  • 2 experienced programmers (P4, P5)

The full participant characteristics can be found here: Participants.

Tasks Summary

Eight tasks were focused on testing following:

  • Task 1: Browse the IDE
  • Task 2: Create and run a Hello World application
  • Task 3: Open and run a project
  • Task 4: Create a project from existing sources and build its JAR
  • Task 5: Set up classpath dependency between projects
  • Task 6: Create resource file, package, main class and run the project
  • Task 7: Set runtime arguments and run the project
  • Task 8: Run a project

The full task's script can be found here: Task Script.

Task Analysis

Task 1: Browse the IDE

All participants had neutral comments about the IDE - at the first look it as they expected.

Task 2: Create and run a Hello World application

Objectives

  • Test creating and running a simple desktop java application.

Goal

  • G2.1: 80% of participants would use the expected steps to complete the task.
  • G2.2: 80% of participants who decide to modify the project location would do not confuse project location with project folder.
  • G2.3: 100% of participants would complete the task without facilitator's help.

Expected Steps

  • Invokes File | New Project
  • Selects "Java Application" template
  • Fills in a project name
  • Optionally modifies the project location
  • Optionally modifies the automatically created main class name
  • Finishes the wizard
  • Modifies the main class
  • Invokes Run | Run Main Project (or through the main toolbar)

Results

  • G2.1: Passed (87.5% passed - P8 failed)
  • G2.2: Passed (100% passed - 2 participants modified the location)
  • G2.3: Failed (87.5% passed - P8 failed)

Notes

  • P6 expected to see a file in the editor after creating a new project [BS-3]
  • P6 created a new main class because he hadn't noticed the automatically created main class in Projects Window (or in the editor) [BS-2]
  • P8 selected "J2SE Library" template - which means that the main project wasn't set, also a main class wasn't set and main project actions were disabled [BS-4]
  • P8 used contextual Run (because the main menu Run was disabled - and he couldn't figure out why) [BS-4]
  • P8 Failed: Facilitator had to help with explaining why it didn't run - because the main class hadn't been set up [BS-1]
  • P8 expected that Project Properties would allow him to select the main class (not to write it in manually) [BS-5]

Task 3: Open and run a project

Objectives

  • Test opening and running an existing project.

Goal

  • G3.1: 80% of participants would use the expected steps to complete the task.
  • G3.2: 80% of participants would understand that unlike in other IDEs they are expected to locate the project folder in NB.
  • G3.3: 100% of participants would complete the task without facilitator's help.

Expected Steps

  • Invokes File | Open Project
  • Selects a project folder
  • Opens a project
  • Invokes Run | Run Main Project (or through the main toolbar)

Results

  • G3.1: Failed (37.5% passed - P1, P3, P6, P7, P8 failed)
  • G3.2: Failed (50% passed - P1, P3, P6, P7 failed)
  • G3.3: Failed (87.5% passed - P1 failed)

Notes

  • All participants were looking for a project file in Open Project dialog [BS-6]
  • P1, P4, P9 opened Open File dialog from the toolbar first [BS-7]
  • P1 opened and closed Open Project dialog without success [BS-6]
  • P1 closed the main project which made the Anagram project the only open (but it wasn't the main project) [BS-8]
  • P1 Failed: Facilitator had to explain the concept of main project, because P1 didn't know how to run the project (the main project actions were disabled) [BS-8]
  • P1, P3, P6, P7 Failed: Participant didn't select the project folder in Open Project dialog (they went into the project folder and then hit the open button, which surprisingly opened the project) [BS-6]
  • P4 said he missed the Open Project action in the toolbar [BS-7]
  • *P8 Failed:" He executed the main project from the contextual menu (this was most likely the result of using the contextual actions in the previous task - he didn't have the main project set then) [BS-4]

Task 4: Create a project from existing sources and build its JAR

Objectives

  • Test creating a project from existing sources.
  • Test creating of JAR.

Goal

  • G4.1: 80% of participants would use the expected steps to complete the task (only the first part - creating a project).
  • G4.2: 80% of participants would learn that creating a JAR is part of the build process.
  • G4.3: 100% of participants would complete the task without facilitator's help.

Expected Steps

  • Invokes File | New Project
  • Selects "Java Library" template
  • Clicks "Use Existing Sources" (not implemented)
  • Finishes the wizard

Alternative:

  • Invokes File | New Project
  • Selects "Java Library" template
  • Finishes the wizard without clicking "Use Existing Sources"
  • Adds sources using the contextual menu "Add Existing Sources" (not implemented)

Note: As "Use Existing Sources" and "Add Existing Sources" were not implemented, facilitator stopped participants, when he clicked "Use Existing Sources" or he tried to use the contextual menu to add existing sources after creating a project.

Results

  • G4.1: Failed (50% passed - P1, P2, P3, P6 failed)
  • G4.2: Failed (25% passed - P1, P2, P3, P4, P6, P8 failed)
  • G4.3: Failed (37.5% passed - P1, P2, P3, P6, P8 failed)

Notes

  • P1, P2, P3, P6 Failed: Participant didn't understand that the project folder have to be empty and tried to create the project folder at the location of existing sources. [BS-9]
  • P1, P2, P3, P6, P8 Failed: Participant didn't know at all how to create a JAR [BS-10]
  • P3 built a different project without noticing that [BS-11]
  • P4 Failed: Participant didn't notice that JAR was created during build so he invoked "jar" target from =build.xml='s contextual menu [BS-10]
  • P7 couldn't find the location of project he just created [BS-12]
  • P8 couldn't find where the jar has been created [BS-13]

Task 5: Set up classpath dependency between projects

Objectives

  • Test setting up classpath dependencies between projects

Goal

  • G5.1: 80% of participants would use the expected steps to complete the task.
  • G5.2: 100% of participants would complete the task without facilitator's help.

Expected Steps

  • Modifies the GUI project sources
  • Invokes File | Project Properties (or contextual Project Properties) on GUI project
  • Adds the library project into compilation classpath
  • Closes Project Properties
  • Builds and/or runs the GUI project

Results

  • G5.1: Failed (0% passed - All failed)
  • G5.2: Failed (37.5% passed - P2, P3, P6, P7, P8 failed)

Notes

  • P1, P5 Failed: Participant didn't add Project but JAR to the classpath. [BS-14]
  • P2, P8 Failed: Participant couldn't find Project Properties dialog (he invoked contextual Property Sheet to modify the classpath). Facilitator had to help. [BS-15]
  • P2 Failed: Participant ran the project with "No Class Def Found Error" (the main class was wrong) but he couldn't figure out what was wrong. Facilitator had to help. [BS-16]
  • P3 Failed: Participant added JAR into runtime classpath only. Facilitator had to point him to compilation classpath. [BS-17]
  • P3 invoked File | Project Properties for wrong project (he noticed that afterwards). [BS-18]
  • P4 Failed: Participant copied the library jar into the GUI project and then he added this JAR into the compilation classpath. [???]
  • P5 added JAR also into the runtime classpath. [BS-19]
  • P6 Failed: Participant didn't have a clue how to set up dependency (wasn't even looking for project properties). [???]
  • P6, P7 Failed: Participant ran a different project which didn't have a main class set. He didn't understand what was wrong (he thought something was wrong with the classpath), facilitator had to help. [BS-20]

Task 6: Create resource file, package, main class and run the project

Objectives

  • Test working with resource files, creating a package, main class and setting up the main class.

Goal

  • G6.1: 80% of participants would use the expected steps to complete the task.
  • G6.2: 100% of participants would complete the task without facilitator's help.

Expected Steps

  • Invokes File | New File (or contextual New File)
  • Selects Text File template
  • Fills in a text file name and location
  • Finishes the wizard

  • Modifies the library sources

  • Invokes File | New File (or contextual New File)
  • Selects Java Package template
  • Fills in a java package name
  • Finishes the wizard

  • Invokes File | New File (or contextual New File)
  • Selects Java Main Class template
  • Fills in a java class name
  • Selects Source Packages package root
  • Selects correct package
  • Finishes the wizard

  • Sets up the main class in project properties of library project
  • Runs the library project

Results

  • G6.1: Failed (37.5% passed - P1, P2, P5, P6, P7 failed)
  • G6.2: Failed (50% passed - P1, P2, P5, P6 failed)

Notes

  • P1, P5, P6 Failed: Participant didn't set the main class for library project and couldn't figure out what was wrong. Facilitator had to help. [BS-1]
  • P2 Failed: Participant couldn't find how to create a package. Facilitator helped. [BS-21]
  • P2 created the resource file directly underneath the project folder and couldn't find it because it wasn't shown in Projects window. [BS-22]
  • P5 didn't know where to put the resource file to have it on classpath. Was looking for something like "rsc" folder, was surprised that "src" was the right place. [BS-23]
  • P7 Failed: Participant didn't find "Java Package" template, so he used "Folder" template. [BS-21]
  • P7 Failed: Participant copied the resource file manually into the "classes" folder. [BS-24]
  • P8 needed to copy the resource file into the default package, but the default package was hidden so he did it outside of the IDE. [BS-25]

Task 7: Set runtime arguments and run the project

Objectives

  • Test using the project properties and running a project

Goal

  • G7.1: 80% of participants would use the expected steps to complete the task.
  • G7.2: 100% of participants would complete the task without facilitator's help.

Expected Steps

  • Modifies the main class in library project
  • Invokes File | Project Properties (or contextual Project Properties)
  • Sets up the argument
  • Closes the project properties
  • Runs the library project

Results

  • G7.1: Passed (100% passed)
  • G7.2: Passed (100% passed)

Notes

  • P2 used contextual menu for running the library project even if it was the main project [???]

Task 8: Run a project

Objectives

  • Test running a project using contextual or main menu

Goal

  • G7.1: 80% of participants would use the expected steps to complete the task.
  • G7.2: 100% of participants would complete the task without facilitator's help.

Expected Steps

  • Invokes Run | Run Main Project (if the GUI project is the main project)
  • Invokes contextual menu item (if the GUI project is not the main project)

Results

  • G7.1: Passed (100% passed)
  • G7.2: Passed (100% passed)

Notes

  • P5 ran a wrong project on the first try. [???]

Findings and Recommendations

[BS-1] Misleading warning about missing main class

Running a project that didn't have a main class set ended up with message written into the Output window. None of the participants found this message helpful, because they couldn't finish the task.

Severity

Because of this problem:

  • 1 participant couldn't finished Task 1
  • 3 participants couldn't finish Task 6

Recommendation

If user runs a project which doesn't have a main class set, following dialog appears.

+--------------------------------------------------------------+
| Warning - Running "My Library" Project                       |
+--------------------------------------------------------------+
|                                                              |
| My Library project doesn't have a main class set.            |
|                                                              |
| Please select the main class:                                |
| +----------------------------------------------------------+ |
| | com.mylibrary.test.Test                                  | |
| | com.mylibrary.test.InteractiveTest                       | |
| |                                                          | |
| |                                                          | |
| +----------------------------------------------------------+ |
|                                                              |
|                                            [ OK ] [ Cancel ] |
+--------------------------------------------------------------+

If a project doesn't have a main class then the same dialog appears with OK button disabled.

+--------------------------------------------------------------+
| Warning - Running "My Library" Project                       |
+--------------------------------------------------------------+
|                                                              |
| My Library project doesn't have a main class set.            |
|                                                              |
| Please select the main class:                                |
| +----------------------------------------------------------+ |
| | <No main class found in the project>                     | |
| |                                                          | |
| |                                                          | |
| +----------------------------------------------------------+ |
|                                                              |
|                                            [ OK ] [ Cancel ] |
+--------------------------------------------------------------+

[BS-2] Automatically created main class wasn't expanded

An automatically created main class wasn't expanded in Projects window. Few participants commented this fact, and one participant didn't notice at all that this class was created so he created another main class.

Severity

Because of this problem all participants were slowed down in completing the task.

Recommendation

Newly created project should always be expanded in Projects window and if a file is automatically created with the project, then it should be selected.

In case of Java Application template a newly created project would be expanded like this:

[ ] My Application
 - [ ] Source Packages
    - [ ] com.myapp
       - [ ] Main.java  - selected
 + [ ] Test Packages

Java Library template would be expanded like this:

[ ] My Library
 - [ ] Source Packages
    - [ ] <default package>  - selected
 + [ ] Test Packages

[BS-3] Automatically created main class wasn't open in the editor

One participant (student with VS.Net past experience) expected a file open in the editor after creating a new project. He created another main class without looking into the Projects window.

Severity

Because of this problem one participant was significantly slowed down.

Recommendation

If a file is automatically created with project, we should consider opening an automatically created file in the editor.

[BS-4] Menu items working with main project were disabled

If participant used J2SE Library template for creating a first project, the Run Main project and Build Main project menu items were disabled. Participant couldn't figure out why they were disabled, because he didn't notice the concept of main project yet. So, he finally used contextual menu items.

Severity

Because of this problem one participant was significantly slowed down.

Recommendation

There are two ways how to solve this problem:

  1. Don't allow the situation when none of the open projects is set as main. So, if user creates any project and there is no project open yet, then make this newly created project as main automatically. This might not work very well for finding [BS-8]. If users closes the main project, it would be incorrect to set one of the open projects as main automatically.
  2. Never disable the main project menu items if there is at least one project open. If the user invokes the menu item and there is no main project set it would ask the user to set the main project (or select the main project directly in some kind of dialog).

I propose to use the second solution and show this dialog:

+--------------------------------------------------------------+
| Warning - Running Main Project                               |
+--------------------------------------------------------------+
|                                                              |
| The main project hasn't been set.                            |
|                                                              |
| Please select the main project:                              |
| +----------------------------------------------------------+ |
| | My Application                                           | |
| | My Library                                               | |
| |                                                          | |
| |                                                          | |
| +----------------------------------------------------------+ |
|                                                              |
|                                            [ OK ] [ Cancel ] |
+--------------------------------------------------------------+

[BS-5] Project Properties didn't propose a list of main classes

Most participants had problems with filling the main class name in Project Properties dialog manually. Few participants didn't know at first weather to fill in a full package name or just a class name.

Severity

Because of this problem all participants were slowed down in completing the task.

Recommendation

Allow the user to select the main class from all main classes in the project.

Alternative 1 (from the UI Spec):

Main Class: |_org.myapp.Main______________| [  Browse... ]

Alternative 2 (with editable combo box):

Main Class: |_org.myapp.Main___________________________|v|

[BS-6] Project didn't have a project file

All participants were looking for a project file when they wanted to open a project. Some of them finally opened the project but they didn't select the project folder directly. They went into the project folder and selected nothing - looks like FileChooser? misbehavior.

Severity

Because of this problem all participants were slowed down in completing the task.

Recommendation

I propose to introduce a project file. It would have the same name as project folder and NB specific extension. Its only purpose would be to serve the role of open project handle from inside and outside of the IDE. It should also have a NB specific icon.

[BS-7] It wasn't possible to open a project using the main toolbar

Few participants used the open file icon in main toolbar as the first choice when they wanted to open a project.

Severity

Because of this problem few participants were slowed down in completing the task.

Recommendation

Three alternatives:

  1. Put only open project action into the toolbar.
  2. Put both - open project and open file actions into the toolbar.
  3. Put only open file action into the toolbar, and allow to open project files using this action. It would most likely not open the required projects.

[BS-8] Menu items working with main project were disabled (2)

After closing the main project and not setting another project as main, the main project menu items were disabled and participant couldn't figure out why.

Severity

Because of this problem 1 participant didn't finish Task 3.

Recommendation

Please, look at the recommendation for finding [BS-4].

[BS-9] Project Folder and Project Location were not explained well.

Few participants tried to put project folder of newly created project at the location of existing sources. They couldn't figure out why it didn't work.

Severity

Because of this problem 4 participants didn't finish Task 4.

Recommendation

???

[BS-10] Create JAR action was hidden.

Few participants couldn't find a way how to create JAR file for project.

Severity

Because of this problem 5 participants didn't finish Task 4.

Recommendation

  1. We need to make the Create JAR action visible in the contextual menu of
project.
  1. We might need to show Create JAR action also in the main menu.

[BS-11] Build output wasn't understood well.

A participant built a different project from the main menu without noticing it by reading the build output.

Severity

One participant was slowed down by this problem.

Recommendation

??? Build output formatting needs to be specified.

[BS-12] Location of project is not shown in IDE.

A participant couldn't find location of newly created project. As a workaround he went to New Project wizard to look at what was the default location of newly created projects.

Severity

Because of this problem 1 participant was slowed down in completing the task.

Recommendation

Show a project folder non-editable field in project customizer:

+----------------------------------------------------------+
|                                                          |
| Project Folder: |_/home/me/work/My Library_____________| |
|                                                          |
| Java Platform: |_JDK 1.4_________________v_| [ Edit... ] |
|                                                          |
| Required Projects:                                       |
| +------------------------------------------------------+ |
| |                                                      | |
| |                                                      | |
| |                                                      | |
| |                                                      | |
| |                                                      | |
| |                                                      | |
| |                                                      | |
| |                                                      | |
| |                                                      | |
| +------------------------------------------------------+ |
| [x] Rebuild Required Projects                            |
|                                                          |
+----------------------------------------------------------+

[BS-13] Location of created JAR wasn't presented well.

A participant couldn't find where was the JAR file created after using Build action.

Severity

Because of this problem one participant was slowed down in completing the task.

Recommendation

It is shown as part of the output messages during the build. If might be shown as a message in Project Properties.

[BS-14] Project wasn't a choice when adding to the classpath.

All participants added JAR not a project to the classpath when defining dependencies.

Severity

Because of this problem all participants defined incorrect dependency between projects.

Recommendation

The results of Task 5 are a bit misleading. All participants added JAR into the classpath because in previous Task 4 they were explicitly asked to locate the library JAR. It means that all participants were informed about existence of the JAR file and therefor the easiest way how to modify the classpath was to choose the first button "Add JAR". Therefor I propose only a small change in ordering of buttons:

Classpath for Compiling Sources:                          
+----------------------------------+                      
| [l] xerces.jar                   | [  Add Project...   ]
| [p] My Lib: my-lib.jar           | [  Add Library...   ]
| [l] XML Advanced Parser          | [ Add JAR/Folder... ]
|                                  |                      
|                                  | [      Remove       ]
|                                  |                     
|                                  | [      Move Up      ]
|                                  | [     Move Down     ]
|                                  |                     
|                                  | [  Edit Library...  ]
|                                  |                      
+----------------------------------+                      

[BS-15] Project Properties dialog was too hard to find.

Two participants couldn't find the project properties dialog. They invoked contextual Properties over the project's node and thought it was the dialog they were looking for. Other participants were confused by having two kinds of properties dialogs.

Severity

Because of this problem 2 participants didn't finish Task 5.

Recommendation

Contextual "Properties" over a project node should open the Project Properties dialog. The special contextual "Project Properties" menu item should be removed.

[BS-16] "No Class Def Found Error" after specifying wrong main class.

A participant couldn't recover from "No Class Def Found Error" written into the output window. He didn't understand that it was a problem of incorrect main class defined in Project Properties.

Severity

Because of this problem 1 participant didn't finish Task 5.

Recommendation

???

[BS-17] Confusion between Runtime and Compilation classpath.

A participant added JAR into the Runtime classpath and couldn't figure out why it wasn't working during compilation.

Severity

Because of this problem 1 participant didn't finish Task 5.

Recommendation

???

[BS-18] File | Project Properties was project-sensitive.

A participant invoked Project Properties from main menu on wrong project. He noticed that after some time.

Severity

Because of this problem 1 participant was slowed down in completing the task.

Recommendation

???

[BS-19] Duplicate JAR on runtime classpath.

A participant wasn't sure about the runtime classpath including also the compiling classpath so he added JAR also to the runtime classpath.

Severity

Because of this problem one participant was confused.

Recommendation

We should expand the compilation classpath in the runtime classpath list. So that all compilation classpath items are directly visible in the runtime classpath list.

Classpath for Running Project:                          
+-------------------------------------------+                      
| [l] xerces.jar [Compiling Sources]        | [  Add Project...   ]
| [p] My Lib: my-lib.jar [Compiling Sources]| [  Add Library...   ]
| [l] classes [Compiled Sources]            | [ Add JAR/Folder... ]
| [l] XML Advanced Parser: xmlparse.jar     |
|                                           | [      Remove       ]
|                                           |                     
|                                           | [      Move Up      ]
|                                           | [     Move Down     ]
|                                           |                     
|                                           | [  Edit Library...  ]
|                                           |                      
+-------------------------------------------+                      

[BS-20] Running incorrect project without main class wasn't well communicated.

Two participants ran a different project which didn't have a main class set and they couldn't find out what was wrong.

Severity

Because of this problem 2 participants didn't finish Task 5.

Recommendation

???

[BS-21] Java Package template was hard to find.

A participant didn't find how to create a package. Another participant used Folder template because he couldn't fine the Java Package template.

Severity

Because of this problem 1 participant didn't finish Task 6.

Recommendation

  1. Implement the New submenu from the UI spec.
  2. Implement the first panel of New File wizard from the UI Spec.
  3. Put "Java Package" template into all java class related categories.

[BS-22] A file created underneath the project folder wasn't shown in Projects window.

A participant couldn't see the file he just created underneath the project folder in Projects window.

Severity

Because of this problem 1 participant was significantly slowed down.

Recommendation

???

[BS-23] "src" folder wasn't expected to be used for resource file

A participant expected different than "src" folder to be used for resource text file. He thought about "rsc" or similar.

Severity

Because of this problem one participant was slowed down in completing the task.

Recommendation

???

[BS-24] It was unclear how to add a resource file on the root of classpath.

A participant copied the resource text file manually into the classes folder to make sure it would show up on the classpath.

Severity

Because of this problem 1 participant didn't really finish Task 6.

Recommendation

???

[BS-25] Copy-paste into the hidden packages doesn't work in Projects window.

1 participant wanted to paste a class into the hidden default package in Projects window but he couldn't, so he achieved that outside of the IDE.

Severity

Because of this problem 1 participant was slowed down in completing the task.

Recommendation

???

Project Features

About this Project

ui was started in November 2009, is owned by Jiří Kovalský, and has 37 members.
By use of this website, you agree to the NetBeans Policies and Terms of Use (revision 20140418.2d69abc). © 2013, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo
 
 
Close
loading
Please Confirm
Close