Author: Jan Rojcek
Study Dates: April 7th - 9th, 2004
Table of Contents:
Usability study of NetBeans IDE based on new build system with J2SE project
type showed that after fixing few problems the UI of new build system would be
easy to use during the first hour of usage (if performing the same type of
tasks as were used during the study). In the study we noted 25 UI problems
directly related to UI of new build system and J2SE project type.
The most visible problems included:
- Participants didn't understand warning message about main class not being set. [BS-1]
- Participants couldn't find out why the main project menu items were disabled. [BS-8]
- Participants confused project folder with project location. [BS-9]
- Participants couldn't find how to create JAR. [BS-10]
- Participants couldn't find Project Properties dialog. [BS-15]
- Participants didn't understand "No Class Def Found Error" - meaning the main class wasn't found. [BS-16]
- Participants didn't know how to create a package. [BS-21]
The study was focused on observing the first hour experience when using
NetBeans IDE based on new build system with J2SE project type. Eight
participants were asked to try to complete 8 tasks using NetBeans IDE based on
new build system. All problems and struggles observed while trying to complete
the tasks were noted down.
We could categorize participants according to programming experience as:
- 2 students (P3, P6)
- 4 junior programmers (P1, P2, P7, P8)
- 2 experienced programmers (P4, P5)
The full participant characteristics can be found here:
Participants.
Eight tasks were focused on testing following:
- Task 1: Browse the IDE
- Task 2: Create and run a Hello World application
- Task 3: Open and run a project
- Task 4: Create a project from existing sources and build its JAR
- Task 5: Set up classpath dependency between projects
- Task 6: Create resource file, package, main class and run the project
- Task 7: Set runtime arguments and run the project
- Task 8: Run a project
The full task's script can be found here:
Task Script.
Task 1: Browse the IDE
All participants had neutral comments about the IDE - at the first look it as
they expected.
- Test creating and running a simple desktop java application.
- G2.1: 80% of participants would use the expected steps to complete the task.
- G2.2: 80% of participants who decide to modify the project location would do not confuse project location with project folder.
- G2.3: 100% of participants would complete the task without facilitator's help.
- Invokes File | New Project
- Selects "Java Application" template
- Fills in a project name
- Optionally modifies the project location
- Optionally modifies the automatically created main class name
- Finishes the wizard
- Modifies the main class
- Invokes Run | Run Main Project (or through the main toolbar)
- G2.1: Passed (87.5% passed - P8 failed)
- G2.2: Passed (100% passed - 2 participants modified the location)
- G2.3: Failed (87.5% passed - P8 failed)
- P6 expected to see a file in the editor after creating a new project [BS-3]
- P6 created a new main class because he hadn't noticed the automatically created main class in Projects Window (or in the editor) [BS-2]
- P8 selected "J2SE Library" template - which means that the main project wasn't set, also a main class wasn't set and main project actions were disabled [BS-4]
- P8 used contextual Run (because the main menu Run was disabled - and he couldn't figure out why) [BS-4]
- P8 Failed: Facilitator had to help with explaining why it didn't run - because the main class hadn't been set up [BS-1]
- P8 expected that Project Properties would allow him to select the main class (not to write it in manually) [BS-5]
- Test opening and running an existing project.
- G3.1: 80% of participants would use the expected steps to complete the task.
- G3.2: 80% of participants would understand that unlike in other IDEs they are expected to locate the project folder in NB.
- G3.3: 100% of participants would complete the task without facilitator's help.
- Invokes File | Open Project
- Selects a project folder
- Opens a project
- Invokes Run | Run Main Project (or through the main toolbar)
- G3.1: Failed (37.5% passed - P1, P3, P6, P7, P8 failed)
- G3.2: Failed (50% passed - P1, P3, P6, P7 failed)
- G3.3: Failed (87.5% passed - P1 failed)
- All participants were looking for a project file in Open Project dialog [BS-6]
- P1, P4, P9 opened Open File dialog from the toolbar first [BS-7]
- P1 opened and closed Open Project dialog without success [BS-6]
- P1 closed the main project which made the Anagram project the only open (but it wasn't the main project) [BS-8]
- P1 Failed: Facilitator had to explain the concept of main project, because P1 didn't know how to run the project (the main project actions were disabled) [BS-8]
- P1, P3, P6, P7 Failed: Participant didn't select the project folder in Open Project dialog (they went into the project folder and then hit the open button, which surprisingly opened the project) [BS-6]
- P4 said he missed the Open Project action in the toolbar [BS-7]
- *P8 Failed:" He executed the main project from the contextual menu (this was most likely the result of using the contextual actions in the previous task - he didn't have the main project set then) [BS-4]
Task 4: Create a project from existing sources and build its JAR
- Test creating a project from existing sources.
- Test creating of JAR.
- G4.1: 80% of participants would use the expected steps to complete the task (only the first part - creating a project).
- G4.2: 80% of participants would learn that creating a JAR is part of the build process.
- G4.3: 100% of participants would complete the task without facilitator's help.
- Invokes File | New Project
- Selects "Java Library" template
- Clicks "Use Existing Sources" (not implemented)
- Finishes the wizard
Alternative:
- Invokes File | New Project
- Selects "Java Library" template
- Finishes the wizard without clicking "Use Existing Sources"
- Adds sources using the contextual menu "Add Existing Sources" (not implemented)
Note: As "Use Existing Sources" and "Add Existing Sources" were not
implemented, facilitator stopped participants, when he clicked "Use Existing
Sources" or he tried to use the contextual menu to add existing sources after
creating a project.
- G4.1: Failed (50% passed - P1, P2, P3, P6 failed)
- G4.2: Failed (25% passed - P1, P2, P3, P4, P6, P8 failed)
- G4.3: Failed (37.5% passed - P1, P2, P3, P6, P8 failed)
- P1, P2, P3, P6 Failed: Participant didn't understand that the project folder have to be empty and tried to create the project folder at the location of existing sources. [BS-9]
- P1, P2, P3, P6, P8 Failed: Participant didn't know at all how to create a JAR [BS-10]
- P3 built a different project without noticing that [BS-11]
- P4 Failed: Participant didn't notice that JAR was created during build so he invoked "jar" target from =build.xml='s contextual menu [BS-10]
- P7 couldn't find the location of project he just created [BS-12]
- P8 couldn't find where the jar has been created [BS-13]
- Test setting up classpath dependencies between projects
- G5.1: 80% of participants would use the expected steps to complete the task.
- G5.2: 100% of participants would complete the task without facilitator's help.
- Modifies the GUI project sources
- Invokes File | Project Properties (or contextual Project Properties) on GUI project
- Adds the library project into compilation classpath
- Closes Project Properties
- Builds and/or runs the GUI project
- G5.1: Failed (0% passed - All failed)
- G5.2: Failed (37.5% passed - P2, P3, P6, P7, P8 failed)
- P1, P5 Failed: Participant didn't add Project but JAR to the classpath. [BS-14]
- P2, P8 Failed: Participant couldn't find Project Properties dialog (he invoked contextual Property Sheet to modify the classpath). Facilitator had to help. [BS-15]
- P2 Failed: Participant ran the project with "No Class Def Found Error" (the main class was wrong) but he couldn't figure out what was wrong. Facilitator had to help. [BS-16]
- P3 Failed: Participant added JAR into runtime classpath only. Facilitator had to point him to compilation classpath. [BS-17]
- P3 invoked File | Project Properties for wrong project (he noticed that afterwards). [BS-18]
- P4 Failed: Participant copied the library jar into the GUI project and then he added this JAR into the compilation classpath. [???]
- P5 added JAR also into the runtime classpath. [BS-19]
- P6 Failed: Participant didn't have a clue how to set up dependency (wasn't even looking for project properties). [???]
- P6, P7 Failed: Participant ran a different project which didn't have a main class set. He didn't understand what was wrong (he thought something was wrong with the classpath), facilitator had to help. [BS-20]
- Test working with resource files, creating a package, main class and setting up the main class.
- G6.1: 80% of participants would use the expected steps to complete the task.
- G6.2: 100% of participants would complete the task without facilitator's help.
- Invokes File | New File (or contextual New File)
- Selects Text File template
- Fills in a text file name and location
- Finishes the wizard
- Modifies the library sources
- Invokes File | New File (or contextual New File)
- Selects Java Package template
- Fills in a java package name
- Finishes the wizard
- Invokes File | New File (or contextual New File)
- Selects Java Main Class template
- Fills in a java class name
- Selects Source Packages package root
- Selects correct package
- Finishes the wizard
- Sets up the main class in project properties of library project
- Runs the library project
- G6.1: Failed (37.5% passed - P1, P2, P5, P6, P7 failed)
- G6.2: Failed (50% passed - P1, P2, P5, P6 failed)
- P1, P5, P6 Failed: Participant didn't set the main class for library project and couldn't figure out what was wrong. Facilitator had to help. [BS-1]
- P2 Failed: Participant couldn't find how to create a package. Facilitator helped. [BS-21]
- P2 created the resource file directly underneath the project folder and couldn't find it because it wasn't shown in Projects window. [BS-22]
- P5 didn't know where to put the resource file to have it on classpath. Was looking for something like "rsc" folder, was surprised that "src" was the right place. [BS-23]
- P7 Failed: Participant didn't find "Java Package" template, so he used "Folder" template. [BS-21]
- P7 Failed: Participant copied the resource file manually into the "classes" folder. [BS-24]
- P8 needed to copy the resource file into the default package, but the default package was hidden so he did it outside of the IDE. [BS-25]
- Test using the project properties and running a project
- G7.1: 80% of participants would use the expected steps to complete the task.
- G7.2: 100% of participants would complete the task without facilitator's help.
- Modifies the main class in library project
- Invokes File | Project Properties (or contextual Project Properties)
- Sets up the argument
- Closes the project properties
- Runs the library project
- G7.1: Passed (100% passed)
- G7.2: Passed (100% passed)
- P2 used contextual menu for running the library project even if it was the main project [???]
- Test running a project using contextual or main menu
- G7.1: 80% of participants would use the expected steps to complete the task.
- G7.2: 100% of participants would complete the task without facilitator's help.
- Invokes Run | Run Main Project (if the GUI project is the main project)
- Invokes contextual menu item (if the GUI project is not the main project)
- G7.1: Passed (100% passed)
- G7.2: Passed (100% passed)
- P5 ran a wrong project on the first try. [???]
Running a project that didn't have a main class set ended up with message
written into the Output window. None of the participants found this message
helpful, because they couldn't finish the task.
Because of this problem:
- 1 participant couldn't finished Task 1
- 3 participants couldn't finish Task 6
If user runs a project which doesn't have a main class set, following dialog
appears.
+--------------------------------------------------------------+
| Warning - Running "My Library" Project |
+--------------------------------------------------------------+
| |
| My Library project doesn't have a main class set. |
| |
| Please select the main class: |
| +----------------------------------------------------------+ |
| | com.mylibrary.test.Test | |
| | com.mylibrary.test.InteractiveTest | |
| | | |
| | | |
| +----------------------------------------------------------+ |
| |
| [ OK ] [ Cancel ] |
+--------------------------------------------------------------+
If a project doesn't have a main class then the same dialog appears with OK
button disabled.
+--------------------------------------------------------------+
| Warning - Running "My Library" Project |
+--------------------------------------------------------------+
| |
| My Library project doesn't have a main class set. |
| |
| Please select the main class: |
| +----------------------------------------------------------+ |
| | <No main class found in the project> | |
| | | |
| | | |
| +----------------------------------------------------------+ |
| |
| [ OK ] [ Cancel ] |
+--------------------------------------------------------------+
An automatically created main class wasn't expanded in Projects window. Few
participants commented this fact, and one participant didn't notice at all that
this class was created so he created another main class.
Because of this problem all participants were slowed down in completing the task.
Newly created project should always be expanded in Projects window and if a
file is automatically created with the project, then it should be selected.
In case of Java Application template a newly created project would be expanded
like this:
[ ] My Application
- [ ] Source Packages
- [ ] com.myapp
- [ ] Main.java - selected
+ [ ] Test Packages
Java Library template would be expanded like this:
[ ] My Library
- [ ] Source Packages
- [ ] <default package> - selected
+ [ ] Test Packages
One participant (student with VS.Net past experience) expected a file open in
the editor after creating a new project. He created another main class without
looking into the Projects window.
Because of this problem one participant was significantly slowed down.
If a file is automatically created with project, we should consider opening an
automatically created file in the editor.
If participant used J2SE Library template for creating a first project, the
Run Main project and Build Main project menu items were disabled. Participant
couldn't figure out why they were disabled, because he didn't notice the
concept of main project yet. So, he finally used contextual menu items.
Because of this problem one participant was significantly slowed down.
There are two ways how to solve this problem:
- Don't allow the situation when none of the open projects is set as main. So, if user creates any project and there is no project open yet, then make this newly created project as main automatically. This might not work very well for finding [BS-8]. If users closes the main project, it would be incorrect to set one of the open projects as main automatically.
- Never disable the main project menu items if there is at least one project open. If the user invokes the menu item and there is no main project set it would ask the user to set the main project (or select the main project directly in some kind of dialog).
I propose to use the second solution and show this dialog:
+--------------------------------------------------------------+
| Warning - Running Main Project |
+--------------------------------------------------------------+
| |
| The main project hasn't been set. |
| |
| Please select the main project: |
| +----------------------------------------------------------+ |
| | My Application | |
| | My Library | |
| | | |
| | | |
| +----------------------------------------------------------+ |
| |
| [ OK ] [ Cancel ] |
+--------------------------------------------------------------+
Most participants had problems with filling the main class name in Project
Properties dialog manually. Few participants didn't know at first weather to
fill in a full package name or just a class name.
Because of this problem all participants were slowed down in completing the task.
Allow the user to select the main class from all main classes in the project.
Alternative 1 (from the UI Spec):
Main Class: |_org.myapp.Main______________| [ Browse... ]
Alternative 2 (with editable combo box):
Main Class: |_org.myapp.Main___________________________|v|
All participants were looking for a project file when they wanted to open a
project. Some of them finally opened the project but they didn't select the
project folder directly. They went into the project folder and selected nothing
- looks like
FileChooser? misbehavior.
Because of this problem all participants were slowed down in completing the task.
I propose to introduce a project file. It would have the same name as project
folder and NB specific extension. Its only purpose would be to serve the role
of open project handle from inside and outside of the IDE. It should also have
a NB specific icon.
Few participants used the open file icon in main toolbar as the first choice
when they wanted to open a project.
Because of this problem few participants were slowed down in completing the task.
Three alternatives:
- Put only open project action into the toolbar.
- Put both - open project and open file actions into the toolbar.
- Put only open file action into the toolbar, and allow to open project files using this action. It would most likely not open the required projects.
After closing the main project and not setting another project as main, the main
project menu items were disabled and participant couldn't figure out why.
Because of this problem 1 participant didn't finish Task 3.
Please, look at the recommendation for finding [BS-4].
Few participants tried to put project folder of newly created project at the
location of existing sources. They couldn't figure out why it didn't work.
Because of this problem 4 participants didn't finish Task 4.
???
[BS-10] Create JAR action was hidden.
Few participants couldn't find a way how to create JAR file for project.
Because of this problem 5 participants didn't finish Task 4.
- We need to make the Create JAR action visible in the contextual menu of
project.
- We might need to show Create JAR action also in the main menu.
A participant built a different project from the main menu without noticing
it by reading the build output.
One participant was slowed down by this problem.
??? Build output formatting needs to be specified.
[BS-12] Location of project is not shown in IDE.
A participant couldn't find location of newly created project. As a workaround
he went to New Project wizard to look at what was the default location of newly
created projects.
Because of this problem 1 participant was slowed down in completing the task.
Show a project folder non-editable field in project customizer:
+----------------------------------------------------------+
| |
| Project Folder: |_/home/me/work/My Library_____________| |
| |
| Java Platform: |_JDK 1.4_________________v_| [ Edit... ] |
| |
| Required Projects: |
| +------------------------------------------------------+ |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| +------------------------------------------------------+ |
| [x] Rebuild Required Projects |
| |
+----------------------------------------------------------+
[BS-13] Location of created JAR wasn't presented well.
A participant couldn't find where was the JAR file created after using Build
action.
Because of this problem one participant was slowed down in completing the task.
It is shown as part of the output messages during the build. If might be shown
as a message in Project Properties.
All participants added JAR not a project to the classpath when defining
dependencies.
Because of this problem all participants defined incorrect dependency between
projects.
The results of Task 5 are a bit misleading. All participants added JAR into
the classpath because in previous Task 4 they were explicitly asked to locate
the library JAR. It means that all participants were informed about existence
of the JAR file and therefor the easiest way how to modify the classpath was to
choose the first button "Add JAR". Therefor I propose only a small change in
ordering of buttons:
Classpath for Compiling Sources:
+----------------------------------+
| [l] xerces.jar | [ Add Project... ]
| [p] My Lib: my-lib.jar | [ Add Library... ]
| [l] XML Advanced Parser | [ Add JAR/Folder... ]
| |
| | [ Remove ]
| |
| | [ Move Up ]
| | [ Move Down ]
| |
| | [ Edit Library... ]
| |
+----------------------------------+
Two participants couldn't find the project properties dialog. They invoked
contextual Properties over the project's node and thought it was the dialog
they were looking for. Other participants were confused by having two kinds of
properties dialogs.
Because of this problem 2 participants didn't finish Task 5.
Contextual "Properties" over a project node should open the Project Properties
dialog. The special contextual "Project Properties" menu item should be
removed.
A participant couldn't recover from "No Class Def Found Error" written into the
output window. He didn't understand that it was a problem of incorrect main
class defined in Project Properties.
Because of this problem 1 participant didn't finish Task 5.
???
A participant added JAR into the Runtime classpath and couldn't figure out why
it wasn't working during compilation.
Because of this problem 1 participant didn't finish Task 5.
???
A participant invoked Project Properties from main menu on wrong project. He
noticed that after some time.
Because of this problem 1 participant was slowed down in completing the task.
???
[BS-19] Duplicate JAR on runtime classpath.
A participant wasn't sure about the runtime classpath including also the
compiling classpath so he added JAR also to the runtime classpath.
Because of this problem one participant was confused.
We should expand the compilation classpath in the runtime classpath list. So
that all compilation classpath items are directly visible in the runtime
classpath list.
Classpath for Running Project:
+-------------------------------------------+
| [l] xerces.jar [Compiling Sources] | [ Add Project... ]
| [p] My Lib: my-lib.jar [Compiling Sources]| [ Add Library... ]
| [l] classes [Compiled Sources] | [ Add JAR/Folder... ]
| [l] XML Advanced Parser: xmlparse.jar |
| | [ Remove ]
| |
| | [ Move Up ]
| | [ Move Down ]
| |
| | [ Edit Library... ]
| |
+-------------------------------------------+
Two participants ran a different project which didn't have a main class set and
they couldn't find out what was wrong.
Because of this problem 2 participants didn't finish Task 5.
???
A participant didn't find how to create a package. Another participant used
Folder template because he couldn't fine the Java Package template.
Because of this problem 1 participant didn't finish Task 6.
- Implement the New submenu from the UI spec.
- Implement the first panel of New File wizard from the UI Spec.
- Put "Java Package" template into all java class related categories.
A participant couldn't see the file he just created underneath the project folder
in Projects window.
Because of this problem 1 participant was significantly slowed down.
???
A participant expected different than "src" folder to be used for resource text
file. He thought about "rsc" or similar.
Because of this problem one participant was slowed down in completing the task.
???
A participant copied the resource text file manually into the classes folder to
make sure it would show up on the classpath.
Because of this problem 1 participant didn't really finish Task 6.
???
1 participant wanted to paste a class into the hidden default package in
Projects window but he couldn't, so he achieved that outside of the IDE.
Because of this problem 1 participant was slowed down in completing the task.
???
