WebApps Project Usability Study Report

Author: Jan Rojcek

Study Dates: May 20th - 21st, 2004

Table of Contents:

Executive Summary
Introduction
Participants
Tasks Summary
Task Analysis
Findings and Recommendations

Executive Summary

Usability study of NetBeans IDE based on new build system with WebApps project type showed that users were able to easily and quickly create simple web application. Participants positively commented the code completion in JSP files and possibility to debug JSPs. The most important problems found were related to creating new project from existing sources and library management.

The most visible problems included:

Participants didn't know how to add a library to project.
Participants didn't know how to use existing sources in NetBeans.

Introduction

The study was focused on observing the first hour experience when using NetBeans IDE based on new build system with WebApps project type. Eight participants were asked to try to perform 7 tasks using NetBeans IDE based on new build system. All usability issues observed while participants were performing the tasks were noted down.

Participants

We could categorize participants according to JSP/Servlet programming experience as:

1 non-JSP/Servlet programmer - P8
2 junior programmers - P4, P7
5 experienced programmers - P1, P2, P3, P5, P6

The full participant characteristics can be found here: Participants[Fix the link].

Tasks Summary

Eight tasks were focused on testing following:

Task 1: Browse the IDE
Task 2: Create and launch a Hello World web application
Task 3: Create and launch a more complicated web application
Task 4: Create a web project from existing sources
Task 5: Modify the source code of web application
Task 6: Use custom tag library
Task 7: Describe how you would use NB with your current project

The full task's script can be found here: Task Script.

Task Analysis

Task 1: Browse the IDE

Objectives

Observe participants' behavior when browsing the IDE for the first time.

Notes

P1 clicked links in the welcome screen and created and ran a new Java Application without problems.
P2 created a new Web Application but didn't launch it.
P3 clicked links in the welcome screen and noticed that this NB already has projects.
P4 clicked links in the the welcome screen and observed the menus thoroughly.
P5 invoked New Project wizard and looked into the menus.
P6 created and ran new Java Application without problems. He also ran the project in debugger.
P7 created a new Java Application and browsed the menus.
P8 browsed the main menus.

Task 2: Create and launch a Hello World web application

Objectives

Test creating and launching a simple web application.

Expected Steps

Invokes File | New Project
Selects "Web Application" template
Optionally modifies the project name
Optionally modifies the project location
Optionally modifies the context path and J2EE specification level.
Finishes the wizard
Modifies the generated index.jsp
Invokes Run | Run Main Project (or through the main toolbar)

Goal

G2.1: 80% of participants would use the expected steps to complete the task.
G2.2: 100% of participants would complete the task without facilitator's help.

Results

G2.1: Failed (62.5% passed - P1, P4, P8 failed)
G2.2: Passed (100% passed)

Notes

P1, P4, P8 Failed - participants didn't notice generated index.jsp and created a new start page. [WP-1]
P1 was creating a new JSP document and in the new file wizard checked the "Create as JSP Document (XML Syntax) even though he didn't understand the XML syntax and had to create another JSP afterwards. [WP-2]
P1 didn't know how to run his own web page which wasn't set as the main page of project. So he supplied the page in browser's address field. [WP-3]
P1 compiled JSP with wrong tag and the output window showed a link to build-impl.xml. [WP-4]
P2 and P5 couldn't find a way how to turn off Tomcat. [WP-5]
P3 mentioned that port 8084 was "non-standard". [???]
P4 created index.html.html. [WP-6]

Task 3: Create and run a more complicated web application

Objectives

Test creating of JSP file.

Expected Steps

Modifies index.jsp using code completion.
Invokes File | New File (or contextual alternative)
Selects "JSP" template
Fills in a JSP file name.
Finishes the wizard.
Modifies the JSP file using code completion.
Invokes Run | Run Main Project (or through the main toolbar)
Optionally iterates between modifying files and running project.

Goal

G3.1: 80% of participants would use the expected steps to complete the task.
G3.2: 100% of participants would complete the task without facilitator's help.

Results

G3.1: Failed (37.5% passed - P1, P2, P3, P4, P5 failed)
G3.3: Passed (100% passed)

Notes

P1, P4 Failed - participants didn't have the main page set, so they modified the address value in web browser. [WP-3]
P2, P5 Failed - participants created JSP with double extension something.jsp.jsp. [WP-6]
P2 Failed - participant created JSP with XML syntax, but he wanted just plain JSP, so he deleted the file and created new one. [WP-2]
P3 Failed - participant created JSP with XML syntax, but he wanted just plain JSP. He thought the problem was in the J2EE level of project, so he closed this project and created new one. [WP-2]
P6 had to redeploy the project, because it didn't appear correctly in the web browser after Run Project. [???]
P1, P4, P7, P8 created only static pages without passing the name parameter. [???]

Task 4: Create a web project from existing sources

Objectives

Test creating a web project from existing sources.

Expected Steps

Invokes File | New Project
Selects "Import Web Module" template
Fills in "C:\Work\FlipCoin" location
Optionally modifies the project name
Finishes the wizard

Invokes File | Project Properties (or contextual alternative)
Clicks Add Library button in Compiling Sources category
Selects JSTL1.1 library and adds it to classpath
Closes the Project Properties
Optionally invokes Build | Build Main Project (or through the main toolbar)
Invokes Run | Run Main Project (or through the main toolbar)

Goal

G4.1: 80% of participants would use the expected steps for importing web module.
G4.2: 80% of participants would use the expected steps for adding the library to project.
G4.3: 100% of participants would complete the task without facilitator's help.

Results

G4.1: Failed (50% passed - P2, P4, P6, P8 failed)
G4.2: Failed (37.5% passed - P3, P4, P5, P6, P8 failed)
G4.3: Failed (25% passed - P2, P3, P4, P5, P6, P8 failed)

Notes

P1 invoked Properties from the Files window and it opened the Property Sheet.
P2, P4, P6, P8 Failed - participants created new Web Application and wanted to "add directory" into this project. [WP-7]
P2, P4, P6, P8 Failed - facilitator had to point participant to correct project template. [WP-7]
P2 had to use Redeploy because the web app didn't work after adding the library. [???]
P3 selected FlipCoin/web folder as web module root in Import Web Module wizard. [WP-8]
P3, P5, P6, P8 Failed - facilitator had to explain how to add the library to project. [WP-9]
P3 was confused from "Edit Library" button in Project Properties. [???]
P4 selected WEB-INF folder as web module root in Import Web Module wizard. [WP-8]
P4, P6 Failed - participant added library to Packaging list. [WP-10]
P6 wanted to change Project Location for imported web module, but Finish was disabled without notifying why. [WP-11]
P7 selected Web Application template first, but returned from the second panel and picked a correct template. [???]

Task 5: Modify the source code of web application

Objectives

Test the code completion usage and regular editor usage.
Test repeated project launching.

Expected Steps

Modifies the source code using code completion
Invokes Run | Run Main Project (or through the main toolbar)
Optionally iterates between modifying files and running project.

Goal

G5.1: 80% of participants would use the expected steps to complete the task.
G5.2: 100% of participants would complete the task without facilitator's help.

Results

G5.1: Passed (100% passed)
G5.2: Passed (100% passed)

Notes

P1 used debugger because Run didn't work at first (because of a bug in Run?). [???]
P3 wanted to stop "container" but couldn't find how. [???]
Run didn't work for P3, P4 (because of a bug?). [???]
P4, P6 used debugger. [???]
P5 was looking for "hot deploy" feature - compiling the jsp should upload it? [???]
P5 wasn't clear in what was really being done by Run vs. Redeploy. [???]
Code completion didn't work in jsp for P6. [???]

Task 6: Use custom tag library

Objectives

Test using a custom tag library in project.
Test using the code completion with custom tag library.

Expected Steps

Invokes File | Project Properties (or contextual alternative)
Clicks Add JAR button in Compiling Sources category
Locates the JAR file and adds it to the classpath
Closes the Project Properties
Modifies the source code using the code completion
Runs the project

Goal

G6.1: 80% of participants would use the expected steps to complete the task.
G6.2: 100% of participants would complete the task without facilitator's help.

Results

G6.1: Failed (75% passed - P4, P6 failed)
G6.2: Failed (75% passed - P4, P6 failed)

Notes

P1 wanted to backup index.jsp and copied it into project root what resulted in exception dialog. [???]
P1 created double extension jsp index.jsp.jsp when he wanted to backup the file. [WP-6]
P1 wanted to browse the library to read the URI and found it in All Files window. [???]
P3, P5, P7 wanted to browse the library to read the URI but couldn't find it. [WP-12]
P4, P6 Failed - participant added JAR to the Packaging list. [WP-10]
P4, P6 Failed - facilitator had to explain where to add the JAR file. [WP-10]