WebApps Project Usability Study Report
Author: Jan Rojcek
Study Dates: May 20th - 21st, 2004
Table of Contents:
- Executive Summary
- Introduction
- Participants
- Tasks Summary
- Task Analysis
- Task 1: Browse the IDE
- Task 2: Create and run a Hello World web application
- Task 3: Create and run more complicated web application
- Task 4: Create a web project from existing sources
- Task 5: Modify the source code of web application
- Task 6: Use custom tag library
- Task 7: Describe how you would use NB with your current project
- Findings and Recommendations
- [WP-1] Automatically created JSP wasn't expanded in Projects window nor opened in the editor
- [WP-2] "Create as JSP Document (XML Syntax)" check box wasn't understood
- [WP-3] Run file was hard to find
- [WP-4] Output window showed link to
build-imp.xml
- [WP-5] Hard to find how to turn off Tomcat
- [WP-6] New file wizard created files with double extensions
- [WP-7] Web Application project was the first choice when creating a project for existing sources
- [WP-8] "Web Module Root" button was confusing when locating existing web module
- [WP-9] Adding library to project is confusing
- [WP-10] Packaging project was preferred choice when adding library to classpath
- [WP-11] Import Web Module wizard didn't show why wizard buttons were disabled
- [WP-12] Impossible to browse library contents
Executive Summary
Usability study of NetBeans IDE based on new build system with WebApps project type showed that users were able to easily and quickly create simple web application. Participants positively commented the code completion in JSP files and possibility to debug JSPs. The most important problems found were related to creating new project from existing sources and library management.
The most visible problems included:
- Participants didn't know how to add a library to project.
- Participants didn't know how to use existing sources in NetBeans.
Introduction
The study was focused on observing the first hour experience when using NetBeans IDE based on new build system with WebApps project type. Eight participants were asked to try to perform 7 tasks using NetBeans IDE based on new build system. All usability issues observed while participants were performing the tasks were noted down.
Participants
We could categorize participants according to JSP/Servlet programming experience as:
- 1 non-JSP/Servlet programmer - P8
- 2 junior programmers - P4, P7
- 5 experienced programmers - P1, P2, P3, P5, P6
The full participant characteristics can be found here: Participants[Fix the link].
Tasks Summary
Eight tasks were focused on testing following:
- Task 1: Browse the IDE
- Task 2: Create and launch a Hello World web application
- Task 3: Create and launch a more complicated web application
- Task 4: Create a web project from existing sources
- Task 5: Modify the source code of web application
- Task 6: Use custom tag library
- Task 7: Describe how you would use NB with your current project
The full task's script can be found here: Task Script.
Task Analysis
Task 1: Browse the IDE
Objectives
- Observe participants' behavior when browsing the IDE for the first time.
Notes
- P1 clicked links in the welcome screen and created and ran a new Java Application without problems.
- P2 created a new Web Application but didn't launch it.
- P3 clicked links in the welcome screen and noticed that this NB already has projects.
- P4 clicked links in the the welcome screen and observed the menus thoroughly.
- P5 invoked New Project wizard and looked into the menus.
- P6 created and ran new Java Application without problems. He also ran the project in debugger.
- P7 created a new Java Application and browsed the menus.
- P8 browsed the main menus.
Task 2: Create and launch a Hello World web application
Objectives
- Test creating and launching a simple web application.
Expected Steps
- Invokes File | New Project
- Selects "Web Application" template
- Optionally modifies the project name
- Optionally modifies the project location
- Optionally modifies the context path and J2EE specification level.
- Finishes the wizard
- Modifies the generated index.jsp
- Invokes Run | Run Main Project (or through the main toolbar)
Goal
- G2.1: 80% of participants would use the expected steps to complete the task.
- G2.2: 100% of participants would complete the task without facilitator's help.
Results
- G2.1: Failed (62.5% passed - P1, P4, P8 failed)
- G2.2: Passed (100% passed)
Notes
- P1, P4, P8 Failed - participants didn't notice generated index.jsp and created a new start page. [WP-1]
- P1 was creating a new JSP document and in the new file wizard checked the "Create as JSP Document (XML Syntax) even though he didn't understand the XML syntax and had to create another JSP afterwards. [WP-2]
- P1 didn't know how to run his own web page which wasn't set as the main page of project. So he supplied the page in browser's address field. [WP-3]
- P1 compiled JSP with wrong tag and the output window showed a link to
build-impl.xml
. [WP-4] - P2 and P5 couldn't find a way how to turn off Tomcat. [WP-5]
- P3 mentioned that port 8084 was "non-standard". [???]
- P4 created
index.html.html
. [WP-6]
Task 3: Create and run a more complicated web application
Objectives
- Test creating of JSP file.
Expected Steps
- Modifies
index.jsp
using code completion. - Invokes File | New File (or contextual alternative)
- Selects "JSP" template
- Fills in a JSP file name.
- Finishes the wizard.
- Modifies the JSP file using code completion.
- Invokes Run | Run Main Project (or through the main toolbar)
- Optionally iterates between modifying files and running project.
Goal
- G3.1: 80% of participants would use the expected steps to complete the task.
- G3.2: 100% of participants would complete the task without facilitator's help.
Results
- G3.1: Failed (37.5% passed - P1, P2, P3, P4, P5 failed)
- G3.3: Passed (100% passed)
Notes
- P1, P4 Failed - participants didn't have the main page set, so they modified the address value in web browser. [WP-3]
- P2, P5 Failed - participants created JSP with double extension
something.jsp.jsp
. [WP-6] - P2 Failed - participant created JSP with XML syntax, but he wanted just plain JSP, so he deleted the file and created new one. [WP-2]
- P3 Failed - participant created JSP with XML syntax, but he wanted just plain JSP. He thought the problem was in the J2EE level of project, so he closed this project and created new one. [WP-2]
- P6 had to redeploy the project, because it didn't appear correctly in the web browser after Run Project. [???]
- P1, P4, P7, P8 created only static pages without passing the name parameter. [???]
Task 4: Create a web project from existing sources
Objectives
- Test creating a web project from existing sources.
Expected Steps
- Invokes File | New Project
- Selects "Import Web Module" template
- Fills in "C:\Work\FlipCoin" location
- Optionally modifies the project name
- Finishes the wizard
- Invokes File | Project Properties (or contextual alternative)
- Clicks Add Library button in Compiling Sources category
- Selects JSTL1.1 library and adds it to classpath
- Closes the Project Properties
- Optionally invokes Build | Build Main Project (or through the main toolbar)
- Invokes Run | Run Main Project (or through the main toolbar)
Goal
- G4.1: 80% of participants would use the expected steps for importing web module.
- G4.2: 80% of participants would use the expected steps for adding the library to project.
- G4.3: 100% of participants would complete the task without facilitator's help.
Results
- G4.1: Failed (50% passed - P2, P4, P6, P8 failed)
- G4.2: Failed (37.5% passed - P3, P4, P5, P6, P8 failed)
- G4.3: Failed (25% passed - P2, P3, P4, P5, P6, P8 failed)
Notes
- P1 invoked Properties from the Files window and it opened the Property Sheet.
- P2, P4, P6, P8 Failed - participants created new Web Application and wanted to "add directory" into this project. [WP-7]
- P2, P4, P6, P8 Failed - facilitator had to point participant to correct project template. [WP-7]
- P2 had to use Redeploy because the web app didn't work after adding the library. [???]
- P3 selected
FlipCoin/web
folder as web module root in Import Web Module wizard. [WP-8] - P3, P5, P6, P8 Failed - facilitator had to explain how to add the library to project. [WP-9]
- P3 was confused from "Edit Library" button in Project Properties. [???]
- P4 selected
WEB-INF
folder as web module root in Import Web Module wizard. [WP-8] - P4, P6 Failed - participant added library to Packaging list. [WP-10]
- P6 wanted to change Project Location for imported web module, but Finish was disabled without notifying why. [WP-11]
- P7 selected Web Application template first, but returned from the second panel and picked a correct template. [???]
Task 5: Modify the source code of web application
Objectives
- Test the code completion usage and regular editor usage.
- Test repeated project launching.
Expected Steps
- Modifies the source code using code completion
- Invokes Run | Run Main Project (or through the main toolbar)
- Optionally iterates between modifying files and running project.
Goal
- G5.1: 80% of participants would use the expected steps to complete the task.
- G5.2: 100% of participants would complete the task without facilitator's help.
Results
- G5.1: Passed (100% passed)
- G5.2: Passed (100% passed)
Notes
- P1 used debugger because Run didn't work at first (because of a bug in Run?). [???]
- P3 wanted to stop "container" but couldn't find how. [???]
- Run didn't work for P3, P4 (because of a bug?). [???]
- P4, P6 used debugger. [???]
- P5 was looking for "hot deploy" feature - compiling the jsp should upload it? [???]
- P5 wasn't clear in what was really being done by Run vs. Redeploy. [???]
- Code completion didn't work in jsp for P6. [???]
Task 6: Use custom tag library
Objectives
- Test using a custom tag library in project.
- Test using the code completion with custom tag library.
Expected Steps
- Invokes File | Project Properties (or contextual alternative)
- Clicks Add JAR button in Compiling Sources category
- Locates the JAR file and adds it to the classpath
- Closes the Project Properties
- Modifies the source code using the code completion
- Runs the project
Goal
- G6.1: 80% of participants would use the expected steps to complete the task.
- G6.2: 100% of participants would complete the task without facilitator's help.
Results
- G6.1: Failed (75% passed - P4, P6 failed)
- G6.2: Failed (75% passed - P4, P6 failed)
Notes
- P1 wanted to backup
index.jsp
and copied it into project root what resulted in exception dialog. [???] - P1 created double extension jsp
index.jsp.jsp
when he wanted to backup the file. [WP-6] - P1 wanted to browse the library to read the URI and found it in All Files window. [???]
- P3, P5, P7 wanted to browse the library to read the URI but couldn't find it. [WP-12]
- P4, P6 Failed - participant added JAR to the Packaging list. [WP-10]
- P4, P6 Failed - facilitator had to explain where to add the JAR file. [WP-10]
Task 7: Describe how you would use NB with your current project
???
Findings and Recommendations
[WP-1] Automatically created JSP wasn't expanded in Projects window nor opened in the editor
3 participants hadn't noticed that index.jsp
was created together
with new web application and created additional start pages.
Severity
Because of this problem:
- 3 participants were significantly slowed down in completing Task 2
Recommendation
Expand the automatically generated index.jsp
in the editor.
[WP-2] "Create as JSP Document (XML Syntax)" check box wasn't understood
3 participants checked the check box even though they wanted to create only plain JSP. Participant didn't understand the XML syntax.
Severity
Because of this problem:
- 1 participant was significantly slowed down in completing Task 2
- 2 participants were significantly slowed down in completing Task 3
Recommendation
Reword the checkbox label and move it to less prominent location.
[WP-3] Run file was hard to find
3 participant couldn't find how to launch one file in browser, so they filled the address field in browser manually.
Severity
Because of this problem:
- 1 participant was significantly slowed down in completing Task 2
- 2 participants were significantly slowed down in completing Task 3
Recommendation
???
[WP-4] Output window showed link to build-imp.xml
Output window showed a link to build-impl.xml
when participant compiled the source code with wrong tag.
Severity
Because of this problem:
- 1 participant was confused when performing Task 2
Recommendation
Don't show the links to metadata files in Output window.
[WP-5] Hard to find how to turn off Tomcat
2 participants were looking for a way how to turn off Tomcat but couldn't find any.
Severity
Because of this problem:
- 2 participants were confused when performing Task 2
Recommendation
???
[WP-6] New file wizard created files with double extensions
4 participants created file with double extensions.
Severity
Because of this problem:
- 1 participant was slowed down in completing Task 2
- 2 participants were slowed down in completing Task 3
- 1 participant was slowed down in completing Task 6
Recommendation
If the user supplies extension in the file name then don't append another one automatically.
[WP-7] Web Application project was the first choice when creating a project for existing sources
4 participants created Web Application and afterwards were looking for a way how to add existing sources into the project.
Severity
Because of this problem:
- 4 participants couldn't finish Task 4
Recommendation
???
[WP-8] "Web Module Root" button was confusing when locating existing web module
1 participant specified "web" folder and 1 participant "WEB-INF" folder as web module root in Import Web Module wizard.
Severity
Because of this problem:
- 2 participants were significantly slowed down in completing Task 4
Recommendation
???
[WP-9] Adding library to project is confusing
4 participants couldn't find how to add a library to project.
Severity
Because of this problem:
- 4 participants couldn't finish Task 4
Recommendation
???
[WP-10] Packaging project was preferred choice when adding library to classpath.
4 participants added library to packaging list instead of compiling classes.
Severity
Because of this problem:
- 2 participants couldn't finished Task 4
- 2 participants couldn't finished Task 6
Recommendation
???
[WP-11] Import Web Module wizard didn't show why wizard buttons were disabled
Wizard button didn't show inline error messages explaining why the button were disabled.
Severity
Because of this problem:
- 2 participants were slowed down in completing Task 4
Recommendation
[WP-12] Impossible to browse library contents
3 participants wanted to browse library contents but couldn't find where.
Severity
Because of this problem:
- 3 participant were slowed down in completing Task 6
Recommendation
???