OOBE Usability Test Report

Tested by: Jindra Dinga, Rudolf Bock
Tested: November 2-4, 2004

Date Prepared: November 2004
Prepared by: Jindra Dinga, Gabriel Tichy

Table of Contents:

  • Method
  • Performance Results
  • Findings and Recommendations
  • Executive Summary

    All participants did not have any problems with completing tasks. Two of nine participants got a free task, the rest of them got a task to test a new prototype of GUI editor. Even if all participants could not do whatever they wanted, they were attracted by NetBeans and they wanted to try it more at home. In addition to that, at the end of that session participants had to make a decision about using NetBeans IDE, which was based on "playing" with this IDE during that study. A scale, where 1 meant "the IDE is the best and I want to use it" and 5 meant "the IDE is really bad I do not want to use it", was given to them and the result is that proffesionals evaluated NetBeans by average mark 1.75 and beginners by 1.5. On the question, if they recommend NetBeans to their friends, everybody said yes. To sum it up, we can say that all participants had a good first impression with NetBeans, so it makes sense to continue with 3rd and 4th step(observing users who using competitive IDE, and usres who migrated from NetBeans) of evaluating NetBeans.

    Introduction

    Purpose

    The usability study was focused on observing the OOBE & first hour experience in NetBeans IDE. 9 participants were asked to complete 7 tasks using NetBeans IDE 4 Beta2 build.

    Mostly qualitative data (problems and their causes, user expectations) were collected during the tests.

    Method

    Participants

    For our purposes we distinguished two groups of users:

    • Java Beginners
      • People who have up to 1 year of experience in Java.
      • People who have an experience in non-java programming language(s) at least for 5 years. They are thinking about using a Java IDE for their next project.
    • Java Professionals
      • This group of people has an experience with Java. They work with a competitive IDE, even they might migrated from NetBeans IDE.

    For the test, 10 people were recruited, 4 were beginners, 6 were professionals. The other characteristics of the participants that might influence usability were recorded, together with the age group and gender.

      Group Age Gender Experience with NetBeans Experience with Java
    1 Professional 20-35 M YES YES
    2 Beginner 20-35 M NO YES
    3 Professional 20-35 M NO YES
    4 Beginner 20-35 M YES YES
    5 Professional 20-35 M YES YES
    6 Beginner 20-35 M NO YES
    7 Beginner 20-35 M YES YES
    8 Beginner 20-35 M NO YES
    9 Professional 35-50 M NO YES
    10 -        

    *Participant #10 did not arrive on the evaluation

    Tasks

    The tasks were selected to reflect the out-of-the-box tasks, including tasks making the first impressions during the actual use of the product.

    The tasks selected for the test were:

    • Task 1: Visit www.netbeans.org and find out, if this IDE is really suitable for you.
    • Task 2: Download and install the latest version of NetBeans 4.0 IDE.
    • Task 3: Start IDE, browse the menu and functions and tell us, what do you expect behind.
    • Task 4: Select the topic related to this IDE, which you might be interested in, and explore it with Help.
    • Task 5: Write and run Hello World program. You can choose if the program display Hello World message a) into console b) into a dialog or c) into a web browser.
    • Task 6: Now you have about 45 minutes for trying of NetBeans 4.0 IDE out. After finishing, you should make a decision if you use it or not and why. Please, do everything that you think is important for evaluating this IDE. This task was completed only by first two participants. Then they were asked to try a new prototype of GUI editor.
    • Task 7: Uninstall NetBeans 4.0 IDE and delete all files you created.

    Every task contained also a bunch of questions asked right after the task completion. For details, see Appendix A.

    Test Setting

    Test was done in the usability laboratory in Prague. Sessions were recorded.

    The test setting was as follows:

    • The participant was seated at a typical work desk, moderator sat next to him.
    • A standard PC running Windows XP and NetBeans 4.0 Beta 2 was used.
    • A control person observed the test in the other room, logging the tasks. Simultaneously, a screen-capturing software was running.

    Procedure

    A complete test session consists of 4 sections: Participant general instructions and orientation, pre-task questionnaire, user test and post-test questionnaire.

    Whole test session takes approximately 120 minutes. The participants will not be „pushed“ by the moderator; instead they should perform the tasks as they would do it in normal situation.

    Participant General Instructions

    The participants receive a short, verbal introduction and orientation to the evaluation by explaining the purpose and objective of the evaluation, and additional information about what is expected of them. They are assured that the IDE is the center of the evaluation and not themselves, and that they should perform in whatever manner is typical and comfortable for them.

    Pre-Task Questionnaire

    The pre-task questionnaire collects Java experience, frequency of use, etc.

    User Test

    In the main part, the intrinsic usability test in the form of a moderated single interview will be performed. In doing so, the monitor provides the particular tasks and instructs the participant that she/he should try to complete them. In addition, the 'speak-aloud' method will be applied, which means that the test person will be asked to express her/his thoughts and expectations loudly, as well as to comment each single step she/he performs. This allows for identifying causes and reasons of user behaviour, particularly in case of problems. The monitor poses questions if required, but helps only in exceptional cases (e.g. to allow for the continuation of the test). After the completion of each task, the participants are asked to rate the perceived ease of that task or differences with their experience.

    Post-test questionnaire

    The post-test questionnaire collects general satisfaction with the IDE.

    Performance Results

    Surprisingly, all tasks were successfully done.

    Complete results, i.e. filled questionnaires (pre-, task, post-), together with graphs and times, are here.

    User Comments

    1 Website

    Participant
    Comments
    1.
    1. too much text, hard to find important things
    2.
    1. does not find NetBeans is for free
    2. looking for information about features, platforms, plugins,etc.
    3. prefers to download files immediately and then try them
    3.
    1. does not know NetBeans is for free
    4.
    1. looks for info about system requirements, platforms, etc.
    5.
    1. missing info about Refatoring
    6.
    1. prefers to download files immediately and then try them
    2. searches for offline manuals(in PDF format)-did not find, expected them in “Support” section, in “Download”
    3. looks for some basic information
    4. the website is not clear, czech version is missing
    7.
    1. does not find NetBeans is for free
    2. czech version of the website is missing
    8.
    1. does not find info about CVS
    2. czech version of the website is missing
    3. expects “Mouse Over Menu”
    4. missing screenshots
    5. searches for offline manuals(in PDF format)-did not find them
    9.
    1. searches for offline manuals(in PDF format)

    2 Downloading

    Participant
    Comments
    2.
    1. not sure about downloading files(explanation is missing)
    6.
    1. does not trust to “Beta” versions
    7.
    1. too many steps for getting the file
    8.
    1. confusing information(they talk about Beta version, in a different place they talk just about normal version)

    3.Installation

    Participant
    Comments
    8.
    1. wondering why there are two licence agreements(one on the website, another in installation proces)
    2. missing choice between standard/custom installation
    9.
    1. does not like to have icons on desktop-he wanted to select them in installer

    4 Browsing IDE, doing simple tasks


    Participant
    Comments
    1.
    1. appreciates generating of JavaDoc
    2. missing Move functions for Refactoring, not satisfied with implementation of refactoring
    3. complains about not showing any info during searching “ClassPath”(such as, where it is looking for it, etc.) - can not do anything during searching
    4. wants to have CodeCompletion more intuitive(if it is clear, it should not show all possibilities, just inserts the right code)
    5. does not like some fullscreen features(like in Help)
    2.
    1. not sure, what refactoring means
    2. complains about “Scanning of ClassPath”-does not know, what it is doing, missing info about it, not sure if it is doing just once(when he creates a project for the first time)-can not do anything during searching
    3. does not appreciate “Guard blocks” in GUI editor-thinks it is a limitationfor him
    3.
    1. appreciates JavaDoc
    2. GUI of NetBeans is dull for him-has seen better IDEs
    3. does not like “Scanning of ClassPath”-would appreciate estimated time-can not do anything during searching
    4.
    1. not sure, what refactoring means
    2. appreciates UpdateCenter
    3. complains about “Scanning of ClassPath”-does not know, what it is doing-can not do anything during searching
    5.
    1. does not know, why “Refresh all files” is enabled, even there are no files
    2. does not find more info about J2EE, tutorials for it
    3. missing a topic “if you want to do this.... you have to do this...” in Help
    6.
    1. missing a “Step-by-Step” tutorial
    7.
    1. complains about “Scanning of ClassPath”-can not do anything during searching
    2. did not notice “Output window”
    8.
    1. surprised by displaying of Welcome screen, ctritisizes the size of that screen does not match with the window size-not ready for 1024x768 resolution
    2. missing some basic shortcuts(e.g. for Exit in menu File) and other shortcuts, which are not dislayed, even they have them
    3. missing “Next/Previous Warning”, “Stop” in menu Build
    4. “Undo/Redo” in menu Refactor should be in menu Editor
    5. criticizes terminology of CVS, VCS, etc.
    6. menu View contains many fields
    7. does not like, Help has forwarded him to the website without any notice
    8. missing screenshots in Help
    9. finds IDE has Remote function, even this info is not on the website
    10. not satisfied with “Scanning of ClassPath”-expects it on the background-can not do anything
    11. criticizes speed of compilation
    9.
    1. not interested in Welcome screen
    2. does not know what Documentation Indicies”, in menu View, means
    3. “Web browser”, in menu Toolbar, is redundant
    4. does not like many version controllers, thinks in options there should be a possibility to select certain version controlles
    5. did not expect Runtime and Project window after starting IDE
    6. Help should not start in fullscreen
    7. missing information, in Help, what will happen with sources after importing them to IDE(will they be deleted, copied, etc.)
    8. not satisfied with “Scanning of ClassPath”-expects it on the background-can not do anything

    5 Uninstallation

    Participant
    Comments
    1.
    1. missing uninstall in ProgramFiles
    2. did not notice where the projects are saved
    2.
    1. missing uninstall in ProgramFiles
    2. looks for his projects in “My Projects” folder(IDE does not show, where the projects are saved)
    3.
    1. missing uninstall in ProgramFiles
    4.
    1. did not notice where the projects are saved
    5.
    1. missing uninstall in ProgramFiles
    6.
    1. missing uninstall in ProgramFiles
    7.
    1. missing uninstall in ProgramFiles
    8.
    1. missing uninstall in ProgramFiles

    6 Others


    Participants
    Comments
    1.
    1. would welcome articles in magazines, info material with screenshots, QuickStart Guide, etc.
    2.
    1. attracted by Welcome screen-likes it
    2. icons are big in toolbar
    3. Beta version should not have any bugs
    3.
    1. does not like Welcome screen-it is redundant
    4.
    1. likes Welcome screen-expected blank windows
    2. after starting IDE, he would appreciate an info about using it-wants to have more tutorials
    5.
    1. would appreciate more screenshots on the website
    2. IDE would start in fullscreen
    3. icons are big in toolbar
    4. does not like Welcome screen
    5. likes tutorials
    6. if something is not mentioned in Help, it is just a ad's bullshit
    6.
    1. does not like icons in toolbar
    9.
    1. ”Release Notes” is important source of information
    2.  prefers to try his own projects

    Findings and Recommendations

    Main Findings

    1 Website

    1.1 Users, who were mostly beginners, they did not know, that NetBeans is free of charge, they did not know that OpenSource means free of charge, or they were looking for the price of NetBeans and they could not find it. (3 participants)
    1.2 Most of users were looking for any offline manuals and they could not find it. They prefer to read about the program at first. (3 participants)
    1.3 Most of users said, that NetBeans website is not so clear, mostly because of too much text and missing screenshots (3-5 participants)
    1.4 Most of users were looking for some basic information at first. They had to browse many pages to find them. (3 participants)
    1.5 Most of users want to download the program immediately and try it on their own without longer browsing of website. (2 participants)
    1.6 Other inconsistency between website and NetBeans (users could not find that NB offers CVS, refactoring, etc. They would like to have website in czech version)

    2 Downloading

    2.1 Most of users do not trust to Beta versions. Maybe, we should not publish for all users only for community. (2 participants)
    2.2 Most of users complained about number of steps to download the file. Some steps can be joined together. We can also omit the "License agreement", it is in install. (2 participants)
    2.3 Some users were confused which file they should download. In the downloading process, there are two files: one file is NetBeans installer
    and the second one is NetBeans Mobility Pack installer. The description of these files are missing. (1 participant)

    3 Installation

    3.1 Some users want to have oportunity to choose if they want to add a shortcut on the desktop or not. (1 participant)

    4 Browsing IDE

    4.1 Almost all users complained about Scanning Classpath. It takes time and they can do nothing during this process. (7 participants)
    4.2 Two users complained Help starts in fulscreen. (2 participants)
    4.3 Some users are missing information what has happened with sources after importing them to the IDE (1 participant)
    4.4 Some users were wondering why there are tabs Runtime, Projects after staring NB for the first time. (1-2 participants)

    5 Uninstallation

    5.1 All users were looking for "Uninstall" in Program Files. The Shortcut is missing there. It might be because of trying Beta version. (7 partsicipants)
    5.2 Most of users did not know, where the projects are saved or they looked for them in different folders (such as My Projects). (3 participants)

    6 Others

    6.1 Half of participants like Welcome screen, another part do not.
    6.2 Some users want to read about NB in magazines, they want to have some info material with screenshots, QuickStart Guide, etc. (2 participants)
    6.3 Some users complained about icons in toolbar (they are big, not so clear) (3 participants)
    6.4 Users want the IDE to start in fullscreen. (1 participant)
    6.5 Two users want to have more tutorials. (2 participants)

    Recommendations

    1 Redesign Website

    a) Highlighting the fact that NetBeans is for free
    b) Add quick visible link to download most recent versions of NetBeans
    c) Omit "Licence agreement" before downloading ("LA" is in installation process as well)
    d) Include "off line" versions of documents
    e) Think over using of "mouse over" menu effect

    2 Downlaoding

    a) Add a short description of downloadable files
    b) Decrease number of steps for downloading of latest versions of NetBeans (now it has about 5 steps)

    3 Installation

    a) Add a choice of having an icon on desktop and in Program Files

    4 Browsing the IDE

    a) Let users work during scanning of classpath (it can run on background)

    5 Uninstallation

    a) Let users know where their projects are saved
    b) Add "uninstall" to Program Files

    To see our suggestion how the Wizard should look, please click here.

    Appendix A: Task Instructions

    Task 1: Visit www.netbeans.org and find out, if this IDE is really suitable for you. Note: Remember to think aloud, please.

    Questions:

    1. What kind of information are you interested in?

    2. Did you find them?

    Yes (continue with question no.3)

    No (continue with question no.4)

    3. Were you satisfied with these information?

    Yes (1) 2 3 4 No (5)

    4. What kind of information you didn't find?

    5. Give us a list of three attributes, which are the most important for you, to start using the IDE.

    1.

    2.

    3.

    6. Did you find these attributes?

    Yes

    No

    7. What do you think about NetBeans website?

    Clear (1) 2 3 4 Not clear (5)

    8. Give us a list of three points what do you like/dislike on NetBeans website.

    1.

    2.

    3.

    Task 2: Download and install the latest version of NetBeans 4.0 IDE.

    Questions:

    1. Was it easy/complicated to reach the desired file or not?

    Easy (1) 2 3 4 Complicated (5)

    2. How confident are you that you downloaded the correct file?

    Confident (1) 2 3 4 Not sure (5)

    3. Was the installation process clear?

    Clear (1) 2 3 4 Not clear (5)

    4. What would have made this task easier?

    Nothing

    The task would be easier, if (please, specify):


    Task 3: Start IDE, browse the menu and functions and tell us, what do you expect behind.

    Questions:

    1. What is the first thing that springs into your eye?

    2. What windows did you expect to open when you start the IDE?

    3. What is your overall first impression of this IDE?

    • layout

    Good (1) 2 3 4 Bad (5)

    • appearance

    Good (1) 2 3 4 Bad (5)

    • other (specify):

    Good (1) 2 3 4 Bad (5)

    Task 4: Select the topic related to this IDE, which you might be inetersted in, and explore it with Help.

    Questions:

    1. At first what kind of information in program do you look for?

    2. Why did/did not you use the tutorial?

    3. Do you have any suggestions for us?

    Task 5: Write and run Hello World program. You can choose if the program display Hello World message a) into console b) into a dialog or c) into a web browser.

    Questions:

    1. Have you found out any problems with completing this task?

    Yes (specify:)

    No (continue with question no.3)

    2. What would have made this task easier?

    Nothing

    The task would be easier, if (please, specify):

    3. Do you have any suggestions for us?

    Task 6: Now you have about 45 minutes for trying of NetBeans 4.0 IDE out. After finishing, you should make a decision if you use it or not and why. Please, do everything that you think is important for evaluating this IDE.

    Note: You can make notes on the paper if you want.

    Questions:

    1. You have already tried NetBeans IDE. Please, could you provide some recommendation, what would be improved? (give us a list of at least three points).

    1.

    2.

    3.

    2. If you compare NetBeans IDE to your favourite IDE, what are the main differences that you see?

    1.

    2.

    3.

    4.

    5.

    3. If you just downloaded and started NetBeans IDE at your home or work, in order to check it out, what would you do? Would have you same steps as you have done in this task?

    Yes

    No (specify:)

    4. Your decision:

    • about using NetBeans IDE

    Yes (1) 2 Not sure (3) 4 No (5)

    • about recommending NetBeans IDE to your friends

    Yes

    No

    Task 7: Uninstall NetBeans 4.0 IDE and delete all files you created.

    Questions:

    1. Have you found out any problems with completing this task?

    Yes (specify:)

    No (continue with question no.3)

    2. What would have made this task easier?

    Nothing

    The task would be easier, if (please, specify):

    3. Do you have any suggestions for us?

    Appendix B: Pre-task Questionnaire

    1. Are you a male/female?

    2. Are you a student/an employee?

    3. What is your education?

    • Elementary school
    • High school
    • College
    • University
    • Other (specify:)

    4. What kind of position/responsibility do you have in the company?

    • Employee
    • Manager
    • Director
    • Other (specify:)

    5. How many people does the company have?

    • 0-25
    • 25-50
    • 50-100
    • 100+

    6. Do you have a personal computer (at home/work)?

    • Yes (specify:)
    • No (continue with question no.12)

    7. What is typical hardware setting when developping applications?

    • Resolution of monitor
    • Operation system
    • RAM

    8. Do you have an Internet access?

    • Yes
    • No (continue with question no.11)

    9. What kind of connection do you have?

    • Dial-up
    • ADSL
    • Cable
    • Other (specify)

    10. What kind of browser do you use?

    • Mozilla
    • MS Explorer
    • Netscape
    • Other (specify)

    11. How often do you use a computer in a week?

    • Once a week
    • Twice-3 times a week
    • 4-5 times a week
    • 6-7 times a week

    12. What programming languages do you know?

    13. Do you have any experience with Java?

    • Yes
    • No (continue with question no.17)

    14. How long have you been using Java?

    15. What kind of projects do you use Java for?

    16. How often do you use Java in a week?

    • Once a week
    • Twice-3 times a week
    • 4-5 times a week
    • 6-7 times a week

    17. What IDE(s) do you know?

    18. What IDE do you use for your work?

    19. What advantages/disadvantages does the IDE have?

    20. What kind of tasks do you use your IDE for?

    21. Did you work in a different IDE in the past?

    • No (continue with question no.23)
    • Yes (specify:)

    22. What was the reason for migrating?

    23. What do you think the “right IDE” would have?

    24. Who is the decision maker, what IDE will be used for developping, in the company?

    25. What kind of IDE(s) do your colleagues use for developping?

    26. Have you ever heard about NetBeans IDE before?

    • No (exit)
    • Yes

    27. Where did you get the information about NetBeans IDE?

    • Press
    • Internet
    • Friend (occupation?)
    • Colleague (occupation?)
    • Other:

    28. What was the information?

    Good (1) 2 3 4 Bad (5)

    29. Have you ever worked in NetBeans IDE before?

    • Yes
    • No


    Appendix C: Post-task Questionnaire

    1. What do you like/dislike on this study?

    • like:

    • dislike:

    2. What do you like/dislike on NetBeans IDE?

    • like:

    • dislike:

    3. Is this session enough to decide whether NetBeans IDE is good or bad?

    Yes

    No (why?)

    4. How much time would the familiarizing with NetBeans approximately take you?

    • 1 day

    • 1 week

    • 1 month

    • 3-6 months

    • more than 6 months

    5. How comprehensible were the menu items for you?

    Clear (1) 2 3 4 Not clear (5)

    6. How comprehensible were the icons for you?

    Clear (1) 2 3 4 Not clear (5)

    7. Do you have any suggestions for us?

    Project Features

    About this Project

    ui was started in November 2009, is owned by Jiří Kovalský, and has 37 members.
    By use of this website, you agree to the NetBeans Policies and Terms of Use (revision 20140418.2d69abc). © 2013, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo
     
     
    Close
    loading
    Please Confirm
    Close