Usability Study Report - Webpage, Installer, Plug-in Manager

Authors: Jakub Franc, Ondrej Langr
Study Facilitators: Jakub Franc, Ondrej Langr
Study Dates: October 8-10, 2007

Executive Summary

Usability study of NetBeans 6.0 web pages, download, installer and plug-in manager proved that users are generally able to perform their tasks. However, their user experience could be still improved. Several minor and medium importance problems associated with plug-in manager concept and organization and content of features pages were identified.

 

Table of Contents:

  • Visual Design
  •  

    Introduction

    Study Setting

     

    • Usability lab in Prague
    • NetBeans Beta 1 (latest milestone release at the time of the study)
    • Microsoft Windows XP operating system

     

    Research sample consisted of 10 participants proportionally representing several strategic target groups.
    Participants were recruited according to these criteria

    • Primary programming language (Java, C/C++)
    • Level of programming experience
    • Applications primarily developed (Web, Desktop, J2ME)
    • Current IDE usage
    • IDE related decision process - the way IDE is chosen in the team
    • Job status (professional/student)

     

    Tasks used for the usability study are available here.

    Study Goals

     

    The main goals of the study were to:

    • prove NetBeans 6.0 web pages feasibility in the scope of IDE evaluation (for both newcomers and current NetBeans users)
    • evaluate user experience of download and installation process
    • examine whether a new plug-in manager concept is comprehensible and pleases users' needs
    • get insight in users' IDE evaluation behavior and usage of IDE web sites

    Participants

     

    Participant Primary programming language Years of programming experience in primary language Applications primary developed Current IDE How he/his team choose the IDE Job status
    P1 Java 5 Web, desktop NetBeans 5.5, Eclipse 3.2 whole team consensus professional
    P2 C/C++ 7 Mathematic console apps Emacs himself student
    P3 Java 6 (C++ 10 years) Web, desktop NetBeans 5.5.1 chosen by manager student/ professional
    P4 Java 7 Web NetBeans 5.5 chosen by manager professional
    P5 Java 4 Desktop IntelliJ IDEA 6 chosen according the customer's platform professional
    P6 C/C++ 1 Database PS pad, Devccp himself professional
    P7 Java 3 Web, desktop NetBeans 5.5 himself professional
    P8 Java 3 Web, desktop Eclipse himslef student/ professional
    P9 Java 2 Web, desktop, database Eclipse 3.2, Websphere 5.1 chosen by manager professional
    P10 Java 8 Mobile, Web IntelliJ IDEA himslef professional

     

    Evaluation and Findings

    Web - Home Page

     

    The homepage worked well in context of given tasks. Users looking for download, description of NetBeans IDE and documentation and support did not experience any major obstacles.

    For the evaluation use-case which was tested, users typically left homepage very quickly towards the features page or download page. Both links to download page and features page were very easy to find for all participants.

    Some parts of the page seemed not to be very frequently used and lacking their purpose in the context of given use-cases. We also asked participants which parts of the page were least useful to them. The most frequently mentioned parts were the Products window/tile, rather frequently also. Top right part with the graphics and news was generally ignored. Users did almost never report it to be one of the least useful parts of the page, nonetheless, when asked, they were almost never able to tell what was in this part of the page.

    Typical navigation from the main page was for a participant to choose very quickly either towards the Features page or towards the download page, depending on the current task. When heading towards the features page, most common navigation element used was the Learn more... link, tag cloud title and tag cloud links were also used.

    Given the evaluative use-case, homepage seems to be a good starting point, but having no informative value at all. Adding Features page functionality to Homepage would serve evaluators' needs better. 

    Furthermore, post-study interviews fully confirmed our hypothesis that evaluation was the most frequent motive for visiting IDE web sites among our participants. They visit their IDE web pages in monthly to yearly frequency. The main reasons for visiting IDE web pages are:

    • Evaluation of new release of the current IDE
    • Evaluation of support for newly employed technology
    • Finding particular topic in documentation (typically via Google)

      Those who follow Java development related blogs (Roumen's blog was mentioned couple times) reported they never visit them via their IDE web pages, but directly or using RSS.

    Features Page

     

    During interviews, we identified two main approaches to IDE evaluation (please consider the blurry frontiers):

    1. Technology based approach. User comes to the page to find out whether the IDE supports particular technologies (required by current project, cool technologies of personal interest). This approach seems to be typical for managers and team evaluators. The task formulation directed participants towards this approach.
    2. Overall impression approach. User comes to the page without any list of specific feature requirements. He/she wants to see what the IDE offers to them. These are often current NetBeans users evaluating new release. Our Highlights page fully corresponds to this approach.

    Some users (often those with technology based approach) would prefer to have information about supported technologies including version numbers in a bulleted list or table. This applies for:

    • Evaluators & Decision Makers: Some users have reported that this is what a person choosing the IDE for their team is doing anyway. This is a must-have for busy managers, who may otherwise not be willing to invest time to find all required technologies in the textual product description.
    • Those who search for technology not supported by NetBeans IDE out-of-the-box: This list/table would make this particular use-case much easier. User would either not find the technology there at all (most likely using browser's search function) or would find it there with a note "available through update-center"
    • Those who search for less common technologies not mentioned on prominent places such as highlights page and paragraph titles

    Recommendation:

    • Create the "Supported Technologies" page containing a list of technologies.

     

    Typical Navigation Flow

    Most participants came to the features page and chose one of the highlighted features matching (or closest to match) the desired technology (as mentioned above, this can be caused by task formulation which asked participants to learn whether NetBeans supports the technologies needed for their current project).

    In case they did not find the desired information on the page they got to, they usually used the left menu to navigate further

    Recommendations:

    • We should make sure that the user understands that Highlights page actually represents top NetBeans features and does not cover all functionality

     

    What worked

    • Excellent discoverability of the Download button. Some users used the download button on the features page despite that the task started with "Return to homepage and download... "
    • UML screenshot with diagram types

     

    Problems and Suggestions (High-level)

    • Users sometimes tend to call technologies by different name than we do. Moreover, too general paragraph titles do not allow users to find desired information at the first glance. This can cause a problem when participant performs a quick scan through a page.
      • "Enterprise" word VS 'Java EE'
      • "Ant" VS 'Project System' - A participant searched for ant support within NetBeans and was significantly slowed down because 'Project System' paragraph title did not provide clear association..

      Recommendation: Be sensitive to users' language and mental concepts.

    • Some participants reported that information presented seemed too much as a "marketing gaff" to them.
      • Participants do not want to hear "easy to use GUI Builder" as it is too generic and anyone can write it about their GUI builder. Much better title is e.g. "GUI Builder with Automatic Alignment and Snapping"
      • Participants wanted to evaluate the product on basis of solid technical information, not according to someone else's (marketing) statements

      Recommendation: Use informative well-descriptive paragraph titles.

     

    Findings (Low-Level)

    Particular pieces of information users had troubles to find:

    • One participant came to the "Web & Java EE" page and mentioned that he does not see any Java EE information
    • Generally, Support for Standards paragraph within the Web & Java EE page seemed to be of a big importance, but not given place prominent enough
      • Supported versions of J2EE
      • Support for JSP
    • Can C++ editor debug? After a while on C/C++ page the user concluded it cannot.
    • What application servers NetBeans supports?
    • Web Services (while at Web & Java EE page)

     

    Finding The name of categories differs between the left link panel and Highlights, mainly in case of SOA, WSDL, XML category
    Recommendation Join into a single category (fixed short after the study)

     

     

    Finding Some current NetBeans users were missing the information about which features are new/improved compared to NetBeans 5.5.
    Recommendation We will add the new/improved stamps over/next to paragraphs describing features which were added or significantly improved in NetBeans 6.0

     

     

    Finding Some users (former or current users of NetBeans especially) were looking for an option to download the "pack" (functionality) they were reading about.
    Recommendation

     

     

     

     

    Finding Support for standards was generally interesting for many participants while it's rather hidden in the text.
    Recommendation

     

     

     

    Finding Some participants attempted to click directly the title on Features/Highlights page.
    Recommendation Make the titles links

     

     

    Finding Some participants left from Features page (particularly it's C/C++ section) quickly to Docs & Training. The reasons behind might be that they did not find the amount of information on that page as sufficient.
    Recommendation The solution is to provide more information on the features pages so that users would not need to search for more information.

     

     

    Finding

    Some participants specifically asked for a flash presentation of the tool/plug-in (UML in particular).

    Recommendation If we do have these demos, they should be linked directly from features page.

     

    Download Page

     

    All participants finished the task of downloading NetBeans without any serious interaction problems.

    In short, the download page does not put too big cognitive load and does not require too long to accommodate the mental model of the download table. However, it does require it's user to invest some time into choosing what NetBeans version they actually want, which is given by rather large number of installation options and their complexity.

    Typical flow was to first read horizontally the NetBeans IDE Bundles, second scan vertically.

    Participants who aimed for the full version downloaded it rather fast. Reasons for choosing the smaller versions of the IDE were available disk space or unwanted dependencies (typical for Eclipse users), download size was rarely considered.

    Because the process of understanding the download page was rather fast and extremely difficult to observe, we believe that use of eye-tracking camera might eventually discover some minor issues, but this equipment was not available.

    Finding Discoverability of additional information about packs was very bad. On the other hand, none of the participants was in a need of this piece of information.
    Recommendation

     

     

    Installer

     

    Unfortunately, many participants were not able to finish the install procedure due to technical problems with the installer during the usability study (For issue report see http://www.netbeans.org/issues/show_bug.cgi?id=118346 ).

    Despite of these problems we consider the qualitative data obtained during the study as valid because the interaction flow is the same until the very last screen no matter whether the last screen is an error message or information about successful installation.

    Not a single participant tried to read or even just scan the License Agreement. When asked, most of them replied they never do so.

     

    What worked

    • most of the installer's flow: Participants did not happen to be in a situation where they would be in doubt what is going on
    • Customize button has excellent discoverability and tends to be used frequently (even though sometimes only for checking installed parts)

     

    Findings

    Finding Some participants (typically those aiming for a single particular functionality) tried to disable Base IDE in customization which was possible.
    Recommendation

    It should not be possible to disable the Base IDE when it is not possible to install NetBeans without it.

     

    Finding One considered weird that NetBeans icon is created on desktop when the installer is in about 15%.
    Recommendation

     

     

    Plug-in Manager

     

    Discoverability of Plug-in manager proved to be sufficiently good, most participants did not have any serious problems finding it. Some did try to search in options or help menu (why?) but finally, all participants got through without moderators' help.

    Once in the plug-in manager dialog, the typical navigation flow for a participant was to scan through all the tabs to associate their names with the functionality which only took a few seconds. Most of the tab names were quickly understood, only Downloaded was not very clear to some of them (some participants even asked how to add a plug-in they downloaded somewhere else, despite this was not part of any task).

    Surprisingly, many participants considered the concept of plug-in deactivation unnecessary, they often reported they would uninstall the plug-in instead.

    The discoverability of deactivate button was not very good. Despite the fact that most participants did at the end find this button, they often struggled with it and some did uninstall the plug-in instead. In many cases, the deactivate button was discovered after the user clicked the uninstall button and moved his/her attention to the new window. In this case, the Deactivate button happens to be right next to the new window:

    Plugin manager screenshot

     

    Problems and Suggestions (High Level)

    On the validation screen when installing a new plug-in, participants often reported that they do trust this particular plug-in because it's being downloaded from SUN's servers (quotation: "I'd expect SUN to have trustworthy things there, despite that they're not signed" [P9])

     

    Findings

    Finding Default size of the plug-in manager should be bigger. Many participants resized the window immediately after coming to this dialog. This puts unnecessary workload on the user and breaks his/her flow by forcing to focus them on adjusting their work environment instead of focusing on their tasks.
    Recommendation

    Make plug-in manager's default window size bigger

     

    Finding The Deactivate button is (as mentioned in the text above) difficult to discover at it's current location. Almost all participants tried to deactivate the plug-in by clicking on the "active" status icon
    Recommendation Make the status icon a control element for enabling/disabling the plug-in

     

    Finding A situation has occurred where a plug-in/feature checkbox and name (BPEL in particular) was disabled and thus impossible to check on. The UI did not communicate why. Was a restart needed? Is this just an edge-case caused by a wrong version of JDK?
    Recommendation  

     

    Visual Design

     

    Users' understanding of navigation and interactions generally was well supported by current visual design (except for issues described in appropriate chapters).

    However, overall evaluation of visual appearance was significantly more modest comparing to last year study. We heard comments like "ok", "good", "standard" or "nothing exceptional".  Couple participants mentioned that NetBeans web pages are losing its own personality and becoming "more standard" or "conforming to system appearance".

    Project Features

    About this Project

    ui was started in November 2009, is owned by Jiří Kovalský, and has 44 members.
    By use of this website, you agree to the NetBeans Policies and Terms of Use (revision 20160708.bf2ac18). © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo
     
     
    Close
    loading
    Please Confirm
    Close