Editor Usability Study Report

Author: Jan Rojcek
Study Facilitators: Jakub Franc, Josef Pavlicek, Jan Rojcek
Study Dates: 25 - 26 August 2005

Table of Contents:

Executive Summary

Primary goal of editor usability study was to observe users writing code in NetBeans IDE 5.0 Beta. We wanted to see how easy/hard it is for them to write simple java code in NetBeans during the first hour experience, identify problematic areas and missing features. We also wanted to check usability of newly introduced editor features: code templates, editor hints and improved code completion.

We simulated situation when study participants evaluated new version of NetBeans IDE (5.0 Beta). Participants were users of Eclipse, JBuilder and NetBeans. They were given 5 tasks, 2 of them focused on code writing.

Main findings of the study are (Please note some of the findings might have been fixed since the study was done):

Our editor is behind competition in a number of editor features. That's the fact. On the other hand, the overall impression from the study is that participants were fighting more with usability of existing features than they missed features they were used to from Eclipse/JBuilder.

Participants

Participants in the study according to their IDE experience:

Task Analysis

Tasks:

The full wording of tasks is available in Task Script.

Task 1: Browse the IDE

Objectives
  • Observe participants' behavior while browsing the IDE for the first time.
  • Observe whether participants notice and/or interact with the welcome screen or not.
Result
  • 2 participants used the welcome screen (P3, P7)
  • 1 participant looked at the welcome screen and closed it without really using it (P2)
Notes
  • P2 closes the welcome screen after quick look at it ("some well-known things" - NetBeans user)
  • P2 sets small toolbar buttons in the main toolbar
  • P2 observes the main menu - notices new refactor items and CVS
  • P3 opens old options dialog and spends a minute looking at options (Eclipse user)
  • P3 clicks the Quick Start Guide in Welcome Screen
  • P4 ignores the welcome screen ("The IDE looks a bit better that it used to" - Eclipse user)
  • P4 is confused from "Scanning in Progress..." label underneath an expanded java file node - he doesn't see the progress bar
  • P4 tries the refactoring ("So refactoring works finally in NetBeans")
  • P4 browses the main menus - tries the CVS checkout dialog, Internationalization wizard, Server manager
  • P4 writes a very simple source code "int i = 1/0;" and runs it from the toolbar and looks at the error output
  • P5 switches to SDI. Can't find the Setup wizard (which was removed) so the facilitator points him to the option dialog ("If I have 3 monitors, I want to organize windows the way I like it.").
  • P6 reads the license text (JBuilder user)
  • P6 browses the main menu and tries New Project wizard
  • P7 thoroughly examines the main menu
  • P7 clicked the Sample Project button in the Welcome Screen
  • [ED-1] P7 creates a sample project and tries to create a new folder (probably a new source root). Created "Mock" folder using the new file wizard but he doesn't see the result in the Project window. He is confused because he found the folder in Win Explorer. Creates the folder again and is confused that he could do it as it already existed. Finds out the source package properties and now understands how to add a source folder.
  • [ED-2] P7 doesn't know how to change the label of source folder. Tries to click in the source package table but it doesn't seem to work. (He has to double-click).
  • [ED-3] P7 opens the source code and comments the source code coloring. He doesn't like that the color of strings is too prominent (red?) and that Javadoc comments have the same color as regular comments.
  • P8 browses through the main menu. Ignores the welcome screen.

Task 2: Create and launch a Hello World application

Objectives
  • Test intuitiveness of creating and launching a simple application.
Expected Steps
  • Invokes File | New Project
  • Selects the Java Application project
  • Optionally modifies the project name
  • Optionally modifies the project location
  • Finishes the wizard
  • Modifies the source code in main class
  • Invokes Run | Run Main Project (or through the main toolbar)
  • Reads the application output in Output Window
Goal
  • All (8) participants would complete the task without facilitator's help.
Result
  • All (8) passed - 0 failed
Notes
  • P1 overlooked the "Hello World!" message in the output window. Finds it after a while.
  • [ED-4] P2 invokes the code completion before the classpath scanning has completed and the completion popup says "Please wait..." instead of "Scanning in Progress..." (Participant says "If it could be a bit faster that would be better."). Then he notices the classpath scanning progress bar.
  • P3 runs the application using the Run File action in the contextual menu in Main.java editor.
  • P5 runs the application using the Run File action in the contextual menu on Main.java node.
  • P6 comments that the code completion is "pretty fast, thus it's not awkward to me".
  • [ED-5] P6 says he doesn't see a way how to open full javadoc from the javadoc popup. He hovers over the buttons in javadoc popup, but tooltips don't show up.
  • P7 intentionally creates main class in the default package and then moves it using refactoring.
  • P7 says "this code completion is the best I've seen so far".
  • [ED-6] P7, P8 would like to see the program output separated from build messages. P8 even searches for an option to switch the build messages off and finds "Verbosity" in old options and switches to "Quite" which results in hiding even the application output.
  • [ED-7] P8 wants to move to the next step by double-clicking on a project template in the new project wizard.
  • [ED-8] P8 expects code completion to be case insensitive. Writes "sys" and invokes code completion for "System". As our code completion is case sensitive by default it doesn't show "System".

Task 3: Write 2 nested "for" loops and "try-catch" block

Objectives
  • Test discoverability of code templates ("for"), editor hints ("try-catch") and observe usage of editor and editor features like surround with, code completion, etc.
Expected Steps
  • Writes "for" loops manually or using the "for" template
  • Uses editor hint for try-catch
Goal
  • a) All (8) participants complete the task without facilitator's help.
  • b) All (8) participants discover editor hints
  • c) 4 participants discover code templates
Result
  • a) 8 passed - 0 failed
  • b) 6 passed - 2 failed
  • c) 3 passed - 5 failed
Notes
  • [ED-9] P1 wants to find a code template for "for", but can't find it in templates manager, keyboard shortcuts, nor in old options dialog.
  • [ED-10] P2 cannot move to the end of line with "End" key while the completion popup is open His comment: "This is dumb".
  • [ED-11] P2 tries to use the try-catch hint on a block of code. He wants to encapsulate both "for" loops so selects them in the code, but at this point the light bulb disappears. He gets back to the try-catch around one line generated with the hint.
  • P3, P4, P8 discovers and uses the for template - writes "for" and invokes code completion.
  • [ED-12] P3 clicks and then right-clicks on error mark icon to get to the editor hints. Expects that editor hints are associated with the error notification. Instead a breakpoint is placed on that line after clicking the icon.
  • [ED-13] All participants have to use mouse to read the error message indicated with error icon in editor gutter. The message is in the tooltip over the error icon.
  • [ED-14] P6 wants to see indication about modified file in the status line ("I prefer '*' in the status line instead of in the tab.")
  • [ED-15] P5, P6 doesn't find the try-catch hint because he is already on a different line in source code so the light bulb doesn't show up. He has to manually write the try-catch block.
  • [ED-16] P6 writes the catch statement manually and wants the IDE to automatically fill in (or suggest to fill in) exceptions thrown in the try-catch block ("I have to manually find it and I'm a very lazy person.").
  • [ED-17] P6 wants to write "ie" as a variable of "InterruptedException" but the IDE expands this string to "interface".
  • [ED-18] P7 wants to create a new method using a new method dialog invoked from editor's contextual menu. He finds the Add menu (but only for class, interface, etc.) in node's contextual menu and goes again to check whether this menu is present in editor. It isn't. Then goes to the main menu to search for it. Gives up and writes it manually ("I will stop searching for it. I don't want to be here till tomorrow.").
  • [ED-19] P7 wants to invoke editor hint with the keyboard bud can't find the shortcut. He looks into the main menu and hovers over the icon but not long enough to show the tooltip with the shortcut.

Task 4: Open and run existing project

Objectives
  • Test whether participants are not confused that instead of project file they have to open project folder
Results
  • 1 participant didn't know how to open project and went to create new freeform project [ED-20], [ED-21]
  • 2 participants had small problems with opening project folder [ED-20]
  • 5 participants had no problems with opening project
Expected Steps
  • Invokes File | Open Project
  • Locates and selects AnagramGame folder
  • Clicks Open Project Folder button
  • Invokes Run | Run Main Project
Notes
  • P1 had no problems with opening the project folder.
  • P2 had no problems with opening the project folder.
  • P3 didn't know how to open the project in Open Project dialog because he was looking for project file: "It seems like the project doesn't exist." [ED-20]. He spent significant effort looking through the menus and browsing the project structure in Total Commander. And when he saw the build.xml file he went to create New Project with Existing Ant Script. The wizard told the user that: Project folder already contains NetBeans project data. He didn't understand the message, because he said "It's strange that the finish button is disabled". After a while the participant selected nbproject folder as the new project folder! Then the wizard said Project location is already owned by NetBeans project Anagram Game. The participant "solved" this by selecting src folder for the project location [ED-21]. The project was created but there were no source files shown in the project window except the build.xml file. The participant ran the project successfully.
  • P4 wants to import project first, but closes the New Project wizard in the first page. He looks at the project folder in windows explorer and then goes to Open Project dialog. He is a bit confused that the doesn't see a project file, but then he selects the project folder and opens the project [ED-20]. He runs the project without problems.
  • P5 had no problems with opening the project folder.
  • P6 tries Open File first: "I don't want to open it as project, because it is an application.". He cancels the dialog "I probably don't understand the task correctly" and goes to Open Project where he selects and opens project folder without any problems. When running the project, he comments that he would like to see description of contextual menu items in status line [ED-22].
  • P7 in the Open Project dialog doesn't select the project folder, but goes inside the project folder and as he doesn't select any folder there yet, the Open Project Folder button works on project folder. He clicks it and opens the project folder [ED-20].
  • P8 had no problems with opening the project folder. He comments that Home and End keys don't work in Go To Class dialog [ED-23].

Task 5: Write code with "for" and "while" loop, "try-catch" blocks and imports

Objectives
  • Test discoverability of for and while code template.
  • Test intuitiveness and effectiveness of error notification features.
  • Test intuitiveness and effectiveness of editor hints.
Results
  • 3 participants successfully used for template; 1 participant had problems finding while template [ED-45]; and 2 participants requested if template [ED-32]
  • All participant had some problem with error notification features. The most visible were clicking on the error annotation icon [ED-12]; missing keyboard-only access to error description [ED-13]; and slow error notification [ED-24]
  • 5 participants were able to use editor hints, even though hints are hard to discover and it's complicated to use them. 2 participants didn't discover that they can do try-catch on block of code from contextual menu [ED-11]; 6 participants expected hints on the error annotation icon [ED-12]; 6 participants had problems to find hints because they didn't put the cursor on the line containing error [ED-15]; and almost all participants had problems with understanding when the hint shows up because of 1 second delay to show it up [ED-27]
Expected Steps
  • This is an open task. No exact expected steps.
Notes
  • P2 writes the source code and he is informed about error on the line after he moves to another line [ED-24]. Writes the while loop manually. Discovers fix import editor hint. Looks at code completion showing code templates (in this case keywords) when it doesn't make sense [ED-25]. Uses create method editor hint for scramble method. The code templates completion that shows up in wrong context is shown above the method completion [ED-25]. Code completion is in the way: "There's no suggestion and I can't move up/down. This is really irritating." [ED-26]. It takes too long to show up the light bulb. He doesn't stay on 1 position long enough so that the light bulb appears [ED-27] or he doesn't put the cursor on the line with error statement [ED-15]. He clicks on an error message in the output window and the cursor is put into the source code and the parameter popup shows up: "What is it telling here? Some kind of parameter?" [ED-28]. Notices the pink background but doesn't understand what it means: "What is this color telling me?" [ED-29] [ED-30]. Participant has interesting idea how he would like to work with try-catch blocks: "I would like to select a block of code and click somewhere and the try-catch block would jump in" [ED-11]. Writes the try-catch block manually but the IDE doesn't offer any help when writing the catch statement [ED-16]. Runs the application and encounters null pointer because he didn't initialized ArrayList fields [ED-31]. Fixes the error. Runs the app.
  • P3 reads the javadoc on java.sun.com. Uses fori template. Clicks on error annotation which puts breakpoint on the line [ED-12]. Expects some code template for if [ED-32]. Wants to write br as variable name and the IDE expands it to break [ED-17]. Uses fori template. Expects keyword completion when writing private keyword [ED-33]. Writes import statement manually because he doesn't find the light bulb [ED-12], [ED-15], [ED-27]. Writes try-catch for block of code manually [ED-11], and expects some help from the IDE [ED-16]. Runs the project successfully even though the code is not correct according to the given task.
  • P4 makes a copy of WordLibrary class first in the same package (WordLibrary_1). Then renames this class using refactoring to WordLibrary2, but he misses the refactoring output to confirm refactoring so he doesn't actually rename the class: "Hmm, it looks like the IDE wasn't very successful." [ED-34]. Clicks error icon a few times and puts there a breakpoint [ED-12]. Wants to add a main method using Add contextual menu on java file node, it's not there [ED-18]. Uses create method editor hint for scramble method. Code completion could be smarter for new statements offering correct constructors [ED-35]. Uses fori template. Uses refactoring for renaming classes. Done.
  • P5 cannot find "Find Usages" contextual menu item, even though he looks at it a few times in contextual menu [ED-36]. He uses Czech characters in method and variable names. He wrote a method usage first and then method declaration and is upset that code completion doesn't offer the method name: "The name is right here" (it's 2 lines above :-). He has to write it manually [ED-37]. Wants to find out why he can't write '{' with Ctrl-Alt-B. Goes to the keyboard shortcuts dialog where he can't find it because it's not ordered alphabetically by shortcuts [ED-38]. Participant selected class field from code completion and the IDE thought it was class keyword [ED-39]. Turned off case sensitive code completion [ED-8]. Waits quite long for parsing to finish. Wants to see the error and hints [ED-24]. Cannot find the light bulb and clicks the error annotation [ED-12]. Made 2 typos and IDE didn't offer to correct them [ED-40]. Writes imports manually [ED-12], [ED-27]. IDE expands br variable name to break [ED-17]. Done.
  • P6 wants to find class in project in Find in Projects dialog and fails [ED-41]. IDE expands br variable name to break [ED-17]. The user comments that code completion should also generate open parentheses when calling constructor [ED-42]. Writes try-catch manually: "There's probably something to generate it, but I like writing it myself". Uses import hint. Done.
  • P7 looks for how to add javadoc to a method he just wrote. Looks into Source and contextual menu [ED-43]. Wants better completion for new statements [ED-35]. Finds and uses import hint. Cannot find javadoc for method he just wrote into the source code [ED-44]. Uses cast hint. Uses try-catch hint.
  • P8 wants to add method with some kind of Add method dialog. Cannot find it in main menu nor in editor's contextual menu [ED-18]. Wants smart new statements [ED-35]. Finds and uses import hint. Has problem locating hint because the cursor is not on that line [ED-12], [ED-15]. Uses import hint. Typed while and expects code template for it [ED-45]. Uses for template. Asks for if template [ED-32]. Cannot find Find Usages [ED-36]. User regular find dialog instead. Comments that he would like to see occurrences located by find dialog in error stripe [ED-46]. Uses cast hint. Done.

Findings and Recommendations

[ED-1] New Source Folder is hard to find

Finding
  • Participant creates a sample project and tries to create a new folder (probably a new source root). Created "Mock" folder using the new file wizard but he doesn't see the result in the Project window. He is confused because he found the folder in Win Explorer. Creates the folder again and is confused that he could do it even if it already existed. Finally finds out the source package properties and now understands how to add a source folder.
Severity
  • 1 participant was confused and slowed down
Recommendation
  • Participant expected to see something like "New Source Folder" option in project's contextual menu. We might consider designing something like that.

[ED-2] Cannot rename label in Source Package Folder with one mouse click

Finding
  • Participant doesn't know how to change the label of source folder. Tries to click on the label value in the source package table but it doesn't seem to work. (He has to double-click).
Severity
  • 1 participant couldn't complete renaming
Recommendation
  • Enable renaming of the label value after one click if the label cell is already selected

[ED-3] Default java code coloring

Finding
  • Participant comments the source code coloring. He doesn't like that the color of strings is too prominent (red/purple?) and that Javadoc comments have the same color as regular comments.
Severity
  • 1 participant didn't like the default java colors
Recommendation
  • Take this participant's opinion into consideration when designing new default color profile

[ED-4] Wrong label in code completion when waiting for classpath scanning

Finding
  • Participant invokes the code completion before the classpath scanning has completed and the completion popup says "Please wait..." instead of "Scanning in Progress..." (Participant says "If it could be a bit faster that would be better."). Then he notices the classpath scanning progress bar.
Severity
  • 1 participant thought for a while that the code completion was too slow
Recommendation
  • Use "Scanning in Progress..." label instead of "Please wait..." during classpath scanning.

[ED-5] Hidden function to open javadoc in browser from javadoc popup

Finding
  • Participant says he doesn't see a way how to open full javadoc from the javadoc popup. He hovers with the mouse over the buttons in javadoc popup, but tooltips don't show up.
Severity
  • 1 participant thought the feature doesn't exist 
Recommendation
  • Show tooltips over the buttons in javadoc popup and consider using textual buttons instead of icons only

[ED-6] Application output is buried between build output messages

Finding
  • 2 participants would like to see application output ("Hello World!") separated from build messages. One participant even searches for an option to switch the build messages off and finds "Verbosity" in old options and switches to "Quite" which results in hiding even the application output.
Severity
  • 2 participants didn't like the way the IDE treats application output
Recommendation
  • Design the output window in a way that the application output is separated from build messages.

[ED-7] Double-click on project template should jump to next wizard step

Finding
  • Participant wants to move to the next step by double-clicking on a project template in the new project wizard.
Severity
  • 1 participant requested this behavior
Recommendation
  • Implement jumping to the next step after double-clicking on project template in the new project wizard (and the same thing in the new file wizard).

[ED-8] Code completion shouldn't be case sensitive

Finding
  • Participants expects code completion to be case insensitive. Writes "sys" and invokes code completion for "System". As our code completion is case sensitive by default it doesn't show "System".
  • Another participant just turned case sensitive completion off in options dialog (old dialog, we didn't have the new one during the study).
Severity
  • 3 participants expected this behavior
Recommendation
  • Consider making the code completion case insensitive. It helps those who expect it and probably doesn't harm those who don't require it.

[ED-9] Code templates are hidden

Finding
  • Participant wants to find a code template for "for", but can't find it in templates manager, keyboard shortcuts, nor in old options dialog.
Severity
  • 1 participant wanted to use code template functionality but couldn't find whether it at all exists.
Recommendation
  • Maybe the new options dialog would solve the problem for this user.

[ED-10] "End" key doesn't work if code completion popup is open

Finding
  • Participant cannot move to the end of line with "End" key while the completion popup is open. His comment: "This is dumb."
Severity
  • 1 participant found this behavior problematic
Recommendation
  • The "End" key should work even if the code completion popup is open.

[ED-11] Try-catch hint should also work on block of code

Finding
  • Participant tries to use the try-catch hint on a block of code. He wants to encapsulate both "for" loops so selects them in the code, but at this point the light bulb disappears. He gets back to the try-catch around one line generated with the hint.
  • Participant: "I would like to select a block of code and click somewhere and the try-catch block would jump in". We have this functionality available in contextual menu, which is not very consistent with editor hints and it's not very well discoverable.
Severity
  • 1 participant couldn't achieve what he wanted and thought that the try-catch hint isn't helpful enough
  • Another participant didn't discover this hint at all, he expected it to work on block of code
Recommendation
  • Design the try-catch hint in so that the user can specify a block of code he wants to encapsulate.
  • Consider providing "block hints". If the user selects a block of code a light bulb would show up offering "block hints". The user could click this light bulb and select an item that's available for this block of code, for example surround with try-catch.

[ED-12] Merge editor hints with error icon in editor gutter

Finding
  • Participant clicks and then right-clicks on error mark icon to get to the editor hints. Expects that editor hints are associated with the error notification. Instead a breakpoint is placed on that line after clicking the icon.
Severity
  • All participants expected this behavior and were confused that clicking the icon put a breakpoint on that line
Recommendation
  • Merge editor hints and error marking in editor gutter.

[ED-13] Keyboard-only access to error messages in source editor

Finding
  • All participants have to use mouse to read the error message indicated with error icon in editor gutter. The message is in the tooltip over the error icon.
Severity
  • All participants were slowed down in completing their tasks
Recommendation
  • Design keyboard access to error messages indicated with red underlines.

[ED-14] Modified status (*) in status line instead of editor tab

Finding
  • Participant wants to see an indication about modified file in the status line ("I prefer '*' in the status line instead of in the tab.")
Severity
  • 1 participant prefers this behavior over the current state
Recommendation
  • Consider showing this information in the status line and remove it from tabs.

[ED-15] Light bulb is hidden if cursor isn't on the same line

Finding
  • Participant doesn't find the try-catch hint because he is on a different line in source code so the light bulb doesn't show up. He has to manually write the try-catch block.
Severity
  • 3 participants didn't find the useful feature and had to write the code manually
  • 3 other participants had problems finding the feature
Recommendation
  • Show editor hints that solve errors always not depending on the position of cursor.

[ED-16] Smart completion of exceptions in try-catch block

Finding
  • Participant writes the catch statement manually and wants the IDE to automatically fill in (or suggest to fill in) exceptions thrown in the try-catch block ("I had to manually find it and I'm a very lazy person.").
Severity
  • 1 participant would find this useful
  • Another participant was irritated that he had to write it manually
Recommendation
  • Design suggestions (maybe through code completion) of exceptions being thrown by the code inside a try-catch block. For example if the user writes "catch(" suggest the exception class. Something like that would be useful if the user already has a try-catch block but modified the code inside and now has to update the caught exceptions.

[ED-17] Code template conflicts with variable names

Finding
  • Participant wants to write "ie" as a variable of "InterruptedException" but the IDE expands this string to "interface".
  • Participant wants to write "br" as a variable of "BufferedReader" but the IDE expands this string to "break".
  • Participant wants to write "ex" as a variable of "IOException" but the IDE expands this string to "extends".
Severity
  • 4 participants had this problem
Recommendation
  • Reduce the number of code templates (replace with keyword completion)
  • Or change the default shortcut to "Shift-Space"

[ED-18] Missing "Add" menu in editor's contextual menu

Finding
  • Participant wants to create a new method using a new method dialog invoked from editor's contextual menu. He finds the Add menu (but only for class, interface, etc.) in node's contextual menu and goes again to check whether this menu is present in editor. It isn't. Then goes to the main menu to search for it. Gives up and writes it manually ("I will stop searching for it. I don't want to be here till tomorrow.").
  • Another participant also wants to add a main method from java file contextual menu, but it's not there (it is on class node)
  • Another participant looked also into the Source menu if it's not there.
Severity
  • 3 participants couldn't find this feature
Recommendation
  • Consider adding the "Add" menu into the editor's contextual menu.
  • Consider adding the "add method" item into java file node contextual menu.

[ED-19] Cannot find shortcut for editor hint popup

Finding
  • Participant wants to invoke editor hint with the keyboard bud can't find the shortcut. He looks into the main menu and hovers over the icon but not long enough to show the tooltip with the shortcut.
Severity
  • 1 participant had this problem
Recommendation
  • Show the shortcut in main menu and consider showing the shortcut inside the editor hint popup

[ED-20] Confusing open project folder

Finding
  • Participant didn't know how to open project. He was looking for project file in Open Project dialog. As he didn't find one, he went to create new freeform project because he saw that the project already has build.xml file.
  • Participant was a bit confused that he didn't see a project file, but then selected the project folder and opened it.
  • Participant doesn't select the project folder, but goes inside the project folder and as he doesn't select any folder there yet, the Open Project Folder button works on project folder. He clicks it and opens the project folder
Severity
  • 1 participant couldn't finish simple action of opening the project and then spent 10 minutes on creating new freeform project.
  • 1 participant was a bit confused due to missing project file.
  • 1 participant didn't use the project chooser properly.
Recommendation
  • Either make it more obvious with icon or something similar that the user needs to select folder, or provide a project file the user should select when opening projects.

[ED-21] Confusing inline error for project folder and location in freeform project wizard

Finding
  • Participant didn't understand the inline error message in new freeform project wizard saying "Project folder already contains NetBeans project data". The participant was still confused why the finish button was disabled. He clicked the help button, but there wasn't any help page associated with this wizard page. The participant then selected "nbproject" folder as the project folder which worked. Then the wizard said Project location is already owned by NetBeans project Anagram Game. The participant "solved" this by selecting src folder for the project location
Severity
  • 1 participant spent a few minutes in the freeform wizard trying to get through the error message he didn't understand.
Recommendation
  • Improve the error messages. Say something like: "Project folder is already NetBeans project <project name> you can open with Open Project dialog". And: "Project location is already owned by NetBeans project <project name> you can open with Open Project dialog". Provide a help page for this wizard page.

[ED-22] Show description of menu items in status line

Finding
  • 1 participant asked for showing description of contextual menu items in status line. He was supposed to run the project: "Some of the IDEs have it. I'm sometimes cautious and would like to see what I'm going to do.".
Severity
  • 1 participant asked for feature already present in other IDEs
Recommendation
  • Show 1 line description of menu items in status line

[ED-23] Home and End keys don't work as expected in Go To Class

Finding
  • Participant comments that Home and End keys don't work in Go To Class dialog. He made a typo and wants to jump to the beginning of text in the text field and it doesn't work. Same for jumping to the end of text.
Severity
  • 1 participant expected this behavior in text field
Recommendation
  • Home and End keys should work in Class Name text field in Go To Class dialog.

[ED-24] Slow error notification in editor

Finding
  • Participant writes the source code and he is informed about error on the line (error underlining) after he moves to another line.
Severity
  • 1 participant is significantly slowed down, because in order to fix the error, he needs to stop writing the next line and move to the previous line to fix the error.
  • 1 participant new there was an error and he wanted to see the error and then use editor hint, end he had to wait quite long.
Recommendation
  • Show error underlining immediately (within 100ms?)

[ED-25] Code template completion should be shown only when it makes sense

Finding
  • Code template completion (including keywords!) is shown in wrong context when it doesn't make sense. For example System.o offers Object, or System.p offers printStackTrace() and private, etc.
  • This wrong template completion is shown above the method completion which complicates access to method items the user wants to use.
Severity
  • Significant problem. All users had problem with it. For the user the IDE looks weak in this area comparing with competition.
Recommendation
  • Offer the code templates completion only when it can be used in current context.

[ED-26] Code completion (javadoc) stays open when it shouldn't be

Finding
  • Participant selected length item from code completion and javadoc window stayed open. Participant couldn't move up/down in the code due to this problem. Participant: "There's no suggestion and I can't move up/down. This is really irritating." The code and cursor position:
private String scramble(String word) {
    StringBuffer buffer = new StringBuffer();
    buffer.append(word.charAt(word.length() - 1));|

                                                  ^ cursor
Severity
  • This is a bug.
Recommendation
  • The code completion and javadoc window should close if not needed.

[ED-27] It takes too long to show the light bulb

Finding
  • Participant never stays long enough with cursor on 1 position to see the light bulb. Currently it shows up after 1000ms.
Severity
  • This is a significant problem. Some participants didn't discover this feature immediately due to this problem!
Recommendation
  • Show the light bulb for fixing error immediately.

[ED-28] Parameter popup shows up when not needed

Finding
  • Participant clicked on an error message in the output window and the cursor was put into the source code and the parameter popup showed up: "What is it telling here? Some kind of parameter?". The position of the cursor:
url.openStream|()
              ^ cursor position
Severity
  • Participant was confused, because he expected hint how to fix the error (unreported exception), not parameter hint.
Recommendation
  • Do not show parameter tip in this case.

[ED-29] Confusing color notification of errors

Finding
  • Participant clicked on error description in output window which marked portion of the line with pink background. He didn't understand what was it for. It's indeed confusing because we have red underlining and pink background which should be used more consistently.
Severity
  • Participant was confused and didn't understand the meaning of using the color.
Recommendation
  • Consider if it makes sense to indicate which is the current error. Or make it more consistent with underlining. Currently it probably makes sense because it better indicates position of error. The error underlining underlines whole line, while this paints pint background only behind error statement.

[ED-30] Hidden current error annotation in editor gutter

Finding
  • Clicking on error description in output window puts an annotation icon into editor gutter. But this annotation was not visible because editor gutter could show only one icon. The cycling annotation icon was visible though.
Severity
  • This is confusing because user doesn't understand what are we trying to tell with the cycling icon.
Recommendation
  • Do not indicate current error with annotation icon.

[ED-31] Warning about not initialized fields and variables

Finding
  • Participant encountered null pointer exception because he didn't initialize ArrayList field. His code called a method on such uninitialized field.
Severity
  • Participant had to run the application in order to get to this error which could be an unnecessary step.
Recommendation
  • Warn the user about unitialized variables in the source code.

[ED-32] Code template for "if"

Finding
  • Participant wanted to use code template for "if" statement.
Severity
  • 2 participant were used to this template from Eclipse.
Recommendation
  • Provide "if" templates.

[ED-33] Keyword completion

Finding
  • Participant wants to use keyword completion when writing private keyword.
Severity
  • Participant is used to keyword completion from Eclipse
Recommendation
  • Provide keyword completion.

[ED-34] Hard to notice refactoring output

Finding
  • Participant overlooks the refactoring output after clicking the Next button in refactoring window. He thinks that something went wrong because the class is not renamed: "Hmm, it looks like the IDE wasn't very successful."
Severity
  • Participants coming from Eclipse are not used to confirming refactoring in output window.
Recommendation
  • Redesign the refactoring flow in a way that the refactoring is performed by default without preview. Also rename the buttons, so that it is clear when the user performs preview and when actual refactoring.

[ED-35] Smart code completion for constructors in new statements

Finding
  • Participant comments that code completion could be smarter when writing constructors in new statements. E.g. StringBuffer buffer = new ... doesn't offer StringBuffer as the first choice.
Severity
  • All participants coming from Eclipse were used to this feature.
Recommendation
  • Design compelling smart code completion for new statements and constructors.

[ED-36] Cannot find "Find Usages"

Finding
  • Participant couldn't find "Find Usages" menu item even though he looked a few times into contextual menu. The problem might be in wrong ordering of menu items.
  • Another participant looked also into the editor toolbar and clicked on "Find Next Occurrence" button.
Severity
  • 2 participants couldn't find very useful functionality and thought it didn't exist in NetBeans
Recommendation
  • There's "Web Service Client Resources" item that isn't a navigation item but is in the same group as "Select in" and "Find Usages". The menu during the test looked like:
    			Go To                        >
    			Web Service Client Resources >
    			Select In                    >
    			Find Usages
    			Show Javadoc
    			-------
    			Refactor                     >
    			
    I suggest to change it to (Navigation item are separated from those modifying the source code):
    			-------
    			Go To                        >
    			Select In                    >
    			Show Javadoc
    			Find Usages
    			-------
    			Web Service Client Resources >
    			Refactor                     >
    		
    Btw, I think "Web Service Client Resources" could be shorten to "Web Service Client".

[ED-37] Code completion doesn't offer method names when declaring method

Finding
  • Participant wrote method usage first and then wanted to write its declaration. Expected that the code completion would offer the method name. It didn't. Participant said: "The name is right here" (it was 2 lines above :-). He had to write it manually. The code and cursor position when he invoked code completion was:
    		static {
    		    readWords();
    		}
    		
    		private static void rea|
    		                       ^ cursor
    		
Severity
  • 1 participant expected this behavior.
Recommendation
  • Provide code completion in this situation offering the method name.

[ED-38] Cannot find what is keyboard shortcuts mapped to

Finding
  • Participant wants to find out why he can't write '{' with Ctrl-Alt-B. Goes to the keyboard shortcuts dialog where he can't find it because it's not ordered alphabetically by shortcuts. In the new options dialog this would be even harder.
Severity
  • 1 participant couldn't solve his problem with the keyboard
Recommendation
  • Design and implement "view by shortcuts" feature in keymap panel in options dialog.

[ED-39] Class.class looks like class keyword

Finding
  • Participant wrote "Class.cl" and invoked code completion and after selecting the "class" item, it put a space after the class character so the user couldn't continue with ".", he had to delete the space. The code before code completion:
    		Class.cl|
    		        ^ cursor
    		
    Code after code completion:
    		Class.class |
    		            ^ cursor
    		
    Participant also commented the blue color of class field which looked like class keyword.
Severity
  • 1 participant was confused from this behavior
Recommendation
  • Do not put space and do not use keyword color for class field.

[ED-40] Hints for correcting typos

Finding
  • Participant doesn't use touch typing thus he has to look at the keyboard when typing. This is very error prone. He made 2 typos: ArrazList and ArrzList in declaration of 2 list variables.
Severity
  • 1 participant would consider it helpful if the IDE corrected typos
Recommendation
  • Offer correcting typos in editor hints.

[ED-41] Find in Projects dialog doesn't look like it searches for class names

Finding
  • Participant wanted to search for a class containing"Word" in "Find in Projects" dialog. He didn't finish it because he thought that "Search for Object Names Containing" means searching for something that "contains" the string, not something that is "named" like that.
Severity
  • 1 participant didn't search for class
Recommendation
  • Before redesigning the whole dialog, it would be good to at least change the wording in Object Name tab. Exact wording TBD.

[ED-42] Generate parentheses for constructors from code completion

Finding
  • Participant comments that he would like code completion to generate parentheses when selecting constructor items. The user wrote: "new InputS", invoked code completion and it generated "new InputStream", he said he would like to see: "new InputStream(|)" with cursor in the middle or at least the left parentheses should be there.
Severity
  • 1 participant requested this behavior
Recommendation
  • Consider providing this behavior.

[ED-43] Add Javadoc to current method

Finding
  • Participant wanted to add Javadoc to method he just wrote. He looked into the Source menu and contextual menu in editor. He couldn't find anything. He expected some feature that would generate basic javadoc for the method.
Severity
  • 1 participant requested this feature
Recommendation
  • Consider designing and implementing this feature

[ED-44] Hard to find javadoc for given method

Finding
  • Participant wanted to look at javadoc of a method he just used in the source code. He wanted to check whether it returns null. He invoked Shift-F1 which opened the javadoc tab, but it just said: "No Documentation Found.". So he tried some other shortcuts: F1 which showed basic help, Ctrl-F1 did nothing. Then he went to the help menu: "It's not clear how to find quick javadoc for given method. Maybe it's not set.". So he tried contextual menu on JDK node and "Show Javadoc" was disabled. After a while: "Maybe I should set it somewhere.". He went to options. "I don't know. I don't like it that I cannot see javadoc." And continues without it.
Severity
  • 1 participant was quite upset that he couldn't access javadoc and he didn't know why and couldn't find where to set it
Recommendation
  • Every place that shows up javadoc should tell the user where to set it if it's not found. In this case, if the javadoc tab explained that to the user he could have found it. Also the "Show Javadoc" menu item should be enabled even if javadoc is not set and open up a dialog that would tell the user how to setup javadoc.

[ED-45] Cannot find "while" code template

Finding
  • Participant type "while" and invoked code completion which said "No Suggestions". He expected it would offer "while" template.
Severity
  • 1 participant couldn't find while template
Recommendation
  • Offer code template in code completion if the user writes the beginning of code template. In this case "while".

[ED-46] Show find occurrences in error stripe

Finding
  • Participant used regular find dialog instead of find usages because he couldn't find "Find Usages". He requested to see the find occurrences in error stripe.
Severity
  • 1 participant requested this functionality.
Recommendation
  • Consider showing find occurrences in error stripe.