Author: Jan Rojcek, Jakub Franc
Study Facilitators: Jakub Franc, Jan Rojcek, Stepan Doubrava
Study Dates: 6 - 7 June 2007
Table of Contents:
Primary goal of editor usability study was to verify improvements in editor features and test the new fonts and color scheme targeted for NetBeans 6.0 release. We observed how 8 participants used code completion, hints and surround with features while they were working on basic coding tasks. As we did the same editor study with the same tasks 2 years ago (NetBeans 5.0) we had a chance to do a simple comparison of the results.
Overall the results were very positive this time. Participants in most cases discovered and used editor features effectively. They could focus on solving the coding tasks instead of figuring out how to use the features like code completion, hints and code templates. It simply worked as they expected it to work. A few participants mentioned that "code completion is smarter than Eclipse". They positively commented new fonts and colors and improved icons. A participant said that the whole NetBeans "looks nicer than previous versions".
Comparing with the previous editor study the best indication of improvements is the number of tasks participants finished during the study. While 2 years ago with NetBeans 5.0 it was on average 5 tasks, this time almost all participant finished all 7 tasks.
We also recorded a few areas for improvements. Unlike in the previous study, we didn't notice occurring problems happening to every participant. This time all findings were mostly subtle issues.
|Participant||Years of Java programming experience||Applications primary developed||Current IDE||Job status|
|P1||7||Web, J2EE||NetBeans 5.5||professional|
|P3||3||Console apps, Web, J2EE||IntelliJ IDEA||student|
|P4||5||J2ME, Swing||NetBeans, Eclipse||professional|
|P5||3||Console apps, Database, J2EE||Eclipse||professional|
|P7||5||Swing, J2EE||NetBeans 5.5||professional|
|P8||2||Web, J2EE||IntelliJ IDEA||professional|
General satisfaction with NB Editor
All participants were generally satisfied with the NB Editor. Users of other IDEs find NetBeans editor as "comparable" to their IDE's editor. P8 (IDEA user) misses "variety of tiny useful features" offered by IDEA editor.
All participants bless code completion. They find it "very helpful", "workflow accelerating"and "smart". Although couple of them criticize it for low promptness.
P1 would appreciate immediate Code Completion, otherwise he is confused and not sure that Code Completion appears at all.
P2 finds NetBeans Code Completion "as smart as Eclipse Code Completion".
P5 finds it even smarter than in Eclipse - because of its very smart parameters completion. P5 particularly likes imports completion and positioning cursor behind the completed import.
P7 "loves" completion of variables. P7 (advanced NB user) is slightly confused, because Code Completion works "a bit differently" than he is used to.
Colors used in editor were generally evaluated as "pleasant", "comfortable" and "nice".
Couple NetBeans users did not notice the change in colors at all. Purpose of grey wavy line was not clear for most participants. P8 was even slightly confused.They generally assumed it indicates and suggests "some improvements", not any serious error. Moreover participants criticize that they do not have chance to find out what is its purpose (some mention they miss an icon that they would click on for more information)
P1 mentions that some icons are not self-explanatory enough. Particularly he dislikes the compile icon with spanner depiction.
P3 likes Eclipse Start Manager that allows him to run his application in multiple modes differing in parameters.
P8 likes the IDEA feature that allows him to add directly key bindings to particular actions.
1 Double-click on project template should move the wizard to the next step
A participant double-clicked on a project template in New Project Wizard and expected it would move to the next step. It didn't move.
Double-click should move to the next step.
2 Classpath scanning takes too long
A participant commented that scanning of classpath takes too long. He mentioned that in Eclipse it's faster.
Compare our classpath scanning to Eclipse. Maybe it's just a perception of slow progress because when the classpath is scanned the very first time it blocks opening of the first project, so it looks like the whole IDE is stuck for a while.
3 "new" completion doesn't offer types that aren't imported
A Participant complains about completion not offering "WordLibrary" constructor when it is clear he wants to use it. The problem is that the WordLibrary isn't imported when the user types:
WordLibrary w = new |
Maybe if WordLibrary is not imported, the completion should import it automatically. If it is ambiguous, it should ask the user what should be imported. Not sure now how the UI for it should look like.
Or if WordLibrary isn't imported, it should automatically fallback to "All Items" completion and preselect "WordLibrary" in completion list.
The former approach is probably better.
4 Coloring of live templatesA participant complains about live template coloring (for). There's a very subtle indication which item is selected (slightly different grade of blue) and no visual indication there's more active items the use can change. Recommendation: Consider using rectangle indication for active items and better distinguish the selected item.
5 Folding of long String array definitionA participant complains he cannot fold long array definition that was present in source files we used during the study. Recommendation: Consider folding also for long array definitions.
6 Cannot create ME application out of the box
Participant was interested in looking at NetBeans ME support, but he couldn't create a Mobile Application project because the wizard said there was no compatible platform installed in the IDE.
I think this is a serious OOBE problem because it's a huge barrier for those who want to learn ME programming using NetBeans IDE.Recommendation: As a minimum, provide clear instructions where to get a J2ME platform from (as a link to download page or similar). Ideally, bundle a J2ME platform with the IDE.
7 Tomcat samples don't work if Tomcat isn't installed
A participant wants to create a sample tomcat project, but he cannot proceed because there isn't tomcat installed.Recommendation: Probably the sample tomcat projects should only be available if tomcat is installed.
8 Cannot "Surround with" from contextual menu
A participant wanted to use "surround with" from contextual menu, but it wasn't available.
Consider adding the surround with action into contextual menu.
9 Basic "for" template
A participant requested a code template that generates a basic "for" loop. Something like:
for (int i = 0; i < 10; i++)Recommendation: Provide this template.
10 Completion should replace existing code
A participant commented that completion should replace existing code instead for inserting code. For example:
if the user invokes completion at this position, and selects
MyDialog from completion window, it should replace the existing code so that it looks like this:
But currently the end result is the following:
Consider the new behavior. There might be use cases when replacing is not the right thing to do.
11 No default constructor in Main class template
A participant comments that the public constructor in the default Main class template should not be there.
Consider removing the default constructor from the Main class template.
12 Hard to find Reformat Code in contextual menu
A participant looked for reformat code action in contextual menu and it took him a while until he found it. He actually closed and reopened menu a few times before he found it.
The contextual menu is poorly organized and there's too many items.
Fix the menu naming and ordering according to the latest permanent UI spec. Another idea not yet used in the spec is to rename "Reformat Code" to simple "Format", as it would not clash with "Refactor" menu item and that's how it is called in other IDEs.
13 Manual completion is not used
A few participants almost never invoke completion manually. They use it only when it pops up automatically after typing ".".
This is an interesting finding as there's places where completion is very handy, like:
WorldLibrary lib = new |
We should consider showing completion automatically after "new" and maybe also on other places. To make it pop up automatically every time we would need to make it very fast and also make the completion window much smaller. The javadoc window can be a problem too as it overlaps a big portion of editor which can be quite obtrusive.
14 Cannot get to Button listener from GUI builder
A participant wanted to get to a button's listener so he double-clicked the button in GUI builder which initiated renaming of the button. He expected it would switch into the source code and show the listener method. He tried other ways how to get to the listener (contextual menu, property sheet) but he wasn't successful.
There should be an easy way how to get to the source code. I think that double-click is a standard way of doing it and "slow" double-click is used for renaming components. That's how it works in all explorers AFAIK: double-click opens the item; "slow" double click renames it.
15 Cannot find "Go to Source" in contextual menu
A participant cannot find "Go to Source" menu item in contextual menu. He tries F3 (Eclipse shortcut), which doesn't work. Finally he uses Ctrl-click.
Maybe the reason why participant didn't find the action was poorly ordered and too long contextual menu.
16 Mouse is a way to get to error description
All participants used the mouse to get to error description indicated by error underlines. It's a huge flow stopper.
Advertise somewhere in the error popup or suggestion popup or in tooltip that the user can use "Alt-Enter" shortcut to get to error description.
Show the one line error description in status line.
17 Hard to find how to set run arguments
A participant cannot find how to set a run argument named "-console". He uses Help and searches for "argument" which points to too many results but the right one doesn't seem to be at the top. Then he searches for "argument run" which interestingly shows even more results, but the right one is at the top.
I'm not sure how to improve discoverability of run arguments. Our approach is simply different from other IDEs.
Just a side note: The help search should *reduce* the number of results if the user types more search words.
18 Run Profiles are hard to find
A participant commented he couldn't find Run Profiles in NetBeans. He said he likes profiles a lot in Eclipse.
We use a different approach to run profiles than Eclipse. Maybe what might help a bit is to put the word "configuration" into the combo box in main toolbar. Currently it says just "default" which doesn't say anything about the purpose of the combo box.
It probably would not help all Eclipse users, but something to consider to improve discoverability of configurations.
19 No tooltip over waved underlines
A participant didn't know how to get to error description indicated by red underlining. He hovered mouse over the underline and expected tooltip with description.
Other participants were not sure what gray underlining indicated. If they hovered mouse above the underline it didn't explain it.
Provide error descriptions and descriptions of gray underlines in tooltips.
20 Template options accessible from editor
A participant says he likes "Alt-Enter" and "Alt-Insert" a lot. He just commented that it would be nice if he could access template options directly from editor (probably as a "Change Template..." item in hint popup?).
Consider providing such access.