Debugger Usability Study Report

Author: Jan Rojcek
Study Facilitators: Jan Rojcek, Josef Holy
Study Dates: 6 - 7 September 2006

Table of Contents:

Executive Summary

In Debugger usability study we tested easy of use of newly designed expression stepping feature and overall experience when using NetBeans debugger. We asked 8 participants to solve 4 simple debugging tasks (pdf).

The expression stepping feature was positively commented by all participants. They found the feature "very nice", "helpful", "interesting". In most cases participants used the feature unassisted which was also very positive. As this feature is unique among Java IDEs, participants (except one) could not find the feature on their own and facilitator had to explain that such feature is present in the IDE. The most visible usability problems related to expression stepping were around visibility of sub-expression node (findings DB-10, DB-13, DB-22) and around coloring of stepping annotation in editor (DB-23). Participants also requested a shortcut associated with the expression stepping action (DB-6).

The testing of overall experience uncovered several usability problems related to basic debugging scenarios. Even though participants solved all tasks unassisted, they encountered problems that disturbed them and slowed them down. The most visible problems were the following:

  • Participants accidentally ran multiple debugging sessions (DB-1).
  • Selected debugger tab wasn't persisted between debugging sessions (DB-2).
  • Participants didn't use the main debugging toolbar effectively (DB-4).
  • Participants had various problems with adding a line breakpoint (DB-9, DB-11, DB-14).
  • Debugger console covered the program output which contained important runtime exceptions (DB-19).
  • And again, participants (Eclipse users) had problems with opening projects (DB-24).

See all findings and recommendations.

Participants

Table: Experience of participants

Participant Developing applications Years of Java prog. (all prog.) Current IDE Frequency in using debugger Uses debugger when...
P1 Non-gui, Web, Java EE, DB 4 (6) NB 5.5, Eclipse 3.2 Once a week bug is hard to fix
P2 Web, Java EE, DB 2 (2) NB Once a week bug is hard to fix
P3 Rich client, Web, DB 5 (6) Eclipse, JBuilder Every day bug is hard to fix
P4 Web 5 (10) NB 4.0 Once a week bug is hard to fix
P5 Web, DB, Java ME 1 (1) Eclipse Every day always
P6 Rich client, DB 3 (6) Eclipse 3.0 Every day always
P7 Web, Java EE, DB 4 (6) NetBeans 5.0 Once a month bug is hard to fix
P8 Web, Java EE, DB 3 (4) Eclipse 3.2 Every day always

Findings and Recommendations

[DB-1] Running multiple debugging sessions accidentally

Finding
  • Several participants ran multiple debugging session accidentally. They either hadn't noticed that the current session was still running or they expected that the debug project action would restart the current session. They ended up with multiple tabs of program output and debugger console in Output window which was very confusing.
Severity
  • 3 participants encountered this problem
Recommendation
  • Should we ask the user what to do if there's multiple debugging sessions running: restart the existing session or start a new one?
  • Or maybe introduce an explicit action that would start a new session. The "Run Main Project" would always do the restart.

[DB-2] Debugger doesn't remember selected tab

Finding
  • Selected debugger tab wasn't persisted between debugging sessions. For example participant selected the Watches tab and restarted debugging session which automatically selected the Local Variables tab. Several participants commented this "buggy" behavior.
Severity
  • This is a bug. Almost all participants were affected.
Recommendation
  • Remember selected tab between debugging sessions (and also IDE sessions).

[DB-3] Click and evaluate expression in Watches/Variable window

Finding
  • Two participants searched for easier way of evaluating expressions. They would like to click into Watches or Variables windows probably into a text field of some sort where they can write expression directly and get results.
Severity
  • Two participants wanted such behavior, other participants would probably also benefit from it.
Recommendation
  • Try to put evaluation of expressions into the Watches or Variable windows.
  • Or maybe just make it easier to add a new Watch by clicking into a text field in Watches window

[DB-4] Main debugger toolbar is ignored

Finding
  • Two participants completely ignored debugger toolbar during the study and used solely the debug menu or shortcuts for all actions including stepping.
  • One participant didn't use the toolbar during task #2, but he used it for the next tasks.
Severity
  • This finding is an indication we might consider changing the debugger toolbar behavior. But we need to make sure that the new behavior is also tested.
Recommendation
  • Consider other ways of showing the debug toolbar.

[DB-5] Error line annotation too similar with breakpoint

Finding
  • Clicking on a assertion error hyperlink in Output window changes background color of corresponding line in editor to red/pink color. This color was very similar to breakpoint color which was a bit confusing.
Severity
  • Several participants encountered this problem.
Recommendation
  • Consider whether it is necessary to annotate editor lines after clicking on error hyperlink. Or change the annotation so that it's not so similar to breakpoint.

[DB-6] No shortcut for Expression Stepping

Finding
  • Several participants commented there's no shortcut for expression stepping.
Severity
  • This seems obvious.
Recommendation
  • Assign a shortcut to Expression Stepping action.

[DB-7] User-defined "toString" watches for certain types

Finding
  • One participant requested user-defined watches, where he can define display of watched object in debugger. For example he can specify that JTable objects would display number of rows in watches window.
Severity
  • One participant explicitly requested this feature. This would be very useful when debugging complex objects.
Recommendation
  • Try to design and implement such feature.

[DB-8] Conditional breakpoint for "number of iterations"

Finding
  • One participant requested separate option in conditional breakpoint for number of iterations so that he doesn't have to write a conditional expression.
Severity
  • One participant requested this feature that would be very useful also when there's no iterator to use in conditional expression.
Recommendation
  • Design and implement such feature.

[DB-9] Two clicks needed to add a breakpoint

Finding
  • If the editor tab is not active, it takes 2 clicks to add a breakpoint. The first click activates the editor, the second click adds the breakpoint.
Severity
  • Several participants encountered this behavior. This is a bug.
Recommendation
  • One click should add a breakpoint in this scenario.

[DB-10] Sub-expression node is hidden after the first step

Finding
  • Sub-expression node wasn't visible right after the first expression step. Participants expected "something" to happen after the first expression step so that they get a hint what the feature is good for and how it should be used.
Severity
  • Majority of participants encountered this problem.
Recommendation
  • Make the sub-expression node visible right after the first step.
  • Maybe it should have a subnode saying something like "Beginning of expression - no results". The subnode should be also visible.

[DB-11] Line breakpoint is not in the New Breakpoint dialog

Finding
  • One participant wanted to define conditional breakpoint the New Breakpoint dialog. He didn't know how he was supposed to add a line breakpoint first and then modify it's properties. He went directly to "New Breakpoint" dialog and expected he can setup everything there.
  • Another participant also used the New Breakpoint dialog. He added "Method" breakpoint.
Severity
  • Two participants expected such behavior. For completeness, it seems correct to show all breakpoint types in the New Breakpoint dialogs even though the line breakpoint would most likely by used with toggle action.
Recommendation
  • Add "Line" breakpoints into the New Breakpoint dialog.

[DB-12] Debug and Run icons in toolbar are too similar

Finding
  • One participants clicked Run toolbar icon instead of Debug icon. It happened a few times. Participant commented: "This happens to me also at work. Make the icons more distinct!"
Severity
  • The icons are indeed quite similar which might cause problems.
Recommendation
  • Consider designing icons that are more distinct.

[DB-13] Sub-expression node is not visible in watches

Finding
  • One participant had the Watches window selected when he invoked the first expression step. Therefore he couldn't see the node even if it was displayed in Local Variables, which it wasn't ;-). Participant then clicked into the Local Variables tab.
Severity
  • This might be a potential problem with discoverability of how the expression stepping works.
Recommendation
  • Consider showing the sub-expression node also in Watches window (probably not a good idea).
  • Or show the expression result tooltips automatically, which would point users attention away from sub-expression node, but maybe that's okay.

[DB-14] Two breakpoints on the same line

Finding
  • One participant double-clicked into the editor gutter which put 2 line breakpoints on the same line.
Severity
  • This is probably a bug. Note that Eclipse users are used to adding breakpoints by double-click.
Recommendation
  • Double-click should only put one line breakpoint on the same line.

[DB-15] Unclear how to customizer breakpoint

Finding
  • One participant looked at the contextual menu but overlooked the "Customize" item.
  • Another participant didn't invoke the contextual menu at all.
Severity
  • Two participants didn't know how to customize breakpoint.
Recommendation
  • Consider showing the breakpoint window by default. This might make some users go there instead of searching for it in contextual and/or main menu.
  • Consider adding "Edit Breakpoint..." item into the main menu.
  • The problem is that the editor contextual menu contains unnecessary submenus. Probably better would be to change the contextual menu and "Customize" item to something like this:
Toggle Bookmark
------------------
Toggle Breakpoint
Disable Breakpoint
Edit Breakpoint...
------------------
Show Line Numbers
Show Toolbar
	

[DB-16] Hard to find list of breakpoints

Finding
  • One participant wanted to use a list of all breakpoints to check how many breakpoints he had and delete those he didn't need. He couldn't find the breakpoints window in window menu. Probably also due to the fact that Window menu was completely rearranged in Retouche build we used in the study.
Severity
  • One participant couldn't find the list of breakpoints.
Recommendation
  • Consider showing the breakpoints window by default next to the call stack window.

[DB-17] Hard to find how to modify watch expression

Finding
  • Two participants used the contextual menu to modify Watch expression, but it took them quite while to click the "Properties" menu item. One of them said "I would expect 'Edit...' item".
Severity
  • Two out of three participants who wanted to modify watches encountered this problem.
Recommendation
  • Consider using "Edit..." item instead of "Properties". Take a look at the other debugger windows (Variables, Call Stack, Breakpoints, etc.) to keep consistency between them if it makes sense.

[DB-18] Code completion doesn't work in New Watch dialog

Finding
  • Code completion didn't work in New Watch dialog.
Severity
  • This is probably a bug.
Recommendation
  • Provide code completion in watches window. Use the context of current line containing cursor in editor.

[DB-19] Console hides the program output

Finding
  • Two participants hadn't noticed a runtime exception thrown into the program output tab because the debugger console was in front.
Severity
  • This is quite a big problem given that the console window wasn't useful at all during debugging.
Recommendation
  • Consider making the program output tab selected by default.
  • Or put the program output tab to front when a new exception is thrown.
  • Or consider not showing the debugger console by default and make it easily accessible for those who need it.

[DB-20] Cannot find "Find Usages" in editor contextual menu

Finding
  • One participant couldn't find "Find Usages" item even though "Find Usages" was the first item in editor's contextual menu. He said "I would like to find references" while looking at the contextual menu.
Severity
  • One of two participants who used find usages had this problem.
Recommendation
  • Consider renaming "Find Usages" to "Find References".

[DB-21] Move breakpoint with drag and drop

Finding
  • One participant put a breakpoint on a wrong line. Then he wanted to drag and drop it onto the next line.
Severity
  • This is a feature request.
Recommendation
  • Design and implement DnD of breakpoints.

[DB-22] Sub-expression node is easy to overlook

Finding
  • One participant completely overlooked the sub-expression node even though it was visible in the Variables window and participant work with the local variables during solving the task.
Severity
  • Several participants had problems with locating the sub-expression node. One participant didn't find it at all.
Recommendation
  • Expand the sub-expression node automatically and show all evaluated expression values.

[DB-23] Coloring for expression stepping doesn't work properly

Finding
  • A few bugs related to expression stepping coloring showed up during the study when debugging multi-line expression:
  • The green underlines indicating computed expressions were hidden when the program counter moved to the second line in the expression.
  • If there was a breakpoint on the first expression line and the program counter moved to the second line then the first line was displayed with breakpoint background color.
Severity
  • The coloring needs to be improved.
Recommendation
  • Make sure the green underlines are always visible while evaluating the same expression. It should not matter whether it is on multiple lines.
  • The breakpoint color should be shown after leaving the whole expression, not after leaving one line. In any case the green underlines should be always visible.

[DB-24] Unclear how to open a project

Finding
  • One participant didn't know how to open a project. He expected a project file, when he didn't find it, he opened the main class with Open File dialog.
  • Several participants (Eclipse users) had problems with using the Open Project dialog. Most of them double-clicked the project folder.
Severity
  • Almost all participants who were not NetBeans users had problems with using the Open Project dialog.
Recommendation
  • We're already working on fixing it.

[DB-25] Hourglass cursor above welcome screen looks like the user needs to wait

Finding
  • One participant thought that the IDE was not yet ready after the startup because it showed the hourglass cursor above the welcome screen. The participant had to wait until the welcome screen loads its content even thought it was not his intention.
Severity
  • This looks like improperly used hourglass cursor.
Recommendation
  • Do not show hourglass cursor above the welcome screen.

Project Features

About this Project

ui was started in November 2009, is owned by Jiří Kovalský, and has 37 members.
By use of this website, you agree to the NetBeans Policies and Terms of Use (revision 20140418.2d69abc). © 2013, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo
 
 
Close
loading
Please Confirm
Close