Debugging With NetBeans

A Usability Study Summary Report

David-John Burrowes, Jennifer Gove
October 10, 2001

Introduction

In the last week of August, 2001, Sun's Forte Tools organization ran a usability study on its "closed source" module for C/C++ and Fortran debugging. This module is intended to be used "on top of" a standard NetBeans distribution.

While the majority of the usability problems encountered by the participants during the study were in the "closed source" module, several were in parts of NetBeans that the participants used while completing the tasks required by the study. This report discusses just the issues related to NetBeans.

Results

The details of how the study was run are included in the Study Design appendix (on another page). A complete list of issues encountered in the study can be found in the List of Issues appendix (also on another page)

Usability Successes

It is easy, when reading (or writing) a report like this to forget that for every usability problem encountered by a participant, there were many things that simply did the right thing in the right way (so participants didn't comment on them), as well as things that participants liked enough to comment on. Some of the things participants said they liked include:

• A participant commented that he liked the way NetBeans automatically switched workspaces when starting a debugging session.
• A participant that had used an older version of Forte for Java was pleasantly surprised at the performance of this version of NetBeans compared to FFJ.
• Another participant commented that she liked the user interface and thought the whole IDE was "very intuitive".

Usability Problems

This section contains a list of the primary usability problems encountered during the study. A full list of all issues are included in an appendix (on another page) to this document. There were 9 participants in the study, but not all used the same features of the IDE. Consequently, each issue below lists the number of participants that had the trouble, and the total number of participants that used that feature.

Note that these issues are grouped by task (general "debugging", then searching, then finding, and then seeking-help tasks).

Debugging: Participants Wanted to Click To Place Breakpoint

8 of 9 participants (88%)

During the study, participants routinely tried to place a line breakpoint by clicking in the gutter of the source editor.

Recommendation: Allow a click in the gutter to add or remove a breakpoint there. This has been filed as Issuezilla issue 15133/Defect/P3 [fixed].

Debugging: Participants Confused by Start New Session Alert

6 of 7 participants (85%)

Our participants often encountered the "Start New Session" alert. However, none of them seemed to understand what they were being asked, and they all used the "Start New Session" button, rather than the "Finish & Start" which was more appropriate for the task.

Recommendation: Revise the wording of the alert so participants are less inclined to click the "Start New Session" button. This has been filed as Issuezila issue 15721/Defect/P2 [fixed].

Debugging: Watch Dialog Doesn't Show Contents of Selection

2 of 5 participants (40%)

If the user chooses "Add Watch..." from the Debug menu, or uses its corresponding shortcut key, the Add Watch dialog appears, but the entry field is left blank, rather than being initialized with the current selection in the Source Editor (when "Add Watch..." is chosen from the contextual menu in the source editor, it is initialized with the selection).

Recommendation: Minimally, when the Source Editor is frontmost, use its selection to initialize the text area in the "Add Watch" window. Better would be to use the text selection from the frontmost window (whichever it is) to initialize the text area. This was fixed before an issue was filed.

Debugging: Participants Did Not Understand "Toggle Breakpoint"

5 of 8 participants (62%)

Five of the participants in the study did not recognize the "Toggle Breakpoint" command could be used to set a breakpoint. Instead, these participants ignored it (or considered it and then ignored it) in both the "Debug" menu and the "Debug (full)" toolbar, and used the "Add Breakpoint..." command instead.

When asked about why they had ignored this command, one participant said that the command sounded like it would transfer them to the next breakpoint, while another said that it sounded like it would be used to disable an existing breakpoint. Another participant clearly thought "Toggle Breakpoint" could only be used on breakpoints that already existed (consistently using "Add Breakpoint..." to add a breakpoint, and "Toggle Breakpoint" to remove one).

As many NetBeans users will agree, this command seems perfectly understandable once you understand what it means, and it may be less of an issue once we allow breakpoints to be set by clicking in the gutter of the text editor. And, in its defense, both Visual Basic and JBuilder use a "Toggle Breakpoint" command. However, it is undeniable that this is a significant hindrance for new users and previous usability studies found this to be a problem as well (see a message from a thread on nbui).

Recommendation: Rename this item to "Insert/Remove Breakpoint", or if possible make it context sensitive so it says "Insert Breakpoint" and "Remove Breakpoint" depending on whether there is already a breakpoint on the line. These names have the benefit of sounding like they will create a breakpoint, but still seem distinct from the "Add Breakpoint..." command. This has been filed as Issuezila issue 17627/Defect/P4 [submitted].

Debugging: Participants Had Trouble Distinguishing the Step Into and Run to Cursor Toolbar Buttons

2 of 7 participants (28%)

While debugging, participants clicked "Run to Cursor" in the toolbar when they meant "Step Into", and vice versa, on the "Debug (full)" toolbar.

Recommendation The icons should be made more distinct, or perhaps not placed next to each other. This has been filed as Issuezila issue 17628/Defect/P4 [submitted].

Debugging: Participants Had Trouble Determining When the Debugee Was Executing or Paused

? participants

We believe that participants had trouble determining what state the debugee and debugger were in at times. Most of our data comes from problems in our own debugging interface. However, we believe that the problem will still be manifested in the NetBeans debugger interface and so are worth mentioning here.

The only ways for the user to reliably identify whether the debugee is in a Paused/Stopped or in an Executing state are to:

• try to interact with the debugee and see if it responds,
• look at the debug toolbar and menu and deduce the debugee's state based on which actions are enabled/disabled.
• display the Sessions component and look at the state in parenthesis there.

In practice, users do not always determine application state from the enabled/disabled state of controls in the interface, and many won't show the sessions component. However, it is also true that when focused on their task, the user will be keeping track of the overall state of the debugee in their mind, so will not need additional feedback. The trouble arises when the user is interrupted or distracted from their task and need to reorient themselves.

Recommendation: We do not claim to have a complete answer for how to solve this problem. However, we did notice that both the NetBeans Debugger Window (and our own debugging components) didn't change their appearance when the debugee switched from paused to executing. For instance, the snapshot shown above is from a program in an executing state which is identical to its paused appearance. We believe that some kind of visual change here (e.g. drawing all text and icons in a "dimmed" color, with appropriate accessible descriptions) may help make the overall debugging state clearer. This has been filed as Issuezila issue 17630/Enhancement/P3 [submitted].

Search Filesystems: Participants Did Not See or Use the Position Information on the Right

4 of 4 participants (100%)

The "Search Filesystems" result window caused the study participants some frustration. Because of the width of the window, two participants never noticed the "Position" information on the right side (often this was mostly obscured by a source editor window. The other two participants did not realize that they could double click on an individual position entry to go to that occurrence. Instead, all participants double-clicked on the file names in the left panel and then tried to locate the relevant text within the source window by hand, with the Find command or by looking up the line number listed in the Position list.

Recommendation: It might be good to intermix the information in the left and right panels in some manner. For example, one might be able to expand each file with a single click to show the individual "positions" within it. This would have the effect of making the window more compact, and would make it easier for users to discover the list of positions. By putting them in the same panel, it may encourage users to click on the positions more. This has been filed as Issuezila issue 17633/P4 [submitted].

Search Filesystems: Participant Not Certain Which Filesystems Would be Searched

1 of 4 participants (25%)

One participant commented that the "Search Filesystems" dialog did not indicate to them what would be searched.

Recommendation: A note at the top of this dialog which informed the user that this would search all filesystems shown in the Filesystems tab might be useful. This has been filed as Issuezila issue 17634/P4 [submitted].

Find: Participants Surprised by Behavior of Find Dialog

3 of 3 participants (100%)

None of the participants that used the "Find..." command got behavior that they expected. The problems they encountered included:

• after typing some letters in the "Find" dialog, the Source Editor gave feedback indicating that it had found something. Satisfied that the task was completed, the participants clicked on "Close", at which point the editor scrolled back to the starting point, and discarded any indication of what it had previously found.
• after clicking the "Find" button in the "Find" dialog, participants were annoyed to have the window disappear (particularly because they could not figure out how to find the next occurrence. See the issue below).

That is, this dialog did not behave as any of the users expected.

Recommendation: This dialog could be redesigned to give more expected feedback. For example, it might give intermediate feedback (as it is finding what the user is typing) that looks less like the selected results of the find (e.g. it might merely draw an outline around the text it has found). The "Close" button might also be renamed "Cancel" to give more indication that clicking it will discard the partially found result. This has been filed as Issuezila issue 17635/Defect/P3 [submitted].

Find: Participants Could Not Figure Out How To Find Next Occurrence

2 of 3 participants (66%)

Two of the three participants that used the "Find..." command wanted to find more than the first occurrence of their string in the current document. However, they could not figure out how to do this. Of course, F3 is bound to a "find next" kind of action in the editor, but participants did not think to try F3. Instead, after finding no command in the menu, they used the "Find..." command repeatedly.

Recommendation: Add a "Find Next" menu item in the "Edit" menu with shortcut F3. This has been filed as Issuezila issue 17636/Enhancement/P3 [submitted].

Help: Help Window Doesn't Show Useful Contents

4 of 5 participants (90%)

All of the participants that opened the help window and then left it open had the window reappear and rearranged itself whenever a dialog box was opened. Clearly, the intent was to make help visible to the user while they were using the dialog. This is a good idea, however in no instance during the study did the help window automatically show any information about the dialog that was just opened.

Recommendation: Any time a dialog is opened and the help window is positioned onscreen alongside it, automatically display information about the dialog in the help window. This has been filed as Issuezila issue 17638/Enhancement/P4 [submitted].