Appendices to Debugging With NetBeans

A Usability Study Summary Report

David-John Burrowes, Jennifer Gove
October 17, 2001

Appendix: Study Design

Tasks

Participants were asked to perform this series of tasks:

1. Given a collection of source files, create a makefile that can construct a program from them.
2. Build the program, fixing any compilation errors encountered
3. Execute the program
4. Locate and fix several bugs in the program.

A full listing of the instructions given the participants is included in an appendix to this document.

Participants

Nine people participated in the study. Each of the participants had been doing software development for at least two years, and had done C/C++ development. The following charts show the expertise levels of the participants as they rated themselves, on a scale of 0 (novice) to 6 (expert).

Note that Participant 4 didn't fill out the experience survey, and thus is not included in these charts. Participant 5 answered the language experience questions, but not the OS experience questions.

Setup

Participants used the August 24, 2001 build of NetBeans with Sun's "closed source" module.

The study machine was an Ultra 60 system running Solaris 8 with 512mb of RAM.

The study was conducted in the Sun Microsystems Usability Labs in Menlo Park, California, USA.

After filling out a brief questionnaire, participants were introduced to the study by Jennifer Gove, and given a set of instructions. They were then given approximately an hour and a half to do as many of the tasks as possible. Each session concluded with a discussion about their experiences.

Appendix: Instructions

These were the instructions given to participants in the study.

Introduction

You have been hired by Sun's new Entertainment 'n Games division. Your group is responsible for getting various existing games to run on Solaris.

Your first assignment is to get a version of Tetris (itetris) to run on Solaris and fix any obvious bugs that you encounter using the new Forte Developer 7 IDE.

Tips

To Run the IDE

• Use the command "fortedev" from the command line

Where to Find the Source Files

• /fd/study/start

How to Play the Game

• Press "Enter" twice to start a game
• Use the arrow keys to move the game pieces as they drop.

Tasks

Access Source Files

To get at the source files from within the IDE, you must "mount" the directory into the Filesystems tab in the Explorer window. This gives you access to all the files in the source directory within that window.

To do this, do the following steps:

1. Choose "Mount Filesystems..." from the File menu
2. In the wizard that appears, choose "Local Directory"
3. Click on the "Next" button
4. From the drop-down list near the top of the new page, choose the root directory ("/")
5. Open the "fd" directory
6. Open the "study" directory
7. Select the "start" directory
8. Click on the "Finish" button
9. Note that "/fd/study/start" appears in the Explorer window
10. Click on the handle to the left of the name to show all the source files

Create Makefile

Using the IDE, create a Makefile (not a project) to build the tetris game from them.

• The source files must be compiled with -DX11
• The game must be linked with X Windows libraries (e.g. -lX11)

Make the Game

Within the IDE, construct the game. Do not worry about "warnings" generated by the compiler (there will be many), but do fix any errors.

Run the Game

Now, run the game.

Debugging

Overview of some of the code

• tetXfigure contains functions related to the movement and positioning of the falling shapes/figures
• tetXmain contains the main() function and other general stuff.
• tetXgame contains some general game-related functions, and responds to keyboard input.
• tetXoper contains functions for changing game levels, for doing some drawing and for creating new games and showing help information
• tetXdraw and tetXboard contains drawing routines

Bugs:

• The game crashes any time the user fills a single row
• Pieces of only one sort are dropped in the game, but there should be seven differently shaped pieces, chosen at random.
• The background is drawn in black rather than as a pattern as it should be.
• If you have time after doing the previous ones, you may want to pursue this one: sometimes when quitting the game, the message "Alarm Clock" or "Signal 14" is printed and a core file is dumped

Appendix: Lists of Issues

This section contains lists of all the NetBeans related issues encountered during the usability test. Some of these are not listed in the issues, above, because at the moment there isn't enough data to draw any conclusions from them. They are included here because they might be useful in future studies.

Building Description Participants Resolution When doing a build, P doesn't see any feedback (such as a cursor change) about whether it was building. 6 Issuezilla 17639/D/P4 P annoyed that the output window doesn't keep showing the most recent output 7 Already fixed

Starting a Session Description Participants Resolution P confused by the choices in the Start New Session alert. The wrong choice is often made. 1,3,5,7,8 Issuezilla 15721/D/P3 Not sure that P understood the consequences of choosing "Do Not Show This Dialog Box Next Time" in the Start New Session alert 8 Issuezilla 17640/D/P3 P chose "Execute" instead of "Start" to debug a program 5 None P doesn't know (and, later, doesn't remember) to select an executable before starting debugging. 6,others? None (Assume because P is used to using projects, but not using here)

Pausing/Stepping/Continuing Description Participants Resolution Step Into and Run to Cursor icons on the toolbar are not distinguishable enough, as P repeatedly clicks on the wrong one. 1,7 Issuezilla 17628/D/P4

During Executing Description Participants Resolution Two PC lines shown in the source file at once 2,4,7 Already fixed Current PC highlight line still shown while the program is executing, which leads P to believe that the program isn't executing. 4, 7 Already fixed P has trouble determining the current debugging state (paused vs. executing) 6,others? Issuezilla 17630/E/P3

Adding/Removing Breakpoints Description Participants Resolution P wanted to add a line breakpoint by clicking in the gutter 1,3,5,6,8 Issuezilla 15133/D/P2 "Toggle breakpoint" often was not a meaningful item name (P7 thought it would jump between breakpoints. P1 thought it would disable a breakpoint) 0,1,3,6,7 Issuezilla 17627/D/P4 Glyph and highlighted line are not aligned intermittently various Already fixed P thinks it would be useful to drag glyphs (especially when accidentally creating a line breakpoint on the wrong line) 6 Issuezilla 17641/F/P4 P looks for a command called "Set Breakpoint" 7 None

Examining Data Description Participants Resolution Watch dialog does not come up with the contents of the current selection in the source editor when it is frontmost. 3,6 Already fixed

Searching Description Participants Resolution In the Search Filesystems dialog, participants often didn't see the position information on the right (sometimes it was obscured, sometimes they just didn't notice it). When they did see it, they generally didn't realize that they could click on the items to go to that line. 1,4,6,8 Issuezilla 17633/D/P4 In the Search Filesystems dialog, there are two radio buttons shown, but neither is slected by default. observer noticed Issuezilla 17642/D/P4 P wasn't sure which filesystems would be searched by the Search Filesystems dialog. 1 Issuezilla 17634/D/P4

Finding Description Participants Resolution P surprised that clicking on Close in the Find dialog canceled the find (because it looked like it had been finding something as they typed). 6,8 Issuezilla 17635/D/P3 P's were not happy that the Find window went away after they clicked the Find button. 7, 8 Issuezilla 17635/D/P3 P doesn't identify that Find has found something. 8 None P's had trouble figuring out how to find the next occurrence of a string without doing a new Find. 7, 8 Issuezilla 17636/E/P3 P assumed would need to use the Find command manually in each file to search through a set of files (didn't recognize that Search Filesystems would do this) 7 None

Examining Source Description Participants Resolution P wants to open two source files at the same time, but can't figure out how 2 None

Managing Windows and Workspaces Description Participants Resolution In MDI mode, users had to rearrange default location of windows in order to get the scroll bar 6,7,8 Already fixed Help window un-minimizes and displays itself when the Add Breakpoints or Finish window is displayed, but no information relevant to the dialog is shown in the help window 2,3,8 Issuezilla 17638/E/P4 P switches to the Editing workspace in order to access files (doesn't realize could open the Explorer in the Debugging workspace) 3 None P does most debugging in the Editing workspace, because must switch to Editing workspace to locate the program to start a new session. 7 None P did most source editing in the GUI Editing workspace because P closed the Explorer in the Editing workspace and later rediscovered it in the GUI Editing workspace. 5 None

Miscellaneous Description Participants Resolution Solaris Copy and Paste keys do not work 1 Issuezilla 17643/D/P3 P found it odd to use a "New" wizard to mount a filesystem 6 None P wanted to filter the list of files shown in the filesystems tab 1 None