Refactoring Usability Study Report

Author: Dusan Pavlica
Version: First draft - 04/23/2004
Study Dates: April 16th and 19th, 2004

 

Table of Contents:

• Introduction
• Participants
• Tasks Summary
• Task Analysis
  • Task 1: Browse the IDE
  • Task 2: Create and run a Hello application
  • Task 3: Rename Hello to HelloCoders
  • Task 4: Try to invoke Undo refactoring action and Move class action
  • Task 5: Change package name in more files
  • Task 6: Move more classes from directories to another one
  • Task 7: Create getter and setter for field badGuess
  • Task 8: Huge amount of renaming in Classfile package
  • Task 9: Put back some changes and find occurrences
  • Taks 10: Change order of parameters by method
    
    
• Conclusions and Recommendations
  • [C-1]  Users didn't understand MDR Browser at all
  • [C-2]  Users expected shortcuts from their IDEs for Rename action
  • [C-3]  Users had problem to find simple Rename in Contextual menu
  • [C-4]  Undo of refactoring action is really hard to find and isn't integrated with Undo of Editor
  • [C-5]  There is no support for automatic refactoring after D&D action
  • [C-6]  Package path should be filled in the Move class dialog
  • [C-7]  Rename action doesn't work in comments
  • [C-8]  There is no way how to create new package directly from Move class dialog
  • [C-9]  There is no intelligent import cleaning after Move class action
  • [C-10] Inconsistency between contextual Refactoring menu and main Refactoring menu when form file is selected
  • [C-11] There is no way how to rename all package all at once, not only in parts
  • [C-12] Participants had problems with scanning of code after start of IDE
  • [C-13] Content of Preview window isn't so clear to users
  • [C-14] There is no support for multiple Move class action but without D&D
  • [C-15] Ability to use renaming of packages with classes instead of move classes to another package
  • [C-16] Users couldn't find Encapsulate Fields or doesn't understand its behavior
  • [C-17] Duplication of action for generating getters and setters
  • [C-18] Encapsulate Fields action doesn't work well with guarded blocks
  • [C-19] Position of getter and setters isn't clear  before generating them in the Preview window
  • [C-20] UI of Encapsulate Fields dialog should be improved a bit
  • [C-21] Undo of refactoring action hasn't bigger history, then only one step
  • [C-22] Where used action is too hidden and it isn't integrated with normal Find in Files action
  • [C-23] Structure in the Preview window is too complicated, should exist ability to switch it into more simple one
  • [C-24] Diff functionality is necessary in Preview window in some cases (for not so known actions)

   

Introduction

The usability study was focused on observing the first hour experience when using refactoring features included in NetBeans IDE. Eight participants were asked to complete 10 tasks using NetBeans IDE Refactoring Alpha build.

Usability study reveal some UI problems which could cause really substantial problems to user. E.g. some of most visible:

• Participants expected automatic refactoring after Drag&Drop actions
• Participants couldn't find Undo for refactoring actions and they used Undo in Editor
• Participants couldn't find Where Used action

Another group of problems aren't so big, but after their fixing refactoring in Netbeans would be easy to use during the first hour of usage. Then usability of refactoring features could be increased too. E.g. some typical problems:

• Participants where confused from Preview output a bit
• Participants need multiple Move Class action often
• Participants doesn't understand name of action Encapsulate Fields

In the study we noted 19 UI problems (till this time) and obstacles directly related to UI of Netbeans Refactoring Alpha build and all are described in detail below.

 

Participants

We could categorize participants according to programming experience as:

• 1 student (P3)
• 2 junior programmers  (P2, P7)
• 4 experienced programmers (P1, P4, P5,  P6, P8)

The full participant characteristics will be found here: Participants.

 

Tasks Summary

Ten tasks were focused on testing following:

• Task 1: Browse the IDE
• Task 2: Create and run a Hello application
• Task 3: Rename Hello to HelloCoders
• Task 4: Try to invoke Undo refactoring action and Move class action
• Task 5: Change package name in more files
• Task 6: Move more classes from directories to another one
• Task 7: Create getter and setter for field badGuess
• Task 8: Huge amount of renaming in Classfile package
• Task 9: Put back some changes and find occurrences
• Taks 10: Change order of parameters by method

The full task's script can be found here: Task Script.
All participants finished all tasks but with problems mentioned below.

 

Task Analysis

Task 1: Browse the IDE

Objectives

• User takes note of Refactoring menu and actions hidden inside by first observation of IDE.

Notes

• All participants had different comments about the IDE based on their experiences with other IDEs.
• P1,P3,P4,P8 found Refactoring main menu
• P2 found refactoring actions in Context menu
• P7 explored Refactoring menu really in detail.
• P5,P6 found MDR Browser and didn't understand it [C-1].

 

Task 2: Create and run a Hello application

Objectives

• Only check if user understands Edit/Compile/Run cycle importantly for next tasks.

Notes

• P2 used Contextual menu for creating new class
• P5 made a mistake in class name and used Rename action from Refactoring Contextual menu automatically.
• P6 create class easily but was confused from sensitive Code Completion in Editor. He prefers insensitive.
• P8 created new java class from contextual menu above package folder

 

Task 3: Rename class Hello to HelloCoders

Objectives

• Check user's method of renaming (manually or by refactoring Rename)
• Check typical place for invocation of Rename action

Notes

• P1,P2 renamed it manually in Editor and Filesystems too
• P3 expected SHIFT F6 shortcut for Rename action
• P3, P4, P5, P6, P7 invoked Rename action from Refactoring contextual menu in Filesystems
• P8 invoked Rename action from Editor

Problems

• Users expected shortcuts from their IDEs for Rename action [C-2]
• Users had problem to find simple Rename in Contextual menu [C-3]

 

Task 4: Try to invoke Undo refactoring action and Move class action

Objectives

• First we would like to know whether user could find Refactoring Undo action.
• Then we need to know how and if user distinguishes between Editor Undo and Refactor Undo.
• Check how user moves one class to another package (manually by Copy+Paste, by D&D or by Move class refactoring action).

Notes

• All participants used Editor undo action (some of them CTRL+Z shortcut)
• Most participants tried to use D&D instead of refactoring Move class action.
• P2 found refactoring Undo action in main Refactoring menu accidentally.
• P1, P3 tried Editor Undo action but without success. Then they used Rename action again instead of Undo action.
• P3 was confused because of empty grey border in Rename dialog (at a place for progress indication)
• P3 criticized usability of Move class dialog (dialog is empty, textfield hasn't focus, tree is collapsed, there is no possibility to create new folder, no code completion)
• P4 moved class by Copy and Paste action and repair package manually in Editor
• P5 realized that renaming doesn't work in comments and criticized it
• P6 tried to find refactoring Undo action in Refactoring main menu and he didn't find it even he had it opened.
• P6 expected creating of new folder in Move class dialog (in package tree).
• P7 was confused from error messages in Editor (simple click doesn't work in left border of Editor).

Problems

• Undo of refactoring action is really hard to find and isn't integrated with Undo of Editor [C-4]
• There is no support for automatic refactoring after D&D action [C-5]
• Package path should be filled in the Move class dialog [C-6]
• Rename action doesn't work in comments [C-7]
• There is no way how to create new package directly from Move class dialog [C-8]

 

Task 5: Change package name in more files

Objectives

• First we would like to know in this bigger project, whether user uses Find Usages and Rename action or makes all changes manually.
• We observe the invocation place for Rename action too.

Notes

• Most participants expected multiple D&D of files and automatic refactoring then.
• Most participants had problems with imports after Move class action in a case they moved classes from different folders.
• All participants had problem with missing Refactoring context menu above form file in the Filesystems window. Some of them found inconsistency among contextual menu and main Refactoring menu
• P1 used Find in Files from main menu and repair everything in Editor
• P2 tried Find in Files, but then invoke Move class action from Context menu. He was vex with to small close button in Editor too
• P3 tried to use combination SHIFT CTRL N for class searching (from Idea). He was confused because of message "Invalid package declaration" even everything was ok. Opening form file instead of source file in Editor was another confusing thing for him. Then he used Find in Files and repaired code manually.
• P4, P5, P6, P8 repaired everything manually and used Find in Files
• P5 tried to find global replace action but without success.
• P6 tried to find package repair action.
• P7 used Move class action instead of package correction. He found Undo refactoring action accidentally. Content of Preview output window wasn't so clear to him.
• P8 was confused from opened form designer instead of source editor and couldn't even find source editor.

Problems

• Participants had problems with scanning of code after start of IDE [C-12]
• Again...There is no support for automatic refactoring after D&D action [C-5]
• There is no intelligent import cleaning after Move class action [C-9]
• Inconsistency between contextual Refactoring menu and main Refactoring menu when form file is selected [C-10].
• There is no way how to rename all package all at once, not only in parts [C-11]
• Content of Preview window isn't so clear to users [C-13]

 

Task 6: Move more classes from directories to another one

Objectives

• We would like to know whether user uses Move class action and in which way in this bigger project with more classes.

Notes

• P1 used D&D action without success.
• P2 tried to rename ui directory to games and expected movement of all classes from ui directory to the games directory and deleting of ui directory then.
• P3 wanted to use mnemonics in Preview output window (for Do Refactoring action). He was disappointed from missing multiple Move class functionality. He lacked Refactoring context menu in the Filesystems window.
• P4 tried to use move package, but then used Move class action from editor
• P5 trusted to Move class action (main menu), so he didn't want to read Preview carefully. He found Undo refactoring action accidentally in main menu. He was confused from opening form editor instead source editor.
• P6 had problems with imports after Move class action and he couldn't find Move class action above form file node.
• P7 knew about problems with Move class action in contextual menu, so he used this action from main menu. He tried to rename folder with classes instead of Move class action.
• P8 tried to rename package in the Editor instead of usage Move class action. It doesn't work, so he moved and repaired all classes manually.

Problems

• Again...There is no support for automatic refactoring after D&D action [C-5]
• Inconsistency between contextual Refactoring menu and main Refactoring menu [C-10].
• There is no support for multiple Move class action but without D&D [C-14]
• There is no intelligent import cleaning after Move class action [C-9]
• Ability to use renaming of packages with classes instead of move classes to another package [C-15]

 

Task 7: Create getter and setter for field badGuess

Objectives

• We test whether user uses (and how exactly) Encapsulate Fields action or writes all code manually.
• We observe if he understand Preview in Output too.

Notes

• P3 tried keyboard combination ALT+INSERT (it imports some constructions in Idea IDE). He didn't find Encapsulate Fields at all. He expected two versions of generating getters and setters (quick without dialog and more detailed with dialog). Result in Preview Output is annoying and too abstract. He thought, that Encapsulate Fields dialog should be improved.
• P4, P7, P8 generated getter and setter by Generate RW property form Tools menu. Then tried the same with Encapsulate fields and it generated it into guarded blocks.
• P5 knew the Encapsulate Field action, but expected something more intuitive related to getters and setters. He was confused from generating into guarded blocks.
• P6 wanted to see suggested position for getters and setters before finishing Encapsulate Fields action by Do refactoring.
• P7 was confused by sensitive Code Completion.
• P8 used Generate R/W property for field action from Tools menu even he saw Refactoring menu in detail. Then I directed him to the Encapsulate Fields. He didn't know, that it's intended for generating getters and setters. Encapsulate Fields dialog was too complex for him.

Problems

• Users couldn't find Encapsulate Fields or doesn't understand its behavior [C-16]
• Duplication of action for generating getters and setters [C-17]
• Encapsulate Fields action doesn't work well with guarded blocks [C-18]
• Position of getter and setters isn't clear before generating them in the Preview window[C-19]
• UI of Encapsulate Fields dialog should be improved a bit [C-20]
• Diff functionality is necessary in Preview window in some cases [C-24]

 

Task 8: Huge amount of renaming in Classfile package

Objectives

• We test whether user uses Rename and Where Used actions in the case of more bigger package then in previous tasks.
• Preview in the Output window contains more information, so we need to know if user understand it easily.

Notes

• P1 didn't like structure view in Preview window. He wanted to switch it to the different simple view without so many levels in a tree.
• P3 didn't find scanning of code after mounting and was confused, because IDE didn't react. He didn't find Undo of refactoring action.
• P4 used Where Used action first and then Rename action. He mentioned great ability of Idea IDE to provide rollbacks directly from the Editor and clear indication of that suggested rollbacks in the left column of the Editor.
• P1, P2, P5, P6, P7, P8 invoked Rename from Refactoring context menu and used it without any problems.
• P7 used preview by another Rename action instead Where Used action. He experimented with Undo action in the Editor and needed longer history by refactoring Undo. He was confused from
• P8 had problem with scanning of code after mounting of package.

Problems

• Participants had problems with scanning of code after start of IDE [C-12]
• Undo of refactoring action is really hard to find and isn't integrated with Undo of Editor [C-4]
• Undo of refactoring action hasn't bigger history, then only one step [C-21]
• Where used action is too hidden and it isn't integrated with normal Find in Files action [C-22]
• Structure in the Preview window is too complicated, should exist ability to switch it into more simple one [C-23]

 

Task 9: Put back some changes and find occurrences

Objectives

• Testing of usages refactoring Undo action or reverse Rename action.
• We test Where Used action too.

Notes

• P1 was confused from Undo&Redo actions in Editor once again. Then he used reverse Renaming instead of refactoring Undo
• P1, P2, P6, P7, P8 used Find in Files action instead of Where Used action
• P2, P3, P5 used refactoring Undo action easily because he knew it from previous tasks
• P4 used refactoring Undo action, but he had some problems during it. He suggested to order refactoring actions by frequency of using
• P3, P4, P5 used Where used action, but didn't expect it in Refactoring menu
• P1, P6, P8 didn't find refactoring Undo action, so he used reverse Rename
• P7 found refactoring Undo action during previous tasks, but he used reverse Rename rather

Problems

• Undo of refactoring action is really hard to find and isn't integrated with Undo of Editor [C-4]
• Undo of refactoring action hasn't bigger history, then only one step [C-21]
• Where used action is too hidden and it isn't integrated with normal Find in Files action [C-22]

 

Taks 10: Change order of parameters by method

Objectives

• Test usage of Change Method Parameters action.

Notes

• P1, P2, P3, P4, P7 invoked Change method Parameters action from Editor and used it easily
• P5, P6, P8 found Change Method Parameters after a while of searching
• P5, P7 mentioned that Preview is confusing for him. He wanted to see diff

Problems

• Diff functionality is necessary in Preview window in some cases (for not so known actions) [C-24]

 

Conclusions and Recommendations

We found following problems during usability study and separated them into more blocks by priority:

Highest priority (6)

• Undo of refactoring action is really hard to find [C-4]
• There is no support for automatic refactoring after D&D action [C-5]
• Inconsistency between contextual Refactoring menu and main Refactoring menu when form file is selected [C-10]
• Participants had problems with scanning of code after start of IDE [C-12]
• Encapsulate Fields action doesn't work well with guarded blocks [C-18]
• Where used action is too hidden and it isn't integrated with normal Find in Files action [C-22]

Higher priority (9)

• Users had problem to find simple Rename in Contextual menu [C-3]
• Rename action doesn't work in comments [C-7]
• There is no intelligent import cleaning after Move class action [C-9]
• Structure in the Preview window is too complicated, should exist ability to switch it into more simple one [C-23]
• There is no support for multiple Move class action but without D&D [C-14]
• Diff functionality is necessary in Preview window in some cases (for not so known actions) [C-24]
• There is no way how to rename all package all at once, not only in parts [C-11]
• Users couldn't find Encapsulate Fields or doesn't understand its behavior [C-16]
• Duplication of action for generating getters and setters [C-17]

Lower priority (9)

• Users didn't understand MDR Browser at all. [C-1]
• Users expected shortcuts from their IDEs for Rename action [C-2]
• Package path should be filled in the Move class dialog [C-6]
• There is no way how to create new package directly from Move class dialog [C-8]
• Content of Preview window isn't so clear to users [C-13]
• Ability to use renaming of packages with classes instead of move classes to another package [C-15]
• Position of getter and setters isn't clear before generating them in the Preview window[C-19]
• UI of Encapsulate Fields dialog should be improved a bit [C-20]
• Undo of refactoring action hasn't bigger history, then only one step [C-21]

We will describe them in detail in the following part of document.

 

[C-4] Undo of refactoring action is really hard to find

Users tried to use Editor undo action (some of them CTRL+Z shortcut) instead of refactoring Undo action, which is too hidden in Refactoring menu.
But when user uses Editor undo action, is impossible to use refactoting Undo action then.

Severity

Because of this problem:

• All (8) participant were confused and they couldn't invoke Undo of refactoring actions
• 6 participants used Undo of steps of Editor in main Toolbar
• 2 participants used shortcut CTRL+Z

Recommendation

Undo action of Editor and Undo action of refactoring features should be merged together and should have one general history of steps.
When user invokes undo of refactoring action from this history, then small informative dialog about bigger Undo changes should be shown.

 

[C-5] There is no support for automatic refactoring after D&D action

Most participants expected multiple D&D of files and automatic refactoring then.

Severity

Because of this problem:

• 7 participant tried to use D&D for Move class action and then realized, that refactoring doesn't work.

Recommendation

When user finishes D&D action with one or more files in Explorer, then refactoring flow should start.
First panel with parameters is different a bit, but rest of refactoring steps are the same.
First panel should contain summary about target package and list of moved files.
Last and important step is summary of founded errors. This summary will be shown in the Output area and user could open them in the Editor and fix them manually.

 

[C-10] Inconsistency between contextual Refactoring menu and main Refactoring menu when form file is selected

This inconsistency causes serious problem  with discoverebility of refactoring actions.

Severity

Because of this problem:

• 5 participant were confused because of missing contextual Refactoring menu above form nodes and were loosing trust in function of refactoring at all.

Recommendation

Simply, Main menu and Contextual menu should be consistent.

 

[C-12] Participants had problems with scanning of code after start of IDE

It blocked them from work and they didn't understand that scanning at all.

Severity

Because of this problem:

• 3 participant tried to use IDE immediatelly after start but they were blocked by non-modal scanning of files.
• 2 participants noticed scanning of code in status line, but didn't understand it.

Recommendation

Scanning of code should be more visible and progress indication is necessary.
This progress indication should be situated in global status line ideally.
If global status line isn't prepared for this case, then modal dialog with short information about scanning should be shown.

 

[C-18] Encapsulate Fields action doesn't work well with guarded blocks

Setters and getters are generated into guarded block section and then they aren't accessible for further adjustments.

Severity

Because of this problem:

• 3 participant found getters and setters in guarded block area and were suprised because of it
• When they would like to adjust code in generated getters and setters, then it's impossible.

Recommendation

Simply...Encapsulate Fields action should generate code out of guarded block area.

 

[C-22] Where used action is too hidden and it isn't integrated with normal Find in Files action

 

[C-1] Users didn't understand MDR Browser at all

Users were confused from that MDR browser.

Severity

Because of this problem:

• 2 participant were confused

Recommendation

Browser should be hidden, not visible for users.

 

[C-2] Users expected shortcuts from their IDEs (for refactoring actions)

Users tried to use the same shortcut which they know from other IDEs.

Severity

• Because of this problem participants (that use shortcut) were slowed down in completing the task.

Recommendation

Try to use the same shortcuts for refactoring actions like in other competitive IDEs (e.g. Eclipse and Idea) as much as is possible.

 

[C-3] Problem to find simple Rename in Contextual menu

Users were confused, because they couldn't find Rename action easily. Most of them found it in Refactoring menu, but some of them only accidentally.

Severity

• Because of this problem participants were slowed down in completing the task.

Recommendation

There is no simple solution, should be discussed more

 

 

Rest of conclusions and recommendations will be filled continuously...