HOME > Community > Projects > ui

Usability Study Report: NetBeans Enterprise Pack 5.5

Author: Jiri Kopsa, Jakub Franc
Study Dates: 8 and 9 August 2006

    Executive Summary

    Participants

    Findings

       XML Schema Editor
           General
           Design View
           Inherited Children
           Tree and Columns View
           Property Windows and Customizers
           All Views

       WSDL Editor

       BPEL
          BPEL Module and Composite Application Project Types
          BPEL Editor
          BPEL Mapper

       Interesting Quotes

Executive Summary

XML Schema Editor

The usability study has revealed many usability issues related to visual appearance, interaction and workflow. Fixes for these are usually quite straightforward, these have been walked through with the team lead and filled as bugs or RFEs. Besides these, there is a couple of significant issues related to the fact that the knowledge of users about XML Schema is quite shallow. This is signified in the findings related to complex types and elements. We are realizing that the difference between these two contructs is unknown to the users, the tools does not help them to understand. It seems that two constructs are somewhat redundant and that users intuition would accept better just one. That would be parallel to data modelling in OOP (class diagrams) or data modelling practice (ER). It seems that if the tool provided a high level view of the XML Schema domain and made the difference between complex types and elements transparent (less significant), the entry barrier would be nicely lowered.

Of course, we may not go that far. There are gaps in the knowledge of XML Schema domain also on other places - in content models and other XML Schema constructs. These are signified in the findings related to Customizers and Property sheets. In these places, the XSD is also exposed in its rawness. And that's very unpleasant to the users; especially those who would be happy to adopt the ease of use of Design View. So perhaps, something can be done in this area as well. Perhaps, the user can be shielded from some of the XSD wiring and can be provided with a higher level UI that would allowed the user to fulfill the most common use cases more easily - and intuitively; with a general notion of data modelling and without a prior knowledge of XSD specialites.

We may also argue about who is the user of the product, as this has not been defined fully. We may think that the user of the XSD Editor knows the XSD domain and if that had been true, the issues mentioned would have not apply. However, the signals that we were getting so far (customer visits, screening users for the usability studies) show that our average user does not know XSD domain well and that's why the findings from this usability study most probably do apply.

In the end these findings and thoughts cannot stand alone, but need to be put into context together with cost of design and development of a higher level UI, cost of gaining the knowledge about XML Schema and also significance of possible competitive advantage and opportunity lost.

WSDL Editor

The findings in WSDL editor are also (as well as in XSD Editor) related to the missing knowledge of the WSDL domain. Also here we may ask the question, who are the users of the product and what level of knowledge can we expect. And in parallel to XSD Editor, here are also smaller areas for improvement that would ease the adoption of the WSDL Editor for less knowledgable users and perhaps eased the use for a more advanced user.

So we present small (minimal) recommendations that can be done for NetBeans 5.5, and also express concerns to be considered for the future design.

BPEL Project Templates, Editor and Mapper

The most fundamental findings in BPEL domain are related to imports of WSDL files, partner links and partner link types. Besides that we have encountered smaller issues related to visual apperance and smaller or bigger issues related to interaction. Some of them have been already fixed or are being fixed.

How To Read This Document

In the further sections, we present the findings ordered by priorities similar to those used in NetBeans issuezilla. P1 is usually a serious issue, a blocker, P2 means significant slow down, P3 is an inconvenience and P4 is just a small issue with minimal impact. Besides the priorities, we also suggest action to be made on the finding. That may be one of the following:

To be implemented - Issue is clear, it only needs to get fixed.
To be desined - Requires some design work.
To be evaluated - Usability study has given some signals, but we don't have enough data to start a design work. Further evaluation needs to be done and then a decisions should be made on whether this is an issue to be fixed or not.
To be considered - These findings don't require any immediate action; these are only interesting informative findings that should be considered in future work.

For most of the findings we also include recommendation that contains some suggestions on how to fix or proceed with the issue. And in the end we also include raw observation notes for readers reference. In the observations sections we use the notation P1 to P6 to denote participants, not priorities.

Update: Some of the sections have been already discussed with the team leads. In that case, the findings may be appended with Decision, commenting on what was the decision on further action.

Usability Study Settings and Participants

The usability study was conducted in Sun Microsystems Prague usability lab on 8 and 9 August 2006. The sessions lasted 90 minutes. The participants were given a scenario (Czech, English) covering definition of a web service and its implementation in BPEL.

The participant profiles were as follows.

Participant	Profile	SOA experience
P1	RPC-style Web Services developer, using NetBeans.	Using Java Studio Enterprise to create RPC-style Web Services. WSDL files are generated by the tool. Has a notion of what is WSDL and XML schema, but does not need to touch them.
P2	Integration developer using Tibco.	Using Tibco (about a year), using XML Spy. Creates tens of XML schemas in 6 months (with tens of elements). Uses NetBeans for Java and Tibco designer for integration.
P3	Integration developer with a custom infrastructure.	Wanted to use BPEL for an integration project (evaluated TPR and SeeBeyond); in the end they developed their own framework. They regret they haven't gone with BPEL though. Has a notion of what is WSDL and XML Schema.
P4	Novice Java developer (a student).	No experience with SOA.
P5	Java Developer	Sometimes develops WS clients, says that often uses XML Schema (authors them in Altova XML Spy), but the level of skills is quite low.
P6	Java Developer	Does not consume or create WS, knows XML Schema, but does not use it in the projects.

Findings - XML Schema Editor - General

As mentioned in the Executive Summary, one of the biggest sources of findings is the fact that the knowledge of XML Schema language is not present. A quote of P2 may shed some light on this: "When a project starts, that's about once in 3 or 6 months, we are creating XML Schemas at first. And then we go on integration." and he further clarifies "So I always forget the details and I need to lookup some documentation and refresh the knowledge".

Finding: The difference between complex types and elements is unclear to the users.
Status: P2 - To be evaluated.

All the users had problems with the concept of Complex Types and Elements. It seems that having a dual construct is somewhat redundant from the user perspective and the tool does not help the users to understand these concepts enough. When asked "what is the difference between elements and complex types" none of the users was able to answer properly.

Finding: The tool does not help the users to understand the difference between elements and complex types; they don't know what to pick in the palette; they don't understand the representations in Design View.
Status: P2 - Filled as #84155 and #84156 and #84157.

Furthermore, the tool does not help the users to understand the difference between these two concepts; user's don't know what to pick, when they want to create a new data structure. They are confused by the two sections "Global Elements" and "Global Complex Types", which apparently contain same kind of artifacts. The colour distinction does not help to understand the differences. Also the Palette category names "XML Components" and "XML Schema Components" are not helping them to understand

Decision: RFE: Rename "Global Elements" and "Global Complex Types" to "Elements" and "Complex Types". RFE: Change the names of the artefacts in palette.

Recommendation (minimal):
#
Use the colour distinction in the Design View to distinguish between inherited and locally defined elements and attributes instead of the current distinction between complex types and elements (which is not being understood). This is also supporting a finding related to inherited elements.
Note: This was actually recommended in the UI specification (see "Element" and "Element - Externally defined").

Perhaps a stronger visual distinction between Complex Types and Elements would also help. Do something with the palette category names.

Recommendation (optimal):

It seems that going further towards "instance document" style authoring would help the users to author data type definition for SOA domain. Perhaps the difference between complex types and elements could be handled more transparently; perhaps users could be shielded from that in Design View.

Observations:

P1: Difference between Elements and Complex Types – not sure.
P1: (In Design View) Expects also to be able to DnD Complex Types to Element Section in Design View.
P1: "It seems to me that the two sections in Palette are related to the two sections (Elements and Complex Types) in Design View."
P2: Complex Type vs Element: Elements: library of elements to be used, Complex Types: these are more complex for real usage.
P2: Is not sure about Elements vs Complex Types.
P4: .... doesn't know – guessing in sense of type vs instance.
P4: Dark blue vs light blue? – root vs leaf element in XSD? (... but does not get the meaning)
P3: Icons in Navigator helped me to understand what is complex type and what is element in the Design View.
P3: It seems to me that complex types and elements are sort of the same.
(I would expect to be able to define the stuff as in Java. The difference is not so significant from user perspective.)
Moderator: Would this be a valid use case: Transform element into a complex type?
P6: Element vs Complex Type? Goes to source, tries to find out in Source; Type – like a Class; Element – is like an attribute? -- does not get it.
P1: In Design View I expect different colour for inherited elements.
Moderator: User cannot see how the inheritance is visualised. In the end, user understands the two sequences (one inherited, one local)
P6: Expects the inherited elements to have different color.

Finding: The meaning of individual content models may not be clear to the users.
Status: P2 - To be considered

Recommendation:

The fact that this knowledge may not be present needs to be kept in mind. At least, the tooltips in palette of Design View should describe their meaining and F1 key press should lead to relevant help topic.

Observations:

P1: User knows sequence and choice, has a notion about all.
P2: User thinks that Choice is an attribute types with limited set of values.
P3: User understands the concept of content models.
P4: Has no idea of what the content types are.
P5: Knows only sequence, the rest understands "intuitively"

Findings - XML Schema Editor - Design View

Finding: There is no way to create a subtype in Design View, although users expect that (4 of 6 users expected a contextual menu action, 6 participants expected to be able to set the base type in properties window - see Finding related to Properties Window).
Status: P1 - filled as #84158. (To be designed and implemented (this is partially covered with an issue related to Properties Window))

Recommendation:

Primary solution of this general issue probably lies in the property sheet. Contextual menu action may be also considered.

Decision: Put an entry into property sheet and add contextual menu action.

Observations:

P1: Expects “Create Inherited” in Design View element contextual menu. Comments: Right click is always the first thing to do.
P2: Expects to find "base on" kind of action in the contextual menu of a complex type.
P2: DnD base type on the new type with pop up to choose to inherit or copy or so ...
P3: Wants to define base type in Properties. Then maybe action in contextual menu.
P6: Expects “create a sub type” in contextual menu of complex type.

Finding: It's not clear that the content model can be changed with a double click.
Status: P1 - filled as #84159 and #84160. (To be designed and implemented)
Recommendation:

Design and implement an enhancement. For example, the sequence/choice/all pulldown could appear when the container is selected. And perhaps there could be also done for the unselected state (and/or mouse-over).

Decision: BUG: In properties window and customizers, change "compositor name" to "Compositor" (or better) and update the description to describe more what's going on there. RFE: On canvas editing (switch to pulldown on compositor selection is the initial idea).

Observations:

None of the six participants realized (without moderator's help) that the content model can changed with a double click in Design View.

Finding: Blank space (and section labels) in Design View is expected to be more active.
Status: P2 - filled as #84161 (To be implemented).

Recommendation:

A contextual menu should pop up when the user right clicks on the blank space (or label of a section) in Design View. It should contain one action "Add Element" or "Add Complex Type". Also it should be possible to drop an element or complex type to blank space to create a new one.

A single click in blank space should exit in-place editing mode.

Decision: BUG.

Observations:

P1: Expects contextual menu in empty space in Design View with “New Type” action, or so ...
P1: "I would like the label of sections in Design View to be active (to expand / collapse)."
P3: Right click on white space in canvas should pop up a contextual menu – to be able to add new elements and complex types.
P4: Expects contextual menu on white space of Design View canvas to contain “Add >” entry (there's no menu today).
P5: Expects to see contextual menu in white space of Design View to be able to add new stuff.
P6: Expects contextual menu in blank space of Design View.

P1: Expects to be able to drop Complex Type to empty space in Design View

Finding: The verbal explanations on the drop targets are very useful, explanations in editor assistant are easy to miss.
Status: P2 - filled as #84162. (To be designed and implemented + P1 - To be evaluated)

Recommendation:

The visual feedback when an artefact is being dropped as a child or to an incompatible target should be improved. The explanation or error message could be moved from top of the editor (editor assistant) closer to the drop target. Also a visual aid that an element will be added as a child could be expressed. Also this pattern could be reused accross all the editors (to be evaluated).

Decision: RFE

Observations:

P1: User likes the Editor Assistant.
P1: (talking about inserting a new element/complex type and specifying a base type) Also there could be pop up after DnD. Because I don't know how to start, when I DnD and nothing happens besides the new element on the canvas.
P3: “Drop line is important for me. I understand that.” ... Moderator: sometimes the tooltip on the drop line is missing ---- the error in the code assistant would be better placed on the line,.... it's inconsistent.
P3: Again – editor assistant should be there on the drop line consistently.
P4: Didn't see editor assistant messages at first, then does not pay attention to them anyway.
P5: Does not see editor assistant it at all. "I think it's dangerous that I can redefine super definition on it's usage place." (although the explanation is given in the editor assistant)

Finding: Users are sort of hesitant to use Design View. It seems it renders "unreliable" impression, perhaps it has someting to do with the visual appearance (may looks too fat).
Status: P2 - filled as #84163. (To be evaluated)

Recommendation:

We should think about how to make the design view look more "reliable", more visually appealing, more "oh, this is all I ever wanted".

Decision: RFE as low priority. Let's wait for more feedback.

Observations:

P1: I prefer Schema View to create the new type, because I'm used to interaction with the tree, contextual menus, etc. Design View looks foreign to me.
P2: Finds Design view components "visually thick"; "wasting too much space with graphical components", He would prefer to see them thinner and smaller, otherwise he warns against "scrolling out to user's death", when a schema is large (moreover he expresses his fellow feelings about his colleagues having it even harder with their small notebook displays)
P3: I'm happy that I can look at the data structure without decorations (XSD language) in Design View. I like that the elements look like XML <>.
P5: Likes the schema view more, Design view is not so common. "I'm more used to tree."
P6: I would pick schema view because it's closer to me. In design view it's too ?thick? ... I don't like DnD style.
P6: I like the visual appearance of the Design View better than XML Spy.

Finding: Interaction with palette may not be obvious, 2 of 6 participants knew how to use palette immediately, 2 participants understood after some time, for 2 participants the concept of DnD from palette was completely foreign.
Status: P2 - To be evaluated

Recommendation:

Today, there is a tooltip suggesting to DnD on all the palette items. However, it would be also usefull (and consitent) to use tooltip to give more information about the palette element. That would even be consistent with NetBeans practice and aligned with a more severe finding about the missing knowledge of content models. So the recommendation is to change the tooltip to give more information about the palette elements and do a further research on palette usage (as it does not only impact XML tools but many other features).

Decision:

Observations:

P1: Expects left click in Palette to create new type in the file. Also tries right click in the Palette. Then finds tooltip to DnD - that helped.
P2: Get's it quickly, just not sure about Elements vs Complex Types.
P3: “User not sure if this is modeling or instance dokument authoring tool.” Then he realizes.) ... Didn't see the palette in the beginnig, so it didn't help him to understand that it's a modelling tool.
P4: Didn't see palette in the beginning. Expects click on palette to add a new object on the canvas. P4 (Moderator): Contents of the palette is not that much clear, the sections are even less clear.
P5: Does not see palette in the beginning, would not expect to be able to DnD from Palette (even when directed).
P6: Immediately starts working with Properties window in Design View. Also gets Palette immediately.

Finding: Attributes are sort of hidden in Design View.
Status: P3 - filled as #84164. To be further evaluated.

Recommendation:

Perhaps, if they were shown by default it would help the users to understand.

Decision: RFE: Let's put a toggle button with attribute icon to show / hide all attrtibutes.

Observations:

P1 would prefer to see all attributes displayed by default.
Other participants also had problems with this.

Finding: One user was not sure if the Design View is a modelling tool (schema authoring) or instance document authoring tool.
Status: P3 - To be considered (not enough data).

Observation:

P3: “I'm not sure if this is modeling or instance document authoring tool.” Then he realizes. Didn't see the palette in the beginnig, so it didn't help him to understand that it's a modelling tool.

Finding: One user expected F2 key press for in-place editing in Design View.
Status: P3 - filled as #84165. To be implemented

Recommendation:

Implement this functionality.

Decision: RFE

Observation:

P3: Expects F2 for in place rename in Design View.

Finding: Global Complex Types and Global Elements section separators may be easy to miss.
Status: P3 - filled as #84163. To be evaluated

Decision: RFE

Observations:

P3: Global Complex Types and Global Elements section separators are easy to miss.

Finding: User expected “Go To Definition” in Element contextual menu in Design View.
Status: P3 - filled as #84166. To be evaluated

Decision: RFE: Add contextual menu action and Ctrl + Click gesture.

Observations:

P1: Expects “Go To Definition” in Element contextual menu in Design View.

Finding: Expand / Collapse buttons on elements in Design View may be easy to miss.
Status: P3 - filled as #84167. To be evaluated

Decision: RFE. Wait for more feedback.

Observations:

P1: "I would rather see [+] icon in sections to expand/collapse."
P5: Tries double click on element in design view to expand/collapse.

P5: Does not see expand/collapse section buttons. User was directed by the user. .... than after 5 minutes, user forgets about them again.

Finding: It may be too easy to miss the "No Children" label in Schema View.
Status: P3 - To be evaluated

Decision: Evaluate the conflict with the finding that no column on the right was also a usability issue.

Observations:

P5: User does not see “No Children” in the right most column. He thought that perhaps something is broken.

Findings - XML Schema Editor - Inherited Children

Finding: 4 of 6 participants were confused by the fact that they can see inherited elements in Design View (and Navigator) whereas they cannot see them in Schema View.
Status: P1 - To be evaluated

Recommendation:

Include inherited children in Schema View. There could also be a switch (in the toolbar) that would turn on and off the inclusion of inherited elements. Perhaps it could be shared among the views. Particular UI design needs to be further discussed.

Decision: Evaluate the conflict with the issue from the previous study.

Observations:

P1: “Schema View looks chaotic to me. I don't know what are attributes. And there are no children in Columns View, but I can see children in Design View."

P2: Confused/surprised - Expects to see inherited elements (inherited sequence) in Columns View
... also surprised that he sees two sequences in Navigator but just one in Columns View
P4: Expects to see inherited elements in the Columns View.
P5: User would like to see inherited elements.

Finding: In Design View, 4 of 6 participants dropped a new element into the inherited sequence when they wanted to add a new element locally. In the end they understood the concept, but were arguing that it might be a dangerous feature.
Status: P1 - filled within #84157. To be evaluated.

Recommendation:

Part of the solution might be using proper color distinction for inherited and locally defined elements - which is also suggested elsewhere. Also the visual feedback of the drop target could be improved to better advertise that an element is being added to a complex type. The drop line tooltip should include the complex type icon and the label should be in the form "Drop here to add to complex type <icon> <complexType.name>" - i.e. with sentence capitalised "complex type". Morover, the visual feedback when an element is being added as child also needs improvement.

Decision: BUG.

Observations:

P1: He criticizes that inheritance in Design view is not displayed well. He would prefer more expressive depiction of this relation.
P2: "I think it's dangerous that I can add elements to base type while working with sub type - there should be at least a pop up".
P4: Participant dropped an element into the inherited sequence, but that is not what he wanted.
Moderator: Maybe the highlighting should be better.... maybe usage of light vs blue (light – local, dark – inherited).
P5: I think it's dangerous that I can redefine super definition on it's usage place.
Editor assistant – does not see it at all.
P6: (In Design View) Adds the new element to super type – didn't want that. ..... then in discussion it's weird. I think it would not make me faster. Maybe the tool could hide/show inherited elements for me with a switch.

Findings - XML Schema Editor - Tree and Columns View

Finding: The choice of tree vs columns is more a user preference than per file basis. The choice of the view (tree vs columns) should be persisted.
Status: P1 - filled as #84187. To be designed and implemented

Recommendation:

Persist the choice of tree vs columns view. The last user choice (within the IDE) should be used for a newly opened editor. The choice should be persisted per editor type (for XSD and WSDL editors individually) - see Finding in WSDL editor.

Decision: RFE

Observations:

P1: "TreeView looks better. I don't like Columns view, it's weird. It might be easier/faster to navigate. I might get used to it."
P1: After performing the task he admits there are some advantages of the Column order. "It looked strange on the first time. But when I got used it, the navigation can be faster here. You do not need to scroll down, and expand so many times. Especially when schema is complicated."
P2: Get's columns view quickly but ... P2 feels like "not having control" of Schema view, because new columns appear and disappear "unexpectedly" when he navigates through it.
P2: Prefers tree view, because he can see two neighbors with all their sub-elements at once. So he can see their relations and context.
P3: Going back in columns view is hard.
P3: Columns view – didn't see types, rather went to source.
P3: Columns view heavy - cognitive load, hard to get oriented, losing context, losing feel of structure.
P3: Tree feels better.
P3: Tree view - It allows me to view to siblings at once ... and compare them. - this is important feature. But I'm not sure how important it is.... (i think it would be good) I just like the tree better.
P4: Likes the tree view better.
P5: User likes Columns view.... then says that Tree View is better, it's more transparent / clear. Hates scrolling, ... I cannot see two siblings at once.
P6: Columns view vs Tree Views. I like the tree better / closer to me. I think that Columns view is weird.
P1: User expects to get back to Tree View (preferred). User forgot about the buttons and was surprised that he is back in Columns View. - the choice should be persisted perhaps.

Finding: In columns view there is a usability problem with scrolling backwards. The columns view only scroll when a unselected item is selected. Users expcect the view to scroll even when they select a a previously selected item.
Status: P1 - To be implemented (issue #83317 filled)

Recommendation: Improve the behaviour of the columns view in the sense of the filled issue.

Finding: All the 6 participants liked the tree view better. 3 of them said that seeing the structure of two siblings simultaneously is important to them, however they were not able to describe the use case. For others it was only a question of "feel". It seemed that some of the users might get used to the columns view during some time.
Status: P2 - To be considered.

Recommendation: Evaluate the use cases related to structures of two siblings and apply findings to the competition of column and tree views.
Note: One of the possible scenarios for such use case would be cloning the document editor. A "clone from here" action might be a boost this scenario, if we find out that it's important.

Findings - XML Schema Editor - Properties Windows & Customizers

#
Finding: All the 6 participants expected to be able to specify a base type of a complex type in Properties Window while in Design View.
Status: P1 - filled within #84158. To be designed (this is partial solution to a more general issue related to base types in Design View).

Recommendation:

Extend the property sheet of the complex type in Design View to be able to set the base type (there is currently no scenario allowing the user to specify the base type in Design View). This needs to be done in sync with other improvements to properties windows and customizers.

Observations:

P1: Expects to be able to specify base type in Properties Window of Complex Type in Design View.
P2: Looking for inheritance in Properties Window (in Design View).
P3: Wants to define base type in Properties. Then maybe action in contextual menu.
P4: Expects to see “type” in Properties.
Moderator: Maybe also there should be customize action in contextual menu of complex type in Design View of XML Schema Editor. But that's a UI of a different level?
P5: Expects to see “extends” in properties window of a complex type.
P6: Expects to be able to specify base type in properties window of a complex type in Design View.

Finding: The difference between Customize and Properties actions is not obvious. It's not clear if these contain mutually exclusive pieces of information or if these are just different views on the same (one data based, the other task based).
Status: P1 - To be evaluated

Recommendation:

To be evaluated. Possibly merge Properties Window and Customizers into one or highlight the difference more.

Observations:

P1: Difference between Customize and Properties? - "Don't know, I think they seem equal to me."
P2: Properties vs Customize – Customize is more powerful
P4: Customize vs Properties, ... expectations: Customize – change name or data type; Properties – don't know.

Finding: The contents of Properties Window is often confusing, users don't understand its items. It's not cleat that they can specify the base type in "Structure ..."; there is a disconnect between "Structure" entry in the Properties Window of a complex type in Design View, "Customize" contextual menu action and "Customizer" title of the dialog.
Status: P1 - To be evaluated

Recommendations:

To be evaluated. Perhaps something from the 'structure' customizer could be pulled into the property sheets. There could be a "Structure" category in property sheet with items that would advertise and allow the user to specify the base type. It seems that the words "base type", "extension", "inheritance" or "instance of" in the Properties Window would help the users to fulfill the use case of specifying a base type.

Observations:

P3: Didn't see super type on element node in columns view, didn't see it in properties window.
P3: Structure pop up is suprising, “don't know what that is.” ... "all this stuff is foreign to me".
P5: Properties Window of complex type in Schema View is not clear at all ... to many items, no meaning.
P6: Looks for base type of complex type in properties window and does not like it (besides name... and ID?) ... hits structure and gets customizer.

Finding: The contents of customizers is not clear to the users. The amount of choices (the decision complexity) is overwhelming.
Status: P1 - To be evaluated + P1 - To be implemented (bug with currently selected node in embedded choosers)

Recommendation (minimal):

There are some miminalistic changes that can be done to improve the usability of customizers:

The label "Type Definition" and the labels of the radio buttons could be more explanatory. Also the labels could serve that purpose.
The label "Global Reference" should also express more.
There should be a label above the tree chooser helping the user to understand that it's dependant on the previous choice of a radio button.
The "Inline Compositor" pull down does not need to be that wide (it should be right aligned or narrower).
The label "inlince compositor" also sounds scary.

Decision: BUG

Recommendation (optimal):

For a "Design View" user, this is "completely foreign" - and there's no scenario to work around this. In-place tree choosers are not always clear, perhaps they are adding too much noise. Also the "current selection" is very problematic - selected node is not visible when the customizer pops up - that's a significant usability issue. This also applies to type choosers in Design View (and perhaps other editors)!

Observations:

P1: "I don't like “Use Existing Definition”, I would expect to see the word “inherit” in the option label."
P3: It seems that the tree in the Customizer belongs to Group radio button.... (after 15 secs of playing with that) then sees that drop down is related to the first radio button.
P3: "Preview is useful."
P4: Tries to edit the Preview in Customize editor.
P5: "Prohibited Derivations dialog is confusing ... does not mean anything to me."
P6: Does not like customizer... not sure if he can extend the type here... "Use Existing Definition?" --- .... liked the word “extension” ... "Preview helped me to understand...."

Findings - XML Schema Editor - All Views

Finding: Users may not be aware of or may not keep in mind the restrictions on names (e.g. no spaces).
Status: P1 - filled as #84195. To be implemented

Recommendation:

Design and implement a validation solution.

Decision: BUG

Observation:

P4: user has changed the name of element to “first name” with space! P4: "I expect not to be allowed to add space if it is not valid."

Finding: Sometimes users get lost because they don't know the type (kind) of a node in tree or list.
Status: P1 - filled as #84190 - To be implemented (properties title) + P2 - To be evaluated (help section)

Recommendation:

Provide the information about the kind of node on multiple places if possible. The title of the properties window should be used consistently in format "$node.name [$node.kind] - Properties". Also the help section of the properties window and the kind entry in the property sheet are used inconsistently.

Decision: BUG

Observation:

P1 Is not sure what Country (in Complex types/tree schema view means). He noticed it is not part of the sequence. "Does it mean that it is optional?" "According to the Schema I am not able to recognize whether Country is element or attribute." The schema and icon are not guiding enough for him. He switches to Source to find it out.

Finding: Sometimes the users think that source view is more friendly to them (perhaps in some use/cases); that it's easier to find or understand the definitions.
Status: P1 - To be considered

Recommendation:

While designing logical or graphics views or editors, compare them to source view; match the efficiency of use cases and scenarios in various views.

Observation:

P1: User is not sure what Country (in Complex types/tree schema view means). He noticed it is not part of the sequence. "Does it mean that it is optional?" "According to the Schema I am not able to recognize whether Country is element or attribute." The schema and icon are not guiding enough for him. He switches to Source to find it out.
P2: Uses Source when he gets lost in Schema and Design. "Schema is more friendly" Immediately considers Design view "for editing", Schema view "for viewing" and "Source is raw :)".
P3: Mentions that he is moving towards developing in Design view trying to do less work in source.
P3: "Source is easiest to understand" "Schema provides the best visual representation of element's inheritance" Sometimes he has to click back to source, "Sometimes it is easier to find something in Source than in Schema. I could not find types in Schema, but I found it immediately in Source. "
P2: (Source) In XML Spy, sometimes I got lost (or cannot find what i need in properties or so) in so I switch to source.
P6: Source vs schema vs design. It seems to me that source is more readable than schema view.

Finding: The UI related to order of elements and complex types has a couple of flaws.
Status: P2 - filled as #84192. To be evaluated
Note: This is only partly supported by the data from the usability study.

Currently the order of elements and complex types reflects the order in XSD source. However, it's not clear if that's important to the user of Design View. Morover, if there is a large number of objects in the list, it is very hard find and navigate to the desired one. Traditionally, alphabetical order is used to ease navigation.

Recommendation:

If the order of the artifacts is significant to the user, there should be a way to change it. Also the UI should behave properly if the user wants to add a new complex type to a specific position. Today, complex type is always added to the end (in design view).

If the order is insignificant, the objects should be ordered alphabetically (and newly added element should be placed to proper location after renaming).

Also the relationship between Navigator and editor views should be cleared out (should a navigator view be called 'design view'?).

Decision: RFE: Make the lists (with exception of sequence) alphabetically ordered.

Observations:

P3: Expects alphabetical order where possible. "But I believe that there are also valid use cases where custom order is important. Maybe there could be View switch."
P?: User tried to add a complex type to a particular position (the DnD visual feedback was also suggesting that), but the new type was added to the end.

Finding: Users may expect the selection in schema view and design view to be synchronized.
Status: P3 - To be evaluated

Observation:

P6: Expects the selection to be synchronised between views (selects something in design then goes to source).

Note: This is also related to all the other views (or perhaps to the editor as a whole)

Finding: Users might appreciate enhanced expand/collapse features in Tree View.
Status: P3 - To be evaluated

Observations:

P3: Would like to see expand all or expand from here action on nodes in Tree View.

Finding: "Go To Super Definition" may not be clear to the users.
Status: P3 - filled as #84193. To be evaluated

Decision: RFE. Let's make it more context sensitive.

Observations:

P5: I would like to be able to get to parent type definition. “Go To Super Definition” - is not clear to me.

Finding: Users may be confused or distracted by the fact that results of "find" are visible in choosers in pop up dialogs (in Schema View).
Status: P3 - To be evaluated

Observation:

P3: Find is still visible there in the tree (I'm not sure if this is what I want, if I like it ... and I can get rid of it in the Find bar ... but I'm in the modal dialog).

Finding: "Create Skeleton" might be an interesting feature.
Status: P3 - To be considered

Observations:

P1: Expects “create skeleton” XML document of XML schema.
P3: It seems that instance document would be an interesting feature. And more in comment to design view (also copied in design view section): (“User not sure if this is modeling or instance document authoring tool.” Then he realizes.) ... Didn't see the palette in the beginning, so it didn't help him to understand that it's a modelling tool.

Findings - WSDL Editor

#
Finding: The WSDL tree is too complex. The relationships between objects are not obvious. The types of nodes are not clear neither.
Status: P1 - To be evaluted + P1 - To be implemented (in sense of minimal recommendation). Properties Windows titles filled as #84190.

Recommendation (minimal):

Add kinds of nodes to the label of properties window. The format should be "$name [$kind] - Properties"; for example "part1 [Part] - Properties ". Add details of nodes to the labels of the nodes in gray colour (as it is done in Java, XML or BPEL Navigators); for example the label of a message part could be "itinerary ota:TravelItinerary"

Recommendation (optimal):

Design a UI where the relationships between components are discoverable in another way than through the properties window.

Observations:

P2: Tries double click on node to get properties window, because he does not see it anywhere.
P2: Expects to be able to drill down in the type of message part – to get visual feedback of setting the type, to check the correctness, to confirm that.
P4: Expects to be able to specify type of input operation parameter in the properties window of the operation child node.
P4: Relationship of various nodes in the WSDL tree are not clear to a novice user. (a graph would be better probably).
P4: Looks for type definition in properties window of a message node.
Moderator: Understanding the WSDL tree is really painful for a novice. Relationship should be exposed better than just burried in the Properties window. (maybe relationship between kinds would be enough, maybe we don't need to show relationship between individual entries).
P5: There are just too many levels of indirection .... type – part – message - operation – port type – partnerlinktype – partnerlink – receive - variable.
P6: I need to see what is the signature of a method (operation) ...
Moderator: Something to be able to understand the node kinds are missing significantly.
P6: Being a novice, trying to play with the tree view, trying to add children to various nodes, but still ... not understanding the web services descriptions / concepts much – not understanding what is related to what.
P6: In the end understand the relationship of operation parameters and messages.
Moderator: Perhaps “part” should be named “message part”.
P6: I'm just waiting where I will be able to define the type ... finally. (comment on the way how he is still 'virtually' drilling down)
P6: I just feel like that WSDL is not a tree ... so it should not be a tree.
P6: Maybe it would be better to see operations first and then messages.
P6: Expects to be able to DnD a type to operation signature to define type of parameter.

Finding: The WSDL language and its constructs may not be clear to the users.
Status: P2 - To be considered
Recommendation:

Consider this fact while designing WSDL features.

Observations:

P2: I usually have WSDL generated, so I don't care about binding.
P2: Policy: Changes the name of message part Order to order (lower case)
P2: Expects to see data type for a message in message properties window. Does not expect that message can have parts, directed to it by moderator...
P6: Messages – are like operations, Port Types – related with runtime environment, Bindings – binding to data, Services – calling another web service.
(see also observations in Finding: The WSDL tree is too complex)

Finding: To some users, it may be confusing or annoying if the refactoring dialog pops up automatically on Rename, some users would be fine with that but would expect to be able to turn it off.
Status: P2 - To be evaluated (has this been already fixed?)

Observations:

P2: Understands the refactoring pop up and XML Refactoring window. No surprise. Rename dialogue and left side of refactoring window are clear for him; "And what is this?" (looking on the right pat of refactoring window); double clicks on the diagram inside. User likes the feature (protection against undesired changes). Also thinks that it should be possible to turn it off somewhere (to make it work transparently).

P6: Surprised. Didn't expect to see "confirmation" of renaming.

Finding: The types subtree seems to be too complex. It seems that the fact that imported data types need to be wrapped in a locally defined schema definition is unnatural to the users.
Status: P2 - To be evaluated.

Recommendation:

Perhaps a view that would just show locally and externally defined elements would be better.

Observations:

P1: Surprised that there is “No Target Namespace” in the tree as he was specifying some in the New WSDL File wizard (does not get that it's related to the namespaces wrapping the imported schema. Also expects to see imported Complex Types directly under the Types node, does not see Referenced Schemas.
P4: User (in a couple of first seconds) expects to find imported data types directly under Types (not under referenced schemas).
P4: Understands that grey nodes of imported schema cannot be changed from within the WSDL editor.
P6: Looking into complex types under Types, not finding the stuff he imported.

Finding: The names used in skeleton WSDL are a bit confusing. Users were not sure if these are abstract or concrete names.
Status: P2 - To be implemented.

Recommendation:

Make the names like "sampleRequestMessage", i.e. remove the name of the WSDL file (which is only adding noise) and add the word "sample".

Observations:

P6: "The default names in the skeleton WSDL looks like abstraction, not like real instances."

Finding: Users like the "Import XSD" feature in the New WSDL File wizard.
Status: P2 - To be considered.

Observations:

P1: Expects to be able only to import some elements from XSD.
P2: User likes Import Schema in New WSDL wizard. User likes the created skeleton (or the fact that a skeleton has been created).
P4: Sees and uses “import XSD immediately”.
P6: Does not use “import XML Schema” option in New WSDL file (although the task was explicitly talking about linking to previously created XSD file).

Finding: Users may prefer the tree view. Columns view vs tree view is probably a user preference, rather then use case based and thus the choice should be persisted.
Status: P1 - filled as #84188. To be implemented

This finding is not based on observations in WSDL editor, however we have paid attention to this question in XSD editor and believe that the findings made there can be applied also here.

Findings - BPEL Module and Composite Application Project Types

Findings: WSDL File template is not accessible from the New File wizard in BPEL Module and Composite Application projects (which is a bug and users expect that).
Status: P1 - To be implemented

Recommendation:

Now, the "WSDL File" template is in Web Services category, which is not visible in BPEL Module and Composite Applications projects. It should be moved to a more generic "XML" category, where it would reside with other XML files. There is an impact on NetBeans Java EE features (web services development), but it seems it's a good solution to put logical templates such as "Web Service" or "Web Service Client" into "Web Services" category and low level file templates into the "XML" category.

Also the template should be named "WSDL File" instead of just "WSDL".

Obsevations:

P1: Looks for WSDL file in XML Category of New File wizard.

P6: Expects to find New WSDL template in the New File wizard ... in XML category, maybe also in SOA category ... "I have the association with webservices." Has problems with “retrieve external WSDL”

Finding: Users understand the difference between Composite Application and BPEL Module project types.

Status: P3 - To be considered

Observations:

P2: Picking up “Composite Application” for “integration” project. BPEL module is more concrete. Composite Application is more general.
P5: "got it ,... i just need one module"

Findings - BPEL Editor

Finding: The story of imports, WSDL files, partner link types and partner links is not obvious; the tool does not make it easy.
Status: P2 - To be evaluated, designed and implemented.

Recommendation:

The end to end story needs to be reviewed and improved.

Observations:

P2: User does not know that he needs to add Partner Link Type in WSDL.
P4: Finds Import node in Navigator quite quickly.
P5: User does not understand that in the “Partner Link Type” pop up editor the pulldowns are interdependent.
P5: User expects to see “Add WSDL” somewhere in the business process; imports are not clear enough. User thinks that he can only import an existing BPEL file.
P6: Moderator: The mindswitch – to bring WSDL interface into process/implementation – is very hard to make.

Finding: The visual feedback given when a WSDL file without a partner link type definition is DnDed to the diagram is unclear (as it shows a green box, but nothing appears when the operation is finished).
Status: P2 - To be designed and implemented.

Recommendation:

Enhance the visual feedback that is given when a WSDL file is DnDed to the diagram. A message should be shown (as in XSD editor) saying that "WSDL does not contain partner link type definition. Only import will be added.".

Observations:

P4: Moderator: Also when DnDin a WSDL file that does not have any partnerLinkType definitions to BPEL process, the visual feedback is unclear. Operation is done, but nothing appears on the diagram.

Finding: The visual feedback given on DnD of a WSDL file is not right (red rectangle).
Status: No action required. This has been already fixed.

Recommendation:

Change the visual feedback to a more friendly one.

Observations:

P4: Moderator: Red rectangle of DnD WSDL rendered an impression that it's not correct.
P4: Possibly also visual feedback on DnD of Partner Link from Palette is not right.
P5: User does not know where to put partner link – bad visual feedback.

Finding: User may have weak knowledge of the underlying technologies, educational aspect of the tool is important.
Status: P2 - To be considered.

Observations:

P2: I don't know what's “Correlation Set”. P2: Create Instance, understands that, set to yes.
P4: Likes the tooltips in Palette – finds them very useful.
P5: User does not know anything about partner link type.

Finding: Dual gateways (opening and closing) may be confusing to the users. Also dual "start events" may cause confusion.
Status: P2 - To be considered.

Observations:

P2: "Receive should be start event", user does not like dual “start”.
P5: User thinks that closing gateway of an If elements is another condition evaluation.
P6: Does not understand scope in ForEach; also does not like closing gateway in the BPEL process - user thinks it's another decision point.

Finding: Users may expect mouse-over to work on error button of the validation callout.
Status: P3 - To be evaluated, designed and implemented.
Observations:

P2: Expects mouse-over to work on error button of the validation callout in diagram.

Finding: Users may expect keyboard navigation and modification on BPEL diagram.
Status: P3 - To be considered (ultimately all the visual editors should have equal keyboard navigation).

Observation:

P6: Expects key navigation and modification (insert to add something).

Finding: The concept of placeholders is fine, their visualisation is perhaps still too sadle (for out of the box experience); the concept of required placeholders probably still does not fit.
Status: P3 - To be evaluated, designed and implemented.

Recommendation:

Make placeholders more visible. Evaluate possible design solutions for required placeholders.

Observations:

P2: User likes placeholders. But also: "I didn't like the “required placeholder”; that was confusing to me;"
P4: Get's the concept placeholder on second time.

Finding: The left mouse DnD + shift gesture to create a message flow is hard to discover.
Status: P1 - To be designed and implemented (Note: This is being worked out within issue #83489).

Observations:

P2: User tries left mouse DnD to create message flow; then expects action in contextual menu of partner link like “Link To >”; when directed in the end he tries Shift. Expects plain DnD to work.

Findings - BPEL Mapper

Finding: The concept of literals may not be clear or may seem to be redundant or unnecessary. Users may want to specify the literals directly as "in-place arguments" of the functoids.
Status: P2 - To be evaluated, designed and implemented.

Observation:

P2: Expects to be able to fill in a constant value as input operand of a functoid.
P5: User picks “Number” instead of “Number literal”

Finding: The visual feedback given when a node is being DnDed from the left or right panes in the BPEL mapper but connection point has not been reached yet may not be sufficient.
Status: P2 - To be designed and implemented.

Recommendation:

Perhaps we should send out a negative signal while the mouse pointer is not above valid location (make the line red).

Observations:

P2: In BPEL mapper: Tried to DnD node from element to canvas, expects to see a representant of the node on the canvas.

Finding: Creating the connections between functoids has usability issues (drop zones are not large enough).
Status: P1 - To be implemented. This has been already fixed.

Observations:

P2: User has difficulties while connecting the connection points of functoids (inappropriate drop zones).
P5: User tries to drop the line in bpel mapper to a connection point; it does not work.

Interesting Quotes

P2 in regards to his knowledge of XML Schema domain: "When a project starts, that's about once in 3 or 6 months, we are creating XML Schemas at first. And then we go on integration." and he further clarifies "So I always forget the details and I need to lookup some documentation and refresh the knowledge"

P3: "NetBeans are going deep, I like that. In comparison to Eclipse, where they go broad but don't (never) finish."

P3 in regards to Design View of XSD Editor: "I'm not sure if this is modeling or instance document authoring tool."