Difference between revisions of "User Interface Evaluation"

From MPDLMediaWiki
Jump to navigation Jump to search
 
(61 intermediate revisions by 5 users not shown)
Line 1: Line 1:
{{Template:TemplateEvaluation}}
{{Template:TemplateUIE_Activities}}
 
<div style="float:left; width:70%; margin-bottom:3em;">
 
Detailed evaluation of the human machine interface is an effective method for designing a better user experience. Modern web applications are expected to deliver a high degree of usability. User participation within the development process is essential to meet those expectations. To balance the developer’s perspective, proven methods such as workshops and usability tests are applied to ensure the necessary degree of user participation in interface development.


Interfaces, built at the MPDL are subject to evaluation. Main focus are all interfaces of the current PubMan solution. Evaluation of the User Interface Engineering team has different approaches: Usability Interviews and tests, GUI Workshops and software based evaluation.  
Interfaces, built at the MPDL are subject to evaluation. Main focus are all interfaces of the current PubMan solution. Evaluation of the User Interface Engineering team has different approaches: Usability Interviews and tests, GUI Workshops and software based evaluation.  


{| align="center"
==Usability Interviews (Thinking Aloud)<ref name="Thinking Aloud">[http://en.wikipedia.org/wiki/Comparison_of_usability_evaluation_methods Wikipedia - Comparison of Usability Methods]</ref>==
  | __TOC__
  |}


==Usability Interviews (Thinking Aloud)==
Interface releases are tested together with potential users from institutes of the Max Planck Society. They perform tasks covering important application functionality. An interviewer tracks feedback and observes user interactions. All issues are noted down. The test result is accumulated if at least 8-11 interviews are performed. A summary is given at the end of the interview series which leads to measures and changes in the interface.
 
The following table shows what is needed to conduct an interview at an institute.
 
'''Setup'''


{|{{Prettytable}}
{|{{Prettytable}}
Line 17: Line 23:
!bgcolor = #eee |Supervisor
!bgcolor = #eee |Supervisor
|
|
1 plus 1 visitor (usually developers)
1 plus 1 visitor (optionally, usually developers)
|-
|-
!bgcolor = #eee |Equipment
!bgcolor = #eee |Equipment
Line 26: Line 32:
<li>Tasklist (provided by UIE)</li>
<li>Tasklist (provided by UIE)</li>
<li>Test Form (provided by UIE)</li>
<li>Test Form (provided by UIE)</li>
<li>Recording Device</li>
<li>Recording Device (optionally) </li>
</ol>
</ol>
|-
|-
!bgcolor = #eee |Duration
!bgcolor = #eee |Duration
|
|
1h
45min
|-
|-
!bgcolor = #eee |Results
!bgcolor = #eee |Results
Line 37: Line 43:
Document with notes on user actions for each action/step.
Document with notes on user actions for each action/step.


The results are analysed after 6 - 8 interviews towards a statistic, providing GUI flaws within use cases.
The results are analysed after 8 - 11 interviews towards a statistic, providing GUI issues ordered by functional area and tasks.
|-
|-
!bgcolor = #eee |Steps
!bgcolor = #eee |Steps
Line 47: Line 53:
</ol>
</ol>
|}
|}


Usability interviews are conducted to discover, document and classify usability issues for
Usability interviews are conducted to discover, document and classify usability issues for


* a specific release (e.g. PubMan 2.0.0.1)
* a specific release (e.g. PubMan 2.0.0.1)
* a functional prototype
* a functional prototype<ref> A functional prototype (Model) (also called a working prototype) will, to the greatest extent practical, attempt to simulate the final design, aesthetics, materials and functionality of the intended design. The functional prototype may be reduced in size (scaled down) in order to reduce costs. The construction of a fully working full-scale prototype and the ultimate test of concept, is the engineers' final check for design flaws and allows last-minute improvements to be made before larger production runs are ordered. Definition retrieved from [http://en.wikipedia.org/wiki/Prototype#Basic_Prototype_Categories Wikipedia article on Prototypes]</ref>
* a prototype draft
* a prototype draft


Line 61: Line 66:
#Fatal (Task could not be finished successfully or in a proper way)
#Fatal (Task could not be finished successfully or in a proper way)
#Serious (Task could not be finished on the first attempt or user performance is bad)
#Serious (Task could not be finished on the first attempt or user performance is bad)
#Minor = (The user hesitates, is not sure or does not feel comfortable with the flow)
#Minor (The user hesitates, is not sure or does not feel comfortable with the flow)
#Cosmetics = (No influence on user performance)


For each interview the UIE team provides an interview outcome. The document contains all tasks, observation notes and comments of participants. Additionally the interviews collect demographic characteristics as well to prepare data needed for personas. After a valid number of interviews exists they are analyzed and summarized. See all documents and the summary here:
For each interview the UIE team provides an interview outcome. The document contains all tasks, observation notes and comments of participants. Additionally the interviews collect demographic characteristics as well to prepare data needed for '''[[personas]]'''.  


[https://zim01.gwdg.de/repos/smc/trunk/04_Design/03_GUI_Design/04_Usability_Evaluation/ Interview Protocols]
=== Interview Form Example (Pullout only)===


===Interviews for PubMan Release 2===
{| {{prettytable}} style="font-size:10px; line-hight:0px;"
'''Interview series I'''
| align="center" style="background:#f0f0f0;"|'''Submission'''
*Munich: 2 Interviews
| align="center" style="background:#f0f0f0;"|
'''Interview series II'''
| align="center" style="background:#f0f0f0;"|
*Nijmegen: 9 Interviews
| align="center" style="background:#f0f0f0;"|
'''[[User Interface Evaluation/Workshops/Interview Analysis|Interview Analysis]]'''
| align="center" style="background:#f0f0f0;"|
 
| align="center" style="background:#f0f0f0;"|
===Interviews for PubMan Release 3===
| align="center" style="background:#f0f0f0;"|'''Participant 01'''
 
| align="center" style="background:#f0f0f0;"|
'''Interview series I'''
| align="center" style="background:#f0f0f0;"|
{|style= color:black; background-color:#fff; cellpadding=5; cellspacing=0; border=0
| align="center" style="background:#f0f0f0;"|'''Participant 02'''
|<b>Place</b>
| align="center" style="background:#f0f0f0;"|
|
| align="center" style="background:#f0f0f0;"|
*Max Planck Institute of Molecular Plant Physiology, Berlin: 2 Interviews
| align="center" style="background:#f0f0f0;"|'''Participant 03'''
*Max Planck Institute for Gravitational Physics (Albert Einstein Institute), Golm 2 Interviews
| align="center" style="background:#f0f0f0;"|
*Max Planck Institute for Human Development, Berlin 2 Interviews
|-
| ||'''Nr.'''||'''Task'''||'''Nr.'''||'''Step'''||'''Status'''||'''Issue Comment (Observation)'''||'''Type'''||'''Status'''||I'''ssue Comment (Observation)'''||'''Type'''||'''Status'''||'''Issue Comment (Observation)'''||'''Type'''
|-
| ||'''1'''||'''[[PubMan_Submission|Submit a publication item detailled]]'''||||||||||||||||||||||
|-
| ||||||1||clicks submission in the Main menu||OK||||||OK||No choosed it from startpage content||||OK||||
|-
| ||||||2||clicks on full submission||OK||Automatically chosen||||OK||Easy Submission||||OK||||
|-
| ||||||3||clicks a collection (optional)||OK||Automatically chosen||||||||||OK||||
|-
| ||||||4||identifies group basic||||||||||||||OK||||
|-
| ||||||5||selects genre = journal article||||||||OK||||||OK||||
|-
| ||||||6||fills title||||||||OK||||||OK||||
|-
| ||||||7||identifies group person & org||||||||OK||||||OK||||
|-
| ||||||8||enters author name, role||||||||OK||||||OK||||
|-
| ||||||9||clicks on select or types manually||||||||Serious||Did\'t use Affiliation input (mixed up with source)||Procedure||OK||After automatic detection all authors must be entered||
|-
| ||||||10||clicks an organization in tree (optional)||||||||Serious||Used \"Help\" to get information (not helpful).||Procedure||||||
|-
| ||||||11||identifies group details||||||||OK||||||OK||||
|-
| ||||||12||fills a date||||||||OK||||||OK||||
|-
| ||||||13||identifies group source||||||||Serious||Mixed up source data and publ. Data||Procedure||OK||||
|-
| ||||||14||fills genre and title||||||||Minor||Was looking for \"Sternderl\"||Position||OK||||
|-
| ||||||14||clicks on the button ‘submit‘||||||||||Validation cycle! (Aff. Missing)||||OK||||
|-
| ||||||End||||||||||OK||||||OK||||
|-
| ||'''2'''||'''Upload 2 different files'''||||||||||||||||||||||
|-
| ||||||1||identify group ‘file’||||||||OK||||||OK||||
|-
| ||||||2||click on browse||||||||OK||||||OK||||
|-
| ||||||3||click on upload||||||||OK||||||OK||||
|-
| ||||||4||enter file properties||||||||OK||||||OK||||
|-
| ||||||5||operate ‘add’||||||||OK||||||OK||||
|-
| ||||||6||repeat actions two times||||||||OK||||||OK||||
|-
| ||||||End||||||||||OK||||||OK||||
|-
|-
|<b>Date/Time</b>
| ||'''3'''||'''Save Locator/File'''||||||||||||||||||||||
|August 6/7
|-
|-
|<b>Status</b>
| ||||||1||Type URL in input field||||||||OK||||||OK||||
|Closed
|-
|-
|}
| ||||||2||Click upload link||||||||OK||||||OK||||
 
'''Interview Series II'''
 
{|style= color:black; background-color:#fff; cellpadding=5; cellspacing=0; border=0
|<b>Place</b>
|
*Harnack-Haus, Berlin: 3 Interviews
|-
|-
|<b>Date/Time</b>
| ||||||3||Choose content category||||||||OK||||||OK||||
|Mittwoch, 05. November 2008/09:00-12:00
|-
|-
|<b>Status</b>
| ||||||End||||||||||OK||||||OK||||
|Closed
|-
|-
|}
| ||2||[[PubMan_Func_Spec_Submission#UC_PM_SM_05_create_item_from_template|Use item as template]]||||||||||||||||||||||
 
===Interviews for FACES Release 3 (Prototype test)===
{|style= color:black; background-color:#fff; cellpadding=5; cellspacing=0; border=0
|<b>Place</b>
|
*Max Planck Institute for Human Development
|-
|-
|<b>Date/Time</b>
| ||||||1||go to workspace||||||||OK||Hard to find from Submission back||||OK||||
|[http://www.doodle.com/9zy24f297pkbzsaz To be scheduled]
|-
|-
|<b>Please note</b>
| ||||||2||click on title to enter item version view||||||||Serious||Did not realize the linked title||Procedure||OK||||
|We need an information about your status (expert 50 minutes per interview / non-expert 30 minutes) Please take a note behind your name.
|-
|-
|<b>Status</b>
| ||||||3||click on ‘use as template’||||||||||||||OK||||
|Planned
|-
|-
|}
| ||||||End||||||||||OK||||||OK||||
 
==Expert Interviews==
 
{|{{Prettytable}}
|-
|-
!bgcolor = #eee |Participants
| ||'''4'''||'''Add 2 organizations for an author'''||||||||||||||||||||||
|
1
|-
|-
!bgcolor = #eee |Supervisor
| ||||||1||||||||||OK||||||OK||||
|
1 - 2
|-
|-
!bgcolor = #eee |Equipment
| ||||||2||||||||||OK||||||OK||||
|
<ol>
<li>PC</li>
<li>Internet connection</li>
<li>Interview agenda (provided by UIE)</li>
<li>Audio recording device (provided by UIE)</li>
</ol>
|-
|-
!bgcolor = #eee |Duration
| ||||||3||||||||||OK||||||OK||||
|
2h - 4h
|-
|-
!bgcolor = #eee |Results
| ||||||End||||||||||OK||||||OK||||
|
Document with protocol of interview. Prototype draft.
|-
|-
!bgcolor = #eee |Steps
|  
|
<ol>
<li>Introduction</li>
<li>Q & A (Questions and Answers)</li>
<li>Summary</li>
</ol>
|}
|}


Expert interviews are conducted to get a better understanding of one specific working process and how it can be organized/reorganized within the user interface. A very domain specific knowledge is needed to understand interface requirements better. Functional aspects come additionally and can not be separated from interface needs.
==GUI Workshops==
 
[[Image:IMAG0044.jpg|right|275px|UIE-Process ]]
 
For GUI changes, enhancements or new GUIs interface drafts can be discussed and shaped together with potential users. The most problematic part is to get at least 6 people who are interested and skilled to take part in interface shaping directly.
 
Participants should know about basic interface controls and their mode of operation at the frontend. At the same time they need to have a clear understanding of available functionality.


===Expert Interviews 2007===
UIE Workshops can help clarify


*'''[[User Interface Evaluation/Expert Interviews/2007 Garching]]'''
*how users approach their tasks
*'''[[User Interface Evaluation/Expert Interviews/2007 Munich]]'''
*how they expect the interface to behave
*'''[[User Interface Evaluation/Expert Interviews/Easy Submission Summary]]'''
*employed metaphors


===Expert Interviews 2009===
Two kinds of workshops are applied if necessary:


*'''Max Planck Institute for Human Development (Planned)'''
#Card Sorting <ref>[http://dmt.fh-joanneum.at/projects/ebus2/seite.php?name=Card_Sorting FH JOANNEUM Gesellschaft mbH zu Testverfahren]</ref>
#Wireframes<ref>[http://en.wikipedia.org/wiki/Website_wireframe Wikipedia on wireframes], [http://wireframes.linowski.ca/ The Wireframes Magazine]</ref>


==Workshops==
'''Setup'''


{|{{Prettytable}}
{|{{Prettytable}}
Line 214: Line 231:
|}
|}


For each workshop the UIE team prepares a topic.


For each workshop the UIE team prepares a topic. The following topics are already covered:
==Software Based Evaluation==
 
===Workshops 2008===
*'''[[User Interface Evaluation/Adding Lists of Authors]], MPDL Berlin'''
*'''[[User Interface Evaluation/Easy Submission]], Nijmegen'''
 
===Workshops 2007===
*'''[[User Interface Evaluation/Short Item View]], MPDL Munich'''


==Software based Evaluation==
An open source heat map application has been installed to track user interactions along with their mouse actions. The results did not deliver the required quality of feedback and the solution was not extendible towards more dynamic interfaces. Software based evaluation is on hold since then.


Currently the pubman prototype is monitored by an open source application called [http://www.labsmedia.com/clickheat/index.html ClickHeat]. This solution generates pictures called heat maps. Heat maps show where users click (dots) and how often (colour).
==Cognitive Walkthroughs==


Example:
Cognitive walkthroughs are done collaterally according to all interface drafts and prototypes.


[[Image:clickheat_depositor.png|framed|center|Heat Map of Depositor Workspace]]
==References==
<references/>


Software based evaluation is not conducted for R3. Currently no open source application is available for analysing dynamic pages.
</div>
[[Category:Analysis & Evaluation]]
[[Category: User Interface Engineering]]
[[Category:User Interface Engineering]]

Latest revision as of 14:58, 17 May 2011


Detailed evaluation of the human machine interface is an effective method for designing a better user experience. Modern web applications are expected to deliver a high degree of usability. User participation within the development process is essential to meet those expectations. To balance the developer’s perspective, proven methods such as workshops and usability tests are applied to ensure the necessary degree of user participation in interface development.

Interfaces, built at the MPDL are subject to evaluation. Main focus are all interfaces of the current PubMan solution. Evaluation of the User Interface Engineering team has different approaches: Usability Interviews and tests, GUI Workshops and software based evaluation.

Usability Interviews (Thinking Aloud)[1][edit]

Interface releases are tested together with potential users from institutes of the Max Planck Society. They perform tasks covering important application functionality. An interviewer tracks feedback and observes user interactions. All issues are noted down. The test result is accumulated if at least 8-11 interviews are performed. A summary is given at the end of the interview series which leads to measures and changes in the interface.

The following table shows what is needed to conduct an interview at an institute.

Setup

Participants

1

Supervisor

1 plus 1 visitor (optionally, usually developers)

Equipment
  1. PC
  2. Internet connection
  3. Tasklist (provided by UIE)
  4. Test Form (provided by UIE)
  5. Recording Device (optionally)
Duration

45min

Results

Document with notes on user actions for each action/step.

The results are analysed after 8 - 11 interviews towards a statistic, providing GUI issues ordered by functional area and tasks.

Steps
  1. Short Introduction
  2. Participant solves task independently
  3. Open questions/diskussion

Usability interviews are conducted to discover, document and classify usability issues for

  • a specific release (e.g. PubMan 2.0.0.1)
  • a functional prototype[2]
  • a prototype draft

Up to 8 interviews are usually sufficient to discover the main flaws of a GUI. The interviewer just provides tasks for his participant. The participant does all tasks on his own and should be encouraged to comment on his actions. It is not recommended for the interviewer to interfere in any form. If the participant gets stuck and there are important steps to follow, he gets a hint how to continue.

Interviews can be recorded optionally if participants agree. If a participant is not able to solve a task or step it will be noted as a fatal usability issue. For later interviews with a more standardized set of tasks the issues are to be classified in the following way:

  1. Fatal (Task could not be finished successfully or in a proper way)
  2. Serious (Task could not be finished on the first attempt or user performance is bad)
  3. Minor (The user hesitates, is not sure or does not feel comfortable with the flow)

For each interview the UIE team provides an interview outcome. The document contains all tasks, observation notes and comments of participants. Additionally the interviews collect demographic characteristics as well to prepare data needed for personas.

Interview Form Example (Pullout only)[edit]

Submission Participant 01 Participant 02 Participant 03
Nr. Task Nr. Step Status Issue Comment (Observation) Type Status Issue Comment (Observation) Type Status Issue Comment (Observation) Type
1 Submit a publication item detailled
1 clicks submission in the Main menu OK OK No choosed it from startpage content OK
2 clicks on full submission OK Automatically chosen OK Easy Submission OK
3 clicks a collection (optional) OK Automatically chosen OK
4 identifies group basic OK
5 selects genre = journal article OK OK
6 fills title OK OK
7 identifies group person & org OK OK
8 enters author name, role OK OK
9 clicks on select or types manually Serious Did\'t use Affiliation input (mixed up with source) Procedure OK After automatic detection all authors must be entered
10 clicks an organization in tree (optional) Serious Used \"Help\" to get information (not helpful). Procedure
11 identifies group details OK OK
12 fills a date OK OK
13 identifies group source Serious Mixed up source data and publ. Data Procedure OK
14 fills genre and title Minor Was looking for \"Sternderl\" Position OK
14 clicks on the button ‘submit‘ Validation cycle! (Aff. Missing) OK
End OK OK
2 Upload 2 different files
1 identify group ‘file’ OK OK
2 click on browse OK OK
3 click on upload OK OK
4 enter file properties OK OK
5 operate ‘add’ OK OK
6 repeat actions two times OK OK
End OK OK
3 Save Locator/File
1 Type URL in input field OK OK
2 Click upload link OK OK
3 Choose content category OK OK
End OK OK
2 Use item as template
1 go to workspace OK Hard to find from Submission back OK
2 click on title to enter item version view Serious Did not realize the linked title Procedure OK
3 click on ‘use as template’ OK
End OK OK
4 Add 2 organizations for an author
1 OK OK
2 OK OK
3 OK OK
End OK OK

GUI Workshops[edit]

UIE-Process

For GUI changes, enhancements or new GUIs interface drafts can be discussed and shaped together with potential users. The most problematic part is to get at least 6 people who are interested and skilled to take part in interface shaping directly.

Participants should know about basic interface controls and their mode of operation at the frontend. At the same time they need to have a clear understanding of available functionality.

UIE Workshops can help clarify

  • how users approach their tasks
  • how they expect the interface to behave
  • employed metaphors

Two kinds of workshops are applied if necessary:

  1. Card Sorting [3]
  2. Wireframes[4]

Setup

Participants

5 - 8

Supervisor

1 - 2

Equipment

Equipment provided by UIE Team

Duration

4h - 6h

Result

A rough draft of a graphical user interface part.

Steps
  1. Introduction to a GUI task
  2. Participants work on paper prototypes
  3. Question and Answers
  4. Comparison of Results
  5. Clustering
  6. Rework if necessary
  7. Preparation plus dissamination of results

For each workshop the UIE team prepares a topic.

Software Based Evaluation[edit]

An open source heat map application has been installed to track user interactions along with their mouse actions. The results did not deliver the required quality of feedback and the solution was not extendible towards more dynamic interfaces. Software based evaluation is on hold since then.

Cognitive Walkthroughs[edit]

Cognitive walkthroughs are done collaterally according to all interface drafts and prototypes.

References[edit]

  1. ↑ Wikipedia - Comparison of Usability Methods
  2. ↑ A functional prototype (Model) (also called a working prototype) will, to the greatest extent practical, attempt to simulate the final design, aesthetics, materials and functionality of the intended design. The functional prototype may be reduced in size (scaled down) in order to reduce costs. The construction of a fully working full-scale prototype and the ultimate test of concept, is the engineers' final check for design flaws and allows last-minute improvements to be made before larger production runs are ordered. Definition retrieved from Wikipedia article on Prototypes
  3. ↑ FH JOANNEUM Gesellschaft mbH zu Testverfahren
  4. ↑ Wikipedia on wireframes, The Wireframes Magazine

Max Planck Digital Library

Publication Management

Graphical User Interface

User Interface Engineering

Uniform Resource Locator

Retrieved from "https://colab.mpdl.mpg.de/mw010/index.php?title=User_Interface_Evaluation&oldid=62876"