User Interface Evaluation

APPLICATION AREAS
Home Interface Conception & Design User Interface Evaluation Interface Development
PROJECTS
Pattern Library Publication Management Digitization Lifecycle Research- and Metadata Handling Usability Report Corporate & Interface Design (under Rework) Page Index
edit

Interfaces, built at the MPDL are subject to evaluation. Main focus are all interfaces of the current PubMan solution. Evaluation of the User Interface Engineering team has different approaches: Usability Interviews and tests, GUI Workshops and software based evaluation.

Usability Interviews (Thinking Aloud)

Interface releases are tested together with potential users from institutes of the Max Planck Society. They perform tasks covering important application functionality. An interviewer tracks feedback and observes user interactions. All issues are noted down. The test result is accumulated if at least 8-11 interviews are performed. A summary is given at the end of the interview series which leads to measures and changes in the interface.

The following table shows what is needed to conduct an interview at an istitute:

Participants	1
Supervisor	1 plus 1 visitor (optionally, usually developers)
Equipment	PC Internet connection Tasklist (provided by UIE) Test Form (provided by UIE) Recording Device (optionally)
Duration	1h
Results	Document with notes on user actions for each action/step. The results are analysed after 6 - 8 interviews towards a statistic, providing GUI issues ordered by functional area and tasks.
Steps	Short Introduction Participant solves task independently Open questions/diskussion

Usability interviews are conducted to discover, document and classify usability issues for

• a specific release (e.g. PubMan 2.0.0.1)
• a functional prototype
• a prototype draft

Up to 8 interviews are usually sufficient to discover the main flaws of a GUI. The interviewer just provides tasks for his participant. The participant does all tasks on his own and should be encouraged to comment on his actions. It is not recommended for the interviewer to interfere in any form. If the participant gets stuck and there are important steps to follow, he gets a hint how to continue.

Interviews can be recorded optionally if participants agree. If a participant is not able to solve a task or step it will be noted as a fatal usability issue. For later interviews with a more standardized set of tasks the issues are to be classified in the following way:

1. Fatal (Task could not be finished successfully or in a proper way)
2. Serious (Task could not be finished on the first attempt or user performance is bad)
3. Minor = (The user hesitates, is not sure or does not feel comfortable with the flow)

For each interview the UIE team provides an interview outcome. The document contains all tasks, observation notes and comments of participants. Additionally the interviews collect demographic characteristics as well to prepare data needed for personas. After a valid number of interviews exists they are analysed and summarized.

Interview Form Example

Submission

Participant 01

Participant 02

Participant 03

Nr.

Task

Nr.

Step

Status

Issue Comment (Observation)

Type

Status

Issue Comment (Observation)

Type

Status

Issue Comment (Observation)

Type

1

Submit a publication item detailled

1

clicks submission in the Main menu

OK

OK

No choosed it from startpage content

OK

2

clicks on full submission

OK

Automatically chosen

OK

Easy Submission

OK

3

clicks a collection (optional)

OK

Automatically chosen

OK

4

identifies group basic

OK

5

selects genre = journal article

OK

OK

6

fills title

OK

OK

7

identifies group person & org

OK

OK

8

enters author name, role

OK

OK

9

clicks on select or types manually

Serious

Did\'t use Affiliation input (mixed up with source)

Procedure

OK

After automatic detection all authors must be entered

10

clicks an organization in tree (optional)

Serious

Used \"Help\" to get information (not helpful).

Procedure

11

identifies group details

OK

OK

12

fills a date

OK

OK

13

identifies group source

Serious

Mixed up source data and publ. Data

Procedure

OK

14

fills genre and title

Minor

Was looking for \"Sternderl\"

Position

OK

14

clicks on the button ‘submit‘

Validation cycle! (Aff. Missing)

OK

End

OK

OK

2

Upload 2 different files

1

identify group ‘file’

OK

OK

2

click on browse

OK

OK

3

click on upload

OK

OK

4

enter file properties

OK

OK

5

operate ‘add’

OK

OK

6

repeat actions two times

OK

OK

End

OK

OK

3

Save Locator/File

1

Type URL in input field

OK

OK

2

Click upload link

OK

OK

3

Choose content category

OK

OK

End

OK

OK

2

Use item as template

1

go to workspace

OK

Hard to find from Submission back

OK

2

click on title to enter item version view

Serious

Did not realize the linked title

Procedure

OK

3

click on ‘use as template’

OK

End

OK

OK

4

Add 2 organizations for an author

1

OK

OK

2

OK

OK

3

OK

OK

End

OK

OK

Interviews for PubMan Release 2

Interview series I

• Munich: 2 Interviews

Interview series II

• Nijmegen: 9 Interviews

Interview Analysis

Interviews for PubMan Release 3

Interview series I

Place	Max Planck Institute of Molecular Plant Physiology, Berlin: 2 Interviews Max Planck Institute for Gravitational Physics (Albert Einstein Institute), Golm 2 Interviews Max Planck Institute for Human Development, Berlin 2 Interviews
Date/Time	August 6/7
Status	Closed

Interview Series II

Place	Harnack-Haus, Berlin: 3 Interviews
Date/Time	Mittwoch, 05. November 2008/09:00-12:00
Status	Closed

R3 Interview Analysis

Interviews for PubMan Release 4

Interview series I

Place	Max Planck Digital Library, Munich: 4 Interviews
Date/Time	April - June
Status	Open

Interview Series II

Place	Max Planck Institute for Foreign and International Social Law, Munich: 2 Interviews
Date/Time	April - June
Status	Open

R4 Interview Analysis

Interviews for FACES Release 3 (Prototype test)

Place	Max Planck Institute for Human Development
Date/Time	To be scheduled
Please note	We need an information about your status (expert/non-expert) Please take a note behind your name.
Status	Planned

Expert Interviews

Participants	1
Supervisor	1 - 2
Equipment	PC Internet connection Interview agenda (provided by UIE) Audio recording device (provided by UIE)
Duration	2h - 4h
Results	Document with protocol of interview. Prototype draft.
Steps	Introduction Q & A (Questions and Answers) Summary

Expert interviews are conducted to get a better understanding of one specific working process and how it can be organized/reorganized within the user interface. A very domain specific knowledge is needed to understand interface requirements better. Functional aspects come additionally and can not be separated from interface needs.

Expert Interviews 2007

• User Interface Evaluation/Expert Interviews/2007 Garching
• User Interface Evaluation/Expert Interviews/2007 Munich
• User Interface Evaluation/Expert Interviews/Easy Submission Summary

Expert Interviews 2009

• Max Planck Institute for Human Development (Planned)

Workshops

Participants	5 - 8
Supervisor	1 - 2
Equipment	Equipment provided by UIE Team
Duration	4h - 6h
Result	A rough draft of a graphical user interface part.
Steps	Introduction to a GUI task Participants work on paper prototypes Question and Answers Comparison of Results Clustering Rework if necessary Preparation plus dissamination of results

For each workshop the UIE team prepares a topic. The following topics are already covered:

Workshops 2008

• Scheduled: 09.12.2008 - 10.12.2008, Bibliotheca Hertziana - Max Planck Institute for Art History

Topics: A brainstorming can be found on Faces GUI.

• User Interface Evaluation/Adding Lists of Authors, MPDL Berlin
• User Interface Evaluation/Easy Submission, Nijmegen

Workshops 2007

• User Interface Evaluation/Short Item View, MPDL Munich

Software based Evaluation

Currently the pubman prototype is monitored by an open source application called ClickHeat. This solution generates pictures called heat maps. Heat maps show where users click (dots) and how often (colour).

Example:

Heat Map of Depositor Workspace

Software based evaluation is not conducted for R3. Currently no open source application is available for analysing dynamic pages.

Max Planck Digital Library

Publication Management

Graphical User Interface

User Interface Engineering

Uniform Resource Locator