Publications and deliverables

Deliverable 6.1

Title:

Evaluation and exploitation report

Due date:

2021-05-31

Executive summary:

The aim of this document is to explicate the plan of the evaluation of the HumBAS Feedback Tool and the process of evaluation at several stages of the HumBAS project. After a brief presentation of the HumBAS Feedback Tool, the methodology of the evaluation is described. The evaluation process in WP6 was established at different stages of the project: during the design and development of the components and interaction elements of the HumBAS Feedback Tool; during the deployment of the final HumBAS Feedback Tool at the user sides. During the design and development of the HumBAS Feedback Tool, the evaluation has been carried out in several steps: Step 1 - Individual Interviews; Step 2 - Semiotic Analysis of Icons; Step 3 - Exploring Types of Interaction including the characteristics of the interaction form and interaction setting, as well as the choice of interaction language; Step 4 - Exploring Types of System Responses. Based on the data gathered during the evaluation process, we can summarize this part of the evaluation as follows: Some symbols suggested were clear to the users because these were standard symbols used in several settings. The users were familiar with such symbols. In case of new designed symbols we suggested they were partly confused, but also partly clear after trying to understand a possible meaning of the symbol in context during the evaluation process. Some of the users were very detailed in their reaction to the symbols and also compared them among each other to cover all possible settings. Their feedback helped us to redesign the symbols. That means we made small adaptations or additions to the current evaluated version of the symbol. This is also shown in the tables in case we had a new version of the same symbol. A challenging design issue was for us to use symbols to implicitly communicate their intention / affordance in a use setting: Shall it show the perception of the user or shall it tell to the system what to change and what to adjust to positively improve the perception of the user? This fact was considered centrally in the further design of the HumBAS Feedback Tool. In WP6 we deployed the final HumBAS Feedback Tool that we have evaluated summative. For this, a one-week diary study with users have been conducted. The central idea was to integrate the feedback tool developed and built in the course of the project, including a streamlined version of the backend system, into the test persons’ working environment. The aim was to create a setting as close to reality as possible, with the most significant trade-off being the effects of the feedback on the working environment. Unfortunately, we did not have a test facility available to connect the HumBAS system to a BAS control software. Thus, the feedback provided by the test subjects did not affect indoor conditions. Nevertheless, important information about the design of the interaction flows, the systems’ responses, and the general configuration could be obtained through these evaluations. The central questions of this evaluation were as follows: To what extent were the designed input and output mechanisms able to adequately communicate the interaction possibilities between the users and the system? To what extent did the forms of feedback provided in the design reflect the real needs of the users? How had the overall system to be configured so that it provided the intended functionality but at the same time did not interfere too much with the user’s work rhythm? The functionality and interaction design of the HumBAS Feedback Tool were satisfactory for the end user. The use was easy, from the beginning of the evaluation process, and clear in most cases. Sometimes, there were ambiguous reactions of the tool, at least, it was not clear to the user why the reaction of the system was in a certain way. Learnability and adaptability were well provided by the tool. The main positive effect is that the tool helps increase the people’s awareness of the room and air situation in an office environment and provide new and innovative ways of dealing with it, especially from the users’ point of view. We know that the evaluation was limited, also due to the Covid-19 situation and restrictions in this period, but it can be seen as a qualitative one to consider for the further improvement and development of the HumBAS Feedback Tool. This document is closely related to other tasks (in other WPs like in WP3) and deliverables of the project. On the one hand, it uses the prototype created and documented in deliverable D3.3, which is the final design of the user feedback tool. It is based on the technical specifications finalized in the D3.2. Due to the tight connection to the WP3, it is advisable to become familiar with the contents of D3.3.