Paper Reading #32: Taking advice from intelligent systems: the double-edged sword of explanations

Reference

Authors: Kate Ehrlich, Susanna Kirk, John Patterson, Jamie Rasmussen, Steven Ross, Daniel Gruen

Affiliation: IBM Research, 1 Rogers St, Cambridge, MA 02142

Presentation: IUI 2011, February 13–16, 2011, Palo Alto, California, USA.

Summary

Hypothesis

Research on intelligent systems has emphasized the benefits of providing explanations along with recommendations. But can explanations lead users to make incorrect decisions? This paper explored this question in a controlled experimental study with 18 professional network security analysts doing an incident classification task using a prototype cyber security system.It further addresses the role of explanations in the context of correct and incorrect recommendations, in a mission- critical setting.

Methods

Nineteen analysts with minimum of three years experience participated in the study on the effects of a user's response to a recommendation made by an intelligent system.Following the training, analysts completed 24 timed trials where they were instructed to complete each trial within two minutes and to give their best guess if they ran out of time.The analyst’s task was to categorize and prioritize the alert. They completed this task by first selecting an Issue Type from a pull-down list of 11 items and then selecting a priority from a list of 2 items (Low, Medium). There was a focus group after the study and during the study the participants were asked to make additional comments and observations. NIMBLE software was used for the study.

Results

The performance of users was slightly better with a correct recommendation than without one. Results indicated that justifications grant benefits to users when a correct response is available. When there is no correct response available, neither suggestions nor justifications made a difference in performance. There was a common trend among analysts that they seemed to discard the recommendations anyway, relying on their own inclinations. In the separate study concerning analyzing users' reactions, it was found that users typically follow the recommendations given and that the influence between the recommendation and the user's action is high. Analysts who valued the recommendations the most were also more likely to follow them, which was to their advantage when there was a correct recommendation but to their disadvantage when there was no correct recommendation. Analysts strongly valued self- sufficiency, independent analysis, and individual judgment while responding to the recommendations.

Contents

This paper addresses the role of explanations in the context of correct and incorrect recommendations, in a mission- critical setting. The issue was whether explanations of system reasoning made it easier to detect erroneous recommendations—by, for instance, letting users discover flaws in the system’s reasoning—or whether explanations made both correct and erroneous recommendations appear more plausible. To address these issues the authors conducted a controlled empirical study in which they systematically varied the accuracy of available recommendations and the presence of explanations. The study was carried out with network security analysts who needed to respond quickly and accurately to alerts generated by intrusion detection systems.

Discussion

As per the results, it seems like the authors did a pretty good job in figuring out the accuracy of recommendations and the relationship between accuracy and the influence of those recommendations on users' actions. I feel like there are different external factors that might affect the responses from the analysts, like how many hours they have been working and what was the intensity of the work. This research can be a foundation for designing better intelligent systems in future but it is limited for general use or for applications on day to day gadgets.