Paper Reading #20: The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only ANOVA Procedures

Reference

Authors and Affiliations:

Leah Findlater,Jacob O. Wobbrock ,The Information School DUB Group University of Washington Seattle, WA 98195 USA

Darren Gergle, School of Communication Northwestern University Evanston, IL 60208 USA

James J. Higgins,Department of Statistics Kansas State University Manhattan, KS 66506 USA

Presentation: CHI 2011, May 7–12, 2011, Vancouver, BC, Canada.

Summary

Hypothesis:

The paper present the Aligned Rank Transform (ART) for nonparametric factorial data analysis in HCI. They propose a preprocessing step that “aligns” data before applying averaged ranks, after which point, common ANOVA procedures can be used, making the ART accessible to anyone familiar with the F-test. They hypothesize that

researchers only familiar with ANOVA can use, interpret, and report results from the ART.

Contents

Conover and Iman’s Rank Transform (RT) uses the parametric F-test on the ranks, resulting in a nonparametric factorial procedure producing inaccurate results for interaction effects making the RT method unsuitable for factorial designs.ART corrects this problem, providing accurate nonparametric treatment for both main and interaction effects. ART users have two opportunities for ensuring correctness. First, every column of aligned responses Y′ should sum to zero; ARTool and ARTweb verify this for the user. Second, a full-factorial ANOVA performed on the aligned (not ranked) responses Y′ should show all effects stripped out (F=0.00, p=1.00) except for the effect for which the data were aligned.

Methods

1)Compute residuals :For each raw response Y, compute its residual as

residual = Y – cell mean

2)Compute estimated effects for all main and interaction effects.

3)Compute aligned response Y′.

The calculation is

Y′ = residual + estimated effect, i.e.

result from Step 1 + result from Step 2.

4)Assign averaged ranks to a column of aligned observations Y′ to create Y′′. The smallest Y′ receives rank 1, the next smallest Y′ receives rank 2, and so on until the largest of r values receives rank r. In the case of a tie among k values, the average rank is the sum of ranks divided by k.

5) Perform a full-factorial ANOVA on Y′′ but the result corresponding to the effect for which Y was aligned as Y′ should be considered.

Results

The use of ART for analyzing the data in three different research in HCI in past show better results in analysis of data. ART reports nonsignificant main effects for Accuracy (F1,22=0.01, p=.920) and Interface (F2,44=0.65, p=.529) and a significant interaction (F2,44=4.12, p=.023) for Findlater research in 2009.

In MacKenzie & Zhang research with the ART, the main effects of Session (F19,76=5.19, p<.001) and Method (F1,4=10.64, p=.031) are still significant, and now their interaction is also significant (F19,76=1.85, p=.032). In Wobbrock research, ART reduces the skew in the data, gives the same significant main effects, and gives both significant Recognizer*No. Train (F2,757=42.49, p<.001) and Recognizer*Speed (F4,757=4.37, p=.002) interactions.

Discussion

The research provides a new dimension in analysis of data in HCI. I was really impressed with the ease of use of the web and desktop application and how it compared with other techniques that have been used till now. I can see a lot of applications of this tool not only in HCI but in other sectors as well where non parametric data needs to be analyzed. I will surely try this tool if needed in future while doing research or any other tasks of analyzing non parametric data.