Publications

Several publications have come out of the work we have been doing on the project.

Major papers:

  • Gould, S. J. J., Cox, A. L., & Brumby, D. P.. (2016). Diminished Control in Crowdsourcing: An Investigation of Crowdworker Multitasking Behavior. Acm trans. comput.-hum. interact., 23(3), 19:1–19:29.
    doi:10.1145/2928269
    [BibTeX] [Abstract] [Download PDF]

    Obtaining high-quality data from crowds can be difficult if contributors do not give tasks sufficient attention. Attention checks are often used to mitigate this problem, but, because the roots of inattention are poorly understood, checks often compel attentive contributors to complete unnecessary work. We investigated a potential source of inattentiveness during crowdwork: multitasking. We found that workers switched to other tasks every 5 minutes, on average. There were indications that increasing switch frequency negatively affected performance. To address this, we tested an intervention that encouraged workers to stay focused on our task after multitasking was detected. We found that our intervention reduced the frequency of task switching. It also improves on existing attention checks because it does not place additional demands on workers who are already focused. Our approach shows that crowds can help to overcome some of the limitations of laboratory studies by affording access to naturalistic multitasking behavior.

    @article{gould_diminished_2016,
    title = {Diminished {Control} in {Crowdsourcing}: {An} {Investigation} of {Crowdworker} {Multitasking} {Behavior}},
    volume = {23},
    issn = {1073-0516},
    shorttitle = {Diminished {Control} in {Crowdsourcing}},
    url = {https://dl.acm.org/authorize?N16948},
    doi = {10.1145/2928269},
    abstract = {Obtaining high-quality data from crowds can be difficult if contributors do not give tasks sufficient attention. Attention checks are often used to mitigate this problem, but, because the roots of inattention are poorly understood, checks often compel attentive contributors to complete unnecessary work. We investigated a potential source of inattentiveness during crowdwork: multitasking. We found that workers switched to other tasks every 5 minutes, on average. There were indications that increasing switch frequency negatively affected performance. To address this, we tested an intervention that encouraged workers to stay focused on our task after multitasking was detected. We found that our intervention reduced the frequency of task switching. It also improves on existing attention checks because it does not place additional demands on workers who are already focused. Our approach shows that crowds can help to overcome some of the limitations of laboratory studies by affording access to naturalistic multitasking behavior.},
    number = {3},
    urldate = {2016-07-04},
    journal = {ACM Trans. Comput.-Hum. Interact.},
    author = {Gould, Sandy J. J. and Cox, Anna L. and Brumby, Duncan P.},
    month = jun,
    year = {2016},
    keywords = {crowdsourcing, cuing, data entry, Human performance, interruptions, Methodology, multitasking, online experimentation, transcription},
    pages = {19:1--19:29},
    }

  • Gould, S. J. J., Cox, A. L., Brumby, D. P., & Wickersham, A.. (2016). Now Check Your Input: Brief Task Lockouts Encourage Checking, Longer Lockouts Encourage Task Switching. Paper presented at the Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, New York, NY, USA.
    doi:10.1145/2858036.2858067
    [BibTeX] [Abstract] [Download PDF]

    Data-entry is a common activity that is usually performed accurately. When errors do occur though, people are poor at spotting them even if they are told to check their input. We considered whether making people pause for a brief moment before confirming their input would make them more likely to check it. We ran a lab experiment to test this idea. We found that task lockouts encouraged checking. Longer lockout durations made checking more likely. We ran a second experiment on a crowdsourcing platform to find out whether lockouts would still be effective in a less controlled setting. We discovered that longer lockouts induced workers to switch to other activities. This made the lockouts less effective. To be useful in practice, the duration of lockouts needs to be carefully calibrated. If lockouts are too brief they will not encourage checking. If they are too long they will induce switching.

    @inproceedings{gould_now_2016,
    address = {New York, NY, USA},
    series = {{CHI} '16},
    title = {Now {Check} {Your} {Input}: {Brief} {Task} {Lockouts} {Encourage} {Checking}, {Longer} {Lockouts} {Encourage} {Task} {Switching}},
    url = {https://dl.acm.org/authorize?N15399},
    doi = {10.1145/2858036.2858067},
    abstract = {Data-entry is a common activity that is usually performed accurately. When errors do occur though, people are poor at spotting them even if they are told to check their input. We considered whether making people pause for a brief moment before confirming their input would make them more likely to check it. We ran a lab experiment to test this idea. We found that task lockouts encouraged checking. Longer lockout durations made checking more likely. We ran a second experiment on a crowdsourcing platform to find out whether lockouts would still be effective in a less controlled setting. We discovered that longer lockouts induced workers to switch to other activities. This made the lockouts less effective. To be useful in practice, the duration of lockouts needs to be carefully calibrated. If lockouts are too brief they will not encourage checking. If they are too long they will induce switching.},
    booktitle = {Proceedings of the 2016 {CHI} {Conference} on {Human} {Factors} in {Computing} {Systems}},
    publisher = {ACM},
    author = {Gould, Sandy J. J. and Cox, Anna L. and Brumby, Duncan P. and Wickersham, Alice},
    year = {2016},
    pages = {3311--3323},
    }

Other papers:

  • Gould, S. J. J., Cox, A. L., Brumby, D. P., & Wiseman, S.. (2015). Home is Where the Lab is: A Comparison of Online and Lab Data From a Time-sensitive Study of Interruption. Human computation, 2(1), 45-67.
    doi:10.15346/hc.v2i1.4
    [BibTeX] [Abstract] [Download PDF]

    While experiments have been run online for some time with positive results, there are still outstanding questions about the kinds of tasks that can be successfully deployed to remotely situated online participants. Some tasks, such as menu selection, have worked well but these do not represent the gamut of tasks that interest HCI researchers. In particular, we wondered whether long-lasting, time-sensitive tasks that require continuous concentration could work successfully online, given the confounding effects that might accompany the online deployment of such a task. We ran an archetypal interruption experiment both online and in the lab to investigate whether studies demonstrating such characteristics might be more vulnerable to a loss of control than the short, time-insensitive studies that are representative of the majority of previous online studies. Statistical comparisons showed no significant differences in performance on a number of dimensions. However, there were issues with data quality that stemmed from participants misunderstanding the task. Our findings suggest that long-lasting experiments using time-sensitive performance measures can be run online but that care must be taken when introducing participants to experimental procedures.

    @article{gould_home_2015,
    title = {Home is {Where} the {Lab} is: {A} {Comparison} of {Online} and {Lab} {Data} {From} a {Time}-sensitive {Study} of {Interruption}},
    volume = {2},
    issn = {2330-8001},
    shorttitle = {Home is {Where} the {Lab} is},
    url = {http://hcjournal.org/ojs/index.php?journal=jhc&page=article&op=view&path%5B%5D=40&path%5B%5D=54},
    doi = {10.15346/hc.v2i1.4},
    abstract = {While experiments have been run online for some time with positive results, there are still outstanding questions about the kinds of tasks that can be successfully deployed to remotely situated online participants. Some tasks, such as menu selection, have worked well but these do not represent the gamut of tasks that interest HCI researchers. In particular, we wondered whether long-lasting, time-sensitive tasks that require continuous concentration could work successfully online, given the confounding effects that might accompany the online deployment of such a task. We ran an archetypal interruption experiment both online and in the lab to investigate whether studies demonstrating such characteristics might be more vulnerable to a loss of control than the short, time-insensitive studies that are representative of the majority of previous online studies. Statistical comparisons showed no significant differences in performance on a number of dimensions. However, there were issues with data quality that stemmed from participants misunderstanding the task. Our findings suggest that long-lasting experiments using time-sensitive performance measures can be run online but that care must be taken when introducing participants to experimental procedures.},
    language = {en},
    number = {1},
    urldate = {2015-08-12},
    journal = {Human Computation},
    author = {Gould, Sandy J. J. and Cox, Anna L. and Brumby, Duncan P. and Wiseman, Sarah},
    month = aug,
    year = {2015},
    pages = {45--67},
    }

  • Gould, S. J. J., Cox, A. L., & Brumby, D. P.. (2015). Task Lockouts Induce Crowdworkers to Switch to Other Activities. Paper presented at the Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, New York, NY, USA.
    doi:10.1145/2702613.2732709
    [BibTeX] [Abstract] [Download PDF]

    Paid crowdsourcing has established itself as a useful way of getting work done. The availability of large, responsive pools of workers means that low quality work can often be treated as noise and dealt with through standard data processing techniques. This approach is not practical in all scenarios though, so efforts have been made to stop poor performance occurring by preventing satisficing behaviours that can compromise result quality. In this paper we test an intervention – a task lockout – designed to prevent satisficing behaviour in a simple data-entry task on Amazon Mechanical Turk. Our results show that workers are highly adaptable: when faced with the intervention they develop workaround strategies, allocating their time elsewhere during lockout periods. This suggests that more subtle techniques may be required to substantially influence worker behaviour.

    @inproceedings{gould_task_2015,
    address = {New York, NY, USA},
    series = {{CHI} {EA} '15},
    title = {Task {Lockouts} {Induce} {Crowdworkers} to {Switch} to {Other} {Activities}},
    isbn = {978-1-4503-3146-3},
    url = {https://dl.acm.org/authorize?N15390},
    doi = {10.1145/2702613.2732709},
    abstract = {Paid crowdsourcing has established itself as a useful way of getting work done. The availability of large, responsive pools of workers means that low quality work can often be treated as noise and dealt with through standard data processing techniques. This approach is not practical in all scenarios though, so efforts have been made to stop poor performance occurring by preventing satisficing behaviours that can compromise result quality. In this paper we test an intervention – a task lockout – designed to prevent satisficing behaviour in a simple data-entry task on Amazon Mechanical Turk. Our results show that workers are highly adaptable: when faced with the intervention they develop workaround strategies, allocating their time elsewhere during lockout periods. This suggests that more subtle techniques may be required to substantially influence worker behaviour.},
    urldate = {2015-05-21},
    booktitle = {Proceedings of the 33rd {Annual} {ACM} {Conference} {Extended} {Abstracts} on {Human} {Factors} in {Computing} {Systems}},
    publisher = {ACM},
    author = {Gould, Sandy J. J. and Cox, Anna L. and Brumby, Duncan P.},
    year = {2015},
    keywords = {crowdworking, data quality, interface lockouts, interruptions, task switching},
    pages = {1785--1790},
    }

Workshop participation:

At CHI ’16 we participated in the “Productivity Decomposed: Getting Big Things Done with Little Microtasks” workshop. This was our submission:

  • Gould, S. J. J., Cox, A. L., & Brumby, D. P.. (2016). Multitasking, Activity Management and Task Decomposition in the Atomic Age. Paper presented at the Productivity Decomposed: Getting Big Things Done with Little Microtasks (CHI 2016 Workshop).
    [BibTeX] [Download PDF]
    @inproceedings{gould_multitasking_2016,
    Author = {Gould, Sandy J. J. and Cox, Anna L. and Brumby, Duncan P.},
    Booktitle = {Productivity {Decomposed}: {Getting} {Big} {Things} {Done} with {Little} {Microtasks} ({CHI} 2016 {Workshop})},
    File = {Gould et al. - Multitasking, Activity Management and Task Decompo.pdf:/Users/sandy/Library/Application Support/Firefox/Profiles/grjnxbn7.default/zotero/storage/KRQVGV88/Gould et al. - Multitasking, Activity Management and Task Decompo.pdf:application/pdf},
    Title = {Multitasking, {Activity} {Management} and {Task} {Decomposition} in the {Atomic} {Age}},
    Year = {2016},
    url = {http://research.microsoft.com/en-us/um/people/teevan/misc/microproductivity/position/gould-microproductivity.pdf}
    }