Modeling visual search in interactive graphic interfaces: Adding visual pattern matching algorithms to ACT-R

Farnaz Tehranchi, Frank E. Ritter

Research output: Chapter in Book/Report/Conference proceedingConference contribution

5 Scopus citations

Abstract

We provide an update on JSegMan, an interactive system to extend the ACT-R cognitive architecture to interact with dynamic interfaces based on the screen contents and generating input for the operating system directly. Current ACT-R models typically interact with the world through ACT-R's device interface?an abstract representation of the world that is based on a simulated Lisp environment provided with ACT-R, or by instrumenting interfaces. In JSegMan, computer vision pattern matching algorithms and visual patterns extend the ACT-R cognitive architecture. With JSegMan, models directly move the cursor on the screen, click on application GUI objects on PCs, and type through the use of existing Java libraries. Implementing users' visual search strategies and input abilities for different visual objects enables the detailed modeling of interactive tasks on any interface. The visual pattern matching algorithms serve two goals: to simulate user behavior in interactive tasks and to create representations of visual stimuli. We tested our visual pattern matching approach by using it with an existing model for a long spreadsheet task. We found that the revised model more accurately predicted a 20-min task by entirely performing the task on an uninstrumented and unmodified interface.

Original languageEnglish (US)
Title of host publicationProceedings of ICCM 2018 - 16th International Conference on Cognitive Modeling
EditorsIon Juvina, Joseph Houpt, Christopher Myers
PublisherUniversity of Wisconsin
Pages162-167
Number of pages6
ISBN (Electronic)9780998508221
StatePublished - 2018
Event16th International Conference on Cognitive Modeling, ICCM 2018 - Wisconsin, United States
Duration: Jul 21 2018Jul 24 2018

Publication series

NameProceedings of ICCM 2018 - 16th International Conference on Cognitive Modeling

Conference

Conference16th International Conference on Cognitive Modeling, ICCM 2018
CountryUnited States
CityWisconsin
Period7/21/187/24/18

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Modeling and Simulation

Fingerprint Dive into the research topics of 'Modeling visual search in interactive graphic interfaces: Adding visual pattern matching algorithms to ACT-R'. Together they form a unique fingerprint.

Cite this