Procedural Activity Understanding in Videos: An Execution-Centric View

Survey · Preprint 2026

Guodong Ding^1*, Dibyadip Chatterjee^1*, Zhanzhong Pang^1*, Anna Kukleva^2*, Yale Song³, Angela Yao¹ and Fadime Sener²

¹National University of Singapore   ²Meta   ³Google  ·  * equal contribution

Abstract

Procedural activity understanding (PAU) aims to interpret goal-directed tasks as structured sequences of steps. As a central problem in video understanding, it spans a broad range of tasks that have largely been studied in isolation. Despite the rapid growth of these techniques, and the recent shift brought by foundation models, no systematic survey has examined them together. This survey reviews the field from an execution-centric perspective. We distinguish a procedure, which specifies how a task should be performed, from an execution, which captures how it is actually carried out in a particular video. Based on this distinction, we organize existing tasks by the question each answers about an execution: what is happening, what comes next, how it is going, and what to convey. For each category, we review the representative methods and evaluation protocols. In addition, we analyze trends across tasks and show that, while foundation models have largely advanced tasks driven by knowledge of the procedure, those requiring fine-grained perception of the execution remain challenging. We conclude by identifying open challenges toward procedural AI assistants. Finally, we have curated a list of PAU resources, Awesome-PAU.

Citation

@article{ding2026procedural,
  title   = {Procedural Activity Understanding in Videos: An Execution-Centric View},
  author  = {Ding, Guodong and Chatterjee, Dibyadip and Pang, Zhanzhong and Kukleva, Anna and Song, Yale and Yao, Angela and Sener, Fadime},
  journal = {Preprint},
  year    = {2026}
}