Visual-Semantic Graph Attention Networks for Human-Object Interaction Detection

Abstract

In scene understanding, robots benefit from not only detecting individual scene instances but also from learning their possible interactions. Human-Object Interaction (HOI) Detection infers the action predicate on a <human, predicate, object> triplet. Contextual information has been found critical in inferring interactions. However, most works only use local features from single human-object pairs for inference. Few works have studied the disambiguating contribution of subsidiary relations made available via graph networks. Similarly, few have leveraged visual cues with the intrinsic semantic regularities embedded in HOIs. We contribute Visual-Semantic Graph Attention Networks (VS-GATs): a dual-graph attention network that effectively aggregates visual, spatial, and semantic contextual information dynamically from primary human-object relations as well as subsidiary relations through attention mechanisms for strong disambiguating power. We achieve competitive results on two benchmarks: V-COCO and HICO-DET. The code is available at https://github.com/birlrobotics/vs-gats.