SGPT: The Secondary Path Guides the Primary Path in Transformers for HOI Detection

Abstract

HOI detection is essential for human-computer interaction, especially in behavior detection and robot manipulation. Existing mainstream transformer methods of HOI detection are focused on single-stream detection only, e.g., <tex>$image \rightarrow HOI(\mathcal{P}_{1})$</tex>, or <tex>$image \rightarrow HO\rightarrow I(\mathcal{P}_{2})$</tex>. Both paths have their own characteristics of concern, so we propose a novel method, using the Secondary path <tex>$(\mathcal{P}_{2})$</tex> Guides the Primary path <tex>$(\mathcal{P}_{1})$</tex> in Transformers (SGPT). SGPT contains two core modules: the Dual-Path Consistency (DPC) module and the Instance Interaction Attention (IIA) module. DPC keeps human, object and interaction consistent on the dual-path and lets <tex>$\mathcal{P}_{2}$</tex> guide <tex>$\mathcal{P}_{1}$</tex> to learn more meaningful features. IIA fuses human and object to enhance interaction in <tex>$\mathcal{P}_{2}$</tex>, which allows instance to constrain interaction. Our proposed dual-path are employed during training, and only the <tex>$\mathcal{P}_{1}$</tex> path is used for inference. Hence, SGPT improves generalization without increasing model capacity in HICO-DET and V-COCO datasets compared to the state-of-the-arts. The code of this work is available at https://github.com/visualVk/sgpt.git.