Abstract
This paper revisits human-object interaction (HOI) recognition at image level
without using supervisions of object location and human pose. We name it
detection-free HOI recognition, in contrast to the existing
detection-supervised approaches which rely on object and keypoint detections to
achieve state of the art. With our method, not only the detection supervision
is evitable, but superior performance can be achieved by properly using
image-text pre-training (such as CLIP) and the proposed Log-Sum-Exp Sign
(LSE-Sign) loss function. Specifically, using text embeddings of class labels
to initialize the linear classifier is essential for leveraging the CLIP
pre-trained image encoder. In addition, LSE-Sign loss facilitates learning from
multiple labels on an imbalanced dataset by normalizing gradients over all
classes in a softmax format. Surprisingly, our detection-free solution achieves
60.5 mAP on the HICO dataset, outperforming the detection-supervised state of
the art by 13.4 mAP