Efficient search with multi-modality for video commercial retrieval

Efficient and robust retrieval of commercial videos is an important topic for many applications such as commercial monitoring, market investigation. In this paper, we propose a two-step scheme to optimally incorporate the information of both visual and audio modalities into commercial retrieval. Firstly an efficient search method based on the extracted audio fingerprint feature is proposed to yield candidate results, and then visual signatures are extracted fused with audio features to validate the candidate results. The computational efficiency of audio modality and the robustness of visual modality are utilized simultaneously. The proposed optimal path search further guarantees the effectiveness in real applications. Comparison experiments were carried out over 123 video commercials and real TV programs from four channels. Results demonstrate both high efficiency and high robustness of the proposed method.