Interactive Mobile Visual Search and Recommendation at Internet Scale - Multimedia Database Retrieval: Technology and Applications - page 101

action is accomplished by recommending relevant entities based on recognition
results and associated metadata.
The novelty of the chapter lies in the following aspects:

￿ BoW model and context-aware visual search algorithm is designed with a novel

context-embedded vocabulary tree (CVT). The algorithm is able to achieve better
visual recognition performance by embedding the context information around the
“O” region into a standard visual vocabulary tree.

￿ Based on the context-aware visual recognition, a real system TapTell is imple-

mented to understand users' visual intents. The goal is to provide a contextual
entity suggestion for activity completion that provides meaningful and contextu-
ally relevant recommendations. Advanced touch screen technology provided at
the mobile platform is utilized to introduce human experts in loop for a better
visual search. Three different kinds of gestures for specifying object (and text)
of interest are investigated by a user study. It is concluded that “O” provides
the most natural and effective way to interactively formulate user's visual intent
and thus reduce ambiguity. After obtaining the recognition results, a location-
aware recommendation is provided to suggest relevant entities for social task
completion.

In the following, an interactive mobile visual search using the BoW model and
the CVT algorithm is first presented. A viable application, TapTell , is introduced in
detail to show how to accomplish meaningful contextually relevant recommenda-
tions through mobile recognition. Experimental results are provided to demonstrate
the effectiveness of the CVT method.

4.2 BoW-Based Mobile Visual Search Using Various

Context Information

This section presents the mobile visual search with context-aware image retrieval
using the BoW model. Section 4.2.1 briefly reviewed the BoW model and its
potential in large-scale content-based image classification, retrieval and visual
search. Section 4.2.2 introduces the literature and industrial developments of mobile
visual search. Section 4.2.3 describes the framework of context-aware mobile visual
search. Section 4.2.4 presents the algorithm of the visual recognition by search
using the BoW model with image context. Section 4.2.5 discusses a filtering process
adopting sensory GPS context.