Database Reference
In-Depth Information
Interested Object
(object or text)
Recognition by search
(recog. object or text)
Contextual
Recommendation
Database
User Interaction
Object Context
Sensory Context
Fig. 4.2
Framework of mobile visual search and activity completion model using image contex-
tual model, including (1) “O”-based user interaction, (2) image context model for visual search,
and (3) contextual entity recommendation for social activities
images are further analyzed to provide meaningful textual-based social activity and
task recommendation.
Figure
4.2
shows the framework of our visual recognition and activity recom-
mendation model. In general, it can be divided into the client-end and cloud-end.
On the client-end, a user's visual search intent is specified by the “O” gesture on a
captured image. On the cloud-end, with user selected object and the image context
around this object, a recognition-by-search mechanism is applied to identify user's
visual intent. A novel context-embedded vocabulary tree is designed to incorporate
the “O” context (the surrounding pixels of the “O” region) in a standard visual search
process. Finally, the specified visual search results are mapped to associate metadata
by leveraging sensory context (e.g., GPS-location), which are used to recommend
related entities to the user.
The “O” gesture utilizes multi-touch screen of the smart-phone. Users do not
need any training and can naturally engage with the mobile interface immediately.
After the trace (the blue thin line in Fig.
4.2
) has been drawn on the image, sampling
points along the trace-line are collected as
N
j
{
D
|
(
x
j
,
y
j
)
∈
D
}
1
, which contains
=
(
,
)
N
pixel-wise positions
. Principal component analysis (PCA) is applied to
find two principal components (which form the elliptical ring depicted by thick
orange line in Fig.
4.2
). The purpose of this part is to formulate a boundary of the
selected region from an arbitrary “O” gesture trace. Mean
x
j
y
j
ʣ
are calculated, based on
D
and non-correlated assumption along the two principal
components:
μ
and covariances
.
x
˃
0
μ
=[
μ
x
,
μ
y
]
ʣ
=
(4.1)
y
0
˃
Search WWH ::
Custom Search