Applying Machine Learning and Data Fusion to the “Missing Person”
Problem
- KMA Solaiman ,
- Tao Sun ,
- Alina Nesen ,
- Bharat Bhargava ,
- Michael Stonebraker
Abstract
We present a system for integrating multiple sources of data for finding
missing persons. This system can assist authorities in finding children
during amber alerts, mentally challenged persons who have wandered off,
or person-of-interests in an investigation. Authorities search for the
person in question by reaching out to acquaintances, checking video
feeds, or by looking into the previous histories relevant to the
investigation. In the absence of any leads, authorities lean on public
help from sources such as tweets or tip lines. A missing person
investigation requires information from multiple modalities and
heterogeneous data sources to be combined.
Existing cross-modal fusion models use separate information models for
each data modality and lack the compatibility to utilize pre-existing
object properties in an application domain. A framework for multimodal
information retrieval, called Find-Them is developed. It includes
extracting features from different modalities and mapping them into a
standard schema for context-based data fusion. Find-Them can integrate
application domains with previously derived object properties and can
deliver data relevant for the mission objective based on the context and
needs of the user. Measurements on a novel open-world cross-media
dataset show the efficacy of our model. The objective of this work is to
assist authorities in finding uses of Find-Them in missing person
investigation.