TechRxiv
Find_Them_IEEE_Computer_Final_Sep_8_copyright_removed.pdf (5.63 MB)

Applying Machine Learning and Data Fusion to the "Missing Person" Problem

Download (5.63 MB)
preprint
posted on 14.09.2021, 20:41 by KMA SolaimanKMA Solaiman, Tao Sun, Alina Nesen, Bharat Bhargava, Michael Stonebraker
We present a system for integrating multiple sources of data for finding missing persons. This system can assist authorities in finding children during amber alerts, mentally challenged persons who have wandered off, or person-of-interests in an investigation. Authorities search for the person in question by reaching out to acquaintances, checking video feeds, or by looking into the previous histories relevant to the investigation. In the absence of any leads, authorities lean on public help from sources such as tweets or tip lines. A missing person investigation requires information from multiple modalities and heterogeneous data sources to be combined.
Existing cross-modal fusion models use separate information models for each data modality and lack the compatibility to utilize pre-existing object properties in an application domain. A framework for multimodal information retrieval, called Find-Them is developed. It includes extracting features from different modalities and mapping them into a standard schema for context-based data fusion. Find-Them can integrate application domains with previously derived object properties and can deliver data relevant for the mission objective based on the context and needs of the user. Measurements on a novel open-world cross-media dataset show the efficacy of our model. The objective of this work is to assist authorities in finding uses of Find-Them in missing person investigation.

Funding

Northrop Grumman Corporation

History

Email Address of Submitting Author

ksolaima@purdue.edu

ORCID of Submitting Author

0000-0003-1218-9237

Submitting Author's Institution

Purdue University

Submitting Author's Country

United States of America