loading page

A Software Implementation of a Conversational Multilingual Avatar-Based Interactive Multifunctional AI Kiosk and Mobile Application With Computer Vision
  • Udayshankar Ravikumar
Udayshankar Ravikumar

Corresponding Author:[email protected]

Author Profile


Background: In an era of increasing globalization and digital interaction, the demand for efficient, multilingual customer service solutions is paramount. This study introduces a novel software implementation of a conversational multilingual avatar-based interactive multifunctional AI kiosk and mobile application with computer vision, which is designed to address the challenges in delivering seamless, interactive, and inclusive customer service. The system integrates advanced technologies, including natural language processing, computer vision, and artificial intelligence, to create a user-friendly, avatar-based interactive platform. The purpose of this study is to explore the software's architecture, functionality, and effectiveness in providing a multilingual, conversational interface for diverse user demographics, particularly in kiosk and mobile application formats.
Results: The implementation demonstrated robust performance in several key areas. The avatar-based interface, created using ReadyPlayerMe, delivered a highly interactive and engaging user experience, enhanced by computer vision technologies like OpenCV and Google ARCore for accurate face detection and tracking. The system effectively utilized Whisper API and Google Cloud's Speech-to-Text (STT) and Text-to-Speech (TTS) services to facilitate realtime language translation and processing. This ensured seamless communication in multiple languages. The integration with OpenAI's GPT-3.5 model, prompt-tuned to specific business data, provided contextually relevant and accurate responses, significantly enhancing the user experience. Critical issues such as privacy, data security, and real-time processing were effectively addressed, ensuring user trust and system efficiency. Additionally, the system demonstrated an ability to cater to a broad spectrum of industries, making it a versatile tool in varied business scenarios.
Conclusions: The multifunctional conversational AI software effectively bridges the communication gap in customer service interactions, providing a scalable, secure, and user-friendly solution. Its ability to engage users in their native language, coupled with its interactive avatar-based interface, marks a significant advancement in the realm of digital customer service solutions. This implementation holds substantial potential for widespread adoption across various sectors, offering implications for enhancing global customer service standards, business efficiency, and user inclusivity. The success of this system sets the stage for future innovations in AI-driven customer interaction technologies.
25 Jan 2024Submitted to TechRxiv
29 Jan 2024Published in TechRxiv