According to the World Health Organization (WHO), hundreds of million people have some type of visual disability. Vision impairment has a personal impact with lifelong consequences because more than 80% of our perception, cognition, learning, and daily activities are mediated through vision. Moreover, in the era of rapid advancements in artificial intelligence (AI), visually impaired and blind people face challenges at work and in education because of inaccessibility to AI technologies. In this regard, we present an assistive mobile application with an intuitive user interface (UI) for visually impaired and blind people to interact with ChatGPT via natural conversation. The app employs automatic speech recognition (ASR), text-to-speech (TTS), keyword spotting (KWS), voice activity detection (VAD), and a convenient UI to interact with ChatGPT effortlessly. We have made the source code, pre-trained models, and UI publicly available at https://github.com/IS2AI/talk-llm to stimulate the development of assistive mobile applications.

Visually impaired and blind people often face a range of socioeconomic problems that can make it difficult for them to live independently and participate fully in society. Advances in machine learning pave new venues to implement assistive devices for the visually impaired and blind. In this work, we combined image captioning and text-to-speech technologies to create an assistive device for the visually impaired and blind. Our system can provide the user with descriptive auditory feedback in the Kazakh language on a scene acquired in real-time by a head-mounted camera. The image captioning model for the Kazakh language provided satisfactory results in both quantitative metrics and subjective evaluation. Finally, experiments with a visually unimpaired blindfolded participant demonstrated the feasibility of our approach.

There are many factors that can negatively impact worker safety in manufacturing or industry. In particular, hazards such as contact of workers’ hands or other body parts with overheated components can lead to accidents and production losses. Hot surfaces cannot be seen with the naked eye. To detect hot surfaces, thermal cameras can be utilized in work areas. Placing multiple thermal cameras in large work environments, such as factories, is cost prohibitive, and carrying them is not a solution. This paper proposes an augmented reality-based assistant that visualizes hot surfaces by integrating thermal camera information into augmented reality goggles. The major advantages of the system are twofold: (1) it consists of an untethered mobile hardware platform with a thermal camera that can send information through a wireless connection, (2) the visualization is implemented for augmented reality head-mounted displays that do not require the use of workers’ hands, unlike tablet-based augmented reality. Augmenting human sight with thermal vision may also have an impact on other research areas in the future

Augmented reality (AR) offers novel ways to design, curate, and deliver information to users by integrating virtual, computer-generated objects into a real-world environment. This work presents an AR-based human memory augmentation system that uses computer vision (CV) and artificial intelligence (AI) to replace the internal mental representation of objects in the environment with an external augmented representation. The system consists of two components: (1) an AR headset and (2) a computing station. The AR headset runs an application that senses the indoor environment, sends data to the computing station for processing, receives the processed data, and updates the external representation of objects using a virtual 3D object projected into the real environment in front of the user’s eyes. The computing station performs computer vision-based indoor environment self-localization, object detection, and object-to-location binding using first-person view (FPV) data received from the AR headset. We designed a behavioral study to evaluate the usability of the system. In a pilot study with 26 participants (12 females and 14 males), we investigated human performance in an experimental task that involved remembering the positions of objects in a physical space and displaying the positions of the learned objects on the two-dimensional (2D) map of the space. We conducted the studies under two conditions—that is, with and without using the AR system. We investigated the usability of the system, subjective workload, and performance variables under both conditions. The results showed that the AR-based augmentation of the mental representation of objects indoors reduced cognitive load and increased performance accuracy.

Object recognition is a machine learning problem that involves the correct classification and localization of objects in an image. Object recognition has found wide applications in Industry 4.0, surveillance, and autonomous driving. Prominent datasets such as Pascal VOC, ImageNet, and MS COCO have further spurred advances in object recognition. However, the images in these datasets were captured using ordinary perspective lenses. Unlike ordinary cameras, fisheye or hemispherical cameras have a wide FoV reaching 180 degrees. Due to this property, they can acquire wide panoramic images, thus, capturing more objects in a scene as compared to ordinary perspective lenses. In this work, we provide the first benchmark testing dataset of real fisheye images containing commonly-occurring objects in the human environment obtained with hemispherical cameras. The dataset comprises 1,000 images containing 39 classes with a total number of 14,218 object instances. We trained the YOLOv7 model on the original MS COCO and transformed FisheyeCOCO datasets and tested the obtained models with our dataset. The results indicate that training the model on the combination of original and transformed datasets improves the performance by 2.5% compared to using these datasets separately.

Faces in Event Streams dataset contains 689 minutes of recorded event streams, and 1.6 million annotated faces with bounding box and five point facial landmarks. Tha paper presents the dataset and corresponding models for detecting face and facial landmakrs directly from event stream data.

Speech Command Recognition (SCR) is rapidly gaining prominence due to its diverse applications, such as virtual assistants, smart homes, hands-free navigation, and voice-controlled industrial machinery. In this paper, we present a data-centric approach to creating SCR systems for low-resource languages, particularly focusing on the Kazakh language. By leveraging synthetic data generated by Text-to-Speech (TTS) and data extracted from a large-scale speech corpus, we successfully created the Kazakh language equivalent of the Google Speech Commands dataset. Moreover, we also compiled the Kazakh Speech Commands dataset with data collected from 119 participants. This dataset was used to benchmark the performance of the Keyword-MLP model trained using our synthetic dataset. The results showed that the model achieves 89.79% accuracy for the real-world data demonstrating the efficacy of our approach. Our work can serve as a recipe for creating customized speech command datasets, including for low-resource languages, obviating the need for laborious and costly human data collection.

Face detection is a mandatory step in many computer vision applications, such as face recognition, emotion recognition, age detection, virtual makeup, and vital sign monitoring. Thanks to advancements in deep learning and the introduction of annotated large-scale datasets, numerous applications have been developed for human faces. Recently, other domains, such as animals and cartoon characters, have started gaining attention but still lag far behind human faces. The biggest challenge is the limited number of annotated face datasets in these domains. The manual labeling of large-scale datasets is tedious and requires substantial human labor. In this regard, we present an input-agnostic face detector to ease the annotation of various face datasets. We propose a simple but effective data-centric approach instead of building a specific neural network architecture. Specifically, we trained a face detection model, YOLO5Face, on human, animal, and cartoon face datasets. The experiments show that the model can achieve accurate results in all domains. In addition, the model achieved decent results for animals and cartoon characters different from the ones in the training set. This implies that the model can extract agnostic facial features. We have made the source code and pre-trained models publicly available at https://github.com/IS2AI/AnyFace to stimulate research in these fields.

This paper addresses sequential indoor localization using WiFi and Inertial Measurement Unit (IMU) modules commonly found in commercial off-the-shelf smartphones. Specifically, we developed an end-to-end neural network-based localization system integrating WiFi received signal strength indicator (RSSI) and IMU data without external data fusion models. The developed system leverages the advantages of WiFi and IMU modules to locate finer-level sequential positions of a user at 150 Hz sampling rate. Additionally, to demonstrate the efficacy of the proposed approach, we created the IMUWiFine dataset comprising IMU and WiFi RSSI readings sequentially collected at fine-level reference points. The dataset contains 120 trajectories covering an aggregate distance of over 14 kilometers. We conducted extensive experiments using deep learning models and achieved a mean error distance of 1.1 meters on an unseen evaluation set, which makes our approach suitable for many practical applications requiring meter-level accuracy. To enable experiment and result reproducibility, we made the developed localization system and IMUWiFine dataset publicly available in our GitHub repository.

Face detection and subsequent localization of facial landmarks are the primary steps in many face applications. Numerous algorithms and benchmark datasets have been introduced to develop robust models for the visible domain. However, varying conditions of illumination still pose challenging problems. In this regard, thermal cameras are employed to address this problem, because they operate on longer wavelengths. However, thermal face and facial landmark detection in the wild is an open research problem because most of the existing thermal datasets were collected in controlled environments. In addition, many of them were not annotated with face bounding boxes and facial landmarks. In this work, we present a thermal face dataset with manually labeled bounding boxes and facial landmarks to address these problems. The dataset contains 9,982 images of 147 subjects collected under controlled and uncontrolled conditions. As a baseline, we trained the YOLOv5 object detection model and its adaptation for face detection, YOLO5Face, on our dataset. In addition to our test set, we evaluated the models on the external RWTH-Aachen thermal face dataset to show the efficacy of our dataset. We have made the dataset, source code, and pre-trained models publicly available at https://github.com/IS2AI/TFW to bolster research in thermal face analysis.