Development of TRINETRA : A Sensor Based Vision Enhancement System for Obstacle Detection on Railway Tracks

The authors have developed a prototype, which promises easy passage for trains in unknown environments such as zero visibility situations, fog, smog, and heavy rain. The prototype is based upon the integration of a camera, Radio Detection and Ranging (RADAR), and Infrared (IR) Light Amplification by Stimulated Emission of Radiation (LASER).The camera used in the prototype catches a long-range view of the track and presents it live on a mini screen fixed in the loco pilot cabin. A combined short, mid, and long-range radar sensor system is used to detect obstacles continuously in loco pilot blind spots on the track, particularly for collision avoidance assistance at high speed. The present work proposes a long-range laser IR illuminator with a wide range of color and mono cameras to aid clear and precise monitoring in zero visibility conditions, which is fixed on the front portion/engine of the locomotive/engine. The prototype experimental results for 2 m - 2 km distances have been performed on a live running train, which shows that the developed prototype tracks obstacles effectively during fog and smog conditions. The design concept, observation, prototype model, and other technical specifications have been presented, and satisfactory results were found.


I. INTRODUCTION
T RANSPORTATION systems like airways, railways, roadways, etc., have been considered as lifeline for humans. The transportation preferences are completely opted by humans based on their frequent availability, distance, and fare. Also, the preference may vary according to the particular country's population and GDP. Indian railway is one of the largest systems in the world, which includes more than 121 × 10 3 km of railway track with a route of 67 × 10 3 km and 7.3 × 10 3 modernized stations. Nearly 49% of the routes are electrified, and overall, 8 billion passengers and 1.2 billion tonnes of freight are recorded in 2017-2018. The revenue from freight and passengers of the Indian Railways is at the unbelievable stats of 16.5 billion USD and 6.9 billion USD, respectively.
The Indian railways are one of the world's largest railways in terms of 1.308 million employees who can handle 17 zones with 68 divisions. So, whenever such a large organization handles massive people, then reliability, safety, and security are the major concerns for management [1], [2]. Thus, a study on railway system reliability and safety has been performed, which depends upon errors related to the human operators [3].
R.K. Saket Also, a real-time monitoring analysis has been established for railway system safety by implementing the cusp catastrophe model [4].
The railway systems lose thousands of USD due to a train delay, cancellation of trains, and mishaps during fog and smog conditions. Also, people lose their lives and time, which can not be tolerated at any cost [5]- [7]. In [8], the authors have provided a review of the literature on vision-based on-board obstacle detection and distance estimation in railways. The authors [9], [10] discussed the current developments and the performance of LIDAR, RADAR, vision cameras, ultrasonic sensors, and IR and review their capabilities and behaviour in a number of different situations: during daytime, at night, in extreme weather conditions. A collision risk prediction assessment has been performed to avoid train-train, or trainvehicle collisions [11]. Also, a vision-based traffic accident detection method was proposed to investigate the imbalance between regular traffic and traffic accident [12]- [14]. Further, a sliding window framework has been considered for detecting several moving objects by using a monocular moving camera [15], [16]. Because of this, a wide-angle camera-based practical back over warning system has been proposed [17] while, new planar object detection, and recognition method has been proposed on data association of light detection and ranging, and camera [18], [19]. The object detection process can also be performed using a rearview camera [20], [21], and a multiple-camera 3D object tracking system has been developed for traffic data collection at intersections [22], [23].
An Infrared (IR) emitted quantum cascade laser is used to illuminate these adverse environmental conditions properly. A vision-based traffic accident detection method was proposed to investigate the imbalance between normal traffic and traffic accident [12], [24]. A machine learning algorithm was proposed to recognize the road surfaces [25], [26]. In poor visibility conditions, 3D mobile laser scanning data have been utilized to detect side objects such as trains, light poles, and signpost [27]. Further, a sliding window framework has been considered for detecting several moving objects by using a monocular moving camera [15]. Because of this, a wideangle camera-based practical back over warning system has been proposed [17] while, new planar object detection, and recognition method has been proposed on data association of light detection and ranging, and camera [18], [19], [28]. The object detection process can also be performed using a rearview camera [20], and a multiple-camera 3D object tracking system has been developed for traffic data collection at intersections [22].
The Indian railway network is the most preferable and cheapest form of transport affordable by most of the society's classes; there is a huge demand for designing an efficient and reliable system, including signal interpretation, working conditions, and operations of the locomotives. Therefore, the present work has developed a prototype for avoiding accidents, which consists of a CMOS sensor camera, radar antenna, and IR laser illuminator. The design of a sensor using CMOS technology has been described, enabling the system's performance capability in terms of frames per second [29], [30]. Also, the calibration of an omnidirectional IR camera has been investigated for intelligent perception applications [31]. It has been recognized a need to understand and address the present challenges/issues related to loco-pilot comfort and visibility to prevent accidents on the tracks and make railway better, safer, reliable, and enhance pilots' situational awareness [32]. The train accidents can be avoided by equipping the present railway system with safety equipment. The study on the degree of reliability and accuracy becomes necessary where the safety and security fundamentals propose multiple sensor systems. The two salient concepts in the rail transport system are reliability and safety [33], [34].
The research work is proposed to provide a technological solution for the trains' standard performance during abnormal environmental conditions like fog, smog, zero-visibility condition, the obstacle on track when locos are driven in Long Hood Forward (LHF) and Short Hood Forward (SHF) Modes. The present work suggests an unconventional and dynamic concept to provide an experimental product for Indian railways to prevent accidents on the track. In order to maintain standard performance during zero visibility situations, impediments on the track, and locos run in LHF and SHF modes, the technique must perform better in fog. The article provides an innovative technological solution by developing a TRINETRA device for tackling the current operational issues encountered by the Indian railways' loco-pilots. The works described in the literature have not considered the abnormality, including fog, smog, and other bad weather conditions, as mentioned in Table I. The prototype device's development with its realtime experimental verification has not been observed in the literature. In contrast, the authors' main contributions to the research work are as follows.
• TRINETRA has been implemented and tested experimentally. • The results obtained by considering different cameras for various distances. • Live running train has been considered for proper validations. The accomplishment of the mentioned contributions, the article has been organized as follows. Section I presents Indian railway networks and fundamental problems related to train accidents in Indian railways. An overview of the casualties in Indian railways and the requirement of development of TRINETRA to prevent train cancellations and accidents during the foggy sessions have been described in Section II. Essential aspects of design concepts of TRINETRA have been illustrated in the Section III. Section IV presents a brief model arrangement methodology of system components for the reliable operation of the developed TRINETRA. The experimental observations are included in Section V. The overall conclusion and future research plan are described in Section VI. The flow chart of the present research work is described in Fig. 1.

II. AN OVERVIEW OF ACCIDENTS IN INDIAN RAILWAYS
The railway accidents in India are due to the collision, derailments, level crossing, and fire in trains. Derailment is one of the major causes due to which mishaps occur, and the dense fog in winter is also aided to the accidents at level crossings. During India's winter season (November-January), the mist is quite regular, with temperatures approximately 3-4 degrees to 10-12 degrees, and the fog sometimes converts to smog in some polluted places. Either fog or smog may lead to the zero-visibility condition, which depends on the concentration of droplets and this condition leads to multiple collisions of road vehicles and trains. Smog is originally named for the mixture of fog and smoke in the air. The number and type of accidents, accidents per million km run, number of casualties, casualties per million passengers carried, and cases of railway accidents are described in the upcoming subsections. So, to save lives, time, and economy, the present work proposes to equip the Indian railway locomotives with a TRINETRA, which is treated as an anti-accident system to stop derailments, train collisions, and unmanned crossing accidents. The proposed technology is based on a radar-based device that alerts loco pilots whenever an obstacle comes on  the railway track and prevents a collision or any other accident.
The developed TRINETRA will be more useful in foggy nights and smog conditions when drivers continuously look outside the locomotive to assess the weather conditions.

A. Number and type of accidents
The 90% of railway accidents in India are due to derailments and accidents at level crossings during fog and smog conditions, and the remaining ten percent of accidents are due to rail collisions. The number of casualties in Indian railways by type from 2011 to 2020 are listed in using Table II. The graphical representation of railways accidents during 2011 to 2020 is described in Figure 2. The reason of accidents due to collision and level crossing is fog and smog.   2011  5  80  53  2  1  2012  9  55  61  4  2  2013  6  49  58  8  NR  2014  4  53  59  7  3  2015  5  63  56  6  5  2016  3  65  35  0  4  2017  5  78  20  1  0  2018  3  54  3 Table III. The train accidents are well shown graphically in Figure 3. Overall, a trend of tremendous decrease in train accidents is noticed but still a significant concern if a single casualty occurs.

C. Casualties per million passengers carried
As shown in Table IV Figure 4.
In any train accident, some casualties occur either in dead or injury form. So, Figure 4 shows the number of casualties per million km run, and it is concluded that casualties are increased and then decreased in 2017-2018. It is also seen that where the ratio of killed and injured is approaching, one leads to fatal accidents.

D. Cause of accidents in Indian railways
The development of TRINETRA technology is due to the various causes of train accidents where it is found that maximum accidents occur due to human mistakes (87%), as shown in Table V. Other than this, equipment failure, sabotage, and unexpected failure are the leading causes of train mishaps, as described in Figure 4. The factors like fog and smog may invoke a few of the losses.
In innovative TRINETRA, a radar-based device is used to alert the loco pilot to prevent casualties due to accidents. This proposed system is more useful in India during the foggy and smog conditions at night to assess the atmospheric conditions. To tab the signals up to 1-2 km distance, the developed TRINETRA technique uses infra-red and displays  the actual information on the screen, which is fitted inside the locomotive. The screen information suggests the driver stop the train within a prescribed time period.
The proposed system easily senses the presence of any animal, a boulder lying, or any uprooted tree on the railway tracks. If any vehicle or tree has broken down or any animal is present on the rail tracks, the developed system will notice them well in advance. The development of this device will be a part of Indian railways to avoid train accidents. In an attempt to avoid train accidents, Indian Railways should initiate a process to launch the TRINETRA system. This device will alert the drivers of any physical obstruction on railway tracks ahead and thus avert accidents. This will be useful during nights and in foggy conditions and the drivers need not to look outside the locomotive to assess the condition. The only drawback is that it can not visualize the objects coming just behind the curves.

III. DESIGN CONCEPT OF TRINETRA DEVICE
The present work aims to design the TRINETRA to prevent the above severe accidents during the winter season (November to January) in India to make the railways a zerotolerance zone for accidents. The suggestion of this research work is to install an automatic signaling system, anti-collision devices, and train protection and warning system in all trains in the coming years to avoid accidents in India. The proposed technologies of TRINETRA will help the Indian railways to prevent any accident, and there will be no railway accidents in the future. The experimental TRINETRA is offered with three technological components and combinations, as provided in Figure 6. The developed prototype TRINETRA system uses bullet camera + lens: IR cut filter (color to black) 1.2 megapixel Complementary Metal Oxide Semiconductor (CMOS) sensor + the electric zoom lenses with Fixed Focal Length (FFL) of 3.5 mm to 100 mm built-in digital HD module to image up to 1080P. Brief descriptions of proposed design components like thermal camera, obstacles sensing radar, and laser infrared illuminator are given below.

A. The optical camera system
It is necessary to incorporate a quality high definition night vision camera in TRINETRA, as shown in Figure 7. The experimental camera is a Pan-Tilt-Zoom Internet Protocol (PTZ IP) type with high resolution, long-distance lens, and infrared illuminator to identify the object details up to 2 km. The illuminator with a 2000 m range and the security system is integrated with the camera, and the TRINETRA setup is attached to the Indian railway network. The thermal imaging camera is used for night vision in bad weather conditions like fog, and also the optical zoom camera with laser technology provides details of the Indian railway tracks. According to the availability of models, the list is illustrated in Table VI.

B. The obstacle sensing radar system
The requirement of a hindrance sensing radar system for the Indian railway is important to enhance lives' security and safety. The radar system of high resolution provides enough details of obstacles; hence it is compatible with Indian railway applications. An obstacle sensing radar system for a railway crossing application has been implemented to operate trains normally. The combined short, mid, and long-range radar antenna chipset for 76 GHz to 77 GHz Frequency-Modulated Continuous-Wave (FMCW) radar sensor system is used in this experiment. A phased array receiver module at 77 GHz is needed to design and manufacture to achieve the ray beam steering capacity. The comparison between the passive element antenna array and measure gains of phased array receiver is made for performance enhancement of system [42].

C. The laser infrared illuminator
A long-range laser infrared illuminator equipped with a camera for use with a wide range of color and mono cameras to aid clear and precise monitoring in the dark. It has long safely dispersed the beam at a wide 4 0 angle to light up a huge area of 400 m 2 at a 300 m range, as shown in Figure 8. A 3D outdoor scene scanner has been presented to acquire km deep scenes in night conditions [43]. Its imaging system includes a low-cost pulsed laser illuminator and a Charge Coupled Device (CCD) camera equipped with a light intensifier as it is possible to get portions of the scene at a certain distance by accurately synchronizing the camera and illuminator. Hence, a long-distance scanning with smaller portions allows camera adjustment concerning the accuracy and enhanced range. The image pixel intensities produce active image systems that are varied with the square of the distance and limit of image sensors. The concept of TRINETRA, an indigenous cost-effective camera equipped with laser rays and display unit, is an outcome of our research efforts to address the current operational visibility issues faced by the loco pilots of Indian railways during abnormal weather conditions, as shown in Figure 9. According to outer visibility, the train speed has been considered, as described in Table VII. This approach is especially applicable during the fog, smog, and rain and locos driven in LHF and SHF Modes, as described in Figures 10  and 11.

IV. INSTALLATION OF TRINETRA COMPONENTS
Equipped with a CMOS sensor camera, radar trans-receive antenna, and IR laser illuminator integration is known as TRINETRA, a hybrid system is fixed on the front portion of the locomotive engine in Figure 12. It will catch a longrange view of the track efficiently and present it live on a mini   screen fixed at the loco pilot cabin, as described in Figure 13. Infrared (IR) laser illuminator will stay OFF when there are enough light sources, such as daytime and typical weather conditions. The Opto light sensor will trigger the illuminator during poor weather conditions such as fog, smog, and heavy rain. Radar detects the distance of obstacles located at the engine's locomotive and sends an early warning and indication to loco pilots in the mini screen arranged in the cabin before a collision occurs.

V. EXPERIMENTAL OBSERVATIONS AND RESULTS
The TRINETRA system is proposed to fix on the front portion of the engine/locomotive. It catches a long-range view of the track efficiently and presents it live on a mini-screen set at the loco pilots' cabin. The train pilots can see the long range of tracks on screen. The tracks can be seen clearly using a camera-radar-laser combination. The observations related to the designed TRINETRA device have been made experimentally, illustrated in Tables VIII to Table XI. The TRINETRA device has been implemented on train number 12229, namely, Lucknow-New Delhi, described in Figure 14. The developed TRINETRA technique uses infra-red and displays the actual information on the screen, fitted inside the locomotive. The obtained results from the first experiment on designed TRINETRA indicate that the 1.2 megapixel CCD camera, infrared (810 nm) 20 mW diode laser, 4.3 inch Thin Film Transistor (TFT) Liquid Crystal Display (LCD) color monitor can catch and present the view up to 2 m distance in zero visibility situation. Obtained results from the first experiment are illustrated in Table VIII. The obtained results from the second experiment on TRINE-TRA indicate that the 10 megapixel HD CCD camera, infrared (810 nm) 20 mW diode laser, 4.3 inch TFT LCD color monitor is able to catch and present the view up to 200 m distance in zero visibility situation. Obtained results from the second experiment are shown in Table IX. The obtained results from the third experiment on innovative TRINETRA indicate that the TVIP SGMA HD camera, infrared (810 nm) 20 mW diode laser, 4.3 inch TFT LCD color monitor is able to catch and present the view up to 998 m distance in zero visibility situation. The obtained results from the third experiment are illustrated in Table X. The obtained results from the fourth experiment on developed TRINETRA represents that the TVIP OPTH HD camera, infrared (810 nm) 20 mW diode laser, 4.3 inch TFT LCD color monitor can catch and present the view up to 2000 m distance in zero visibility situation. The obtained results from the fourth experiment are illustrated in Table XI. It is observed that using better resolution camera with higher Mega Pixel helps in enhancing the visibility to a larger range. This is working as a third eye for the train to see the tracks clearly.

VI. CONCLUSION
The developed TRINETRA device has been dedicated to Indian railways after successful real-time experiments on running train. The railway system can continue to operate normally in foggy conditions and zero-visibility situations like smog, heavy rains, and storms. After performing a rigorous survey, it was concluded that railway accidents are very prone to abnormal environmental conditions. Thus, the present work has performed real-time experiments to verify the usability of the developed prototype. During the real-time experiments, it was found that the device performed better for distances up to 2 km.
TRINETRA provides the predictive detection of obstacles on tracks and collision warning to loco-pilots has driven in long hood forward and short hood forward modes. The output would automatically keep loco-pilots at a distance from the locomotive in front and contribute to preventing accidents on the tracks by identifying potentially dangerous situations and responding accordingly.
The present work aims to install an automatic signaling system, an anti-collision device, and a warning system to avoid train accidents. Further, the developed prototype can also be helpful as a surveillance technology during these abnormal conditions. The proposed technologies of TRINETRA would help the Indian railways in preventing the accidents, and there would be no related casualties in the future. Further, the device can be improved in terms of distance and visibility, and the economic aspects can be considered the future research scopes.