Next Generation Smart Fridge System using IoT

— Rapid development in technology has driven our attention towards intelligent and smarter regime. Current lifestyle of people involves spending less time at home and more time at work, a quick responsive smart kitchenware can be designed to save time and money during shopping. This paper focuses on developing a smart fridge that will notify the user of which food items are present in the fridge via email. This smart sensing application will capture image of the items present in the fridge and recognize the list of the items using Amazon Web Services (AWS) Rekognition API. The AWS Polly will translate the list of the items into speech format, an audio .mp3 file, which is played by the Bluetooth speaker. Further, this image of the items in the fridge and the audio file is sent to the user via email. Thus, helping users to avoid food wastage and overspending on unnecessary items. This system is accessible anywhere and anytime by the user.


I. INTRODUCTION
Technology is important in our day to day life. In a dynamic world like today, life without technology is meaningless. Technology connects tools which efficiently ease the generation, use and exchange of information. It not only makes activities easier to implement but also has the solution to many society problems [1]. It has touched every aspect of life thus making it simpler, better and different. A simple example would be the way communication system has evolved from pigeons delivering messages to emails and messages received within a fraction of second. This advancements from treetops to laptops has made our life's more convenient and adaptive.
We have all heard about Internet of Things (IoT) which allows devices like car, mobile phones, security systems, commercial environments to communicate collecting and exchanging information through internet. It uses devices with sensors and software that can be accessed remotely in order to enable computer systems to be directly integrated. Current rapid development in technology has driven our attention towards intelligent and smarter regime [6]. Devices need to be smarter in order to match with our daily requirements. One of such devices is a fridge. The lifestyle of people involves spending less time at home and more time at work, a quick responsive smart kitchenware can be designed to save time and money during shopping. Moreover, when it comes to shopping in the grocery store people are less prepared and eventually are trapped into buying groceries more than required and thus wasting food and time.
In this report, we have proposed a smart fridge to increase healthy lifestyle with minimum effort and time. The system is designed to identify and manage the food items in the fridge. It identifies which type food is stored in the fridge like fruits, vegetables. The smart fridge has intelligent embedding system to help it communicate with the devices as well as the user. We have used raspberry pi with Raspbian OS, to mound the devices, raspberry pi camera to recognize the food item, Amazon Web Services Polly to convert user's text to speech, AWS Rekognition to recognize items in the fridge and Bluetooth speaker which plays the audio of the list of the items on the fridge. The user can access and run the program remotely using app, Terminus and VNC Viewer. Further, the list of the items, both in image and audio, mp3 format are sent to the user via email. Thus, helping users to avoid food wastage and overspending on unnecessary items. This system will assist people who have busy schedule and help them save a lot of money and time [12].
We integrated the current fridge system, which holds no automation and user interaction and build on it, to convert it into a smart IOT device. The current fridge system is taken to a whole new level of automation with the raspberry pi model, making it user interactable as the next Generation system [3].

II. PROBLEM STATEMENT
Spoilage of waste food is one of the biggest constraints we are facing in the world. As soon as the food is purchased it is stored for a long term in the refrigerator thus making its depowering its freshness. There are times when one forgets the items are already present in the refrigerator and end up buying the items again from the grocery store thus supporting food wastage.
Example, there are times when people forget the items like strawberries or milk already present in the fridge and end up buying these items in bulk. This significantly effects the food wastage if the product is thrown away since these are perishable food which spoil faster than non-perishable food perishable. This widely effects the expenditure rate spend on buying these food items thus unnecessarily shopping in bulk. There are few systems which exists in the market for the consumers in the meal control and dietary planning area.
However, these traditional meal control applications do not focus in terms of the management the food and assisting the consumers in grocery management. Therefore, in this project we have designed and implemented a system which allow its users to determine the product (fruit and vegetable) in the refrigerator and simultaneously notifying the user regarding the items available in the refrigerator remotely via email.

III. LITERATURE SURVEY
In order to give light to our project, we have referred research papers similar to smart refrigerator to help in creating a better and efficient smart fridge for the consumers. In research paper published by Emily [5] she focuses on identifying the food products in the fridge. It compares the status of food products with criteria's like expiry date, quality, weight, volume, etc.
Customized notifications can be sent to users. In the second research paper published by Subramanya Nayak [8], the proposed system automatically detects and alerts the users as well as shop owner for the restocking and replenishing supplies in the fridge. Buzzer and LED were used for alerting the users.
There was different technology used in the third paper written by Xing [18] the intelligent system allows users to locate and manage the food in the fridge efficiently. RFID technology is used to collect food information, identify the users, locate the food.
In the fourth referred research paper introduced by Dhananjay V. Kulkarni [15] he focuses on tries to reduce the food wastage around the globe in the proposed system. It uses IOT and advanced kitchen smart evolution focusing mainly on smart fridge. Finally, the system proposed by Mukesh Mahajan [4] tries to generate automatic list of food item for the users based on user's frequent food selection. It also notifies users scares product through android notification.
IV. COMPONENTS While building the system, it's important to setup the components properly to avoid any mishap during the experimentation process. Most of the components mentioned below will only be used during the setup stage.
A. Hardware Components: 1. Amazon Web Services Polly: Apps using voice to communicate are becoming more common. However, Naturalness of speech is a key element, which is a key aspect of AWS Polly.

Amazon Web Services Rekognition: Amazon
Rekognition makes it easy to add image to any applications using proven, scalable, deep learning technology that requires no machine learning expertise to use.

V. SYSTEM ARCHITECTURE
We start by physically connecting external devices, power bank, SD Card and camera to the Raspberry Pi device. An external speaker is then attached to the Raspberry Pi device using Bluetooth or audio cable.
The SD Card is pushed inside the Raspberry pi model to read the Raspbian OS. The power cable is attached in the USB port to give power to the raspberry pi model. The raspberry pi camera is attached in the camera module port of the raspberry pi model. These external components are mounded to the main raspberry pi device.
Further, VNC Viewer and Terminus mobile apps are installed to remotely access the internal view of the fridge anytime and anywhere. Raspberry Pi device's IP address is used to build connection with Remote mobile application. The user can easily access and view the live status of the food items in the fridge by using remote access apps. Fig. 1 shows, given below shows the proposed system architecture.

VI. WORKING
The program is divided into 4 parts: capturing the image through the raspberry pi camera module, recognizing the items captured in the images, converting the list into audio file and sending user the email containing the image and the audio mp3 file. The flow of the program for all the 4 processes is given below.

Fig 2. Flow to capture Images
In this section, the Fig. 2 above shows the flow for capturing the image. The camera module of the raspberry pi which is attached under the raspberry pi latch should be enabled in order to call the module. First, call the raspberry pi camera module. Then the image of the items stored is captured by the camera. Lastly, this image is stored on the desktop.

Fig 3. Flow for Rekognition of Images
The above Fig. 3 shows the flow for the recognition of the image captured by the camera module of the raspberry pi module. The image stored on the desktop is retrieved to recognize the items of the image. The connection built with the AWS Rekognition is called by the credentials using the access key and the secret key which was captured in the above steps.
With the help of AWS Rekognition, items are recognized and printed on the screen as a list. Further, this list is appended with the following text "Following items are present in the fridge".

C. Flow for AWS Polly
The below Fig. 4 indicates the flow that is followed for creating an audio file of the items present in the fridge. We take the appended text from the recognition procedure. Then, call the AWS Polly using the credentials which was mentioned earlier.
We use the access key and secret key to build connection with AWS polly. We call the AWS polly using the boto3 library. AWS Polly performs the conversion process: it takes the text and synthesizes it into an audio output sing Emma voice id. This audio is stored on the Desktop of the raspberry pi. The above Fig. 5 shows the flow for the email, we first call the python libraries: yagmail and smtplib which will automatically send the email to the user. Then we simply pass the gmail user and the password from which the email is supposed to be sent. Pass the content to be followed with the subject line for the email and send it using yagmail.send(). The user will receive the image and the audio within 1-2 secs.

Fig 6. Input
In our case, the input will be items in the fridge. The system should be properly placed to capture the image without any background hindrance. We need the Raspberry pi, raspberry pi camera, LED Light and the power cable to be placed in the fridge. The entire connection to be placed in the fridge is shown above. The above Fig. 6 shows the input and the system setup. The above Fig. 7 shows the image captured by the Raspberry pi camera which is later sent to the user via email. We have used a set of fruits and vegetables for this experiment.

VIII. OUTPUT
The below Fig. 8 shows the audio .mp3 file generated by the AWS Polly. This audio file is then played through the Bluetooth speaker.
The Fig. 9 shows list of the items generated as an output on the screen. The Fig. 10 depicts the email sent to the user containing the image of the items in the fridge and the audio mp3 file.

IX. BENEFITS
The system has been placed in the fridge and tested successfully. In this section, the benefits of the system are identified and listed below: A. Saves money: Most of the hardware are easy to find in market and are not costly than other hardware available in the market like raspberry pi and when it comes to software most of our software like Linux, AWS Polly and AWS Rekognition are open source or have free trial like python.

B. Quick access:
It is less time consuming to check the quantity of the food available in the fridge using the automatic recognition technique rather than traditional techniques. It also automatically finds the type of food available in the fridge as quickly as possible. Example: If there is apple in the fridge, then the system will detect apple within seconds.

C. Food waste:
It minimizes the food wastage by checking the quantity of the food available in the fridge and thus avoiding the available food to be shopped since food wastage is one of the strongest and major aspect of our project. This will help the consumers to have a regular check on their food items which will include both perishable and non-perishable food items present in the fridge.

D. Avoid unnecessary shopping:
Unnecessary shopping and directly linked with wastage of food. The system allows users to only buy item which are required to be refilled in the fridge by avoiding unnecessary shopping for food items. Example, if the user already knows the list of items present in the fridge then they would have a track of things to buy and compute the missing items in the fridge automatically.

E. Remote Access:
One of the most important benefits apart from food wastage is remote access. This will give the user secondary means of access to the system specially when the user is not around the system. This will mainly happen when the user beings to grocery shop and forgets the items in the refrigerator thus asking the system. The remote access come in play with the help of email. The user gets an email by the system along with audio output of the items available in the fridge.

A. Focus on different aspect of food:
While developing the smart fridge we have only focused on the perishable items which are mainly vegetables and fruits. This makes our smart fridge recognize fruits and vegetables accurately and efficiently. Therefore, because of the lack of time we couldn't focus on different aspects of food like milk carts, eggs, meat, fish, grains, bread and many more. This project can proceed with advancements in the range of items to be recognized by the system for the users accurately.

B. High resolution camera for high quality photos:
While using raspberry pi and raspberry pi camera to recognize the fruits and vegetables we had used raspberry pi camera module which had resolution of only 5 mega pixels. This reduces the sharpness in the quality of the image as well as the accuracy of reading the image efficiently. Therefore, by increasing the pixel of the raspberry pi 3 camera by using higher model of raspberry pi, that is raspberry pi 4, we can increase the sharpness and the accuracy of the images which will increase better recognition of the items in the fridge.

C. Focus on updated hardware and software:
In this project we focused on raspberry pi 3 model in order to build connection with the fridge and detect the items in the fridge. However, there are various other hardware one can use like using a higher and updated version of raspberry pi, that is raspberry pi 4. The project can be also done using Arduino as the main hardware instead of raspberry pi 3.

XI. CONCLUSION
This will give the user a virtual image of the items presents in the fridge as well as the system will convey the output via audio which can be helpful for users who are visually impaired. The user can also remotely access the camera and view the item present in the fridge. This application will help users save time in shopping and live healthy lifestyle. We are certainly that such type of system will be extremely beneficial for avoiding food wastage.

ACKNOWLEDGMENT
The Next Generation Smart Fridge System Using IoT final year project took us about a year to develop the system from scratch and to execute the system successfully. This entire process brought us to unlimited passion and harvest in our academic and professional careers. We would like to thank our supervisor Dr. Jinan Fiaidhi for supporting us and helping us. We have given a lot of efforts in the Raspberry pi setup throughout this project.
Secondly, we would love to thank the scholars cited websites and blogs for the inspiration. It is precisely because of the inspiration and help of research results that we can better understand and write the final report.