Embedded System
A Raspberry Pie with 4 gravity sensors and camera is used to collect item information. In order to ensure accurate weight measurement, we placed two hard plates at the bottom of the shopping cart. Gravity sensors were installed at the four corners between the two plates. This ensures that the weights of items do not change when moving in the cart. We encountered some challenges when pursing precise weight: Items becomes bumpy when the shopping cart moves. This may recognized as multiple putting or taking actions. To solve that problem we set up that user can only add or remove items when the shopping cart is stable (gravity sensors fluctuate within a certain range). In addition, we assume that most items in the mall contain RFID labels. For items without labels, such as fruits, we use camera to identify and measure its weight. The camera points to a corner of the shopping cart. When the item is identified, user should move it to another place. When item is being removed, cart will also take a picture and update the item list. All database updating requests or item images are sent to AWS cloud server by Raspberry Pi.
Mobile Phone App User Interface
The mobile phone app mainly serves as a user interface front end for the intelligent shopping cart. When a user pick up a shopping cart, he or she need to read the cart ID and enter in the shopping cart phone app. This manually entering may be replaced by QR code scan in the future work. During the regular shopping phase, the mobile app will continues poll information through AWS Lambda functions for current cart ID every 300 millisecond. It will then compare the newly polled information with the past data. If any items get updated in the shopping cart, the app will re-draw the UI throw it's UI threads and user will get a audio notification on the phone. Currently, this constant poll model may trigger unnecessary power consumption. In the future it will be replaced by server publishing notification model.
For sake of reducing development cost and coherence user experience between iOS and Android, our team developed the mobile application based the Ionic web app framework. The whole app was coded in Angular JS, HTML, CSS and the phone native API was handled by Cordova framework. In this manner the app can by easily deployed to both iOS and Android platforms. As the shopping cart front end, the web app architecture won't be the bottleneck of any performance issue.
For sake of reducing development cost and coherence user experience between iOS and Android, our team developed the mobile application based the Ionic web app framework. The whole app was coded in Angular JS, HTML, CSS and the phone native API was handled by Cordova framework. In this manner the app can by easily deployed to both iOS and Android platforms. As the shopping cart front end, the web app architecture won't be the bottleneck of any performance issue.
Micro-services Supported by AWS
The cloud services mainly provide three functionalities, image recognition, data manipulation, and intelligent recommendation. They are implemented by utilizing the following components: Cognito, Simple Storage Service (S3), API Gateway, Rekognition, three Lambda functions, and two DynamoDB tables. This sub-section provides detailed explanation of the workflow of each process.
As mentioned in the above sections, Raspberry Pi sends the image it captures to S3 bucket when no RFID tag is scanned. This image also carries user ID and item weight information. A lambda function is set to be trigger by S3 bucket adding event, such that it would automatically run when an image arrives at the linked bucket. This lambda function then internally calls AWS Rekognition API that provides image recognition service. Then, this lambda funcition would analyze the tags returned by Rekognition and determine the name of added or deleted items. The next step is to determine whether a user is adding or deleting item based on the weight change information carried by the uploaded image. If the weight change value is positive, it will check whether this entry exists in DynamoDB. Similarly, If the weight change value is negative, it will check whether the remaining weight is larger than zero. After all these procedures, it will update the database accordingly.
The API Gateway is connected to two lambda functions, one for updating listed item on mobile phone application and the other for customer checking out. When the content of the shopping list displayed on users' phone needs to be updates, the first function would be called. It would first fetch all the data corresponding to this specific customer from DynamoDB, then run the recommendation algorithm to find new recommendation item. Information fetched from database and the calculated result would be returned to the mobile application in JSON form.
The recommendation algorithm of our project is a primitive implementation based on Prem Melville and Vikas Sindhwani's paper[3] about collaborative filtering in recommender systems. The basic idea is that, if customer A's shopping preference is close to customer B, then for those item A haven't pick yet, it is very likely that he will have the same choice as customer B. Directed by this ideology, every time the shopping list is updated, we will fetch all history shopping records and place them in a matrix. We find those history shopping record vectors having smallest distance with current shopping list vector, and provide recommendation based on them.
The recommendation algorithm of our project is a primitive implementation based on Prem Melville and Vikas Sindhwani's paper[3] about collaborative filtering in recommender systems. The basic idea is that, if customer A's shopping preference is close to customer B, then for those item A haven't pick yet, it is very likely that he will have the same choice as customer B. Directed by this ideology, every time the shopping list is updated, we will fetch all history shopping records and place them in a matrix. We find those history shopping record vectors having smallest distance with current shopping list vector, and provide recommendation based on them.