Authors:
Adam Archambault
Abstract:
This disclosure proposes mitigating the issue that human personal shoppers may select fresh food items of inconsistent quality -- whether the issue is caused because of time pressures, personal biases of the shopper, or other factors. Produce recognition technology certainly already exists even within Toshiba, but this invention would further refine existing machine learning/neural networks that merely recognize produce and other fresh foods to also score the fresh food item based on intake of "ideal" image representations of the product, whether from publicly-available images or those provided by a retailer. This invention would also allow the consumer to assist the AI device/robot in the decision of which item(s) to select, thereby further reducing the chance that the device selects a flawed product or that does not meet the consumer's quality demands.
Background:
During the time of the COVID-19 pandemic, there has been increased consumer demand for personal grocery shoppers through companies that provide those services for those who do not wish to risk possible exposure to the virus in a store setting. These companies rely on human shoppers to fulfill online customer orders. As demand for such services increase and as demand for human labor outweighs the available labor force, retailers may begin to augment human staff with AI devices such as bots to roam stores or distribution centers and pick/fulfill orders.
One challenge with using AI for grocery order fulfillment concerns picking fresh food items such as produce, meat, poultry, seafood, deli, and bakery items. Grocery customers are often choosy about these items and analyze them based on several sensory factors such as visual appeal, texture, and smell. When consumers use personal shoppers to shop on their behalf, there is a real possibility that the personal shopper will not consistently give the same priority to these sensory factors as a consumer would while shopping on his or her own. Possibly because of a competing priority to fulfill as many online orders as the personal shopper can as quickly as possible.
With this problem in mind, AI/robotic technology could be adapted to automatically analyze the visual qualities of fresh food items in a store or distribution center. With the plethora of product images available on the Internet or with a retailer's own provided images of "ideal" versions of its fresh food products, an existing neural network that already can recognize produce and other fresh food items in a store could be further enhanced to also analyze/score the products based on how well an image captured of the item matches an image of the ideal item in terms of color, shape, size, and lack of visual defects such as spotting. Before the order is fulfilled, the AI device would send the consumer the top 3-5 images of the product that it found via text or email, taking into consideration whether there is enough of the top-scoring items remaining to fulfill the consumer's requested quantity.
Description:
1. A retailer would load its catalog of fresh food items such as meat, produce, bakery, deli, and seafood items to a database that the machine learning application would use. The catalog data would need to include, at minimum, the product's SKU, and description.
2. Optionally, the retailer would also provide a set of images for each catalog item representing the "ideal" form of the item in terms of color, shape, size, and lack of visual defects.
3. The machine learning application, using a combination of publicly available product images that it finds via GIS (Google Image Search) and the retailer's provided product images, would build a cloud-based neural network to allow AI devices deployed in stores and distribution centers to locate ideal fresh food items. The GIS search would use search keywords that include each product's DB description and adjectives such as "good", "best", "great", etc.
4. When a consumer submits an order through an online retail order fulfillment application and the order reaches the "picking" phase of fulfillment, this would trigger an AI device/robot to begin traversing a store/distribution center to fulfill the order. If the AI device encounters a SKU in the order associated with a fresh food product, the device will move to the area where the product is known to be located from a store's layout map (not covered in this disclosure) and begin to take image samples of the product there, persisting quantitative attributes of the image such as color RGB values and exact product dimensions (height, length, width). If the consumer requests a specific quantity of a pre-packaged fresh food product (ex. 1 lb. of ground beef), the AI device may also analyze the product's label to ensure that the image samples only include items that are at or within a configurable tolerance of the consumer's requested weight/quantity.
5. The AI device will pass the images to the machine learning application, which will score the images compared with the "ideal representation" images that are part of its neural network and return the top 3-5 candidates.
6. When the AI device receives the candidate images, it will send them to the phone number or email address associated with the consumer's loyalty account with the retailer.
7. The device will not complete fulfillment of the order until the consumer responds with his or her choice of item via text message or email reply. The retailer may also choose to cancel the order if a response is not received within a configurable time window.
8. Once the consumer responds, the device will attempt to pick the item associated with the image. If the product is no longer available because of a delay in responding, the device will submit new candidate images to the consumer. Because of response delays, the AI device originally responsible for fulfilling the order may need to place the order in a "suspended fulfillment" state so that another AI device may resume the fulfillment once the consumer responds with a selection.
9. The AI device will pick the product that the consumer selected and mark the item in the order as picked or fulfilled.
10. After an order is picked up, consumers could respond to a survey rating the quality of the fresh food items in the order. The results could be aggregated and used to improve the machine learning application's scoring algorithm.
TGCS Reference 2265
