In-Store Detection of Damaged Products

Authors: 
Abby Beizer, Zachary Darden, Gary Harper, Dan Kane

Abstract:
A method for real-time detection of damage and/or tampering to products in a retail environment using image recognition. This invention promotes safety of food items and prevent hazards created by spillage of liquids. Comprised of:

1. A machine learning component that is trained on product data to identify different products in images.
2. Cameras to monitor products in an area. This component gathers image data and feeds it to a software component.
3. A software component that processes image data and utilizes the trained ML model to identify when a product in the image is physically damaged, and flags the product for employee attention.

Background:
This invention aims to detect physical damage to any stocked product in a range of products at the soonest possible point in time from when the product was affected. Existing patents tend to focus on damage prevention rather than response. Furthermore, the use of image recognition for identifying products tends to be part of larger inventory-tracking solution with no intention of classifying a product’s physical condition as damaged.

One such invention, the “smart shelf”, analyzes deviation in size and weight for accuracy of quantity-tracking wherein outliers are treated as a different product altogether – there is no focus on determining whether these deviations indicate damage. While size and weight can be used as additional indicators to support this type of analysis, they offer much less reliability than image recognition as a main factor. For one, the smart-shelf sensors analyze only the bottom of a product, which cannot detect damage to the product’s other areas. A series of cameras can easily monitor aisles from different angles to give a more complete picture. Secondly, damage to a product may exist without affecting that product’s weight – the smart-shelf may fail to detect damage in cases such as these. And lastly, analyzing the dimensions and weight of a product may indicate that the product has been damaged, but image recognition can additionally indicate the extent or severity of the damage done (for example, whether the contents of a product have spilled into the aisle and pose a slipping/tripping hazard to customers).

Description:
Product images are used to train a machine learning model to determine the identity of a product and any significant deviations in its expected physical appearance that indicate damage.

Cameras are installed to monitor products contained in various shopping areas and periodically report images to an in-store server. Images are accompanied by the camera’s identity, allowing the server to collect images from all cameras in an area into batches before feeding data into the trained machine learning model. Whenever damage is detected, an alert containing detailed information about the location of the damaged product and the extent of the reported damage is raised for an employee to address

Usages:

1. Monitor safety of food items for consumption
2. Prevent injuries due to spillage of product contents
3. Quality assurance for a wide variety of products
4. Detect instances of tampering and theft

Enabling Technology:

1. Machine learning
2. Image recognition

Related Prior Art:

https://patents.google.com/patent/US10586208B2

As previously mentioned, prior art for inventions such as smart shelves focuses on identification of items for the purpose of inventory management rather than assessing quality of the product, and tends to ignore products that deviate from expected parameters rather than qualifying them as damaged.

https://patents.google.com/patent/US10586208B2

This patent outlines a method of inspecting a variety of engines for physical defects. This differs from the proposed mechanism for use in a retail environment in the following ways:

1. The proposed mechanism must be able to monitor many products in multiple layouts (shelves, displays, deli counters, etc.), while US10586208B2 inspects parts in isolation and is highly specialized to engine blades.
2. US10586208B2 inspects mechanical parts for deviations from strict specifications, such as improper measurements, physical damage, warping, etc. The proposed mechanism for retail environments demands more flexibility in detection of damage due to inherent inconsistencies in printing and packaging processes. Furthermore, food products such as fruits, vegetables, meats, etc. have a range of acceptable colors, shapes, and sizes.

TGCS Reference 2372