AI, as the name suggests, is a computer that can perform human-like processing. If speech recognition (Speech to Text) is the ear and speech synthesis (Text to Speech) is the mouth, image recognition is the function of the eyes. From now on, let's take a look at how image recognition is being used by industry.
If you look at the online article about image recognition, it is explained that it is classified into the following three types.
However, regardless of the classification of object detection and classification, it feels strange to pick up only faces and characters from recognition and line them up. Face recognition is one of the biometrics, along with fingerprints, irises, retinas, and veins, and character recognition includes barcodes and QR codes.
Actually, the recognition target is wide, besides faces and letters, lesions (cancer, inflammation, etc.), types (animal and plant names, distinguishing between weeds), products (used in unmanned stores), people (crime prevention). , failures (products, infrastructure deterioration, etc.), and so on.
There are various classification standards, but the three above give the impression that the classification by technology and the classification by subject are mixed up. Therefore, in this paper, we will classify image recognition from the following three perspectives and systematically understand it.
Image recognition can be broadly classified technically into object detection, which detects the position of a specific class in an image, and object detection, which detects feature points. It is divided into classifications that analyze and recognize what instances it is (Fig. 1).
Figure 1: Object detection and image recognition
The class identified by Object Detection is English for "a collection of similar things". For example, "human", "car", and "dog" are classes. On the other hand, the word "instance" identified by Classification is the English word "genutsu". The target is specified as a more specific real thing (instance), such as "Mr. Yamada", "Supra", or "Saint Bernard".
Figure 2 is a demo of the face recognition service "Face API" provided by Microsoft's Cognitive Services. Object Detection is the technology that encloses the position (and size) of the face in each image, and Classification is the technology that analyzes the feature points of two faces and determines that they are the same person. In this demo, it is judged to be the same person with a confidence level of 0.90852. If you have a photo registered in your profile like Facebook, you can even name it "Umeda".
Figure 2: Demonstration of Microsoft "Face API"
Image Recognition technology is huge and deep, and there are various accompanying technologies such as segmentation (area recognition), scene (situation recognition), and emotion (emotion recognition), but here we will focus on the two types of technology that are familiar with smartphones. Just understand the difference in roles.
This time, we will introduce examples of image recognition utilization, focusing on the second category, "classification by industry." I think you can imagine the third, "classify by target", together in the explanation.
・Amazon GoUnmanned convenience store opened by Amazon in January 2018 "Amazon Go" surprised me. If you install the app in advance, register your credit card, hold the QR code over the gate and enter the store, all you have to do is put the product from the shelf in your bag and leave the store. It is a mechanism that allows you to pay by credit card without paying at the cash register (Fig. 3).
Figure 3: Shopping on Amazon Go [Source] NEC website
The image recognition used here consists of "behavior tracking" that follows the location of the person after matching the QR code and the person, and "link between the product and the person" that determines who has taken the product. With "Tsuke", payment will be automatically made at the moment the person leaves the exit gate. In addition to cameras, weight sensors on product shelves are also used to determine who took the product.
In the beginning, we had a goal of opening 2,000 Amazon Go stores across the United States in 10 years, but we have not yet reached 30 stores at this time, probably due to the impact of the new corona. I don't know if it's stagnant or if it's going to accelerate, but smart stores are opening one after another in the United States, China, and other countries. Recently, the number of cases in Japan has been increasing little by little, so let me introduce some of them.
・TOUCH TO GO Co., Ltd. TOUCH TO GO Co., Ltd. is a company established in July 2019 as a joint venture between JR East Startup Co., Ltd. and Sign Post Co., Ltd. After demonstration experiments at Omiya Station in 2017 and Akabane Station Kiosk in 2018, the unmanned AI payment store "TOUCH TO GO" opened in Takanawa Gateway Station in March 2020 and became a hot topic. The basic mechanism (tracking a person's behavior, deciding whether to pick up an item, etc.) is the same as Amazon Go, but this method does not require QR code authentication when entering, and instead uses a touch panel to process payments when exiting. is.
TOUCH TO GO has developed this mechanism into a solution and is expanding sales to retail stores and restaurants.
＜＜Note＞＞Smart store and customer base buttonI used to call a store like Amazon Go an unmanned store, but in reality, there are clerks doing work other than the cash register. For this reason, the terms “smart store” and “smart convenience store” have recently been used.
By the way, old convenience stores used to have a "customer class button" that the clerk could press based on their appearance. There were buttons for about six levels of age groups for men and women. Recently, cashless payments have increased, and there are issues with credibility, such as entering the age of 49 for all busy days, and Lawson and Family Mart have abolished it.
It seems that 7-Eleven will continue even if it is replaced with the self-checkout that we often see recently, but as it becomes smarter, more information can be obtained accurately with image recognition, so I expect that it will be abolished eventually. I'm here.
・Seven-Eleven Seven-Eleven has been working with NEC to demonstrate facial recognition payment (limited to Seven-Eleven employees) at the Kojimachi Ekimae store since March 2020. If you register your face photo, credit card for payment, and confirmation code in advance, you can make a face pass payment at the face recognition self-checkout. However, since products require a barcode scan, I have the impression that it is more efficient to scan all at once at the end of a normal self-checkout.
・NEC In February 2020, NEC opened the “NEC SMART STORE” in its headquarters building. This is also limited to NEC Group employees, but if you register your photo in advance, the gate will open with a walk-through (face pass) by facial recognition when entering the store. As with Amazon Go, the camera and weight sensor are used to link the product with the purchaser.
・Lawson and Fujitsu In cooperation with Fujitsu, Lawson will open the “Fujitsu Shin-Kawasaki TS cashierless store” for a limited period of three months from February 2020, and conduct a demonstration experiment of a cashierless store. I was. It uses the same method as Amazon Go, where you scan a QR code to enter the store and do not need a cash register when you leave, and the store operation system is "Zippin" by VCOGNITION TECHNOLOGIES in the United States.
・Koyo Shop-Plus In January 2021, Fujitsu is conducting a demonstration experiment at Koyo Shop-Plus and the cashierless store "Green Lives Plus Yokohama Techno Tower Hotel". Here, multi-biometric authentication is combined with "Zippin" so that not only a QR code on a smartphone, but also palm veins and facial information are combined to identify an individual and enter the store.
・Daiei In July 2019, Daiei will conduct a demonstration experiment of week-through payment (payment without a cash register) with the same mechanism as Amazon Go at the Showa Women's University student lounge for one week only. I did. In addition, in September 2021, NTT DATA's "Catch & Go" service will be used to open a "walk-through store" at NTT DATA headquarters to verify the smart store.
・DIME LOUNGE STOREIn April 2021, an AI unmanned store opened in Shinjuku for a limited time in April 2021 as a demonstration project by the Shogakukan DIME editorial department, Maruzen Junkudo, and Secure. "is. The system uses Secure's AI STORE LAB. Only those who have pre-registered can enter the store with a face pass, but registration can be done with the authentication machine installed at the entrance. This method also uses a camera and weight sensor to determine whether the product has been picked up, and finally allows payment to be made using face recognition.
・BakeryScan A smart convenience store prepares dozens of cameras in the store and installs weight sensors on product shelves. The device is magnificent, such as preparing The AI image recognition cash register is a simpler mechanism that allows you to make a quick payment simply by installing a camera at the cash register. Since 2013, Brain Co., Ltd. has been developing BakeryScan, a cash register for bakeries.
Mechanism is a typical machine learning model. The AI learns the image of the bread in advance, and when the tray is placed under the camera, the type, quantity, and price of the bread are displayed at a glance. The company is expanding this system horizontally, and has also released the AI cash register "Sweets Scan" dedicated to Western confectionery. It's the perfect solution for breads and cakes that are difficult to barcode.
・ViscoveryTaiwan's Viscovery also develops and sells an image recognition cash register "Visual Checkout" using AI. The principle is almost the same as BakeryScan, and the stores that use it are also covered by bakeries, pastry shops, and delicatessens.
・Kyocera's unmanned checkout systemThe system that allows you to make a quick payment with a checkout camera like BakeryScan or Viscovery is likely to spread in the future. One of the new entrants is Kyocera, which announced in June 2021 a "smart unmanned checkout system" equipped with object recognition AI technology. This is the stage where research and development is completed, and it is planned to be put into practical use by 2023.
・NTT DATA LEWEAVE CORPORATIONNTT DATA LEWEAVE CORPORATION has also released an automatic cash register “CoolRegi” that uses AI image recognition. This is mainly targeted for use in student cafeterias and employee cafeterias. The object of image recognition is not the food, but the plate. It is a mechanism that performs machine learning on tableware in advance, recognizes the image of the tableware that has finished eating, and immediately calculates the price.
・Robot Mart The “Robot Mart Hatchobori Store” opened by Robot Security Police Co., Ltd. in April 2020 also has AI image recognition that automatically detects the purchased products when the products are arranged on the cash register. I am hiring a cash register.
As the installation of in-store cameras has spread, the number of solutions that analyze the behavior of customers and clerks using image recognition has increased. In the past, it was common to analyze traffic lines using a beacon method that detects the position of a customer when they approach a beacon installed in the store, or a Wi-Fi access point. However, beacons and Wi-Fi have hurdles such as the need to install apps and connect to Wi-Fi, which has hindered their spread. With the evolution of image analysis, the number of camera methods is increasing, and the following analyzes are possible by customer face recognition and behavior tracking.
a. Customer analysis, visitor attributes by time zone (gender/age), customer flow line, stay time
b. Sales floor/aisle analysis ・Congestion situation in the store and sales floor ・Number of people stopping in front of the shelf, time, line of flow to the sales floor, time, sales floor stop-by rate, purchase rate, aisle traffic
c. Product shelf analysis ・Product contact and products returned to the shelf ・Purchase rate ・Customer gender/age and linkage of purchased products
d. Store analysis ・Equipment usage (fitting room, etc.) ・Congestion in front of the cash register (queue/time/prediction) ・Store entry rate, number of people entering the store, length of stay ・Purchase rate (number of purchasers/number of customers)・Comparison between stores
e. Clerk analysis, customer service frequency and customer service time, work analysis by clerk
f. Crime prevention, mask wearing check, temperature measurement, suspicious person detection, notification, unauthorized entry detection
g. Number of passers-by in front of the store by location (camera outside the store), time of day, customer base, stopping in front of the store, store entry rate
There are many vendors that provide behavior analysis services using camera images. However, we are still at the stage of practical use in actual stores, so here we will introduce four services that introduce case studies on the website.
Table: Camera (image authentication) type customer behavior analysis tool
| Service Name | Vendor | Published on Vendor HP Main examples |
|---|---|---|
| Flow | Flow Solutions Inc. | BAYCREW'S, Triumph, Toys "R" Us, DAYTONA, … |
| ABEJA INSIGHT for Retail | ABEJA Inc. | Sanyo Shokai (Macintosh Philosophy /LOVELESS), Tokyo Shirts, BEAMS, ICI Ishii Sports, … |
| Moptar | Supreme System Inc. | Kirarina Keio Kichijoji , AQUAIR, … |
| FollowUP | Data Section Inc. | Ships, Cosme Next, Amer Sports, Kojisanso, … |
In-store cameras were originally mainly for crime prevention purposes. By taking pictures of shoplifting by customers and part-time terrorism by staff members, evidence of crimes is left in the form of images, which also serves as a deterrent. Since it will be too late after it is stolen, AI can be expected to detect shoplifting in real time if it can recognize the action of putting the product in your bag like in a smart store.
You can go one step further and use it for prevention. It is a system that uses machine learning to detect shoplifting behavior and notifies the store clerk in real time if there is a suspicious customer. The store clerk who receives the notification will be effective just by asking, "What do you want?"
Recent supermarkets have self-checkouts and I often use them. It is convenient because there is little waiting time at the register, but unlike Japan, in the United States, it seems that there are many thefts called skip scans, where cheap items of about the same weight are scanned and then expensive items are put in a bag. For that reason, Walmart uses an AI image recognition system called Missed Scan Detection to detect fraud in real time whether the product is properly scanned.
Image recognition functions have been added to digital signage used inside and outside stores. For example, as soon as you pick up a product, detailed product information will be displayed on the signage, or the customer base (gender and age group) will be determined, and advertisements and information that suit the customer will be displayed.
It's a little fun to imagine a scene where a signage is placed at the front of a restaurant, and the elderly are tempura zaru, if it's a tough type, it's a katsudon set, and if it's a young woman, it's an omelet rice. but).
Signage with speech synthesis is also coming out. Although it is not a store, Toppan Printing's multilingual AI guidance signage at Takanawa Gateway Station uses AI Talk, the voice synthesis engine introduced last time, for Japanese voices.
The first installment of AI image recognition was its use in stores. There are many development cases and utilization cases where vendors have announced that they can do something like this, but I have the impression that there are not many introduction cases yet. Vendor releases tend to be announced by the Imperial General Headquarters, but seeing is believing, so please take a look at the introduction examples and get a feel for how it is being used. However, I have the utmost respect for each vendor's stance of desperately taking on challenges toward a new society. I pray that the flowers will bloom soon and that the efforts of each company will bear fruit.