One service that has become popular in the last few years is AR. AR is an abbreviation of "Augmented Reality", and when you look at the real scenery through a smartphone or smart glasses, you can see images and data that are not actually there, that is, virtual visual information. are superimposed to extend the real landscape.
In conventional AR, characters and text information can appear in the same way no matter where you are looking at the scenery with a smartphone or smart glasses. On the other hand, what is necessary to make AR appear in a specific place in the landscape is the "VPS (Visual Positioning Service)" being developed by KDDI Research Institute. VPS is a technology that identifies in real time "what direction you are facing from where" when viewing the actual scenery with a smartphone or smart glasses camera.
Then, what are the advantages of specifying the position where AR appears? In what situations will VPS be useful in the future? And how does it work? Let's take a look at some concrete examples.
Please watch this video first. Demonstration in the "Shibuya Entertainment Tech Promotion Project" launched by KDDI, Shibuya City Tourism Association, and Shibuya Mirai Design to make the city of Shibuya more interesting by making full use of advanced technology centered on 5G. It's made by experiment. I am experiencing AR displayed at the scramble crossing in Shibuya through my smartphone, but VPS is used here.
A shark swims in front of a building with a large vision on the glass. And then tropical fish pass by the shark. If you turn your eyes to the sky, the temperature and humidity are arranged so as to look down on you.
Whales swim along the streets of the scramble crossing, and balloons float in the correct position pointing toward Aoyama and Harajuku because the VPS knows where you are and which direction you are facing. It is possible because it is specified.
Create a 3D map of the area around the Shibuya scramble crossing in advance, and set which animations and characters will appear at which positions in the map. If you look around the target area with a smartphone or smart glasses camera, the VPS will compare the building such as "SHIBUYA TSUTAYA" projected there with the 3D map to find a matching point, and the smartphone or smart glasses will " Find out where you are and what direction you are facing.
By accurately specifying the position and orientation of the user, it is possible to display the various visual information that makes up the AR, superimposing it on the real landscape without any sense of incongruity.
The feature of VPS is that users only use the camera and do not need a special sensor. By accessing a pre-created 3D map with a smartphone or smart glasses and finding points that match the scenery actually seen through the camera, you can accurately identify your position and orientation.
With VPS, you can accurately identify your location and orientation not only outdoors like the Shibuya Scramble Crossing, but also indoors.
For example, in March 2021, the demonstration experiment "HYPER LANDSCAPE" was conducted in collaboration with "National Museum of Emerging Science and Innovation" and KDDI. When you wear smart glasses, a virtual human "coh" appears in AR that has visuals that are as close as possible to humans and explains with natural gestures and intonation. It will guide you through the exhibits. Here is a video that summarizes the event.
As you go through the spiral corridor in the museum, coh will appear several times and explain the contents of the exhibition. The position is specified by VPS, and coh appears in the optimal position without shifting from the corridor of about 2m in width.
How exactly does this VPS identify the location of smartphones and smart glasses? We asked Kenji Komorita of KDDI Research Institute, who has been working on system development.
Kenji Komorita, Media Recognition Group, Media ICT Division, Advanced Technology Research Institute, KDDI Research Institute By analyzing the 3D map created in advance and the image captured by the smartphone camera using AI, it is possible to determine the exact position and orientation of the smartphone.The production of 3D maps differs depending on the VPS technology. In the AR video example at the beginning, the area around Shibuya's scramble crossing is created from satellite images. Also, in the demonstration experiment of the virtual human "coh" at the "National Museum of Emerging Science and Innovation", I walked around the venue with a 360-degree camera and created it.
Taking the National Museum of Emerging Science and Innovation as an example, a spiral corridor surrounds a gigantic globe suspended in the hall's atrium.
Around "Geo-Cosmos" at "Miraikan" where 3D map was created for use in VPS
If you make a 3D map representing this place with a point cloud, it will look like the video below.
In the 3D map created in this way, if you launch an AR app on your smartphone or smart glasses and take a picture of the surroundings, you can identify in real time where you are and which direction you are facing. You can.
"AI recognizes the scenery captured by smartphones and smart glasses as 'patterns.' , judge against the 3D map.
This is a picture of our office. To the naked eye, it looks like a normal room like the one on the left, but VPS finds many features like the one on the right and recognizes it as a point cloud.
Left: Image taken with a camera inside KDDI research atelier Right: Point cloud when VPS analyzes the image , and if there are employees or other people moving here, the AI will exclude them.”GPS is a well-known technology for recognizing one's position using a smartphone. I wonder if there are strengths unique to VPS.
Kenji Komorita, Media Recognition Group, Advanced Technology Research Laboratories, KDDI Research Institute The location is determined by the time difference between the arrival of radio waves from multiple satellites orbiting the earth to the terminal, but the error is up to several tens of meters.Also, it is not possible to identify the orientation without using a compass built into the terminal.KDDI General In the case of the National Museum of Emerging Science and Innovation, the error of the VPS technology at the laboratory is about 35 cm on average, and the angle is about 1 degree.Moreover, GPS that uses satellites is not good at estimating positions indoors, but in the case of VPS can accurately determine its position and orientation even indoors.”Such a VPS is beginning to attract attention, and various companies are working on its development.
KDDI Research Institute's technology judges from a 3D map with patterns as feature points, as explained in the example of the National Museum of Emerging Science and Innovation. It is possible to create a 3D map simply by photographing the environment with a 360-degree camera, etc., and the strength is that the position and orientation can be specified without being affected by the weather or the brightness of the surroundings.
Also, like Shibuya's initiative at the beginning, there is also a method of generating a 3D map from satellite images with AI and identifying the location from the shape of buildings instead of "patterns". You can experience it using the application called "XR CHANNEL" with the technology promoted by the American company "Sturfee", which has a partnership with KDDI.
This technology was also experienced at the home game of soccer J1 Nagoya Grampus in September 2020. As KDDI "au 5G Special Match Day!", players and mascots appeared in AR around Toyota Stadium and the nearest Toyota City Station.
The AR was perfectly matched to the station square, the wall of the building, the entrance of the stadium, etc., and I was able to see it without any discomfort. On the day of the event, many fans were seen holding up their smartphones over buildings and stadiums to enjoy the event.
VPS is a technology for accurately identifying a location. In particular, in cooperation with AR, which has a high affinity, services that are easy to use and can be enjoyed will be developed.
By using this technology, you can not only view services that are convenient in your life, such as navigation that guides you to your destination with arrows, but also images of office layouts and furniture installations. It is thought that the movie, stage, and event information in the movie can be displayed as an advertisement in the city, and the images and videos that can be displayed in a specific place can be incorporated into escape games, etc., and the fun can be expanded as a new entertainment production. .
On the other hand, VPS can be used for things other than AR. Just by looking at the surrounding scenery with a smartphone or smart glasses, the location can be pinpointed, so for services such as delivery, for example, instead of entering the address, just share the surrounding scenery and it will be delivered to the spot. becomes possible. Even if you're sitting on a bench in a large park and you don't have an address, and you can't even explain where it is, you can still get it delivered.
In fact, in the KDDI Research Institute facility called "KDDI research atelier", an experiment in which robots deliver food using VPS has also begun. In the first place, there is no address in the office, but by sending a video of the “landscape” of the place where you are, such as your desk, you will be able to deliver the food.
A delivery robot using a VPS that is being tested at "KDDI research atelier". You can order food from any place in the facilityVPS is a technology that is difficult to see from the outside. However, with the advancement of such technology, the services that make our lives more interesting and convenient will continue to evolve.
Let's go see AR Snoopy! ] Currently, when you launch the XR CHANNEL app in some 5G areas of au, a three-dimensional Snoopy using VPS will appear. The corresponding area will be expanded gradually. * We use the VPS technology of "Sturfee".
Please enjoy posting photos and videos taken with Snoopy that appeared on SNS.