3D VISION – AN OVERVIEW OF TECHNICAL ADVANCEMENT AND APPLICATIONS

3D camera vision systems have a remarkably diverse range of applications across multiple sectors. From machine vision and robotics, through to smart cities, autonomous vehicles and healthcare, the ability to extract 3D information from 2D images holds the key to numerous exciting technological advances and use cases.

The capability is underpinned by three primary technologies – stereo cameras, Time-of-Flight (ToF), and LiDAR. Each approach has its essential characteristics and, therefore, its own strengths and weaknesses, making it more suitable for some applications than others. In each case, rapid advances in the image-sensing systems that underpin approaches to 3D camera vision are opening up new possibilities.

So, let's look at the three primary approaches to 3D camera vision.

1. 3D stereo camera

3D stereo cameras operate similarly to human vision. With the pupillary distance of our eyes set apart by between 50-70mm, our eyes capture two-dimensional images from slightly different perspectives. Our brains combine these views and work out their disparities to perceive depth and construct a 3D representation. In image sensing, 3D stereo cameras use more than one camera at established positions to produce 2D images from varied angles. These images are then processed through techniques including triangulation to establish depth and create the 3D interpretation.

One of the core benefits of stereo vision is its ability to perform real-time processing, making it a neat fit for dynamic environments where fast decision-making is critical to operations. That is a beneficial attribute for machine vision, which underpins a broad range of activities in automation and robotics.

Stereo cameras can, for instance, help robots understand their environment and, therefore, navigate and interact with objects for careful manipulation. A stereo camera system might allow a robot arm to select and pick up an object from a conveyor line to confirm quality or for bin picking or crate handling. There is also an emerging role in mobile robotics, which is finding widespread adoption in warehouses for material transportation, with stereo cameras capable of offering high frame rates, low latency, wide field of view, and high dynamic range for indoor and outdoor applications. Stereo vision lends itself well to environments such as manufacturing and logistics, as it is relatively low-cost and can be ruggedized for harsh operations. Yet potential downsides include low-light performance and high computational requirements.

2. ToF sensors

ToF cameras use a different technique to achieve 3D representation based on the principles of light illumination. Here, the camera sensor emits light – usually but not exclusively from the infrared spectrum - to the target object and collects its reflection. The distance or depth to the target object is then calculated using the time it takes for the light to return to the sensor. Sophisticated algorithms then create a fully-formed depth map, creating a 3D representation of the scene.

ToF sensors offer numerous operational benefits, including rapid depth measurement and high accuracy in different lighting conditions. Consequently, automotive is an exciting emerging application area, especially for use cases such as advanced driver-assistance systems (ADAS) that can support obstacle detection, adaptive cruise control, and parking assistance.

There are plenty of in-cabin monitoring applications for automotive, too. For instance, the VIZTA academic research project has used Tof F sensing to monitor the driver's presence, position, and movements. This data can improve safety features such as driver alert systems while enabling gesture control for vehicle functions. It uses advanced sensors and machine learning to keep a close eye on what is happening inside the car to boost safety and enhance the driving experience.

Intriguingly, such technologies could become self-learning and self-adaptive, understanding vehicle occupants' unique behaviours and preferences and leading to a smarter and more responsive in-car environment. This would augment efforts to develop more intelligent and autonomous vehicles with better safety and convenience.

3. 3D LiDAR

Finally, 3D Light Detection and Ranging (LiDAR) uses pulsed light from a laser light source. Hundreds of thousands of laser pulses are fired off every second, reflecting off objects in the surrounding field of view. The LiDAR system then calculates the time each pulse returns to build up a precise 3D map of a dynamic environment. Cameras can then be used with LiDAR to capture visual data fused with the depth data from the LiDAR to create a more detailed 3D representation.

Now imagine LiDAR systems deployed in a smart city to manage complex urban situations. It is accurate, fast, relatively low-cost and can provide consistent results. Such systems could be used for traffic and pedestrian monitoring, as well as for understanding the movement of people and objects in 3D. They could be used for security management, tracking suspicious activity or overseeing crown management and disaster response. It could be used for city planning, allowing urban designers to visualize the state of infrastructure accurately and better optimize public spaces such
as roads and buildings.

In summary, 3D camera vision systems—including 3D stereo, ToF, and LiDAR technologies—are advancing rapidly. Sony Image Sensing Solutions (ISS) is a trusted partner and key supplier for this market, providing many fundamental building blocks for these exciting technologies. As innovation in 3D vision technology accelerates, Sony ISS will continue to play an essential role in driving breakthroughs to meet industry needs.