Introduction of SDK for Polarization Camera

■ Introduction
Polarization was discovered by Christiaan Huygens in 1690. In the more than 300 years since, this physical phenomenon has been researched and applied in many fields. In the field of photography, polarization filters are attached to cameras
when taking photographs, to suppress the reflection of light off surfaces of water and glass. In industrial applications, polarization filters are utilized in manufacturing inspection processes to reduce the unwanted reflection of light used when checking for problems such as foreign material, defects, and contamination.
At present, polarization cameras are attracting a great deal of attention in the field of industrial cameras. In September 2018, Sony Semiconductor Solutions began mass production of the IMX250 polarization image sensor equipped with
"Polarsens" polarization technology. In December 2018, we participated as exhibitors at the International Technical Exhibition on Image Technology and Equipment held in Yokohama, where more than 10 companies exhibited polarization cameras. We observed strong interest from visitors at the event.

Sony developed the XCG-CP510 polarization camera, which is a 5-megapixel machine vision camera with GigE interface, along with the XPL-SDKW specialized SDK (for Windows) for the polarization camera. Both items began shipping in December 2018. Here, we introduce some of the features and functions of the polarization camera and SDK.

■ Transition from polarization filter to polarization camera
As mentioned above, polarization filters (also called PL filters) are a useful accessory for reducing the unwanted reflection of light, such as when taking photographs of natural scenery with a consumer-use camera. To reduce reflection, the user is required to rotate the filter to find the most effective angle. Since the optimum filter angle varies according to position of the sun or the angle of the camera relative to the subject, it is necessary to make the adjustments each time.
When time is not an issue, such as when taking photographs of natural scenery, the time required to adjust the polarization filter is not a major problem. However, in situations such as when checking manufactured products on a production line, it is not realistic to make the adjustments each time. In such situations, it is important to set the lighting, positional relationship with the subject, and filter angle in advance, to establish shooting conditions in a constant environment where the need to perform adjustment for each individual product item is eliminated.
In the past, inspections for defects and contamination were performed visually through a polarization filter. However, the appearance of open source image recognition processing software now makes it easier to embed image recognition
technology into software, which has led to greater progress in the automation of inspection operations. Since images captured through the filter comprise static information that includes the state of the filter, and because it is not possible to
adjust the filter angle after the image is captured, the installation of the filter is now more important than ever.
For application other than reflection removal, previously it was necessary either to capture multiple images while changing the angle of the polarization filter, or to install multiple cameras to obtain the degree of polarization. This
presented problems in terms of the amount of time and effort and the degree of accuracy due to parallax. In that respect, a major feature of our polarization camera is the ability to calculate the state of the polarization filter at an any angle in one shot, by combining a four-directional polarizer with subsequent signal processing. Since the information of the four-directional polarizer is saved for each captured image, the image information can later be used to recalculate the filter state at any angle, and images can be viewed offline after they are captured.

■ Configuration and functions of SDK
Here, we introduce the polarization camera SDK for Windows (XPL-SDKW), which Sony developed.
XPL-SDKW, which is provided by Windows Installer, consists of the following components:
 SDK main files (dll)
 API documents
 Sample source code
 Tool for obtaining license ID
 Sample viewer (XPL Viewer)
One feature of the SDK is that it provides seven polarization algorithms for handling transmitted and reflected light. In the sections below, we introduce the various SDK functions that are available when using the sample viewer included with the SDK.

■ Degree of Polarization (DoP)
This function calculates the degree of polarization of the subject, and displays the image so that areas with a higher degree of polarization are indicated by a higher degree of whiteness. For subjects such as bundled power cord, which are difficult to visually inspect due to low contrast and underexposed areas in images, or subjects that are difficult for the camera to recognize because they are the same color as the background, the shape of the subject can be checked by obtaining the degree of polarization.

■ Surface Normal
This function can calculate the normal directions for each surface of the subject based on the polarization information. In the SDK, the surface normal is calculated for each pixel of the subject. There is also a function to apply colors to he calculated surface normals, so that the different surface directions can be expressed in various colors. As with the Degree of Polarization, this function can be used for objects that are difficult for an ordinary camera to recognize when they are the same type of color. Since the surface directions are made more visible and the object has a three-dimensional appearance, this function can be used for 3D measurement or shape recognition.

■ Retardation
Retardation refers to the stress that is applied inside an object. Retardation occurs when an external force is applied to an object, and it can remain in the object even after the external force is removed (residual stress). Materials in which the amount and direction of birefringence changes according to the amount and direction of distortion when retardation occurs are called photoelastic materials. When polarized light is applied to a transparent or translucent photoelastic material, a phase difference occurs in the birefringent polarization based on the amount of distortion. The amount of retardation occurring in an object can be checked by measuring this phase difference. The observed phase difference is visualized and displayed in color in the sample viewer.

■ Reflection Cancel / Enhance / Extract
A property of unpolarized light is that it becomes polarized in a specific direction after it is reflected off the surface of an object. A polarization filter cancels reflected light by blocking polarized light in this specific direction. Unlike polarization
filters, which are only capable of canceling reflected light, our polarization camera also makes it possible to enhance the display of reflected light or extract only the reflection component, by performing numerical calculation processing in the
SDK. One usage of reflection cancellation is that it can improve visibility by removing reflected light from objects such as the windshield of a motor vehicle or the surface of water. On the other hand, reflection enhancement makes it
possible to identify the position or shape of an object with high transmittance, such as a glass plate, which is difficult to achieve with a normal camera, by enhancing the reflection component. Manual and Auto modes are available for the reflection cancellation function in the polarization camera SDK, so that the optimum cancellation method can be selected to suit the application.
 In Manual mode, it is possible to cancel reflected light, which can by nature only be completely canceled at Brewster's angle, by manually specifying the angle of the reflection surface.
 In Auto mode, the polarization (reflection component) of the subject is obtained for each individual pixel, so that reduction can be performed automatically.

■ Stokes Vector
The Stokes vector consists of four components called Stokes parameters, S0, S1, S2, and S3, which are combined into a column vector to represent the state of polarization, expressed as S=(S0, S1, S2,S3)t. Each Stokes parameter represents
the intensity of light that can be measured. By defining the Mueller matrix M and performing matrix calculation in the Stokes vector, it is possible to understand the change in state of light after it passes through the optical element (S’=MS).
This Stokes vector can be obtained from the SDK, and the state of each Stokes parameter can be displayed in the sample viewer. Note, however, that since S3 cannot be obtained by the polarization image sensor, it is not handled in the
SDK.

■ Characteristics of Sony SDK
This section describes the unique characteristics of Sony's SDK (except for the polarization algorithms). 

■ Use of demosaic processing to improve image quality
Although the polarization image sensor has the advantage of obtaining the polarization in four directions in one shot, it also has a disadvantage in that the image in each polarization direction has 1/4 the total resolution in terms of total
number of pixels, because the numerical calculations are performed for the information in four directions. In the case of the IMX250 polarization image sensor in the XCG-CP510, the maximum resolution is 2464 × 2056 with approximately 5.07
million effective pixels. However, since each pixel contains the polarization information in four separate directions, each corresponding image has a resolution of 1232 × 1028.
In addition, since manufacturing tolerances allow for some variation in each directional polarizer in the polarization image sensor, the sensitivities may vary slightly. A grid pattern can be observed in the sample viewer if Raw output is selected and the image is enlarged. This is due both to the polarization information and the variations in sensitivity. As a result, the information from the polarization image sensor is different from the image that is viewed by the human eye, and the image appears as if mosaic processing has been applied.

For this reason, demosaic processing is performed in Sony's SDK to convert the mosaic image so that it more closely resembles what is visible, by applying pixel interpolation in each direction and adjusting the spatial phase shift before the polarization algorithm calculation is performed. Our unique demosaic processing, which is optimized for the optical element array, enables the amount of information to be restored to the same amount as for the effective pixels of the polarization image sensor, while simultaneously improving the image quality.

■ Flat Field Correction (FFC)
As described above, the polarization image sensor element has some variations in sensitivity in each pixel and in each polarization direction. In addition, the phenomenon of "shading", which causes the outer edges of the image to appear
darker, may occur when using a camera with a lens. To correct for this, the FFC function is implemented in the SDK. The camera itself is equipped with a shading correction function, but a large amount of calculation is required for the camera
to handle 5.07 million pixels, which makes it difficult to perform complete correction. High-quality correction is achieved by performing it on a personal computer that uses a GPU for the calculation resources.

■ Use of GPU to achieve high-speed processing
The polarization algorithm is required to perform numerical computation for all pixels of images that are output from the camera, which makes it difficult even for the most advanced CPU to perform real-time processing for the entire image.
When processing is performed solely by a CPU, the only available option is to reduce the calculation amount, either by reducing the size of the image that is handled, reducing the frame rate, or simplifying the polarization algorithm. However,
depending on the application of the polarization algorithm, these measures may result in some limitations for the application. This is not appropriate for the SDK, which does not identify the application. In recent years, the role of GPUs in personal computers has evolved beyond simply performing image processing, so that
it is now common to use the high parallel computing capabilities of GPUs for large-scale numerical computations. In the polarization camera SDK, high-speed processing is achieved by using a GPU for calculation processing, for the purpose
of performing real-time polarization processing of images output from the polarization camera. In the past, when using GPU (a GPU for calculation), the implementation method depends on the GPU vendor or platform (CUDA, AMD Stream, DirectCompute, etc.). Presently, however, GPU can be performed independent of GPU vendors by using OpenCL (standard developed by Khronos Group). Since OpenCL is used for the polarization processing in the polarization camera SDK, recent Windows personal computers can achieve processing at higher speeds in many environments, compared to systems that use only a CPU. For processing at full resolution and full frame rate, use of a high-performance GPU such as the NVIDIA GeForce GTX 1070/1080 is recommended.

■ Online and offline support
In addition to online use (where the polarization camera is connected), the polarization camera SDK can also handle offline Raw images (which are saved as files). Raw images can be saved as files that can be loaded later, and all polarization processing can be reproduced. This enables users to take time when, for example, selecting the optimum angle for reflection cancellation.

■ Conclusion
As exhibitors at the International Technical Exhibition on Image Technology and Equipment, we had the opportunity to speak with many visitors about polarization cameras. We were struck by the realization that the potential market for using
polarization cameras is much greater than we imagined, for a vast array of subjects ranging in size from microscopic cells to large areas of land, and in a broad spectrum of fields spanning medicine, manufacturing, transportation, research,
space, and sports. Although we have long known about the world of polarization, the recent emergence of new polarization camera technology has led to the development of new application methods and prospects for significant growth moving forward. It is our hope that the camera and SDK developed by Sony will play a key role in this regard. 

Please refer to the following URL for product details.
 https://www.sony.co.jp/ISPJ/

Polarsens is trademark of Sony Corporation. All other trademarks are the property of their respective owners.