How do the robots in Schubert’s fully automatic packaging machines know which product they are picking up and placing down, when, where and how? They can actually see. In TLM systems, image processing performs the function of the human eye. The team supporting Dr Abdelmalek Nasraoui, head of image processing development at Schubert, works every day with these vision systems to ensure that the robots have everything in view that is necessary for the efficient and high-quality packaging of products from almost any industry. Schubert is regarded as a pioneer in image processing in the packaging sector and continues to advance its in-house technologies.
The new 3D camera system brings to life the concept of fully automatic “bin-picking” for complex products in any position. Whether unsorted, shifted or even lying on top of each other, the 3D image processing system’s four cameras and two light pattern projectors accurately recognise fed products of all shapes and colours in terms of position, height and orientation. This means that the robot tool can precisely pick up components that fed in an unformatted pattern, eliminating the need for complex sorting and separating systems. After each robot intervention, the objects are again recorded three-dimensionally, and the position and orientation of the next product to be picked up is calculated. For this reason, the product intake also functions reliably if the arrangement of the other products changes due to the removal of one product.
The new powerful camera system generates very high-resolution 3D images within fractions of a second. This makes it possible to quickly recognise the height and quality of an object in addition to its position and orientation, which ensures that the robots can pick up products in any position in three-dimensional space. The two camera pairs offer the best all-round view and fewer gaps in the image than conventional single-shot 3D camera systems. Additionally, there is a projector between each camera, which throws a random light pattern onto the products. Light pattern projection and stereo image processing are intelligently combined to create optimal point clouds. The height and surface control is a central function in the recognition of products in 3D space and in their qualitative evaluation.
The vision system’s image data and the height profile are displayed on the operator terminal. Each product to be detected is characterised by a corresponding set of parameters, which is accessible to the plant operator via an associated menu. The type and number of parameters are application-dependent and can be individually adjusted. Depending on the task, they are defined in advance and stored in the control system. No adjustments are necessary during operation.
Schubert engineers developed this new 3D camera system on the basis of their own existing image processing systems. Schubert has been working on the development of image processing for packaging robots since the early 1980s. In 2014, the transmitted-light, reflected-light and colour-light scanners were followed by the 3D scanner, which raised product recognition and quality control to a new level of performance. The specialists in Crailsheim also developed and recently further optimised this 3D scanner, which had been successfully launched on the market. In the future, the 3D scanner will deliver an even higher resolution. In addition to improved image quality, the optimised 3D scanner also impresses with a higher throughput. “Through intelligent hardware implementations of 3D image processing algorithms, we are now processing large amounts of 3D image data even faster,” says Abdelmalek Nasraoui. The Schubert 3D scanner is unique worldwide: it provides colour and height images and is available in various “lengths” as a compact system component, depending on the system’s bandwidth. Every 100 millimetres, high-resolution colour line scan cameras and light pattern projectors are alternately installed in the 3D scanner. Each camera shares half of its field of view with its neighbour. In accordance with the principle of stereoscopy, there are two images of each product from different viewing angles. Product height can be calculated based on these, as well as from the light samples. This system offers maximum system integrity. The 3D data is available as soon as the products leave the area under the scanner.
The image processing experts at Schubert are also currently researching an intelligent system for quality assurance. Artificial intelligence (AI) and deep learning are the buzzwords here: neural networks (in particular CNN, Convolutional Neural Networks) are to be trained in advance with several thousand product images and can then recognise whether a certain product meets the quality criteria and should be further processed. “Baked goods exhibit a high variance in shape and surface. Additionally, there are a variety of possible defects per product class. These two factors make the mathematical description of the shape and quality characteristics a challenge. Neural networks are being used increasingly as a powerful instrument to accomplish such complex tasks,” said Abdelmalek Nasraoui. As with the 3D scanner and the new 3D camera system, the implementation of neural networks in Schubert’s packaging machines represents further milestones in the company’s own image processing.