D IGITAL IMAGE STABILIZATION FOR THE EXPERIMENTAL VERIFICATION OF OBJECT MOTION TRAJECTORIES

– The article presents digital stabilization procedure of the image recorded with a camera performing unintentional spatial movement in relation to the scene. This method consists in carrying out translational and rotational transformations and scaling all frames in relation to the first frame, using the tools of the Matlab environment. The image stabilization procedure allows to effectively improve the image to make it suitable for the analysis of the tracked objects trajectories on the scene, observed from the camera located on the rotor drone.


INTRODUCTION
Computer processing, analysis and recognition of images is successfully used in many fields of science and technology, such as: geology, petrography, crystallography, materials engineering, metallography, robotics, meteorology, medicine, forensics, transport and others [1], [2]. 1 Computer vision systems (machine vision systems) relieve people in many processes of analyzing and recognizing certain classes of images, such as fingerprints in access security systems, or banknotes in vending machines. Vision systems extend the range of perception of our sense of sight, e.g. [1]: -in relation to observation in infrared, ultraviolet or X-rays or the perception of differences not detectable by direct observation, -enabling automatic, mass analysis and classification of millions of images, based on numerical criteria, with the use of devices that are not subject to emotions or fatigue, also where humans cannot reach. An important feature of computer image analysis is the non-invasive (non-contact) nature of examining physical objects and phenomena. During the examination, there are not disrupted the properties of the object nor its functioning.
In case of using a high-speed camera, it is also possible to 1 Article was created as part of the project NCBR POIR.04.01.04-00-0006/19. "Cross-belt system for automated sorting of unit loads by a two-stage method" register and analyze fast-changing processes, e.g. the process of automatic sorting or positioning of unit loads in warehouse picking systems [3], [4]. In laboratory applications, the cameras are attached to tripods (Fig. 1a), which ensure that the camera remains stationary in relation to the observed scene. a) b)

Fig. 1. Image registration with a digital camera using: a) a tripod, b) a rotor drone
In this way, the introduction of image disturbances due to possible camera movement is avoided. Thanks to this, the paths of movement, velocity and acceleration of objects determined on the basis of image analysis refer only to the observed objects, and not additionally to the movement of the camera relative to the scene.
In case of testing processes carried out in active production halls of companies, the installation of racks or scaffoldings for mounting cameras may be expensive or impossible due to the requirement to maintain the company's production continuity. A solution is expected that will allow for the omission of building infrastructure for mounting the camera and will allow it to be easily positioned in relation to the observed scene. The answer to this demand may be a rotor drone with a camera (Fig. 1b). Commonly available rotor drones have the ability to hover at a height and assume an angular position adjusted to the observed object on the scene.
The disadvantage of this solution, similarly to insufficiently rigid tripods, is the possibility of the camera lens moving relative to the observed scene. This imperfection can be self-compensated to some extent by image stabilization systems, often built into modern cameras, and even met in mobile phone cameras. There are two variants of this stabilization: --optical stabilization -it is implemented by a stabilization system with a gyro sensor that detects camera vibrations and moves the lens in the opposite direction to minimize the vibrations of the optical image directed to the image sensor, -digital stabilization -it is carried out by digital comparison of individual captured frames, automatically matching them to each other, creating the best possible frame [5][6][7][8][9].
Despite the use of cameras with integrated image stabilization systems, it may turn out that the recorded images will still show stabilization deficiencies. Then you can use digital stabilization methods that are available in programs designed to CAE (Computer Aided Engineering), e.g. Matlab [10], [11].
The presented article proposes a solution for image stabilization recorded with a digital camera using the Image Processing Toolbox and Computer Vision Toolbox functions available in the Matlab environment. The following sections present the procedure and results of the applied stabilization for two sample images recorded in the production hall. One image was recorded with a camera mounted on a rotor drone, and the other -with a camera attached to a frame with insufficient rigidity, which intensified the vibrations of the production hall floor.

I. DIGITAL IMAGE STABILIZATION PROCEDURE
In the proposed image stabilization procedure, the first frame is treated as the base frame, with which the remaining frames are compared.
The stabilization procedure consists of two main steps: -stage 1 -selecting control points P on the scene unmovable areas of the first frame ( Fig. 3, Fig. 9b) and searching for the coordinates of these points on the remaining frames, -stage 2 -applying a transformation matching subsequent frames to the first frame, based on the coordinates of control points P (Table 1). Stages 1 and 2 are performed in turn for each frame separately, except the first frame. The first frame is not transformed, as it is used only to define the initial position of control points P.
Selection of control points P in stage 1 in the first frame consists in indicating rectangular fields, which in Fig. 3 and Fig. 9b are marked in red. These fields act as templates searched on the remaining frames, using the TemplateMatcher function of the Matlab environment. Templates should not contain a uniform background, but details that are distinguishable from the background. To reduce frame processing time, the search for templates is limited to the ROI (Region of Interest) around the templates, marked in green in Fig. 3 and Fig. 9b. The geometric center of the ROI in subsequent frames coincides with the coordinates of the geometric center of the template detected in the previous frame. This means that the ROI on each frame has a different position, following the position of the template found in the previous frame.
In step 2, the frame is transformed using the fitgeotrans function, which applies a combination of three basic geometric transformations: translation, rotation and scaling.
This function can implement various types of transformations related to the expected geometric deformation of the frames (Table 1).
In the further part of the article, among the various types of transformation (Table 1), the projective type was used, which additionally takes into account image deformations caused by lens tilts relative to the recorded scene. The projective type requires four P control points to be defined.

STABILIZATION
The procedure of digital image stabilization described in the previous section was applied to two movies, which were recorded with a camera changing its position relative to the observed scene. Fig. 2 shows summary frame, which was created by overlapping 2636 frames recorded by the camera from the rotor drone.

Fig. 3. Control points P1-P4, marked in the first frame: 1 -identification templates (red rectangles), 2 -search areas for identification templates (ROI -green rectangles), 3 -traces of geometric centers of identification templates, located on 2636 frames
This drone was hovering above the prototype of the carousel drop of the two-stage sorter [12], during the commissioning tests of one of the most important components of this sorter [13]. Fig. 2a shows the image before stabilization, and Fig. 2b -after stabilization. The summary frame in Fig. 2a, created from the image before stabilization, is blurred, which means that the drone accidentally changed its position relative to the scene during image recording. The fact of the drone drifting is also confirmed by the traces of the geometric centers of the templates found in all frames -marked in yellow and by the reference (3) in Fig. 3. The diagram of scaling and rotation used during the transformation of individual frames, in relation to the first frame, is shown in Fig. 4. Scaling was carried out in the range of approx. 3%, and rotation -in the range of 3 o . The effectiveness of the image stabilization procedure is shown in Fig. 2b, which presents much greater image sharpness than the "raw" frame in Fig. 2a.
Displacements and translational velocities of control points P1 and P2 are shown in Fig. 5-Fig. 8. Displacements of these points before image stabilization exceeded 0.1 m (Fig.  5, also in Fig. 3 for P1 and P3). The displacement velocities of points P1 and P2 between the original frames reach 0.06 m/s - Fig. 6. The use of image stabilization resulted in about a tenfold reduction in both displacements (Fig. 7) and velocity (Fig. 8).  Fig. 9 and Fig. 10 show the registrations of tests of dropping an object to a slide in a two-stage sorter [12], [13].
The camera above the stage was attached to a frame with insufficient rigidity, which made the camera subject to vibrations taken over and amplified from the floor of the production hall. The amplitude of these vibrations is smaller than in example 1 -comparing the trajectories of points P marked with the yellow curve in Fig. 3 and Fig. 9b or plots in Fig. 5 and Fig. Fig. 11. For this reason, the total raw frame in Fig. 9a shows less blur than the frame in Fig. 2a. Camera vibrations, however, cause very large amplitudes of vibrations of the motion velocity of points marked on the corners of the sorted object (Fig. 10b or points P shown in Fig. 12), determined by differentiation of their motion trajectories ( Fig. 10a and Fig. 11). The use of image stabilization smoothes the motion trajectories of these points ( Fig. 10c and Fig. 13) and significantly reduces the amplitudes of vibrations of the motion velocity Fig. 10d and Fig. 14). The motion trajectories of the object corners were determined using the BLOB technology -also available in the Matlab environment [14], [14].

III. CONCLUSIONS
The paper proposes a procedure for digital stabilization of the image recorded with a digital camera, changing the position relative to the observed scene. There are two stages in this procedure, in which the TemplateMatcher and fitgeotrans functions, available in the Matlab environment, are used respectively. The image stabilization tests show that the proposed procedure allows to significantly compensate the vibration amplitudes of the camera relative to the scene. One of the most important conditions required for effective image stabilization is the selection of control points in the first frame of the image, indicating rectangular patterns located on unmovable areas of the scene. It is important that the defined templates have individual features that allow to stand out them from background of each frames. The proposed procedure makes it possible to conduct credible experimental studies of the motion paths of objects based on the analysis of the image recorded with the camera, moving relative to the observed scene. This means that in cases where the use of a rigid frame to mount the camera above the stage is impossible, image recording can be successfully applied, e.g. a camera from a drone. The consequence of this choice, is the need to perform digital image stabilization (e.g. using functions available in the licensed Matlab environment, or open-source Python), before the proper analysis of the tracked objects' motion trajectories.