Image Stitching And Basics Of OpenCV

Written this blog with the collaboration of Yagnik Poshiya 

Blog Outcomes

OpenCV is originally developed by Intel

What is OpenCV?

OpenCV is a huge open-source library. In OpenCV, the CV is an abbreviation form of a Computer Vision, that helps computers to understand the content of digital images such as photographs and videos. OpenCV is mainly focused on computer vision applications such as image processing, artificial intelligence, video capture, and analysis of object detection.

Basic Functions Of OpenCV

1. imread(): used to load an image from a specified file and returns it. if the image cannot be read by the imread() function then it will return an empty matrix. And it supports bmp, dib, jpeg, jpg, jpe, jp2, sr, ras, png, pic, and many other file formats.
2. imshow(): used to display an image in the specified window.
3. waitKey(): used to hold image window for a particular time duration. waitKey(0) will hold the image window until you press any key. waitKey(12) will hold the image window for 12 ms.
4. destroyAllWindows(): used to destroy all opened high GUI windows.
5. cvtColor(): used to convert an image from one color space to another color space. The default color format in OpenCV is often referred to as RGB but it is actually BGR. The first byte in a standard (24-bit) color image will be an 8-bit Blue component, the second 8-bit will be a Green component, and the third 8-bit will be a Red component.
6. split(): used to split multi-channel images into single-channel images. The conversion process of colored images to single-colored images is very time-consuming.
7. merge(): used to merge single-channel images into multi-channel image.
8. resize(): used to scale dimensions of an image in terms of height, width, and both. resize() function preserves the aspect ratio after a process of image resizing.
9. imwrite(): used to save an image to the specified location. The format of the image is chosen based on the given filename extension.
10. shape: used to get dimensions, height, width, and a number of channels of an image based on the indexing method. img.shape is used to get only dimensions of an image, img.shape[0] for height, img.shape[1] for width, and img.shape[2] for number of channels of image.
11. Canny(): used to detect edges of ROI (Region Of Interest) based on minVal (Minimum Intensity Gradient) and maxVal (Maximum Intensity Gradient).
12. GaussianBlur(): used for image smoothing. image smoothing is a process used to remove noise and sharpness from an image.

Applications Of OpenCV

1. acial Recognition System
2. Human-Computer Interaction
3. Motion Understanding
4. Motion Tracking
5. Augmented Reality
6. Mobile Robotics
7. 2D and 3D Feature Toolkits
8. Image Panorama Stitching

Image Panorama Stitching

Image Stitching is a process of combining multiple photographic images with overlapping fields of view to create a segmented panorama or high-resolution image. Image Stitching is also known as Photo Stitching.

Applications Of Image Stitching

1. Video Stitching
2. Document Mosaicing
3. Medical Imaging
4. Panorama Creation

Process Flow

Code Explanation

For the image stitching process common region is required in both images. Given solution for image panorama stitching is robust in case of pictures have different in one or more aspects namely: Scaling, Angle, Special Position, Capturing Devices.

Let’s start with code by importing libraries.

Corners are invariant to the rotation but they are variant to the image scaling process. So, for robustness, we require kinds of features that are invariant to image rotation as well as image scaling, and for this purpose, we are going to use SIFT, SURF, or ORB kinds of methods. We have two ways to match features first one is Brute Force and another is k-Nearest Neighbour. Here we have used the Brute Force method.

After that read images using imageio and apply the grayscale transformation. By default, OpenCV defines the color channel in the order BGR. But it requires transforming images from BGR to RGB to be compatible with matplotlib.

Input Colored Images

Now, we run detectAndDescribe() method on both, queryImg_gray and trainImg_gray to get key points and feature descriptors. Here if we use SIFT method as the feature extractor then it returns a 128-dimensional feature vector for each key point. and SURF will return a 64-dimensional feature vector for each key point.

Now, Display the key points and features detected on both images.

In the image (a) and (b), key points are highlighted in green color

Let’s create matches and cross-check with the help of the createMatcher() function based on the given name of the method. Here match of the first image will cross-check with all other matches of the second image.

To match key points using the Brute Force method matchKeyPointsBF() method is defined which creates matches of a particular image and returns sorted matches. It will sort the feature based on distance means the features of both images with small distance difference (more similarity) are ordered first in the vector.

The below-defined matchKeyPointsKNN() method also did the same work as matchKeyPointsBF() method. In addition, KNN validates pair of features using Ratio Testing. Now let’s see how ratio testing works with an example: for each pair of features (f1,f2), if the distance between f1 and f2 falls within a certain ratio then we kept it otherwise we throw it away.

Based on the feature matching technique it will call the relevant method (Brute Force or KNN). Now, To indicate matches of both images, it uses the drawMatches() inbuilt method.

See carefully, there are colorful lines to show matches

getHomography() method converts key points to NumPy array and based on that it constructs two distinct sets of points. Here getHomography() method uses the RANSAC algorithm to estimate homography between those sets of points and returns matches, transformation matrix (also known as Homography matrix).

Here RANSAC algorithm is used because it is more robust to outliers than the Linear Regression algorithm.

In the below line of code warpPerspective() inbuilt function of OpenCV is used to merge two images based on the Homography matrix (Transformation Matrix).

Generated panorama before applying resizing process

At last we apply some function of OpenCV like cvtColor(), findContours(), threshold(), boudingRect() and see final output contains panorama image of input images. Here we have to use the cvtColor() method to convert an image to grayscale because the threshold() method takes the only grayscale image as input.

hreshold() applies a particular intensity value to each pixel if the value is less than applied the value (known as Threshold Value) then the pixel intensity value is set to zero otherwise it will set pixel intensity value to maximum.

findContours() is used to extract contours from an image.

Contours are defined as the line joining all the points along the boundary of an image that are having the same intensity.

imutils.grab_contours() resize the contours and max() will find contour with maximum value and boudingRect() make rectangle border to contour which has the maximum value.

Final panorama view

References

1. Basic Functions Of OpenCV: opencv.org
2. Applications Of OpenCV: wikipedia.org
3. Image Stitching Definition: wikipedia.org
4. Applications Of Image Stitching: wikipedia.org

Thank You… Keep Learning✌!!!