Description
Please follow the homework guidelines in your submission.
will be your main interface for running your code. Parameters for the different programsrunHw3. or unit tests can also be set in that file. Before submission, make sure that you can run all your programs with the command runHw3(“all”) with no errors.
Vectorization
MATLAB is optimized for operations involving matrices and vectors. Avoid using loops (e.g., ) in MATLAB whenever possible – looping can result in long-running code. Instead,
youfr, shouldwile vectorize loops to optimize your code for performance. In many cases, vectorization also results in more compact code (fewer lines to write!). If you are new to MATLAB, refer to th is article for some ideas on how to optimize MATLAB code.
ImageMosaickingApp (13 points)
Your task is to develop an “Image Mosaicking App” that stitches a collection of photos into a mosaic. Creating image mosaics and panoramas has become one of the most popular features on smartphones. Outside the realm of smartphones, similar applications have also been developed to create stunning panoramas seen on GigaPan. With the concepts and algorithms presented in the Image Alignment lecture, you too can create an image mosaic.
Before we create the mosaicking app, we will go over the individual tools required to build it.
After that, you will create these tools as separate programs which you write and submit.
We will take one photo (any image is okay) from the collection as the reference image for the mosaicking process. Given this reference image, the steps to construct the mosaic are:
registration: estimate the geometric transformation relating each image in the collection
1. to the reference image, “ ”
warping : for each image in the collection, we undo the transform estimated from the
2. step above, so that each image is in the same “frame of reference”. Still, the different
photos will cover different regions of the field of view, for which we perform…
3. blending : combine the warped images into a single panoramic view.
The registration method is prone to errors, so we usually apply the RANSAC (RANdom Sampling And Consensus) technique to make it more reliable. In this homework, the basic registration is already implemented for you, but you will need to implement RANSAC.
Each step will be built as its own program, and we will test them individually on some simple inputs so that we are sure that they work correctly. Once we do this, we will put them together to implement the entire stitching pipeline. This homework has five parts: Challenge 1a through 1e.
Registration (already implemented)
NOTE: We strongly recommend checking out the section in the script. It is a test of the registration implementation, and will bedebug1important to understandruHw3when.m you get to
RANSAC.
The registration program calculates the homography between a pair of images. In our case, the homography is estimated as a transformation relating individual keypoints across images. Assume that you have 2 sets of ncorresponding pixel locations – the first set are the n pixel in the reference image, and the second are the corresponding pixels in the other image (the one we want to register to the reference). We use the equation in the Image Alignment lecture to estimate the homography between these sets of points, which you can find in the computeHomography.m file.
incomputeHomographytwoimages: calculates the homography between the two sets of corresponding pixels It uses Hthe= MATLABcomputeHomography(srcfunctionto_pts,computedestthepts)homography matrix.
eig
A second program named applyHomography applies a homography to a set of points:
dest_pts = applyHomography(H, test_pts)
The function helps to view (and debug) the mapping between correspondingshowCorrespointsndenceinthe two images:
Here, result_andimg = showCorrespondence(srcaretwoimagesrelatedimg,bydestahomography,img,srcpts,and dest_pts) is an
src_img dest_img result_img
image showing and side-by-side, with lines connecting and
. The followingsrcimg imagedshowstimg an example output of src_pts.
dest_pts showCorrespondence
Challenge 1a: Warping (4 points)
Here, we will develop a program to warp an image, and use it to warp and paste a portrait of
Vincent ( ) to the empty billboard in the image shown below ( ).
To help youportraitgetstarted,smalla.skeletonpng code has been provided in challenge1a. osaka.png
-
-
-
Before
After
(source)
-
-
First, you need to estimate the homography from the portrait to the target billboard
• frame. Fill in the missing. parts in challenge1a where appropriate using (1 point) computeHomography
Second, we need to replace the “Warp Image Here” part of the target image with the
• portrait image. This will be achieved using backward warping : we will create a new image, such that only the area corresponding to the “Warp Image Here” region in the billboard image, gets exactly filled with the portrait, while the rest is left “blank” (pixel value = zero).
Write a program named backwardWarpImg that warps an image based on a
[mask, dest_img] = backwardWarpImg(src_img, dest_to_src_homography,
The input argument is the sourcedestimagecanvas(the_portraitwidthheiginthist)case). Because you
will implement backwardsrcimgwarping, is the inverse of the
homography that maps source to destinationdestto.src_homography specifies
width and height of the destination image. dest_canvas_width_height
Now the output arguments. is the output image. In this case, is a
warped portrait on a black canvasdest.img is a binary mask indicating thedestarea_ofimgthe
warped image on the canvas. ismaskneeded because the warped image is often not the
final result, but used as an inputmaskto other post-processing tasks (in this case, you will
post the warped image to another image). (3 points)
You are allowed (and expected) to use the MATLAB built-in function interp2.
After generating a warped portrait image and its associated mask, superimpose the image on the given billboard image. The code to perform this task is already included in challenge1a.
Functions not allowed: ALL im*, cp*, tf* functions.
Challenge 1b: RANSAC (3 points)
When stitching together real images, we will not select keypoints manually as you did in Challenge 1a. We will use automatic keypoint extraction methods such as SIFT. Here too, a
function has been provided to compute a set of points ( ) in the source
image andgenSIFTMatchasetofcorresponding points ( ) in the destination imagesrc._Typically,pt these matches will include many outliers, that is,destthe_ptmatches are not correct (they correspond to different scene points). We have discussed the outlier problem in the Image Alignment lecture.
RANSAC (RANdom Sampling And Consensus) is a popular method to robustly estimate
quantities in the presence of outliers. Write a function named to robustly compute homography using RANSAC to address the outlier problem. runRANSAC
[inliers_id, src_to_dest_H] = runRANSAC(src_pt, dest_pt, ransac_n,
specifies the maximum number of RANSACransaciterations,eps) and specifies the acceptableransacn alignment error in pixels. Use Euclidean distance to measureansactheerroreps. Your
program should return a homography that relates the to the .
In addition, return a vector srcthat tolistsdestthe_indicesH of inliers, srci.e.,_thept indicesdestofthept
rows in and inliers.Thefollowingid two images show the matches between two
images, srcbeforept and afterdest runningpt RANSAC. Functions not allowed: ALL im*, cp*, tf* functions
Before RANSAC
After RANSAC
Hint: We recommend you to initially try the values = 100 and = 3,
before experimenting more with them. ransac_n ransac_eps Implementation tip: The RANSAC loop can be implemented in nine lines of MATLAB, including the ‘ ‘ and ‘ ‘ lines. You may find the following snippets helpful, though it is not necessary toforuse themendto receive full credit for this homework.
-
A = H * B; % A and B are 3 x n matrices and H is a 3 x 3 matrix
-
dist = sqrt(sum((A-B).^2)); % A and B are nx3 matrices and dist is a 1xn
-
-
-
matrix
-
-
-
inds = randperm(n, 4); % inds is a vector of 4 random unique integers
-
-
in [1, n]
-
Challenge 1c: Blending (3 points)
Write a program named blendImagePair that blends two images into one:
img1 and img2 are two input images. mask1 and mask2 are the corresponding binary masks |
||||||
result = blendImagePair(img1, mask1, img2, mask2, blendi g mode) |
||||||
indicating the regions of interest. blending_mode, a string, is either “overlay” or “blend”. |
||||||
• |
In the case of “overlay”, copy img2 over img1 wherever the mask2 applies. This case is |
|||||
already implemented for you. |
||||||
• In the case of “blend”, perform weighted blending as discussed in class. You may use |
||||||
the MATLAB function bwdist to compute a new weighted mask for blending – use the |
||||||
‘ |
verlay” |
’ methodblend”ifyou do so. The figure below shows example outputs (left: |
||||
“ |
, right: |
“ |
). Also see the figure on the next page, which gives a hint on |
|||
euclidean |
to compute the weighted mask. After computing the distance |
|||||
how to use |
||||||
transform, you also need to make sure that the mask weights sum to one in the valid |
||||||
bwdist |
region of the output image.
Functions not allowed: ALL im*, cp*, tf* functions
Challenge 1d: Stitching (2 points)
Now you have all the tools to build the mosaicking app! Write a program that
stitches the input images into one mosaic. stitchImg
Your programstitchedshouldimg=acceptstitchImg(img1,anarbitrary numberimg2,…of,imagesimN) (you can use the MATLAB
function to take variable number of inputs). You can assume the order of the input images matchesvararginthe order you wish to stitch the images. Also, in this assignment we will only stitch images of a single row/column. Use “blend” mode to blend the image when you call
.
FunctionsblendImagePairnotallowed: ALL im*, cp*, tf* functions
Challenge 1e: Your own photos (1 point)
Capture images using your own (cell phone) camera and stitch them to create a mosaic. Note that the mosaic need not be a horizontal panorama. Submit both the captured and stitched images.