Another Application of the Four Point Algorithm

I mentioned the mathematical underlying of the Four Point Algorithm in my previous post, with an example showing tiles in the NY subway under projective distortion and the same tiles without the distortion.

Another application of this algorithm is used in stadiums, on the sidelines. You'll often see ads that appear "straight" to the viewers, although the camera is located high, on one side of the field, and thus should display major distortion.

Using the Four Point Algorithm, one can remove that distortion such that the user sees the ads as undistorted.

Here's an example:
I snapped a picture of a blank piece of paper, at an angle such that there is visible projective distortion.

I calculated the homography by clicking the 4 points of the trapeze above and putting them in correspondence with 4 points forming a rectangle of dimensions equal to the original piece of paper. If you put anything on the piece of paper, it'll undergo that homography transformation as well - hence the alien look.

Printed that picture out, and put it back in the same setting, roughly the same camera pose. The projective distortion is the inverse of the homography which produced the alien look... We end up with no distortion.

The 4 Point Algorithm

The Four Point Algorithm allows you to remove projective distortion for planar points. One application is those sideline ads in stadiums: they appear perfectly straight to the tv viewer, but if you were to examine them from up close they'd look fairly distorted.

This algorithm is a special case of epipolar geometry, which aims to find the projective transformation from one camera to another one, with the restriction that the points under consideration are planar. The transformation is called a homography.

Suppose we have Camera 1 and Camera 2, and \textbf{X}_i is a point with coordinates (x,y,z) for camera i: \begin{equation} \textbf{X}_2 = H \textbf{X}_1 \\ \begin{bmatrix} x_2\\ y_2\\ z_2 \end{bmatrix} = \begin{bmatrix} H_{11} & H_{21} & H_{31}\\ H_{21} & H_{22} & H_{32}\\ H_{31} & H_{23} & H_{33} \end{bmatrix} \begin{bmatrix} x_1\\ y_1\\ z_1 \end{bmatrix} \end{equation}
The goal is to determine H. There are 9 elements, but 8 degrees of freedom since it's up to a scale factor. Keep in mind that you can write the equations in inhomogeneous form (dividing x2 and y2 by z2 to produce x'2 and y'2). You end up with 2 equations linear in H_ij, for each point in correspondence. Thus, all the elements of H are determined with 8/2 = 4 points in correspondence between camera 1 and 2.

\begin{equation} (H_{31} x_1 + H_{32}y_1 + H_{33} z_1) x'_2 = H_{11}x_1 + H_{12}y_1 + H_{13} z_1 \\ (H_{31} x_1 + H_{32}y_1 + H_{33} z_1) y'_2 = H_{21}x_1 + H_{22}y_1 + H_{23} z_1 \end{equation}
From here, use SVD to solve for the coefficients of H.
H allows us to "fix" projective distortion for points that are planar. Here's an example, where the selected 4 points were the 4 corners of the Clinton-Washington Av sign (and the 4 points in correspondence are the 4 corners of a rectangle which I ballparked to be the same ratio):

One requirement is that all 4 points are planar. That's the reason my arm is severely distorted. Same goes for the beams supporting the ceiling.