The 4 Point Algorithm

The Four Point Algorithm allows you to remove projective distortion for planar points. One application is those sideline ads in stadiums: they appear perfectly straight to the tv viewer, but if you were to examine them from up close they'd look fairly distorted.

This algorithm is a special case of epipolar geometry, which aims to find the projective transformation from one camera to another one, with the restriction that the points under consideration are planar. The transformation is called a homography.

Suppose we have Camera 1 and Camera 2, and \textbf{X}_i is a point with coordinates (x,y,z) for camera i: \begin{equation} \textbf{X}_2 = H \textbf{X}_1 \\ \begin{bmatrix} x_2\\ y_2\\ z_2 \end{bmatrix} = \begin{bmatrix} H_{11} & H_{21} & H_{31}\\ H_{21} & H_{22} & H_{32}\\ H_{31} & H_{23} & H_{33} \end{bmatrix} \begin{bmatrix} x_1\\ y_1\\ z_1 \end{bmatrix} \end{equation}
The goal is to determine H. There are 9 elements, but 8 degrees of freedom since it's up to a scale factor. Keep in mind that you can write the equations in inhomogeneous form (dividing x2 and y2 by z2 to produce x'2 and y'2). You end up with 2 equations linear in H_ij, for each point in correspondence. Thus, all the elements of H are determined with 8/2 = 4 points in correspondence between camera 1 and 2.

\begin{equation} (H_{31} x_1 + H_{32}y_1 + H_{33} z_1) x'_2 = H_{11}x_1 + H_{12}y_1 + H_{13} z_1 \\ (H_{31} x_1 + H_{32}y_1 + H_{33} z_1) y'_2 = H_{21}x_1 + H_{22}y_1 + H_{23} z_1 \end{equation}
From here, use SVD to solve for the coefficients of H.
H allows us to "fix" projective distortion for points that are planar. Here's an example, where the selected 4 points were the 4 corners of the Clinton-Washington Av sign (and the 4 points in correspondence are the 4 corners of a rectangle which I ballparked to be the same ratio):

One requirement is that all 4 points are planar. That's the reason my arm is severely distorted. Same goes for the beams supporting the ceiling.