Lens Distortion Corrector

Photograph a tiled wall with a wide lens and the wall comes back swollen. Rows of tiles arch near the top of the frame and sag near the bottom. Nothing in front of the camera was curved.

That bend is usually treated as a blemish to be rubbed out with a slider. It is better understood as a measurement. Each bowed edge records how far the lens strayed from an ideal one at that distance from the center, and a handful of them is enough to work out the correction.

Why straight things bend

The ideal camera is a pinhole. In that model every straight line in the world lands on the sensor as a straight line, which is the property that makes a photograph of a building look like the building. Real lenses, wide ones most of all, break the rule: their magnification changes with distance from the center of the picture.

When magnification falls toward the edges, the picture is squeezed inward at the rim and straight lines bow outward. That is barrel distortion. When magnification rises toward the edges the lines bow inward, which is pincushion distortion. A lens can also do both, barrel near the middle turning into pincushion near the rim, so that a horizontal line in the upper half of the frame takes the shape of a handlebar mustache.

The point that matters is that the error is radial. A point is pushed along the line that joins it to the center, by an amount that depends only on how far out it is. That is why a line passing through the center stays straight no matter how bad the lens is, and why the lines nearest the frame edges bend the most.

Two numbers and a center

Because the error depends on radius alone, it can be written as a short series. The form used in camera calibration takes the radius a point should have, r, and gives the radius it actually has in the photo as r times (1 + k1 r squared + k2 r to the fourth). Zhang’s widely used calibration method models radial distortion, and the OpenCV library writes it in this multiplying form with a third term added. The first terms matter most, because radial distortion is dominated by its low-order components.

So k1 does most of the work, and its sign names the fault.

A negative k1 shrinks the outer radii, which is barrel; a positive one stretches them, which is pincushion. The second coefficient, k2, bends the curve again farther out, and a k2 of the opposite sign to k1 is what draws the mustache. Barrel and pincushion grow with the square of the distance from the center, while the mustache needs that fourth-power term.

A warning belongs here. The numbers mean nothing until the unit of radius is stated. One convention calls the corner of the picture radius 1, which is half the diagonal. The panorama tools community uses another, in which radius 1 is half the shorter side. The same lens has different coefficients under each, and a profile copied from one program into another without converting it will be wrong.

There is a second formula worth knowing. Fitzgibbon proposed dividing rather than multiplying: the corrected point is the photographed point divided by (1 + lambda r squared). In his comparison against a carefully measured laboratory lens, the one-parameter division model fitted with an error of 0.65 px where the traditional one-term model managed 0.77 px. It also has a closed-form inverse, which the polynomial does not.

Then there is the center. The formula assumes distortion spreads from one point, and that point is normally close to the middle of the frame but not exactly on it. Brown’s earlier work on what he called decentering distortion dealt with lens elements that are not perfectly aligned, and the practical lesson survives: if a correction straightens the left side of a photo and leaves the right side slightly bowed, the center is probably off.

Reading the lens from its own mistakes

In 1971 Duane Brown described a way to calibrate a camera with almost no equipment. He photographed a field of plumb lines, strings held straight by hanging weights, and used the lack of straightness in their images to work out the distortion analytically. No measured target was needed. The only fact he relied on was that the strings were straight.

The same idea works on an ordinary photo. Mark several points along an edge that should be straight: a roofline, a door frame, a horizon. For any trial pair of coefficients, move the marked points to where the correction would put them, lay the best straight line through each group, and add up how far the points miss. The right coefficients are the ones that make that total smallest.

Finding the minimum is a least squares problem with no direct solution, so it is solved by trial and improvement. The Levenberg-Marquardt method is the standard tool. It blends two strategies, cautious downhill steps when far from the answer and bold steps based on the local curvature when close, and it adjusts the blend after each attempt.

The method has blind spots, and they follow from the geometry. A line through the center tells it nothing. Short lines near the middle bend too little to measure. A set of parallel lines cannot locate the center along their own direction. With only a few points, k1 and k2 can trade against each other and produce a pair that fits the marks and is wrong everywhere else. A careful fit refuses these cases instead of returning a number, and says which of them it ran into, because a wrong correction that looks confident is worse than none.

Here is a case to try. The sample photo in the tool above is 960 by 640 pixels, so its half diagonal is 576.9 pixels. Set k1 to -0.1 and leave k2 at zero. A point at the corner of the photo has radius 1, and the correction moves it to the radius r where r times (1 - 0.1 r squared) equals 1. That is r = 1.1535, because 1.1535 times 0.8669 is 1.0000. The corner travels outward by 0.1535 half diagonals, which is 88.5 px. The radius check in the tool, set to 100 percent, reports the same move.

What the correction costs

That corner now lies outside the original frame. The corrected picture is no longer a rectangle; its corners reach outward and its sides curve in. There are three honest ways to deal with that. Keep the original frame and lose the corners. Enlarge the canvas to keep every pixel and accept empty wedges along the sides. Or crop to the largest clean rectangle, which in the worked example is 1,046 by 662 pixels and discards 3.8 percent of the picture.

The second cost is sharpness. Moving the corner outward means spreading the same recorded pixels over more output pixels, about one and a half times more in the corners of that example. No detail is created. Every output pixel is an average of its neighbors in the source, computed by an interpolation filter, so the stretched corners come out softer than the center, and saving the result as a lossy file compresses it a second time.

A single percentage is often quoted for distortion, and it needs the same caution as the coefficients. One definition, from a mobile imaging standard, compares the height of a box at its sides with its height through the middle, negative for barrel and positive for pincushion. Imatest notes that this SMIA figure is 2 times the older television definition for the same lens. A percentage without its definition is not comparable with another.

Fisheyes, and corrections you never see

A fisheye lens is a different case. It is not a flawed pinhole. It follows another rule on purpose, because a pinhole picture would need infinite width to reach 90 degrees off axis. In an ordinary lens the radius grows with the tangent of the angle from the axis. In the common equidistant fisheye it grows with the angle itself, and in the equisolid type with twice the sine of half the angle.

Converting a fisheye photo to an ordinary perspective is therefore a change of projection, and it needs the field of view to set the scale. It is also where the costs become extreme. The outer part of a wide fisheye frame is stretched enormously in the conversion, and the rays near 90 degrees have no place in the result at all.

Finally, much of this happens without the photographer knowing. Some camera systems store correction parameters in each lens, and the camera and the raw converter apply them automatically; such lenses are often designed with more optical distortion because the software will remove it. Raw developers can read corrections embedded in the file or look the lens up in a profile database. A photo that has already been corrected should not be corrected again.

What remains for a hand tool are the cases those systems miss: an adapted or unlisted lens, a phone add-on, a scan of an old print, a frame grabbed from video. There the bent lines are all the calibration data there is. They are usually enough, provided the result is reported with what was lost at the edges.