Green Screen Remover

Removing a green screen sounds like a search. Find the green, delete it, keep the rest. That is true for most of the picture and false along the outline, where a pixel is some of each.

Those in-between pixels are the whole craft. A key that only sorts pixels into two piles leaves a hard, crawling outline with a green rim. A good key answers a finer question for every pixel: how much of this is subject? The answer is a number between zero and one, and the picture made of those numbers has its own name.

The matte is a second picture

In film, a matte was a separate strip of monochrome stock, clear where the foreground should print and opaque elsewhere, laid against the color film so light reached only the wanted parts. Partly dense areas let some light through. The digital version is the alpha channel, a gray picture the same size as the color one: full value where a pixel is to be seen, zero where it is not, a fraction where it is partly transparent.

Keeping the matte as its own picture is the single most useful habit in this work. You can look at it. A gray haze in an area that should be pure black is leftover screen that will veil the new background. A gray patch inside the subject is a hole that the new background will shine through. Neither is easy to see in a finished composite and both are obvious in the matte.

The matte and the colors meet in one line of arithmetic, which Thomas Porter and Tom Duff set out in 1984 as the “over” operation. For a foreground color F in front of a background B, with coverage a, the recorded color is C = aF + (1 - a)B. Shooting against a screen is that equation with B known. Putting the subject somewhere new is the same equation run forward with a different B.

One color, and why it is green

The method that still underlies most keying is older than digital pictures. Petro Vlahos devised the color-difference technique for film and then built it in electronics; his 1971 patent describes eliminating the blue of the backing by limiting the foreground’s blue signal to a function of its green signal. The observation behind it is modest. Most things people film, skin above all, do not have much more blue than green. A pixel with a large excess of blue over green is therefore probably screen.

Turn that excess into a number and you have a matte. For a green screen the difference is green minus the larger of red and blue. Divide it by the same difference measured on the bare screen, subtract from one, clamp to the range zero to one. A pixel of pure screen gives zero. A gray, white, black, red or skin-toned pixel has no excess and gives one. Everything between slides smoothly, which is why this key handles soft edges better than a threshold does.

Alvy Ray Smith and James Blinn examined these formulas in 1996, when the basic patents had run out; the most recent of them, they wrote, expired in July 1995. Their paper is blunt about the mathematics. Each pixel supplies three measured numbers, red, green and blue. Each pixel hides four: the three foreground channels and the coverage. That is 3 equations, 4 unknowns, so there is no unique answer, and they conclude a person must stay in the loop to recognize a correct matte.

The color-difference key works because it adds an assumption that supplies the missing equation. Smith and Blinn show that the problem can be solved exactly when the foreground is gray, and nearly so for flesh, whose red is roughly twice its green and blue across all skin tones. Where the assumption holds, the key is right. Where it fails, no slider rescues it.

Blue came first because film practice settled on it. Video moved to green for practical reasons that reinforce one another. Blue clothing is common and green clothing is not. A common camera sensor layout, described by Bryce Bayer in a 1976 patent, places green-sensitive elements at every other position, so 1 in 2 sensor sites records green and the green channel is the most finely sampled. And in the standard conversion to luma and two color-difference signals, green carries a weight of 0.587, against 0.299 for red and 0.114 for blue, so it is also the channel that compression protects best.

Soft edges, hair and spill

A single hair is narrower than a pixel. The sensor element it crosses records mostly screen with a little hair mixed in, and the correct coverage might be one tenth. Motion blur and shallow focus do the same thing on a larger scale, smearing an edge across many pixels, each holding a different blend. This is not noise to be cleaned away. It is the true picture, and the matte must carry those fractions or the subject will look cut out with scissors.

Here is one such pixel worked through. Take a mid-gray subject, 128 in each channel, covering half of a pixel in front of a screen measured at red 34, green 184, blue 62. Light mixes in linear terms, so convert first: the screen’s green is 0.4793 and its blue 0.0482, the gray is 0.2159. Half of each gives 0.1170, 0.3467 and 0.1329, which a camera stores as about 96, 159 and 102.

Now key it. The screen’s difference is 0.4793 minus 0.0482, or 0.4311. The pixel’s is 0.3467 minus 0.1329, or 0.2138. One minus their ratio is 0.504, against a true coverage of 0.5. Run the same sum on the stored numbers without converting and the result is 0.533, because stored values are not proportional to light. The error is small here and grows on darker subjects, which is why this tool offers the conversion and uses it by default.

Knowing the coverage is half the job. The stored color of that pixel is still a dull green, and pasted over a sunset it would leave a green fringe. Solve the equation for F: subtract the screen’s share, then divide by the coverage. With the numbers above the recovered color is 129, 128, 128, the gray we started with, off by one level from rounding. The division is the risky step. Where coverage is tiny, small errors are multiplied, so the result has to be clamped.

Spill is a different contamination. The screen is a large, bright, colored light source, and it tints whatever stands in front of it, most visibly on white collars, blond hair and shiny surfaces. That light has reflected off the subject, so it is genuinely part of the foreground color and no matte removes it. Vlahos’s remedy was a clamp: do not let the screen channel exceed what the other two channels justify. The tool applies the same idea, limiting green to the average or the larger of red and blue. The strength is adjustable, because a subject that really is green loses color too.

Spill also damages the matte itself. A white collar lit green has an excess of green, and the key reads that excess as transparency. On the built-in test picture the collar comes out about 91 percent opaque.

Lighting is the cheapest fix

Every method above divides by, or measures distance from, one screen color. A real screen is not one color. Smith and Blinn note that even carefully prepared backings have slight brightness variations and some impurity. A shadow in a corner is a darker green, which looks to the key like a thin layer of something in front of the screen.

The classic answer is to photograph the empty screen and use that record pixel by pixel. A later patent from Vlahos and his son describes synthesizing such a clear frame afterward from the shot itself. This tool does a simpler version: click several patches of bare screen and it blends them by distance, so each pixel is compared with a screen color estimated for its own position.

It is still better to fix the light than the arithmetic. Light the screen evenly and separately from the subject, use a cloth that does not shine, and move the subject forward so less green bounces onto it. A wider tolerance will swallow an uneven screen, but it eats into hair and translucent edges on the way.

What no slider fixes

Green clothing vanishes. The key cannot tell a green scarf from the screen, because by its only measure they are the same thing. On the test picture with a scarf in the screen’s color, all three methods in the tool make it about 85 to 92 percent transparent. The remedy is a garbage shape painted over the scarf to force it opaque, or a blue screen and a reshoot. Strongly colored edges fail the other way. A magenta garment has far less green than red, so at its outline that deficit cancels the screen’s excess, and a pixel that is truly 41 percent covered is reported as fully opaque. It is the test picture’s largest error, one pixel wide.

Compressed sources carry a quieter problem. The usual JPEG arrangement stores the two color signals at half resolution in both directions, so color at a sharp edge is blocky. Start from the least compressed file you have.

The remaining controls are repairs, each with a cost. Shrinking the matte by a pixel, a morphological erosion, removes a fringe and also thins every strand of hair. Feathering blurs the matte with a Gaussian kernel and softens a jagged edge along with a sharp one. Despeckle deletes small islands, and a distant bird is a small island.

One last detail belongs to the file. PNG stores color and transparency separately and unmixed, while a browser canvas holds color already multiplied by transparency, a conversion the HTML standard calls lossy for pixels that are not fully opaque. A cut-out written through a canvas can therefore change the very edge colors that were just cleaned. This tool writes the file directly and reads it back to confirm that nothing moved.

Use the status view before trusting any result. The built-in green sample, at its default settings, reports about 63 percent of pixels keyed out, about 18 percent partly clear, and a matte error of 1.23 percent against the known answer. On your own picture there is no known answer, only your eye, which is the conclusion Smith and Blinn reached in 1996.