Using Channels to separate an object from its background

[Midgardsormr]

I started in on using the channels to fix this, but ultimately it's just easier to use the brush tool to clean up that edge. A better Levels adjustment might have made it unnecessary, though.

Now, on to the water. The difference between the water and the rock is not as pronounced in the water as it is in the sky, so we're going to explore a new function: Image > Calculations.

This brings up a new dialog where two channels can be combined arithmetically to produce a third with different properties. As noted before, the rocks have a lot of information in the red channel, and the water has a lot in the blue, so those are the two channels you should experiment with.

Spend some time choosing different blending modes and turning the Invert box on and off in each one. You're looking for a combination that accentuates the contrast between water and rock without destroying the detail of the edge.

In this image, you can see that I've chosen to combine the channels with the Overlay mode, and the blue channel inverted. It looks like I may have done well to try this before working on the sky matte—I might have been able to do the entire matte in one go. You'll notice, though, that using the red channel has added a lot of noise to the sky, so that matte might not have been as clean as I could have gotten had I worked the sky matte a bit more.


Now you can play with Levels and make the water matte the same as you did the sky matte. Here's my finished product:


Now make a Marquee selection (M, then draw a box) around the bottom part of the sea matte channel. Don't let the top of the marquee include any uncertain parts of the rock. Copy (Ctrl-C), switch to the matte channel, and Paste (Ctrl-V). You have finished pulling your matte.

Ctrl-click the little thumbnail in the channels list, switch back to Layers, click on your original layer to get the RGB image back, and Layer > Layer Mask > Reveal Selection. Or just click the "Add Layer Mask" button at the bottom of the Layers palette.

All done! If you now save as a PNG, you'll have an object with transparency. The mask acts as a fourth channel, which indicates which pixels should be transparent, like so:

[Midgardsormr]

This was a fairly simple image to manipulate, but the technique works quite well for images that have very fine detail on the edges, especially hair. It takes some work and experimentation, but it can be done. If you do have very fine detail, it may be better not to clamp the levels quite so hard as I did here, as a bit of translucency is desired in that case.

One other technique you can try is to use the dodge and burn tools on the edges of the matte. Set the Dodge tool's range to "highlights" and the Burn tool's range to "shadows", reduce the Exposure to around 20%, and brush each tool lightly over the edge. Since you've restricted the range, the dodge will act only to lighten the lights, and the burn to darken the darks. You can get a more subtle edge with this method, though if you apply it several times, you'll wind up with the same kind of 1-bit (two values: black or white) matte I just demonstrated.