Tutorial: Triple Pass Keying in After Effects

May 28, 2016

Tutorial: Triple Pass Keying in After Effects

I have often noticed that those who are new to compositing and VFX go about their green screen keying the wrong way. There’s a misconception that keying is an almost one click, five minute process and that if the results are unsatisfactory that there is something wrong with the keying software. But that’s very seldom the case, pulling a matte takes WORK. Work and time which you need to put into your shots to pull good quality mattes.

In this tutorial we’re going to explore the triple pass keying method in After Effects. Many times artists will try and do a key in one pass. In the context of After Effects that’s one Keylight effect on one piece of footage. They’ll be forced to push the clip white and black sliders too far and end up with a harsh, unsatisfactory matte. But good keys require multiple passes: garbage mattes, core mattes, edge mattes, spot fixes and yes, possibly rotoscoping. For this tutorial we’ll only cover the essential three passes: the super tight garbage matte, the core matte and the edge matte. This will take our keys to a level above single pass keying and will still be quick enough to do even on the tightest deadlines.

It should be said that the most important aspect of getting a good key is the one you might not be able to control: the footage. The colour resolution, pixel resolution, green screen lighting and recording format play a huge role in the keying software’s ability to do it’s job. You’ll want a camera that can record 444 colour resolution to a native or RAW format. For this tutorial I’m using footage from Hollywood Camera Work which you can download here. This footage is actually shot at 422 at 1280×1080 with a non-square pixel aspect ratio, which is exactly what we’re looking to avoid. However you’ll be faced with poor quality footage in your career and three pass keying methods will help you deal with them, so let’s begin.

Super tight garbage matte

The first of our three passes will be the super tight garbage matte. Great for cutting out tracking markers, uneven screen lighting and folds in cloth screens. To make it we first duplicate our footage layer, apply a Keylight effect and pick our screen colour. It sometimes helps to choose a slightly darker screen colour by picking an area of the screen in shadow. You should experiment on a shot by shot basis.

Green Screen Keying 1

Your goal here is to adjust the clip white and clip black values to eliminate as much of the junk on your green screen as possible while keeping a solid foreground, the edges and any holes in the matte do not matter at all. I’ve used values of 55 on the black and 67 on the white while in screen matte mode to monitor the black and white matte. Finally we use the Screen shrink/grow parameter to grow our matte outwards beyond any fine detail like stray hairs.

Green Screen Keying 2

Lastly for this shot I’m going to use a mask to get rid of all the lighting equipment around the screen. All your additional garbage masking should happen on this top garbage matte layer. I’m also going to rename my layers for neatness, set our Keylight mode back to final result and finally set our bottom layer to use an alpha matte. This should be our result:

Green Screen Keying 3

Core matte

With our garbage matte completed we only have to worry about a thin local area of green around our subject. The second of our three passes is the core matte. For this pass we only care about the main mass of our subject and we are not concerned with fine details like hair or edge quality. We simply want to make sure that the core of our matte is solid.

Apply a Keylight effect to the bottom footage layer and pick the screen colour. Switch into screen matte mode so we monitor our matte.

Green Screen Keying 4

Notice how the inside of the matte has some grey areas which we want to eliminate while paying no attention to the edges or fine detail. Adjust the clip white and clip black values to create a solid inside matte.

Green Screen Keying 5

Lastly we’ll adjust the screen shrink/grow and screen softness parameters to bring the core matte in a bit and away from our edges. We only want this pass to affect the core of our subject and not interact with the edges at all. So shrink and soften the matte slightly.

Green Screen Keying 6

Set the mode to intermediate result and our core matte is complete.

Green Screen Keying 7

Edge matte

The final pass for this technique is the edge matte. For this pass we’re only concerned about the edge quality and fine details of our matte. Apply another Keylight effect onto your bottom layer and pick the screen colour once again. We’re able to do both the core and edge mattes on the same layer. In order to make this work simply drop down the inside mask sub menu and set Source Alpha to “Add to inside mask”.

Green Screen Keying 8

The core matte instance of Keylight must be set to intermediate mode for this to work properly. For this final pass we’re going to adjust the clip white and black values as little as possible so as to maintain a nice edge quality and keep all our fine hair detail. Remember you’ve already taken care of the inside of your matte with your core matte. Even though it will show the grey areas you’ve done already in the screen matte mode, you don’t need to worry about this. For this shot I only needed to adjust the clip black to 6 and left white at 100, keeping them as far away from each other as possible.

Green Screen Keying 9

Lastly I’m going to add an advanced spill supressor to take care of the green spill in the hair.

Green Screen Keying 10

With our three passes now completed we can go back and adjust any values of our previous passes while viewing the final result of all of the passes working together. In this case I adjusted the core matte again to shrink the core matte even further away from the edges, bringing further detail back into the hair. I also placed a solid behind the footage to check the quality against a background. It should be noted that the raspy edges in this particular example are due to this footage being displayed in pixel aspect ratio correction, because the original footage was not shot with square pixels.

Green Screen Keying 11

Cheat sheet

If you need to refer back to this tutorial at any time you use this cheat sheet instead of reading the entire tutorial again.

Garbage matte layer:

1x Keylight with heavily adjusted clip white and black values brought close together.

Screen Shrink/Grow adjusted outwards with a positive value.

Keylight set to Final Result

Core and edge matte layer:

Layer set to use an alpha matte.

2x Keylight effects, one for the core and one for the edge.

Core matte has heavily adjusted clip white and black values brought close together.

Core matte has Screen shrink/grow set to a negative value to bring the matte in away from the edges and softened with Screen softness.

Core matte Keylight effect set to Intermediate Result.

Second Keylight effect has Source Alpha under Inside Mask set to “Add to inside mask”.

Edge matte has lightly adjusted clip white and black values, kept as away from each other as possible.

Edge matte has no screen shrinking or softening at all.

Edge matte Keylight effect set to final result.

Add a spill suppressor if necessary and review all values for all the mattes and make adjustments where necessary.


LEAVE A COMMENT