Tutorial: Advanced VRay render elements compositing in After Effects

Compositing VRay render passes is an important subject for the purposes of both quality and flexibility in our compositing workflow. Being able to adjust colours, reflections and even individual lights in the scene long after the render is complete is not only exceptionally powerful, but also sometimes necessary for speedy turnaround on client changes. In this tutorial I’m going to get straight to point with quick yet powerful formulas for advanced render element compositing in After Effects using VRay for 3ds Max. The same principles will apply to other compositing and 3D packages as well. We’ll start off with a basic composite using only 5 passes and further on we’ll tackle an advanced method using 13 render elements.

Linear Workflow

The first step is to set up our linear workflow. This is a necessary evil for all rendering and should be adhered to whether you’re rendering in passes or not. I won’t go into why we have to do gamma correction, just how we do it.

Open up the Gamma and LUT settings from the Rendering menu in 3ds Max and check the following: “Enable Gamma/LUT Correction”, set the gamma to 2.2, check both “Materials and Colors” boxes, and set the bitmap input gamma to 2.2 and it’s output to 1.0.

Next we need to make sure all our textures and images are set to the correct gamma. This can be a pain if you’ve already completed a scene so in future make sure you set up your linear workflow first before doing anything else. When you choose an image as a texture select “Override” in the gamma settings and set it to 2.2. This will be correct for most image file types however HDRs and other linear colour space files should be set to 1.0.

If in doubt, use 2.2.

In the “Color Mapping” section of VRay’s main rendering panel we need to set the gamma to 2.2 and check “Don’t affect colors (adaptation only)”. Do NOT check the linear workflow box despite it’s name.

Lastly we need to use the VRay frame buffer instead of the Max frame buffer. Simply enable it in the VRay frame buffer drop down menu. Once you start rendering be sure to click the sRGB button at the bottom of the VRay frame buffer window.

We are now working in the correct gamma. To make sure we can take advantage of that we need to render to EXR files in at least 16 bit half floating point. When choosing your output file type in your render settings simply choose EXR and the defaults should suffice. In this tutorial we aren’t going to go about combining all our passes into one file, which while neater is a bit more complicated.

Still struggling with the linear workflow process or want to learn more about it? Here’s a great tutorial on linear workflow for 3ds Max. If you’re on Maya try this one, here’s an XSI LWF tutorial and lastly one for Cinema4D users.

Setting up refractive materials

If you’re rendering refrative materials like glass or water against a plain environment background you may have issues with the alpha channel for transparent objects. To fix this we need to change the affect channels setting in our transparent material under the refraction section to “All channels”.

 

Setting up VRay render elements

Now we’re ready to get to down to business. I’ve got a basic scene with some shiny coloured balls, glass, self illumination and some textured balls to cover most situations in one scene. It is lit by two lights, GI and an HDRI reflection. Go over to your render elements tab in the render settings dialogue and let’s add the following render passes for a basic composite. We’ll get to an advanced composite using many more layers later on.

• VRayReflection
• VRayRefraction
• VRaySpecular
• VRayGlobalIllumination
• VRayLighting

These are the render passes required to recreate any given beauty pass. But for this scene we also need to add:

• VRayCaustics
• VRaySelfIllumination

Evaulate your scene to determine which additional passes you may need. In this scene I have glass materials creating caustics and a VRay light material casting self illumination and so I need those elements to recreate the beauty pass. Your scene may not need them or it may need others like VRaySSS2 for sub-surface scattering.

We also want to setup the VRay frame buffer to save a separate colour and alpha channel for us. To do that we simply check “Save separate render channels”, tick all the boxes and choose where to save the files. This also solves the problem of VRay not saving render pass files.

 

Taking the render elements into After Effects

Now we can begin our basic comp in After Effects. To begin we need to set up our project. Create a 16 bit composition and in the project settings window (click on 16 bpc) change the working space to sRGB and check the “Linearize working space” box.

Import all your render layers, go into the Interpret dialogue and premultiply the alpha with black if you’re rendered on a black background. Then in the “Color Management” tab check the “Preserve RGB” box. To speed up the interpretation of multiple layers just interpret one correctly then right click on the layer -> Interpret -> Remember interpretation. Then shift select all your remaining layers and right click -> Interpret -> Apply Interpretation.

Now simply drag all of your layers into your new 16 bit comp and set all blending modes to “Add”. As a general rule of thumb any non-raw, basic render element will use the “Add” blending mode or operation. Order does not matter when all blending modes are set to Add. Order only matters when there are other layers with multiply, divide and subtract modes in play as well.

Your render should now look identical to your beauty pass. The last thing we need to do is pre-comp all the elements into one layer and correct the alpha. You may find you have a bit of a black edge matte on your alpha right now. To fix this drag in the separated alpha channel that VRay should have saved for us and use it as an alpha or luma matte for our pre-comp. It may also help to uncheck “Preserve RGB” in the interpretation of the alpha pass ONLY.

If anything looks dark, bright or incorrect make sure you have linearized your workspace correctly, interpreted your footage correctly and that you have rendered in a linear colour space from your 3D application with the linear workflow tutorials.

 

Alternative gamma correction: If you find that your 8-bit layers are a little bright there is an alternate method to correct the gamma for your render elements only. Simply set your After Effects project settings to the following:

 

Then use an adjustment layer above all your of your render elements and apply an Exposure effect (Effects -> Colour correction -> Exposure).

 

Set the Gamma Correction to 2.2.

This should give you the same result as linearizing the workspace but only affects your render layers. Any 8-bit layers you use elsewhere are unaffected by the Exposure adjustment and will not be altered in any way.

Advanced VRay render elements compositing

If everything went well with your basic composite then you’re ready for more complex render passes. Let’s go back to 3ds Max and use the following render passes:

• VRayCaustics
• VRaySelfIllumination
• VRayShadows
• VRaySpecular
• VRayLightSelect
• VRayExtraTex (Ambient occlusion)
• VRayZDepth
• VRayDiffuseFilter
• VRayRawGlobalIllumination
• VRayRawReflection
• VRayReflectionFilter
• VRayRawRefraction
• VRayRefractionFilter

Some of these require a little more setup:

VRayLightSelect

This incredibly powerful pass let’s us control our lighting in After Effects or any other compositing package.

Start by adding as many LightSelect passes as you have lights or for as many groups of lights as you’d like to control. I have two lights in my scene that I want to control separately so I have two VRayLightSelect passes, you can add as many as you need by simply adding more as you would any other pass. Select the light or group of lights you’d like to control with each render element.

It’s important that you include all your lights in these elements as we’ll be recreating our VRayLighting element with these passes. VRayLightSelect passes are regular VRayLighting elements for selected lights only. Add all your lights to LightSelect passes, add all the passes together and you’ll have the same result as a VRayLighting pass.

 

VRayExtraTex

This will be our ambient occlusion pass. Place a VRayDirt map into the texture slot and play with the radius settings of the VRayDirt to suit your scene.

 

VrayZDepth

Use the tape measure tool in Max to determine the distance of the closest and furthest objects in your scene. Use those values as your minimum and maximum distances in the VRayZDepth pass settings. Many also believe that ZDepth passes should be unfiltered (aliased) but your mileage may vary depending on the scene.

 

Advanced VRay render elements composite

The difference between this advanced composite and the basic formula is that we’ll be using raw render elements to create our base passes. For instance using VRayRawReflection and VRayReflectionFilter to create a VRayReflection pass. We’ll also use additional elements like shadows, ambient occlusion, zdepth and more. Below is a flow chart of how we’ll be compositing these passes:

Note how we’re breaking down some of our previous basic passes. Instead of using a VRayReflection pass we’re creating it by using VRayRawReflection and multiplying it by the VRayReflectionFilter element and precomposing those layers together, then setting the precomp to “Add” like we would a regular VRayReflection pass.

VRayReflection = VRayRawReflection*VRayReflectionFilter (Order does not matter when all blending modes are set to multiply inside a pre-comp)

VRayRefraction = VRayRawRefraction*VRayRefractionFilter (Precompose to create VRayRefraction element and set to add in your main comp)

VRayGlobalIllumination = VRayRawGlobalIllumination*VRayDiffuseFilter (Precompose to create VRayGlobalIllumination element and set to add in your main comp)

VRayLighting = VRayRawLighting*DiffuseFilter (Demonstration only  – We’ll be creating our lighting passes with LightSelect elements instead of RawLighting)

 

 Using VRayLightSelect passes

LightSelect passes are very powerful so I’m going to make sure I cover how to comp them correctly. In my scene I have two lights separated into two VRayLightSelect passes. Each LightSelect element equals a VRayLighting element for that light only. Therefore:

VRayLighting = VRayLightSelect01 + VRayLightSelect02 (Use “Add” blending mode)

You’re welcome to use VRayLightSelect passes as they are to control your lighting but I’m going to delve even deeper to get even more control from our lighting passes. We’re going to create RawLighting passes for each light from their LightSelect pass. VRay doesn’t provide a ‘RawLightSelect’ pass so we have to backwards engineer it ourselves.

To create a RawLighting element from our LightSelect pass we divide it by our DiffuseFilter:

‘VRayRawLightSelect’ = VRayDiffuseFilter/VRayLightSelect (DiffuseFilter MUST be on top and set to “Divide”)

Precompose those elements together. Now we can recreate our Lighting pass from our newly created ‘RawLightSelect’ element.

VrayLighting01 = ‘VRayRawLightSelect’*VRayDiffuseFilter (Both set to “Multiply)

 

Precompose these elements to create a ‘VRayLighting01’ layer that we can use to control that light. We also have access to DiffuseFilters for changing object colours. I did this process a second time for my second VRayLightSelect02 pass and ended up with two VRayLighting elements in my main comp.

Let’s recap that with some formulas and screenshots:

VRayLightSelect01 = ‘VRayRawLightSelect'(VRayDiffuseFilter/VRayLightSelect01)*VRayDiffuseFilter

VRayLightSelect02 = ‘VRayRawLightSelect'(VRayDiffuseFilter/VRayLightSelect02)*VRayDiffuseFilter

VRayLighting = VRayLightSelect01 + VRayLightSelect02

 

Using the VRayShadows and ExtraTex elements

Compositing the VRayShadows and ambient occlusions passes are a little different from our other layers. They will be using different blending modes which means their order is important. VRayShadows can also be used in two different ways.

By default your VRayShadows pass should be composited above your Lighting passes which should be at the very bottom of your comp. We can use VRayShadows in either “Add” or “Subtract” depending on what we want to do and the layer should be turned off if you do not want to modify the shadows in the image.

Turning VRayShadows off will leave shadows in the image unaltered.

Using VRayShadows with “Add” at 100% opacity will remove all shadows from the image.

 

Using VRayShadows with “Subtract” at 100% opacity will double the shadow value in the image.

Compositing VRayShadows above the GI layer will darken the shadow areas more than usual in “Subtract” mode and distort colours.

In order to change the colour of the shadows first add the VRayShadows at 100% and then duplicate the layer on top of the first and use the element in “Subtract” mode at 100%. This will remove the shadows and add them back again allowing us to control them directly. Next, add the curves effect. Remember that everything is inverse with this layer, if you want to add more blue to shadows you need to pull the blue down with curves to reduce the amount of blue that is being subtracted from the image.

Moving onto our ambient occlusion pass I originally placed my ExtraTex layer on top of everything but below my SelfIllumination element. I’ve now revised that to placing the ExtraTex layer on top of everything in the comp and setting it to “Multiply”. This gives my glowing sphere a little bit of shading. These positions aren’t set in stone so if you find something that gives you a look that you’re liking then go with it. This is part of why we’re rendering out all these layers to begin with.

This means that our final formula looks like this:

Beauty Pass = ((VRayDiffuseFilter/VRayLightSelect01)*VrayDiffuseFilter) + ((VRayDiffuseFilter/VRayLightSelect02)*VrayDiffuseFilter) + or – VRayShadows + (VRayRawGlobalIllumination*VRayDiffuseFilter) + (VRayRawRefraction*VRayRefractionFilter) + (VRayRawReflection*VRayReflectionFilter) + VRaySpecular + VRayCaustics * VRayExtraTex + VRaySelfIllumination

*Phew!*

Correcting the alpha channel and using VrayZDepth

Our main comp is now complete. Precomp all of your elements so far into one layer and place your separated alpha channel element above it. Set  your main precomp to use the alpha layer as a luma or alpha matte. I also find it beneficial to uncheck “Preserve RGB” in the interpretation of the alpha channel only but your mileage may vary.

 

 

Precomp both of these layers into one final composite. Place your backplate, ZDepth pass and any further adjustments you’d like to make.

To make use of your ZDepth pass you’ll need to add either After Effects’ Camera Lens Blur effect to the Composite Alpha layer or use 3rd party plugins like the Frischluft Depth of Field effect I’ve used here. You’ll then be able to select a point to focus on and the amount to blur. Be sure to turn the ZDepth layer off as we don’t want to actually see it in our image, just use it’s data for the depth of field effect.

 

Making adjustments to the composite

At this stage everything is primed and ready to be tweaked and edited to our hearts content. There is a limitless amount of possibility so I won’t cover all the things you can do with these layers but here are a few ideas:

• Use opacity values on layers to decrease their effect. Reducing the reflection pass for instance reduces the intensity of the reflections. Reducing opacity on one of our LightSelect composites decreases the intensity of that one light only.
• Use curves to adjust colours of reflections, shadows and GI.
• Duplicate layers to increase their effect or use Levels on VRayReflectionFilter and VRayRefractionFilter elements.
• Use Hue/Saturation effects on VRayDiffuseFilter passes within our GI and Lighting elements to change the colours of objects in our scene. Use the same effect on VRayRawReflections and VRayRawRefractions.
• Add a glow to the SelfIllumination layer.

I’ve done those exact things with some grading to complete this simple image. It’s now up to you to experiment and discover all the incredible things rendering in passes can do for you!

Before

 

After

---