Compositing VRay render passes is an important subject for the purposes of both quality and flexibility in our compositing workflow. Being able to adjust colours, reflections and even individual lights in the scene long after the render is complete is not only exceptionally powerful, but also sometimes necessary for speedy turnaround on client changes. In this tutorial I’m going to get straight to point with quick yet powerful formulas for advanced render element compositing in After Effects using VRay for 3ds Max. The same principles will apply to other compositing and 3D packages as well. We’ll start off with a basic composite using only 5 passes and further on we’ll tackle an advanced method using 13 render elements.
The first step is to set up our linear workflow. This is a necessary evil for all rendering and should be adhered to whether you’re rendering in passes or not. I won’t go into why we have to do gamma correction, just how we do it.
Open up the Gamma and LUT settings from the Rendering menu in 3ds Max and check the following: “Enable Gamma/LUT Correction”, set the gamma to 2.2, check both “Materials and Colors” boxes, and set the bitmap input gamma to 2.2 and it’s output to 1.0.
If in doubt, use 2.2.
Setting up refractive materials
If you’re rendering refrative materials like glass or water against a plain environment background you may have issues with the alpha channel for transparent objects. To fix this we need to change the affect channels setting in our transparent material under the refraction section to “All channels”.
Setting up VRay render elements
Now we’re ready to get to down to business. I’ve got a basic scene with some shiny coloured balls, glass, self illumination and some textured balls to cover most situations in one scene. It is lit by two lights, GI and an HDRI reflection. Go over to your render elements tab in the render settings dialogue and let’s add the following render passes for a basic composite. We’ll get to an advanced composite using many more layers later on.
These are the render passes required to recreate any given beauty pass. But for this scene we also need to add:
Evaulate your scene to determine which additional passes you may need. In this scene I have glass materials creating caustics and a VRay light material casting self illumination and so I need those elements to recreate the beauty pass. Your scene may not need them or it may need others like VRaySSS2 for sub-surface scattering.
We also want to setup the VRay frame buffer to save a separate colour and alpha channel for us. To do that we simply check “Save separate render channels”, tick all the boxes and choose where to save the files. This also solves the problem of VRay not saving render pass files.
Taking the render elements into After Effects
Now we can begin our basic comp in After Effects. To begin we need to set up our project. Create a 16 bit composition and in the project settings window (click on 16 bpc) change the working space to sRGB and check the “Linearize working space” box.
Alternative gamma correction: If you find that your 8-bit layers are a little bright there is an alternate method to correct the gamma for your render elements only. Simply set your After Effects project settings to the following:
Then use an adjustment layer above all your of your render elements and apply an Exposure effect (Effects -> Colour correction -> Exposure).
Set the Gamma Correction to 2.2.
Advanced VRay render elements compositing
If everything went well with your basic composite then you’re ready for more complex render passes. Let’s go back to 3ds Max and use the following render passes:
- VRayExtraTex (Ambient occlusion)
Some of these require a little more setup:
This incredibly powerful pass let’s us control our lighting in After Effects or any other compositing package.
Start by adding as many LightSelect passes as you have lights or for as many groups of lights as you’d like to control. I have two lights in my scene that I want to control separately so I have two VRayLightSelect passes, you can add as many as you need by simply adding more as you would any other pass. Select the light or group of lights you’d like to control with each render element.
It’s important that you include all your lights in these elements as we’ll be recreating our VRayLighting element with these passes. VRayLightSelect passes are regular VRayLighting elements for selected lights only. Add all your lights to LightSelect passes, add all the passes together and you’ll have the same result as a VRayLighting pass.
This will be our ambient occlusion pass. Place a VRayDirt map into the texture slot and play with the radius settings of the VRayDirt to suit your scene.
Use the tape measure tool in Max to determine the distance of the closest and furthest objects in your scene. Use those values as your minimum and maximum distances in the VRayZDepth pass settings. Many also believe that ZDepth passes should be unfiltered (aliased) but your mileage may vary depending on the scene.
Advanced VRay render elements composite
The difference between this advanced composite and the basic formula is that we’ll be using raw render elements to create our base passes. For instance using VRayRawReflection and VRayReflectionFilter to create a VRayReflection pass. We’ll also use additional elements like shadows, ambient occlusion, zdepth and more. Below is a flow chart of how we’ll be compositing these passes:
VRayReflection = VRayRawReflection*VRayReflectionFilter (Order does not matter when all blending modes are set to multiply inside a pre-comp)
Using VRayLightSelect passes
LightSelect passes are very powerful so I’m going to make sure I cover how to comp them correctly. In my scene I have two lights separated into two VRayLightSelect passes. Each LightSelect element equals a VRayLighting element for that light only. Therefore:
VRayLighting = VRayLightSelect01 + VRayLightSelect02 (Use “Add” blending mode)
To create a RawLighting element from our LightSelect pass we divide it by our DiffuseFilter:
‘VRayRawLightSelect’ = VRayDiffuseFilter/VRayLightSelect (DiffuseFilter MUST be on top and set to “Divide”)
VrayLighting01 = ‘VRayRawLightSelect’*VRayDiffuseFilter (Both set to “Multiply)
VRayLightSelect01 = ‘VRayRawLightSelect'(VRayDiffuseFilter/VRayLightSelect01)*VRayDiffuseFilter
VRayLightSelect02 = ‘VRayRawLightSelect'(VRayDiffuseFilter/VRayLightSelect02)*VRayDiffuseFilter
VRayLighting = VRayLightSelect01 + VRayLightSelect02
Using the VRayShadows and ExtraTex elements
Compositing the VRayShadows and ambient occlusions passes are a little different from our other layers. They will be using different blending modes which means their order is important. VRayShadows can also be used in two different ways.
By default your VRayShadows pass should be composited above your Lighting passes which should be at the very bottom of your comp. We can use VRayShadows in either “Add” or “Subtract” depending on what we want to do and the layer should be turned off if you do not want to modify the shadows in the image.
Turning VRayShadows off will leave shadows in the image unaltered.
Moving onto our ambient occlusion pass I originally placed my ExtraTex layer on top of everything but below my SelfIllumination element. I’ve now revised that to placing the ExtraTex layer on top of everything in the comp and setting it to “Multiply”. This gives my glowing sphere a little bit of shading. These positions aren’t set in stone so if you find something that gives you a look that you’re liking then go with it. This is part of why we’re rendering out all these layers to begin with.
Beauty Pass = ((VRayDiffuseFilter/VRayLightSelect01)*VrayDiffuseFilter) + ((VRayDiffuseFilter/VRayLightSelect02)*VrayDiffuseFilter) + or – VRayShadows + (VRayRawGlobalIllumination*VRayDiffuseFilter) + (VRayRawRefraction*VRayRefractionFilter) + (VRayRawReflection*VRayReflectionFilter) + VRaySpecular + VRayCaustics * VRayExtraTex + VRaySelfIllumination
Correcting the alpha channel and using VrayZDepth
Our main comp is now complete. Precomp all of your elements so far into one layer and place your separated alpha channel element above it. Set your main precomp to use the alpha layer as a luma or alpha matte. I also find it beneficial to uncheck “Preserve RGB” in the interpretation of the alpha channel only but your mileage may vary.
Making adjustments to the composite
At this stage everything is primed and ready to be tweaked and edited to our hearts content. There is a limitless amount of possibility so I won’t cover all the things you can do with these layers but here are a few ideas:
- Use opacity values on layers to decrease their effect. Reducing the reflection pass for instance reduces the intensity of the reflections. Reducing opacity on one of our LightSelect composites decreases the intensity of that one light only.
- Use curves to adjust colours of reflections, shadows and GI.
- Duplicate layers to increase their effect or use Levels on VRayReflectionFilter and VRayRefractionFilter elements.
- Use Hue/Saturation effects on VRayDiffuseFilter passes within our GI and Lighting elements to change the colours of objects in our scene. Use the same effect on VRayRawReflections and VRayRawRefractions.
- Add a glow to the SelfIllumination layer.
I’ve done those exact things with some grading to complete this simple image. It’s now up to you to experiment and discover all the incredible things rendering in passes can do for you!