Automating Maya render layers with Python

I had hoped to take a break from writing up Maya python related articles on the blog, but hey - when there's plenty to share, may as well keep popping it up.  This time, its all about building a tool (rather then a UI) for automating the render layer setup process...

Creating render layers with the click of a... eh, script...

If you're working with a small team of generalist artists on a project, and you need them all to set up their own renders, its easy for human error to creep into the process and mess up the production pipeline.

On top of that, should you want to apply custom shader overrides to a layer say, for doing an occlusion pass or perhaps a shader that uses facing ratio for faking rim lighting then the time to do this can be wasteful for a group of artists.

This is a perfect example where scripts can save time and productions from accidental screw ups.  I set up this type of tool just for this exact reason, and I'm sharing snippets of the process here for anybody needing to create their own tool...

Note : For those who may or may not be familiar with scripting in Maya, you need to ensure you import the maya.cmds module first.  A common namespace to use is cmds, and you'll see it mentioned all through the code.  Figured its worth just pointing that out for some of you...

Creating a render layer

Creating a render layer is crazy simple... its the cmds.createRenderLayer command.  Yes - that kind of crazy simple...

cmds.createRenderLayer(name="renderLayer")

But you can't render nothing...

Obviously empty render layers are going to generate empty renders.  I kept my tool simple, expecting the artist to take care of selecting those objects needed first.  Scripts should try and at least catch simple things like this to prevent the process from breaking.  Checking that items have been selected is relatively simple with a check using the cmds.ls command, passing the parameter sl=True to test that the selection list is not empty.

if cmds.ls( sl=True ):
  print "Thank you, you selected an item"

Its also not a bad idea to also check to make sure that your render layer doesn't already exist as well... If it does exist and you try to create it, then the new one will have a number appended to the end of it - that may be useful, but in terms of the pipeline you probably want to avoid having this type of naming convention by making render layers unique and not duplicate names.

We can check if a named node exists in Maya easily (a node is anything related to your scene).  This looks in the scene for the name, and its not just for render layers.  We can use this to see if shaders are installed, etc. which we'll do later on.

if cmds.objExists("myRenderLayer"):
  print "myRenderLayer already exists"
else:
  cmds.createRenderLayer(name="myRenderLayer")

Be specific with layer names

In general, you should name anything you create in Maya projects.  Using Maya's default naming and numbering leads to confusion and inability to quickly identify things (ie. "I need to edit the material for the toes, but is it Lambert67 or Lambert199?")

In my team production, all file assets were named with a "prefix_" when saved (I had a custom save script). I used a check of this prefix to determine if the render layer was for a character, or an environment.  By identifying this, I could use it when naming the render layer.

# Check for the prefix "prefix_" in at least one of the selected items
for item in cmds.ls(sl=True):
  if "prefix_" in item:
rlayerName = "prefix_RenderLayer"
  else:
rLayerName = "myRenderLayer"

Its easy to make a render layer, but...

Once its created, how do we use it?  In our team production, render layers were used to render out different shader passes using shader overrides.  Layers would render fake rim lights, occlusion, etc.

However, there is no direct python way (as far as I know) to add a shader override.  It took me a bit of research and scouring the net, but I did discover that there is a MEL command specifically to attach (hook) a shader to a render layer called hookShaderOverride.  We'll look at this one shortly.  By the way, before I get to that step, to access mel from Python you should import maya.mel...

import maya.mel as mel

Adding multiple shader overrides

I had a few shaders I wanted to apply to a selection, each as a separate render layer.  All the shaders I created, then exported from the hypershade as a file everybody could access.  This is relatively simple to do (in Hypershade, select all your shaders and use file > export selected networks) and make sure that they have clear and unique names that match what you are going to use.

When a new scene is loaded, bringing these back in before rendering is as simple as a general file > import (as you would with objects, etc).  But there is one important tweak you need to make to the Namespace Options.  When importing, make sure that you deselect Use namespaces and set the Resolve option to Clashing nodes so that the shaders come in with their names only (and not the scene name as its name space)

Once these are loaded, I could then run a script and create the render layers with their own shader overrides as needed.  While my final script is a lot more complex, the steps I went through to do what I needed are covered in the following code snippets...

I kept the shader names in a list in the script that I could refer to and apply as I needed.  I did of course make sure that I tested the artist had loaded them first by using cmds.objExists

# In this example, I'm using two shaders called rl_Rim and rl_Occ
# These shaders should have been imported by the artist before this
# script is run...
shaderList = ['rl_Rim','rl_Occ']

# First I test to make sure that they exist.  If even one is missing
# I set a flag to indicate this and abort the layer creation...
flgShaderLoaded = True
for shdr in shaderList:
  if not cmds.objExists(shdr):
  flgShaderLoaded = False

Once tested, I created the render layers and applied the shader overrides using our mel command hookShaderOverride.  Note that import maya.mel as mel also needed to be added at the top of the script or this step won't work.

# Do this next step as long as all of the shaders were imported
if flgShaderLoaded:

  # Test that something is selected BEFORE doing this
  if cmds.ls( sl=True ):
 
  # Iterate through each shader
    for shdr in shaderList:

  # Create our render Layer.  I'm using the shader
      # name as part of the render layer name
lyrName = shdr + "_renderLayer"
cmds.createRenderLayer(n=lyrName)

# Now apply our shader override.  For this we use
# the mel command hookShaderOverride.  We pass the
# following parameters:
# ("name of layer", "", "shaderName")
#
# Pay attention to all those " characters!
mel.eval('hookShaderOverride(\"'+lyrName+'\",\"\",\"'+shdr+'\")')
    print "Done!"
  else:
    print "Nothing selected - nothing created"
else:
  print "Shaders need to be imported first"

And that's it...

Its fairly straightforward to create renderlayers, and now that you know about the mel command, you've seen its pretty simple to also apply shader overrides as well.  Obviously to build a tool like this, you would likely want to implement additional checks, perhaps some automation in selection of items for adding to your render layers - maybe even create the shaders as well while you're at it.

I hope that this post has been again useful for you.  I know having this process automated saved not just the time and human error, but the overall sanity of the team!