What is MMUFPHK?

Yes, what is MMUFPHK?  I recently ran into a situation regarding file management and team production which resulted in a task I decided to call Massive Multi User File Path House Keeping.  This is where you (hopefully don't) find multiple Maya project folder structures that have become buried within a singular project on a team server, and worse - production files that are referencing or importing elements from a variety of these individual project folders.

As you can guess, when a large project has to be handled by a team or placed on a Render farm then having a single project made up from a variety of other project sources means a lot of broken file path headaches.  One example I've seen in the past went something like this...

One example of "Scene-ception" (ie. Don't do this)

On a plus - as long as Maya is set to the base project, all scenes are still stored relative to the project root, although it's messy and generally bad practice (ie. project root\Project_Characters\scenes vs project root\scenes).

Naughty!

While the previous example can be used without creating too many headaches, its this structure below where the mess becomes a much more serious issue.  Accessing files from separate projects altogether (outside the main project).  File paths start to be used (rather than relative folders) which breaks things and becomes messy to deal with.  This is the type of problem that this article was written to deal with.

Noooooooo! What ARE you doing?

In a majority of cases, this is a situation nobody would put themselves under.  However to be brutally honest about this, I love it when issues like this occur.  It means I have an excuse to do some coding. 👍

Walking around, doing Housecleaning

Recently we had a team project where this issue occurred.  It wasn't just a collection of project folders within folders, but also a large collection of random file types - files that were generated at render time (but not needed in the project itself) and a variety of original sculpts, test renders, etc.

The best thing for this would be to bring everything together into one singular project folder structure, stripping out any unnecessary files.  Obviously a good handful of scenes will be broken once we do this (they will contain file paths that no longer exist in the new structure, breaking references, multiple texture images, etc) .  That is a task that can be mostly handled using Maya's File path Editor to re-link files...  Though I could see this as being a cool scripting project to handle scenes as a batch process...  Maybe in the next big project.

It was extremely satisfying to do, and as it highlights a variety of techniques and concepts that will definitely be useful for other scripts, I felt it was another blog sharing opportunity.

This article is still written using Python 2.7.  It's worth noting that this version is no longer being updated or supported (you are encouraged to move to Python 3.x) however a lot of applications such as Maya still rely on it for scripting. (I believe Maya 2020 is still running under 2.7.11)

TIP - for those curious about what version of Python is being user by their host application, you can query the python version using sys.version.

import sys
print sys.version

Let's get started - What & where?

Obviously, first thing is to dig about and find all the files and folders on the server that are in need of a little Kondo-ing. For that we need to travel from a start file path and collate together the contents.  This means looking inside folders, sub-folders, sub-folders in sub-folders.  Luckily for me, python has the tools for the job under its os module - os.walk()!

Lets start with importing the module and we'll look at just how cool this is to work with.

import os

We'll be copying and modifying files from one place to another, so we'll also import shutil - shutil contains all of our high level file management operations.

import shutil

We need to start from a root location.  And we want to copy and organise all of this into a nice clean folder.

sourceFolder = "D:/kevsproject/messy"
destFolder   = "D:/kevsproject/clean"

As mentioned, there can be a lot of unused random file types and to help clean up things I tell the script there are certain types I want to ignore completely.  I defined a list that contains the unwanted file extensions. Obviously, when doing this you do need to know what file types are definitely not in use anymore.

# Ignore list allows us to specify what file extensions we DON'T want to bother with.
igList = ['.zpr','.mtl','.obj','.swatches','.swatch','.db',
          '.json','.iff','.py','.mov','.avi','.mp4','.spp','.rib','.xml',
          '.wav','.rlf','.dll']

I only want to process a subset of Maya's standard project folders - those ones that were actually being used (noting that Maya creates a lot of additional folders, as do other's such as Renderman). 

# Subfolder scan list - specify what Maya project subfolders you want to copy.
sfList = ['sourceimages','scenes','images','assets']

I created one more list which lets me specify what sub-folders within any Maya project folder to ignore - the core ones being the tmp folders and .mayaswatches...   We also create an empty list called allFldr. 

Before we do a lot of messing about in the file system, we'll use the lists we've defined to do a little pre-processing and create a list of the files/folders that need to be dealt with.  We'll look at this pre-process step a little later on...

# Ignore subfolder list.  If these exist in the path, we just ignore them
igsfList = ['tmp','.mayaSwatches','edits']

# Collect the details in a list called allFldr
allFldr = []

Walk the walk...

Our lists we just defined will act as "rules".  These rules will now be applied as we start to process the file structure... So how do exactly does one process a file structure?  Easy - os.walk (as its name says) 'walks' through the folder structure, gathering what it finds at each step and returns it as a tuple containing 3 entries.

• > The folder path that os.walk has just read
• > A list of any sub-folders inside this folder path
• > A list of any files inside this folder path

Note that os.walk iterates its way through all the sub-folders so there's no need to manually process the sub-folders list in that first tuple.

# Step through the folders found by os.walk()
for fldr in os.walk(sourceFolder):

As I plan on updating some of the data in each of the tuples returned by os.walk,  being a tuple means it would be immutable (unable to change the contents).  To change that, I simply typecast it as a list.

    updateFldr = list(fldr)

Pre-processing using those lists of rules

I previously mentioned something about keeping a pre-processed list of files to deal with using a series of lists.  I would use these lists and store the details in a list called allFldr.  The reasoning behind this thought was to optimize time - by stripping out unwanted files before doing any physical file management, we then skip dealing with a lot of unnecessary data.  So, lets do that...

Our first pre-process step is to ignore any folders that contain no files (which are stored in the last list of the tuple). Simple enough - we just check if the files list was blank (False) or if it had contents (True)

    # Only care about folders that contain files.
    if updateFldr[2]:

We then filter out the unwanted file types from the list of file extensions to ignore in igList.  The files are filtered using a simple list comprehension into a new list called stripFN, then this updated list replaces the one within the now-typecast-to-a-list tuple

        # Ignored files removed
        stripFN = [fn for fn in updateFldr[2] if os.path.splitext(fn)[1] not in igList]
        updateFldr[2] = stripFN
        # And append to allFldr if the folder still has non-ignored files to copy
        if stripFN:
            allFldr.append(updateFldr)

Basically if this folder was not empty after cleaning out any unwanted files, we then add it to list allFldr.

I'm a file-scanning guru!

We have a nice, lean and clean list to now process.  We can now collate everything back together into one single Maya project folder structure. For example, all scenes folders found would come back into a singular scenes folder in our destination path. i.e. sourceImages to a single sourceImages folder, and so on.

We added the import shutil to the start of the script.  We now loop through our pre-processed list of files and folders stored in allFldr...

for d in allFldr:

We're going to construct a path from the source location's sub-folder structure.  We start by first setting a default (pathSF is our sub-folder, destLoc is the base path to our destination folder we defined at the top of our code)

# Set defaults
pathSF = ''
destLoc = destFolder

Another list comprehension lets us find sub-folders that sit inside the Maya sub-folder list (ie. the ones we set up in the sfList (e.g. 'scenes', 'sourceImages', etc).

# Scan for items in sfList within file path
chkSF = [sf for sf in sfList if sf in d[0]]

If there were folders within the Maya sub-folder(s) then we extract the sub-folder name into pathSF, we set the descLoc to point to the Maya sub-folder.

if chkSF:
pathSF = d[0][d[0].index(chkSF[0])+len(chkSF[0])+1:]
destLoc = destFolder + ('\\%s' % chkSF[0])
# Clean up any whitespace around path name
pathSF = pathSF.strip()

Before we proceed, lets double-check that the sub-folder itself is not one of the Maya sub-sub-folders we don't care about...  If it was, just make the pathSF blank (ie. will be set to the root Maya folder)

# Ignore Maya's own subfolders (ie. we don't need to copy these)
if pathSF in igsfList:
pathSF = ''

Ok.  Those things done, let's finish constructing the full path with the sub-folder added.

# If there was a path, then add it to the end of the destination path
if pathSF:
destLoc += ('\\' + pathSF)

Obviously we can't just copy files to a non-existent file path, so we'll create it if it doesn't already using os.makedirs() - Note that this is pretty cool.  It will create an entire file path, even multiple sub-folders within the path in one go...

# Check to see if path exists, and if not, create it (but only if there are files in it)
if not os.path.exists(destLoc) and d[2]:
try:
os.makedirs(destLoc)
except Exception as e:
print str(e)

Path found, destination path created - its now just a case of copying our files across.  As we're copying from a collection of multiple Maya projects, its likely we'll get duplicate file names at some point.  For now, I'm going to just warn the user that there may be some overwriting going on.

# If there were files inside this folder, then copy them.
if d[2]:

# Loop through the files to copy
for ef in d[2]:
fName = d[0] + '\\' + ef
dName = destLoc + '\\' + ef

# Alert the user that there may have been a duplicate filename overwritten.  At this stage we
# aren't stopping overwrites...  But it would be relatively easy to do so...
try:
if os.exists(dName):
print "WARNING - Multiple files exist with same name. Overwriting may have occurred"
print "      >> %s\n" % dName
except:
pass

Next, the command shutil.copyfile() is used to copy each file.  If the operation failed, we just alert the user.  On an interesting note - a common reason for failed copying is where a file path is excessively long.  We can notify the user of this fact as well - then it will simply be a manual process for them to deal with.

try:
    shutil.copyfile(fName,dName)
except Exception as e:
    print "FAILED : Could not copy %s" % fName)
    if len(fName) > 200:
        print "File path was possibly too long??\n")

And we're done!

When the script finishes, we should have all files from a server that were found under a scenes, sourceimages, images or assets (basically the contents in sfList) folder now brought together under one single Maya project folder (with sub-folders inside)

As I mentioned earlier, there will be some further clean up required, but this can be left to the user (as punishment for making the mess in the first place 😈) to look into duplicate, overwritten or missing files as well as repairing broken paths in scene files...

However - this script really did a great job in cleaning house in one foul swoop...  And it was a lot of fun to write as well.

TIP - It's better than bad - it's good...

With any tools that you write to manage files, analyse scenes, store analytics of processes or just manipulate data, you should be creating log files.  A log that tracks your scripts progress is also a great way to assist in debugging complex programs.

If you're still a little new to Python, start your script using a with statement to open your log like in the example below.  The nice thing about doing it this way is that it closes the file automatically should your script fail or stop because of an error.

# Define the filename for our log...
logFile = "mylogfile.txt"

# Open the logfile for writing
with open(logFile,"w") as lFile:

# This is the main script in here.
print "This example just writes these three strings into the log file"
for errMsg in ["an error", "another error", "last error"]:
lFile.write(errMsg)

# After the script is done, the lFile is closed automatically
print "Script finished - logFile was saved as %s" % logFile

TIP - keeping an eye on progress

One thing to note is that a script processing a lot of data can tie up Maya and at some stage you might wonder if Maya has crashed.  One way to make sure that Maya appears to be doing something is to throw up a progress bar to indicate it's working - and Maya's UI controls just happens to have such a feature.  

Here's how you can produce a handy progress bar that updates as you go.  Note that the progress bar does need a max value to work with (ie. based on how many things you are processing).  In this example I've simply opted to use a for loop to demonstrate how to use this tool.  However in something like our housekeeping tool, use the length of your file list as the max value, and let it update as each entry in the list is processed...

import maya.cmds as cmds

Our progress display will be a simple window with our progress bar.  We don't plan on having anything else in here, but obviously you could make it more interesting with a banner graphic or other visual elements to give it some professional polish.

window = cmds.window(t='Doing something')
cmds.columnLayout()

Its as easy as creating a progressBar() control.  We us maxValue to set the amount of increments that it will accept to reach the 100% mark.  width is how wide the bar is in the window (in pixels).  In here I've set the bar max to 1000.

progressControl = cmds.progressBar(maxValue=1000, width=300)
cmds.showWindow( window )

That's the window done and displayed. Now its just a case of updating your progress bar in your script.  Here's an example using a simple loop...

for prog in range(0,1000):
cmds.progressBar(progressControl, edit=True, step=1)

And then when completed, close the window before you exit the script, and your tool is ready to roll out to your team...  Fun stuff!

And that's it... again...

As always, this article is the way that I have approached things, and while it works for me, there are always going to be better ways and other more advanced python techniques that could be employed here.  My goal however is to share some ideas and I hope this all made at least some sense.

Until next time...  Which may be a while, as I've recently started to obsess over writing retro games on retro computer hardware.