On a downside, the Maya feature simply zips up files - retaining their file path structure. To extract the project cleanly, often its a case of digging down in your archive tool until you reach the project folder and extracting that section so you don't end up with a messy folder structure.
So - I decided to look at my options, and see if it could be easy enough to create a cleaner archive. My students needed a way to streamline the process of getting their work up onto the render farm and if this mean't that they didn't have to manually dig through the zip and extract the right data, it was going to be a real time-saver (and minimize human-error as well)
 |
| Mine (top) or Maya's (bottom) |
Here is a comparison between the result of the script presented in the blog post here (top), and the same project using File > Archive Scene (bottom). Note the path displayed which shows how far down the folder structure we had to drill to get to the main project folders...
Hey Python, howzabout you just zip it!
Yes! Python has its own zip module that we can use to archive data to disk, called zipfile. Its got all the core features you'd need - archiving, testing archives/CRC checks and extraction. Its also relatively easy to use.
import maya.cmds as cmds
import os.path
import zipfile as zf
# ---------------------------------------------------------------------
# Example : Create a new zip file
# ---------------------------------------------------------------------
myArchive = zf.ZipFile("myArchive.zip","w")
# ---------------------------------------------------------------------
# Add a file to the zip. You can specify the file, and the
# name/path of the file inside the zip. In this example
# I'm zipping a file called 'blah.txt'. In the zip file,
# I've decided to store it under a subfolder 'zipped'
# ---------------------------------------------------------------------
myArchive.write('blah.txt','zipped/blah.txt')
# ---------------------------------------------------------------------
# Close the zip file
# ---------------------------------------------------------------------
myArchive.close()
import os.path
import zipfile as zf
# ---------------------------------------------------------------------
# Example : Create a new zip file
# ---------------------------------------------------------------------
myArchive = zf.ZipFile("myArchive.zip","w")
# ---------------------------------------------------------------------
# Add a file to the zip. You can specify the file, and the
# name/path of the file inside the zip. In this example
# I'm zipping a file called 'blah.txt'. In the zip file,
# I've decided to store it under a subfolder 'zipped'
# ---------------------------------------------------------------------
myArchive.write('blah.txt','zipped/blah.txt')
# ---------------------------------------------------------------------
# Close the zip file
# ---------------------------------------------------------------------
myArchive.close()
Zipping up is one thing. The other is extracting it back out. We can also do this relatively easily too.
# ---------------------------------------------------------------------
# Extracting a zip file
# ---------------------------------------------------------------------
unArchive = zf.ZipFile("myArchive.zip","r")
# We can test the zip first to ensure its valid. A test will return none if
# the file was ok.
if unArchive.testzip():
print "Invalid zip file - there was a problem"
else:
# Unzip the file to the specified path (in this example, the root of C:)
unArchive.extractall("C:\\")
# Extracting a zip file
# ---------------------------------------------------------------------
unArchive = zf.ZipFile("myArchive.zip","r")
# We can test the zip first to ensure its valid. A test will return none if
# the file was ok.
if unArchive.testzip():
print "Invalid zip file - there was a problem"
else:
# Unzip the file to the specified path (in this example, the root of C:)
unArchive.extractall("C:\\")
Great! Now, what should I zip?
In Maya, you can also request a list of all the files loaded using the cmds.file() command. By default, the very first entry in the list that gets returned will the the name of the current scene file. If a scene has not been saved yet, this will usually be 'untitled', making it an easy check to remind the user of this fact.We can also ask for the Maya project path using the cmds.workspace() command. If a file in the list we retrieved is in the project, its path will include this. This also allows us to double-check and alert the users of problematic 'external' files that somehow slipped in while they weren't paying attention to what they were doing.
By grabbing these, we can strip off the paths and then just write them to the zip file with the subfolder structure in tact directly from python... Making this tool was surprisingly easy.
# ---------------------------------------------------------------------
# Get a list of all files in Maya right now...
# ---------------------------------------------------------------------
# get a list of all files loaded into scene (images, etc)
allFiles = cmds.file(list=True, q=True)
# Go through loop, and remove the leading project folder
prjFolder = cmds.workspace(fn=True, q=True)
# First make sure this is a saved project:
if allFiles[0] == 'untitled':
print "Please save the scene first!"
else:
# Lets go through the list and just check to make sure its in
# the project structure
for loadedFile in allFiles:
if prjFolder in loadedFile:
print "File found in project - Nice!"
else:
print "ARGH! This file was NOT in the project:"
print loadedFile
# Get a list of all files in Maya right now...
# ---------------------------------------------------------------------
# get a list of all files loaded into scene (images, etc)
allFiles = cmds.file(list=True, q=True)
# Go through loop, and remove the leading project folder
prjFolder = cmds.workspace(fn=True, q=True)
# First make sure this is a saved project:
if allFiles[0] == 'untitled':
print "Please save the scene first!"
else:
# Lets go through the list and just check to make sure its in
# the project structure
for loadedFile in allFiles:
if prjFolder in loadedFile:
print "File found in project - Nice!"
else:
print "ARGH! This file was NOT in the project:"
print loadedFile
Using the os.path module, we can also test to make sure a file physically exists as well.
if not os.path.exists(loadedFile):
print "Even worse. The file also doesn't exist on disk!"
print "Even worse. The file also doesn't exist on disk!"
Putting it all together...
So, as you can see, zipping and checking files in Maya is pretty straightforward. Putting this together was fairly easy. The snippets of code above were for explanatory purposes. By taking that knowledge and wrapping a handful of tests and string tweak-age around it, here's the code that I ended up with. As always, I've tried to comment it as well as I can to explain what is happening...The zip process is written as a function that can be called from other code. The user can pass a filename if they wish to, otherwise the scene filename is used and the function will return the filename to whatever called it... A blank filename returned means there was a problem...
# ---------------------------------------------------------------------
# A Better scene archiver
# Kevin Phillips, Jun/Sep 2018
# ---------------------------------------------------------------------
import maya.cmds as cmds
import zipfile as zf
import os.path
# ---------------------------------------------------------------------
# Zip up the currently open Maya project and its associated files
# An optional filename can be passed to the function, otherwise the project
# name will be used to name the ZIP file
#
# Note that the filename is returned - if no name was provided at the start,
# then this will return the filename that was created.
# ---------------------------------------------------------------------
def zipScene(*args):
# The zip filename to use. Blank it out first
zipPFN = ""
# Check to see if a filename had been passed to the function.
if len(args):
zipPFN = args[0]
# Now, grab a list of all files loaded into Maya
allFiles = cmds.file(list=True, q=True)
# Get the current Maya project path
prjFolder = cmds.workspace(fn=True, q=True)
# Lets split the workspace name from the path. We will use this later
# to find valid files with odd paths and to test file presence.
wsPath = prjFolder.split("/")
prjWSpace = wsPath[len(wsPath)-1]
wsPath.pop()
prjWSPath = '/'.join(wsPath)
# Ready to go. But make sure that the files been saved...
if 'untitled' in allFiles[0]:
cmds.confirmDialog(m='Please save the scene first and try again')
# If its all good, lets do our archival magic!
else:
# Get the scene name. We'll use this to name the archive (maybe)
fPath = allFiles[0].split("/")
fName = fPath[len(fPath)-1]
# Test scene for external files. Part of an error-test
# phase to alert the user that there are issues.
chkFiles = False
invPaths = 0
extFiles = 0
for testFile in allFiles:
if prjFolder not in testFile:
if prjWSpace in testFile:
print "Invalid path used in Maya scene : ",testFile
invPaths += 1
chkFiles = True
else:
print "External file path used : ",testFile
extFiles += 1
chkFiles = True
if chkFiles:
msg = "{0} abs.paths\n{1} ext.files\n".format(invPaths,extFiles)
msg += "\nMaya scene will need to be tweaked."
cmds.confirmDialog(m=msg)
# Build clean list of file/relative paths for the archive
# Store each file physical location, along with the zip
# filename (ie relative paths, not entire drive path)
zipContents = []
# We'll also keep count on issues for reporting later on.
invalidFiles = 0
externalFiles = 0
# Loop through all the files that were loaded into Maya
for eachFile in allFiles:
# Strip {#} from end of filename. When files are referenced
# multiple times, this can occur.
if "{" in eachFile:
eachFile = eachFile[:eachFile.index("{")]
# If path wrong, but file appears to be coming from the same
# project workspace, likely that it will exist.
# We simply remap it to the current relative path.
if prjWSpace in eachFile:
eachFile = prjWSPath+"/"+eachFile[eachFile.index(prjWSpace):]
# Doesn't exist? Set the filename to blank
if not os.path.exists(eachFile):
print "Uh-oh! Non-existant file : ",eachFile
eachFile = ''
invalidFiles += 1
else:
# Test external file paths exist. These
# are the ones NOT in the project workspace...
if not os.path.exists(eachFile):
print "Uh-oh! Non-existant file : ",eachFile
eachFile = ''
invalidFiles += 1
# Store file location, and zip path in a list (of lists)
# Each file is stored as [physPath,zipPath]
# Only do this if the filename is not blank
if eachFile:
# Start the 'pair' list with the physical file path
eachPair = [eachFile]
# Create the zip filepath that is associated with that file...
# Test to see if this path is in the project folder structure.
# Yes? Just store the zip location as the relative path
if prjFolder in eachFile:
# If yes, we can strip workspace path from the filename
eachPair.append(eachFile[len(prjFolder):])
else:
# External files stored under 'externalFiles' folder
# to prevent messy file paths all over someones drive
eachPair.append('externalFiles/'+os.path.basename(eachFile))
externalFiles += 1
# add the file/filename into our main list to be zipped up.
# Make sure to also skip any duplicates...
if not eachPair in zipContents:
zipContents.append(eachPair)
# Create the zip file. If there was no filename provided, then
# lets use the scene filename...
if zipPFN == "":
zipPFN = prjFolder+"/"+fName+'.zip'
else:
zipPFN = prjFolder+"/"+zipPFN
# Create the zip file to write
arcProj = zf.ZipFile(zipPFN,'w')
# Step through the zip file list we collected, and zip each file
for arcFile in zipContents:
# Zip up the physical file, store the relative path
arcProj.write(arcFile[0],arcFile[1])
# This code collects a list of the zip contents so it can be
# displayed at the end to the user as a summary. Zipfile.namelist()
# is used to return this list...
contentsZip = ''
# Limit the list to 16 entries - just to make it display nicely on
# screen when there are a LOT of files zipped up...
if len(arcProj.namelist()) < 16:
# Print the contents of our zip
contentsZip = '\n'.join(arcProj.namelist())
else:
fC = 0
for cz in arcProj.namelist():
if fC < 16:
contentsZip = contentsZip+cz+'\n'
fC += 1
contentsZip = contentsZip+'...and '+str(fC-16)+' other(s)\n'
# Finally, just close the zip
arcProj.close()
# And we finish by notifying the user to the outcome...
if invalidFiles:
msg="{0} non-exist\nand not Archived".format(invalidFiles)
else:
msg="{2}:\n\nContents:\n{1}\n\n({0} external files)".format(externalFiles,contentsZip,zipPFN)
cmds.confirmDialog(m=msg, t="Zip function says...")
return zipPFN
return ''
# A Better scene archiver
# Kevin Phillips, Jun/Sep 2018
# ---------------------------------------------------------------------
import maya.cmds as cmds
import zipfile as zf
import os.path
# ---------------------------------------------------------------------
# Zip up the currently open Maya project and its associated files
# An optional filename can be passed to the function, otherwise the project
# name will be used to name the ZIP file
#
# Note that the filename is returned - if no name was provided at the start,
# then this will return the filename that was created.
# ---------------------------------------------------------------------
def zipScene(*args):
# The zip filename to use. Blank it out first
zipPFN = ""
# Check to see if a filename had been passed to the function.
if len(args):
zipPFN = args[0]
# Now, grab a list of all files loaded into Maya
allFiles = cmds.file(list=True, q=True)
# Get the current Maya project path
prjFolder = cmds.workspace(fn=True, q=True)
# Lets split the workspace name from the path. We will use this later
# to find valid files with odd paths and to test file presence.
wsPath = prjFolder.split("/")
prjWSpace = wsPath[len(wsPath)-1]
wsPath.pop()
prjWSPath = '/'.join(wsPath)
# Ready to go. But make sure that the files been saved...
if 'untitled' in allFiles[0]:
cmds.confirmDialog(m='Please save the scene first and try again')
# If its all good, lets do our archival magic!
else:
# Get the scene name. We'll use this to name the archive (maybe)
fPath = allFiles[0].split("/")
fName = fPath[len(fPath)-1]
# Test scene for external files. Part of an error-test
# phase to alert the user that there are issues.
chkFiles = False
invPaths = 0
extFiles = 0
for testFile in allFiles:
if prjFolder not in testFile:
if prjWSpace in testFile:
print "Invalid path used in Maya scene : ",testFile
invPaths += 1
chkFiles = True
else:
print "External file path used : ",testFile
extFiles += 1
chkFiles = True
if chkFiles:
msg = "{0} abs.paths\n{1} ext.files\n".format(invPaths,extFiles)
msg += "\nMaya scene will need to be tweaked."
cmds.confirmDialog(m=msg)
# Build clean list of file/relative paths for the archive
# Store each file physical location, along with the zip
# filename (ie relative paths, not entire drive path)
zipContents = []
# We'll also keep count on issues for reporting later on.
invalidFiles = 0
externalFiles = 0
# Loop through all the files that were loaded into Maya
for eachFile in allFiles:
# Strip {#} from end of filename. When files are referenced
# multiple times, this can occur.
if "{" in eachFile:
eachFile = eachFile[:eachFile.index("{")]
# If path wrong, but file appears to be coming from the same
# project workspace, likely that it will exist.
# We simply remap it to the current relative path.
if prjWSpace in eachFile:
eachFile = prjWSPath+"/"+eachFile[eachFile.index(prjWSpace):]
# Doesn't exist? Set the filename to blank
if not os.path.exists(eachFile):
print "Uh-oh! Non-existant file : ",eachFile
eachFile = ''
invalidFiles += 1
else:
# Test external file paths exist. These
# are the ones NOT in the project workspace...
if not os.path.exists(eachFile):
print "Uh-oh! Non-existant file : ",eachFile
eachFile = ''
invalidFiles += 1
# Store file location, and zip path in a list (of lists)
# Each file is stored as [physPath,zipPath]
# Only do this if the filename is not blank
if eachFile:
# Start the 'pair' list with the physical file path
eachPair = [eachFile]
# Create the zip filepath that is associated with that file...
# Test to see if this path is in the project folder structure.
# Yes? Just store the zip location as the relative path
if prjFolder in eachFile:
# If yes, we can strip workspace path from the filename
eachPair.append(eachFile[len(prjFolder):])
else:
# External files stored under 'externalFiles' folder
# to prevent messy file paths all over someones drive
eachPair.append('externalFiles/'+os.path.basename(eachFile))
externalFiles += 1
# add the file/filename into our main list to be zipped up.
# Make sure to also skip any duplicates...
if not eachPair in zipContents:
zipContents.append(eachPair)
# Create the zip file. If there was no filename provided, then
# lets use the scene filename...
if zipPFN == "":
zipPFN = prjFolder+"/"+fName+'.zip'
else:
zipPFN = prjFolder+"/"+zipPFN
# Create the zip file to write
arcProj = zf.ZipFile(zipPFN,'w')
# Step through the zip file list we collected, and zip each file
for arcFile in zipContents:
# Zip up the physical file, store the relative path
arcProj.write(arcFile[0],arcFile[1])
# This code collects a list of the zip contents so it can be
# displayed at the end to the user as a summary. Zipfile.namelist()
# is used to return this list...
contentsZip = ''
# Limit the list to 16 entries - just to make it display nicely on
# screen when there are a LOT of files zipped up...
if len(arcProj.namelist()) < 16:
# Print the contents of our zip
contentsZip = '\n'.join(arcProj.namelist())
else:
fC = 0
for cz in arcProj.namelist():
if fC < 16:
contentsZip = contentsZip+cz+'\n'
fC += 1
contentsZip = contentsZip+'...and '+str(fC-16)+' other(s)\n'
# Finally, just close the zip
arcProj.close()
# And we finish by notifying the user to the outcome...
if invalidFiles:
msg="{0} non-exist\nand not Archived".format(invalidFiles)
else:
msg="{2}:\n\nContents:\n{1}\n\n({0} external files)".format(externalFiles,contentsZip,zipPFN)
cmds.confirmDialog(m=msg, t="Zip function says...")
return zipPFN
return ''
Just a note - I've had a few hand-formatting issues with blogger, so if you copy-paste any code here, you may find the odd indent issue. The code does work... Reformatting it to look 'visual' on Blogger isn't always as reliable. lol!
Yay! Done...
Hopefully that code was fairly self-explanatory. Yes - it can be cleaned up a little (or a lot) as this was an initial 'just make it work' code process. I noted after re-reading my code that there were some 'split()' functions used to separate filenames from paths which really didn't need to be in there. In fact, later on I used os.path.basename() which does that for me (Doh!). You can also use os.path.dirname() to get the complete path before the filename (rather than a .join) but that's something for another time when I feel the need to de-hackify my code. lol! :DAnyways...
Needless to say, if you're going to write a zip function, it wouldn't be surprising to also include an unzip function. This function takes two arguments (the zip file and the path to extract into). The original plan was to set this to archive up a project for submission to the render farm, and then let the zip file be cleanly extracted into the render farm jobs folder...
# ---------------------------------------------------------------------
# Unzip a file into a specified path. Zip file and extraction Path are
# passed as arguments to function.
# ---------------------------------------------------------------------
def unzipScene(*args):
# Test to make sure we have arguments - needs two to tango
if not len(args) == 2:
msg = "error : unzipScene - zipfile, extractpath required"
else:
# Get the filename and extraction path
zipFN = args[0]
zipExtract = args[1]
# Ensure the specified zip file path is valid first.
if not(os.path.exists(zipFN)):
msg = "The zip file did not exist:\n{0}\n".format(zipFN)
else:
unzipper = zf.ZipFile(zipFN,'r')
# Also Test the zip file was OK before we extract it
tested = unzipper.testzip()
# If all good, proceed with unzip.
if not tested:
# Extract the zip into the specified path
unzipper.extractall(zipExtract)
msg = "All good - apparently"
else:
msg = "Bad zip file!\n{0}\n\nError : \n{1}".format(zipFN,tested)
# The extract is complete. Just display the message to the user
cmds.confirmDialog(m=msg, t="Unzip function says...")
# ---------------------------------------------------------------------
# ---------------------------------------------------------------------
# Example of how these functions can be used. Simple, but I like simple...
# 01 : Zip the currently loaded Maya project
archiveName = zipScene()
# 02 : Set up path to render farm or other location
extractPath = "C:\\myProjects"
# 03 : Unzip the project into the server
if archiveName:
whatToDo = cmds.confirmDialog(m='unzip files?',b=['yes','no'])
if whatToDo == 'yes': unzipScene(archiveName,extractPath)
# Unzip a file into a specified path. Zip file and extraction Path are
# passed as arguments to function.
# ---------------------------------------------------------------------
def unzipScene(*args):
# Test to make sure we have arguments - needs two to tango
if not len(args) == 2:
msg = "error : unzipScene - zipfile, extractpath required"
else:
# Get the filename and extraction path
zipFN = args[0]
zipExtract = args[1]
# Ensure the specified zip file path is valid first.
if not(os.path.exists(zipFN)):
msg = "The zip file did not exist:\n{0}\n".format(zipFN)
else:
unzipper = zf.ZipFile(zipFN,'r')
# Also Test the zip file was OK before we extract it
tested = unzipper.testzip()
# If all good, proceed with unzip.
if not tested:
# Extract the zip into the specified path
unzipper.extractall(zipExtract)
msg = "All good - apparently"
else:
msg = "Bad zip file!\n{0}\n\nError : \n{1}".format(zipFN,tested)
# The extract is complete. Just display the message to the user
cmds.confirmDialog(m=msg, t="Unzip function says...")
# ---------------------------------------------------------------------
# ---------------------------------------------------------------------
# Example of how these functions can be used. Simple, but I like simple...
# 01 : Zip the currently loaded Maya project
archiveName = zipScene()
# 02 : Set up path to render farm or other location
extractPath = "C:\\myProjects"
# 03 : Unzip the project into the server
if archiveName:
whatToDo = cmds.confirmDialog(m='unzip files?',b=['yes','no'])
if whatToDo == 'yes': unzipScene(archiveName,extractPath)
And there ya have it...
Its pretty easy to archive data from Python. I did this primarily to save myself and my students from messy management of projects and files, but knowing how to zip and archive files in Python open a lot of possibilities for more tools, or including this type of functionality in unrelated projects of your own.There are still some things that could be added or improved (noting that the workspace.mel is not included in my version (though its a relatively simple file to generate/store)), but the main foundation is in place...
Have fun!
0 comments:
Post a Comment