OpenCue integration at Dupp

may 08, 2020
Note: This is a text about software for render management. Enjoy ;)

Well... changing your render management solution isn't easy as pie. And you don't want to make the wrong choice that you regret after tons of hours of integration work. But "luckely" (for us) our previous system developer went out of business and we had to come to a decision on this. Don't want to name our previous system but it filled almost all our needs and we integrated our softwares in to it. But the user interface were lagish and we experienced both server and client side crashes. Also sorting and filtering of jobs weren't very inuitive.

What's our needs?

* a stable system that utilize as much of the cores as possible.
* able to integrate in to our main tools for rendering, transcoding and publishing.
* windows, unix and mac
* sending notifications to artist (e-mail and Slack)
* dependensies to connect series of events.
* search, filtering and manage running jobs
* stability!
* strong community!

Why we choose OpenCue

A year ago my first choice would be Pixar's Tractor. I know it has everything we're looking for and I have years of experience with it. Also Deadline seams to have a nice community and AWS integration. I have only scrapt the surface of Deadline helping students with it and for me it looks the same as our previous solution. And then we have Rush, good old Rush, many peoples favorite, but haven't seen a new release in years and the interface is from the late nineties. We are really excited about Academy Software Foundation and how this will shape the VFX industry the years to come. At the moment OpenCue doesn't have all the needed functionality. The main feature i'm missing at the moment is to control license management more precisely (eg. if you have 10 Nuke render licenses but would like to run 5 instances on each node, that's 50 Nuke instances that should only use 10 licenses). You can't control this at the moment, it's on the road map and there are workarounds. One other thing is that Python 2.7 integration can be problematic. At least from my experience on Windows. That has to do with both gRPC and PySide2 libraries. So if you are using software on Python 2.7 platform you need to build a wrapper for sending jobs to a Python 3.7 environment. Sorry for babbling away from the subject.... why we choose OpenCue?
1. It's free, it's OpenSource.
2. Very scalable
3. Strong contributors: SPI, Google, Microsoft
4. Being an early adopter is a great challenge :)

 
The OpenCue architecture

 

Installation

I will not get in to this in detail. There's a great guide at opencue.io/docs/getting-started. And for windows users the VFX Pipeline channel on Youtube has three videos on how to get started (youtube.com/watch?v=Vk-huejruG0).

Once everything is up and running you can try submitting jobs using CueSubmit. It supports jobs like Shell, Maya, Nuke and Blender by default. Also it's a good idea to get familiar to CueAdmin. With CueAdmin you can create shows and a lot of other stuff. check
cueadmin -server %OPENCUE_SERVER% -h

 
Understanding OpenCue

You can't delete jobs! All jobs will be stored in the database forever and ever. I did get over this quite fast. But quite frustrating when submitting hundreds of jobs when developing. "Eat dead frames" what's this? When a frame fails (the process exits with an error code) it will appear as DEAD in OpenCue. The job will be visible until you either retry, eat or kill it.

  • Retry - stops rendering and retries the frame on another proc.
  • Eat - stops rendering and doesn’t try to continue processing the frame.
  • Kill - stops rendering and books to another proc.

What is booking? You can look at cuebot (the server) as a scheduler. If you turn booking off, the frame/layer/job will not be scheduled. A "proc" is a "portion of cores or memory". Eg if you want a Nuke chunk to book 4 cores or 2gb of ram it will book that space until the chunk is done.

You cant submit a job if you don't have a show registered. So before submitting the first job on a show you need to register the show. You can do this with cueadmin:
cueadmin -server %OPENCUE_SERVER% -force -create-show NAME

What is allocation? It's a way of grouping hosts. A allocation combines a facility with a tag. For example local.general. And what's a subscription? This is like a rule to a show. At what allocations this show can render. Services are a group of default values to a layer (tags, cores and memory). Limits are a way of giving a layer a maximum number of frames to run at the same time. I'm currently missing functionality of licence limitation mentioned above.

 
Jobs - pyOutline

Lets look at some code. I'll share our Nuke integration below. And because we are using Nuke on a Python2.7 platform I send all needed data to a Python 3.7 virtual environment before the job is submitted.

Simplified case submitting one job with one layer:

def buildLayer(layerData, command, lastLayer=None):
    if float(layerData.cores) >= 2:
        threadable = True
    else:
        threadable = False
    layer = Shell(layerData.name, command=command.split(), chunk=layerData.chunk,
                  threads=float(layerData.cores), range=str(layerData.layerRange),
                  threadable=threadable)
    if layerData.services:
        layer.set_service(layerData.services[0])
    if layerData.limits:
        layer.set_limits(layerData.limits)
    if layerData.dependType and lastLayer:
        if layerData.dependType == 'Layer':
            layer.depend_all(lastLayer)
        else:
            layer.depend_on(lastLayer)
    return layer

def buildNukeLayer(layerData, lastLayer):
    nukeCmd = buildNukeCmd(layerData)
    return buildLayer(layerData, nukeCmd, lastLayer)

def buildNukeCmd(layerData):
    writeNodes = layerData.cmd.get('writeNodes')
    nukeFile = layerData.cmd.get('nukeFile')
    cmd = 'Nuke11.2.exe -F #FRAMESPEC# '
    cmd += '-X {} '.format(writeNodes)
    cmd += '-x {}'.format(nukeFile)
    return renderCommand

NUKE_LAYER_DATA = {
    'name': seqName,
    'layerType': JobTypes.JobTypes.NUKE,
    'cmd': {'writeNodes': nodes, 'nukeFile': file},
    'layerRange': fstart+'-'+fend,
    'cores': '4',
    'chunk': '1',
    'limits': ['nuke_lic'],
    'services': ['nuke']
}
jobData = {
    "name": job,
    "shot": shotName,
    "show": projectName,
    "username": usr,
    "layers": [Layer.LayerData.buildFactory(**NUKE_LAYER_DATA)]
}

outline = Outline(jobData['name'], shot=jobData['shot'], show=jobData['show'], user=jobData['username'])

jobs = cuerun.launch(outline, use_pycuerun=False)
    for job in jobs:
        job.setPriority(5)
        job.setMaxCores(1000)
        job.setMaxRetries(3)
This is our set-up:

nukeDispatcher.py

cmd = 'start cmd /K "'
cmd += 'echo Submitting to OpenCue & '
cmd += 'cd /d [PATH TO OPENCUE VIRTUAL ENVIRONMENT]\\Scripts & activate & '
cmd += 'set PYTHONPATH=%PYTHONPATH%;[adding some local libraries]& '
cmd += 'python [PATH TO SCRIPT]\\nukeCueWrapper.py \
    --file '+comp_dirpath+' \ #path to nuke script
    --nodes "'+nodelist+'" \ #write nodes to render
    --fstart '+startFrame+' \
    --fend '+endFrame+' \
    --fps '+framerate+' \
    --ingrid '+folderId+' \ #This is shot id from our asset manager
    --proj "'+projectNameNice+'" \ #name of show
    --shot "'+shotName+'" \ #name of shot (sq010_sh010)
    --job "'+jobName+'" \ #name of task and version (compositing_v001)
    --usr "'+str(userName)+'" \
    --seq "'+input+'" \ #this is the path of a selected read nod on publish
    --mov "'+output+'" \ #where the quicktime should be generated
    --cs "'+colorspace+'" \
    --lut "'+folderLut+'" \ #if shot lut scould be applied to quicktime
    --qc "'+masterReadFile+'" \ #auto generate quicktime for quality check with main plate
    --type "publish"' #what type of job is this, render or publish
pipe = subprocess.call(cmd, stdin=PIPE, stderr=PIPE, stdout=PIPE, shell=True)

nukeCueWrapper.py
This script is mainly based on cuesubmit/tests/Submission_tests.py so that might be a good starting point.

from __future__ import print_function
from __future__ import division
from __future__ import absolute_import

import os
import subprocess
from datetime import date
import argparse

from builtins import str
from outline import Outline, cuerun
from outline.modules.shell import Shell
from cuesubmit import Constants
from cuesubmit import JobTypes
from cuesubmit import Layer

import pyDupp #local stuff, not needed for opencue
import ingrid #local stuff, not needed for opencue

PYTHON_BIN = '' #path to python
os.environ['CUEBOT_HOSTS'] = '192.168.40.101'

def buildLayer(layerData, command, lastLayer=None):
    if float(layerData.cores) >= 2:
        threadable = True
    else:
        threadable = False
    layer = Shell(layerData.name, command=command.split(), chunk=layerData.chunk,
                  threads=float(layerData.cores), range=str(layerData.layerRange),
                  threadable=threadable)
    if layerData.services:
        layer.set_service(layerData.services[0])
    if layerData.limits:
        layer.set_limits(layerData.limits)
    if layerData.dependType and lastLayer:
        if layerData.dependType == 'Layer':
            layer.depend_all(lastLayer)
        else:
            layer.depend_on(lastLayer)
    return layer

def buildNukeLayer(layerData, lastLayer):
    nukeCmd = buildNukeCmd(layerData)
    return buildLayer(layerData, nukeCmd, lastLayer)
    
def buildQtLayer(layerData, lastLayer):
    qtCmd = buildQtCmd(layerData)
    return buildLayer(layerData, qtCmd, lastLayer)

def buildNotifyLayer(layerData, lastLayer):
    notifyCmd = buildNotifyCmd(layerData)
    return buildLayer(layerData, notifyCmd, lastLayer)

def buildFfmpegLayer(layerData, lastLayer):
    ffmpegCmd = buildFfmpegCmd(layerData)
    return buildLayer(layerData, ffmpegCmd, lastLayer)

def buildNukeCmd(layerData):
    """From a layer, build a Nuke Render command."""
    writeNodes = layerData.cmd.get('writeNodes')
    nukeFile = layerData.cmd.get('nukeFile')
    if not nukeFile:
        raise ValueError('No Nuke file provided. Cannot submit job.')
    renderCommand = 'set NUKE_PATH=& ' #your nuke path here
    renderCommand+= 'set foundry_LICENSE=4101@dupp01& '
    renderCommand+= '"C:/Program Files/Nuke11.2v3/Nuke11.2.exe" -F #FRAMESPEC# ' 
    if writeNodes:
        renderCommand += '-X {} '.format(writeNodes)
    renderCommand += '-x {}'.format(nukeFile)
    return renderCommand
    
def buildQtCmd(layerData):
    
    #Using nuke to render quicktimes with burn-ins. This is very facility specific so no big point of sharing this. Please reach out if you want to discuss. 
        
    return renderCommand
    
def buildNotifyCmd(layerData):
    
    #Sending notification to artist, supervisors. We are using slack for all notifications.

    return notifyCommand
    
def buildFfmpegCmd(layerData):
    output = layerData.cmd.get('mov').replace('.mov','.mp4')
    ffmpegCommand = 'l:\\bin\\ffmpeg\\ffmpeg.exe -i '+layerData.cmd.get('mov')+' -y -q:v 0 '+output+''
    return ffmpegCommand

def cuePublish(file, nodes, fstart, fend, fps, ingrid, proj, shot, job, usr, seq, mov, cs, lut, qc, type):
    if type=='render':
        NUKE_LAYER_DATA = {
            'name': seqName,
            'layerType': JobTypes.JobTypes.NUKE,
            'cmd': {'writeNodes': nodes, 'nukeFile': file},
            'layerRange': fstart+'-'+fend,
            'cores': '4',
            'chunk': '1',
            'limits': ['nuke_lic'],
            'services': ['nuke']
        }
    elif type=='rendermov':
        NUKE_LAYER_DATA = {
            'name': seqName,
            'layerType': JobTypes.JobTypes.NUKE,
            'cmd': {'writeNodes': nodes, 'nukeFile': file},
            'layerRange': fstart+'-'+fend,
            'cores': '4',
            'chunk': str(int(fend)-int(fstart)+1), #all frames on one node if .mov
            'limits': ['nuke_lic'],
            'services': ['nuke']
        }
    QT_LAYER_DATA = {
        'name': movName+'_qt',
        'layerType': JobTypes.JobTypes.SHELL,
        'cmd': {'fstart': fstart, 'fend': fend, 'fps': fps, 'ingrid': ingrid, 'usr': usr, 'seq': seq, 'mov': mov, 'cs': cs, 'lut': lut, 'qc': qc, 'type': type},
        'layerRange': fstart,
        'cores': '0',
        'limits': ['nuke_lic'],
        'services': ['nuke'],
        'dependType': Layer.DependType.Layer
    }
    FFMPEG_LAYER_DATA = {
        'name': movName.replace('.mov','.mp4')+'_ffmpeg',
        'layerType': 'ffmpeg',
        'cmd': {'mov': mov},
        'layerRange': fstart,
        'cores': '0',
        'limits': '',
        'services': '',
        'dependType': Layer.DependType.Layer
    }
    NOTIFICATION_LAYER_DATA = {
        'name': 'notify_on_slack',
        'layerType': 'notify',
        'cmd': {'fstart': fstart, 'fend': fend, 'project': projectName, 'shot': shotName, 'usr': usr, 'job': job, 'seq': seq, 'mov': mov, 'cs': cs, 'lut': lut, 'type': type},
        'layerRange': fstart,
        'cores': '0',
        'limits': '',
        'services': '',
        'dependType': Layer.DependType.Layer
    }
    if type=='publish':
        jobData = {
            "name": job,
            "shot": shotName,
            "show": projectName,
            "username": usr,
            "layers": [Layer.LayerData.buildFactory(**QT_LAYER_DATA),
                       Layer.LayerData.buildFactory(**FFMPEG_LAYER_DATA),
                       Layer.LayerData.buildFactory(**NOTIFICATION_LAYER_DATA)]
        }
    elif type=='render' or type=='rendermov': #This is a render
        jobData = {
            "name": job,
            "shot": shotName,
            "show": projectName,
            "username": usr,
            "layers": [Layer.LayerData.buildFactory(**NUKE_LAYER_DATA),
                       Layer.LayerData.buildFactory(**QT_LAYER_DATA),
                       Layer.LayerData.buildFactory(**NOTIFICATION_LAYER_DATA)]
        }
    else:
        print('Unknown job type.')
    print('.')
    outline = Outline(jobData['name'], shot=jobData['shot'], show=jobData['show'], user=jobData['username'])
    print('..')
    lastLayer = None
    for layerData in jobData['layers']:
        if layerData.layerType == JobTypes.JobTypes.MAYA:
            layer = buildMayaLayer(layerData, lastLayer)
        elif layerData.layerType == JobTypes.JobTypes.SHELL:
            layer = buildQtLayer(layerData, lastLayer)
        elif layerData.layerType == JobTypes.JobTypes.NUKE:
            layer = buildNukeLayer(layerData, lastLayer)
        elif layerData.layerType == 'notify':
            layer = buildNotifyLayer(layerData, lastLayer)
        elif layerData.layerType == 'ffmpeg':
            layer = buildFfmpegLayer(layerData, lastLayer)
        else:
            raise ValueError('unrecognized layer type %s' % layerData.layerType)
        outline.add_layer(layer)
        lastLayer = layer
    if 'facility' in jobData:
        outline.set_facility(jobData['facility'])
    print('...')
    jobs = cuerun.launch(outline, use_pycuerun=False)
    for job in jobs:
        job.setPriority(5)
        job.setMaxCores(1000)
        job.setMaxRetries(3)
        #job.addComment('Subject','Comment')
        print('....\nJob \"'+job.name()+'\" submitted!')
        print('(you can close this window)\n')


def parseArgs():
    parser = argparse.ArgumentParser()
    parser.add_argument("--file", help="File path to Nuke scene file.")
    parser.add_argument("--nodes", help="Write nodes sepperated by ,")
    parser.add_argument("--fstart", help="Start frame.")
    parser.add_argument("--fend", help="End frame.")
    parser.add_argument("--fps", help="Framerate.")
    parser.add_argument("--ingrid", help="Ingrid folder ID.")
    parser.add_argument("--proj", help="Project name")
    parser.add_argument("--shot", help="Shot name path")
    parser.add_argument("--job", help="Name of job.")
    parser.add_argument("--usr", help="Name of user.")
    parser.add_argument("--seq", help="File path to output files")
    parser.add_argument("--mov", help="File path to output quicktime")
    parser.add_argument("--cs", help="Colorspace of output sequence.")
    parser.add_argument("--lut", help="Apply LUT to quicktime. Enter LUT name")
    parser.add_argument("--qc", help="Path to source file for QC use")
    parser.add_argument("--type", help="Job type. render or publish")
    return parser.parse_args()

if __name__ == '__main__':
    args = parseArgs()
    cuePublish(args.file, args.nodes, args.fstart, args.fend, args.fps, args.ingrid, args.proj, args.shot, args.job, args.usr, args.seq, args.mov, args.cs, args.lut, args.qc, args.type)
 
Contact

Let me know your thoughts or reach out if you have any questions.

Best,
Anders
This email address is being protected from spambots. You need JavaScript enabled to view it.

© 2020 Dupp Film. Web design It’s just me! Development Optimal media.