Hello USD - Part 7: Where my error messages at????
Yesterday my script outputted file(s) passed the Pixar Library’s usdchecker
but not the Apple’s USDZ Tools usdARKitChecker
. usdARKitChecker
was very coy about why.
The TL:DR:
usdchecker --arkit $FILENAME
does a lot of whatusdARKitChecker
does incase you don’t have that.usdcat -o $OUT_PUT_NAME.usdc --flatten $ROOT_SCENE_FILE.usda
will smoosh all the usda files that$ROOT_SCENE_FILE.usda
sees into a single.usdc
file for ARKit compliance.
System Info:
- MacOS 13.4 (Intel & Mac)
- VSCode 1.79.2 (Swift plugin from server working group v1.4.0)
- swift-driver version: 1.75.2 Apple Swift version 5.8.1 (swiftlang-5.8.0.124.5 clang-1403.0.22.11.100)
- Pixar USD Library 23.05, (python 3.9.17 PySide 6.4.3)
- Added
**/__pycache__/
to .gitignore
What’s in the Apple script?
First thing I wanted to know was why the verbose mode usdARKitChecker
really wasn’t that chatty. No clarity of why my file failed. Also, since I’m new to so much of this I wanted a lot of feedback as to what was being checked and what could go wrong.
So I went poking through the file.
Code Apple’s (license in linked file later in post). Comments mine.
## should by python3?
#!/usr/bin/python
## Python libraries
import subprocess, sys, os, argparse
## pixar usd library python scripts
from pxr import *
## validateMesh.py is file in same directory
from validateMesh import validateMesh
## validateMaterial.py is file in same directory
from validateMaterial import validateMaterial
## Note - these files are linked to later in post
def validateFile(file, verbose, errorData):
## UsdStage "owns the scenegraph and provides access to a composition."
## https://github.com/PixarAnimationStudios/OpenUSD/blob/b53573ea2a6b29bc4a6b129f604bbb342c35df5c/pxr/usd/usd/stage.cpp#L866
## https://openusd.org/release/api/usd_page_front.html
stage = Usd.Stage.Open(file)
## true until an error fires
success = True
## https://github.com/PixarAnimationStudios/OpenUSD/blob/b53573ea2a6b29bc4a6b129f604bbb342c35df5c/pxr/usd/usd/primFlags.h#L576
## https://github.com/PixarAnimationStudios/OpenUSD/blob/b53573ea2a6b29bc4a6b129f604bbb342c35df5c/pxr/usd/usd/primFlags.h#L135
## Prims appear to have several booleans (flags) relating to their configuration / how deep they are, etc. This "Usd_PrimFlagsPredicate" seems to gather them all together in a set and lets the summoner filter out non-relevant prims. (I suspect `Usd.PrimIsActive` etc are in fact bit masks as this really is a boolean and, not a logical and)
predicate = Usd.TraverseInstanceProxies(Usd.PrimIsActive & Usd.PrimIsDefined & ~Usd.PrimIsAbstract)
for prim in stage.Traverse(predicate):
if prim.GetTypeName() == "Mesh":
success = validateMesh(prim, verbose, errorData) and success
if prim.GetTypeName() == "Material":
success = validateMaterial(prim, verbose, errorData) and success
return success
def runValidators(filename, verboseOutput, errorData):
## https://github.com/PixarAnimationStudios/OpenUSD/blob/b53573ea2a6b29bc4a6b129f604bbb342c35df5c/pxr/usd/usdUtils/complianceChecker.py#L912
## RIGHT HERE THE ABOVE IS THE THING TO READ.
checker = UsdUtils.ComplianceChecker(arkit=True,
skipARKitRootLayerCheck=False, rootPackageOnly=False,
skipVariants=False, verbose=False) ## <-- Here is (a) verbose problem?
# TODO: checker.DumpAllRules()?
checker.CheckCompliance(filename)
errors = checker.GetErrors()
failedChecks = checker.GetFailedChecks()
for rule in checker._rules:
error = rule.__class__.__name__
failures = rule.GetFailedChecks()
if len(failures) > 0:
errorData.append({ "code": "PXR_" + error })
errors.append(error)
usdCheckerResult = len(errors) == 0
## Where are the errors being printed?
## Do the custom checks from the other files
## TODO: understand them
mdlValidation = validateFile(filename, verboseOutput, errorData)
success = usdCheckerResult and mdlValidation
print("usdARKitChecker: " + ("[Pass]" if success else "[Fail]") + " " + filename)
def main(argumentList, outErrorList=None):
## Read in any arguments / options
parser = argparse.ArgumentParser()
parser.add_argument("--verbose", "-v", action='store_true', help="Verbose mode.")
parser.add_argument('files', nargs='*')
args=parser.parse_args(argumentList)
verboseOutput = args.verbose
## Validation succeeds until it fails.
totalSuccess = True
for filename in args.files:
errorData = []
## filename: from the loop
## verboseOutput: From the flag
## errorData: Just defined.
runValidators(filename, verboseOutput, errorData)
## outErrorList: defined in function call as None by default
## When is it ever updated to not None? Is it just an optional out pipe?
if outErrorList is not None:
outErrorList.append({ "file": filename, "errors": errorData })
# We're still good only if no new errors and no old errors
totalSuccess = totalSuccess and len(errorData) == 0
## Program exits with no errors (code 0) if totalSuccess
if totalSuccess:
return 0
else:
return 1
if __name__ == '__main__':
argumentList = sys.argv[1:]
sys.exit(main(argumentList))
What’s in the Pixar script
An invaluable clue in the usdARKitChecker
script - how to find the Pixar script that drives usdchecker
, complianceChecker.py
.
So much information! I do not understand a lot of these terms yet, but I feel like I’m getting closer. Text copied from file linked to above:
- Files within a usdz package must be laid out properly, i.e. they should be aligned to 64 bytes.
- Files within a usdz package should not be compressed or encrypted.
- The composed USD stage should not contain any unresolvable asset dependencies (in every possible variation of the asset), when using the default asset resolver. "
- All stages should declare their ‘upAxis’ and ‘metersPerUnit’.
Stages that can be consumed as referencable assets should furthermore have a valid ‘defaultPrim’ declared, and stages meant for consumer-level packaging should always have upAxis set to ‘Y’ - Texture files should be readable by intended client (only .jpg, .jpeg or .png for consumer-level USDZ).
- Supported Image Formats: (“jpg”, “jpeg”, “png”), special ARKit mode: (".exr")
- Unsupported Image Formats[“bmp”, “tga”, “hdr”, “exr”, “tif”, “zfile”, “tx”]
- Check for basic prim encapsulation rules:
- Boundables may not be nested under Gprims
- Of course we must allow Boundables under other Boundables, so that schemas like UsdGeom.Pointinstancer can nest their prototypes. But we disallow a PointInstancer under a Mesh just as we disallow a Mesh under a Mesh, for the same reason: we cannot then independently adjust visibility for the two objects, nor can we reasonably compute the parent Mesh’s extent.
- Connectable prims (e.g. Shader, Material, etc) can only be nested inside other Container-like Connectable prims. Container-like prims include Material, NodeGraph, Light, LightFilter, and exclude Shader
- UsdUVTexture nodes that feed the inputs:normals of a
UsdPreviewSurface must ensure that the data is encoded and scaled properly.
Specifically:
- Since normals are expected to be in the range [(-1,-1,-1), (1,1,1)], the Texture node must transform 8-bit textures from their [0..1] range by setting its inputs:scale to (2, 2, 2, 1) and inputs:bias to (-1, -1, -1, 0)
- Normal map data is commonly expected to be linearly encoded. However, many image-writing tools automatically set the profile of three-channel, 8-bit images to SRGB. To prevent an unwanted transformation, the UsdUVTexture’s inputs:sourceColorSpace must be set to “raw”.
- A prim providing a material binding, must have MaterialBindingAPI applied on the prim.
- A prim providing skelBinding properties, must have SkelBindingAPI applied on the prim.
- ARKit Checker:
- If the root layer is a package, then the composed stage should not contain references to files outside the package. In other words, the package should be entirely self-contained.
- Allowed Layer Formats: (‘usd’, ‘usda’, ‘usdc’, ‘usdz’)
- _allowedPrimTypeNames = (’’, ‘Scope’, ‘Xform’, ‘Camera’, ‘Shader’, ‘Material’,‘Mesh’, ‘Sphere’, ‘Cube’, ‘Cylinder’, ‘Cone’, ‘Capsule’, ‘GeomSubset’, ‘Points’, ‘SkelRoot’, ‘Skeleton’, ‘SkelAnimation’, ‘BlendShape’, ‘SpatialAudio’)
- UsdGeomPointInstancers and custom schemas not provided by core USD are not allowed.
- Shader nodes must have "id" as the implementationSource, with id values that begin with "Usd*". Also, shader inputs with connections must each have a single, valid connection source.
- All material binding relationships must have valid targets.
- The root layer of the package must be a usdc file and must not include any external dependencies that participate in stage composition.
Getting more error messages out usdARKitChecker
Running
usdchecker --arkit ~/Developer/GitHub/USDHelloWorld/Part_06/multiball_20230709T222354.usda
supplies the following error:
The stage uses 2 layers. It should contain a single usdc layer to be compatible with ARKit's implementation of usdz. (fails 'ARKitRootLayerChecker')
Root layer of the stage '~/Developer/GitHub/USDHelloWorld/Part_06/multiball_20230709T222354.usda' does not have the '.usdc' extension. (fails 'ARKitRootLayerChecker')
Why was I not getting that from usdARKitChecker --verbose
? Well now I am! ALL THE MESSAGES ALL THE TIME!!!
Links to raw:
Also: https://github.com/carlynorama/USDHelloWorld/tree/main/explorations/Part_07
#!/usr/bin/python3
## changed to python3
import subprocess, sys, os, argparse
from pxr import *
from validateMesh import validateMesh
from validateMaterial import validateMaterial
## No changes
def validateFile(file, verbose, errorData):
stage = Usd.Stage.Open(file)
success = True
predicate = Usd.TraverseInstanceProxies(Usd.PrimIsActive & Usd.PrimIsDefined & ~Usd.PrimIsAbstract)
for prim in stage.Traverse(predicate):
if prim.GetTypeName() == "Mesh":
success = validateMesh(prim, verbose, errorData) and success
if prim.GetTypeName() == "Material":
success = validateMaterial(prim, verbose, errorData) and success
return success
## More verbose verbose mode
def runValidators(filename, verboseOutput, errorData):
checker = UsdUtils.ComplianceChecker(arkit=True,
skipARKitRootLayerCheck=False, rootPackageOnly=False,
skipVariants=False, verbose=verboseOutput) #<- changed to pass verboseOutput through
checker.CheckCompliance(filename)
if verboseOutput:
print("Rules Being Checked by usdchecker:")
# checker.DumpAllRules() line failed with error:
# for ruleNum, rule in enumerate(GetBaseRules()):
# NameError: name 'GetBaseRules' is not defined
for rule in checker._rules:
print(rule)
print("-----")
errors = checker.GetErrors()
failedChecks = checker.GetFailedChecks()
warnings = checker.GetWarnings()
if verboseOutput:
for warning in warnings:
print(warning)
for error in errors:
print(error)
for failure in failedChecks:
print(failure)
print("-----")
for rule in checker._rules:
error = rule.__class__.__name__
failures = rule.GetFailedChecks()
if len(failures) > 0:
errorData.append({ "code": "PXR_" + error })
errors.append(error)
if verboseOutput:
print("Will be exported to outErrorList")
print(errorData)
print("----")
usdCheckerResult = len(errors) == 0
mdlValidation = validateFile(filename, verboseOutput, errorData)
success = usdCheckerResult and mdlValidation
print("usdARKitChecker: " + ("[Pass]" if success else "[Fail]") + " " + filename)
## No changes
def main(argumentList, outErrorList=None):
parser = argparse.ArgumentParser()
parser.add_argument("--verbose", "-v", action='store_true', help="Verbose mode.")
parser.add_argument('files', nargs='*')
args=parser.parse_args(argumentList)
verboseOutput = args.verbose
totalSuccess = True
for filename in args.files:
errorData = []
runValidators(filename, verboseOutput, errorData)
if outErrorList is not None:
outErrorList.append({ "file": filename, "errors": errorData })
totalSuccess = totalSuccess and len(errorData) == 0
if totalSuccess:
return 0
else:
return 1
if __name__ == '__main__':
argumentList = sys.argv[1:]
sys.exit(main(argumentList))
Why use usdARKitChecker
at all?
Unclear to me, someone new to the party, what hasn’t been ported to the Pixar library yet. I haven’t tested anything with materials or meshes and I don’t know if the extra files provided in the USDZ Tools still give a needed extra layer. They seem to check models fairly extensively. But cross referencing usdchecker
on some of the terms seems to reflect the scripts are feeling around similar areas in some cases. ALSO haven’t even begun to poke at ARKit
, ModelIO
, the new betas, etc to see what tools are baked in to the frameworks that might be more on the beaten path.
Can I fix my scripts compliance?
My problems with making my files ARKit compliant are both easier and harder to fix than I was hoping.
Easier in that I can still use primitives and don’t need to make a Scene, the problems I thought I was having. Switching over to a meshes and embedding things into scenes or creating assemblies would have been new territory.
Harder because I am not going to be writing a Swift USDA -> USDC compressor algorithm anytime in the near future and if I want to embed a .usda generator in a project that will punt it to ARKit I don’t know how to do that yet and now it seems without usdc, there will be no joy. I’d like to do a demo project of getting a Swift wrapper around C++, but the OpenUSD Pixar tools have a lot going on.
In the mean time I will live with a little post processing.
My first successful attempt at force-fixing my files was:
usdview ../Part_06/multiball_20230709T222354.usda
# File > Save flattened as... (choose USDC option)
usdchecker --arkit multiball_20230709T222354_flattened.usdc
> Success!
python3 usdARKitChecker_revised.py multiball_20230709T222354_flattened.usdc
> usdARKitChecker: [Pass] multiball_20230709T222354_flattened.usdc
This requires actual interaction from a human, so not tenable in the long run.
Second successful attempt:
# note just the ONE input file name, the root scene file
usdcat -o output_test_2.usdc --flatten ../Part_06/multiball_20230709T222354.usda
python3 usdARKitChecker_revised.py output_test_2.usdc
> usdARKitChecker: [Pass] output_test_2.usdc
See.. its a USDC now!
Size savings: (5KB + 415 bytes) for uncompressed 3KB for compressed.
Failed attempts involved:
- using
usdzip
right away (still fails because the zipped up files were still more than 1 file, and not usdc) - trying to use
usdcat
to flat AND convert at the same time while giving it all the files associated with my scene instead of just the root file. - trying
python3 /Applications/usdpython/usdzconvert/usdzconvert ../Part_06/multiball_20230709T222354.usda
is interesting. Right at the top there was a warning about an unflattened file, but then the script just brazened it’s way through with tons of warnings, creating a .usdz file that passed the ARKit validation check but contained nothing to load. The script’s behavior reflects the correct “do your best” failure mode, I think, for a complex file where some assets might be available and some not, but kind of amusing in this simple case. I do not yet understand what “composition arcs” do.Warning: in UsdUtilsCreateNewARKitUsdzPackage at line 1329 of $USDREPO/pxr/usd/usdUtils/dependencies.cpp -- The given asset '/var/folders/qb/g6s3spx57x7f0nbrt3_613nh0000gn/T/tmpri1etvnvusdzconvert/multiball_20230709T222354.usdc' contains one or more composition arcs referencing external USD files. Flattening it to a single .usdc file before packaging. This will result in loss of features such as variantSets and all asset references to be absolutized.
Before I found the right order / combination of usdcat
parameters I started to write a shell script. It works, but I leave it here only for the record since the one liner is better.
#!/bin/sh
# Usage: give the script the name of the root file and
# the root file only, not the full list.
# can be replaced with
# `usdcat -o $OUTPUT_NAME --flatten $INPUT_ROOT_FILE``
#PATH_TO_USDCAT="somepath"
REMOVE_TMP_FLAG=false
TMP_FILE_NAME="smash_tmp.usda"
OUTPUT_NAME="smash_output.usdc"
# using $@ (all args) is iffy here since really
# should be passing _one_ file name after all.
usdcat --flatten $@ > $TMP_FILE_NAME
usdcat -o $OUTPUT_NAME $TMP_FILE_NAME
if $REMOVE_TMP_FLAG; then
rm $TMP_FILE_NAME
fi