My site-local Shotgun cache is coming together quite nicely; you can see the progress on GitHub, as well as the ongoing results of my reverse-engineering efforts.

As work progresses, there is a greater and greater need for a through understanding of Shotgun's database schema so that I can replicate its behavior. One example that I'm starting to tackle: when you query an entity or multi_entity field, the entities will return the (required) type and id, but also a name, however, if you query for that name field you will find that it doesn't actually exist:

>>> # Lets grab the pipeline step of any task:
>>> sg.find_one('Task', [], ['step'])
{'step': {'type': 'Step', 'id': 4, 'name': 'Matchmove'}, 'type': 'Task', 'id': 2}
>>> # Notice the name...           ^^^^^^ here.
>>> # Let's grab that name directly:
>>> sg.find_one('Step', [('id', 'is', 4)], ['name', 'code'])
{'code': 'Matchmove', 'type': 'Step', 'id': 4}
>>> # No 'name' to be found. Huh.

Another example is the description of back-references of many multi_entity fields. E.g. if you set Task.entity to a Shot, that task will end up in Shot.tasks as well.

I have reached out to Shotgun support, and they have confirmed to me that the schema returned by the public API's schema_read (and related) methods would need to be expanded to return the information I need.

There must be another way to get this information, because the Shotgun website works, and it must have it. So lets go digging there...

Read more... (3 minutes remaining to read.)

I have been working on a site-local Shotgun cache in order to (1) eliminate the round-trip time to Shotgun's servers, and (2) allow for more expressive queries. This cache primarily operates as an API proxy; it satisfies any requests it can, but if it is missing anything it will forward the request to the real server, and cache the results for next time.

Schema changes are a major concern of mine. Instead of trying to perfectly anticipate how Shotgun will internally mutate data during a schema change, I have opted to simply discard any data affected by such a change.

This still requires me to be able to watch for these changes. My first thought was to watch the event log (by polling for new EventLogEntry entities), and this does show us a schema change, e.g. here is the entry for changing a field's type from "text" to "number":

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

{
    'type': 'EventLogEntry',
    'id': 1996928,
    'event_type': 'Shotgun_DisplayColumn_Change',
    'attribute_name': 'data_type',
    'entity': {'id': 2836, 'name': 'sg_text_field_1', 'type': 'DisplayColumn'},
    'meta': {
        'attribute_name': 'data_type',
        'entity_id': 2836,
        'entity_type': 'DisplayColumn',
        'field_data_type': 'text',
        'new_value': 'number',
        'old_value': 'text',
        'type': 'attribute_change'
    },
    'user': {'id': 108, 'name': 'Mike Boers', 'type': 'HumanUser'}
}

The problem is that this doesn't mention anywhere what entity type this field is on! Where is CustomEntitity02 in all of this?

"It's okay, Mike", you might say. "You can just query the DisplayColumn entity!". Lets try:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

>>> sg.find_one('DisplayColumn', [('id', 'is', 2836)])
Traceback (most recent call last):
...
shotgun_api3.shotgun.Fault: API read() invalid/missing string entity 'type':
Valid entity types:
["ActionMenuItem",
 "ApiUser",
 "AppWelcomeUserConnection",
 "Asset",
...

It turns out that DisplayColumn, along with PageSetting (and potentially more entity types) are considered private, and cannot be queried via the standard API.

So... how can we get that data?

Read more... (2 minutes remaining to read.)

Knowing how and when to deal with colour spaces in your textures files is crucial to a realistic rendering workflow.

In short: since monitors are usually non-linear devices, image files are generally stored with the inverse non-linearity baked in (the general example of which is sRGB). Since rendering calculations work in a linear space, you must convert your images back to the linear space before using them otherwise your renders will not be accurate.

While it is key that you linearize your colour textures (e.g. diffuse, specular colour, etc.), you must be very thoughtful of doing the same with your data textures (e.g. displacement, bump, etc.).

For example, lets look at a displacement:

Read more... (1 minute remaining to read.)

I've been building a VFX pipeline for Shadows in the Grass that allows for artists to work in whatever applications they are the most skilled.

One little thing which has been a bit annoying is that somewhere between Blender and Houdini (of all pairings), geometry transferred via Collada does not have properly qualified normals. This results in some wacky behaviour when you start transforming your models:

Note that when spinning the above sphere, the normals are getting dragged along with the surface, but they remain pointing in the same direction in world space; they are not spinning with the surface. A very obvious effect of this is that the light appears to be spinning around the sphere, even though the light is stationary.

There is a simple enough fix, however.

Read more... (1 minute remaining to read.)

On a limited number of occasions I have had need to reach directly into some of the raw files produced by Autodesk's Maya. There isn't much documentation I could find on the web, so I will try to lay out what I have learned here.

The generic structure is based on the IFF format, but with enough small changes to warrant this exploration (with lots of kudos to cgkit's implementation, which helped with some of the gritty details).

Read more... (6 minutes remaining to read.)