Querying Private Shotgun Entities

Deep linking your way into the internal API.

I have been working on a site-local Shotgun cache in order to (1) eliminate the round-trip time to Shotgun's servers, and (2) allow for more expressive queries. This cache primarily operates as an API proxy; it satisfies any requests it can, but if it is missing anything it will forward the request to the real server, and cache the results for next time.

Schema changes are a major concern of mine. Instead of trying to perfectly anticipate how Shotgun will internally mutate data during a schema change, I have opted to simply discard any data affected by such a change.

This still requires me to be able to watch for these changes. My first thought was to watch the event log (by polling for new EventLogEntry entities), and this does show us a schema change, e.g. here is the entry for changing a field's type from "text" to "number":

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
{
    'type': 'EventLogEntry',
    'id': 1996928,
    'event_type': 'Shotgun_DisplayColumn_Change',
    'attribute_name': 'data_type',
    'entity': {'id': 2836, 'name': 'sg_text_field_1', 'type': 'DisplayColumn'},
    'meta': {
        'attribute_name': 'data_type',
        'entity_id': 2836,
        'entity_type': 'DisplayColumn',
        'field_data_type': 'text',
        'new_value': 'number',
        'old_value': 'text',
        'type': 'attribute_change'
    },
    'user': {'id': 108, 'name': 'Mike Boers', 'type': 'HumanUser'}
}

The problem is that this doesn't mention anywhere what entity type this field is on! Where is CustomEntitity02 in all of this?

"It's okay, Mike", you might say. "You can just query the DisplayColumn entity!". Lets try:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
>>> sg.find_one('DisplayColumn', [('id', 'is', 2836)])
Traceback (most recent call last):
...
shotgun_api3.shotgun.Fault: API read() invalid/missing string entity 'type':
Valid entity types:
["ActionMenuItem",
 "ApiUser",
 "AppWelcomeUserConnection",
 "Asset",
...

It turns out that DisplayColumn, along with PageSetting (and potentially more entity types) are considered private, and cannot be queried via the standard API.

So... how can we get that data?


Exploiting Deep Links

It turns out that while these entities are private, you can still retrieve data from them via deep links; you just need to know where to look.

Lets start from that log entry, and hope there is an "entity_type" field on the DisplayColumn:

1
2
>>> sg.find_one('EventLogEntry', [('id', 'is', 1996927)], ['entity.DisplayColumn.entity_type'])
{'entity.DisplayColumn.entity_type': 'CustomEntity02', 'type': 'EventLogEntry', 'id': 1996927}

Success!

Getting Away from the Event Log

Given the way the event log appears to work, there is bound to be at least one log entry for every entity that exists in the database, regardless of its public visibility. Since deep-links have recently become arbitrarily deep, you can now effectively bypass any privacy restrictions by linking through any log entry!

Lets run effectively the same query, but without needing a specific log entry:

1
2
>>> sg.find_one('EventLogEntry', [('entity.DisplayColoumn.id', 'is', 2836)], ['entity.DisplayColumn.entity_type'])
{'entity.DisplayColumn.entity_type': 'CustomEntity02', 'type': 'EventLogEntry', 'id': 1996918}

You do still need to know what fields are available. Since the Javascript interface also needs to know this, if you take a look through the HTML source of any Shotgun page you will find something like:

1
<script src="/page/schema?page_id=1234"></script>

If you load that script and un-minify it, you'll have a complete dump of the site's schema, including the private entities! Now, for example, we can see there is a "settings_json" field of PageSettings, so we can query how Shotgun represents everything about any given page:

1
2
>>> sg.find_one('EventLogEntry', [('entity.PageSetting.page.Page.id', 'is', 1234)], ['entity.PageSetting.settings_json'])
# insert huge dump of JSON here

But is it safe?

After asking Shotgun support this question, they replied:

Hmm, no that is not safe to count on (linking from events to other entities in general, and in particularly these internal ones). Expect that to go away at some point, soon. =) Aside from not wanting to leak internal API stuff out (we can disable that any time), allowing linking from the events table could lead to some incredibly bad performance problems, so we shouldn't have allowed that originally, and will need to figure out a nice way to deprecate it the future.

They may add the entity_type to the event log in the future, but in the mean time this is all we seem to have.

Posted . Categories: .