Static Libraries in a Dynamic World

There is a common pattern when using bindings for C libraries that work against the dynamic language I'm working in: the conceptual packages in the library (which are exposed as header includes, and so provide no extra overhead to use in your code) are actually packages in the binding.

Unless they are flying in the face of good practices, the dynamic language user actually has to work harder to identify a specific function/variable than the static language user would!

The Problem

Let's look at an example Qt program written in C++:

1
2
3
4
5
6
7
8
9

#include <QApplication>
#include <QPushButton>

int main(int argc, char *argv[]) {
    QApplication app(argc, argv);
    QPushButton hello("Hello world!");
    hello.show();
    return app.exec();
}

If we translate directly to Python, we are left with code that feels not only too specific, but puts more of a burden on the programmer than they would have with the static version of the library since they need to know exactly where every object comes from:

1
2
3
4
5
6
7

import sys
from PyQt4 import QtGui

app = QtGui.QApplication(sys.argv)
hello = QtGui.QPushButton("Hello world!")
hello.show()
exit(app.exec_())

This is something I encounter all the time. For example, in PyQt/PySide, the Adobe Photoshop Lightroom SDK, PyOpenGL, Autodesk's Maya "OpenMaya", PyObjC use of Apple's APIs, etc..

Why is "import *" bad?

Most tutorials and examples for PyQt/PySide replicate the feeling of the static code by importing everything into the global namespace:

1
2
3
4
5
6
7

import sys
from PyQt4.QtGui import *

app = QApplication(sys.argv)
hello = QPushButton("Hello world!")
hello.show()
exit(app.exec_())

This is an anti-pattern, as we know that from whatever import * is bad:

Since Python lacks an "include" statement, and the self parameter is explicit, and scoping rules are quite simple, it's usually very easy to point a finger at a variable and tell where that object comes from -- without reading other modules and without any kind of IDE (which are limited in the way of introspection anyway, by the fact the language is very dynamic).

The import * breaks all that.

Answer to "Why is import * bad?" on Stack Overflow

The "Honest" Solution

I tend to use the dynamic nature of the host language to build a more "honest" version of the API I am using. This is what it looks like for our example in PyQT:

1
2
3
4
5
6
7

import sys
from honestpyqt import Q

app = Q.Application(sys.argv)
hello = Q.PushButton("Hello world!")
hello.show()
exit(app.exec_())

... or sending a notification in OS X via PyObjC:

1
2
3
4
5
6

from honestfoundation import NS

notification = NS.UserNotification.alloc().init()
# configure the notification
center = NS.UserNotificationCenter.defaultUserNotificationCenter()
center.scheduleNotification_(notification)

This feels like a totally natural extension of these APIs into Python. Mozilla (and various working groups) apparently agree, as expressed by the WebGL API:

1
2
3
4
5

gl = canvas.getContext("webgl")
gl.clearColor(0.0, 0.0, 0.0, 1.0)
gl.enable(gl.DEPTH_TEST)
gl.depthFunc(gl.LEQUAL)
gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT)

Writing a Dynamic Module Proxy

So, how do I do this? I have a ModuleProxy object in Python which searches a given list of modules with a given list of prefixes when you ask for an attribute it does not know about, and caches the result so this search only takes place the first time:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

class ModuleProxy(object):

    def __init__(self, prefixes, modules):
        self.prefixes = prefixes
        self.modules = modules

    def __getattr__(self, name):
        for prefix in self.prefixes:
            fullname = prefix + name
            for module in self.modules:
                obj = getattr(module, fullname, None)
                if obj is not None:
                    setattr(self, name, obj) # cache it
                    return obj
        raise AttributeError(name)

Using this for PyQt/PySide:

1
2

from PyQt4 import QtCore, QtGui
Q = ModuleProxy(['', 'Q', 'Qt'], [QtCore.Qt, QtCore, QtGui])

Using this for the OS X API's via PyObjC:

1
2
3

import AppKit
import Foundation
NS = ModuleProxy(['NS'], [Foundation, AppKit])

For Lightroom (whose bindings are in Lua) I use:

1
2
3
4
5
6
7
8

local Lr = {}
local meta = {}
meta.__index = function(table, key) 
    key = string.sub(key, 1, 1):upper() .. string.sub(key, 2)
    Lr[key] = import('Lr' .. key)
    return Lr[key]
end
setmetatable(Lr, meta)

Finally, our dynamic code feels as it should.