WARNING (2017-04-06): I've recently found that something in this setup is corrupting NFS traffic between my machines. I'm going to try to isolate it at some point, but be wary.

I've recently built a personal file server (running Ubuntu 16.04), and elected to try connecting it to my main workstation (a macpro6,1 running macOS Sierra) with a pair of bonded 1Gb/s Ethernet links.

Using the default settings does work, however I don't seem to have any control over how the link is negotiated with LACP. It results in using one of the packet hashing modes that puts entire TCP connections on one link or the other. This seems great for stability, and aggregate throughput, however my use case has be mounting the server via NFS or SMB (which runs via a single connection).

In that case, I'm not getting much benefit out of the bond for my primary use case of setting up the bond at all!

Early on, I discovered that if I set both sides of the bond to be "static", then they would both default into a round-robin style packet distribution. In this mode, I can hit 220-230MB/s while doing file operations!

The problem: macOS spontaneously reverts the bond mode back to LACP.

Read more... (1 minute remaining to read.)

A few years ago I started writing PyHAML, a Pythonic version of HAML for Ruby.

Since most of the HAML syntax is pretty straight forward, PyHAML's parser uses a series of regular expressions to get the job done. This proved generally inadequate anytime that there was Python source to be isolated, since Python isn't quite so straight forward to parse.

The earliest thing to bite me was nested parenthesis in tag definitions. In PyHAML you can specify a link via %a(href="http://example.com"), essentially treating the %a tag as a function which accepts keyword arguments. The very next thing you will want to do is include a function call, e.g. %a(href=url_for('my_endpoint')).

At this point, you are going to have A Bad Time™ with regular expressions as you can't deal with arbitrarily deep nesting. I "solved" this particular problem by scanning character by character until we have balanced the parenthesis, with something similar to:

1
2
3
4
5
6
7

def split_balanced_parens(line):
    depth = 0
    for pos, char in enumerate(line):
        depth += {'(': 1, ')': -1}.get(char, 0)
        if not depth:
            return line[:pos+1], line[pos+1:]
    return '', line

And things were great with PyHAML for a long time, until a number of odd restrictions starting getting in the way. For example, you can't have a closing parenthesis in a string in a tag (like %img(title="A sad face looks like ):")), you can't have a colon in a control statement, and statements can't span lines via unbalanced brackets.

If only you could use Python to tokenize Python without fully parsing it...

Read more... (1 minute remaining to read.)

which is a handy utility for discovering the location of executables in your shell; it prints the full path to the first executable on your $PATH with the given name:

$ which python
/usr/local/bin/python

In a similar stream, I often want to know the location of a Python package or module that I am importing, or what entry points are available in the current environment.

Lets write a few tools which do just that.

Read more... (1 minute remaining to read.)

One of the local pubs styles itself after geek/nerd culture (e.g. sci-fi, fantasy, and board games). The back of the coasters feature a word-search. It reports to contain 45 "geek words and phrases":

Writing code to solve a word-search isn't particularly tricky (if you remember to use a prefix trie) as long as you have a list of words to find, but in this case we are given no such clues.

But, since we are tremendously lazy, how can we solve this with code anyways?

Read more... (2 minutes remaining to read.)

The last few times I overhauled an execution environment I required people to execute the bulk of their tools via python -m package.module instead of python package/module.py (to enable the development environment).

The downside is that you lose shell autocompletion, which can be a big deal if you have dozens of tools that you only occasionally use.

This addition to your ~/.bashrc fixes that.

Read more... (1 minute remaining to read.)

Lets define a classproperty in Python such that it works as a property on both a class, and an instance:

class classproperty(object):

    def __init__(self, func):
        self.func = func

    def __get__(self, obj, cls):
        return self.func(cls, obj)

It can be used thusly:

class Example(object):

    @classproperty
    def prop(cls, obj):
        return obj or cls


x = Example()
assert x.prop is x
assert Example.prop is Example

Is this a good idea, or a bad idea?

(Hint: I don't know.)

Real world photography is a fantastic and easy source of data in computer graphics and visual effects, be it for textures, backgrounds, or light maps. There is one complication that is very easy to overlook, and tricky to get right: linearity.

In real world light transport (and the simulated version of it in our various renderers) the math of light operates in a linear manner. That is, light source A with intensity 1 and light source B with intensity 1 will combine to intensity 2 (in whatever units those numbers are).

However, image capture and display devices do not work in a linear space. This is mostly historical, and maintained for backward compatibility for the content that has been produced in the past, but we must still deal with it to this day. In order to be immediately useful for 99% of cases, non-RAW image formats (as produced by your camera or the RAW converter) have the inverse curve already baked into the image so that they appear similar to the real world when viewed on a display.

It is this curve that must not be applied in order for our photography to be representative of the world (as far as the math is concerned).

I won't get into what "gamma" means, or scene-referred vs. display-referred imagery, so for an in-depth look at linear workflows see Sony Pictures Imageworks' 2012 SIGGRAPH course nodes (PDF) and this FXGuide article.

So how do we get linear output from our DSLR?

Read more... (3 minutes remaining to read.)