PxrLMLayer Blending is Broken

NOTE: This problem has been fixed by Pixar and released in RenderMan 20.7. It was a interesting bug, so read on anyways!

While working on Shadows in the Grass I've been exploring building RenderMan shaders out of multiple PxrLMLayers. I've been having issues composing shaders of multiple translucent layers.

I've reduced it to a very simple case:

Example 1 (top stripe): I have a base diffuse material with an overlay. The base material is black, and the overlay is red with a gradient mask. Looks like I would expect.

Example 2 (middle stripe): I have a base diffuse material, and two overlays on top. The base material is black, the first overlay is green with a mask of 0, and the second overlay is red with a gradient mask.

I would expect the middle stripe to appear like the top stripe, but the green is somehow getting into the calculation despite its mask being zero.

Example 3 (bottom stripe): An SeExpr I created that I believe matches the behaviour:

R = [1, 0, 0];
G = [0, 1, 0]; 
r = s;

Rm = r^3 * R;
Gm = r^2 * (r - 1) * G;
Rm + Gm

Hand-made RIB which exhibits this problem is:

Pattern "PxrSeExpr" "expr"
        "string expression" "[s, t, 0]"

    Pattern "PxrLMLayer" "red"
        "reference float layerMask" "expr:resultR"
        "color diffuseColor" [1 0 0]
        "color specularColor" [0 0 0]

    Pattern "PxrLMLayer" "green"
        "float layerMask" [0]
        "color diffuseColor" [0 1 0]
        "color specularColor" [0 0 0]
        "reference struct lmlayer" "red:result"

    Bxdf "PxrLMDiffuse" "base"
        "color frontColor" [0 0 0]
        "color backColor" [0 0 0]
        "reference struct lmlayer" "green:result"

I've started digging into the source code, compiling my own PxrLM plugins, etc., and I'm coming to the conclusion that the implementation of the layer-to-layer composite is incorrect, as it always assumes that the bottom layer is fully opaque.

The over operation (defined in lmParam.h and used in lmLayer.h) appears to be:

\[ over(fg, bg, mask) = mask \times fg + (1 - mask) \times bg \]

With R/G/B as my three layers, and r/g as the two layer masks, I get:

\[ B*(1 - r^2 - g \times (1 - r)) + (G \times (1 - r) + R \times r) \times (g \times (1 - r) + r^2) \]

Setting b and C to zero (to match my example above), this simplifies to:

\[ r^3 \times R + r^2 \times (r - 1) \times G \]

Which is the same as my SeExpr above.

So... "fixing" this is likely way out of the range of the time I can commit to this. I'm likely going to be doing my own composites and feeding them into a single BXDF.

Does anyone have any other thoughts or insights for how I can get what I am looking for?

UPDATE 1: Pixar has replied to my report on their forums:

I have identified the problem. Thanks for the repro case.

At least it isn't just me.

UPDATE 2: The fix was going to be in RenderMan 20.5, but it was pulled during QA due to unresolved issues. Hang tight!

UPDATE 3: The fix was released into RenderMan 20.7 (see the release notes). Hooray!