Everything you wanted to know about the stencil buffer, but were afraid to ask.

The Bugaboo of Multipass Rendering

Multipass rendering might scare off indie game devs; it seems to tell us tales of bump mapping, realtime shadows, or other incredible, complex, tweaky effects that strain our minds and put the fear of math into us. We are meek fellows who want to make fun games, and the idea of having to think carefully about various stencil, pixel, and depth buffers seems like what we'd spend our days doing if we worked for a mainstream game developer.

And it's sort of true.

But at the same time, properly structured and understood multipass rendering is nothing to be afraid of! Well, perhaps not *much* to be afraid of. Or perhaps, what I should say is that I've implemented some semi-fancy effects and now need to sort out what my rendering passes actually are.

My Render Order (currently)

Here is what I worked out:

Pass Function

---- --------

rp_clear Clear depth, color, stencil buffers; this is the sky "pre draw"

rp_solid Render solid things, and write 1's to stencil bit 0 where we DON'T want the sky to appear; DO write to the depth buffer

rp_sky Render the sky, to whever we have a 0 in stencil bit 0; do NOT write to the depth buffer at all

rp_transparent Render transparent things, including object shadows and water terrain overlayers; do NOT write to the depth buffer

rp_lenseffects Render the lens effects, such as blinkies, tints, wavies, etc.

rp_uiobjects Render special "UI" layer objects, such as arrows and targetting reticules that must be drawn after everything

rp_ui Render the user interface

Not so bad, only six (!) rendering passes. Haha! =) How to manage all of this? Well, I really just made a list of enums, and once I had a solid picture in my head of what was supposed to happen, I reworked some of the render () calls to take a (Pass) parameter, and based on that the render call could decide what to render. Not everything is rendered in every render pass, so it's not so much stressing the GPU as it is organizing things. In fact, I may have caught a bug which involved rendering all objects TWICE. Oops!

Stencil Buffer Primer

The fancy effects in question pertained to the stencil buffer. If you're like me, you may regard the stencil buffer as a bit of a fancy thing. In fact, a little research turns up that it's not! In fact, it's really just part of the depth buffer, and probably (fingers crossed) supported on any 3D hardware you might be interested in.

The way OpenGL defines operations into the stencil buffer can be a little bit confusing, but ultimately it makes sense. Here is a handy checklist for you:

  • Decide how many bits of stencil buffer you want. You might only need one bit, but you might expect to have 4 or 8 to play with on most hardware. To determine how many you really need, first you need to decide what each bit will mean, i.e., why you need it or what it will be for.
  • You can group bits together to create small unsigned integers, as well. You could have part of the stencil buffer be a 4-bit integer, perhaps representing some type of transparency or offsetting effect.
  • You need to clear the stencil buffer at some point. For me that happens in Pass 0. Just pass in GL_STENCIL_BUFFER_BIT to glClear, e.g., glClear (GL_STENCIL_BUFFER_BIT | GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT) will clear your stencil, color and depth buffers.
  • Whether or not you want to use, or draw to the stencil buffer, you first must decide which bits you are interested in and create the bitmask for it. Remember it's a bitmask, not a bit number; this is so you can combine bits to make little ints which can be incremented/etc.
  • Set up the behaviour of the stencil buffer using glStencilFunc and glStencilOp

Understanding glStencilFunc and glStencilOp

Imagine this C code, for each pixel P.

// helper function: calculate each stencil parameter given a stencil operation GL_KEEP, etc. and the current stencil value

int StencilParam (StencilOp, CurStencil) {

switch (StencilOp) {

case GL_KEEP: return CurStencil;

case GL_ZERO: return 0;

case GL_REPLACE: return REF_VALUE;

case GL_INCR: return clamped (CurStencil + 1); // clamped clamps it to min/max range based on your bitmask

case GL_DECR: return clamped (CurStencil - 1);

case GL_INVERT: return ~CurStencil; // this is a bitwise inversion



// get our current (starting) stencil value from the stencil buffer

CurStencil = (StencilBuff[P] & MASK);

// calculate whether, based on current stencil, FUNC, and our REF_VALUE, the stencil test passes

if (FUNC == GL_NEVER) StencilPass = false;

elseif (FUNC == GL_LESS) StencilPass = (CurStencil < REF_VALUE);

elseif (FUNC == GL_LEQUAL) StencilPass = (CurStencil <= REF_VALUE);


elseif (FUNC == GL_ALWAYS) StencilPass = true;

// decide, based on whether the stencil, and depth test passes/fails what the new stencil buffer value here should be

if (!StencilPass) NewStencil = StencilParam (OP_FAIL, CurStencil);

elseif (DepthTestFailed) NewStencil = StencilParam (OP_ZFAIL, CurStencil);

else NewStencil = StencilParam (OP_ZPASS, CurStencil);

// voila!

StencilBuff[P] = NewStencil;

As an additional note, if StencilPass = false, after this code runs, then the color buffer won't be written to either. The depth test/depth buffer write happens before this code executes.

The parameters, GL_* are GL constants. The parameters FUNC, REF_VALUE, MASK, OP_FAIL, OP_ZFAIL, and OP_ZPASS are set up before your rendering pass:

glStencilFunc (Func, RefValue, Mask)

glStencilOp (OpFail, OpZFail, OpZPass);

Hopefully, the above code/explanation will help someone understand what the official documentation is trying to say:



How it works in Texas

For me, bit 0 is the only bit I'm interested in (so far) and it means, basically, "there is foreground drawn here.". So my MASK is always 1; I always call glStencilFunc (?, ?, 1) regardless what I'm doing.

Because the sky rendering operates on a totally different transformation (think: sort of a 3D analogy to a 2D parallax layer) I can't reliably use the depth buffer to determine where the sky should be drawn. What I was doing before, was drawing it *first* so it would just be there in the color buffer, and then rendering terrain. But I had more in mind for it, that required drawing it after the terrain and certain special effects had already been rendered.

So what I need to do is:

  • In rp_clear, clear the stencil buffer
  • Next, in rp_solid, draw the terrain. For this pass, I set it up with glStencilFunc (GL_ALWAYS, 1, 1) and glStencilOp (GL_KEEP, GL_KEEP, GL_REPLACE). That means, stencil test always passes (FUNC == GL_ALWAYS), our reference value is 1 (REF_VALUE == 1) and our stencil mask is 1 (MASK == 1). And it means, if the stencil or z test pass, keep the current value, otherwise write the reference value to the stencil buffer.
  • Next, in rp_sky I render the sky. For this pass, I want it to render only where the stencil value is 0; i.e., there is no terrain or other solid objects. So I set it up with glStencilFunc (GL_EQUAL, 0, 1) and glStencilOp (GL_KEEP, GL_KEEP, GL_KEEP). I.e., we are going to draw the sky only where the stencil buffer is 0.

There's a bit more to it than that. In particular, I have a "rp_transition" which, if we're transitioning from screen to screen, wipes the stencil buffer in an animated windshield-wiper pattern. This causes the sky to be drawn overtop of that part of the terrain, making it look like we've wiped the terrain off in a paper mario-esque way. As well, within rp_solid I sometimes intentionally poke "holes" into the stencil buffer, so the sky will be drawn. These holes end up being a nice reflective layer which I use really only for water. It makes it look like the water is reflecting the sky.

Solid, Sky, and Transparent Layers Screenshots

This image shows the screen after rp_solid. Not visible here is the stencil buffer. The stencil buffer is 0 wherever the screen is black, but also in some places on the center to the left where there is going to be a pond later on. We carefully rendered some ripples, directly and only into the stencil buffer, where we want the sky to reflect.

This image shows the screen after rp_sky. You can see that the sky drew to wherver the stencil buffer was 0, including the ripples in the pond.

Finally, rp_transparent has rendered the water. The water layer is rendered overtop and after the solid and the sky has been rendered. This way, the ripples aren't totally blindingly bright but are mixed with the water color. You'll notice that the player's feet are underneath the water. If there was a reflective ripple, it would completely obscure the player's feet because we would have drawn a stencil value of 0 to the stencil buffer there, which would have had the sky drawn overtop of it.


This is an epic blog post. Hopefully it will help you, as an indie developer, not to be afraid of stencil effects, or having a properly defined multipass rendering pipeline. That said, if you don't really need it or want it, don't sweat it. At least up to a point, it really is OK to just throw everything at the GPU and let the depth buffer sort it out.

Here is your special reward, it's a video of Japanes Techno Geniuses Denki Groove performing N. O. (Nord Ost):


◀ Back