Fog on the Frog on a Log in the Bog

Been tuning my fog implementation a little more.  For the D3D9 version I had used a single effect file which was loaded and compiled twice - once with a #define to switch fog off, the second time to switch it on, and then selecting which version to use at runtime.

This worked well enough, and I was planning on doing similar for D3D11 but as I found out, the Effects framework is now semi-officially in deprecation-land (it's no longer provided in compiled format with the SDK), and for other reasons I wanted to move away from it anyhow, so I had to find an alternative solution.

There are a few different ways of doing fog, and the one that struck me as being quite neat was depth-buffer-based fog.  You just read the depth-buffer as a texture, reconstruct the view-space position from it, and run the result through your fog equations, blending over the main scene as a fullscreen quad.

I actually have this working under OpenGL in a test setup, and - while it works - there are a number of failings with it.  For one it's hellishly slow - when coupled with the somewhat heavier fillrate overhead of D3D11 it would be enough to add almost 10ms to each frame (remember that in order to hit 60fps you have a 16.666ms per frame budget and that 10ms extra suddenly starts looking huge).

Secondly it doesn't work correctly in all cases.  There are instances in Quake where you don't write to the depth buffer while drawing - particles, translucent water, etc.  So these will pick up fog depths from the geometry behind them, rather than from their own geometry, and - while it actually looks OK in a lot of cases - there are enough visual discontinuities to make it unpleasantly jarring.

The next approach I tried was to transform each vertex by it's local-space matrix then output that result as an additional vertex shader output.  That's then read in the pixel shader and run through a fog equation.  Again, this worked, and it even worked correctly, but I was unhappy with the extra matrix multiply per-vertex and unhappy with the extra interpolator used.

Enter Shader Model 4 and the SV_Position semantic.  While rummaging around through various help sites I uncovered two interesting facts - (1) SV_Position is actually readable in the pixel shader, and (2) the w component contains the view-space position of the fragment.  Awesome!

So the only things left to do were plug that into a standard fog equation and clean up all the older crap I had, and out came perfectly shaded per-pixel exp2 fog.



Of course, the fact that you can actually use SV_Position.w for fogging seems to be completely undocumented, but oh well...

From here I'm still in a position where I have a slight combinatorial explosion of pixel shaders, but at least the extra per-vertex work is now totally gone.  An interesting learning experience indeed.