Fun with Particles

I moved particles across to my new render recently, and so far have hit a slow down with them. Right now though it's sub-optimal and definitely not the final code, so there is room for improvement. I have 2 main ideas that should prove effective - one relating to alpha depth sorting, the other relating to more optimal batching of particles, that will definitely bear fruit.

For the season that's in it there won't be any updates for a short while, so let me wish all my readers and everyone who's downloaded my engine, played it, given me good feedback, or nagged me to get a feature in, a happy Christmas/Holiday/Solstice/Larentalia/Emperor's Birthday or whatever! You've all contributed to make this engine an enjoyable experience for me to develop.

Speeds Up Again

The earlier version of the SDK seems to run more efficiently with the way DirectQ is coded, so there's a few extra frames from using it. I'm not quite certain what causes it, but I strongly suspect it's the D3D9 shader compiler in the earlier version vs the D3D10 (even on XP) shader compiler in recent versions.

Sky has just gone in to the new rendering framework, which is increasingly turning out to be very nice indeed. DirectQ can now properly handle sky surfs on ammo boxes, including proper depth clipping for non-hlsl modes (which includes sky boxes).

I can think of at least one situation where this might be useful, which is a sky room entity stored as a BSP model. Done well on the QC side you could in theory have changing skies in a single map. This is one for mappers and modders to explore if they want.

A report of slowdowns in certain busy scenes and I'm almost certain that this is particles. The DirectQ particle engine is fairly well optimized, but I've completely removed the upper limit on the number of particles. You could throw 100,000 particles at it and it will quite happily set them up and render them - slowly, but happily. ID Quake only lets you have 2,048 particles on-screen at any time, and I think I'm going to add a cl_maxparticles cvar to let you have some control over this. For comparison, of the ID demos, demo1 is the particle monster (yes, I've counted the number of particles in each demo) and it tops out at less than 4,000.

This is one situation where more widespread availability of D3D10 and geometry shaders would have been great, as particles - being totally dynamic - need to send quite a lot of data to the GPU each frame. In fact, of the entire scene, it's perfectly possible for the overhead from particles to be a number of times higher than everything else combined.

I have a totally crazy idea brewing in the back of my mind that, if I can pull it off, might just turn out to be something of a Holy Grail for Quake engine games. Right now I haven't even written any exploratory code, never mind proof of concept or production quality, so I'm not willing to go into much detail for fear that I might generate some false expectations (it might not even turn out to be that big a deal, depending on how it works in practice).

Imagine huge outdoor scenes with massive draw distances but no real slow down. That's what I'm hoping to aim towards. Even if I do start coding something I don't think it's likely to be a 1.8 feature; but - depending on how it goes - it could happen soon-ish.

More Nice Changes coming in

I've switched DirectQ back to an older version of the D3D9 SDK (September 2004).

The big thing about this is that it will no longer be required to upgrade your DirectX on Vista and Windows 7, and older D3D9 installs on XP should also work fine.

With hindsight I should have done this a long time ago, but better late than never.

DirectQ Speeds Up

DirectQ 1.8 is now faster on Windows 7 that 1.7.3 was on Windows XP. We're still not quite back to 1.6.x speeds, but we're inching closer each day.

This is all a side effect of the new rendering code I've been doing. I've solved several major problems, including efficient and effective state batching, handling multiple different types of input formats, multiple entities sharing the same model, and merging model rendering into the world without unpleasant side effects.

As an added bonus the code is really clean and elegant, and nice to work with. It should be possible to port this virtually unchanged (aside from the D3D API specific stuff) to just about any engine and see immediate benefit. Of course it's also reliant on the Windows API VirtualAlloc function, but if your chosen OS has a functional equivalent it Can be Done.

I'm really enjoying the work I'm doing on DirectQ at the moment, far more than at any time for a long time. It's a rare thing in any development project to be getting immediately measurable results from nice easy simple code, and also to be throwing out horrible old legacy code while doing so, and also to know that what you're getting is far more functional and robust than what you had before, but this one just keeps on giving.

I wouldn't make any claim to be particularly talented here; it was a happy accident that I stumbled across the right way, and there were 3 aborted versions of the main code along the way here. But things are on track and moving towards a good end result.

DirectQ Bug Alert!

Two bugs in current versions of DirectQ have recently come to light.

The first one is that all brush models are being drawn twice. I had suspected that there was something aberrant in my code as I had a small but significant performance decrease when I made the original 1.7; I had been putting it down to the overhead of state changes from the fixed render path. This was discovered when I had removed some of the brushmodel rendering code during the construction of my new renderer (which is coming along quite nicely).

Fixing it would involve taking a whole heap of code that is already a mess and making an even bigger mess of it. This will have to wait on 1.8, which will be a lot cleaner and more flexible (did I say it's coming along quite nicely?)

The second is a crash bug. It just happens that if when - and only when - you start up DirectQ and enter a map for the first time (and it's only on the first map you enter after starting the engine) there are no alias models in the view aside from the gun, DirectQ will crash.

This is pretty obscure; for a whole load of people the first map is e1m3 - the Necropolis demo - which has a nice big whopping torch in the view to prevent this. The majority of other first-run maps will also have some kind of models visible.

It's a one-line fix, but right now I'm not going to. I haven't had anyone report such a crash yet, so for now the fix is to not modify your autoexec to skip the Necropolis demo.

Regarding 1.8, the sands have shifted a little. My original plan was to finish off my D3D8 port of QRack and then start on 1.8, but nobody can predict the way things will turn out. 1.8 is actually well underway, and D3D8 QRack is now on the backburner.

Apologies to anyone who was looking forward to that.

The good news however is that I'm starting to see 1.8 as happening relatively soon. What I'm currently planning on releasing will primarily contain just the restructured renderer and the alias model cleanup. These two are both significant enough to justify the version number bump, and overall it seems to be the path of wisdom to release a less radically developed engine; it means fewer places to go looking for bugs, after all.

Happy Birthday to the Quake Source Code!

This is the major anniversary I wrote about a coupla days ago. It's now 10 years since the Quake source code was released. 10 years, imagine that. Not only is the game itself still going reasonably strong, but engine mods are arguably stronger than ever before these days - there may be fewer, but the stuff that's being done is well ahead of what was ever done before. It's a testament to the inspiration this rickety little old game provides to people.

Rock on.

Cleaning out the Alias Model format

Just done some work on DirectQ 1.8 today (when I should have been working on QRack).

It's no secret that the Q1 alias model format is horrible. It's horrible to load, horrible to work with, horrible to render from. Even it's own mother doesn't love it.

Having to work with it was the main reason why I delayed doing shadows in DirectQ for so long.

The on-disk format is obviously beyond redemption but there are things we can do with the in-memory format. I've been meaning to do this for a long time, but finally knuckled down to it on a lazy Sunday afternoon.

No more commands, no more order, no more negative vertex counts, no more not knowing anything about what you're going to draw in advance. An in-memory alias model is now a linked list of polygons, each of which contains a primitive type, numverts, an array of verts and a next pointer.

Each vert contains s, t and an index into the alias verts array. The alias verts array does not need to be pre-expanded to strips or fans, but is exactly as it comes off disk.

Side effects of this include a substantial memory saving on alias models, removal of a lot of the temporary buffers that were used to shift data around, faster loading, and an extremely clean render function (it looks quite similar to a brush poly render function now). Even the loader and structs are cleaner.

Functionally it still supports everything that the old in-memory format did, but it's far more pleasant to work with and maintain. Result I think.

Happy Birthday DirectQ!

I completely missed this, but DirectQ has recently had it's first birthday. On December 14th 2008 the first release of what became DirectQ (it was called something else then) was let out into the open. It's quite amazing to think how far it's come in that year, and equally amazing to think that this all happened after a few years during which I had released nothing.

Unfortunately time was against me and 1.7.666c didn't make it out on the same date, 2009 (it was 2 days later).

There's a major Quake anniversary also coming up in a few days time; I hope you all know what it is.

DirectFitz is Released!

Go get it.

This is a milestone release as it's the engine which I developed over 90% of the wrapper on, so I've got a nice warm glow inside from getting it out.

In the end I decided not to bother with either scissor test or r_oldwater 0 mode. I did however spot some major optimization potential in liquid surface subdivision (which should be able to be carried forward to GL FitzQuake) so that's something new for you. Basically what this means is that water surfaces will tesselate much less for a given value of gl_subdivide_size, but retain much the same quality.

The next release will be DirectRook (QRack). This is already mostly done, but there is quite a bit of work in bringing on the remaining items (I may even need to drop some QRack features, but I'd prefer not to - we'll see).

Following that I'm going to consider my work largely complete. I've had a look at the DarkPlaces code, and while a DirectHavoc is certainly possible, creating one would require considerable disruption of it's OpenGL setup code. To my mind that really defeats the purpose of creating this wrapper in the first place. I might do it sometime just as an exercise (everyone really should read more of LordHavoc's code) but I make no guarantees.

I'll probably also release updated versions of the wrapper from time to time, but nowhere near as regular as now.

DirectFitz is almost done

This is going to happen sometime over the next 2 days. All that's needed now is for me to go through the code and pick up on anything I've accidentally missed out. How many of those "accidentally"s there are will determine how long it takes.

I've had tremendous fun implementing GL_QUAD_STRIP in Direct3D (FitzQuake uses this for r_showbboxes). The OpenGL documentation could certainly be a lot better here - I was working off the basis that the documented vertex indexes were 0-based, and couldn't figure out what was going on - turns out they were 1-based all along.

Hint to OpenGL documenters - use the same array base as programming languages do, please. You're writing for programmers, and programmers don't actually care if you're technically correct according to some criteria that's irrelevant to them.

Sheesh, I know I harp on a bit about the quality of D3D documentation, but here's evidence that OpenGL ain't perfect either.

---

For the most part I like to try get a port as complete as possible, but this will be one where I might fall short. Two main things will likely be missing; the first will be on account of the fact that D3D8 doesn't support scissor test (although I might try to hack something up), the second will be on account of the fact that I didn't feel like bothering with implementing FitzQuake's render-to-framebuffer water (although I will need to implement the API calls used for other engines, so I might knuckle down and do it yet).

---

I also haven't done the stencil buffer; FitzQuake 0.85 requests a stencil buffer, clears it if it has one, but doesn't actually do anything else with it.

---

Polygon offset was also fun. D3D8's implementation was very basic (it would have been much better if I had used D3D9, but I wanted to retain compatibility with as much horrible old hardware as possible), just being a single Z bias factor which is only capable of biasing towards the viewer (sometimes we need to bias away from the viewer). In the end I hacked a little and set the default to the midpoint of D3D's range, thereby allowing me to bias towards or away at the expense of some range (which in practice wasn't really used).

---

More next time (which will hopefully be the release).