As I'm getting ready to release 1.7, it's kinda inevitable that I've been thinking a bit about the overall quality. There's no doubt in my mind that it's the best DirectQ I've ever made, but it's not the one I like the most. That was 1.0, when everything seemed fresh and wonderful and new.
This one does remind me a lot of 1.0, but there are parts of it I genuinely dislike. I don't like the renderer, and I don't doubt that bugs will surface as a result of it's complex evolution and current rather messy state. Mostly I don't like it because the code is ugly. It's down for a total rewrite for 1.8, and not just for aesthetic reasons. Hacking in entity alpha support really was a trial, and I want something that I can more easily extend if another new feature that needs to be supported comes along.
I didn't get to rework the menus as I had intended. It's only a coupla hours work, but in the end other higher priorities took over. I could easily delay the release another day and get it done, but I don't see it as an absolute must-have feature right now.
There are bugs I decided to fix by just ignoring. I removed backface culling from bmodels because it didn't work and I just had too much going on to be able to invest time in it. The sky sphere also needs non-visible surf culling - it's certainly fast enough, but it's using unnecessary bandwidth. Texture loading is starting to look clunky and hacky, and could do with being pulled back to basics and built up again in a more clean manner.
I'm fully satisfied that moving back to fixed functionality was the right call, and am now happy that I've adequately replicated everything as correctly as possible. When I think of the mess I would have had to make to add in even something as simple as fog to the old shaders, it's all vindication. Being able to target a wider potential user base is also really nice.
The stability and security enhancements are good to have, Nehahra support is something I was edging towards for a while and finally got to be at least able to run it, and I'm still astonished by how well the automap turned out - sometimes I just wander around a map toggling it on and off and just looking at it.
1.8 is something that I'm also thinking about at times. I'm not totally certain what's going to be in it aside from the reworked render, but one thing I have realised some time ago is that I get more personal satisfaction from working on behind-the-scenes code than I do from graphical stuff, so maybe expect some of that.
Nehahra will move further towards completion, and any other mods I uncover that have issues will have those issues resolved. PK3 support (likely just a straight copy of the Q3A code) will almost definitely happen, and I'd like to add dzip but it's lower priority right now.
I'd also like to modernize the sound code and maybe port it to something like SDL, but I don't really like the idea of using an external library that's not guaranteed to be installed on all PCs. Release 1.6.3 is currently at 287 downloads of the engine binary, and it's a fairly reasonable bet that a good portion of those are not active in the Quake community. I've something of a responsibility to those people, and requiring them to install an external library they may not have even heard of is just not treating them with respect.
Monday, August 31, 2009
As I'm getting ready to release 1.7, it's kinda inevitable that I've been thinking a bit about the overall quality. There's no doubt in my mind that it's the best DirectQ I've ever made, but it's not the one I like the most. That was 1.0, when everything seemed fresh and wonderful and new.
Posted by mhquake at 11:27 PM
I've been throwing a lot of heavy abuse at 1.7 trying to break it, and so far it seems as though it can handle anything I can come up with without even batting an eyelid. Even ultra-heavy firefights in SuperDuper Quake don't faze it, and the unlimited particle engine really shows off it's stuff there. When 5 or 6 Devastator (I think) blasts go off at once things really get impressive looking.
A few more last minute changes have crept in; I've optimized memory usage in alias model loading a little bit, added a peak usage counter to the heap_report command, and have switched lightmap handling back to 1.6.3 style for faster map loading and (sometimes quite dramatic) memory saving, as well as cleaner code.
64 bit lightmaps are staying gone though. In the end I don't think the improvement they gave was worth the overhead, both in terms of performance and in terms of keeping the code paths clean, stable and maintainable.
Still on track for a Wednesday release, expect it sometime during the evening (GMT). I think I'm going to class this one as "stable", despite the radical changes. But I'm really looking forward to letting you lot loose on it.
Posted by mhquake at 10:26 PM
While bringing on Nehahra support I've noticed that it's maps are positively riddled with textures which have fullbright pixels in the wrong places. Here's one of the worst examples, from the Realm of the Ancients map:
It's occurred to me that one possibly viable approach to resolving this could be to examine the texels in the second miplevel for fullbright pixels, and base the decision of whether or not to load a luma on that. It's purely at the "hey, that's a neat idea" stage right now - I don't know if the creators even bothered to add full mipchains to their BSP textures - and it's fully possible that there may also be spurious texels in that too, which may even be in different places. But at the same time, the filtering down to half size and subsequent blurring, then selecting of a colour from the Quake palette, should give a more accurate result.
I could even go one step further and examine the third miplevel too, then take a wild guess that if 2 out of 3 have appropriate texels we're likely to have a genuine luma texture.
Anyway, I'm going to put this together later on today and see how it works out. More news as it happens!
Edit: done it. It's resolved the issue in the screenshot above, but there are other areas in other maps where the bad fullbrights persist all the way down to the fourth miplevel. Oh well, good enough for now, and at least better than it was.
Posted by mhquake at 7:12 PM
I was wondering if maybe I was a little unfair in my previous comments about Nehahra, but I've decided in the end to let them stand. For a mod that is viewed as groundbreaking, it's certainly a strange and clunky beast. OK, it achieves a lot, and that's good, but the grief is with the way it achieves it.
I believe that it leans a little too much towards being a slave to the Great God of QC. There are so many situations where alternative solutions should have been pursued, but the designers instead chose a QC solution, with the inevitable hackery and fooling around required to support it. Examples: instead of using proper coloured light and LIT files in the engine, it spawns persistent dynamic light entities; instead of setting a new cl.intermission state and selectively showing/hiding on-screen content based on that, it stomps over user cvars to get the result.
Full credit is deserved for an imaginative approach to a problem, but there is a limit. In the days before the engine source was available this kind of thing might be acceptable, but these guys released an engine too. There really is no excuse.
Posted by mhquake at 11:17 AM
Sunday, August 30, 2009
I've just added fairly rudimentary Nehahra support to DirectQ. Right now it doesn't actually do a lot of the extra stuff in Nehahra, but it does blindly parse them and therefore can run without crashing.
It turned out that most of what I needed was already present in the engine in some shape or form, and only a few extra server messages were required (along with SPR32 support, which was fairly trivial to add).
I'm not going to invest too much effort into completing this properly for 1.7, but will work some more on it for 1.8 - the important thing for now is that it runs.
Edit: done some more work, and I'm starting to realise that Nehahra could be a collection of vile hacks to rival even Hipnotic! A great gameplay experience, perhaps, but just look at what crawls out from under the rocks when you turn them over.
Edit 2: Remove "could be" from that last comment and replace with "is definitely". I don't know why modders do this, but stomping all over user cvars and then not restoring them properly is either clumsy or evil - take your pick. I'm taking notes for my upcoming "things I wish modders and mappers wouldn't do" post, and Nehahra on it's own is providing me with enough ammo for the entire post.
In this case I'm going to vote for "evil"; there are enough stuffcmd calls in there which just blindly stomp all over cvars without so much as a by your leave. I haven't even been able to determine how they're disabling the crosshair; it's not coming from one of the standard QC interfaces for sure, so I guess they're bypassing the interfaces here and actually sending svc_ messages directly from QC. Anyway, I have client-side code to detect this and stop it from happening now.
Repeat after me, nice and slow: The crosshair cvar belongs to the PLAYER, not to your poxy mod.
OK, you may have reason to disable it for cutscenes, but when you're done, for crying out loud PUT IT BACK THE WAY YOU FOUND IT.
Sheesh, did your mother never teach you anything?
Posted by mhquake at 8:47 PM
Saturday, August 29, 2009
I'm gonna write sometime about some of the things I wish mappers and modders wouldn't do, but before I do that I need to say how great it is that there have been mods like Warpspasm, SuperDuper Quake and Remake Quake in recent times. I believe that any sane engine coder should find these mods to be inspirational, as they provide encouragement to build a more capable engine. This is a great example of one section of the community feeding off the work of another, and even though things aren't always seen eye-to-eye, it's stuff like this that helps the whole thing hang together.
Posted by mhquake at 11:09 PM
It's not something I've really brought forward very much, but DirectQ is probably the highest capacity engine there is. There have been a few bugs along the way to this position, and the most recent (making it impossible to complete warpc without cheating) were eradicated as recently as yesterday.
In a lot of places the limits have been raised all the way to infinity. When I say "infinity", I mean of course that there are practical limits such as how much RAM your PC or graphics card has. Another practical limit is 32767 leafs or nodes - because a signed short is used to store leaf/node indexes in the BSP format itself (with negative values indicating a leaf).
This has been achieved without requiring any user intervention at all. There is no need to set -heapsize when running DirectQ, no need to specify a server-side edicts limit yourself, no need to specify a particle count limit, and so on. The engine itself will look after all that nasty messy stuff, and you can just play the game without having to worry about it.
In addition, I don't just increase limits, where possible I remove them entirely and dynamically allocate as needed - sometimes one object at a time, sometimes in batches (where avoiding runtime overhead of many small dynamic allocations is desirable).
All of this is possible because of the memory architecture of DirectQ, which I said I was going to write a bit about. I'm not going to go into any deep technical discussion, but will describe the evolution of it a little bit, as well as how it works in more general terms.
The first release used the very same hunk/cache/zone architecture as classic ID Quake, with a slightly raised limit. From there it moved to removal of the cache and zone, and using a tagged heapblock system in a linked list. The upcoming 1.7 restores the cache and zone, removes the tagged heapblocks, and runs everything through the Windows API virtual memory system.
What this means is that I can reserve a pretty damn huge chunk of process memory at startup (currently about 1 GB) but not actually use any of it until I need to. This chunk is partitioned off into various pools, each of which are dedicated to a single allocation type. There is a pool for permanent objects, a pool for the current game, a pool for the current map, and so on. The size of these pools imposes an upper limit on memory available to the engine, but because it's only reserved memory - not actually used - this upper limit can be set quite high. The current map pool, for example, is set to 128 MB - as warpc only uses about 45 MB of that, it's not likely to be ever exceeded.
Cache memory is used exactly as it was in classic ID Quake - for storing objects that can be reused between maps, such as alias models and sounds. The system is vastly simplified however, and does provide for a very significant boost in loading speeds.
Zone memory is used for all other allocations, not just small strings. The Win32 Heap functions provide identical functionality to the classic ID Quake zone system but with all of the messing around with concatenating blocks moved back to the API rather than shoving ugliness in your face.
In addition, a lot of the objects that were previously stored in big static arrays have now been moved over to dynamic allocation as required. In this way, a more accurate count of how much memory is actually being used (and how much is needed) can be obtained.
The end result is a very flexible system that takes responsibility for ensuring that Quake has enough memory out of the players hands and puts it back to the engine - where it belongs. It's not portable, but then neither is Direct3D, so I'm not too concerned about that. What it is is a whole lot more player-friendly, and it also plays nicer with the underlying OS, giving a more solid and stable game overall.
Posted by mhquake at 10:29 PM
Friday, August 28, 2009
Wednesday, August 26, 2009
Just did a playdemo on some SDA demos and have sorted out one crash bug already. Nice. Also made some small changes; integrated the world render path with all of the bmodel render paths (I had about 95% of the work done for this so it only needed a few minutes) and added the ability to clear down cached models and sounds on a map change.
I must write sometime about how the new memory architecture works. I think I've achieved something interesting and good here, but oddly enough it's come almost full circle and there is now a cache and a zone again. Funny how things work out like that.
I've been thinking - and having doubts - about my proposed new render architecture. Right now I think that I'm still going to do it, but I suppose I'm wondering exactly how much I'll achieve at the end for the work that will be put in. I'm probably not going to make any commitment either way just yet, but will likely migrate a small part of the render - something like sprites - over to it first (in the forked off codebase, of course) just to see what's involved and how it's going to work out.
Whether or not I go all the way with it, it will be obvious when you see the code post-release that a fairly huge cleanup is required. This is a direct result of migrating the initial fixed path to HLSL and then back over the past 6 months or so, with incremental changes along the way to facilitate each step. With hindsight, going back to the last fixed path release and bringing that forward rather than migrating the HLSL structure back to fixed would have probably been a better approach, but what's done is done for now.
Still on track for release next week, by the way!
Posted by mhquake at 9:51 PM
Monday, August 24, 2009
I've decided that they're going to have to wait on the next full version. I have pretty big plans for restructuring the entire main render in order to get things cleaner and more consistent across the board, and so I don't want to add in even more code right now that's going to need to be rewritten when I come to that.
I've done some experimental work on this restructuring already, which has mostly involved planning it out on paper and writing some skeleton code in a standalone app. It's enough to satisfy me that the structure is sound and the approach is valid. The next thing to do will be to fork off a branch of the 1.7 code base and work on that. I'm intending that I'll have almost everything I need isolated in a single module, so that I can just bring it in and then make the necessary changes elsewhere.
The main remaining thing to do with 1.7 is a whole batch of testing, but I'm going to back away from it a little for maybe the rest of this week before starting on that. Sometimes one can get too close to one's own code that one can't see obvious things that need attention, and taking the opportunity to view it from a distance is the best thing to do.
Testing will primarily involve SDA demos and some gameplay, although I have been using it for general play quite a bit during development. Maybe towards mid/end next week we'll be in release country, assuming of course that nothing blows up. That's roundabout 2nd or 3rd September, folks!
Posted by mhquake at 6:06 PM
Sunday, August 23, 2009
I think pretty much everything has been marked off my lists now. I'm going to do shadows tomorrow, then I need to run through a few maps and mods in order to make certain that nothing has been missed.
I'm not at all satisfied with the current structure of major parts of the render. For now the focus has been on bringing up something that (1) doesn't crash, and (2) draws stuff correctly. Unfortunately that means that there are huge swathes of code where performance and elegance took a back seat. Not that we've got slideshow framerates, but it has dropped about 5%-10% in relation to 1.6.3, and does need to be gutted, overhauled and rewritten.
I'm not however going to delay 1.7 for much longer on account of that. It's a very significant jump forward from 1.6.x in many other majorly important areas, which deserve release sooner rather than later. So watch this space!
Post 1.7 I think I'm going to follow the same pattern as for 1.6, and have some fairly rapidly emerging follow-up releases: 1.7.1, 1.7.2, and so on. I fully expect that there WILL be bugs I haven't caught this time round, especially given the completely new render required by the switch back to fixed functionality and doing proper alpha entity support.
Here's everything that's new since the last update list:
- Resolved all animating texture Host_Errors by just switching the offending texture to non-animating and devprinting a message.
- Resolved 16 char long texture names error.
- Implemented non-recursive BSP node/leaf parent matching to resolve extreme BSP depths (this is clever...)
- Resolved various automap rendering bugs and omissions, and implemented some optimisations.
- Added reporting of which lump has the funny size (this is now a Host_Error so you can continue playing on another map).
- Made r_novis take effect immediately instead of on a viewleaf change.
- Added r_lockfrustum cvar (default 0).
- Resolved long-standing issue of entities in server fatpvs showing through water surfs.
- Added several optimized recursion cases.
- Fixed crash bug where a static entity doesn't have a model.
- Renamed sv_antiwallhack to sv_cullentities (compatibility).
- Added r_normqrain cvar to enable user to disable Remake Quake rain model (default 0).
- Generalised brush render functions for inline and instanced with or without alpha.
- Added r_instancedlight cvar, set to 0 to revert to old ammo box lighting mode (default 1).
- Added key_automap state with special automap keys to scroll and zoom.
- Resolved several rendering issues and general all-round weirdness with the automap.
- Resolved issue of inline brush models that have not been sent by the server not being visible in the automap.
- Documented automap control cvars and commands.
- Removed library replacement functions.
- Switched all vsprintf to _vsnprintf to make buffers safe.
- Replaced all uses of sprintf with _snprintf to make buffers safe.
- Restored gl_flashblend 1 mode (0 is the default).
- Removed net_vcr code; removed more old grotty modem and IPX code.
- Removed 64 bit lightmaps.
- Removed disgusting and buggy old file handles system.
- Ensured that all memory allocated by DirectQ is marked non-executable.
- Added autocompletion to changelevel, game, save and load commands.
- Fixed sky fog (looks ugly on skyboxes because of corners, will need to project the skybox onto a sphere for to fix this).
- Increased cl.levelname to 128 for those really long level names.
Posted by mhquake at 11:36 PM
Saturday, August 22, 2009
Previously I hadn't been certain, but had suspected. Now I know it for sure:
DirectQ will not run on Windows 95, Windows 98, Windows 98SE or Windows ME.
This is confirmed by the fact that I am now using Windows API calls that are not supported on those OSs. You might be able to work around it with some DLL hackery, but you're on your own, and I won't be doing anything to "fix" DirectQ so that it will run on your mouldy old Windows 98SE machine.
I don't consider this a bug; it's now 2009 and Windows 95 is a 14 year old OS, Windows 98 is an 11 year old OS, Windows 98SE is a 10 year old OS and Windows ME is a piece of dirty rubbish that should have never even gotten on to a release schedule, never mind off one.
DirectQ is also not supported and will definitely not work on Windows NT4. The lack of any real DirectX on that platform is the primary reason here. Again, if you hack around and get it working, more power to you, but I don't want to know.
DirectQ may run - and run very well - under WINE on Linux or other OSs, and this is especially true since I've moved it back to fixed functionality. Release 1.0 ran fine, I am told, but the HLSL versions were problematic. I am very interested in learning more about anyone's experiences with this.
DirectQ may or may not run on Vista. I don't have a Vista machine to test on, and can't say any more in that regard. With Windows 7 now at RTM I also don't really intend ever testing on Vista.
I have tested a previous release of DirectQ on both the Beta and the RC of Windows 7, and it ran fine. I don't see any reason why it won't run on RTM, but I don't have a Windows 7 RTM machine to test on at present. I do however fully intend to make the switch to Windows 7 on at least 2 of my machines before the end of the year, so I will be able to have some more info then.
Posted by mhquake at 9:58 PM
I've just spent some time closing several hundred potential buffer overrun cases in DirectQ; most of these were inherited from the original GLQuake code, but a few I had introduced myself. I pretty much took the opinion that any such case, even if it did not result in the ability to inject arbitrary code into an executable memory location, was worthy of closing, as it's possible that a stray pointer could cause a jump to an area where it would pose a threat. There are still some left as of now, and it's likely that there will be some forever, but at least things are a lot more solid than they were.
All memory allocated by DirectQ (with the possible exception of Zone memory - I must check that one up) is explicitly marked "non-executable" also, so that's another plus.
I always laugh when I hear people arguing the case that Free Software is more secure because vulnerabilities will get patched faster. This is not necessarily a fair and accurate description; what's actually happening is that Free Software assists in creating a scenario where the response time to a known vulnerability can be potentially decreased. That's a whole lotta maybes, before you even factor in the fact that those who are potentially decreasing that response time don't have to depend on doing so for food and shelter. Security is a serious matter and one should not play games of political oneupmanship with it, but unfortunately most Free Software proponents seem hell-bent on doing just exactly that.
The whole area of security and Linux is full of similar misleading info. The simple truth, and genuine cause of the whole security scare, is that the number of Windows users is several orders of magnitude larger. If I was a worthless piece of dirt writing a virus that let me get at people's personal stuff, of course I'm going to pick on the larger target. If in 20 years time the Free Software revolution happens and most people are growing their own vegetables, weaving their own beards, wearing sandals and using Linux, then the opposite would be the case. Linux would be riddled with viruses and Windows would be seen as the secure alternative.
Anyway, I don't have a problem with the concept of Free Software in general or Linux in particular - DirectQ itself is Free Software, after all. My main problem in OS-land is with Unix; and this also applies to HP-UX, Solaris, AIX and other non-Free software. And yes, it's based on practical real-world hands-on experience, not what I read on SlashDot or have been led to believe by others.
Posted by mhquake at 6:37 PM
Some more fine-tuning has been added to the automap - proper frustum culling (well it's not really a frustum but the end result is the same), removal of the "culled objects" count (which was just there for test purposes) and drawing of inline brush models that had previously been visible to the player. This latter will get rid of the large hole in the middle of the start map as seen on the previous screenshot. The implementation is slightly hacky but for now it stays as I think I've achieved pretty much what I set out to do with it. So I'm considering the automap code "done", unless I can think of anything else that it needs.
I know I said that I was doing a feature freeze, but I think I'm going to add shadows to this version. Not volume shadows, just classic Quake shadows. I have most of the code I need already in the Quake II engine, although the MDL architecture and scaffolding is quite different (even more so with this version of DirectQ) but it should be something that's quick and easy to implement all the same (famous last words). I'm not going to do it straight away, as I intend continuing with integrating the whole translucent brush model setup next (which proved to upset the whole engine quite a lot more that I would have liked), but I'm going to save it for when I get fed up of doing that and fancy something different (just like I used the automap for).
Of course, the fact that Direct3D actually does have both a stencil buffer and polygon offset (sssshhh - nobody tell this person) makes this easier.
Posted by mhquake at 12:53 AM
Friday, August 21, 2009
A lot of bugs and glitches that were present in the release 1.6.3 version of it have been resolved. I'm also making it very configurable, and will be providing a readme for it. Here's the bulk of what you can do:
Switches the automap on or off. You should bind a key to it (it's been added to the Customize Controls menu to make this easier) for your own convenience. Switching the automap on will pause the game client-side, the same way as the menu or console does. When it is switched on it will only accept a limited subset of keys: navigation keys (see below), whichever key has been bound to "toggleautomap", whichever key has been bound to "screenshot" and the ESC key, which will toggle the automap off (in case you forgot to bind a key).
"r_automapscroll_x" and "r_automapscroll_y" (cvars, default to -100)
Automap navigation uses the arrow keys to scroll and the Home and End keys to zoom in and out. By default the sense for the arrow keys is inverted; if this bothers you then you can change these cvars to positive numbers. They can also be changed to lower or higher numbers if you'd like to adjust the scroll speed.
"scr_automapinfo" (cvar, default to 3)
When the automap is drawn a certain amount of information is displayed above and below it. Setting this cvar to 0 will disable the display of that information. Set it to 1 for the bottom information only, or to 2 for the top information only.
"r_automap_nearclip" (cvar, default 48)
Used to control the positioning of the near clipping plane when drawing the automap; anything higher than this amount of units above the player won't be drawn.
"scr_automapposition" (cvar, default 1)
An indicator is drawn to show the players position on the automap; set this to 0 to turn it off.
Posted by mhquake at 1:44 AM
Wednesday, August 19, 2009
It's a cliché, but I managed to drag a silver lining out of the brushmodel cloud. I've now been able to generalise a lot of my brush rendering code, so I should be able to send the world, instanced brushes (with and without alpha) and inline brushes (with and without alpha) through broadly the same render path (which is turning out to be very clean indeed). That was nice to do.
I'm also going to be able to restore the ability to use the lighting in instanced brushmodels, probably as an option. This will also be nice to do.
Posted by mhquake at 9:59 PM
I've been spending a lot of time working on translucent entity depth sorting. The sorting itself is trivial, it's an already solved problem: just store translucent entities in a separate list (not with cl_visedicts), calc the distance from the viewpoint (no need to sqrt it) and qsort the result (if there's only 1 or 2 then you can skip the qsort). You could alternatively do something nice with a BSP tree walk, but the qsort method is gonna be faster and simpler in most cases where not every visleaf has a translucent entity in it. The consequences in terms of DirectQ's current rendering structure are however quite appalling. To keep the code mostly simpler, DirectQ (and MHQuake before it) have always merged inline brushmodels (doors, plats, etc) with the main world texturechains, and just drawn everything in a single pass. That won't work with translucent inline brushmodels (windows, etc) as it breaks the sorting and draws them out-of-order from other entities. So the end result is that I'm gonna need to implement something similar to original GLQuake, which means yet another different entity drawing function.
If instanced brushmodels (health and ammoboxes) didn't exist this wouldn't be such a big deal, but right now it's annoying me. I almost cracked and started work on one of my longer term goals, which is to implement a common in-memory format for all model types, but in the end decided that the inevitable delay isn't worth it. I'll live with ugly code for 1.7, and keep the nicer solution in mind for the future.
Posted by mhquake at 6:00 PM
Monday, August 17, 2009
I've decided that yesterday's feature list is going to be the complete list for release 1.7, and I'm now into bugfixing and tuning mode. Aaaaahhhh, feels good.
I expect this to take some small amount of time as the radically overhauled memory system has some fairly wide-ranging implications; for example I'm almost 99% certain that at least some of the menus are now overflowing buffers (those that use cvars in textboxes). I've already tuned the memory allocation quite a lot, and the good news is that map loading times are now almost on a par with GLQuake. I think that the remaining speed loss there is coming from a combination of texture compression (takes longer to load a texture) and D3D texture loading (which goes through various different COM interfaces before getting down to texture memory).
I might make disabled texture compression the default in the new release; it'll mean a small drop in performance, but not dramatically so, and those huge maps will be faster to load, which I think is worth the tradeoff (if the perf drop was much more I would say "default on", of course).
Posted by mhquake at 6:53 PM
Sunday, August 16, 2009
Been a while since we had one of these...
- Cleared open file handles on a game change.
- Fixed .alpha "illegible server message".
- Renamed vid_mode cvar to d3d_mode (prevent invalid values in a config from causing problems).
- Removed display of total monsters or total secrets when either is 0 (remake quake compatibility).
- Lightened default contents cshifts for better underwater visibility.
- Fixed automap scaling and positioning.
- Cleaned out matrix handling.
- Transitioned 2D rendering back to fixed functionality.
- Transitioned particles back to fixed functionality.
- Transitioned underwater warp back to fixed functionality.
- Transitioned sprites back to fixed functionality.
- Transitioned alias models back to fixed functionality.
- Transferred initial camera position to view matrix so that default world matrix is always identity (easier save/restore).
- Transitioned instanced brush models back to fixed functionality.
- Completed transition to fixed functionality for all remaining items and removed HLSL framework.
- Ported FitzQuake's "old style" "correct warp" water warp render.
- Removed MH sky sphere warp.
- Split opaque water drawing to a separate dedicated pass.
- Signficantly reworked alias model drawing.
- Added .alpha support to brush models, alias models and sprite models.
- Reworked edict allocation and fixed potential crash bug when loading edicts from saves.
- Fixed (I hope!) long-standing DirectQ crash bug in Draw_CachePic.
- Added simple sky alternative for r_skywarp 0.
- Fixed static entity removal was not working.
- Added permanent gibs cvar (sv_permagibs, default 0).
- Reworked entire Heap memory allocation system to use virtual pools instead.
- Reverted to compressed vis (for faster map loading and memory overhead reduction).
- Added gl_compressedtextures cvar, default 1, set to 0 to disable texture compression on new textures.
- Restored alias model caching.
- Restored sound caching.
- Added sprite caching.
- Fixed crash bug in multiplayer setup menu.
- Fixed colour change doesn't take effect until after map change.
- Added gl_maxtextureretention cvar (default 3) to control how long unused textures are retained for.
- Switched particle render to use indexed primitives for performance.
- Implemented particle filtering by contents to fully resolve particle above water/below water issues.
- Implemented entity depth sorting for alpha resolving.
- Added fog (usual cvars) with parsing from worldspawn.
- Added zone memory using new memory system.
- Completely relocated all malloc calls to new memory pool system for better reporting of memory use.
- Added fog options menu.
- Added correct contents colour shifts depending on colour of liquid texture.
I think that when I do release I'm going to revert it back to "beta" status; there have been so many changes to how so many things work that I can't honestly call it a stable engine until it's been out there and used by people.
Posted by mhquake at 11:14 PM
Thursday, August 13, 2009
I've been checking out - and doing some things with - the way DirectQ handles it's memory allocations. Currently there are 7 memory pools used by the engine:
- Memory that is allocated one time only at startup and is permanent for the duration of the run, even if the game or maps are changed.
- Memory that is allocated for the current game and is released and rellocated as games change.
- Memory that is allocated for the current map and is released and reallocated as maps change.
- Memory used for loading temp files.
- Generic scratch memory used for general purpose stuff.
- Memory reserved but not yet committed for loading server-side entities; this is committed as entities are loaded.
- A dedicated walled-off pool for the progs.dat to keep it separate from everything else.
It's past time to make this more robust, so I'm going to overhaul this code and get a single interface at least for as much as possible.
Looking at how much memory is actually used by the engine, the Heap system maxes out at about 75 MB for warpc.bsp - so it's safe enough to assume that no other map is gonna exceed that. I currently don't have any reporting or stats on the other pools, so I honestly have no idea about how much (or little) impact on memory usage they have.
I have some ideas in my head about what the final system is going to look like - most probably a group of reserved but not yet committed pools like what's currently used for sv.edicts; but I need to work on how some of the ways DirectQ uses memory are done, particularly in relation to the huge particle support, lightmap memory mixing with texture resources, and so on.
Nonetheless, it's something that can be gradually transitioned, and will be initially a good place to be, and getting better all the time.
It's done, and the old heap system has been removed. I'm using VirtualAlloc for this, meaning that I can reserve HUGE blocks in the address space (I'm taking 256 MB for the map) then commit them as required. This gives all the advantages of both the old Hunk system and my Heap system. Because the blocks are just reserved in the address space, the amount of memory you have in your PC is not a factor - there are 2 GB of address space.
As an added bonus the code is now much more robust, and cleaner too, so hopefully a lot of the crazy crashes I was getting will be gone.
I've also reverted back to using compressed vis. Running directly off decompressed vis was nice for a while, but like a lot of ideas of this nature, what seems simpler on paper is not necessarily viable in the real world. In this case using decompressed vis was slowing down map loading and was also incurring a huge memory overhead - 20 MB on warpc, less on smaller maps.
Speaking of map loading times, they have been improved a lot by the new memory system. The previous one was prone to lots of small allocations syndrome, whereas this one is more able to cope with it. It's not quite at GLQuake speeds, but colossal amounts of time have been shaved off the loading speed after the server comes up, which was a major time killer in the previous release.
Posted by mhquake at 6:47 PM
Wednesday, August 12, 2009
Also known as ".alpha support"; just been added. Right now it's not very robust or performant, but it's a start. This is literally just a "first cut", and has quite a way to go; I don't even sort entities at all yet. No screenshots until I get that all done.
I'm also going to add in a second alpha support option, which is for Hipnotic window hacks. These are present in some of the Hipnotic maps, but they don't work at all, not even in WinQuake. They are just a regular Quake texture with an alpha stipple pattern on them so I guess the guys at Hipnotic tried it out to see if it worked, and when it didn't they didn't bother reverting it to a normal texture.
Most of the other work today involved setting up state changes in a slightly more intelligent manner, reworking alias model drawing, and splitting opaque water out to a separate pass.
Some people will be disappointed in this, but I've also removed the "MHQuake" sky sphere warp. The only sky warp options now are either a skybox or the classic warp. I intend adding a "simple sky" option to these.
Posted by mhquake at 1:00 AM
Monday, August 10, 2009
An acceptable non-HLSL solution for water warps has been found! I can't claim any credit for this - what I did was go through a few different engines and examine their implementations. The FitzQuake "old water" solution, with a non-power-of-2 subdivide size less than 32 was very very good indeed, so I plugged it into my 1.0 codebase to see how it looked there (just in case metlslime had made any changes elsewhere, I wasn't ready to modify my live codebase yet) and the end result was perfect. Some testing on various ID1 maps that are known to have bad water warps in certain areas followed, and confirmed the robustness of the solution.
So that has now migrated across to DirectQ, and the full fixed path has become viable. I made some changes to it, such as calculating the old gl_warpsin.h table in the engine rather than storing it in a header file, and cvar-izing a lot of the parameters for better flexibility and user control. I don't know yet if I'll expose these in the menus, they are really there for "fun" rather than anything else, and messing too much with them could have undesirable effects. The casual audience won't want to know about them and the hardcore audience will find them through the cvar system and examining the code anyway.
One further change is addition of more granularity to the sin table. The old GLQuake default was 256 values; I'm intending to go up to 1024 or 2048 here. This will result in much smoother warps overall, which - as always - can only be a good thing.
Performance wise, which is important, it's quite a bit faster than the pixel shader method (because sin calcs come from a lookup and are per vertex, rather than being "raw" and per pixel).
Speaking of performance, one odd effect of moving back to fixed functionality is that things in general have become a coupla percent slower. I'm assuming that this is because I now have state change overhead rather than just being able to do calculations on raw data in shaders. On the other hand the fixed functionality path is not optimized at the moment, so there is room for some gains.
Anyway, credit and thanks to metlslime is very much due here.
Posted by mhquake at 9:44 PM
After my recent fun with render to texture I've decided to backtrack a little. I'm a firm believer that the wise man knows when to walk away from something that isn't working out, and the render to texture issues were something that I had encountered before, and were - in fact - the primary reason why I started moving to HLSL in the first place!
So rather than a full changeover to fixed functionality we're now looking at a partial changeover. Water will still use HLSL where available, and I'm hoping to have a fallback software mode which uses something else (what that is going to be I'm not certain yet).
Just to elaborate on the rationale behind changing back to fixed, as the reasons are scattered over various posts going back over a month. There are in fact two main reasons behind this:
- Compatibility with older machines. There are a few older PCs that don't have the required PS 2.0 support, and locking them out of DirectQ defeats one of the reasons why DirectQ exists. I also believe that there are compatibility issues when running on Linux under WINE; this is not a target audience, but it's still nice to restore full support.
- Code simplification. The HLSL path wasn't that bad, but I found myself in a situation where if I wanted to add features like .alpha support or even fog I would have to do major shader rewrites with extra passes and even more infrastructure around them to support it. This doesn't happen with fixed.
It's not the first time I've done this backtracking thing (release 1.2 came very close to having a HLSL path and I still have experimental code from that build) and it likely won't be the last. It's a useful exercise when doing any major code changes, as it allows you to make a dry run at the changes, learn what works (and what doesn't) and then bring your knowledge forward to a fresh clean codebase.
Sunday, August 9, 2009
I'm starting to feel as if I'm in a time warp here...
Today I completed the render to texture water warp updates. The bad news is that we're right back to DirectQ 1.2 performance with them; i.e. slow. Note slideshow slow, and still very playable, but slow nonetheless.
Further analysis reveals that the whole render to texture framework has minimal FPS loss right up the the moment you start to render something. What this means is that I can retrieve a surface from the rendertarget texture, set it as the new rendertarget, release it when done, and subsequently use it as a texture for the normal render, with about a 1% FPS drop. Sweet. However, as soon as I draw to the new rendertarget I get something like a 30% FPS drop.
This is across the board, consistent, and - funnily enough - static. It doesn't matter if I update 1 surface or update 50, the drop is a steady 30%.
Now what this says to me is that there is something else at work here, most probably some flag or option which is required but which isn't as well documented as it should be.
The next step is to go hunting for some practical real-world RTT examples and compare my code to them so that I can get a better feel for what's happening (and why).
Posted by mhquake at 4:51 PM
Saturday, August 8, 2009
The standard Quake sky warp is now completed. Once again I ran into an almost undocumented "feature" of Direct3D, which I had encountered before but which hadn't become an issue until now. This time it's texture matrix transforms and the issue is to do with where in the matrix the transforms are stored.
To elaborate, here's a standard translation matrix which is used by everything else:
And here's a translation matrix for the texture matrix:
The big deal here is that you can't use the same code for transforming the texture matrix as you would use for anything else. And where is it documented? Where would you or I document it? I know I'd put it in the description of maybe the D3DXMatrix* functions, and perhaps in the general overview of matrices, and perhaps also in the Texture Coordinate Transformations section. Microsoft however chose to mention it in the "Special Effects" section, purely in passing, in sample code rather than in a general discussion, and without drawing any attention whatsoever to the fact that hey! this might be different!
Here is the sum total of the documentation of this "feature", preserved for posterity:
Posted by mhquake at 10:34 PM
Thursday, August 6, 2009
Here we go. Everything is kind of done but as you can see there is still a way to go. Right now neither sky nor water animate at all, and the front sky layer isn't being drawn either. But still, it's a significant milestone all the same and I'm reasonably happy to be there with it.
Posted by mhquake at 10:15 PM
I think the only things remaining to be done are sky and water. Sky should be straightforward enough, but I'm still reasonably uncertain enough about how to proceed with water. Right now my thinking is that I'm just going to say "to hell with it" and go with render to texture. Calculations indicate that for a native 64 x 64 texture we're not really getting too much extra overhead on video RAM; the big killer is when you're using large external textures where it becomes about 20 MB per texture for a 512 x 512 texture. I might clamp liquid textures at 256 x 256 to mitigate this (overhead shrinks to 5 MB per texture).
I've previously done tests where I've updated mipmaps for these at runtime (using D3DXFilterTexture) and performance has been good, so I'm not too concerned about that part.
64 bit lightmaps have also yet to be tackled. I've followed up on my intention to use a modulate with a default diffuse of 0.5 though, so a semi-arbitrary sliding scale for lightmap intensity has been restored. This is achieved via a combination of the sliding scale (0 to 2, default 1, r_lightscale cvar) and the gl_modulate cvar (defaults to 2), giving full lighting range of 0 to 8 x. The default here is 2, so you'll be able to boost brightness quite a bit using these if you want.
Once I have those done I'm going to do a further review of the overall new renderer just to clean up unoptimized paths and shake out any bugs. Fortunately I've been able to just lift huge chunks of code from the Quake II engine (which was the reason I wrote it, so I'm happy with obtaining that objective) so most of it should be fairly good.
A lot of people will be happy that I've reduced the intensity of the underwater contents colour shifts quite a bit. They really were too strong, and it was virtually impossible to see where you were in many cases. The strange thing is that when I was recently fooling around with WinQuake I discovered that my colour shifts turned out to be identical to WinQuake's, so if GLQuake has lighter shifts it's GLQuake that's wrong. This however is one area where I'm prepared to say that the way WinQuake did it does NOT have priority.
One other thing I want to do - unrelated to the fixed pipeline path - is overhaul the menus. Parts of them are heading in the direction I want, but other parts use a very old-style menu division. Where I want to be is to have submenus of Options divided by what each submenu relates specifically to, so I'm talking about a lighting menu, a models menu, a texturing menu and so on. This will result in most menu options being more logically grouped together, whereas right now they're a bit scattered in places. I don't know yet if this will happen for the upcoming release or if it will happen later.
It's still too early in the game to indicate a release date, but I'm hoping it will be a small number of weeks.
I haven't said it yet but it's long overdue, so I want to yell out a HUGE "THANK YOU!" to everyone who downloaded DirectQ 1.6.3. I'm both shocked and flattered by the number of downloads here; 293 downloads (215 of which were the engine binary) is really good for what is at the end of the day a minority-interest engine for use with an ancient game. Even if you only ran it once and then deleted it, I don't mind. It's things like this that can help encourage me to keep going when I'm feeling a bit "meh" about it, so - again - cheers y'all.
Massive thanks also due to the regulars who - poor deluded souls that they are - seem to like this engine, seem to want to use it, and who offer me feedback on it. Testing is the hardest part of any development job, and any comment is worth it's weight in gold.
Posted by mhquake at 7:37 PM
Wednesday, August 5, 2009
2D rendering has been completed, and the world render is coming close to completion. One thing I've had to wave goodbye to is my ability to set an arbitrary lightmap scale, as Direct3D modulate scales are restricted to values of 1, 2 or 4 (the same is true of OpenGL's GL_RGB_SCALE with combines; at least D3D is honest about the limitation). The gl_modulate and intensity cvars from Quake II are playing a visit however, by way of compensation.
An alternative way I might tackle this is to set a base diffuse colour of 0.5 and modulate the lightmap with that; D3D's default texturing behaviour is a modulation with 1, so there's no perf loss here, and it will open up greater flexibility (as well as giving better lightmap texture granularity). The base diffuse could then be cvar-ized in order to give back some flexibility.
I'm not certain where 64 bit lightmaps are going to fit into this setup. I did them in QII, but I'm looking for more flexibility with DirectQ so if they become an issue I'll just drop them. They're more of a luxury feature than anything else anyway, and are an addition to the required functionality rather than something the engine must have. However, I will be making every reasonable effort to be able to keep them in.
I'm quite surprised by how clean the 2D render in particular has turned out to be; definitely an indication that I'm on the right track with this. With hindsight, HLSL was an interesting experiment but not really long-term viable. I'm certain that there is a clean way of implementing it; just that I haven't discovered it.
I think I'm going to do alias models next; that should be good fun as I'll need to do them in multiple passes - normal/fullbright/alpha/alpha+fullbright.
Posted by mhquake at 10:53 PM
Today I started the process of switching DirectQ back to the fixed pipeline. Right now the entire render is in a fairly savagely gutted state, with whole chunks of code commented out and new code crudely slapped into some places. This doesn't bother me too much though as the objective is to just get stuff on the screen right now.
I've decided that I'm going to go for everything here, so 1.7 will be a full switch back. It's just too messy having a mixed render, and I want to clean out a lot of the really bad code I had in from release 1.0 so that I can move forward with the good and clean stuff.
What this means is that I'm not going to be waiting on an improved method of handling water warps before release. If I can get a good one it'll be in there, but if I can't, so be it.
The other change from what I said before is that I'm starting on the world and 2D drawing. This was required so that I could establish the correct baseline transforms for everything else, as there's a lot of interdependency in there.
Going through the old code sure is interesting, it really brings home how messy the original 1.0 setup (much of which has been largely retained even up to now) was. Hopefully 1.7 will be a lot cleaner, meaning better consistency in the render and less weird bugs.
Posted by mhquake at 8:13 PM
To celebrate my return to DirectQ I'm going to fire off a rant. Aaaaah - just like I've never been away...
But before I say why I don't like it, I want to discuss what I think is good about about it. It's nice to be able to give the user a generic performance vs quality option without expecting them to have any technical knowledge of the API. glHint is great for this; GL_NICEST or GL_FASTEST, job done. Also - and this is a more general strength of OpenGL - when developing test apps you can use it to get a quick and easy result without having to delve too deep into details. It's sorta like "I need this part to be fast, I want this part to look good" and so on.
However, that last one is one of precise reasons why it fails, because in so many cases OpenGL goes no further. The setting is purely binary, there's no intermediate shading. It's like if you were saving a JPEG and could only select 0 or 100 for the quality level.
Even worse, and I'm quoting from the OpenGL spec here:
Though the implementation aspects that can be hinted are well defined, the interpretation of the hints depends on the implementation.In other words, no matter what you select for your hint, an implementation is not bound to honour it. So back to the JPEG example, it's not even 0 or 100; it could potentially be any intermediate value. The implementation could even totally ignore the hint and just do it's own thing. So here we have a situation where code that does one thing on one 3D card could very well do something entirely different on another.
Control is completely removed from the developer and you are at the mercy of your driver. GL_FASTEST may be optimal on one vendor's card but ugly as a box of frogs on another's, GL_NICEST could give good performance on one but be a slideshow on another, there might be no difference at all between them on a third. You have no control over what your code does.
Now, call me old-fashioned but I think this is a Bad Thing writ large.
Tuesday, August 4, 2009
I've just got the kick up the behind that I needed to get back into the engine - a classic/momentous/etc map has been released that DirectQ breaks on!
I am of course talking about the mighty ijed's Remake Quake Demo, which everyone should download and run in the engine of their choice (although not DirectQ just yet, obviously).
Anyway, after fixing it and doing a brief run through the map, I'm wanting to add fuller support for some of the things that were missing or half-done on the previous release. So expect more regular updates and potential release dates over the next few weeks!
Posted by mhquake at 11:36 PM
Monday, August 3, 2009
OK, it's been about another week and I still haven't gone back to the engine. There are a number of reasons for this, with the main one being that I've been re-reading an old favourite - Peter F Hamilton's Night's Dawn Trilogy (thoroughly recommended); but also that I haven't really been feeling like it. If this was a job I'd just knuckle down and get on with it, but it ain't.
Anyway, roundabout now I'm starting to feel like getting stuck in again. It might take another day or two, but I think sometime this week I'm going to be firing up the old Visual C++ again and commencing the move back to the fixed pipeline.
One consideration is where to begin? There are a few logical starting points, but I'm thinking it's going to be something reasonably small and self-contained at first, just to get back into the groove. The obvious one then becomes sprites and particles, both of which use a broadly similar render path and both of which can be done in isolation. From there - who knows?
Posted by mhquake at 12:47 PM