I still haven't gone back to DirectQ as I'm currently waiting for the coding "itch" to strike me again. I know it's gonna come soon, but right now I just don't have it.
Nothing much else to say, just giving a brief status update as it has been a week. :)
Monday, July 27, 2009
I still haven't gone back to DirectQ as I'm currently waiting for the coding "itch" to strike me again. I know it's gonna come soon, but right now I just don't have it.
Posted by mhquake at 7:35 PM
Tuesday, July 21, 2009
OK, as I said earlier the plan is to now get back to DirectQ and start on moving it back to fixed functionality. I've decided not to do a big-bang apporach on this one though, so the next release will most likely be a half-and-half affair. The major outstanding item is water warps - I still don't have a satisfactory fixed functionality replacement. Render to texture warp updates are closest in terms of quality, but the video RAM overhead (as well as the per-frame mipmap regeneration) precludes this from being a realistic prospect.
I'll post later about exactly what the problem is with water warps.
Anyway, before I start on that I need to do a total rewrite of video startup. It's clear to me now that I have too much legacy baggage in there from the old GLQuake startup (which was based on WinQuake startup, and which assumes that certain things happen in certain a certain order). This needs a fresh new beginning to get a stable baseline going forward.
But before I even do any of that I'm taking a break. I burned myself out a little towards the end of Quake II, so it's time for one of my periodic recharges. It's a good opportunity to reflect on things learned during the QII process rather than diving straight into code. I'll continue to post some thoughts on the direction of the engine, things to come, and problems to be solved, during this break; but I won't be actively working on anything for a short while.
Posted by mhquake at 11:35 PM
DirectQII that is.
Get it here.
I'd encourage any potential downloaders to take a good read of the release notes on that page, by the way. They're very important, and they should hopefully leave the status of this particular port in no doubt.
Having said that, the engine should run reasonably fine on most hardware, and may be significantly faster than OpenGL Quake II for some. But bear in mind that if it does crash or otherwise misbehave, it's not a release that will be either supported or developed any further going forward. It was written for a specific purpose, and has achieved that; if anyone uses it and finds it useful that's a bonus.
I'll post later today or tomorrow about where things stand with the main DirectQ codebase, which I'll be going back to exclusively from this point on.
Posted by mhquake at 9:47 PM
Monday, July 20, 2009
After that last crash bug I'm holding off on releasing until I'm reasonably certain that there aren't any more nasty little surprises waiting for me. The main reason is that I'm committing to this being a one-time-only release. I don't have the capacity to maintain two codebases concurrently, and the objective of the Quake II build was always to feed results from it back into DirectQ. So even if it does make it out with bugs, and even if it doesn't run on people's cards, there won't be updates.
The other option is not to release at all; my original plan didn't really involve any commitment to releasing, and I'll still achieve my objective anyway. However, having come this far it would be a shame not to.
Anyway, I'm also going to provide a replacement gamex86.dll with it, but probably won't be providing any source code for that. Why not? Because it's just a straight recompile of unchanged ID gamex86 code, and therefore if you want the source code you can just download it from ID. No sense in confusing things by releasing unchanged source code; the dll release will just be as a convenience (cos I'm such a nice guy) for anyone who doesn't have the necessary resources to compile ID's release themselves.
Before anyone goes howling about "GPL violations", I'd encourage you to actually read the text of the GPL. I am "offering equivalent access to copy the source code from the same place" (GPL section 3c) as I originally received it, and this is the very same source code that was used to compile the gamex86.dll I'll be distributing.
So let's get this nice and confusion-free now: The DirectQII gamex86.dll will NOT be in any way different to ID's 3.21 gamex86.dll, it is just being provided as a convenience for those who can't compile it themselves, the source code is NOT being released by me because I don't want anyone getting confused into thinking that it is different, because it is NOT different. If you want the source code you can download it from ID, it is THE VERY SAME.
Posted by mhquake at 8:20 PM
Saturday, July 18, 2009
It turns out that the most likely cause of the recent crash bug was an earlier version of the gamex86.dll (possibly 3.19) I was using and which inadvertently slipped through, and that recompiling the dll based on the released 3.21 source made it go away.
Similarly, running a freshly compiled stock ID Quake2.exe and ref_gl.dll with the older gamex86.dll also repro'ed the bug.
In one sense it's a relief as it means my code is good. On the other hand, the perfectionist in me would like to be able to find out exactly what was happening in the older game dll. I guess I'll just have to accept things as they are.
My gut instinct however is that free was being called twice on the same block of memory; all the symptoms match. There's no way of verifying it however.
This is a bit of a pickle now. I really should release a gamex86.dll with the engine binary release, but in an ideal world I shouldn't have to. Oh well, if it's what's needed...
Posted by mhquake at 9:38 PM
Friday, July 17, 2009
I've found a bad crash bug just immediately after the Big Gun map, when you switch from the intermission (cheesy big bare room with lots of explosions going off in it) to the cinematic for the next unit. Just on the brink of releasing too. :(
Anyway, it's to do with Zone memory allocation, and - specifically - freeing. So far I've moved everything out of the Zone aside from gamex86 stuff, and have switched the Zone code back to C, suspecting that there may be an incompatibility between the game C code and the engine C++ code. If I comment out the free call it works fine, of course, but I'm leaking memory by doing that.
So I need to pay a serious visit to this code, bringing up a gamex86.dll in debug mode, and see what's going on. I suspect that I'm going to have to do some serious rewriting of it (I don't like this code anyway - lots of tiny allocations, and the struct that manages them is bigger than most of the malloc sizes).
Anyway, release is off now. Sorry.
Posted by mhquake at 12:37 AM
Thursday, July 16, 2009
The previous "What Needs To Be Done With OpenGL" post (available here) made some suggestions that - in my opinion - would improve the state of OpenGL and make it an API worth considering again. I deliberately kept away from issues like future evolution and platform compatibility; in the case of the one I believe that the changes can be made now and in such a way so as to make them available to all current versions (say by rolling some of them into GLU); in the case of the other it goes without saying, and anyway I had always intended to focus specifically on Windows so that I could do a fair and like-for-like comparison with Direct3D.
I'm not of the "Direct3D Good, OpenGL Bad" mindset - in a lot of ways OpenGL remains a far superior API to Direct3D (procedural vs COM based is a matter of personal opinion and doesn't come into it). There are certain things that are also wrong with Direct3D, and to balance the argument a discussion of them is also required. That's what I'm here for today.
Stop using Direct3D as a Vista marketing tool
OpenGL has proven that most D3D10 features actually can in fact run successfully on Windows XP. There is no reason whatsoever for features like geometry shaders to be restricted to Vista, and the arguments supplied by Microsoft don't wash. Yes, they might want to sell as many copies of their OS as possible, and that's fair, but how about letting it succeed or fail on it's own terms, rather than giving it artificial leg-ups like this?
Keep the API consistent between versions
Every time a new version of D3D comes out it's almost like you have to learn a new API all over again. There's no graceful evolution, and legacy work becomes worthless. It can't easily port to the new version, and is of minimal value to coders looking for samples of something they are trying to implement.
The current state of D3D documentation is a case of "when it's good it's very very good, but when it's bad it's horrid". There are far too many places where related information is not grouped together, and where a developer has to pull together info from multiple sources scattered throughout the help file in order to get a clear picture of something. My understanding is that MS documentation is written by technical writers following a spec, then reviewed by programmers who design the API. How about getting it also reviewed by someone who actually uses the API? With regard to the samples provided, it should be ensured that they actually compile and run as written, and that they use the core API instead of some sample framework - it's their job to document an API, not to show off how good their programmers are. Having to delve through 5 layers of abstraction just to find out how to draw a triangle (exaggerated example) is not what somebody trying to learn the API from their documentation wants.
Sort out some of the basic data types
The implementation of Matrixes in D3D sucks. A big one. Hard. In the core API you basically have to set up each row and column yourself - by hand. You have to perform your own matrix multiplications. The D3DX library helps a little, but it's still obscure enough to trip up someone who just wants to rotate an object (having to instantiate a vector class just to do this is OOP overkill too...). Likewise, it should be possible to draw a primitive by just passing individual vertexes one at a time; too much of D3D assumes that your data is already set up in the correct format it expects, and as a result it's potential for use in experimental work is reduced. Colours need to be standardised; use of ARGB here, ABGR there, BGRA in the other place, and RGBA elsewhere, and with no reliable correlation between the component ordering and the format used (e.g. one texture format is D3DFMT_A8R8G8B8 but it actually expects data in BGRA order; others don't) just leads to confusion.
Move some of the heavy lifting that programmers have to do back into the API or runtime
Here I'm talking about stuff like handling device resets, loading textures with data, gracefully handling API call failures, and so forth. Some of these should be handled automatically by the runtime without any programmer involvement required at all, others should have API calls merged - I'd love to see a D3D equivalent of glTexImage2D, which just created a texture object and filled it with supplied raw data (and even generated mipmaps) in one fell swoop. There's something like it in D3DX, but it needs to be rolled into the main API and made more flexible. It's great that D3D is more low-level than OpenGL and gives you more specific control over how things work, but sometimes you also want a quick and dirty.
Anyway, that's that. Some of these may already be in D3D 10 or above; I don't know as I don't have Vista and have no intention of ever installing it (Windows 7 on the other hand looks a huge improvement), and given the development strategy for D3D it's reasonable enough to say that there is no chance of anything like these ever appearing in D3D9. It should be obvious at a glance that most of these also relate to areas where D3D compares unfavourably to OpenGL.
One conclusion that can be drawn from this post and the previous one is that there is potential for a truly great 3D API that can be all things to all people, but neither D3D nor OpenGL are currently there. Each needs more evolution in the direction of the other to get close, but some of the central philosophies underpinning each may well prevent it from ever really happening.
Wednesday, July 15, 2009
What is - I think - the last code change has just gone in. All this one does is just center the window properly when starting up in a windowed mode instead of relying on the vid_xpos and vid_ypos cvars. I don't know if that's going to be to everyone's taste, but you can always manually move the window after startup anyway. No big deal.
What's going to follow now is a few more days of playing Quake II in debug mode to see if any gremlins fall out. This - together with the SDA demos - was a most useful exercise, and definitely helped to root out some bugs.
Then we're onto release. The engine's going to be called DirectQII, but then you already knew that, eh?
I'm not certain if there are going to be any QII releases beyond this one. The objective was to use the knowledge I'd gained over the past half year to build a proper fixed-functionality D3D app in a codebase that's broadly compatible with Quake but that has some of the basic stuff implemented in a more sensible manner, and that has now been achieved, so it's back to DirectQ itself shortly.
I've established a good list of items that are scheduled to be ported over to the main DirectQ codebase, and which will involve some pretty major rewrites, but that's a good thing. Although 1.6.3 got things to a stage where I was very happy with the end result, there remained a lot of cruft left behind from the original OpenGL codebase, as well as from version 1.0, where - to be honest - I really had no idea what I was doing. Evolving beyond 1.6.3 would involve building on a foundation of this cruft, so it's a good time to strip back and get something more solid in before moving forward again. So that's going to be the main focus of 1.7. I'm going to let the ideas percolate in my mind for a few more days before mentioning any specifics.
Posted by mhquake at 9:35 PM
Tuesday, July 14, 2009
I think the graphics work is now complete. Better water warp remains outstanding, but it's not so much an issue in QII as it is in Q1 (I half suspect that the QII qbsp may do some slightly more intelligent stuff with splitting water surfs on nodes). There are still some places where the poor quality warp is noticeable but you really have to go looking for them. In any event, unless a good solution presents itself quickly enough it's not something I'm going to worry about here, although I do want an eventual solution nonetheless.
Coming up next is some work on the sound code. I've already stripped out the Primary Sound option (although you can do it, DirectSound does not advise writing to the Primary Buffer - it's much the same as writing to the front buffer in graphics) and upgraded the support to DirectSound 8. I need to remove some of the remaining scaffolding around this, then consider removal of the wave only option. As with DirectQ, I'm of the opinion that wave only was a good idea back in the mid/late 90s when DirectSound wasn't yet a mature and established standard and when there wasn't any proper driver certification available. Time has moved on since then, and a sound card that doesn't support DirectSound is more likely to be badly broken than anything else.
I have some Managed DS code for setting effects on buffers so I may try to hack in a cheesy underwater gurgle effect. I tried this in DirectQ but all I had to work from was the SDK docs, and it didn't work. I'll see how it goes.
I'm also chewing over the possibility of removing the joystick code. I'm in two minds about this one though owing to the fact that there is no real gain in simplification from doing so.
Posted by mhquake at 6:21 PM
Monday, July 13, 2009
Just added fake fullbrights to the world render, and will be doing alias models shortly. They're not really fullbrights as they exist in the Quake 1 engine, just a greyscaling, bias and squaring of the base texture, but they certainly look effective.
It's one thing that always bothered me about the Quake II engine, but thankfully it's resolved now. Of course there's bad news - because they exist on pretty much every texture, using them tends to slow things down quite a bit. I've halved the size of them compared to the base texture to compensate a little, but be warned.
Of course they're optional, use "gl_fakefullbright" to switch them on or off. The default is off by the way, as they're an addition to the baseline QII render.
Oh, and I wish I'd discovered D3DTSS_TEXCOORDINDEX before. Oh well, I have done so now. That brings me to the next topic - I've ranted enough in the past about deficiencies in the Direct3D documentation, but - to balance matters - it's getting near time for a rant similar to the OpenGL one previously posted.
Back to fullbrights, and the topic of using them in DirectQ. For DirectQ I decided that fullbright colours are a baseline software Quake feature, and therefore they are an "always on" feature. I haven't provided an option to switch them off, and have no intention of doing so. The same applies to overbrights.
I've read that some feel that fullbrights and overbrights are something that should be optional, and even that perhaps the reason why software Q1 had them was because of technical limitations. This is not true. Software Quake uses it's colourmap for calculating lighting, which is basically a great big lookup table with 256 colours across the top and 64 gradations down the side, each of which is an index into the Quake palette. It looks like this:
Now, ID could have chosen anything from the Quake palette (in fact there is headroom as the colourmap only has 243 colours) but instead they chose the colours that they did. As far as I'm concerned that ends the story, and DirectQ will continue to have enforced fullbrights and overbrights.
Posted by mhquake at 10:23 PM
The purpose of this post is not to be rude or argumentative about OpenGL, but instead to highlight some specific issues with it that were instrumental in my move to the Dark Side, and to suggest some practical solutions. I am not speaking from any position of authority or influence here, but I am someone who was frustrated enough in the past to finally take some decisive action regarding my own continued use of OpenGL.
I don't propose to discuss items relating to the future of the API. These are problems that exist now, and - in my opinion - would be better off being tackled now before any further advances are made. This would give a clean and solid baseline to move forward from, instead of trying to patch up the mess that is currently there on a piecemeal basis.
So without any further ado:
Implement driver certification.
Conformance testing is not good enough as it only validates that outputs are correct for given sets of inputs. We all know of implementations that don't have all entry points supported but yet somehow get away with advertising as a certain version. We all know of implementations that don't support all core features of the version they advertise as. Direct3D drivers have WHQL certification, OpenGL drivers need Khronos Group or ARB or WHATEVER certification. Users of a certified driver can then be confident that the driver supports all entry points and features for the version it advertises. Developers can target certified drivers and not have to worry about writing fallback paths for where a feature that should be there isn't (or doesn't work right).
Implement proper capability querying.
As things stand now the only way to get any halfway decent capabilities info is to try something and see if it fails. This is not good enough. At the very least proper capability querying should mean that you can pass an extension string or a #define to a glGetCapability function and get a return of GL_NOT_SUPPORTED, GL_SOFTWARE_EMULATED or GL_HARDWARE_ACCELERATED. Cards that advertise non-power-of-two textures as a supported feature but that don't hardware accelerate them are a perfect example here. Without proper capability querying this kind of feature is USELESS to a developer.
Publish a proper SDK.
This means something that you can download and install, including a good help file, practical examples and tutorials, hints, tips and proper procedures, tools, sample applications, the works. Look at the DirectX SDK and learn from what it does right (as well as from what it does wrong). Full sets of headers and libs for each version on each platform are REQUIRED. Having to go through wglGetProcAddress contortions just to get core entry points DOES NOT CUT IT ANYMORE. These should be automatically handled by a statically linked LIB, so that all the developer has to do is call the damn function where required.
Establish stronger ties to the windowing system.
This means a glAttachToWindow function which takes a window handle or equivalent and creates the required rendering contexts and everything for you. It can be kept reasonably generic in the API itself, with implementation details being left to the driver vendor. Driver certification would ensure that it was done right. It can provide a reasonable set of safe defaults for getting up and running with good performance and quality, and the more hardcore way can be retained for those who want specific precise control.
Accept some impurity or less-than-perfection where it is pragmatic to do so.
Classic example - the use of the "bottom-left is the origin" paradigm throughout OpenGL. This is AWKWARD and DIFFICULT and UNINTUITIVE to USE, no matter how "correct" it may be in terms of classical Cartesian coordinate systems. EVERYTHING else - image editors, Direct3D, even the way you read a book - uses "top-left is the origin". Another example is the glHint mechanism. In theory it's nice, but in practice any glHint may as well translate to GL_DONT_CARE every time, as implementations don't have to honour them. Give us the ability to say "I want this feature to work this specific way".
None of these suggestions need be dependent on any future revisions of the API; there is no reason why all of them cannot be backported to all previous versions. It's a short enough list - only 5 items - but it's my belief that implementing these would make OpenGL a great and rock solid API again. Right now developing on it is Just Not Fun. You spend more time doing workarounds for driver bugs and dealing with awkwardness than you spend doing productive stuff. Back in the days when Direct3D sucked, this was acceptable enough; in fact it wasn't even an issue as the alternative was just so much worse. OpenGL was seen as a great API in those days, and it could be a truly great one again.
Another week or so of shaking down the code and I'm going to release. I could have done so 4 days back, and it would have fulfilled it's initial objective, but it would have gone out with subtle (and some not so subtle) bugs in. It's going to be a much better release for this extra time.
Most of the development work now consists of actually playing Quake II using this engine, making notes of what's not quite right, and then spending some time fixing it. Porting and feature wise it's pretty much complete, with perhaps the addition of a hardware VBO for certain world surfaces still to come.
Posted by mhquake at 1:22 AM
Sunday, July 12, 2009
Lost device handling is now done. This was a BITCH to code, but in the end I decided to go for the simplest (but not necessarily most elegant) solution and just restart the entire refresh. It means that there is a single code path for everything, and handles resolution changes after an Alt-Tab properly too. It also opens up the possibility of switching everything from the managed to the default pool, which should improve performance a little (although I'm going to need to be a little careful with lightmap uploads).
If you were astute enough you'll have noticed that the models in my last screenshot were rendered fullbright. This wasn't a bug in my shadows code, but was instead a bug in handling a "no hit" R_LightPoint result for the Strogg ships flying past the windows in that room. The solution I chose was to set alias model lights to 0.5 if there is no hit. I'm not sure how well that will work overall, but right now I don't see any conceivable problems with the approach. The only time it should happen is if there are no solid surfs under the model, meaning it's in empty space. Obviously as solutions go it's a bit cheesy, but it seems better than rendering the thing in full black to me.
Speaking of R_LightPoint, I'm really satisfied with how the new aproach worked out. A combination of reading the results of R_BuildLightmap and using LordHavoc's "interpolation" code has given me a really smooth and accurate end result. I've set "gl_lightmap 1" mode to also apply to alias models so that I can do some good testing on the lighting for them, and this is definitely the best I've ever seen.
Posted by mhquake at 9:25 PM
I've decided that I'm not going to force the user to double their gl_modulate for 64 bit lightmap support. This means that the additional dynamic headroom is lost, but after having done some comparisons of with and without it, it doesn't add enough to warrant making things awkward or unintuitive for the user.
I've added in LordHavocs interpolated R_LightPoint code for alias model lighting. This is an important piece of code, but it's not really "interpolation" as is understood elsewhere in the engine; instead it smooths out transitions between different parts of the lightmap. Averaging the current shadelight with a last shadelight is another option I'm considering.
Those who base their reality on SlashDot posts and old writings from 1996 had better look away now.
Here's Quake II running under Direct3D with stencil-buffer shadows and polygon offset. :)
Posted by mhquake at 2:23 PM
Saturday, July 11, 2009
Just added 64 bit lightmaps (for hardware that supports them) to the Quake II codebase. These are advantageous for two major reasons, with reason 1 being that you can configure them to use a much higher dynamic range, so that light doesn't get clamped to the max as quickly, and reason 2 being that they provide a much finer degree of quantization between individual steps of that range. This means that lighting looks smoother, and brighter spots are able to really bright.
Now the bad news - you'll have to double your value of gl_modulate in order to use them. So if you normally use gl_modulate 1, set it to 2, and if you normally use 2, set it to 4. Because gl_modulate now operates in hardware, there is no option above 4, so if you normally use 4 you will have to compensate by doubling your value of intensity. It's not quite the same end result, but close enough. If on the other hand you use intensity 4 and gl_modulate 4 all the time, you really need to get a brighter monitor!
I could remove the requirement to do this by simply changing one occurance of 127 to 255, but doing so would lose the extra dynamic range headroom. I'd rather sacrifice the ability to set gl_modulate too high than do this.
Of course, in DirectQ it's all handled using pixel shaders, so none of this is a problem.
I've decided to not get too hung up about performance with this engine. Naturally that doesn't mean that I'll intentionally write slow code for it, but it does mean that if something worthwhile impacts on performance, then the worthwhile thing is staying in. The exception of course is if a slideshow is the result, but something in the order of 60 FPS would be acceptable on the hardware I do most of my development on (it's currently getting almost double that, by the way, so there's a long way to go).
Posted by mhquake at 9:48 PM
Thursday, July 9, 2009
I've just spend the last hour writing a code formatter. Now, enough hot air and wasted energy has been expended arguing the relative merits and demerits of each style, and I'm not going to join the fray. The sole relevant point is that the style you are used to is important. It helps make code more readable, and - as it's a known fact that reading code is harder than writing it - anything that helps make code more readable is a bonus.
My own style has changed and evolved over the years, just like anyone else, and if I was willing to put in the effort I could probably adapt to the styles used in the Quake codebases in a matter of a coupla months. But until then I'm working at a lower speed, reading code that looks strange to me, and wasting time by reformatting it as I go. Even worse, the Quake codebases can be a mish-mash of different styles, depending on who was working on them at any given time.
I'm not doing this for money and it's not my day job; if it was I would put in the effort to adapt, but that's not the case.
So after hunting down various reformatters on the web I've decided to write my own. Why? Cos the others are all rubbish. The only GOOD reformatter I've ever seen is the one built into Visual C# (and presumably VB, but I've no exposure to it) 2005 and above. Precise control over all the important things. The others are too limited, and/or impose their own idiosyncracies, and sometimes even add injury to the insult by making you pay for them. This way I can target specific things that drive me batty in the Quake codebases and adjust them to specific outputs that I feel more comfortable with. What other people like or dislike is their own business.
Posted by mhquake at 10:16 PM
Wednesday, July 8, 2009
It's starting to take some good shape now and mosey towards release status. Still a while off, but it seems eminently stable with baseq2 functionality. Once again I'm using the Speed Demos Archive material for testing of functionality and stability.
Today I did a very worthwhile adjustment to how R_LightPoint works. I'd always smelt a rat about this function, suspecting that the colour it returned was not quite the same as the correct surface light colour, particularly with respect to dynamic lights. Then it occurred to me that - as I've shifted lighting to something more similar to Classic Quake and am therefore storing all lightdata in main memory - I could just read the data from the final generated lightmap texture instead of having to rebuild it from the raw lightdata. This means that I can skip the addition of dynamic lights in R_LightPoint and have a precise result. This is definitely something that's going into the main DirectQ codebase.
I've also added in stencil buffer shadows (if you have a stencil buffer available) for a nice improvement to the default shadows. The "gl_shadows" cvar can use floating point values between 0 and 1 to control the intensity of the shadows.
"scr_showface 1" will replace the "Red Cross" health icon on the sbar with the face picture for your chosen skin. I've hacked the sbar code slightly to shift the y position up by 4 units and also to add another 4 units between the stat icon and value. It might not be 100% mod compatible, but how many mods actually customize this in such a way that it would be affected?
"gl_modulate" and "intensity" are still hanging around, but the way they work has been changed. Only values of 1, 2 or 4 are allowed, and if you set them to anything else they will just pick the nearest of the legal values. They now take effect immediately, and the behaviour of gl_modulate with alias models has been changed slightly so that it's the same as the behaviour with the world (it was different in stock ID Quake II).
"gl_monolightmap" is still there but now has no effect. The whole R_BuildLightmap function has been cleaned out of the various optimized cases; what may have been required in order to get it running on a P133 in 1997 no longer applies in 2009, and cleaner, simpler code is definitely preferable in this case.
Anisotropic filtering ("gl_anisotropic_filter") has been added. The old gl_texturemode, gl_texturesolidmode and gl_texturealphamode cvars are still there but no longer functional. Texture filtering is now either anisotropic or trilinear.
Hardware gamma has been added, using the old "gamma" cvar to control it. It works the same as in DirectQ but without the independent red/green/blue sliders, as I want to keep things simpler for this one.
The correct video mode formats supported by your adapter are enumerated and made available. Integrating the refresh DLL with the main engine helped a LOT here. As a consequence of this, the first time you run the engine it may start up in a weird and unexpected mode (it will however always respect the value of vid_fullscreen). It makes an effort to adjust to a supported mode if your cvars are outside of the ranges. I might change the cvar names and supply a safe mode (something like 640 x 480 windowed) as a default, just to avoid this first-time-startup issue.
DirectInput has been added for the mouse. I know a lot of Quake players prefer to use DirectInput, so now it's in Quake II also. Use "m_directinput" to control it, default is 1. If it seems too fast or too sluggish, use "m_diboost" to apply an additional scale modifier (default 3).
I'm going to add in BMP/PNG/JPG screenshots, just like in DirectQ. These are so easy to code in Direct3D that it would seem crazy not to. It will be a string cvar called "scr_screenshotformat", with a default of "tga".
External texture support might be added. I'm in two minds about this one; it seems a nice feature to have, but is it really used often enough in Quake II engine mods? I think I'll do it anyway just for completeness sake.
I might also add in additional external crosshair support; seems easy enough in the Quake II codebase.
I'm also going to see about removing the hard-limit on the number of textures (I've already upped it to 2048, but I want it gone). Texture-wise, proper bilinear filtering (from the D3DX library) is used for resampling (if required), and 8 bit textures are uploaded direct to the hardware in 8 bit + palette format (for faster texture uploads, which roughly evens out the overhead from using compressed textures).
The other main thing is that I still haven't gone back to fixing up the "lost device" case. It's annoying enough to have to do it in the first place, so I'll save it for a time when I just feel like grinding out code rather than doing anything creative or productive.
Think that's all for now.
Posted by mhquake at 7:35 PM
Tuesday, July 7, 2009
I did some DirectQ work today; ported over the new lightmap uploading code from Quake II. It's ever so slightly slower than my previous method, which I guess is most likely API overhead rather than anything else, but I'm happier with it as the previous method had always given me a slight case of the heebeejeebies (it worked Ok, but was it the Right Thing to do?) I might shoot for a hybrid method to try get the best of both.
I've also decided that DirectQ will stay with HLSL and won't get "enhanced" particles. I'm really just too far down the HLSL route to change right now, and on the particle front I've decided that the original particles are too finely tuned for the game to modify them. Of course, that latter has been done by many others, but I like the more traditional look of DirectQ so there we go.
Both of these are of course subject to change if inspiration for either strikes (or if I get the itch - right now I just don't have it).
On the Quake II front it's gained a few bugfixes as well as anisotropic filtering. It's one thing saying that the port to Direct3D is "complete", which it is, but a whole other thing is sorting out rendering bugs and slowdowns, as well as other bugs that may have been brought in by changes to data structures and code structures. This exactly mirrors the DirectQ situation where the original port was done over a weekend (I think it took 2 days this time) but was then followed by 2 weeks of shaking down the code. Of course I know more about what I'm doing this time, so I was able to avoid some basic mistakes. The flipside of that is that I can spend more time implementing things that I just wouldn't have been able to last time.
A final interesting development is the use of DirectQ 1.0 as an experimental codebase. I ported LIT support to it earlier on, just for laughs, so it's starting to build up a more comprehensive feature set which may see the light of day sometime. Maybe that will compensate for retaining HLSL in the current DirectQ codebase. I wouldn't however hold any breath over this potential "DirectQ Zero" release.
Posted by mhquake at 1:28 AM
Monday, July 6, 2009
In order to test the old Software Quake waterwarp I fired up the original DirectQ 1.0 source code and wrote it in. Conclusion: it's ugly, and has been removed. Performance-wise however it was a nice surprise, giving me only a 1 FPS drop.
It's actually quite nice having this codebase available, as it gives me a good platform for experiments such as this without the risk of polluting any of the main codebases. I'm going to port my new lightmap updating code from Quake II over to it shortly so I can do a good (and valid) performance comparison between the two.
It's also really strange going back to it and seeing all the things that were wrong with Quake in general and GLQuake in particular - a nasty trip into the past that was just full of bad vibes.
Posted by mhquake at 8:00 PM
I've been thinking about water warps, and I think the solution is to use a method similar to software, which updates the texture every frame from a sintable lookup. Textures would update from a default 64x64 tile into a new 128x128 tile, meaning that the typical case will be no worse than a coupla lightmaps being updated per frame (better in fact as I can send 8-bit data plus palette down rather than 32 bit data, meaning a 17K upload instead of 64K), although factoring mipmapping into the equation will be an interesting thing to resolve.
Where this may get complicated and even more interesting is with high-res external textures that are not guaranteed to be powers of 2 in DirectQ, but for the Quake II engine it's at least worth exploring to see how it goes.
Posted by mhquake at 1:45 PM
I've successfully sped up cinematic handling at large resolutions by a factor of about 50. I had originally (and misguidedly, as it turned out) thought that writing the raw data directly to the backbuffer would be the optimal solution, and - while it did work OK at 800x600 - at anything higher it went to a total crawl. Using regular texturing turned out to be the Correct Thing to do, after all. I'm not certain if it was the use of a fullscreen Device or the higher resolution, in fact; but at this stage I'm not really going to bother finding out, as it now works very fine indeed.
I've optimised it beyond the GL handling in that it caches the texture and only needs to recreate it if the size changes or a new cinematic begins. This is the D3D equivalent of a single glTexImage followed by glTexSubImage until the texture params change.
Just for laughs I also changed the texture update to only happen every other frame, and the difference in animation was completely invisible, but perf was boosted even further still. I kinda suspect that default QII cinematics were playing at some crazy high frame rate.
All in all, this has been a good one to mark off the list.
Posted by mhquake at 1:26 PM
Sunday, July 5, 2009
Not as bad as the title may lead you to believe. I finished video startup (Quake II has a proper mode list now) and started into lost device handling, but very quickly ended up getting nowhere with that. It's frustrating, and I hit the same issues with DirectQ; having to ensure that you release everything properly before resetting the device, then bringing them back on. I thought I could probably just drop the device and recreate it, same as a vid_restart, but there are a few complications there too. The really annoying thing is that this is something that everybody has to write code for, and everybody ends up writing broadly the same code, so why is it not handled automatically by the runtime?
Another issue I'm hitting is that cinematics are really really slow at higher resolutions. These were handled OK by 3DFX cards at 800 x 600 10 years ago, so I just need to find a way to do them faster. I suppose most people don't even bother with cinematics, but for the sake of completeness I want to have them.
Posted by mhquake at 10:23 PM
The Quake II render is now complete. No screenshot, as the previous one was 95% there and the remaining items were stuff like sprites, gl_flashblend 1 mode, the null model and so on. Trust me, you wouldn't see any difference.
I did say nothing new was going in, but the following did make it:
- Underwater warping.
- Z-Fail Sky Box.
- scr_consize adjustable 2D render scale.
- Removal of functionality of "intensity" cvar.
- Removal of functionality of "gl_picmip" cvar.
- Compressed textures.
This poses an interesting problem for DirectQ. I've satisfied myself that not only is it possible to replicate the render using the fixed pipeline, but that speeds should be comparable. What I'm now thinking is that I'm going start transitioning DirectQ back to the fixed pipeline. The main reason for me to go HLSL in the first place was to do water warping, but if I can resolve that item then the reason is gone.
Posted by mhquake at 2:24 PM
Saturday, July 4, 2009
The render is now virtually feature-complete. I think the only remaining items are Sprites and Screenshots, and then we will have a direct port from OpenGL to Direct3D.
As mentioned before, I got rid of gl_modulate and intensity, and instead used a 4 x modulate blend to replicate these at their default settings. I'm not overly pleased with some of the stair-step artefacts this gives however, so I may well restore gl_modulate. Both cvars have been left hanging around anyway so that this engine can play nice with existing configs.
State change optimizations are in and - as predicted - we're at 100 FPS on the Intel.
Following this I'll need to work on video startup - right now it just assumes that you're running in an 800 x 600 window and doesn't support anything else, which was very handy for development but not good enough for Real World Use. Then I want to add in the "false fullbrights" and underwater warp, so that we're feature-compatible with Quake 1. It'll be missing the scrolling sky of course, but that's all vertex shader work rather than pixel shader, so I can live without it.
Posted by mhquake at 11:51 PM
This is the full Quake II Alias Model render, including shells, shadows, left-handedness and all the other stuff. It was incredibly pleasurable to write, and took about 2 hours to port from OpenGL.
Just a quick word about the porting process. I'm pretty much replicating the way I did it with DirectQ, as in I've commented out all the GL calls and then have been gradually replacing them with the D3D equivalents. No optimization is being done at this stage, so there are redundant state and texture changes everywhere. That's going to be left till the very end, and will have a similar state change filtering system to DirectQ.
I'm going to do particles next, then probably launch into finishing off the world (i.e. lightmaps). I'm not really looking forward to the latter owing to the sheer volume of cruddy code that needs to be cleaned up - before I even begin! I might yet chicken out of it and do some other things before it.
There is one change to the render I do want to make, which is that I want to add in some "false fullbrights". Remember that a major objective of this port is to get a fixed path render that will be usable in DirectQ. In order to achieve that I need fullbrights. However, I'll probably make these optional, as I'm feeling that I am going to release this.
Posted by mhquake at 1:45 PM
Thursday, July 2, 2009
So far so good, we have the 2D stuff completed but in need of tweaking in places, and just starting on the world right now. Getting this far is a MAJOR landmark position, as it means that we now have meaningful stuff on-screen to work with.
God, the code a mess. It's been a long long time since I'd hacked on the QII codebase, and I'm starting to get unpleasant memories. The surface refresh in particular is fairly badly in need of a major gutting, reorganisation and simplification. I suppose a certain part of that is down to the nasty array of 3D cards which were available at the time, and which QII had to run on. I had originally intended to port over everything, even the gl_monolightmap stuff, but somehow I doubt I'll be doing that. I can also see the "attempting to be all things to all people all of the time" cvars like intensity and gl_modulate falling by the wayside.
I haven't even bothered optimising for now, I'm just shoving bits of code together and getting a result. I think I can get 60 FPS with an unoptimised codebase, then tune it for close to 100. Will be interesting to see how well the completed one does run.
Posted by mhquake at 9:59 PM
Nothing dramatic, and virtually impossible to reproduce (so no major cause for panic); this happens very rarely while navigating the menus after a game change, and is triggered by a Draw_CachePic call loading a new pic for a menu banner. I've seen it maybe three times over the past few months, but have never been able to determine the cause.
I think I've got it licked now; it was just a matter of clearing down any open file handles in sys_win (which would have remained persistent after a game change).
It's not enough to prompt a 1.6.4 release, and I just found it while playing rather than actively working on the code, so current plans remain on track.
Regarding those: I have a QII codebase converted to C++ and with the ref_gl integrated into the main engine from a while back, and have so far been pulling out the OpenGL code and putting in basic Direct3D support. Right now it just creates a Device and shoves some of the 2D menu and HUD pics onscreen, the most simple basic requirement to confirm that things are working.
Posted by mhquake at 5:26 PM
As is customary by now, I'm giving advance notice of my intentions regarding the next release shortly after the previous. :)
I've pretty much decided that it's definitely going to be 1.7; 1.6.3 certainly had enough going on behind the scenes to justify a full version number bump, but I want to have something more substantial than behind the scenes changes for it.
Of course, with a new full version number we get new features, and I'm going to follow the model of using rapid incremental releases to fine tune them. As to what those new features will be: once again I've left particles outstanding, and I see that a sample CSQC implementation on top of a vanilla Quake codebase is now available. No promises though, I'm definitely not going to bring CSQC into my main codebase until I'm satisfied that it's going to integrate cleanly, and these days I just seem to be rubbish at particles (4 aborted attempts in DirectQ speak for themselves).
One thing that's almost certainly on is a fine tuning of the standard QC interpreter. I've already done some work on it in previous releases, but it's time to really get stuck in.
Otherwise, I think that in terms of the renderer, the engine is "done". I still want to add in a fixed function path, but the HLSL path is about as feature-complete as I want to make it.
I've toyed with the idea of going back to my Release 1.0 code for the fixed path, but some issues with it occurred to me. The 1.0 code worked fine, and gave me lots of pleasure in writing it, but looking back with the benefit of what I know now, there are a lot of things I would have done different (and better).
I was then going to port ProQuake (which I thought would be a nice present for Baker) and use the end result as a base, as it's already set up for a D3D renderer, but doing so would involve some major gutting of it's D3D code as it's structured very differently to what I would use. The temptation to bring some of that structure across to the port would be too much. I wanted a clean, fresh, but broadly compatible codebase to work from.
Where this is all leading is that I'm going to port Quake II to D3D. The render is almost identical to classic Quake (although with a fair bit of "second system effect" in), and it'll be a valuable exercise in providing a good codebase to bring forward to the DirectQ fixed path. Who knows, the end result might even be releasable in it's own right. ;)
It will be a classic pure port, just the baseline rendering stuff, and nothing fancy in effects, no changes elsewhere in the engine. One time only, one version only. No Vertex or Index Buffers, no HLSL, no particle effects, nothing beyond what Quake II already does. I did the original Quake 1 port in a single weekend, but I don't expect to do the same here. About a month seems a reasonable upper limit.
This also gives me a time window should any last-minute unexpected bugfixes be required in DirectQ 1.6.3; not really starting on 1.7 until after this port is done will mean that any such fixes can be brought directly into the unchanged 1.6.3 codebase.
Posted by mhquake at 12:46 AM
Wednesday, July 1, 2009
Just released 1.6.3; if you don't see it in the menu to the right just yet, it's because you've visited here in the time interval between when I posted this update and when I added the link. Be patient!
The big news for this one is that I've switched the "Development Status" to "Stable". It's gotten some very extensive testing courtesy of the demos available at http://speeddemosarchive.com/quake/ - in particular the "100% Secrets 100% Kills all of ID1 on Easy" demo set. There may still be some esoteric mods that crash the engine in unlikely or contrived circumstances, or where they do something completely unexpected and in an unintended manner, but the same applies to any Quake engine out there (although the circumstances may differ).
I won't post the full feature list here as most of it was given a coupla days back. Three other things that you should be aware of are:
- Release 1.6.3 is about 20% faster than the previous release. Some of it came from fine-tuning parts of the code, but most of it is from using compressed textures. I really like the extra degree of finer control D3D gives you and this is a good example. OpenGL just has a hint mechanism which implementations are free to honour or not as they see fit, whereas with D3D you can specify exactly what's done and how it's done. It does mean more code and more checking, but you get the precise result you want at the end of it all. It just feels closer to the metal.
- Release 1.6.3 has an integrated QC debugger! Yes! It's fairly simple in that it just logs QC related info (function calls, commands and cvars, crashes, etc) to either console or file (or both), but it should be useful for anyone who wants to use it (or port it to their own engines).
- The importance of a good test suite has really come home to me here. The SDA demos helped me catch a bug that I would likely never have been able to reproduce otherwise. I'll be using these more in future.
Posted by mhquake at 8:59 PM