03-31-09 - Spring Cleaning and Mucus

Living up here reconnects me with the idea of "Spring Cleaning". Living in the warm south it never made any sense - why pick Spring to do a big cleaning once a year? Just do little cleaning all the time. (actually I'd like to have a schedule of one major cleaning item to do each month, like Jan - launder drapes, Feb - remove grates and dust inside air vents, Mar - move appliances and clean behind them, etc).

Anyway, up here it's cold enough that you close everything up for the winter, and I guess in the really cold places that's even more true. Heck in the old days when Spring Cleaning became a custom, they were still putting boards over windows and bringing livestock inside for the winter to stay warm. You would close up for the winter and swaddle in blankets and live in the stale musty air and dead skin and dust.

Here it's getting stale and dusty inside. I prefer to live with my windows open all the time. Sadly I haven't been able to do that for quite a while, since here it's cold and wet, and even in San Francisco if I opened my windows a pound of filth would come in the window every day. Anyway, I'm looking forward to when this cold and gray and wet finally ends so I can open up all the windows and shake everything out.

As I've gotten older something has changed in my nose geometry. As a child I always had a leaky nose, causing me to frequently wipe on my sleeves and t-shirt when I couldn't find a kleenex fast enough. Finally I gave in and started carrying a handkercheif like my mom (which I always found disgusting). Gradually something in the cartilage has changed, and now the snot drips down my throat instead of out the front (maybe this is just because it's constantly plugged with boogers). Anyway, it now makes me spit up loogies instead of dripping snot, which is also gross, but is better I guess because I can at least control it a bit and decide when to let it out.

Anyway, while my constant spitting of loogies is gross, and I sympathize with spitters, there are some spitting habits that people have which I find completely repellant and mind boggling :

1. The constant hocking / sniffling. This guy has got a mucus problem but for some reason is not going ahead and expelling the offending matter. He just keeps hocking or snorting over and over without ever spitting. This is sometimes done by people who were told by their mother not to spit, so they're trying to have good manners - well bud, hocking over and over is much worse than just doing it once and getting the spit out.

2. The gratuitously thorough hock. This guy reaches deep and grunts and rumbles to stir up mucus from deep inside then does a massive hock and spits out a liter of saliva. If you really needed to hock that hard to get the stuff up then you really didn't need to spit that bad in the first place. It's like they do it as a show intentionally to hock as loud as possible, as if it's manly or tough or something. This is usually done by working class males; it's sometimes used as punctuation in conversation.

3. The dripper. This guy hocks and then doesn't spit a tight wad with force the way you're supposed to, he just sort of leans over and opens his mouth and lets it drip out. Most of it comes out quickly, but there's usually a long trail of sticky slime that hangs off his mouth for many seconds, and he just sits there and lets it slow stretch and finally detach. Come on man, first of all spit it harder so that doesn't happen, but if you get a hanging trail, use your hand to detach it or something. This is often done by smokers or tobacco chewers.

03-31-09 - GDC Middleware

There's a game engine for every day of the week now. Personally I don't think big game engines is the way to go. It just puts you as a developer too much at the mercy of the engine developer, and if it has major flaws that you don't like you're screwed. I'd much rather license lots of little pieces I can tack together as I see fit.

SpeedTree's tools get better and better but their runtime is still not awesome. That seems to be a pretty common thread. It's kind of weird because for me the tool is the hard part that I never want to do. Hell I could write the runtime for *all* these middlewares if they would write the tools and the do the sales and support. Some other kind of similar ones :

Fork Particle Middleware has a pretty nice tool. It's funny I was talking to Sean at the show about what other middlewares RAD might do someday and I mentioned particles. We figured you'd want to have a nice tool for artists where they could compose arbitrary particle systems, then the ideal thing would be to output a sort of "particle HLSL" for each system, then you could compile that to CPU code or GPU code or SPU code. The idea there is that all the toggles hierarchy and such that the artists set up, you don't have in the code. That is, you want to avoid winding up with code that's like :

if ( m_particleSystemDef->m_doSpin )
{
    SpinParticle();
}
... more ifs for each possible effect ...

instead you just compile down optimized code for each system type. You also would want to automatically provide particle LOD and system scalability so it can target different machine capabilities.

Anyway, I doubt it's a good RAD product because it's too tool and artist heavy, it's not a big enough piece of pure technology. The Fork demos look pretty super-duper asstastic, like shockingly bad, but I don't know if that's just because they have bad artists setting it up or because their tech actually sucks. I do think there's a lot of value in actually making an impressive good looking demo - it proves that it's at least possible to do so with your system.

Allegorithmic Substance was at the Intel booth showing their procedural texture middleware. This is another one that I had the idea of doing some day maybe, so I wanted to see their action. Procedural texturing is obviously compelling in the future era of 8+ CPU cores and SVT infinite textures and so on. And you clearly need a nice artist-driven tool to define them.

Allegorithmic has a really super nice polished tool for writing shaders. I have no idea how fast their runtime is; I tried to ask some questions about how they run the shaders and the guy demoing didn't seem to be an actual programmer who knew WTF was going on. Again just like particles you would want to be running code generation so that you don't have like 100 per-texel branches for all the options. In fact, this is really just like rasterization in a software rasterizer like Pixo. The procedural texture code is just like a pixel shader and you want to output some kind of HLSL and have it compiled to a CPU/GPU/SPU shader.

I also don't think their way of defining shaders is right. You want something that is intuitive to artists and easy for them to tweak visually. You don't want to have to hire someone who is a procedural texture specialist. The Allegorithm shaders are just like Renderman shaders (or as Sean pointed out - like the stuff that demo coders do for 64k demos). You get a bunch of functions that you can compose and chain together and you tweak them to make it look like something. For example you can take Perlin Noise and apply curves and powers to it and threshold it and all that kind of stuff. Or you take a "mother wavelet" shape kernel and tile it, randomly rotate and offset it, etc.

That stuff is powerful and all, but it's just not intuitive and hard to tweak and not good for artists. If I was doing procedural texturing it would be example-driven with little bits of functional stuff. Some stuff you could do is all the shape-by-example synthesis stuff. You can do "detail textures" the correct way by using different tiling textures at different frequencies for the different wavelet levels of the output. You can use multiple tiling source textures and randomly compose them using Perlin Noise or an artist supplied blend texture. You can do things like the old "splatting" technique from Surreal with blend-alpha channels in tiling textures. You could even do things like Penrose Tiles where you have the artists paint the tiles and then you can create infinite non-repeating tilings. Or you can use sample textures to create tiles automatically like the Hoppe lapped texturing stuff. etc.

It just seems like the Allegorithmic tool is aimed at an audience that doesn't exist. It's too technical for real game artists to use. But if you're going to have a technical artist / programmer write the procedural textures, they would rather just have a text HLSL type language like Renderman. And having text shaders like that is much better because it's easier to share them on the web. In fact, the only way a tool like this could ever really take off is if a community develops and they post shaders on the web and share them, because writing all the shaders yourself is too much work. (in raytracing there are tons of prewritten shaders that are free or sold in commercial procedural texture kits).

Actually what you want from the tool is just a general attribute plug editor. You want to write text shaders like Renderman but have them take various parameters as scalars, colors, or images. Then you want to expose those to the artists with nice GUI tools and show them how they affect the shader with realtime preview. Something like :

input parameter brick_color : type color;
input parameter mortar_color : type color;

color diffuse_shader(vec2 uv, vec3 worldpos)
{
    ... do shader maths using uv, brick_color, mortar_color ...
}

In fact having a generic parameter editor set is a handy thing that all studios should have and everyone reinvents, but again it's too small of a piece to sell.

03-31-09 - Taxes

I did my taxes this year with TaxACT . Mainly because I put it off too long to get a decent accountant, but also because they were so simple this year it seemed worth it to just do myself.

I liked TaxACT, I recommend it. I used TurboTax years ago and it was a bloody nightmare. By the time I was half way through I wished I had just used pen and paper. TurboTax was a weird annoying interface that didn't match the real form exactly and I couldn't figure out how its questions matched up to the data I had to enter.

TaxACT starts out with the same kind of annoying retarded interface where it just asks you questions; it tries to be like a "Wizard" for tax info, like "what kind of taxes do you want to do today?". Fortunately you can just close that down and go directly to the forms. It has all the forms with number entry boxes, and it does all the bits of copying numbers from one box to another and adding and subtracting and whatnot.

Sometimes when I do taxes by hand I feel like the IRS has created one of those math puzzles where you do a bunch of steps and always come up with the same answer, like :

enter your pre-tax W2 income in box 41
take the last digit and subtract it from nine and enter that in box 42
divide box 32 by box 33 and put the remainder in box 43
evaluate the gamma function of box 44
take the last three digits of box 44 and add them together, enter in box 45
copy box 41 to box 46
that's your income

anyway, TaxACT does all that bit for you.

Some caveats about it :

1. It claims to be free but it's not. I didn't need the premium version ($10) but if your taxes are complex at all (eg. itemized deductions) then you do. I did have to pay for direct deposit ($8) (it says free efile but doesn't mention you have to pay if you want the DD) and the California add-on ($14). Still cheaper than TurboTax.

2. Every time you try to enter a derived value it says "this is a computed value, do you wish to do the worksheet" , say yes and go do the worksheet. If you don't, it messes up its spreadsheet computation tables.

3. It can be a little hard to go "back" to forms you've previously done and want to edit. The best way I found is just to browse the index of all forms, and select them manually. Another trick is if you want to go to a worksheet that filled in a certain derived quantity, you can just try to edit the result and it will pop up the dialog mentioned in #2 and then you can say "take me to the worksheet".

Anyway, still not a huge advantage over just using pen and paper, and it does have the disadvantage that you have to copy in all the data off your 1099's and W2's and whatnot manually if you want to efile.

03-31-09 - Recent TV Reviews

I just started watching "Life on Mars" (BBC) with Alissa. It's decent so far. It is rather ham-handed with the whole is-it-real existential crisis. It also beats you over the head with the setting over and over just like the horrible "Mad Men" ; like oh look they're smoking how quaint, oh look there's an 8-track in the car, how cute. It would be so much better if they just told the story straightforwardly and the setting was just *there* without all the over-emphasis. Life on Mars wins over Mad Men just because it actually has stories - the Not sure if recommend yet. I think I would rather just see a good cop show rather than this self-conscious highly affected play on a cop show.

(BTW of course if you have to option between an American remake and the BBC original - always watch the original. The American formula is to basically steal BBC shows, put beautiful retarded actors in them that don't match the character, rip out everything that's real or honest or edgey or controversial at all, make them just like every other bland TV show, use tacky overly slick fancy production and lighting and makeup and everything).

I've been watching a bit of "Shameless". It's basically a Northern England soap opera, and I feel a bit dirty watching it, but it's very well made. It's got a great pace and rhythm to it and the characters are very lively. Recommended for losers like me who wish they were watching soaps but would never admit it.

Watched all of the first season of "That Mitchell and Webb Look". I must say watching it all in one go was not friendly to it. The running gags are very repetetive. The first episode I really liked and it just went downhill. Still a few bits really made me lose it. I love the realtor bit, and the natives in the gardening supply was laugh-out-loud gold.

I watched an episode and a half of "15 Storeys High" then gave up on it. I just don't get it. Is it supposed to be a comedy? I don't think I've seen a funny bit yet. It's just sort of weird and grim. I guess there are some bits that are obviously supposed to be funny, like when the guy doing the relaxation tapes gets mad and yells, but that was just so obvious and set up and not at all in context that it completely misses. Do not recommend.

Recent Movies (just the good ones) :

"Man on Wire" was beautiful, inspiring. I've always wanted to live more like that - living every day artistically, beautifully, doing something crazy and pointless and dramatic with a band of friends that are united by the sheer drive to be doing *something* this these sad boring lives of ours.

"Milk" was quite good. A bit heavy handed & cheezy at times (the flash back at the very end to the foreshadowing at the beginning was just inexcusable), but overall very moving, well acted, fun. It made me really miss San Francisco (we saw it before GDC). All the crazy fun people, the liberal politics, all the protests and parties in the streets.

03-31-09 - Recent Food Reviews

I've become a big fan of braising greens as I get older. The Trader Joe's braising green mix is reasonably priced and convenient, and unlike most bag salad products, it's not all rotten because braising greens hold up well. At Restaurant Zoe we had some collard greens that were almost delicious but in fact disgusting. They were smothered in so much butter, and it they were near room temperature so the butter was semi-solidified, so that it was almost like a quiche with just butter instead of milk and egg. If you're going to add bacon to your braising greens, don't be tempted to do it first. Yes, cooking things together is nice, but it's better to cook the bacon last and just sprinkle it on at plating so it stays crispy.

The best thing I ate in San Francisco during GDC was the bone marrow at Two. Bone marrow is a lot like butter-collards, it can be disgusting if it's too plain, because it is basically pure fat, and it does have a slightly weird vitamin-like bone taste. It's been trendy to cut it with a bit of persillade or gremolata or something like that, but at Two they serve it in a kind of reduced-broth onion gravy, which was perfect. It enhances the meatiness and tames the vomit factor, and provides some extra nice tasty bread dip.

(the Raviolo at Two was also excellent - a soft cooked egg yolk in a pasta wrapper with brown butter sauce, very simple but delicious. It would've been better with some crunchy pancetta bits sprinkled on top. That also continues a trend I've seen a lot recently of really interesting and well made delicious appetizers, leading up to rather boring and lazily made main courses.)

03-30-09 - Price Comparison Ruins Quality

Online price comparison is very harmful.

Meh I'm bored of this rant but you can fill in the blanks cuz it's obvious. Here are some key words for you :

Expedia - airlines competing only on price, Nextag, eHealthInsurance / Progressive / Geico - insurance companies competing only on price.

Obviously quality goes to shit.

03-28-09 - GDC

... was really rough. I didn't get to any talks so I don't have anything good to tell you. I got to walk around the floor a bit, but didn't see anything interesting really. There is a ton of middleware out there - a procedural texture product (Allegorithm or something), FMOD, SpeedTree, a particle system middleware - and all of them have really really polished tools, super nice UIs. There are also just a ton of game engines now, some of them have very flashy demos.

I miss San Francisco. Even just flying in, I looked out the window and could make out the weird shape of Point Reyes and thought of all the great biking out there, the fun people who go swim Bass Lake, the sun shine, the aromatic smell of all the sage and bay laurels. The city has such a happy vibe to me, so many different types of people all jammed together and getting along, hobbos, hippies and hipsters. People biking around, sitting in the park, everyone seems to be energetic and full of life.

It was interesting seeing the break down and set up. It's amazing how fast they do it. Literally the second the 3:00 closing bell is chimed on the last day, the union labor guys start pouring in and picking up carpet. The RAD Veterans immediately started tearing apart the booth and boxing it up so it would be ready and they could get out of there. In a few hours the whole place was torn down.

Also, a lot of people have mentioned "OnLive" in their GDC blogs. It's 100% retarded and I don't want to hear about it any more. I would bet any amount of money against them ever making any money.

... and I got the GDC Disease from shaking hands with weirdos from around the World. As soon as I got home I was hit with a brutal cold, headaches and mucus.

03-28-09 - Old Man Comfort

When I was young I was really nervous around people or in new situations; I wanted to show that I knew how to handle myself, I never wanted to make a mistake, I wanted to prove that I deserved to be with the A-list. It made me a total loser, really self conscious, condescending, stiff, and just awkward and antsy. I was also a huge a flaker and would frequently bail on social situations or just go very briefly.

I was always jealous of the older men who just seemed comfortable in any situation. They were usually somewhat wealthy, but not rich, relaxed, maybe a bit fat, graying, balding, but just confident and at ease. While I'm still certainly more like the youthful me, I'm starting to get a bit of the Old Man Comfort.

When I was a kid I thought that I would become comfortable over time once I learned how to act in all those situations, how to interact with the bell hops, how to act when you order wine, etc. etc. Now I know that actually learning how to act has nothing to do with it. It's more just about not caring. As you get older you care less and less what people think and you know it's just not a big deal if you fuck up. For me there have been two big factors :

1. I've given up on the hope of the Ivory Tower. I used to imagine that there were amazing people somewhere, smart, beautiful, fun, having great conversation, great parties, starting businesses together, making art, cooking ,traveling. I wanted to be one of them, and I wanted to prove that I deserved it. I always thought most people were shit, but I idolized certain heroes in my fields that I thought were just total rock stars and I wanted to act cool around them and get their approval.

Now I no longer believe in that. Yes, there absolutely are different classes of people, and yes there is a AAA group that is way better than the rest in most ways. But they're shit too in their own way and not worth impressing. Basically everyone is shit and there's no magic circle you can get admittance to so you don't need to try.

2. I'm no longer as interesting to myself alone as I used to be. When I was young, hanging out with other people was borderline excruciating because you'd just be having some horrible boring conversation and never getting into the real issues; even if you did find a smart person to talk to, you'd just spend the whole time in misunderstandings and talking around the same point. On the other hand, if I bailed out on that social situation, I could wander the streets and look at the beautiful plants and compose poems in my head while isolating various of my senses one by one and focusing on them to turn up their intensity. I might go home and work on a programming project for a while, or crack a notebook and work on some physics theorems, or read some textbooks by brilliant people. Every day I spent alone was full of activity and thought and projects and learning and excitement. Pretty much any minute of that spent with someone else was a loss, and it made me feel really antsy and impatient with people and want to rush through conversations and cut them off as quickly as possible.

While that's all still true to some extent, the big thing that's changed is not that I've discovered the value of other human's conversations (no, they're still as useless as ever). The difference is the value of my own alone time has plummetted, bringing them closer to parity. I'm now lazy and tired and bored of myself, I have lots of project ideas but I no longer have the enthusiasm to do them justice.

03-24-09 - Cute but retarded

There's an article in last Sunday's New York Times called "Try, Try Again, Or Maybe Not" that I think is a great example. It's sort of cute, it comes across as smug, it interviews experts. And it's just completely wrong.

The core point goes like this : 22% of venture capitalists succeed. Those who have previously succeeded have a 34% success rate. Those who have previous failed have a 23% success rate. The conclusion is "you don't learn anything from failure, the success rate is basically the same".

Totally retarded. They completely miss the basic point that there are varying populations and the previous outcomes weight for the different populations. Furthermore the success rate of the group with experience is (0.22*34 + 0.78*23) which is clearly better.

Let's run some example numbers. Let's say the population is made up of good entrepreneurs and bad ones.

Initial population :

X good
(1-X) bad

success if good = P(G)
success if bad = P(B)

Initial success = X * P(G) + (1-X) * P(B) = 0.22

X * G + (1-X) * B = 0.22

In the successful group,

fraction successful Y = ( X * P(G) ) / 0.22

success rate = E + Y * P(G) + (1-Y) * P(B) = 0.34

E = additive benefit from experience

0.22 * E + ( X * P(G) ) * P(G) + (0.22 - ( X * P(G) )) * P(B) = 0.34 * 0.22

In the unsuccessful group :

fraction successful Z = ( X * (1 - P(G)) ) / 0.78

success rate = E + Z * P(G) + (1-Z) * P(B) = 0.23

0.78 * E + ( X * (1 - (G)) ) * (G) + (0.78 - ( X * (1 - (G)) )) * (B) = 0.23 * 0.78


This is three equations in 4 unknowns, so there is a variety of solutions.
X can be in [0,1]
For each X there is only one valid {G, B, E}

... meh I started to solve this but I got bored. The solution is something like G = 0.4, B = 0.1, E = 0.05.

Anyway it's absolutely clear that E is significantly greater than zero, and in fact the whole thesis of the article is wrong. Actually something the numbers are telling me that I think is perhaps more interesting is that the "Bad" population is *very* bad. The reason why the group that failed once is not too bad the second time is because there are still quite a few "Good" people in it that just got unlucky on their first roll, and everyone learned a bit from experience.

03-23-09 - A Better Log Window

I just put a LogWindow in cblib and I thought I'd make some notes about it. The reason it exists is that stdio on Windows is so ungodly slow. The console is actually a seperate process, and when you printf to stdout, it creates an interprocess communication packet and fires it over. I'm sure every programmer knows that doing lots of stdio printing can be the bottleneck in your app if you're not careful. We all take this for granted and take a lot of steps to avoid printing too often. For example my file copy percent display routines look like :

    int lastPercentShowed = -1;

    .. copy N more bytes ..
    int percentDone = (100 * bytesDone + (bytesTotal/2) ) / bytesTotal;
    if ( percentDone != lastPercentShowed )
    {
        fprintf(stderr,"%d%%\r",percentDone);
        fflush(stderr);
        lastPercentShowed = percentDone;
    }

where the whole purpose of lastPercentShowed is to avoid doing too many prints. (you could also use timers and check enough time has passed, etc).

But sometimes it's a pain to limit this kind of stuff at the high level. For example Oodle logs all the files it opens. Normally that's fine because you don't open files too often. But sometimes you do something that opens a ton of tiny files, and suddenly the printf to stdio becomes the bottleneck. Well you know what, that's retarded, it's just putting text on the screen, it shouldn't be such a huge perf hit.

So I just did my own Log Window.

It's just a simple Window on a thread. It runs its message pump for that Window on that thread (BTW to do this you must also Create the window on that thread - Windows automatically associate themselves with the thread they are made on, unless you do AttachThreadInput which you don't want to do).

The LogWindow::Puts from the main thread just uses my lock-free SPSC FIFO to send the string as a packet to the LogWindow thread. That makes it nice and nonblocking and fast for the main thread. (I just duplicate the string to make it safe to send to the thread).

So that's all nice and simple and fast and the main thread can just fire tons of logs and not worry about it. The other big speed improvement is in the LogWindow update.

The LogWindow thread sleeps when it has no windows messages. It needs to wake up occasionally if it got a mouse click or a paint that it needs to respond to. It also needs to pull new strings off the FIFO and add them to its list. When it does that, it can just keep pulling off the FIFO while it's not empty - it doesn't cause a redraw for every new string it gets, it only redraws once for a whole bunch of strings. This is much neater, if the main thread is firing tons of strings at it really quick, it will wind up just grabbing them all and doing one big step instead of tons of little ones.

The LogWindow thread also needs to wake up when there are new messages in the FIFO even if it hasn't gotten a paint message. I guess I should do that with an Event signal, but right now I'm just using a timer. The timer just wakes it up and makes it pull the fifo. Yeah in fact I'll switch that to an Event right now and use "MsgWaitForMultipleObjectsEx" that lets you wait on either an Event or a Windows HWND Message.

Though actually they optimize slightly different cases. The Event is better if you log very rarely or with big gaps, because it lets the LogWindow completely sleep when it's not being updated (with the Timer it still wakes up on the timer interval all the time and costs you some thread switch penalty even though it does nothing). The Timer is better if you are logging very heavily, because just setting the Event for each printf is not at all free. Maybe there's a way to get the best of both worlds, but really either one is miles ahead of using a standard console.

03-23-09 - Tech Image Series 5

This time we see some error delta images. These are created by taking an image, compressing, decompressing, then subtracting from the original, and finally multiplying up by some constant so that you can actually see the errors (normally there are values like 0,1,2,-1,-2, so they would just appear black and not visible to the human eye).

Wavelet :

Lapped DCT :

03-23-09 - Oodle GDI and Strings

Some random Oodle thoughts/questions :

My ThreadViewer is currently using GDI. It's pretty good, I like it. I did originally because I consider it a favor to the user. There are some nice things about using GDI instead of a 3D API - it plays nicer like a normal windows app, you know it gets Paint messages with rects and only repaints what it needs to, it can be dragged around and respects the user's paint while dragging or not request, and of course it doesn't gobble CPU when it's just sitting there. And of course it doesn't interfere with other 3D apps using the same device driver and memory and whatnot.

There are some disadvantages though. GDI is butt slow. If I draw much at all it really dogs. It's possible I could speed that up. For example I'm drawing individual lines and it might be faster to draw line-lists. I also wonder if it would be a win to batch up states. Right now I do SetPen/SetBrush and then draw, over and over, I could batch up draws with the same settings to avoid state changes. Another disadvantage is fancier blend modes and such are tricky or slow with GDI.

One option is to do my own 2d drawing by locking a DIB and using my own rasterizer. Probably crazy talk. The other option is just to be rude and use 3d.

---

The issue of strings in Oodle is quite interesting. The way you refer to resources is by name (string). Obviously the file names that I load have to be strings because that's what the OS uses. The main data structures that map resources to files are thus big string->string maps. (at some point I'm going to rewrite this to make it a bidirectional map, a database that can go resource->file or file->resource, which is just string<->string with hashes both ways).

Currently what I have is a ref-counted string that can optionally also just be a plain C-string. It's a 4-byte pointer and I stuff a flag in the bottom bit to indicate if it's got a ref-count or not. If not, it's assumed to point at const memory. If it does have a ref count, the count is right before the string data. Thus getting to the characters is always fast. I also almost always pass around a 4-byte hash with the string, which lets me avoid doing string compares in almost all cases. I'm usually passing around 8-byte (64 bit) "HashedString" objects. I only actually do a string compare if the hashes match but the pointers don't, which in my world is basically never. This is very good for the cache because it means you never actually follow the pointer to the string data.

One problem with this is that the strings can still wind up taking a lot of memory. The most obvious solution to that we've discussed before here. It's to explicitly use {Path/Name} instead of {Full Path}. That is, break off the directory from the file name and share the directory string for all files in the same dir. That would definitely be a big win and is one option.

Another option is to have a "Final mode" where the strings go away completely. The 4-byte hash would be kept, and instead of a 4-byte pointer to string data there would be a uniquifier to break hash ties. In order to make the right uniquifier you would have to have a full list of all the strings used in the game, which you of course do have with Oodle if your game is done. Then the uniquifier has to get baked into all the files, so there would have to be a "destring" pass over all the data that may or may not be reversible. This could be made relatively transparent to the client. One annoyance we had with this kind of thing at Oddworld is that once you bake out the strings, if you have any errors you get messages like "Resource 0x57EA10BF failed to load!". Still this would make the super-low-memory console wonks happy.

Another option is to combine the strings very aggressively. I already am running all the strings through a single global string table (the Game Dictionary that has the resource<->file map also makes all strings unique - this is a big win with the refcounted string - whenever you read a string from a file, rather than allocating a new buffer or that string you see if you have it already and just use the one you already have and bump the refcount). I'm not actually enforcing that to be required, but a lot of possibilities open up if you do require that.

For one thing, a "string" can just become an index or handle to the global string table. I almost never care about the contents of the string, I just use them as a key to find things, so if the char data is converted to a unique index, I can just compare index equality. Then, that index in the string table need not point at plain old char data. For example it could be an index to a leaf of a Trie. Each string gets added to the Trie and only unshared parts of the string make new chars. Each unique string is a leaf, those get labels and that's the handle that is given back to index the global string table.

Another option would be just some sort of LZ type compression. Some strings get full data, but other strings are just prefix matches, like "use the first N bytes from string M and then add these chars".

I think in the end I'll probably have to do the zero-strings option to please the masses. It also has the advantage of making your memory use more controlled and calculable. Like, you use 32 bytes per resource, not 32 bytes + enough to hold the strings whatever that is.

03-23-09 - Consumer Reports

I broke down and bought a Consumer Reports membership to check up on cars. It's reasonably cheap, you can get a $5.95 for one month, but of course you have to remember to cancel and there are many reports of that being a pain and then contuining to charge you.

More importantly it's just crap. It's worthless. There's not really much on there that you can't find for free around the web. The only thing of value at all is the reliability survey, the actual articles are pure piss. I was actually shocked at the lack of information, I kept clicking around the web site trying to find the information that I assumed I was somehow not seeing. And with the growth of free sites like TrueDelta and even just the forums of places like Edmunds, you can pretty much get the reliability information elsewhere.

I mentioned to Ryan this concept : I think you can trust car owners complaining about their cars. A lot of stuff you see on web forums is just whiners and cranks, but people are so biased to defend their purchase, after spending $20k or more on a car they really don't want to admit that they fucked up, so when you see someone say that they hate their car and regret buying it, there must be real problems.

03-23-09 - Neighborhood Spring

The lovely Big Picture series about spring reminded me that I've really enjoyed all the daffodils and crocuses that have been blooming here of late. It certainly doesn't feel like spring here - it's still cold and gray - but the plants seem to think it is. I guess daphodils and crocus are classic northern European flowers, and Seattle is a lot like northern Europe; it's different than the desert wild flowers of spring I'm used to in Texas and Southern California, like the poppies, bluebonnets, indian paint, all that kind of stuff.

Anyway, I took some pictures walking around the neighborhood the other day. I think there's a very modest beauty to the little patches of flowers.

Also, this is right near us and always makes me laugh :

03-22-09 - Intelligent Hedonism

Often in life you find that really stupid people actually basically do the right thing, then semi-intelligent people reason about things but get it all wrong and wind up doing totally the wrong thing, and then the truly intelligent do the same thing as the dummies. You see this in poker of course a ton, people who just "bet when they have a hand" are actually sort of doing the right thing even though they have no concept of what's going on. Then semi-intelligent people start thinking all these cock-eyed things like "wait that makes no sense, the goal is to maximize EV weighted by the probability of each outcome, so let me estimate and do some mental math..." and they totally cock it all up. Another example I believe is the pursuit of base physical pleasures.

The most animal of pleasures are the greatest. Animal physical and mental pleasures are our reward for getting through this life of ours. The physical pleasures are the taste of food, the glow of exercise, the tingle of drugs, the haze of booze. The base mental pleasures are things like ego boosts, the feeling of acceptance by friends or peers, the feeling of being loved or wanted.

Now, the dumb obviously have no problem accepting the base pleasures. But they get it all wrong. They do it vulgarly, without discretion or sense. For example the dumb boys who chase every bit of tail in the bars. Yes, I applaud them for seeking out the wonderful physical pleasure of sex, but they make it vile, they make it unpleasant for the good girls to participate, and they cheapen the whole experience. It should be subtle, full of anticipation and hints. Similarly the dumb fatties all over the mid west. Yes, I agree, food is delicious, but they do it all wrong, just stuffing their gobs full of cheap filth all the time. Better to eat less and seek out new exciting pleasures all the time instead of just trying to get more and more of the same thing.

Intelligent Hedonism is about pursuing the physical pleasures in a smart way. Not over-indulging, not over-using one particular pleasure so that you tire of it, not pursuing one when it's not wanted, and not just doing it for a fix. The Intelligent Hedonist is always striving to find new pleasures, and to enhance an experience. You might not get it very often, but when you do it's really good.

It can be simple things - like the smell of the air after a rain, or seeing a sunrise from the top of a hill - those are the moments you stop and savor, and you might spend days or weeks working to set them up.

The Intelligent Hedonist doesn't just throw back wine to get drunk. They enjoy every moment. First the joy of the hunt - reading, searching, shopping, learning. Then just touching the bottle and seeing the label - the moment of anticipation, wondering what it will be like. The ritual of the opening and pouring, the sound of the pop as the cork comes out and the glug glug of the first pour (at a restaurant the whole ritual of the taste test). Then the nose the taste the feeling. But not just wine - obviously a lot of people pretend to do this with wine now because TV and Movies have told them to. You do this with everything because you love it. With a chocolate bar, you admire the wrapper, the sound of it tearing open - hopefully it has a nice gold foil under the paper and you get to lovingly fold that back; then the color of it, the shape it's pressed, the feel of it just slightly melting in your fingers, the scent of the cocoa notes, then the feel in your mouth, on your tongue, the melting velvet, and then the taste.

The Intelligent Hedonist is not just a complainer or a crank. He/she doesn't just moan about the shitty restuarants or the bad movies. He seeks out the good stuff. His life is an endless quest for things that are made with care and skill.

A lot of semi-smart people see things like typical drug users and rightly think "that's gross, that's a cheap way to get a high, I don't need that, I'm above that". Well, sort of. You're right that they way the dumb hedonist is using drugs is wrong, but you're wrong in thinking that the highest goal is to avoid them. The same thing goes for casual physical sex, many semi-intelligent people see it and think of it as manipulative, degrading, cheap, unfullfilling, and thus swear off it completely and spurn all those who partake of it. Well, yes, again you are right that the way dumb people pursue that pleasure is in fact gross, but that's because they're doing it wrong.

03-22-09 - GDC 2009

... is gonna suck. Tons of people that I usually look forward to seeing aren't even going.

I'll be at the RAD booth much of the time, so stop by and say hi. I'm not really pushing my product (Oodle) yet, but there will be an early preview of it, and of course you can see all the other great RAD libraries (and a preview of Sean's Flash/GUI product too).

Obviously, the Larrabee talks are going to be awesome, but I haven't heard of anything else too exciting yet. I'll add to this post if I find anything worth going to.

03-21-09 - The Marvels of German Marketing

On all these car sites you constantly read about the "fine German engineering". What? I mean the stuff is really nice, I am very sympathetic to their aesthetic, and I love that they are driver-centric. But that's just the design choices. Their build quality is just awful. You get quotes like this :

"It is assembled with typical BMW care and craftsmanship with high quality materials used throughout"

the real marvel is the incredible German Marketing. Holy crap they have done a number on people.

You also get lots of auto writing that just doesn't make any sense at all :

"The acceleration, especially from a standing stop, is super responsive and yet restrained enough to delight."

What? In what world does restrained equal delight ? Oh yes, please restrain the acceleration more, oh how delightful. So a Hyundai Elantra makes you jizz in your pants?

03-21-09 - Search Awareness

It's very important to be aware of how your product name will work in search these days. If you name your band something like "Soap" you're never going to show up.

For example apparently there's a company called "Rad Technologies" in Hyderabad, and they posted a job listing on Oodle.com (a big classified site) , so if you search "Oodle Rad" you find them. There's also a Developer dot Ooodle dot com (for the Oodle classified site) which has an Oodle API and so on.

Anyway, the thing that's annoying me today is I'm trying to search for differences between Directx 8 and 9, and it's just impossible to search for anything related to DirectX any more.

What they should have done is made a different name for the runtime and the developer API. Call the runtime DirectX but call the developer interface apiX or whatever. That way when developers are talking about the internals, we can find each other.

03-21-09 - Actions for Outcomes

URG FUCKING TAKE YOUR SHOES OFF YOU COCK

I wonder what the outcome would be if I yelled that out at the top of my lungs. Improvement or no?

03-20-09 - Car Buying Mentality

Car buying is one of those things where you can easily fall into the trap of worrying about what other people will think of it. I don't just mean buying a car to impress others, that's the obvious trivial way and not a big deal. The more subtle bad thought process is mentally justifying your purchase to others. People who buy a Honda Civic are preparing their rationale in their heads as they buy it, imagining conversations where they tell people "it's just so practical, it's very reliable, it's fast enough for me for now" or whatever.

I've always been biased against Porsches because they're underpowered for the money, and they are so often the car of middle aged mid-life crisis men. So what? Getting as much horsepower per dollar is not the goal. It's to get the car that you enjoy. In my head I can hear the taunts of the stupid public saying "oh but some dumb American car has way more power at half the price, how could you buy that?" and I prepare a response in my head. Fuck you, I don't even have to respond to you. Stop thinking of rationales.

In my head I keep hearing people say "why would you get a 1 series when you can get a 3 series for just a few k more?" Well, voices in my head, I'll tell you why - because I like it better. It's smaller, lighter, faster, stiffer, more nimble, more fun. But really the point is that I don't need to make reasons for you, voices in my head.

In the end there are just way too many conflicting factors to weight them sensibly against each other in any objective way. You just have to go with your instincts.

03-19-09 - Test Drives

So I went test driving with Ryan yesterday. Here's what we saw :

Nissan GT-R : it looks way bigger in person than it does in images, it's really a hulking beast. The cockpit is very tight, you really feel like you're wedged in, which I guess is good if you're gonna push it. Unfortunately this salesman was a complete dickwad retard and wouldn't let me drive it. WTF I'm not gonna buy it if I can't drive it. If I had felt how it moves and loved it I might have considered it, but it is rather impractical. The douche kept telling me about how rare it was and the premium on the price. Look dickwad, there are *tons* of GTR's for sale on Ebay right now, in fact it's easier to find than an Audi RS4 or a new M3 or a BMW 135.

Infiniti G37 : this is the "luxury" version of the Nissan 370, it's the same engine and chassis, just different body and nicer interior. I use quotes on luxury because there is not much luxury to be had here. Everything feels like a cheap Japanese car; actually it's worse than that, the interior is *way* worse than the interior of my Prelude, because the Infinity is all cheap plastic, but it tries to look fancy, so it's got this retarded plastic analog clock in the dash (LOL?) and these shiny fake-metal plastic pieces everywhere. The whole thing kind of reminds me of like a strip mall with roman columns in front of it. Anyway, the interior isn't that important to me, how does it drive? Meh. They only make the 4WD in auto so I tried the RWD manual. The shifter is okay (though we smoked the clutch up pretty good). It's definitely got speed, but you don't really feel it. The power delivery is weak at low revs, and the torque just isn't there when you want it. There is a huge excess of Infinitis right now so they can be had very cheap (perhaps below invoice).

Audi S5 : this thing really looks beautiful in person. The styling is sporty but subtle. It actually really reminds of the BMW styling, but just smoother and better. The interior is also (mostly) nicer than BMW, nicer feeling leather and better arm rests and just nicer trim all around. The only exception is the center console control stuff which is oddly bangled and blingy looking with all kinds of shiny metal and studded pieces that look like they belong on a rapper's necklace. Unfortunately the drive quality just sucks. It feels like a boat. It feels really huge and heavy, it's got a big powerful engine but it doesn't really feel fast. The steering was way too loose, by far the loosest of any car we drove that day. Throttle response was laggy. Visibility is very poor - the rear view mirror is tiny and the blind spots on the side are big. I really hate this trend in car styling where they make the body taper to the front so that the butt is really big, but they also make the roof taper down in the back, so that the rear window is just microscopic (the 370Z has the same problem, in fact a lot of these cars do). In the Audi you get a rear view camera to help you back up despite the bad window and the huge beastliness of this road-boat. Yuck. The sales guy told me there's a selective sport option that tightens up the feel but I don't believe him, and it would still be huge and heavy. It's a fucking V8 with 350 horsepower and it just felt slow. Plus it's a fucking Audi which is just one of the worst made cars around right now.

BMW : the dealer guy was nice and we drove a ton of cars and they were all quite pleasing. In all of them the interior is pretty nice and solid feeling. The steering controls are direct and responsive. Personally I'm not a big fan of the iDrive computer stuff, however it is better than the ghetto-iDrive-knockoffs that were in the Audi and Infiniti. It seems hard to find a car now without all kinds of unecessary gadgetry, and if you are going to have them the BMW gadgets seem to be the best. The non-iDrive interfaces to the stereo and such are awfully minimal, like it just looks like it's only partially finished, with buttons not even labelled and such.

335xi : this is the 4WD twin-turbo 3L V6 that doe 300 hp and 300 torque. This one was a little disappointing. Steering felt good, but throttle response was very laggy. I would jam on the gas and it would take a good "one missisipi" before the power kicked in. Not sure what the deal was, maybe the AWD was slowing it down? The RWD 135 we tried later had the same engine and didn't seem to suffer from the safe sluggishness. Also it was an automatic, even though I put it in sport mode it was hard to coax aggressive gearing out of it. It does feel a little on the big & heavy side to me, but suspension and steering were still very crisp. I'd like to try a manual but they didn't have any.

M3 : for the fuck of it I tried the new M3 with 4.0L V8 414 HP ; this one had the dual-clutch automatic / paddle shifter setup. The transmission was very quick and smooth. I'm sure I could drive much faster with this semi-automatic kind of setup than with a true manual, but I did miss the feeling of the manual a bit. I dunno if I'd get used to the semi-automatic over time. I think the steering wheel paddle shifts are a bit of a gimmick and not very practical. It's nearly impossible to keep your hands on them through a hard turn, which means you get stuck in a bad gear. Maybe on the track they would be useful. The engine sound in this thing is glorious, it's deep and rumbly and loud, it sounds like the beast it is, and it's got tons of power. It definitely does feel big and heavy, you have to muscle it around. The "M" button is indeed exciting. There's just something pleasing about flicking a switch to turn on the better mode (like the computer "turbo" buttons of old). It is kind of just a gimmick for me though since I think I would just drive it in sporty mode all the time. (with the "M" turned off it feels like a big heavy luxury sedan). While it was good, I didn't really love it. If you want the feel of an American muscle car but in a German luxury sedan, and some tight road feel - this is for you. But that's not really what I want. I want the feel of a go-kart, but with low end grunt, and some decent build quality and comfort. I do like the controls to turn traction control on and off and all that stuff.

135i : this is the same engine as the 335xi, but in a slightly smaller, lighter RWD package. Whoah, this was a revelation and probably our favorite car of the day. The rear seats are pretty tiny, there's no AWD. Again we were stuck with an automatic, but the semi-auto sport shift control thingy was pretty good. The front cabin is plenty spatious, you can sit up straight and spread out. It's got twin turbos like the 335, but you can't feel them kick in at all, there's no real power gap, it feels very smooth and always available.

While the 135 is a bit lighter, it's by no means light. It weighs 3,373 lbs , (vs 3,571 lbs for the 335i and 3,759 lbs for the 335xi). But it felt really nimble and stiff and peppy. It's got a slightly shorter wheelbase than the 3 series, which I think makes it feel tighter. On the down side, it is a new line which means it is somewhat rare and it doesn't look like prices are too great (I'm seeing it go for MSRP - close to $40k; I'd like to see it go for Invoice - close to $32k). Also because it's a new line there seem to be some 1 series reliability problems . Hrrumph.

The steering feel and response on all the BMWs was excellent. The double clutch automatic was superb, super quick and smooth, and the manual up-down control was okay.

Overall there wasn't anything I fell in love with. It was fun - my god, all these cars have tons of power, and many of them have great screaming engine sounds, but they just feel heavy and laggy. The 135 was the most pleasing and I think I could be happy buying that right now, but it's not totally ideal.

At the end of the day we got in Ryan's WRX and it reminded me what a nice car that is. Very light snappy feel, good control. If only it had a little more low end grunt and slightly more comfortable interior it would be pretty perfect. Eh, I also feel like you sit a bit too high and upright in the WRX, I prefer to be slung down in a cockpit a bit more. Fuck maybe I should just get one anyway.

As usual I was just shocked at how shitty the salesmen were. The BMW guy was nice, but only in a relative sense in that he didn't actively deter us from buying his cars. At every dealership we'd ask basic questions about the cars - what's the engine in this one, what transmission are available, what's the different between the sport package and the base, etc. and time after time we'd get "I don't know" or even worse - just completely wrong answers. We asked the Infiniti guy how much the G37 weighs. After a bit of hmm , err, he wanders around to read the labels on the car and announces 2400 pounds. I'm like, "umm , no, that's not right" , so he umms and errs around a while and announces "4600 pounds". Umm, no, I don't think so. (the real answer is 3,590 lb). I literally had to prompt the salesmen to give me brochures and business cards. You owe me 10% of your comission.

I also did not see any of the desperation that I hoped for. Despite constantly reading about how bad things are in the news, I have yet to see it. I found these nice graphs on the auto industry downturn that show just plummeting sales this year, and there are tons of photos around the net like this or this of huge lots of unsold cars. But I have yet to see big deals or much eagerness from dealers. Wouldn't you rather sell the cars for a tiny profit than just have them sit? It makes no sense to me. I have money and want to spend it, give me a bargain!

BTW fucking foot-pound vs. newton-meter for torque is a disaster. Part of the problem is the conversion is only a 1.35 multiplier so if someone says "300 torque" you can't just guess the units. That's probably 300 Nm = 222 Ft-lb. Of course there's also the horsepower fuckup; there's an "english" or "mechanical" horsepower and a metric horsepower which are off by a factor of 1.01, so at least that's not a huge difference but you have no idea what anyone is talking about. And then of course the standard horsepower that's quoted is "brake" horsepower (bhp) which is really sort of retarded, because it just measures the engine in isolation; what's really more useful is the effective or "wheel" horespower, but car makers never tell you that.

Some other things I might have a look at : previous gen (E46) M3's , they were a bit smaller and lighter, though I find the styling very generic and unexciting. Audi RS4 is very fast with good AWD, but it is an Audi, and very expensive, and awfully heavy and a huge gas guzzler. Lexus IS - the specs are good, but Lexus tends to make cars feel really mushy and boring, plus the manual is only in the 250. The advantage is that Lexus actually makes cars that don't fall apart.

03-19-09 - Seriously-

My god I am so fucking sick of your retarded broken ass software. Fucking bill pay web site can't handle commas or dollar signs in the number entry boxes. WTF I just copy-pasted from the amount you are showing me is due into the payment box and you fucking freak out and can't handle it !? In fact, all web based UI can just bite me. Ten second stall outs to show me the next GUI box are not okay.

03-19-09 - How Not To Invest

You can learn a lot about how to invest by doing the opposite of me. Some of my guiding principles that have been hugely detrimental to me are :

"It's just a bubble - I'll stay out" . Sounds reasonable, but is quite silly. Bubbles are great opportunities to make a lot of money. Staying out because "it doesn't make any sense" or "it's rationally wrong" or "its fundamentally overvalued" is just shooting yourself in the foot. I've intentionally stayed out of tech stocks, real estate, gold, etc. because they are illogical bubbles.

"It's too late to play that now" . Over and over I've seen very obvious trends but figured I missed the good early chance to get in, so now I should pass it up. One example is bubbles, it's so obvious when they're happening, and I would think "oh, this is super obvious to everyone now, it's too late". No, it's not. These things take a long time and there's plenty of chance to get in late. There have been tons of obvious plays recently - like investing in China and India a few years ago - where I thought oh it's already obvious to everyone it's too late to get the good value.

"That's too obvious the market must have compensated for that already" . I always assume that the market is doing a good job of compensating for news and obvious trends. Like if some company announces an awesome new product and everyone loves it - the stock should shoot up based on that news and already have all the good news baked into the value, right? Well, no, not really. Things like "Halliburton will do well under Bush/Cheney" is just such an obvious play that I thought it would never work because the market had already compensated. Similarly stuff like shorting Home Depot in the recession.

"I should do what's been proven to work best over time" . In scientific studies of investment strategy, no portfolio strategy beats buy and hold in the long term (or if it does it's by a very small amount that's less than transaction cost). So you're "supposed" to just find broad market funds with minimum expensenses. Listening to that advice has been a huge mistake.

03-17-09 - The Wrong Blame

As usual the retards that are finally up in arms about our financial disaster have started getting the blame all wrong.

The Daily Show segment about short sellers really pissed me off. (the fact that they mocked the random short seller guy and actually seriously listened to the crazy Overstock.com loony-toon guy is ridiculous). My god, short sellers don't bring down the companies. The companies are failing. Short sellers *might* occasionally help prick the bubble of inflated stocks that are going up purely based on momentum and collective belief in a fantasy.

There actually was a country that legislated that stocks could only go up. You were only allowed to sell stock for equal or greater value. (someone help me find a reference to that). Obviously that's retarded. Keeping bad companies afloat should not be our goal. Taking all the fluidity out of markets is not the goal either.

One bad aspect of the whole short sell / trading thing is how twitchy and over-reactive people have gotten. Not just day traders but professional brokers. Some good news comes in and stocks shoot up way past their real value. Bad news (like a bunch of shorts) comes in and stocks shoot way down. That's obviously bad.

If you're going to get mad about some practices in trading, there are plenty of things to be up in arms about. An obvious one is the intentional spreading of information (false or not) to affect stock prices, which has become quite standard and is borderline illegal and definitely unethical. The collusion at big banks between the investment bank part that offers a certain stock and the brokers and analysts that recommend that same stock is another.

Another very common practice to get mad about is the way the hedge funds stalk the major indexes. Quite a few of the large hedge funds now basically act as friction on the market. Any time the market wants to move anywhere, they take a piece. They act as an energy drag, sucking off a bit for themselves. This works basically because all the large mutual funds track certain indexes. A huge amount of the total stock ownership is through these large mutual funds. The indexes and the funds take some time to react to things, so when something happens, they have to announce it, and then it takes them a while to get things done. The mutual funds often have the problem that they have so much money to move that it takes them a long time to buy or sell the stock they're after.

The simplest and most obvious case occurs with the S&P 500 (or the DJIA). Whenever a stock is added or removed from the list of companies in the index, all the many huge funds that track that index must sell one stock and buy another. These huge quick computer-automated hedge funds jump in and buy up the stock that the funds need to get into, then sell it to them for a higher price. This is basically like if you walk into the grocery store and announce that you're buying apples, and some asshole runs over and buys all the apples and then offers to sell them to you for double the price.

These is just literally stealing money from long term investors. I'm sure there are lots of other practices like this that I don't understand, but plain old short selling is not one of them.

The other retarded fixation of the media was the whole corporate jet issue, and more generally just slight excesses of spending, like million dollar parties and whatnot. It's ridiculous on so many levels. First of all, it's a fucking drop in the bucket. It's not even a huge waste of money because it saves them time and whatnot. Second, the idea that the rich should stop spending lavishly or be embarassed about spending now is completely backward and will only make things worse if they tighten the purse strings. Third, the fuck-tard executives are getting paid way more than that in salary and bonuses and options; in fact there have been a few cases where some executive redecorated his office with a million dollar bidet and the media got all excited, so he just paid for it with his own salary. What? How is that better?

The recent AIG bonuses brings up a related point - yes, it's a bit ridiculous that they're paying out huge bonuses after receiving hundreds of billions of bailout money - but it's our own fucking fault. We just gave away a trillion dollars to all these fucking crooks with absolutely *ZERO* strings attached. No regulation, no requirement about how it be used. Of course they're just going to pocket it, why would they pump it into their failing business? That's what they're good at - giving themselves profits, that's their job to some extent. It's the fucking government's fault for giving out that money without regulation. Now congress and the media gets in a big tizzy and calls them in and says "hey, waaa you're not spending it how we wanted" and the executives are just like "meh, fuck you, you gave us the cash, I'm buying a new yacht".

I think the focus on the hedge funds like Bear Sterns or investment banks like Merrill is a bit out of place too. We should have just ignored them. It's perfectly fine for a hedge fund to take huge risks and sometimes lose on those bets - but we have to just let them fail! There is basically zero public interest in those companies. The invest the money of rich people and big institutions - they're not regulated and have got to just be allowed to fail. In the future we don't need tighter regulation of funds - we need to pledge to never bail them out. That might mean limitting how big they can be and also perhaps limiting how much regulated institutions can invest in them. Basically never want them to be "too big to fail".

The other myth that's being spread by the popular media is that all these poor homeowners are just victims in the crisis. Nonsense, in fact they're a large part of the problem. Now certainly a few people were victimized, given really bad mortgage deals that they didn't understand, and they just wanted a place to live and weren't speculating. But tons of people knew exactly what was going on. They were intentionally buying way out of their means because they wanted to get rich on real estate. They intentionally got neg-am loans because it allowed them to leverage up. They were speculating, they were watching TV shows about flipping properties. Most people knew we were in a bubble and kept buying anyway. In theory I don't blame them, they saw a profit opportunity and took it, but that doesn't mean we should now feel sorry for them or protect them or bail them out. However the reality is that US law has lots of favorable protections for mortgages, and speculating on investments that are protected by the government is definitely unethical and really should be illegal.

Basically my view is that there are two reasonable extremes : (1) completely free market with zero protections and little regulation, or (2) safe protected market with lots of regulation. Either one is okay (morally), but the thing that has gotten us in so much shit is our semi-protected semi-regulated market in which people can speculate and then have their risk covered by the government. That is fucked up. Also the illusion that things are regulated when they really aren't, and the ability of financial institutions to cross the dividing line between the safe market (in US law, a "bank" with FDIC insurace is supposed to be very safe and have plenty of capital and not take too much risk - similarly a mortgage is supposed to be a very safe type of loan that can be covered by the value of the property).

The big problem I see in general is that companies and individuals profits weren't tied to the deals they were making. Mortgage brokers for example were only motivated to do as many deals as possible, because they take a commision on each deal, and then just resell the actual mortgage, so they get none of the risk from the actual property. This gives them zero motivation to actual do a good deal or even turn down borrowers that are bad risks.

It seems like we need some kind of law to make profits more tied to actions. What we don't need is a bunch of micro-regulations about exactly what constitutes a safe mortgage or a good borrower or whatever. Unfortunately that's the way the US government tends to solve problems. It's much better to let the free market work and decide for itself who's a good borrower - the problem is just the mismatch of reward from actions.

Mortgages in particular seem pretty easy to fix. If mortgages could only be issued by institutions that backed the mortgage themself, it would all be fixed right there. Now they don't want to offer bad loans because they are backing the loan and they have to eat the default. Brokers no longer get paid just on volume, but rather from the actual return of the mortgages they offer. No more trading mortgages, no more government Mae subsidies backing the industry. Seems very simple and obvious and I don't really see a downside.

Another example is the credit rating companies. They have a complete conflict of interest and no motivation to rate things right. In fact their motivation is just to over-rate everything because that makes people happy and makes them issue more securities, so you get more business. The easy way to fix this is just to eliminate all of these made up abstract nominal credit ratings. Boom. Instead, the credit rating companies offer insurance on the securities. The value doesn't necessarily come in anyway *buying* that insurance, but simply seeing the price of it gives you the effective rating. That is, the price of the insurance on a given asset *is* the rating of how safe it is. If the rating is wrong, then the insurer can lose money because of the mistake. Thus they are motivated to give it the right rating. And it's in absolute directly measurable units - dollars.

Instead we'll probably have some fucking huge mess of laws about how the rating companies have to work, what exactly constitutes each of the rating scales, blah blah blah. That's terrible. Huge mess of regulation on top of a basically corrupt system is the wrong way to do things. Instead dig up the system and make it non-corrupt. Use the free market, but force the free market to work correctly - that is, make pricing honest and make the people taking the profit take the risk.

I like the idea of forcing people to price things as a way of enforcing honesty. Like if a trader recommends a certain stock - boom you have to buy it. If you recommend shit, you will lose money. Thus you are motivated to only recommend things that are actually good. You could make a law like anyone publicly recommending a stock has to buy it and hold it for at least a year.

03-16-09 - Temp Names

I made the classic error of actually trying to use the APIs you're supposed to use instead of just writing my own code from scratch.

First of all, the stdlib "tmpnam" is just horrifically broken in the MSVC CRT. I don't understand WTF they're thinking, but it's just awful in various ways :

1. It actually puts the files in the ROOT of your damn drive ! WTF.

2. It doesn't work in Vista at all (you can't fopen the name they give you back). Presumably because it's putting the file in the root and the app doesn't have write access to the root (?).

3. The names are awful, like "s38b" , usually with no extension at all. This makes it very hard to delete them safely when you get like 100,000 of them piled up in your fucking root.

Okay, okay, so we're on Windows we can use the Win32 functions. Well, on the plus side, GetTempPath seems to actually work, it gives you the dir specified by the "TMP" environment var (BTW if it can't find a temp setting, it falls back to the Windows directory which is kind of awful, but whatever, it at least finds the temp dir normally).

On the minus side, GetTempFileName is retardedly awfully bad.

If you just glance at the docs is seems like it's pretty reasonable. You stick 0 in for uUnique and rock on.

But then one day your app grinds to a complete halt because of GetTempFileName and you go "WTF?". Well guess what, GetTempFileName only actually makes up to 64k temp file names. And when it actually checks the name for existence and then tries again. This means that if you even get over 10k temp files in your temp dir, it can make names and check existence over and over before it finds an open slot. If you get more than 50k temp files, you app is basically dead.

Now obviously you don't want to have a ton of temp files hanging around, but Windows never cleans up your temp dirs so if you have some bugs at some point or even just other random broken apps, you can easily crud up your temp dir. And the prefix is only 3 characters so you can often run into other people's prefixes!

One improvement would be if they stuck the app's name onto the temp path and made a subdir, so that you at least got your own dir to play in so that one broken app couldn't mess up the whole system.

On the plus side they actually use the extension ".tmp" so you can go delete everything.

Anyhoo, this is all silly because we have this thing called "long file names" and we don't need to be using 4 characters of hex (!? WTF) which makes it so easy to get name collisions.

So I wrote my own; here's my current version of MakeTempFileName :

// stdc tmpnam seems to put files in the fucking root !?
void MakeTempFileName(char * into,int intoSize)
{
    // use better routines on windows

    do
    {
        char tempPath[MAX_PATH];
        GetTempPath(sizeof(tempPath),tempPath);

        static uint32 s_seqNum = 1;
        
        uint32 seqNum = s_seqNum;
        ++s_seqNum; // not thread safe, whatever
        
        // use tsc or something to get a real random int :
        
        Timer::tsc_type tsc = Timer::rdtsc();
        uint32 tscLow = (uint32)tsc;
            
        char tempFile[MAX_PATH];
        sprintf(tempFile,"cb_%d_%08X.tmp",seqNum,tscLow);
        
        CombinePaths(tempPath,tempFile,into);
        into[intoSize-1] = 0;
        
        // retry while this name exists :
        //  (would be very bad luck)
        
    } while( FileExists(into) );
    
}

yes, yes, this calls lots of cblib stuff.

03-16-09 - No place for money

My TIPS are down about 10% in the last year. All bonds fell a lot during the crash. I'm not sure exactly what the reason is, I guess corporate bonds lost value because there was a fear of defaults as people realized the ratings weren't what they were supposed to be. Also the yields have plummetted which has made them less desirable. I don't really get it though. I mean I guess even though the bonds are fixed income, the price for them is set by the market, so is prone to fluctuations of supply & demand.

Saving accounts are no good either. Here's the delightful ING reports :

Dec 30, 2008    Interest Rate Change to 2.472% (2.50% APY)
Jan 20, 2009    Interest Rate Change to 2.374% (2.40% APY)  
Feb  3, 2009    Interest Rate Change to 2.178% (2.20% APY)
Feb 18, 2009    Interest Rate Change to 1.835% (1.85% APY)  
Mar  3, 2009    Interest Rate Change to 1.638% (1.65% APY)  

Steadily plummeting to zero. Though really that's just an indicator of falling inflation. The real inflation rate is maybe something like the ING APY plus 2 or times 2 or something like that.

I think almost anything you do with your money is a loss. The reason the bank is giving you X% is because they think they can do better and make money off you. There's no free money in the world, the bank is basically investing for you and taking a big commision. Bank interest rates are almost always below inflation (the only exception is during brief periods when things haven't adjusted yet).

Anyway, since there's nowhere to save I figure I'll just invest in Hookers and Blow.

03-16-09 - Windows NetWorking

Windows "File & Printer Sharing" Networking is really annoying me. It's one of the main culprits of long mystery stalls during boot. It also gives you the awful horrible stalls when you open Explorer for the first time.

What I would like is to be able to boot with Windows Networking completely disabled. Then when I choose at some later point I'd like to be able to turn it on. Can I do this?

The other thing that kills me is if I slip while typing in CMD and accidentally dir to one of my networked mapped drives, I just get a huge stall.

My Windows XP boot takes about 1 minute. I've been using "BootVis" and "BootLog XP" to check it out. BootVis is pretty cute, it makes nice graphs, but the actual important part of boot it just labels as "services" with no breakdown about *which* services. BootLog XP does a bit of a better job because it shows each DLL load and times each one, so you can make some educated guess about which services are taking all that time.

There's tons of little shit, okay, whatever. The AntiVir takes a lot of time, maybe 15 seconds total. Oh well, that's life.

csrss.exe is the next biggest time eater, it takes abour 10 seconds. Apparently most of that is because of loading the registry. I checked "pagedfrg" and my registry is not fragged at all.

The other big thing is svchost starting up the networking services which takes around 10 seconds.

03-15-09 - Hyundai Genesis

One thing I've been shocked by as I read about cars is just how fast they all are now. In the last 10 years the average horsepower has shot from 170 to 230. A 300 HP car used to be a rare performance beast, but they are now extremely common in the luxury/sport segment.

You can see it very nicely in this graph of Horsepower vs. MPG over time . As much as I appreciate this, it's pretty retarded and obviously way out of step with the times. The auto industry seems really really slow to react. People have wanted more economy for 5 years now, but they have just kept churning up horsepower. Now that gas prices are falling back down, the industry is just about to start putting out more efficient cars.

Anyway, what this means is there are lots of ordinary cheap cars that are pretty damn good. For example, the Mazda 6 which is one of the most reliable cars you can buy, costs around $25k and has a 3.7L V6, 24 valves, 272 hp @ 6250 rpm. Not bad !

Another new cheap asian sporty car coming out is the Hyundai Genesis. I know a lot of people have bad associations with "Hyundai", but if you take the sensible position that "build quality" means it won't fall apart and break down when you drive it, then a Hyundai is actually a better built car than any of the supposedly well engineered German cars.

The Genesis has 306 hp and 266 lb-ft of torque from a 3.8-liter V6. It's RWD and is basically a direct compentitor to the Infinity G37 Coupe. But it costs $30k , vs. $40k for the Infinity (and around $50k for a comparable BMW). And actually the Genesis has much more low-end torque than either the G37 or a BMW 3, both of which need high RPMs to make power. (IMO a flat torque curve is much much better).

I also really like the styling of the Genesis. While the Germans seem to be racing each other towards rice-boy baroque over-decoration with unnecessary bangles and slashes, the Genesis has a sweet smooth simple styling, that's maybe a tad boring, but at this point the only choices in car styling is to pick the one that is least bad.

Check out the direct comparison of Hyundai Genesis vs. Infinity G37 .

Sadly the Genesis is only RWD, and while the G37 comes in AWD, only the automatic is available, and the manual is only RWD. It reminds me of the dumb Lexus models that only offer their better engine with automatics. WTF. I think a manual is crucial. Though a no-stall first gear would be nice for rush hour traffic.

It would be really useful to be able to see a history of eBay final sale prices. You can search for "Nissan GT-R" on eBay, but it's hard to tell from the current prices what the actual sale prices are. I'd like to see a chart of sale prices over time like NexTag does. I guess they intentionally hide that info, but it would be pretty easy to scrape if you had a bunch of spiders and a fat pipe.

Also see consumer reports reliability summary and warranty reports by manufacturer.

BTW just looking at the ranking order of warranty reports is a bit deceptive. There's really a huge step :

Group A : < 10% very good

# 1. Mazda - 8.04%
# 2. Honda - 8.90%

(! paradigm shift here !)
Group B : not bad

# 3. Toyota (*) - 15.78%
# 4. Mitsubishi - 17.04%
# 5. Kia - 17.39%
# 6. Subaru - 18.46%
# 7. Nissan - 18.86%
# 8. Lexus - 20.05%

Group C : not good , around 25%

# 9. Mini - 21.90%
# 10. Citroen - 25.98%
# 11. Daewoo - 26.30%
# 12. Hyundai - 26.36%
# 13. Peugeot - 26.59%
# 14. Ford - 26.76%
# 15. Suzuki - 27.20%
# 16. Porsche - 27.48%
# 17. Fiat - 28.49%
# 18. BMW - 28.64%

Group D : very bad - 30% +

# 19. Vauxhall - 28.77%
# 20. Mercedes-Benz - 29.90%
# 21. Rover - 30.12%
# 22. Volvo - 31.28%
# 23. Volkswagen - 31.44%
# 24. Jaguar - 32.05%
# 25. Skoda - 32.12%
# 26. Chrysler - 34.90%
# 27. Audi - 36.74%
# 28. Seat - 36.87%
# 29. Renault - 36.87%
# 30. Alfa Romeo - 39.13%
# 31. Saab - 41.59%
# 32. Land Rover - 44.21%
# 33. Jeep - 46.36% 

(* on Toyota because apparently Toyotas made in Japan are very good and would be in Group A, but Toyotas made in America are shit and bring down their average).

03-15-09 - Seattle Weather

The weather really hasn't been all that bad this winter. In fact since January 1st it's been sunny quite often, and it's absolutely gorgeous up here when it's clear, what with all the green and mountain views around.

Just recently another patch of the interminable drear has set in :

Ugh.

We've had a few bouts of hail and sleet and freezing rain and all that gunk. It's so much better if it just really snows, in fact I rather enjoy the snow if it would just dump and then get sunny again.

I like how the newspaper here does stuff like this :

03-14-09 - Merge

I've created this fucking nightmare for myself where I have like 4 different slightly different versions of the same code. I've got my home Galaxy3, my home cblib, my work Galaxy3, my work cblib, the old Exoddus code, and the Oodle Core, and they're all quite similar but a bit different.

Once a week or so I've been trying to merge the bug fixes I do in one bit to the other bits, and it's been just awful. Today on this miserable Saturday I've got a miserable big bit of merging to do. So I went and got the free trial of Araxis Merge.

On the plus side, the GUI is slightly better than P4 merge. On the minus side, the actual merge is completely miserable.

WTF. It seems completely unable to find identical blocks that are just moved.

Like if I in one file I have

AAAA

BBBB

CCCC

And in the next file I have

CCCC

BBBB

AAAA

it totally freaks and just shows that as a big diff. It should show me that stuff is just swapped in location.

I usually wind up doing the merge just by doing windiff between the dirs and then manually applying the edits. Not fun.

The other thing I'd like to see is detection of synonyms. A common thing I have is "gAssert" in Galaxy is "ASSERT" in cblib and "RR_ASSERT" in Oodle. You should be able to detect that. Even if it was a manual config file of synonyms associated with directories that would be okay.

03-14-09 - Computers get Hot

I'm disappointed with how loud my HTPC is. It's got the quietest of Scythe case fans, and the HD is perfectly silent, but I still hear the drone of the fans and it bugs me.

So today I thought I would try the simple solution - just stick in my home theater cabinet and shut the door.

First problem - it's too big. I have an Antec Fusion case so it looks like a home theater component, but it's just a bit bigger than any standard amplifier, and my cabinet is standard size. Annoying, but that's not really a big deal because -

It's going to get way too hot in there. I figured I would have to cut air holes in the back of the cabinet anyway, so I'll just cut a bigger hole so that the oversize case fits. I would up cutting just a few holes, since I had to use a steak knife to punch the entry holes and then my jig saw to cut them. (I would up breaking the steak knife blade, damn shitty stamped blades).

Now it's been running in there a few hours, and it is indeed much quieter, but it's also getting boiling hot in there even with the big air holes. Damn. What I need is a large solid box with two open ends, so that the sound is muffled but it still gets plenty of air.

My next thing to try is replacing the PSU with a lower-power quiet one. They even make fanless PSU's now though that's a bit scary because it just means they pump more heat into the case.

My other option is to throw this damn thing out the window, burn down my apartment, ride my bicycle around the world and never touch another damn piece of electronics again.

03-13-09 - Automatic Prefs

I wrote briefly at molly about the way I do "hot vars" or "tweak vars". I don't really like them. I had a bunch in my GDC app for tweaking, but it means I have to edit the code to tweak things and I don't like that for various reasons. It's way better to have a real text pref file. There are just so many wins. I can source control it seperately. I can copy it off and save good snapshots of prefs I like for different purposes, like a development prefs vs. final run prefs. I can copy it and change it to make new instances. And I can edit it and have my final app load the changes without a recompile (hot var can do this too if you keep the C file around as "data")

Fortunately I have a prefs system in cblib, so I switched to that. But it made me realize that I'd really like to have an automatic pref system. Basically I want to write something like :

struct MyPref
{

int i = 7;
float x = 1.3;
String str = "hello world";
ColorDW color(200,50,177);
Vec3 v(3.4,0,1.7);

};

and have it automatically get IO to the pref file and construction with those values as defaults.

Now, that all is actually pretty easy if I just make some custom syntax and run a source code preprocess over the file before compiling. I could use a syntax like :

struct MyPref
{

int i; //$ = 7;
float x; //$ = 1.3;
String str; //$ = "hello world";
ColorDW color; //$ (200,50,177);
Vec3 v; //$ (3.4,0,1.7);

};

where anything with a //$ automatically becomes a "pref var" that gets IO'd and tweakability and so on.

The annoyance and something I haven't figured out is just how to deal with generated code in a build setting. Should the code generator stick the generated code back into the original file? Should it make another seperate C file on the side and put the generated code in there? Maybe all the generated code from all the C files in the whole project should go together in one big spot?

I dunno but it seems like a mess. Maybe the easiest thing to do would be to put the autogenerated code in the same file, and run the generator as a pre-build step.

It would also be annoying to have to put the code generator in as a pre-build step on every file one by one. So annoying as to be unacceptable actually. I would want it to automatically run on all my files any time I build, so that if I just go and put the autogen markup in any file it does the stuff and I don't have to open msdev options dialogs.

I know people out there are doing things like this, so I'm curious how you deal with the mod time issues and builds and source control.

BTW the autogenerated code will look something like this :

void MyPref::AutoGen_SetDefault()
{
    i = int(7);
    x = float(1.3);
    str = String("hello world");
    color =  ColorDW(200,50,177);
    v = Vec3(3.4,0,1.7);
}

template <typename functor>
void MyPref::AutoGen_Reflection()
{
    REFLECT(i);
    REFLECT(x);
    REFLECT(str);
    REFLECT(color);
    REFLECT(v);
}

pretty easy to generate just by some text movement. The big win of the shortened syntax is that you only have to write a variable once instead of 3 times.

03-13-09 - Windows Game Development

... blows so bad. Can't we just all be Xenon-exclusive? Please?

In a bit more detail - I've been tracking down a graphics driver bug the last few days. And this is like the easy case - it's on Windows XP and it's happening on my dev machine. I remember the old days of hell at Eclipse/Wild Tangent where we'd get a report that something in the graphics screws up, but only on Windows version XX and only with graphics card YY and only with driver version ZZ and only when you run Media Player at the same time, and we can't repro it any other way.

At WT we had a huge room full of like 50 computers with all kinds of different hardware and OS setups. Any time we did a release it had to be tested on all those boxes. (this was back in the day when you had 3d-only cards like the Voodoo and NVidia had just come out with the Riva 128, and you had to support all kinds of weird ass cards, in comparison the differences between even the most different of cards these days are very minor). I'm actually amazed how many PC game development studios don't seem to have this kind of test room.

We also had 2 full-time employees basically running testing all the time. I'm amazed at how many game studios have zero internal testing.

03-13-09 - Paged Pool Leaks

So I tracked down the paged pool leak. It appears to be a bug in my ATI driver (or perhaps a bug in the DX interface that shows up as a leak in the driver). It's caused by locking POOL_MANAGED textures that are currently in the GPU push buffer. When you do that it causes the hardware texture to get aliased to avoid contention, and it appears something in there is leaking. I don't think that whole textures are leaking because the leak is pretty slow, it must just be some kind of texture book-keeping object (eg. maybe it actually is the "texture" struct, just not the actual surface bits).

Anyhoo, nobody cares about bugs that I see via Directx8, but what is interesting is the cool Poolmon utility.

Poolmon lets you see the kernel allocations and thus track down leaks.

You need to turn on some registry settings .

Run Poolmon in a command line with lots of vertical lines.

Use the keys "d" and "p" to get the view you want

Then track away.

03-12-09 - Fixed Memory Pools

The Windows Kernel Paged Pool shit I'm dealing with is reminding me how much I hate fixed size memory pools.

I have fucking 3 Gigs of RAM in this machine and I'm using less than 1 G ! How can you be running out of memory !!!! Oh, because you have a fucking stupid fixed size page thing. Now, okay, maybe *MAYBE* the OS kernel is one case in which fixed size pools is a good thing.

It's common wisdom in game development that dynamic allocations in games are bad and that "mature" people use fixed size pools because it's more stable and safe from fragmentation and robust and so on. Hog wash!

It should be obvious why you would want variable memory allocation. Memory is one of our primary limiting factors these days, and it should be allocated to whatever needs it right now. When you have fixed pools it means that you are preventing the most important thing from getting memory in some case.

For example, your artists want to make a super high poly detailed background portion of the game with no NPC's. Oh, no, sorry, you can't do that, we're reserving that memory for 32 NPC's all the time even though you have none here. In another part of the game, the artists want to have super simple everything and then 64 NPC's. Oh no, sorry, you only get 32 even though you could run more because we're reserving space for lots of other junk that isn't in this part of the game.

Now, I'm not saying that budgets for artists is a bad thing. Obviously artists need clear guidelines about what will run fast enough and fit in memory. But having global fixed limits is a weak cop out way to do that.

Furthermore, recycling pools and maximum counts for spawned items is a perfectly reasonable thing to do. But I don't think of that as a way of dividing up the available memory - it's a way of preventing buggy art from screwing up the system, or just lazy artists from making mistakes. For example, having a maximum particle count doesn't mean you should go ahead and preallocate all those particles, cuz you might want to use that memory for something else in other cases (and of course the hard-fixed-size pool thing can

In general I'm not talking here about *dynamic* variation. I'm talking about *static* variation. Like anything that can be spawned or trigger from scripts or whatever, stuff that can be created by the player - that stuff should be premade. Anything that *could* exist at a given spot *should* exist. That way you know that no player action can crash you. Note that this really just a way to avoid testing all the combinatorics of different play possibilities.

By static variation I mean, in room 1 you might have resource allocation like {16 NPC's, 100 MB of textures} , in room 2 you might have {8 NPC's, 150 MB of textures}.

Fixed sized budgets is like if you partitioned your hard disk in half for programs and data. People used to do things like that, but we all now realize it's dumb, it's better just to have one big disk and that way you can change how you are using things as need arises.

Now, people sometimes worry about fragmentation. That may or may not be an issue for you. On Stranger on XBox it basically wasn't an issue because we had 64M or physical memory and 2G of virtual address space, so you have tons of slack. Again now with 64 bit pointers you have the same kind of safety and don't have to worry. Sadly, 32-bit Windows right now is actually in a really bad spot where the amount of physical memory roughly matches the address space, and we actually want to use most of that. That is fragmentation danger land.

However, doing variable size budgets doesn't necessarily increase your fragmentation at all. The only thing that would give you fragmentation is if you are dynamically allocating and freeing things of different sizes. Now of course you shouldn't do that !

One option is just to tear things all the way down and build them all the way back up for each level. That way you allocate {A,B,C,D} in order, then you free it all so you get back to empty {} , then next level you allocate {C,B,B,A,E} and there's no fragmentation worry. (if you tried to do a minimal transition between those sets by doing like -D +B+E then you could have problems).

Another option is relocatable memory. We did this for Stranger for the "Contiguous" physical memory. Even though virtual address fragmentation wasn't an issue, physical memory (for graphics) fragmentation was. But all our big resources were relocatable anyway because they were designed for paging. So when the contiguous memory got fragmented we just slid down the blogs to defrag it, just like you defrag a disk. Our resources were well designed for fast paging, so this was very fast - only a few thousand clocks to defrag the memory, and it only had to be done at paging area transitions.

Note that "relocatable resources" is kind of a cool handy thing to have in any case. It lets you load them "flat" into memory and then just rebase the whole thing and boom it's ready to use.

Personally after being a console dev and now seeing the shit I'm seeing with Oodle, I would be terrified of releasing a game on a PC. Even if you are very good about your memory use, your textures and VB's and so on create driver resources and you have no idea how big those are, and it will vary from system to system. The AGP aperture and the Video-RAM shadow eat out huge pieces of your virtual address space. The kernel has an unknown amount of available mem, and of course who knows what other apps are running (if it's a typical consumer machine it probably has antivirus and the whole MS bloatware installed and running all the time).

I don't see how you can use even 512 MB on a PC and get reliable execution. I guess the only robust solution is to be super scalable and not count on anything. Assume that mallocs or any system call can fail at any time, and downgrade your functionality to cope.

Now, certainly doing prereserved buckets and zero allocations and all that does have its merits, mainly in convenience. It's very easy as a developer to verify that what you're doing fits the "rules" - you just look at your allocation count and if it's not zero that's a bug.

---

It's just very frustrating when someone is telling you "out of memory" when you're sitting there staring at 1 GB of free memory. WTF, I have memory for you right here, please use it.

The other important thing is efficiency is irrelevant if you can't target it where you need it. Having a really efficient banana picking operation doesn't do you a lick of good when everyone wants apples. A lot of game coders miss the importance of this point. It's better to run at 90% efficiency or so, but be flexible enough to target your power at exactly what's needed at any moment.

Like with a fixed system maybe you can handle 100 MB of textures and 50 MB of geometry very efficiently. That's awesome if that's what the scene really needs. But usually that is not the exactly ideal use of memory for a given spot. Maybe some spot really only needs 10 MB of geometry data. You still only provide 100 MB of textures. I'm variable and can now provide 139 MB of textures. (I lose 1 MB due to overhead from being variable).

Many game devs see that and think "ZOMG you have 1 MB of overhead that's unacceptable". In reality, you have 39 MB less of the actual resource your artists want in that spot.

03-12-09 - ERROR_NO_SYSTEM_RESOURCES

ERROR_NO_SYSTEM_RESOURCES is the fucking devil.

So far as I can tell, the only people in the history of the universe who have actually pushed the Windows IO system really hard are me and SQL Server. When I go searching around the web for my problems they are always in relation to SQL Server.

I don't have a great understanding of this problem yet. Hopefully someone will chime in with a better link. This is what I have found so far :

You receive error 1450 ERROR_NO_SYSTEM_RESOURCES when you try to create a very large file in Windows XP
SystemPages Core Services
Sysinternals Forums - not enough resources problem - Page 1
Overlapped WriteFile fails with code 1450 [Archive] - CodeGuru Forums
Novell Eclipse FTK file io
How to use the userva switch with the 3GB switch to tune the User-mode space to a value between 2 GB and 3 GB
GDI Usage - Bear
Error Message ERROR_NO_SYSTEM_RESOURCES (1450)
Download details Detection, Analysis, and Corrective Actions for Low Page Table Entry Issues
Counter of the Week Symptoms Lack of Free System Page Table Entries (PTEs) and Error Message ERROR_NO_SYSTEM_RESOURCES (1450
Comparison of 32-bit and 64-bit memory architecture for 64-bit editions of Windows XP and Windows Server 2003

Basically the problem looks like this :

Windows Kernel has a bunch of internal fixed-size buffers. It has fixed-size (or small max-size) buffers for Handles, for the "Paged Pool" and "Non-Paged Pool", oh and for PTEs (page table entries). You can cause these resources to run out at any time and then you start getting weird errors. The exact limit is unknowable, because they are affected by what other processes are running, and also by registry settings and boot.ini settings.

I could make the error go away by playing with those settings to give myself more of a given resource, but of course you can't expect consumers to do that, so you have to work flawlessly in a variety of circumstances.

In terms of File IO, this can hit you in a whole variety of crazy ways :

1. There's a limit on the number of file handles. When you try to open a file you can get an out-of-resources error.

2. There's a limit on the number of Async ops pending, because the Kernel needs to allocate some internal resources and can fail.

3. There's a limit on how many pages of disk cache you can get. Because windows secretly runs everything you do through the cache (note that this is even true to some extent if you use FILE_FLAG_NO_BUFFERING - there are a lot of subtleties to when you actually get to do direct IO which I have written about before), any IO op can fail because windows couldn't allocate a page to the disk cache (even though you already have memory allocated in user space for the buffer).

4. Even ignoring the disk cache issue, windows has to mirror your memory buffer for the IO into kernel address space. I guess this is because the disk drivers talk to kernel memory so you user virtual address has to be moved to kernel for the disk to fill it. This can fail if the kernel can't find a block of kernel address space.

5. When you are sure that you are doing none of the above, you can still run into some other mysterious shit about the kernel failing to allocate internal pages for its own book-keeping of IOs. This is error 1450 (0x5AA) , ERROR_NO_SYSTEM_RESOURCES.

The errors you may see are :

ERROR_NOT_ENOUGH_MEMORY = too many AsyncIO 's pending
Solution : wait until some finish and try again

ERROR_NOT_ENOUGH_QUOTA = single IO call too large
Solution : break large IOs into many smaller ones (but then beware the above)

ERROR_NO_SYSTEM_RESOURCES = failure to alloc pages in the kernel address space for the IO
Solution : ???

So I have made sure I don't have too many handles open. I have made sure I don't have too many IO ops pending. I have made sure my IO ops are not too big. I have done all that, and I still randomly get ERROR_NO_SYSTEM_RESOURCES depending on what else is happening on my machine. I sort of have a solution, which seems to be the standard hack solution around the net - just sleep for a few millis and try the IO again. Eventually it magically clears up and works.

BTW while searching for this problem I found this code snippet : Novell Eclipse FTK file io . It's quite good. It's got a lot of the little IO magic that I've only recently learned, such as using "SetFileValidData" when extending files for async writes, and it also has a retry loop for ERROR_NO_SYSTEM_RESOURCES.

Further investigation reveals that this problem is not caused by me at all - the kernel is just out of paged pool. If I do a very small IO (64k or less) or if I do non-overlapped IO, or if I just wait and retry later, I can get the IO to go through. Oh, and if you use no buffering, that also succeeds.