I've finally completed the new GD-ROM module, though ultimately it turned out to be very similar in design to the last one. With the new code I can freely choose what ATA/ATAPI commands are to be executed immediately and what can be offloaded to helper thread.
Unfortunately a thread is not a very good substitute for a real-time device. In typical multitasking OS there's a sheduler that decides when things will happen, there's no guarantee that the helper will start processing given request right away. Simply put, it can start immediately or only after some tens of miliseconds has passed. I could run the code in a tight loop to reduce this lag but that would waste too much CPU power, always needed elsewhere. Not to mention OS can (and eventually will at some point) interrupt any running process if it's required, so this approach would not solve the problem completly, just reduce the odds of ecountering it. Same thing goes for messing with thread priorities - that's not exactly foolproof and is also very inelegant.

Now, why would mere miliseconds ever pose a problem? In a well-designed system the consumer would just wait a bit for the producer to come up with data and occasional lag wouldn't even be noticable. Some Dreamcast software however (BIOS including) doesn't really check if the data has arrived, rather assume it must be so after a certain amount of time has passed.
This isn't that much of a problem for GD-ROM commands that return small amounts of redily available data - those can be simply executed at the moment they arrive, so there is no lag. Disk reads are another story though, those take time to complete and the whole idea was to run them concurrently so that virtual CPU would not be stalled waiting in the first place.

It gets more interesting at this point. A real Dreamcast GD is not anywhere near as fast as modern hard drives, so why doesn't it work in emulator if it works on an inferior system? The answer, again, is timing. It's simply very crucial to get it right - and you can have accurate emulator, or fast emulator, but not both. And you've guessed it right, I'm aiming for fast at this point. It has to be noted that should the original GD/CD media become too scratched or the laser detoriates beyond certain point, the problems I'm having will manifest themselves on Dreamcast as well - so I still say it's simply bad coding on games part.
What are the options... For starters, I will continue to play with the DMA code and try to mimic Dreamcast behaviour as closely as possible. And if all else fails, there's still the "ideal" DMA mode, that is to make transfers always block to be sure the virtual CPU always gets all the data it asks for. This of course causes jerkiness and audio stuttering, mostly to be felt in games using sequenced music or sound effects very closely timed to video.

This whole re-write took quite some time and didn't really help as much I hoped, but thanks to all that work I'm now much more aware of what has to be done yet. I've a few ideas, it's just that every attempt at fixing one thing breaks other stuff. Sometimes very badly I still need to figure out what upsets so much the MPEG library used on demo GDs and some MIL-CDs.

PS. Why didn't anyone tell me Giana's Return got broken with the addition of MMU-capable recompiler?!
PPS. I'm SOOO behing with testing Yuki's stuff now... I'm going to capture & post few screenshots later today. If I get the games to work with the new GD code that is

UPDATE:
No screens yet. I had a bug that prevented most of the images from booting properly, that's fixed now. But there's a bigger problem - the new module is actually even less compatible, timing-wise, than the old one. I'll have to run more tests but now I'm beginning to suspect that some games require GD DMA processing to be done way more often. Which is bad because it seriously affects emulation speed.
At this point one begins to wonder what exactly is good about this new code Well, it is more reliable now. I was hoping that alone would be enough but apparently I was too optimistic...

UPDATE 2:
A bit closer, still not there.

http://pics.livejournal.com/dknute/pic/000c4ef9

http://pics.livejournal.com/dknute/pic/000c5bed

This is starting to seriously annoy me. DMA changes helped with some tiles but not all of them. Then I inserted a few debug statements into the code and even though it reduced emulation speed, it actually made things more stable. And this is the only lead I have so far...