rpix86 0.18 released!

Here finally is the latest version of rpix86! Some pretty big internal changes in this version, so I hope I did not break any games that used to run in the previous version of rpix86.



1. Implemented FPU support in real mode (TradeWars 2002)

I already mentioned this change on my previous blog post. At that point the TradeWars 2002 game did not start up, as it complained that SHARE.EXE was not loaded or was unresponsive. I was able to use DOSBox to determine that the same problem happens there as well, and it is caused by an unsupported INT 21 AH=5C: FLOCK - RECORD LOCKING DOS call. I changed my code to not report an error (but it still does not actually perform any record locking) and that allowed TradeWars 2002 to progress further. However my version seems to miss a file (or I have not set it up correctly) so I don't know if the game actually runs in rpix86 yet.

But in any case, if there are other real-mode programs that need FPU support to run, those should now run properly in this version of rpix86.

2. Implemented support for Borland RTM (Jazz Jackrabbit, Tyrian 2000)

I described this change pretty thoroughly in my previous blog post. After that post I spent some time debugging a timer I/O port problem I had, which caused Tyrian 2000 to hang intermittently. The timer input port 0x40 MSB/LSB toggle got out of sync, and the game got larger values than it expected and went into a never-ending loop. The problem turned out to be my using the same temp variable both for port 0x40 input and output. I changed the temp variable to be different, and this problem seemed to go away. Tyrian 2000 still seems to hang at launch occasionally, but I haven't been able to determine what causes this. If it gets past the launch time, it seems to run fine (but pretty slow).

3. Implemented new -noems command line parameter (Ultima VII)

After those fixes I still had some time to implement further features, so I decide to add the -noems command line switch to disable EMS memory (and emulation of EMM386 memory manager features) in rpix86. This seemed to fix Ultima VII, as it checks whether EMM386 is loaded at startup, and fails to start if it is loaded.

4. Changed the GPIO RS-232 port handling to emulate a null modem cable connection

I have not heard from rpix86 users having success using the GPIO serial pins to connect serial devices to their Raspberry Pi and controlling them from rpix86. As this should work but I have not been able to test this myself, I thought that maybe my emulation is not quite as good as it should be. In this version I decided to experiment adding loop-back handshaking to my GPIO serial port emulation. For USB-to-RS232 device this is not needed, as the device should take care of the handshaking. However, since there are no RS-232 handshake pins in the GPIO connector, I thought that it might be a good idea to fake the handshake signals in this situation. At least my Telix communications program now thinks that my GPIO serial port is already connected when I start it up. Let me know if you can test this out and see any improvements in the behavior or compatibility. Or if it now works worse than before.

That's it for the changes in this version! I hope you find this new version useful, and again let me know of any bugs you encounter. Thanks again for your interest in rpix86!

May 24th, 2015 - rpix86 work continues!

Okay, after a break of about a year, I am back working on my emulators! For the past several weeks I have been working either on my Windows Phone version of pax86 or on rpix86. I returned to working on my emulators in the beginning of April, when I got an email describing problems running TradeWars 2002 on Raspberry Pi. I have received various bug reports during the past year, and have been adding them to my todo-list, but this bug report was slightly different as it looked like the game should actually already work on rpix86. So, I thought I would spend some time looking into what is going on with this game.

Work on supporting TradeWars 2002

TradeWars 2002 has two versions, one running under DPMI DOS extender and one running without it. The problem was that the DPMI one exited with "Cannot enable A20" and the non-DPMI version with "Numeric co-processor required". What bothered me was that rpix86 should already support enabling/disbling the A20 line, and it also has support for the numeric co-processor. So why does the game not detect them?

I started with the numeric co-processor problem, as there are several ways the game could attempt to enable the A20 line, and I thought it might be difficult to determine which method the game attempts to use. After some looking into the rpix86 code, I remembered that I had only coded complete FPU support for 32-bit protected mode, as all the games I have encountered so far have only needed FPU support in that mode. The real-mode FPU code simply silently skipped all operations. Perhaps the game tried to use FPU opcodes in real mode, and when the results were not correct, it exited?

I disabled the real mode FPU opcode skipping code, and indeed, the game encountered an FPU opcode in real mode:
-----------------------[TW2002]------------------------
TW2002: Unsupported opcode! INTVectors=0x42788000
CPU: REAL, 16, 0000FFFF
EAX=00000000 EBX=00003D38 ECX=00007FBD EDX=00001E6E
ESP=00009FFA EBP=00000000 ESI=00000264 EDI=00000042
ES=02D3 CS=1E6E SS=2494 DS=20BF FS=0000 GS=0000
NV UP EI PL ZR NA PO NC VM=0 IOPL=3
1E6E:09EB D93F fstcw [bx]

That looks like the game attempts to store the FPU control word into memory in real mode. I checked the game code further, and noticed that it does indeed do some floating point calculations in real mode. I decided to spend my Easter vacation working on implementing the real-mode FPU opcodes, as now I had a game to use for testing them. This work mainly consisted of adding the real-mode opcode tables, as the main FPU operations are the same between the real and protected modes. It took several days, though.

After I had implemented the FPU opcodes, the game progressed further, but then exited with the following message:
File Error (1) - TWNODE.DAT : Invalid Function
05/24/27 00:00:08 AM Game Halted:
SHARE.EXE not loaded or unresponsive.

c:\games\2002v309>

I have spent several days trying to debug the game and see how it determines that SHARE.EXE is not loaded (well, it isn't loaded, but I would like to fool the game into thinking it is), but have not yet been able to determine this. I recently tested the game in DOSBox, and it gets the exact same error, so I can not even use DOSBox to debug the game behaviour. Perhaps I will look into FreeDOS implementation of SHARE.EXE to determine the possible detection routines that the game uses. However, I got a bit bored with working on this game so I switched to working on my Windows Phone version of pax86 instead. I wrote a blog post about my work there a couple of weeks back.

Implementing support for Borland RTM DOS Extender

I recently got an email with a debug log from Tyrian 2000. The game crashed in the IRET opcode, which in principle should be supported in both real and protected modes. So, I decided to switch back to working on rpix86 for a while.
-----------------------[RTM]------------------------
RTM: Unsupported opcode! INTVectors=0x7387c000
CPU: PROT, 16, 0000FFFF
EAX=000002A8 EBX=00000FFC ECX=000000B7 EDX=00003000
ESP=00000F94 EBP=00000F8C ESI=00000F9E EDI=00001000
ES=005B CS=0020 SS=0018 DS=0018 FS=0000 GS=0000
NV UP DI PL NZ NA PO CY VM=0 IOPL=3
0020:000015EC CF iret


Since the game runs in protected mode, there are several possible reasons why rpix86 would exit at that point. I improved my error logging so that instead of the generic "Unsupported opcode" rpix86 would report the unsupported situation with IRET that it encountered. Running with the enhanced logging resulted in this log: -----------------------[RTM]------------------------
RTM: Unsupported IRET to outer level!
CPU: PROT, 16, 0000FFFF
EAX=000002A8 EBX=00000FFC ECX=000000B7 EDX=00003000
ESP=00000F94 EBP=00000F8C ESI=00000F9E EDI=00001000
ES=005B CS=0020 SS=0018 DS=0018 FS=0000 GS=0000
NV UP DI PL NZ NA PO CY VM=0 IOPL=3
0020:000015EC CF iret

Okay, now that is pretty clear, as I had not yet added support for interrupt return to outer (less privileged) protection level. The fact that the running program is called RTM made me remember that I actually worked on similar issues when I added support for Jazz Jackrabbit into my zerox86 emulator. Since zerox86 runs on MIPS architecture instead of ARM as in Raspberry Pi, the assembler code is very different and thus I had not added similar support to my ARM emulators at that time. However, I had written several blog posts about supporting Jazz Jackrabbit in both my DS2x86 and zerox86 blog posts, so I thought that using those blog posts and taking the ideas and algorithms from my MIPS emulator code would make it reasonably straight forward to implement the same support to my ARM emulators.

1. Interestingly, when I started up Jazz Jackrabbit, it exited with "Cannot Enable A20". The same problem that the DPMI -version of TradeWars 2002 had. At this point I thought it would help me quite a bit if I got my debug-version of DOSBox running again. I had switched to a new hard drive and installed Windows 8.1 since I last ran Visual C++ 2008 Express for DOSBox, but luckily I still had the install program for that old Visual C++ on my old hard drive. It installed fine, and pretty soon I was able to run Jazz in DOSBox. I noticed that it uses the keyboard controller method of setting the A20 line, and somewhat to my surprise I noticed that I did not have a proper handler for that yet in rpix86. I implemented that, and then got Jazz go drop into debugger in rpix86 at the same point as where Tyrian 2000 drops. Thus, I switched to working on Jazz Jackrabbit, so I could follow my old blog posts step by step.


2. The first step was adding that IRET return to outer priority level case. This was not terribly difficult, as I was able to simply look at my implementation for zerox86, mentally convert the MIPS asm to ARM asm code line by code line. I had to be careful to use the correct CPU registers in my new ASM code, by comparing this code to the normal IRET code I already had implemented, though.


3. After I had implemented the IRET opcode, the game not surprisingly encountered the priority level jump to the opposite direction. This is done using an INT opcode.
-----------------------[RTM]------------------------
RTM: Unsupported opcode! INTVectors=0x42840000
CPU: PROT, 16, 0000FFFF
EAX=00000300 EBX=00000021 ECX=00000000 EDX=000002A8
ESP=00000F9E EBP=00000FD4 ESI=00000F9E EDI=00000FA2
ES=005B CS=00B7 SS=005B DS=0000 FS=0000 GS=0000
NV UP DI PL NZ NA PO CY VM=0 IOPL=3
00B7:000001F7 CD31 int 31

Very similar approach worked here, I implemented the new code based on my zerox86 code.


4. Next, the game encountered a HLT (halt) opcode in protected mode. This opcode is not allowed in any other priority level except 0 (the most privileged), otherwise it will cause a General Protection Fault Exception.
-----------------------[RTM]------------------------
RTM: Unsupported opcode! INTVectors=0x428b0000
CPU: PROT, 16, 0000FFFF
EAX=00000303 EBX=00000735 ECX=00000000 EDX=000000F4
ESP=00000FCE EBP=00000FFA ESI=00002258 EDI=00002195
ES=0000 CS=0113 SS=005B DS=00B7 FS=0000 GS=0000
NV UP DI PL NZ NA PO CY VM=0 IOPL=3
0113:0000000C F4 hlt

The game was running in priority level 3 at that point, so it was actually supposed to cause a GPF. However, I had not implemented any exception handling in rpix86 yet, so that was the next step in my work on supporting the RTM DOS Extender.


5. At this point I had to spend some time hunting down a bug I had typed into the new INT handling code. Such bugs are pretty difficult to hunt down, even when I know the bug is very probably in the code I just added. When staring at the code you have written, you always see it as you meant it, not as you accidentally typed it in. :-) Anyways, after fixing that bug I was back on track (as I kept following my prior blog posts for those MIPS emulators), the next problem was the LES opcode that encountered a segment that was not present.
-----------------------[RTM]------------------------
RTM: Unsupported opcode! INTVectors=0x42830000
CPU: PROT, 16, 0000FFFF
EAX=00000100 EBX=000003F4 ECX=000004AA EDX=00002000
ESP=000003D2 EBP=000003D4 ESI=0000396C EDI=00000000
ES=009F CS=00DF SS=009F DS=00E7 FS=0000 GS=0000
NV UP DI PL ZR NA PO CY VM=0 IOPL=3
00DF:0000C391 C47E06 les di,[bp+06]

Here I added the Page Fault handling using my new exception handling mechanism, and again was able to progress further.


6. There had been several additional steps at this point that I had had to do for my MIPS emulators. However, since those enhancements were done into the C code, I had already implemented them also for my ARM emulators. The MIPS and ARM emulators share most of the C code, but the ASM code is obviously separate. Thus I was able to skip any new changes for reporting no extended memory for the Borland RTM, adding the "last fit" memory allocation strategy, and returning the proper return value from DOS call INT 21 AH=4D.


7. The next step was to handle opcode 0x8E, which loads a segment selector register. This also needed page fault handling, much like the LES opcode above. As you may notice from the debug log, at this point the actual game [JAZZ] is already running, while in the previous log entries rpix86 was still running the [RTM] loader program.
-----------------------[JAZZ]------------------------
JAZZ: Unsupported opcode! INTVectors=0x427d8000
CPU: PROT, 16, 0000FFFF
EAX=00000000 EBX=000003EE ECX=00000540 EDX=00000000
ESP=000003EE EBP=000003F0 ESI=00002386 EDI=000023C7
ES=009F CS=00E7 SS=009F DS=00EF FS=0000 GS=0000
NV UP DI PL ZR NA PO CY VM=0 IOPL=3
00E7:00000593 8E46FE mov es,[bp+FE]




8. Then I had to implementation the Page Fault handling also to the RETF opcode handler...
-----------------------[JAZZ]------------------------
JAZZ: Unsupported opcode! INTVectors=0x427f0000
CPU: PROT, 16, 0000FFFF
EAX=0000FB43 EBX=00000000 ECX=000001C7 EDX=00000000
ESP=00001FFC EBP=00000000 ESI=00000282 EDI=000023C6
ES=01AF CS=00E7 SS=028F DS=01AF FS=0000 GS=0000
NV UP EI PL ZR NA PO CY VM=0 IOPL=3
00E7:000003A6 CB retf




9. ... and next the corresponding Page Fault in the CALL opcode.
-----------------------[JAZZ]------------------------
JAZZ: Unsupported opcode! INTVectors=0x42810000
CPU: PROT, 16, 0000FFFF
EAX=00000000 EBX=00000277 ECX=0000078A EDX=00000277
ESP=00002000 EBP=00000000 ESI=0000028B EDI=0000EA32
ES=0287 CS=01D7 SS=028F DS=0287 FS=0000 GS=0000
NV UP EI PL ZR NA PO CY VM=0 IOPL=3
01D7:0000000C 9AA4395702 call 0257:39A4




10. At this point the game started up, but then dropped to the debugger with an "Unsupported SB DSP command FF!". As I mentioned in the DS2x86 blog post, the game seems to have a bug where it attempts to send the DMA length as parameters to SB DSP command 0x1C (start 8-bit autoinit DMA). Since this command does not actually take any parameters, those extra bytes are interpreted as additional (undocumented) commands.
0417:25DE cmp word [22ED],0200
jbe 25F8 Jump if SoundBlaster DSP version <= 2.00
mov al,1C
call 22F5 Send DSP command 0x1C (start 8-bit autoinit DMA)
mov ax,[22F1] ax = 0x0800
dec ax ax = 0x07FF
call 22F5 Send DSP command 0xFF (undocumented!)
mov al,ah
call 22F5 Send DSP command 0x07 (undocumented!)
0417:25F7 ret

I had hacked these unsupported commands 0xFF and 0x07 to be silently ignored in my MIPS code, so I added a similar hack to the ARM code.


11. At this point the game started into graphics mode, but it kept crashing almost immediately with "Unhandled exception 0008 at 3042 0020" followed by differing ErrCodes. At times this happened even before the game went into graphics mode, but usually during the intro graphics. The game immediately dropped back to DOS at that point, and usually the whole rpix86 went unresponsive after a few key presses. To be sure I also tested Tyrian 2000, and it crashed similarly, so this was obviously caused by some bug in my code. This never happened in my MIPS emulators, so it looked like this was something specific to my ARM implementation.

I googled for that Exception 0008, and it seems to mean "Double fault", which occurs when the processor detects an exception while trying to invoke the handler for a prior exception. I never generate that exception in my code, so it looks like the Borland RTM generates that by itself when it encounters that situation. But how do I find out where and why that happens?

I assumed that RTM considers exceptions and hardware interrupts to be equivalent in that situation, so the most likely cause was that my hardware IRQ emulation interrupts an exception handler. I began looking at how I disable hardware interrupts when an exception occurs. The code seemed fine, the interrupts are disabled as soon as the exception handler starts. I added logging for every situation where the Disable Interrupts flag changes value, but that did not seem to show anything out of the ordinary. Finally I added also logging for whenever an interrupt or exception handler starts, and that produced this interesting code snippet:
0277:000001C0 EAX=00003F60 EBX=05AF0005 ECX=00000060 EDX=00000000 ESP=000016DA Enable interrupts
EBP=000016E4 ESI=00001A0B EDI=00001B3E DS=05FF ES=028F SS=028F 1 1 = interrupts are now enabled
FB9C sti
0020:00000C7A EAX=00000006 EBX=05AF05FF ECX=00000060 EDX=00000000 ESP=00000FEC GPF Exception Handler starts
EBP=000016AA ESI=00001A0B EDI=00001B3E DS=00BF ES=028F SS=0018 0 0 = interrupts are now disabled
558B push bp
0020:00000B52 EAX=00000006 EBX=05AF05FF ECX=00000060 EDX=00000000 ESP=00000FE6 Timer IRQ Handler starts
EBP=000016AA ESI=00001A0B EDI=00001B3E DS=00BF ES=028F SS=0018 0 0 = interrupts are now disabled
558B push bp

It looked like the GPF Exception handler had just started and turned the interrupts off, but then immediately the timer interrupt handler started, even though the interrupts should have been disabled!

I looked into my code that disables the interrupts, and found a potential race condition. If an interrupt happened while the exception was being generated, it was possible for the interrupt code to mark that the next opcode to run needs to be the interrupt handler. The exception handler did not reset that mark when it turned the interrupts off, only the hardware interrupt handler handled this situation correctly. I made the exception handler behave similarly to how the hardware interrupt handler behaves, and after that change the game no longer encountered that Exception 0008 problem.


12. The next step was to add code to read data from file directly to Mode-X VRAM memory. This reading from file directly to graphics RAM is only done in two games that I have encountered, Heimdall which uses this in the EGA graphics mode, and Jazz Jackrabbit in Mode-X. The C call for this routine already existed but was commented out, as I had not implemented the actuall handler in the ASM code yet. Without this code the Jazz character in the 3D intro game did not have any textures and was black, as in this screen copy:

Adding the code was pretty simple, as I had commented the MIPS code pretty well in my zerox86 source code. It took only a few minutes to add the same code to the ARM Mode-X ASM code.


13. My zerox86 blog post was also very useful in implementing the enhancements for the few Mode-X opcode handlers that Jazz Jackrabbit uses in the game menu selector drawing. These images from my zerox86 blog show the problem on the left and the fixed code on the right:


14. I had spent considerable time debugging the Jazz audio problems in zerox86, so I added the same code enhancements to my ARM SoundBlaster emulation code. Thus Jazz Jackrabbit sounded pretty good even the first time I ran it.


Okay, now Jazz Jackrabbit was running fine, so I switched back to checking Tyrian 2000. It too seemed to run otherwise fine, but it did not play any audio. I will need to look into this a bit further.

I have not yet had time to test the new rpix86 properly, so I did not want to release the new version today. However, if nothing major happens, I should be able to release version 0.18 next Sunday! I am still checking my todo-list for additional improvements I might have time to implement, but at least Jazz Jackrabbit will run in the next release. Thanks again for your interest in my emulators!

http://rpix86.patrickaalto.com/rblog.html