wraggster
August 9th, 2011, 12:11
News via http://dsx86.patrickaalto.com/DSblog.html
My summer vacation ended by the previous weekend, so this week I have not had all that much time to work on DS2x86. I spent an hour or so every weekday debugging some problem games, though. I first decided to finally try and find the keyboard problem in Frontier, where the keys seem to get stuck when in the star map, so that you can't properly move around in the map. I debugged the keyboard interrupt routine that the game uses, and found out that it uses separate maps for each normal and each extended key that is either down or up. When the problem occurred, I noticed that the non-extended cursor key was down in the map, while the extended cursor key was up. After checking my keyboard emulation routines, I realized that my key repeat feature always repeats only the non-extended versions of the keys!
What this problem meant for Frontier, was that my keyboard emulation routine first sent an enhanced cursor key down code, and if you kept the key pressed, it began sending the non-enhanced cursor key down codes. When you then lifted your finger from the key, my routine sent the enhanced cursor key up code, but never an up code for the non-enhanced key. Thus, the non-enhanced key stayed down, and Frontier did not actually make a difference between enhanced and non-enhanced cursor keys when looking for keys that were down. This was relatively simple to fix, by always repeating the same key, be it enhanced or not, that was pressed. After this change the Frontier star map began to scroll properly when using the cursor keys, but there is still a problem when using mouse. I need to still work on the mouse problem, I'll see if I can fix it also before releasing the next version.
Next I looked into the weird 360x480 graphics mode problem in Albion. This graphics mode turned out to actually be the 360x240 mode (same as in Settlers, for example). I found out that there are actually two ways to make the VGA card display 240 instead of 480 rows, and I had only checked one of those ways in DS2x86. So, I added a check for the other method as well, and Albion began to look correct.
I had heard reports that the Tap functionality in the TouchPad Mouse (TPM) emulation is broken in DS2x86. I used Albion to test this, but did not find anything wrong with the functionality. There are some difficulties in the key and touchpad reading in general in DS2x86, but that is mostly due to the DSTwo SDK. Besides that, the TPMTap=TRUE setting in the INI file seems to work fine.
I did however notice that the TPM handling did not work very well in Albion, it looked like the horizontal movement of the mouse pointer was about twice that of the distance I moved the stylus, and the vertical movement was about four times that. So, I decided to finally enhance the TPM functionality by adding X and Y scaling factors to the configuration. So, in the next version you can use TPMXScale and TPMYScale game-specific INI file keys to select the mouse movement scaling factor when using touchpad mouse. For example, in Albion you could use this:
[MAIN] ; For Albion!TPMXScale=0.50TPMYScale=0.25The scaling factor should be a floating point number, where 1.00 is the normal default scaling. Note, though, that Albion is a bad example in a sense that it uses the mouse buttons to make the character walk, so using the D-Pad mouse might be easier. After those changes I started making some changes to the code that would allow me to support Paging (as in Virtual Memory) in the future. There are several games that want to turn paging on (including the Wing Commander Armada I tested last weekend), so I want to start working on this feature soon. The first change I made (which should not actually affect anything yet) was to change my memory mapping to use 4KB pages instead of 16KB pages. I had originally used 16KB pages, as that is the smallest granularity needed to support EMS memory. This was also enough to handle the special paging of the VGA memory (where the 64KB memory block at address 0xA0000 is actually accessing 256KB of VRAM). However, to support real paging each page should be 4KB in size. I had used a simple look-up table that had 1024 entries (to access all 16MB of emulated memory with 16KB pages), so I increased the table to 4096 entries so that each page is 4KB, and adjusted all the routines that access this table to handle the new page size correctly.
After I got the code running properly with the 4KB page size, I again spent some time thinking about how to add the actual paging functionality without sacrificing the performance of games that do not use paging. When paging is not in use, I can simply look up the physical memory address from my one look-up table (or Translation Lookaside Buffer (http://en.wikipedia.org/wiki/Translation_lookaside_buffer), TLB), which is reasonably fast. However, when paging is enabled, the physical memory lookup is much more complex, as the following image (from Wikipedia) illustrates:
The linear address (which without paging I can simply shift right 12 bit positions to get the mapping table index, and then add the mapping table value to the remaining bits to get the correct physical RAM address) is split into three fields, where the top 10 bits index the Page Directory table, the next 10 bits the Page Table, and the remaining 12 bits the byte within the page. The CPU register CR3 gives the starting physical address of the Page Directory table, which in turn gives the starting physical address of the Page Table used for the middle 10 bits of the linear address. Each of the page entries contain the 20 highest bits of the physical address, while the low 12 bits contain all sorts of bookkeeping bits, telling if the page has been accessed, is it present in memory, is it dirty (that is, it has been written into), etc. Quite a complex (read: slow) system.
The biggest problem I have been having with this is to figure out a way to make the memory access behave differently when paging is in use. Until now all the linear page lookup code has been inlined into all the opcode handlers, so there is no subroutine that I could change to work differently. And, since my primary goal was to not make the non-paging code any slower, adding a subroutine was out of the question.
The current solution I am working with, is a bit of a brute force method: If I increase my linear page mapping table to have 1048576 entries (to map the whole 4GB of virtual RAM), I can then precalculate all the needed physical page addresses and use the exact same code I use without paging to look up the physical address also when paging is in use! This however means that the linear page mapping table will take a whopping 4MB of RAM! I checked how much memory I currently use, and noticed that my code takes about 4MB, and the emulated RAM takes 16MB. Then there is the allocated EMS memory of 4MB, plus some stack area etc. So, slightly over 24MB of the 32MB of RAM is in use. Increasing the page table to 4MB will still leave a few megabytes free, so it should actually fit. And of course only a small part of the 4MB mapping table is actually accessed, even though it needs to contain the whole 4GB address range. The game can use whatever virtual addresses it wants, as the paging tables then map these to the existing RAM.
I am thinking of using a similar system to what DOSBox uses when paging is turned on: All the lookup-up table values are cleared, so that whenever a new page is accessed it is forced to go though an initialization routine. This initialization routine then checks and fills the bookkeeping bits of the page, and in my case it can also fill the linear look-up table with the proper physical address. I already have code in (almost) all the look-up table access routines to jump to a different handler when EGA or Mode-X RAM is accessed. I just need to add a new case into this opcode-specific jump table so that in the future it selects between RAM, EGA, Mode-X and unitialized page handling routines. This should not make the code any slower, but it does make it somewhat bigger.
There are currently around 500 different locations where this inline code is used, and I'll need to generate handling code for the uninitialized page situation to all of those locations. This will obviously not be done by the next version yet, but this is what I plan to work on from now on. I'll also try to fix some bugs in the code and release new versions, but there will probably not be any other new features until I get this paging system done.
My summer vacation ended by the previous weekend, so this week I have not had all that much time to work on DS2x86. I spent an hour or so every weekday debugging some problem games, though. I first decided to finally try and find the keyboard problem in Frontier, where the keys seem to get stuck when in the star map, so that you can't properly move around in the map. I debugged the keyboard interrupt routine that the game uses, and found out that it uses separate maps for each normal and each extended key that is either down or up. When the problem occurred, I noticed that the non-extended cursor key was down in the map, while the extended cursor key was up. After checking my keyboard emulation routines, I realized that my key repeat feature always repeats only the non-extended versions of the keys!
What this problem meant for Frontier, was that my keyboard emulation routine first sent an enhanced cursor key down code, and if you kept the key pressed, it began sending the non-enhanced cursor key down codes. When you then lifted your finger from the key, my routine sent the enhanced cursor key up code, but never an up code for the non-enhanced key. Thus, the non-enhanced key stayed down, and Frontier did not actually make a difference between enhanced and non-enhanced cursor keys when looking for keys that were down. This was relatively simple to fix, by always repeating the same key, be it enhanced or not, that was pressed. After this change the Frontier star map began to scroll properly when using the cursor keys, but there is still a problem when using mouse. I need to still work on the mouse problem, I'll see if I can fix it also before releasing the next version.
Next I looked into the weird 360x480 graphics mode problem in Albion. This graphics mode turned out to actually be the 360x240 mode (same as in Settlers, for example). I found out that there are actually two ways to make the VGA card display 240 instead of 480 rows, and I had only checked one of those ways in DS2x86. So, I added a check for the other method as well, and Albion began to look correct.
I had heard reports that the Tap functionality in the TouchPad Mouse (TPM) emulation is broken in DS2x86. I used Albion to test this, but did not find anything wrong with the functionality. There are some difficulties in the key and touchpad reading in general in DS2x86, but that is mostly due to the DSTwo SDK. Besides that, the TPMTap=TRUE setting in the INI file seems to work fine.
I did however notice that the TPM handling did not work very well in Albion, it looked like the horizontal movement of the mouse pointer was about twice that of the distance I moved the stylus, and the vertical movement was about four times that. So, I decided to finally enhance the TPM functionality by adding X and Y scaling factors to the configuration. So, in the next version you can use TPMXScale and TPMYScale game-specific INI file keys to select the mouse movement scaling factor when using touchpad mouse. For example, in Albion you could use this:
[MAIN] ; For Albion!TPMXScale=0.50TPMYScale=0.25The scaling factor should be a floating point number, where 1.00 is the normal default scaling. Note, though, that Albion is a bad example in a sense that it uses the mouse buttons to make the character walk, so using the D-Pad mouse might be easier. After those changes I started making some changes to the code that would allow me to support Paging (as in Virtual Memory) in the future. There are several games that want to turn paging on (including the Wing Commander Armada I tested last weekend), so I want to start working on this feature soon. The first change I made (which should not actually affect anything yet) was to change my memory mapping to use 4KB pages instead of 16KB pages. I had originally used 16KB pages, as that is the smallest granularity needed to support EMS memory. This was also enough to handle the special paging of the VGA memory (where the 64KB memory block at address 0xA0000 is actually accessing 256KB of VRAM). However, to support real paging each page should be 4KB in size. I had used a simple look-up table that had 1024 entries (to access all 16MB of emulated memory with 16KB pages), so I increased the table to 4096 entries so that each page is 4KB, and adjusted all the routines that access this table to handle the new page size correctly.
After I got the code running properly with the 4KB page size, I again spent some time thinking about how to add the actual paging functionality without sacrificing the performance of games that do not use paging. When paging is not in use, I can simply look up the physical memory address from my one look-up table (or Translation Lookaside Buffer (http://en.wikipedia.org/wiki/Translation_lookaside_buffer), TLB), which is reasonably fast. However, when paging is enabled, the physical memory lookup is much more complex, as the following image (from Wikipedia) illustrates:
The linear address (which without paging I can simply shift right 12 bit positions to get the mapping table index, and then add the mapping table value to the remaining bits to get the correct physical RAM address) is split into three fields, where the top 10 bits index the Page Directory table, the next 10 bits the Page Table, and the remaining 12 bits the byte within the page. The CPU register CR3 gives the starting physical address of the Page Directory table, which in turn gives the starting physical address of the Page Table used for the middle 10 bits of the linear address. Each of the page entries contain the 20 highest bits of the physical address, while the low 12 bits contain all sorts of bookkeeping bits, telling if the page has been accessed, is it present in memory, is it dirty (that is, it has been written into), etc. Quite a complex (read: slow) system.
The biggest problem I have been having with this is to figure out a way to make the memory access behave differently when paging is in use. Until now all the linear page lookup code has been inlined into all the opcode handlers, so there is no subroutine that I could change to work differently. And, since my primary goal was to not make the non-paging code any slower, adding a subroutine was out of the question.
The current solution I am working with, is a bit of a brute force method: If I increase my linear page mapping table to have 1048576 entries (to map the whole 4GB of virtual RAM), I can then precalculate all the needed physical page addresses and use the exact same code I use without paging to look up the physical address also when paging is in use! This however means that the linear page mapping table will take a whopping 4MB of RAM! I checked how much memory I currently use, and noticed that my code takes about 4MB, and the emulated RAM takes 16MB. Then there is the allocated EMS memory of 4MB, plus some stack area etc. So, slightly over 24MB of the 32MB of RAM is in use. Increasing the page table to 4MB will still leave a few megabytes free, so it should actually fit. And of course only a small part of the 4MB mapping table is actually accessed, even though it needs to contain the whole 4GB address range. The game can use whatever virtual addresses it wants, as the paging tables then map these to the existing RAM.
I am thinking of using a similar system to what DOSBox uses when paging is turned on: All the lookup-up table values are cleared, so that whenever a new page is accessed it is forced to go though an initialization routine. This initialization routine then checks and fills the bookkeeping bits of the page, and in my case it can also fill the linear look-up table with the proper physical address. I already have code in (almost) all the look-up table access routines to jump to a different handler when EGA or Mode-X RAM is accessed. I just need to add a new case into this opcode-specific jump table so that in the future it selects between RAM, EGA, Mode-X and unitialized page handling routines. This should not make the code any slower, but it does make it somewhat bigger.
There are currently around 500 different locations where this inline code is used, and I'll need to generate handling code for the uninitialized page situation to all of those locations. This will obviously not be done by the next version yet, but this is what I plan to work on from now on. I'll also try to fix some bugs in the code and release new versions, but there will probably not be any other new features until I get this paging system done.