burrito
June 15th, 2007, 19:15
I was reading the comments from StrmnNrmn's latest blog update (https://www.blogger.com/comment.g?blogID=13094505&postID=105300388387063087) (June 14), and saw that StrmnNrmn answered many questions about Daedalus.
Do you know whether it's actually self-modifying (as in, selectivelyish), reading a new routine from ROM (as in a loop from $b0000000), or a PI DMA?
How are you handling PI DMA at the moment when it comes to code fragments?
It took a short while to figure out exactly how it was modifying the code. I was worried that it was modifying the code via the cpu (i.e. manually updating instructions) but it turned out to be a DMA transfer. I'll post again with a bit more details about how I'm handling this.
I have been looking at ways to verify dynarec execution against an emulator, for similar reasons (but I'm just starting out in mine), so this is very helpful! :)
Have you considered running both the emulator and dynarec in a semi-lockstep mode? I am going to try this when I get far enough into getting my dynarec working, as I'm not too worried about performance (yet... it's more of a learning experience and for some researchy ideas I have) and this should avoid the need to have a temporary log and the disk access. Of course it will be a lot slower, due to having to run two complete emulations at once, but this may prove quicker than going via disk?
Glad you found the post useful :)
I did consider the lockstep method (I believe Zilmar was using this on pj64) but I think the aproach I use in Daedalus is much simpler to get working. The lockstep approach would cause a few issues for the psp as you'd need to have enough RAM for two copies of the emulated RAM etc.
It probably would be quicker than going via disk though, and less hassle to get everything set up.
Self-modifying code is perhaps the most annoying thing to deal with in a dynarec (that is, if you do in fact have to worry about it beyond cache invalidation). A good number of GBA games are very unfriendly in this regard. Actually, if it isn't checked for in general then a large number of games will fail outright.
Of course, to detect it for the general case you have to add code to either check for it or change state somewhere after every write to RAM. In your case, since this happens relatively infrequently, you can probably manage to special case it only for certain writes in certain games.
If you look at Project64's INI file you can see that there are several self-modifying code handling models (many of which probably don't really apply to PSP as it doesn't have an MMU). This alone indicates that it's a real problem for a number of N64 games, although perhaps not the most popular ones.
One thing I do recommend, if you're not already doing this, is to have separate translation caches for code that is in RAM and ROM (and not directly linking fragments between the two). That way you don't need to flush all of the generated code upon an icache invalidation or other detected modification, rather just the code in RAM. You can also go a step further and split things into dynamic and static regions to track modifications and try to minimize flushing overhead. This is only a concern if the game modifies code heavily - if any N64 games are like some GBA games then there will be ones that modify code many times a frame.
I talked about this issue in my blog, if you're interested: http://gpsp-dev.blogspot.com/
Hiya. Most n64 games seem to be pretty well behaved (they generally invalidate caches where required) which is why I've not spotted too many problems in the past.
They don't generally seem to modify code very frequently either - usually just on specific events (e.g. transfer from menus to game etc) rather than on a frame-by-frame basis.
Separate caches is a good idea though, I'll investigate that this weekend.
In a word: whoa!
Great job StrmnNrmn. Here's a question for ya: can the SSB rom be 'patched' to work correctly? Maybe not the file itself, but during the loading you have game-specific patches that Daedalus adds on the fly?
Also, on an unrelated note:
Is it possible to implement some kind of down-sampling of the N64's textures in order to save ram? The PSP's screen is tiny (large by handheld standards sure, but still smaller than a TV) and I am wondering if people would notice or care if the textures were reduced.
Keep up the great work man. I gotta say, you're regular posts are motivating. I have a wierd urge to try an Atari Jaquar PSP emulator with JIT dynarec! (okay... it passed)
In the past I've gone to the length of scanning the rom for recognisable library functions and patching them (mostly just to help debug what's going on with the rom, but also for optimising a few things like memcpy() etc). There's no reason why I couldn't use the same technique to patch game-specific code which is known to cause problems, but I think this would be pretty labour intensive.
In the end my solution was just to check each DMA transfer directly for an overlap with the fragment cache. This solution should work with all roms that modify code though ROM->RAM DMA, but still won't solve the problem of roms modifying individual instructions directly :(
The N64's textures tend to be pretty tiny anyway - it only had 4KiB of texture memory so 16x16 and 32x32 are pretty typical texture sizes. The ones that I have the most problems with are textures used for backgrounds in menus. Quite often these are really wide and thin - MarioKart's menus use lots of 320x3 textures for instance, which I have to round up to 512x4 for the psp. Down-sampling the textures might help a bit here - I'd have to experiment to see how it looked.
Amusingly I used to have code to support this in the PC build - I can't remember the details, but it was a compatibility fix for certain PowerVR cards which were limited to 256x256 textures IIRC :)
StrmnNrmn so you know I will be setting us a poll at DCEMU as you suggested for which titles are most wanted to be improved on for R13. I'm just taking the liberty in doing this for you so you can concentrate on Daedalus, I know you have enough on your plate as it is so. Anyway good work and I always enjoy reading your posts, keep it up.
Thanks - I look forward to seeing the results :)
Do you know whether it's actually self-modifying (as in, selectivelyish), reading a new routine from ROM (as in a loop from $b0000000), or a PI DMA?
How are you handling PI DMA at the moment when it comes to code fragments?
It took a short while to figure out exactly how it was modifying the code. I was worried that it was modifying the code via the cpu (i.e. manually updating instructions) but it turned out to be a DMA transfer. I'll post again with a bit more details about how I'm handling this.
I have been looking at ways to verify dynarec execution against an emulator, for similar reasons (but I'm just starting out in mine), so this is very helpful! :)
Have you considered running both the emulator and dynarec in a semi-lockstep mode? I am going to try this when I get far enough into getting my dynarec working, as I'm not too worried about performance (yet... it's more of a learning experience and for some researchy ideas I have) and this should avoid the need to have a temporary log and the disk access. Of course it will be a lot slower, due to having to run two complete emulations at once, but this may prove quicker than going via disk?
Glad you found the post useful :)
I did consider the lockstep method (I believe Zilmar was using this on pj64) but I think the aproach I use in Daedalus is much simpler to get working. The lockstep approach would cause a few issues for the psp as you'd need to have enough RAM for two copies of the emulated RAM etc.
It probably would be quicker than going via disk though, and less hassle to get everything set up.
Self-modifying code is perhaps the most annoying thing to deal with in a dynarec (that is, if you do in fact have to worry about it beyond cache invalidation). A good number of GBA games are very unfriendly in this regard. Actually, if it isn't checked for in general then a large number of games will fail outright.
Of course, to detect it for the general case you have to add code to either check for it or change state somewhere after every write to RAM. In your case, since this happens relatively infrequently, you can probably manage to special case it only for certain writes in certain games.
If you look at Project64's INI file you can see that there are several self-modifying code handling models (many of which probably don't really apply to PSP as it doesn't have an MMU). This alone indicates that it's a real problem for a number of N64 games, although perhaps not the most popular ones.
One thing I do recommend, if you're not already doing this, is to have separate translation caches for code that is in RAM and ROM (and not directly linking fragments between the two). That way you don't need to flush all of the generated code upon an icache invalidation or other detected modification, rather just the code in RAM. You can also go a step further and split things into dynamic and static regions to track modifications and try to minimize flushing overhead. This is only a concern if the game modifies code heavily - if any N64 games are like some GBA games then there will be ones that modify code many times a frame.
I talked about this issue in my blog, if you're interested: http://gpsp-dev.blogspot.com/
Hiya. Most n64 games seem to be pretty well behaved (they generally invalidate caches where required) which is why I've not spotted too many problems in the past.
They don't generally seem to modify code very frequently either - usually just on specific events (e.g. transfer from menus to game etc) rather than on a frame-by-frame basis.
Separate caches is a good idea though, I'll investigate that this weekend.
In a word: whoa!
Great job StrmnNrmn. Here's a question for ya: can the SSB rom be 'patched' to work correctly? Maybe not the file itself, but during the loading you have game-specific patches that Daedalus adds on the fly?
Also, on an unrelated note:
Is it possible to implement some kind of down-sampling of the N64's textures in order to save ram? The PSP's screen is tiny (large by handheld standards sure, but still smaller than a TV) and I am wondering if people would notice or care if the textures were reduced.
Keep up the great work man. I gotta say, you're regular posts are motivating. I have a wierd urge to try an Atari Jaquar PSP emulator with JIT dynarec! (okay... it passed)
In the past I've gone to the length of scanning the rom for recognisable library functions and patching them (mostly just to help debug what's going on with the rom, but also for optimising a few things like memcpy() etc). There's no reason why I couldn't use the same technique to patch game-specific code which is known to cause problems, but I think this would be pretty labour intensive.
In the end my solution was just to check each DMA transfer directly for an overlap with the fragment cache. This solution should work with all roms that modify code though ROM->RAM DMA, but still won't solve the problem of roms modifying individual instructions directly :(
The N64's textures tend to be pretty tiny anyway - it only had 4KiB of texture memory so 16x16 and 32x32 are pretty typical texture sizes. The ones that I have the most problems with are textures used for backgrounds in menus. Quite often these are really wide and thin - MarioKart's menus use lots of 320x3 textures for instance, which I have to round up to 512x4 for the psp. Down-sampling the textures might help a bit here - I'd have to experiment to see how it looked.
Amusingly I used to have code to support this in the PC build - I can't remember the details, but it was a compatibility fix for certain PowerVR cards which were limited to 256x256 textures IIRC :)
StrmnNrmn so you know I will be setting us a poll at DCEMU as you suggested for which titles are most wanted to be improved on for R13. I'm just taking the liberty in doing this for you so you can concentrate on Daedalus, I know you have enough on your plate as it is so. Anyway good work and I always enjoy reading your posts, keep it up.
Thanks - I look forward to seeing the results :)