During the past week I have continued trying to get Windows 3.11 running in DS2x86. This work has progressed slowly but steadily. I keep running into new problems, all of which seem to take a day or two of debugging and studying before I understand the cause of the problem and can implement a fix for it.
The problem of dropping back to DOS that I had last Sunday was caused by my RETF opcode not adjusting the CS register properly when in VM mode. It was an easy fix, and the next problem was also reasonably easy to fix, I needed to add support for INT 2F AX=1603 (MS Windows/386 - GET INSTANCE DATA) and INT 2F AX=1607 (MS Windows - DOSMGR VIRTUAL DEVICE API) software interrupts. I looked at DOSBox to see what it returns for those interrupts, and created similar handling into DS2x86.
The next problem was a bit more difficult, Windows 3.11 again dropped back to DOS, but this time with a message "Insufficient memory to run Windows". Windows 3.11 should run fine in 16MB of RAM, so I knew it was not a question of having too little RAM, but some problem with the memory managers or such. After some debugging I noticed that Windows calls the DOS interrupt INT 21 AH=52 (DOS 2+ internal - SYSVARS - GET LIST OF LISTS), and then preforms all sorts of checks of the list pointers. In DS2x86 I had left many of the list pointers (like pointer to directory array, pointer to next device driver after NUL device, pointer to FCB file pointers, and the next pointer of the master file table) as 0xFFFF:0xFFFF, which means the list end. However, Windows assumed all of those to be valid pointers, and attempted to make some size checks and memory allocations using those pointer values, which then of course failed. I needed to create new fake data to the DOS memory segments for many new data structures and have those pointers point to these new structures before I was able to make Windows continue past the checks. I also added proper CON device header, which had been missing until now. This will make the DOS device list look a bit more realistic, so it might be useful for other software besides Windows as well.
After fixing that problem, Windows next dropped back to DOS with a message "Unsupported expanded memory driver". This problem took several days to find and fix, as I was first looking for the problem in the code that gets executed after the previous fix I made. It took me a while to realize that Windows stores the EMM driver features almost as soon as it starts, but only checks these stored values much later in the bootup process. In the end it turned out that this problem was caused by my not supporting the DOS interrupt INT 21 AX=4402 (Memory Managers - GET EMM IMPORT STRUCTURE ADDRESS). I still don't know exactly what Windows checks from within this structure, I just used the same values as what DOSBox puts into that structure. DOSBox creates the structure into the VGA BIOS segment 0xC000, so I assume the contents do not need to actually reflect the current EMS manager state.
The current problem with Windows 3.11 is that it causes some unsupported I/O port accesses (it writes to VGA port 0x3CA, which is a read-only port). This port access does not happen in DOSBox, so somewhere before this happens the execution path has differed between DS2x86 and DOSBox. After a few of these port writes Windows 3.11 in DS2x86 shows a BSOD with a message: "Invalid VxD dynamic link call to device number 0001, service 2484. Your Windows configuration is invalid. Run the Windows Setup program again to correct this problem." I strongly suspect this is caused by the VGA driver, but I have not yet managed to find where the difference between DOSBox and DS2x86 execution begins. After I locate that, I can then check what VGA features my emulation is still missing.
So, all in all, slow progress, and I have no idea yet how many problems are still ahead before I get Windows 3.11 to actually start up in DS2x86. After that it might still take a while before any actual Windows programs will start running properly.