DSx86 - Blog

June 28th, 2014 - New subdomain for pax86

Just a short blog post, letting you know that I have created a new subdomain pax86.patrickaalto.com for my pax86 x86 emulation core, which I have commercially licensed to a few parties. I also moved the blog posts that relate more to pax86 than to DSx86 from this blog to my new blog for pax86. Check out the new pax86 blog if you are interested in it!

Jan 6th, 2013 - Work started on ax86 (Android x86 emulator)!

Happy new year 2013! As I hinted at in my previous blog post, I have now started work on an Android port of DSx86. It has a working name of ax86, and thus I created a subdomain ax86.patrickaalto.com for it. I will be writing blog posts about the development of ax86 on those pages. This will probably be the last Dsx86 blog post, at least for now. I released the first public beta version of DSx86 on the 29th of December 2009, so just a few days over 3 years ago. Big thanks to all of you who have been following my blog and progress of DSx86, and I welcome you all to the ax86.patrickaalto.com site!

Nov 4th, 2012 - DSx86 hiatus continues

This is just a short update about my VHS digitizing project. I am currently digitizing the eight VHS tape (of about 30 tapes I plan to digitize), so it will still take a couple of months before I have digitized everything. With the first ten or so tapes I have a pretty good idea about what I want to save and what not, but the remaining tapes I have not yet looked at properly, so it is possible there is a lot of stuff I can simply skip. But even so, I doubt I will get back to my programming hobbies before the end of the year.

I do have some DSx86-related news waiting for a proper time (which is not up to me) to announce, so I might have some interesting news to write about even before I have finished my VHS digitizing project. Also, I have not yet made up my mind about whether to simply continue with Nintendo DS programming, or try porting DSx86 to some other platforms, but it looks like I still have a couple of months to think about this. :-)

Sep 23rd, 2012 - DSx86 hiatus continues

I am still on my coding break. I decided to write this blog post just to quickly let you know what it is I am doing currently, instead of DSx86. I am digitizing my large collection of old VHS tapes. Many of them of course are quite irrelevant as they contain movies that I have since purchased in DVD or Blu-Ray format, but there is a lot of material that I would like to digitize properly.

I got my first VCR (a Sharp VC-488) back in 1985, and have been recording stuff since that time. My second VCR was a JVC HR-D530EH, which I got in August 1988, to be able to copy stuff from tape to tape. Thus many of my tapes are actually second-generation recordings, sadly. After those VCRs I have had a couple of other machines (JVC HR-D980 purchased in April 1992, Panasonic NV HD 670 in April 1998) up to the latest Super-VHS JVC HR-S8600EU that I bought in May 2001.

I got my first Digi-TV card in May 2004, so after that time I pretty much stopped recording new VHS tapes from TV programs. Then in the summer of 2005 a thunderstorm killed my S-VHS VCR, and I decided not to have it repaired. I still had the Panasonic VCR and could use it to view the great majority of my VHS tapes. I only had a couple of S-VHS tapes and thought that I won't miss those that much. The time of VHS tapes was pretty much over and all the TV channels had switched over to digital.

Then in 2006 I borrowed a friend's "one-click VHS-to-DVD" system and digitized the most important tapes using my Panasonic VCR, but sadly the quality left much to be desired. I wanted to quickly get the tapes digitized and return the device, so I did not spend much time with the digitizing. After that I gave the Panasonic VCR away, so I did not have any working VCR any more (or so I thought).

Then this summer, inspired partly by a friend of mine who inherited a lot of VHS tapes from his father and is in the process of digitizing them, I got interested in an idea of re-digitizing my old VHS tapes, hopefully this time with a better quality. After visiting DigitalFAQ I found out that JVC HR-S9600 (which is a sister model of my thunderstricken HR-S8600) is one of the best VCR machines to use for digitizing old VHS tapes. So, I called a video repair shop and asked whether they still repair old VCRs, and they agreed to take a look at it.

In the mean time I found my old VCRs (both the Sharp VC-488 and the JVC HR-D530EH) from my attic, where they had both spent more than 15 years, exposed to temperatures of -30 degress Celsius during winters. I decided to see whether the 24 years old JVC VCR would work at all (or whether it would simply blow a fuse or something when connected to mains electricity), and to my surprise it still worked fine! I originally moved it to the attic because it began eating tapes when fast-forwarding with picture, but other than that problem it was still working fine at that time.

The best thing about this old VCR still working is that between 1990 and 1992 I used this machine to record a lot of music (mainly from borrowed CDs) in LongPlay format (6 hours per E-180 VHS cassette). I have not been able to digitize them properly because the VCR head noise/distortion has been quite noticeable when playing the tapes in any other VCR machine. However, now that I can use the same machine as the one the tapes were recorded with, there is no noticeable distortion!

The video repair shop did not manage to fix my dead JVC HR-S8600 S-VHS VCR, but I just found a used one on eBay and purchased that. My plan is to possibly combine the electrical components from the eBay machine with the mechanical parts of my own little-used machine, especially if the machine I purchased turns out to be very worn-out. I had only used my machine for about 3 years before it died, so the mechanical parts should be in good condition.

Anyways, this is what I am currently doing, digitizing old VHS tapes, some of them containing only music and some also video. It is very time-consuming work, trying to come up with optimimal settings and filters, and fixing various dropouts and such from the digitized audio. No time to work on DSx86 while I am doing this. :-)

Sep 2nd, 2012 - DSx86 hiatus continues

Nothing much to report, I am still on my "coding break" and focusing on my other hobbies that I have been neglecting during the past three years I have been working on DSx86. I do have some DSx86 -related news, though. Sverx, who already optimized my smooth screen scaling algorithms once before, came up with even faster method of handling this scaling. I will update the DSx86 screen scaling algorithm to use his latest invention in the next DSx86 version (when I get back to coding :-). Meanwhile you can read more details about his scaling method in his blog. Thanks again Sverx!

Aug 19th, 2012 - DS2x86 Status

I have not done anything to DS2x86 during the past week. I am rather busy at work again, and I have to admit that I am starting to lose interest in working on DS2x86. It seems that I am currently pretty much the only homebrew developer programming for the SuperCard DSTwo environment. As other programmers have moved on to other platforms, also the users either have already moved on or will move on soon. Looking at the recent posts on the Supercard SDK forum it is pretty evident that without BassAceGold and myself there would not have been much happening in that scene for the past several months.

The whole NDS homebrew scene is also long past it's peak, I think it was actually on the decline already 3 years ago when I began working on DSx86. Hard to imagine I have already worked on it for more than three years.. I suppose smart phones have largely replaced the dedicated handheld gaming devices nowadays. That is somewhat sad as architecturally Nintendo DS is a pretty neat device to program for. I do want to continue working on DSx86, but I am starting to think that perhaps the Nintendo DS hardware does not offer any more the kinds of interesting challenges and learning experiences that keep me interested in my hobby projects.

It might be interesting to look into Android programming, and possibly even port DSx86 to Android smartphones at some point. The potential user base would be huge, and my x86 ARM assembler code would be simple to port to ARM-based Android phones. I'm not sure how useful an x86 emulator would be on a smart phone that does not have a keyboard, and also any reasonably new smart phone has enough power to run a port of DOSBox, so perhaps my porting DSx86 to that platform would be rather redundant.

But in any case, I feel that after three years of working on DSx86 and DS2x86 I need to take a break. I have been toying with the idea of attempting to port my old LineWars II game to Android environment. After all, I began my Nintendo DS homebrew coding "career" by first porting LineWars II to it, and only after that I started on DSx86. Perhaps it would be a good idea to move into Android programming using a similar approach. If I indeed want to start coding for Android, which I have not decided yet. I'll take a break from hobby programming, at least while I am very busy at work, and then see what would be my next hobby programming project. Or whether I find some new interesting things to do for DSx86, as that is certainly also possible.

Aug 12th, 2012 - Slow week for DSx86

Again a week with nothing much happening on the DSx86 front. I am rather busy at work after my vacation, and after getting home from work, watching the Olympic games interests me more than coding DS2x86. I did however do some tests as to the changes I would need to do to make Wing Commander Armada run. It looks like the biggest problem with it is that it uses VCPI (Virtual Control Program Interface), which is a part of EMM (Expanded memory Manager) features. My current EMS support in DS2x86 is missing the VCPI features, so currently Wing Commander Armada simply drops back to DOS with a message "EMS driver is not VCPI compliant".

To make my inbuilt EMS driver support VCPI, I will need to change the way I currently handle the EMS and XMS memory. Currently I have reserved a separate memory area for EMS memory (the memory that for example 4DOS uses for swapping, accessed using the EMS page frame at 0xE000 segment) and for XMS memory (the extended memory above 1MB). Almost the first thing that Wing Commander Armada does with the VCPI features is to call function 0xDE06 (Get Physical Address of Page) for the EMS page frame 0xE000. This is currently a problem, as my EMS memory (being completely separated from the XMS memory area) does not have any "physical address" that I could return to the game!

Another problem with my inbuilt EMS/XMS manager is that it is very stupid in the way it allocates memory. Currently I only have a value telling how much of the memory has been allocated, so if any game allocates many memory blocks and then frees some other blocks besides that last one, these don't actually get freed. I want to fix this problem as well when combinining the EMS and XMS memory handling to use the same memory area. This is quite a big rewrite of several routines, and it also changes many low-level memory access routines, so I will need to do this very carefully not to break anything. I hope to start working on this during the next week, when there are no more Olympics to distract me. :-)

It would also be interesting to experiment using an actual EMM386 or similar driver instead of building this functionality inside DS2x86, but I fear this might be even more work. This would easily escalate to my having to support full config.sys and autoexec.bat handling, and that is quite a big change. At some point I might try to do that, but currently I feel this is a bit too much work to start working on while I am busy at my daytime job.

Aug 5th, 2012 - DS2x86 Windows 3.11 progress

The last week was my first working week after my summer vacation, and I did not have much time to work on DS2x86 during the evenings. The extra project I worked on before my summer vacation also needed some additional work done, so that too decreased my free time. Thus I only got to work on DS2x86 this weekend, and there has not been much progress. I have managed to find the code that somehow sets up the page table with wrong values, and have been comparing the behaviour with that of DOSBox. The problem here is that the behaviour seems to differ quite a bit, and I'm not sure which differences are supposed to be there (as the memory organisation is slightly different between DOSBox and DS2x86) and which are symptoms of something going wrong. The problem is that it needs a huge amount of work to determine the cause of every single difference.

It is starting to feel like I would spend my time more productively trying to get some other software besides Windows 3.11 working in DS2x86. I will probably work on some games for a while before getting back to Windows 3.11 support. One game I would like to get working is Wing Commander Armada. It does need paging and is running the actual game in Virtual 86 mode, so it will need some of the same enhancements I have already done for Windows. I think I will spend the next couple of days trying to get it running. There are also other games that would need various fixes, and in any case it would be nice to be able to release a new version at some point!

July 29th, 2012 - My summer vacation ends

Sadly, today marks the last day of my summer vacation. The four weeks went by pretty fast again. During this week I have not worked much on DSx86 or DS2x86. I had planned some other things I wanted to do during my summer vacation, and of course I had not gotten around to those before it suddenly was the last vacation week and I really had to do them!

One interesting task I did was to replace the CMOS battery of my old Acer Travelmate 803 laptop. It had started to behave erratically with the system time, so I assumed that the CMOS battery was dead. Strangely the clock had not completely stopped working, instead after I set it to the correct time, it kept time for a little while (for a few hours), then suddenly jumped a couple of hours backwards and then stopped running.

I had purchased the laptop in May 2003, so it is over 9 years old. It just has been working so well that I have not had a need to replace it with a more modern machine. I have replaced the hard disk with a 32GB SSD disk, and I have also upgraded the RAM to 1.5GB, so the machine is reasonably fast (and what is more important to me, it is dead silent when running in Max Battery mode, regardless of CPU usage).

Anyways, I had found a blog post describing some important information about the CMOS battery, like the type (CR1220) and location (underneath the motherboard!) of it. Replacing it meant pretty much disassembling the whole thing, changing the battery, and then trying to put everything back together again. Surprisingly, no screws or other parts were left over after I had done this, and the machine even seems to work (and the clock keeps correct time)!

I was also given a heads up that NeoTeam is holding a Neo Coding Compo 2012, and it looks like also existing projects (like DSx86) are allowed to participate. Thus, I am thinking about possibly taking part in that competition. I would need to add a splash screen to DSx86, and I also would like to do some enhancements to it (I don't think the point of the competition is to simply add a splash screen and be done with it). I have not yet decided whether to take part, but I might.

Anyways, from now on I don't have all that much time to work on DSx86-related stuff, but I still try to continue working on the Windows 3.11 support for DS2x86. It would be fun to get that working, even though at the moment it seems very difficult and frustrating. But, the more difficult it is to achieve, the sweeter it feels when you finally get it working!

Thanks again for your interest in DSx86 (and for reading my blog!) :-)

July 22nd, 2012 - DS2x86 Windows 3.11 progress

Yet another week where I have been slowly trying to get Windows 3.11 to progress further. The progress just seems to get slower and slower, this time all the progress I have managed to get done fits within a few lines of ASM code! Here is the part of the code (in the KRNL386.EXE of Windows 3.11) I have been working on, with the problem locations numbered (1., 2. and 3.): 

8DC8:C357	push	cx
		push	es
		mov	ax,1687
		int	2F			DOS Protected-Mode Interface (DPMI) - INSTALLATION CHECK
		or	ax,ax
		jnz	C43C			Jump to error if DPMI not installed
		xor	bh,bh
		cmp	cl,03			CL = processor type (02h=80286, 03h=80386, 04h=80486)
		jb	C43C			Jump to error if CPU < 386
		mov	bl,04
		je	C373
		mov	bl,08
8DC8:C373	mov	[0324],bx		Save processor type flag to variable
		mov	[1148],di		ES:DI = DPMI mode-switch entry point
		mov	[114A],es
		pop	ax
		add	ax,0010
		mov	es,ax
		add	si,ax
		xor	ax,ax
		call	far word [1148]		1. 2. Call the DPMI mode switch entry point: Switch to protected mode, ring 3
009F:C38D	jc	C43C			Jump to error if mode switch failed
		mov	ax,cs
		and	al,07
		cmp	al,07
		jne	C411			Jump to error if code selector is not ring 3 LDT selector
		mov	bx,cs
		mov	ax,000A
		call	2B42 ($+67A1)		DPMI 0.9+ - CREATE ALIAS DESCRIPTOR
009F:C3A1	mov	[05B0],ax		Save returned alias descriptor selector for CS
		mov	bx,ds
		mov	ds,ax
		mov	[0030],bx		Save original DS selector to new alias data segment
		mov	ds,bx
		push	es
		push	si
		mov	ax,168A			
		mov	si,114C			DS:SI = "MS-DOS",0
		int	2F			3. DPMI 0.9+ - GET VENDOR-SPECIFIC API ENTRY POINT
009F:C3B8	cmp	al,8A
		je	C411			Jump to error "KERNEL: Inadequate DPMI Server" if call not supported
		...

Here are the problems I have been having listed, with the numbers corresponding to the source code above:

  1. The "system halted" problem I mentioned in the previous blog post happened within the call to the DPMI entry point. I finally found that the problem was caused by Windows 3.11 having the LDT (Local Descriptor Table) in virtual memory. When I some time ago fixed my IDT and GDT tables to support having their base address in virtual memory, I of course should have also fixed the LDT table at the same time, but for some reason I did not. So, what goes around comes around (and bites you!).
  2. After I implemented the LDT virtual memory support, I got a drop to debugger with an "IRET to lower priviledge!" error message. This was simply a sort of assertion in my protected mode IRET routine, as I had not yet encountered a program that would return from a higher priority level (ring 0) code to lower priority level (ring 3) code using an IRET operation. Windows 3.11 seems to do that, in the built-in DPMI mode change routine. Supporting that needed some coding into the IRET handler, and I also combined my 16-bit and 32-bit IRET handlers into a single routine while I did this. I have been wanting to do that for some time now, as both of those routines are practically the same, they just load either 16-bit or 32-bit values from the emulated x86 stack.
  3. The next (and current) problem is that Windows 3.11 exits with a message KERNEL: Inadequate DPMI Server. It took me a while to debug where this happens, but in the end I found that it is inside the call numbered 3. above. Windows manages to create an alias descriptor for CS (code segment), and after that it tries to get a vendor-specific DPMI entry point (in this case it wants to make sure it uses Windows' own DMPI code instead of 386Max or Borland 32RTM or some other possibly installed DPMI server). It gives a string "MS-DOS" as input to the DPMI call, but the call returns an error (call not supported). The DMPI server it calls is indeed Windows' own server, so at first I was pretty much at loss as to why it reports it does not support it's own server. After quite a bit of debugging I found out that even though KRNL386.EXE contains the "MS-DOS" string, and it gets loaded to RAM into a correct position, the DPMI server still does not find the string in RAM. For some so far unclear reason the virtual memory page mapping tables have an invalid address for the page where that RAM address should be. I have not yet been able to determine why the page tables change suddenly, as I have determined that initially the value is correct. This is the problem I am currently fighting with.
Rather discouraging seeing that I have only progressed about a dozen ASM opcodes within KRNL386.EXE during the whole week. But I guess I just need to keep debugging the code to see where the problem is, fix that, and then just continue to the next problem...

July 15th, 2012 - DS2x86 Windows 3.11 progress

Okay, I have finally got a little bit forward with the Windows 3.11 support. Still no end in sight for the changes I still need to do, but at least I have managed to get some progress done. I found out the reason for the invalid VxD dynamic link call. I found a list of the device numbers from a Microsoft Knowledge Base article, and the device number 1 means the Virtual Machine Manager. The service number 0x2484 (9348) is larger than the number of services available in VMM, so this is why the blue screen occurred. That had nothing to do with VGA registers. The reason why I suspected some VGA register problem was that I got unsupported I/O port calls that did not happen in DOSBox, but it turned out that those happened while Windows was abruptly switching back to text mode to display the blue screen error message! So the VGA register problem was a symptom, not a cause.

Anyways, after quite a bit of debugging I then found where the invalid service number call happened. Windows 3.11 uses INT 20 software interrupt to perform those dynamic VxD calls. The calls are coded so that the interrupt opcode CD20 is followed by first the service number (in two bytes) and then the device number (in two bytes). Here below are two screen copies from the debugger illustrating what the problem was in DS2x86. On the left is the problem situation, where the INT 20 opcode at offset 8028DFFD is followed by the service number. Here the service number happens to be split into two 4KB pages, with the low byte being at offset 8028DFFF and the high byte at 8028E000 (which is in a different physical memory page). As Windows uses virtual memory, these two pages may not be adjacent in the physical memory, but my movzx opcode (which Windows uses to read the service number and device number within the INT 20 handler) did not handle this situation. It simply calculated the physical start address of the 16-bit value (from the offset 8028DFFF) and then read two bytes from that address. This caused the high byte to be read from whatever page happened to physically follow the current page in RAM, and this page happened to have byte 0x24 in the first offset of the page. Reading the other parameter (the device number 0x0001) in turn worked correctly, as it calculated the physical address from offset 8028E001 and correctly read the value from there, as shown on the right hand debug screen copy.

 

By the way, the screen copies above display another interesting (or annoying, depending on whether you are attempting to debug it!) behaviour in Windows 3.11. In many cases Windows replaces this (slow) interrupt call with an indirect function call after it has been executed once. In other words, Windows seems to use a lot of self-modifying code! For example, the opcode at offset 8028DFCE was originally a similar INT 20 call to device 0001 service 0084 (opcode bytes CD 20 84 00 01 00) but it has been replaced by a call near word [800118D0] (opcode bytes FF 15 D0 18 01 80) after it was once executed. Both of these seem to be some simulated DOS interrupt calls (as they follow a mov eax, 00000021 opcode, which loads EAX register with 0x21, which is the DOS interrupt number). You can perhaps imagine how difficult and frustrating it is to debug code that keeps changing itself while you run it!

In any case, adding a check for page split into movzx opcode handling fixed this problem, but the bigger issue remaining is that there are still a lot of other opcodes that may have the same problem. This is the reason why paging is so difficult (or more accurately, slow) to support. I would need to have a check for this split page handling in every opcode that accesses more than a single byte of RAM, but this will of course make the code much slower. Perhaps eventually I will decide to have two versions of DS2x86, one which does not support paging but is much faster, and another with full paging support but running much slower.

The next problem I ran into was that the code jumped to a real-mode address 8C80:0000, but the processor was in protected mode. So when the code there began with opcodes PUSH CS followed by POP DS, the DS register was loaded with an invalid selector. After some more debugging I realized that the address 8C80:0000 is jumped to when the WIN386.EXE loads and executes KRNL386.EXE (using the DOS LOAD AND/OR EXECUTE INT21 call). The DOS calls can not be run in actual protected mode, only real mode (or VM86 mode). So, if the processor was in actual protected mode after that call, there was certainly a problem in my implementation of it. The call should clear the condion flags when launching the new program, but I had mistakenly made it clear all flags. This meant that also the VM86 flag got cleared, and the processor went into actual protected mode. After fixing that problem the code progressed a little bit further.

The current problem is that Windows 3.11 hangs with the screen in text mode. Looking at the code where it hangs, this seems to be some sort of serious error handler, as the code drops into text mode, prints a message (which in this case is simply an empty string) and then goes into a tight loop. There is an opcode JNE that jumps into itself, so there is no chance that the code will progress further from that point. So, what I am currently trying to determine, is where and why KRNL386.EXE determines something is so badly wrong that it needs to halt the system. Again I need to compare the behaviour with DOSBox starting from the beginning of KRNL386.EXE loading and executing, so this will again probably take many days to figure out and solve.

July 8th, 2012 - DS2x86 slow progress

The past week was my first summer vacation week, and after spending much of my free time with the extra project I have been working on, I wanted to have some actual vacation time for a change. So, I did not code anything for DS2x86 until today, when I finally got a bit bored with simply being lazy. Today I then began again working on the Windows 3.11 support for DS2x86. No major progress yet, I am still debugging it to see why the VGA graphics register handling differs between DOSBox and DS2x86. I believe I first need to determine the cause of this difference before I can progress further, as the problems I am currently having seem to be related to the way that Windows 3.11 accesses the graphics card.

I did however hack together a simple text-mode screen copy routine, mostly just to be able to get some screen copies of my progress to my blog. :-) This is what the actual error message looks like. I believe this is caused by some VGA register handling difference.

So, I will continue working on this issue, hopefully I will eventualy get Windows 3.11 to actually start in DS2x86!

July 1st, 2012 - DSx86 version 0.43 released!

Okay, here is the new fixed version of DSx86, which has the problems I mentioned in the previous blog post fixed. A couple of the problems were actually not in DSx86 but in the tester program, so this list is a little bit different to the list in the previous blog post. Anyways, here are the changes:

My summer vacation is starting now, so I should now have more time to work on DSx86 and DS2x86. I am still working on getting Windows 3.11 running in DS2x86, but I am somewhat stuck with it. I need to compare the behaviour to DOSBox, which is rather tedious and time-consuming work. But, I hope to now finally get some progress done with that, as I can focus on it properly. In any case, thanks again for your interest in DSx86 and DS2x86!

Previous blog entries

Main Page | Downloads | Credits