This is a guest post by A. N. Other.
The following was originally intended as a comment to “Not MSX Either“, the 4th installment in the hunt for the mysterious 0x69 FAT VBR-start byte which was allowed in DOS. Due to the length and content I was asked to turn it into a guest-post, presented below.
Definitive answer to 0x49, leading to new speculations about 0x69.
First let me thank Octocontrabass for the golden tip that got things rolling!
Also bleuge for your marvelous effort (I’m very curious to some info about [46; 2E; 262]…).
I got into this matter as my fuzzy tester uncovered 0x49. Further testing did indeed show that MS was checking for 0x49 for >25 years and that it indeed still does so for Win10, instead of 0x69 as stated on Wikipedia (and corresponding talk page, in fact there is currently no mention of the reality of 0x49).
It is this 0x49 that, for practical reasons, interested me the most. And who knows, it might give some insight into the 0x69 (and I will end with some new theories about this based on the facts regarding to 0x49).
The journey:
First I checked some current popular implementations. They also checked for 0x49 (instead of 0x69). Initially I figured that since ‘everyone’ was talking about 0x69 and not 0x49, and ‘everyone’ did check 0x49 (without informative comment) that it was well known why we check for 0x49.
At that point I had already found the first 3 pages on OS/2 Museum but not yet this page. I decided to ask on reactOS, where I was pretty instantly pointed to this page and 2 devs started to dig with me during a couple of days. (I must say I’m in awe about that community.) Each taking our own approaches, I dug in with the ‘FM TOWNS’ and ‘IPL’ leads from the article.
FM TOWNS quickly makes it clear that we are talking about the Japanese market, and the few number of (relevant) results for ‘IPL’ wouldn’t be at all surprising IIF we are talking about compatibility with a Japanese only (but huge!) market: google would likely conveniently filter out Japanese language links for non-Japanese users. Likewise, it might also explain why bleuge’s massive effort turned up empty.
So the first goal was to find some raw images (preferably floppies for starters) with this ‘IPL’ thing (I mean, how the heck did that work on a PC-Compatible BIOS).
As sheer luck would have it, someone has been working during the past year on an FM TOWNS emulator “Tsugaru” (part of the FM TOWNS Resurrection Project: http://ysflight.com) and shares some very valuable insights, and an empty floppy image (note this wasn’t the only image that was examined, archive.org for example has multiple GB of FM Towns hd/fd image archives).
On his page “Emergency Boot Floppy Disk with English Localization” you can find instructions to use a pre-existing FM Towns emulator called “UNZ” (and download link) to create a genuine “Emergency Boot Floppy Disk” using the official tools/software. It includes screenshots where to click (because Japanese), as well as a ready-made empty (raw) floppy-image for download.
This same page also mentions:
“copy to a FM TOWNS format floppy disk (who is saying PC-9801 format? Call it FM TOWNS format)”
(which according to MS sources I found later appears to be mostly correct but we likely should call it “FujitsuFMR format” or just “FMR format” instead, as MS turns out to do…).
IMPORTANT: this is about the High-Level (file-system) format, NOT the physical Low-Level format which is commonly referred to as the (NEC) PC-98 format, read on!
So, what did we just get (aka what the froyo is a 1.23M floppy and what are “3 mode floppies”)?
First we need to realize that the Japanese PC market was somewhat isolated from the rest of the world during the ’80s and ’90s: Their market was mostly dominated (~50%) by a tri-factor of more or less PC-compatible machines, the Fujitsu FMR/FMTOWNS/FMMARTY being less (dig around a bit in Wikipedia for more info).
The Fujitsu FM-R series (a popular precursor to the FM Towns) was introduced in … 1987 with MS-DOS 3.1 (some say 1986)…
Due to historical reasons based in this set of market dominant machines, combined with practical cost factors for end-users, Japan used a smaller-sized floppy format (compared to the common 1.44M) at 360rpm (instead of 300 rpm), closely resembling the 1.2 MB format with 15 sectors per track previously found on 5.25″ high-density floppy drives.
This in turn (for obvious mass-production reasons) led to 3.5 inch “3 mode floppy disks”: (720KB + 1.2MB + 1.44MB) and 3 mode drives to go with it (if your bios and OS supported it, note one can still buy USB 3 mode floppy drives). The physical low-level format was obtained by simply leaving the inner 3 tracks of the HD format unused. And a buyer could always do a physical low-level reformat (assuming hardware/software support).
*I assume* that just like ‘we’ bought low-level 1.44MB pre-formatted + ‘DOS’ high-level pre-formatted disks right out of the store, the Japanese could buy ‘PC-98’ low-level pre-formatted disks (perhaps even with FMR/NEC/DOS/etc… high-level formatted) disks.
For those interested in how ‘soft-sectored’ low-level formatting works, see this discussion.
Anyhow, with this minimum required background, we continue quickly:
Ysflight’s “Writing 1.23MB Floppy Disk Image back to actual floppy disk in Windows 10” page explains some more detail, including how to get a NEC PC-98 low-level format physical disk and *THEN* overwrite the (newly created low-level sectors) with the FMR high-level format image:
“FM TOWNS (like PC 9801) can only boot from a 1.23MB floppy disk. Let’s call it FM TOWNS format. Don’t call it PC-9801 format.
FM TOWNS format disks were used only in Japan. All other places in the world were using 1.44MB format. If you use Linux you can easily write a 1.44MB disk image back to an actual floppy by using dd command. But, a tool for writing 1.23MB disk image back to an actual floppy disk can be found nowhere. It was a major obstacle to format a 2HD disk in 1.23MB FM TOWNS format and then write a disk image to it.”
See, this is a bit better to understand with the background information I provided (again noting, a better terminology might be along the lines of “FMR format on PC-98 disk”).
He instructs us (on Win10) to:
You need to do it in 32-bit version command mode. Why? Try from a regular 64-bit command prompt. Format command doesn’t take an option /f:1.23m
(1) Start a 32-bit version command mode by typing C:\Windows\SysWOW64\cmd.exe in the Start menu.
(2) Insert a floppy disk in drive A and type:format A: /f:1.23m
Note: I’ve seen other command line arguments (such as specifying 77 tracks etc.) that appear to also work, and other tools do exist, YMMV.
The author of that website then wrote a tool to overwrite the sectors with an 1.23MB FMR image, the tool can be downloaded from the above page (which we are discussing). He notes:
“Actually once you write a valid disk image, Windows 10 will allow you to read/write files to and from the disk. Windows XP had a horrible habit of corrupting the floppy boot sector, but Windows 10 doesn’t seem to do it.”
OK damnit, spill them first (VBR) bytes already:
The image we downloaded and now understand a bunch more about starts with (in hex):
49 50 4C 34 EB 53 00 20 20 20 20 00 04 01 01 00
In plain ASCII the first bytes then decode to “IPL4”. It’s followed by a common jump (to vbr_start + 0x59) inside the OEM_Label field at vbr_start+4. And followed by a common BPB:
1024 byte per sect. 1 sect. per clust. (1024 bytes) 1 reserved sectors (1024 bytes) 2 nbr. of fats 192 Root directory entries (6144 bytes 6 sectors) 1232 Total logical sectors (1261568 bytes) 254 Media descriptor 2 Logical sectors per FAT 8 Physical sectors per track 2 Number of heads
Right….. what trickery demo-scene uber-hacking makes this (IPL4) work ?
http://ysflight.in.coocan.jp/FM/towns/bootloader/e.html shares with us the last (well, almost) piece of the puzzle under the header “Technical Information” where he documents FUNDAMENTAL parts in his experiences in writing his own “IPL = Initial Program Loader” (which is like an umbrella-term to describe the sum of the different parts/files of the bootloader):
“During the initial boot process, FM TOWNS’ system ROM read the first sector of a media to B000:0000, and if the first four bytes are “IPL4”, it jumps to B000:0004. This process is independent of CMOS setting. Therefore if you write your own boot loader and write to the first sector of a media, the system ROM will find your program and jump to offset 0004 even if your CMOS RAM is wiped due to depleted battery. The boot loaders here use this mechanism.
The first stage of the boot loader, or initial program loader (IPL), is limited to (apparently) 2KB, 2 sectors of a 1232KB floppy disk, or 4 sectors of a SCSI hard disk drive. It cannot do too much things in 2KB. So, IPL reads the second-stage boot loader to 8000:0000, and jump there.”
He further describes that from there on the point is to find and load a sector starting with “FBIOS” (roughly the equivalent of DOS IO.SYS).
He documents more, be sure to continue reading on that page if it interests you!!
Wait… what? The bios “jumps directly” to vbr+4 when it starts with “IPL4”? And that’s just it?
aka: Gimme the damned last bytes of that image’s VBR:
vbr+510 = 0x00 and.. 0x00 vbr+1022 = 0x00 and 0x00
But, but.. magic disk signature 0x55 0xAA ?? Ain’t there, get over it. That’s why they say ‘mostly’ PC-compatible and as we can see, this one was the least compatible 🙂
Ok, time for conjecture (proven below) at this point:
So to be able to read/write to a FujitsuFMR floppy/partition the first thing we need to allow is the VBR starting with 0x49 (ASCII uppercase I). AND NOT check the boot signature.
Hey, as already shown in this series on OS/2 Museum, DOS 3.3 already stopped checking the boot signature.
THUS: knowing what we know now (so far), it would not be wrong to state that:
Windows could not read/write these FMR floppies/partitions if it didn’t allow 0x49 and/or required boot-signature!
Indeed, it would be pretty safe to say/comment that: 0x49 allows Fujitsu FM TOWNS.
BUT that doesn’t prove that this was actually MS’s reason to include the check (such proof would be ideal, right?).
*PROOF* (…finally):
(WARNING: scroll down if you do not want to see this, and scroll back up to the “NEW 0x69 THEORIES” header)
A copy of NT4 source is floating online (*very easy to find*) and starting at line 215 of nt4/private/ntos/fsrec/fat_rec.c we can find:
// FMR Jul.11.1994 NaokiM - Fujitsu - // FMR boot sector has 'IPL1' string at the beginnig. if (Buffer->Jump[0] != 0x49 && /* FMR */ Buffer->Jump[0] != 0xe9 && Buffer->Jump[0] != 0xeb) { result = FALSE; // FMR Jul.11.1994 NaokiM - Fujitsu - // Sector size of FMR partition is 2048. //snip by me to only post bare minimum for proof // FMR Jul.11.1994 NaokiM - Fujitsu - // 1. Media descriptor of FMR partitions is 0xfa. // 2. Media descriptor of partitions formated by FMR OS/2 is 0x00. // 3. Media descriptor of floppy disks formated by FMR DOS is 0x01.
NOTE: “IPL1” (instead of our previously discussed “IPL4”).
NOTE: “Sector size of FMR partition is 2048” (not the previously discussed 1024)
NOTE: if you’d look at the full file, you’ll see it’s littered with additional “FujitsuFMR” code, which kicks in when “JAPAN” and “_X86_” are “defined”.
A bunch of other files contain additional FMR_fat checks and comments explaining that for example PC-98 doesn’t need all that many extra kludges (it has some, but they start as regular PC-compatible on the VBR) and the FujitsuFMR needs the ‘most’ kludges for compatibility with the dominant marketshare (feel free to search for them, they are not elementary to the proof above).
NOTE: these include: scsidisk/disk.c, disk/disk.c, diskdump/diskdump.c, fastfat/fsctrl.c.
You’d be looking at flags like “FatData.FujitsuFMR” or:
#if defined(JAPAN) && defined(_X86_) && defined(_FMR_)
// Fujitsu TakaO Oct.20.1994
// Partition information is contained in block1 on FMR
//
(this snippet is relevant for the conjecture about 0x69, based on what we have found)
As far as I’m concerned, 0x49 is clearly specifically added for FujitsuFMR high-level format/partition. We guessed it, understand the mechanics and found confirmation in the M$ source.
Also, almost forgot: YAY !!
Additional: on IRC Michal told me he’s sure that 0x49 wasn’t in NT3.5, he expects it to be added since NT3.51.
NEW 0x69 THEORIES
Remember as shown above, the Fujitsu FMR was introduced in 1987 with DOS 3.1
Recall that MS-DOS 3.21 OAK is from May 1987 while the 3.30 OAK is from July of the same year (see here for details). On IRC Michal told me DOS 3.2 was released April ’86.
Recall, as stated by Michal, that MS “had Japanese support reasonably early on”.
Recall the market dominance of these NEC PC 98 / “3 mode” floppies and interoperability on that huge Japanese market.
Recall DOS 3.21 SRC\BIOS\MSDISK.INC (courtesy of the MS-DOS 3.21 OAK):
;******************************************************************************* ; Put a sanity check for the boot sector in here to detect boot sectors that ; do not have valid BPBs. ; We examine the first two bytes - they must contain a long jump or a short ; jump followed by a NOP. ; If this test is passed, we further check by examining the signature at ; the end of the boot sector for the word AA55H. ; If the signature is not present, we examine the media descriptor byte to ; see if it is valid. ;****************************************************************************** cmp byte ptr cs:[DiskSector],069H ; Is it a direct jump? je Check_Signature ; don't need to find a NOP cmp byte ptr cs:[DiskSector],0E9H ; DOS 2.0 jump? je Check_Signature ; no need for NOP cmp byte ptr cs:[DiskSector],0EBH ; How about a short jump. jne BadDisk cmp byte ptr cs:[DiskSector]+2,090H ; Is next one a NOP? jne BadDisk Check_Signature: ; look for signature word on disk cmp word ptr cs:[DiskSector+510d], 0AA55h jz GoodDsk ; if present disk is good
In my opinion the explanatory comment above does not address the 0x69 “direct jump” at all. As if it was added or intentionally not addressed in the explanation.
The 3.3 version retypes the explanatory comment block making things even weirder by changing one line (introducing the ‘long jump 0x69’):
; put a sanity check for the boot sector in here to detect
; boot sectors that do not have valid bpbs. we examine the
; first two bytes - they must contain a long jump (69h) or a
; short jump (ebh) followed by a nop (90h), or a short jump
; (e9h). if this test is passed, we further check by examining
; the signature at the end of the boot sector for the word
; aa55h. if the signature is not present, we examine the media
; descriptor byte to see if it is valid. for dos 3.3, this
; logic is modified a little bit. we are not going to check
; signature. instead we are going to sanity check the media
; byte in bpb regardless of the validity of signature. this is
; to save the already developed commercial products that have
; good jump instruction and signature but with the false bpb
; informations
; that will crash the diskette drive operation. (for example, symphony diskette).
cmp byte ptr [disksector],069h ; is it a direct jump?
je check_bpb_mediabyte ; don't need to find a nop
cmp byte ptr [disksector],0e9h ; dos 2.0 jump?
je check_bpb_mediabyte ; no need for nop
cmp byte ptr [disksector],0ebh ; how about a short jump.
jne invalidbootsec
cmp byte ptr [disksector]+2,090h ; is next one a nop?
NOTE: in the middle of that BLOB of explanatory comment-text, they continue that “for dos 3.3, this logic is modified a little bit. we are not going to check signature. instead we are going to sanity check the media byte in bpb” !!!
Finally note, this assembly file is filled with special marked sections warning maintainers that the actual code in these sections is going to get patched!!!
Conjecture 1:
They intended 0x49 (not 0x69 as in the code). Bad/outdated (Fujitsu-dev) specification they worked on? Looked at the wrong column for ‘I’ in the ASCII table and entered the hex for lower case ‘i’ ?
I think this is less likely however in an era where such strings and labels were usually uppercase, also I really do assume multiple eyes (including the one that re-typed and modified the explanatory comment in 3.3).
Conjecture 2:
The precursor to FM-R was the FACOM_9450. Could it perhaps have used the string ‘ipl’ (lowercase)?
I have not yet been able to find images for this, also the ‘uppercase’ would have been expected argument. Also, the FMR (as specifically mentioned in the 0x49 proof) co-incided with DOS 3.1
Conjecture 3: (well some versions of the same idea with different reasons):
It’s a placeholder: The binary could be patched by changing the 0x69 check to 0x49, which would be sufficient after dos 3.3 removed the boot signature check. For 3.21 one would additionally modify the subsequent jump function to skip the “Check_Signature” check and move on to the “GoodDsk” label instead!
The lowercase ‘i’ combined with the increasingly ‘odd’ comments AND knowing the FMR bios would indeed “jump directly” to VBR+4 does kind of feel like a ‘logical’ way for me to ‘hide’ it: containing sufficient ‘hints’ to someone knowing what to look for and why, also an almost ‘lazy’ mod from intended 0x49 to 0x69. And if nothing starts with 0x69, then perfectly safe!
Remember, this isn’t a crazy idea, this assembly file is already patched all over the place!!!
Reasons:
- someone sneaked it in to support his own FMR machine
- the programmer was ordered to ‘sneak’ it in (hiding for non-Fujitsu OAK customers)
- it was to test this on machines after patching it (3.21 to 3.3 is a time-frame of months)
- it was already formally there for “JAPAN” “FMR” support, we just need to dig deeper to see it get patched when applicable
- it was a placeholder for future support (Wikipedia notes that MS devs recall that when IBM ‘took’ DOS 3.0 it was far from finished).
My money is on conjecture 3, reason 3 or 5 or 4.
SO, Happy hunting,
588079
PS: I hunted for WHY 0x49. Since that goal is reached (with a more satisfactory answer than I anticipated, I didn’t expect it to be spelled out so extensively), I did NOT inspect if/how OEM_Labels containing null bytes are handled.. something could perhaps be found there.
from Hacker News https://ift.tt/2W8etgJ
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.