I almost forgot to write about the successful execution of Task 3:
I basically patched the SegaLoader (i.e. the Media Board payload) to break after initializing the GD-ROM, i.e. after reading the GD-ROM into Dimm-Board’s memory. Then I just repeated the steps I did for dumping the Media Board (after the Media Board is switched into “Dimm Board”-mode, the same read commands will read the data from the DIMM Board instead of the onboard flash). The result is a (~300 MB, in my case) .gcm file with the game data. It won’t run as it is on a Gamecube due to the main board/media board dependencies, but I’m not going to say that this cannot be patched. An interesting property is that the unused data (i.e. after the game data - the image is not padded with garbage, like usual Gamecube discs are) doesn’t look like uninitialized ram. Instead, it looks like random garbage, but repeating, sometimes changing. I don’t think that the RAM will be intentionally filled with that data - instead I believe that one of the SEGA chips on the DIMM board decrypt the data on the fly, and the data is stored in RAM in encrypted form.
The next step now is to understand the process of decrypting the GD-ROM data. For that, I started with reversing the DIMM Board firmware. I also tried to dump the GD-ROM data directly, with mixed success (due to my setup. I should use a real Dreamcast). But I dumped enough to see repeating 8 byte patterns, which is a strong indication for a 8byte block cipher in ECB mode. Maybe a simple DES? Who knows… This data is probably loaded, as it is, into the DIMM board’s RAM. I’ve searched for decryption functions in the firmware, and didn’t found any block cipher. Thus I’d say that it’s done directly in hardware.
Now the “security PIC” (each game comes with a specific one) comes into the scene: What I did found in the DIMM Firmware was something which looked like something that bitbangs stuff, after a lot of obfuscation, over a GPIO bus. 8-byte strings are send, with 8-byte strings received as the answer. Two example string is “Akeycode” and “Bkeycode”. The result of these commands (7 bytes of the first response + 1 byte of the second respopnse) are written into a specific register. Might that be the decryption key for the game data? There are some other commands, and something which looks like a challenge/response, but that’s nothing I care about at the moment.