The DAT format has changed

[dsdude123]

With the release of Update 2, the TT Games script for QuickBMS can no longer extract the GAME.DAT file. The true question remains though, why the change?

[Fluffy Cupcake]

They did it for the lolz because they like to give reverse engineerers a reason to do more crackin in their free time ...and because they were promised cake.

[lol username]

Yeah, I emailed the guy who made the script (well, and a lot more than just the script...) earlier today. Hopefully it'll get updated soon-ish.

Edit: He's gonna take a look at the file.

If the purpose of the format change was just a general upgrade, we'll probably be fine - but, if they're specifically trying to keep modders out of it, it'd make sense for them to also disable the loading from extracted files. We'll find out...

Edited August 25, 2015 by jamesster

[McJobless]

Can't really do much but speculate right now, but I'm going to assume they're just updating and improving the data file structure as they do with every game release (wouldn't surprise me that their in-development game will face multiple file structure changes over time).

If they are trying to crack down on modding, we still have the power of the Hex Editor and the important bits are still readable/modifiable. For now, just time to sit back and let Luigi work his magic.

[Orgcon]

Has anyone e-mailed luigi? Or does anyone know if he knows about this current problem?

Kinda wanted to tinker around with a few ideas I had in update 1 that I didn't get around to fixing.

Although no real rush since I'm off on holiday tomorrow

[dead_name]

Has anyone e-mailed luigi?

Yeah, I emailed the guy who made the script (well, and a lot more than just the script...) earlier today. Hopefully it'll get updated soon-ish.

Edit: He's gonna take a look at the file.

Pls learn to read.

[Orgcon]

I must have skipped that post while scrolling, my bad

[lol username]

Ok, I have updated the script.
Basically these ZIPX chunks are used only for the TXT files and they
are probably obfuscated/encrypted.
Now the script no longer breaks at that point and extracts everything.
There is no solution yet for these TXT files.

http://aluigi.altervista.org/papers/bms/ttgames.bms

The game can't run on the extracted files (not surprising given the text files come out as gibberish - there's also the possibility they don't allow for reading of anything but compiled .DATs now either; without a way to test with 100% properly extracted files, who knows). But we can at least view the other files now. Thanks to Luigi for getting us that far!

Looks like there's data for a Gladiator and three new mixels in the game. I haven't encountered those, yet, so maybe they're unused, unless somebody else has found them. There's only the one Rock Monster.

If anybody wants to take a shot at figuring out the ZIPX chunks, we've got text files from previous versions of the game that have almost certainly not been altered in the new update, and could be compared with the new encrypted/obfuscated/whatever bits. I replied to Luigi asking if he's planning on doing any more with it.

Edit: Sounds like he's gonna investigate further.

Edited August 29, 2015 by jamesster

[lol username]

(Referring to CHEATS.TXT)

This file was interesting because it's just "meh" and its encrypted
version is 0x1b 0x0b 0x78, so the obfuscation may be simple but it's
not possible to figure it from these 2 files.

The only way to do that is working on the executable/dlls.
Maybe I will wait the demo because currently I'm a bit busy.

If somebody else wants to take a shot, go for it.

[NotARaptor]

If anybody wants to take a shot at figuring out the ZIPX chunks, we've got text files from previous versions of the game that have almost certainly not been altered in the new update, and could be compared with the new encrypted/obfuscated/whatever bits. I replied to Luigi asking if he's planning on doing any more with it.

I've only just found this forum, so don't have any previously extracted files to work off. If you can get me some old extracted files and the new extracted-but-encrypted files I'd be happy to give it a bash. The fact that 3 bytes encodes to 3 bytes is a very good sign, means it's probably a simple encoding.

[lol username]

If anybody wants to take a shot at figuring out the ZIPX chunks, we've got text files from previous versions of the game that have almost certainly not been altered in the new update, and could be compared with the new encrypted/obfuscated/whatever bits. I replied to Luigi asking if he's planning on doing any more with it.

I've only just found this forum, so don't have any previously extracted files to work off. If you can get me some old extracted files and the new extracted-but-encrypted files I'd be happy to give it a bash. The fact that 3 bytes encodes to 3 bytes is a very good sign, means it's probably a simple encoding.

Sure, go for it: https://www.dropbox.com/s/7wf36vx89uw7an6/LEGO Worlds Text Files.zip?dl=0

[NotARaptor]

Ok so it doesn't look like a simple encoding. I've tried all the simple methods I can think of and come up blank - so far at least. Some statistical analysis of the ciphertext suggests it's probably strong encryption. Just looking at the byte frequencies:

	Min	Max	Mean	Std. Dev
Plaintext	0	16952	530.7	1860.2
Ciphertext	464	588	530.7	22.6

The plaintext is as you'd expect for a text file, limited byte range (alphanumeric) and lots of repetitions. The ciphertext is much more evenly spread over the whole byte range, clustered around the average.

Going a bit deeper, we can look at the trigram frequencies (just of the TEXTURESUMMARY.TXT file since it's got lots of repeating text):

	Min	Max	Mean	Std. Dev
Plaintext	1	1384	21.75	98.23
Ciphertext	1	2	1.00	0.05

We can clearly see that the oodles of repeating bytes in the plaintext do not carry on over to the ciphertext at all, so that's probably a dead end.

Making the (huge?) assumption that they are encrypted on a file-by-file basis, it's got to be a stream cipher because of CHEATS.TXT - if it was using a block cipher those 3 bytes would be padded out to a larger blocksize. Based on this, no IV is stored in the ciphertext either, so it's either a static IV, or none, or it's stored elsewhere and the files aren't encrypted one-by-one.

 

Are there any other file pairs (same bytesize, reasonably sure they haven't changed in update 2), where the first character (of plaintext) are the same? Eg FILE1.TXT and FILE2.TXT, old versions (not encrypted) start with the same character (letter, number or symbol). This might give us a way to start, checking if their respective encrypted versions also start with the same character or not.

[Sluicer]

I am not sure, if I get all of what you wrote, but I think it is an offset based encryption.

I searched for texts in the old-files and searched for the matching values in the new-files. Then I sorted them by the new-values and detected equivalent positions. I searched for 0000 and lego. Here are the results:

0000         30 30 30 30 // in the old file
1 0001 695f: 28 4f 9b b4 // in the new file
2 0000 695f: 28 4f 9b b4
1 0001 50cd: 3c 63 fd 65
2 0000 50cd: 3c 63 fd 65
1 0000 81a3: 3f 07 98 4f
2 0000 01a3: 3f 07 98 4f
1 0000 dcaf: 4c e3 99 0a
2 0001 5caf: 4c e3 99 0a
1 0000 b553: 4e eb f8 b3
2 0000 3553: 4e eb f8 b3
1 0000 4a34: 56 90 cd 86
2 0000 ca34: 56 90 cd 86
1 0001 2411: 6c cb b6 49
2 0000 a411: 6c cb b6 49
1 0000 4b78: 8d ef e3 51
2 0001 4b78: 8d ef e3 51
1 0000 e7c4: 8f 56 a5 a5
2 0000 67c4: 8f 56 a5 a5
1 0000 2aee: a2 5c f8 71
2 0001 2aee: a2 5c f8 71
1 0000 205d: a9 d8 4a 3e
2 0000 a05d: a9 d8 4a 3e
1 0000 4d73: b6 67 e1 2b
2 0001 4d73: b6 67 e1 2b
1 0001 3983: be 0f 08 9b
2 0000 3983: be 0f 08 9b
1 0000 0c26: e3 cc 99 fb
2 0001 0c26: e3 cc 99 fb
1 0000 8ba3: ea 0d ae 0a
2 0001 0ba3: ea 0d ae 0a

lego        6c 65 67 6f // in the old file
1 0000edae: 06 fd a8 00 // in the new file
2 00016dae: 06 fd a8 00
1 00010bab: 09 ed cb c7
2 00008bab: 09 ed cb c7
1 0000e201: 11 63 eb 01
2 0001e201: 11 63 eb 01
1 00004d7b: 18 a1 bb bd
2 00014d7b: 18 a1 bb bd
1 0001b41a: 18 ad bf 5f
2 0001341a: 18 ad bf 5f
1 00019692: 1e 81 84 f9
2 00009692: 1e 81 84 f9
1 0001f50a: 20 95 05 8e
2 0000f50a: 20 95 05 8e
1 00013f4e: 31 c6 e7 aa
2 0001bf4e: 31 c6 e7 aa
1 000150d5: 33 48 97 66
2 000050d5: 33 48 97 66
1 0000e7cc: 38 7b 31 df
2 000067cc: 38 7b 31 df
1 0000b55b: 42 21 68 09
2 0000355b: 42 21 68 09
1 0000133a: 42 58 8a 47
2 0001133a: 42 58 8a 47
1 0000ee96: 44 91 15 e4
2 00016e96: 44 91 15 e4
1 0001e823: 49 35 3c 3f
2 0000e823: 49 35 3c 3f
1 0000e0bb: 4d 16 7e 72
2 0001e0bb: 4d 16 7e 72
1 0001571e: 5c fe 5a fc
2 0000d71e: 5c fe 5a fc
1 0000dcb7: 6d fb 9f e9
2 00015cb7: 6d fb 9f e9
1 000020e4: 71 17 4e 89
2 0000a0e4: 71 17 4e 89
1 0000ca3c: 71 20 57 26
2 00004a3c: 71 20 57 26
1 0000202a: 72 c0 db de
2 0001202a: 72 c0 db de
1 00014b80: 78 3c 02 11
2 00004b80: 78 3c 02 11
1 00002997: 78 91 e9 a3
2 0000a997: 78 91 e9 a3
1 0000a419: 80 2b 02 cf
2 00012419: 80 2b 02 cf
1 0000e24d: 83 82 a3 7f
2 0001e24d: 83 82 a3 7f
1 00016e05: 8c d5 55 14
2 0000ee05: 8c d5 55 14
1 00011ace: 9a b1 64 43
2 00001ace: 9a b1 64 43
1 00007816: ad 43 70 2d
2 0001f816: ad 43 70 2d
1 00005fd4: b2 f8 4e 82
2 0000dfd4: b2 f8 4e 82
1 00002065: b7 62 66 04
2 0000a065: b7 62 66 04
1 0001398b: b8 d5 bc a8
2 0000398b: b8 d5 bc a8
1 0001b9c8: be 6d ac 9c
2 000039c8: be 6d ac 9c
1 00002162: c5 36 fc 42
2 0000a162: c5 36 fc 42
1 00001a62: c6 70 cf b7
2 00019a62: c6 70 cf b7
1 00012af6: cc f7 0b d8
2 00002af6: cc f7 0b d8
1 0000169a: d4 e2 40 9a
2 0001169a: d4 e2 40 9a
1 00010c2e: d7 ad a5 94
2 00000c2e: d7 ad a5 94
1 0001c982: d9 94 43 55
2 0000c982: d9 94 43 55
1 00018332: db 73 3c 82
2 00010332: db 73 3c 82
1 00001978: db a0 c3 4b
2 00019978: db a0 c3 4b
1 00002575: df 45 f8 b5
2 00012575: df 45 f8 b5
1 00016f27: e0 a5 70 47
2 0000ef27: e0 a5 70 47
1 000108a0: e8 96 88 06
2 000188a0: e8 96 88 06
1 00016967: fa 07 71 92
2 00006967: fa 07 71 92
1 000081ab: fa 35 1c 5e
2 000001ab: fa 35 1c 5e
1 0001bdb8: ff 58 19 49
2 00013db8: ff 58 19 49

You can see that when the last 15 bits of the position are the same the encryption is also the same. But I did not found the algorithm by now.

Edited September 10, 2015 by Sluicer
changed 7 to 15 bits

[ProfessorBrickkeeper]

Are there any other file pairs (same bytesize, reasonably sure they haven't changed in update 2), where the first character (of plaintext) are the same? Eg FILE1.TXT and FILE2.TXT, old versions (not encrypted) start with the same character (letter, number or symbol). This might give us a way to start, checking if their respective encrypted versions also start with the same character or not. 

As most text-based files are those being encrypted, it might be worthwhile also looking at some of the  .lxfml (LEGO Exchange Format) files, as I can say with certainty that the indivual files, nor the formats have not been modified between the two updates. Additionally, and I'm not sure, but do you think taking a look at the raw GAME.DAT file may lead anywhere?

[Sluicer]

I looked a bit more into this and here are my current results (not the complete research way):

It looks like that the files are encrypted in blocks of the size 0x8000. If a block is full, the encryption key seems to be the following (XOR). Here are only the
first 20 bytes of the 32768 bytes:

19 4d 07 6a 08 48 d7 6a 53 3c 59 05 06 4c a2 07 8e a7 7a b5 ...

If a block is not full it seem to be different every time. Maybe it depends on the size of the block. I have no access to blocks (files) that have the
same size, so I can not cross check. In the samples that were provided by jamsster (thanks again) are only different sizes. [request]It could be helpful, if someone could upload a
few more examples.[/request]

 

[NotARaptor]

I've looked a bit more into this too, I can confirm that your sequence for 0x8000 bytes works for multiple files (for blocks of that size), and even better - the sequence seems solely dependent on the block size! For example for block sizes of 33 bytes, the following sequence works as a key (there are two files of 33 byte size which is how I can test it):

4D 6F AE AC 0A E6 CB 00 02 84 46 00 11 E6 2C ED CE 13 B5 44 F1 62 70 5C 9E 41 0A 28 0E C2 DC E4 D7

Now the question is how to derive this key from the block size - could it be an LCG with the block size as seed? I'll try and run some tests when I get the time.

Update : it doesn't look like an LCG - plotting the values as triplets in 3d space doesn't reveal any patterns. I also checked the keys of a 32 and 33 byte chunk, hoping that there might be an obvious connection (since the seeds are off by one) - I couldn't see one:

 

32 byte key : 2f 61 17 d9 57 57 67 01 4b 11 63 c1 61 97 42 de 75 29 c9 a4 9e 8e 5c 7f 87 ee fa 22 0b 16 55 a9 

33 byte key : 4d 6f ae ac 0a e6 cb 00 02 84 46 00 11 e6 2c ed ce 13 b5 44 f1 62 70 5c 9e 41 0a 28 0e c2 dc e4 d7 

(I did spot the 57 57 in the 32 byte key - probably not significant but you never know)

Edited September 15, 2015 by NotARaptor
Update

[NotARaptor]

I think I'm on to something but had to leave for work before I could follow it through. Will update as soon as possible, there is still hope.

[lol username]

so update 3 is out and

[NotARaptor]

Nothing is unmoddable! I haven't had nearly enough time to spend on it as I would have liked (fulltime job, wife, kid [who loves playing Lego Worlds with me!]), and since Update 3 was looming where they probably would have changed the encryption scheme anyway, I didn't pursue what I was doing. But it's definitely possible - it just involves stepping through disassembly.

I've successfully managed to inject code into the process and monitor all the file input/outputs - pretty sure I could "inject" a replacement file and make LW think it came from the .DAT file, even if it is larger. The text files are encrypted of course, and that's going to be the difficult one.

Nothing is unmoddable! Just more challenging

[dsdude123]

So Update 4 is out and the DAT files can't be opened as usual. (well either that or the script was never updated for Update 3)

[Fluffy Cupcake]

It never was updated for 3, and it probably will continue to be so long as the game is under active development.

[aidenpons]

I wandered over to the Steam threads on LW and asked the devs if they were deliberately trying to stop modding or if the updates were just breaking things. The answer - No, it's just usual TT bugginess.

 

 

Hi,

It's not a deliberate thing, but it is something we did warn about some time ago (that our updates mean most mods will be broken). We're also not yet actively looking into mod support yet (and we might not, it's not really been decided), mainly because a) with us still making changes, it wouldn't make sense to do so, and b) it's something that would need quite a bit of time, which we don't really have at this moment in development.

[Orgcon]

Update 5 Game.dat will successfully extracted all of its juicy goodness.

 

You'll need Quickbms 0.7.4a and the latest version of ttgames.bms.txt in order to extract it.

 

The downside is (so far as I've discovered), the game wont recognise the uncompressed files. So when you delete the Game.dat the game tries to load and fails to desktop. 

 

If you have better luck than I, please post details...

Edited May 12, 2016 by Orgcon
Original was a little OTT

[Mariner]

I keep getting this error with extraction of .dat files LB3, and I'm using QBMS 0.7.4 and the latest tt script. The file is GAME0, and I successfully extracted all the DLC .dat and GAME.dat itself without a problem.

[Fluffy Cupcake]

I think your best bet is post on zenhax (where? Idk) or to send a private message to aluigi.