Modding An Unreleased Game

[JrMasterModelBuilder]

Here is a BLK file extractor prototype (for onua.blk). Unfortunately, I don't entirely understand the file format yet (I'm a little stuck) so it might not work properly, but it works to the extent I understand the file:

http://www.mediafire.com/?kz2rpvd2hvq641v

FILE FORMAT SPECS:

First 4 bytes: 42 4C 4B 46 = BLKF - The file header. Duh.

Next 4 bytes: 01 00 00 00 - Not a clue.

Next 4 bytes: 81 00 00 00 - The number of files, 129 in decimal.

for each file (56 bytes each file)

{

First 40 bytes - The file name (ends at first 00 hex or 0 dec).

Last 16 bytes - (four, four byte blocks listed below).

{

Block 1 - The file size in decimal (info on how this works below).

Block 2 - Not a clue. It seems it must represent something. Always greater than the file size at least in onua.blk. hook.x and lhook.x are duplicates with both have the same bytes for Block 2.

Block 3 - No idea. Always "03 00 00 00" at least in onua.blk.

Block 4 - The offset of the file (relative to the beginning) in decimal (info on how this works below).

}

}

The info below: To find the number that those 4 bytes represent, multiply as so (BYTE# being the byte in decimal form):

({BYTE1}*1)+({BYTE2}*256)+({BYTE3}*65536)+({BYTE4}*16777216)

You probably get the idea, every time it rolls back to 0 after reaching 255, the next byte stores that number.

For easy viewing, I've created a spreadsheet with all the values of the byte blocks for each file (except Block 3):

http://www.brickshelf.com/gallery/JrMasterModelBuilder/random/tlomn/tlomnfilelist.xls

I'm almost positive the *.x files are DirectX binary files. The only problem is there is some extra junk in them I don't understand yet.

All the files in onua.blk start with:

"ÿxof 0303ÿbin 0032ÿ"

However a proper binary .x files should start:

"xof 0303bin 0032 "

The *.x files have a bunch of extra ÿ's throughout the file starting out a 0 then every 9 bytes after that for a little bit, but then it stops (not sure at all how that works or if it's even consistent). Maybe it has something to do with the mystery block. Not sure.

Needless to say, the *.x files fail to open in DirectX Viewer.

Anyway, if anyone can help me figure out the rest of this file, that would be great! At the moment, I'm stumped.

[lol username]

I don't have the slightest idea what any of that means, but I'm glad that it seems we're making progress. :D

[Dovahn]

My suspicion when I saw the extra characters in the .X files was that it had to do with the "compression" of the .blk file. Forgive my unfamiliarity with the .X format (my only knowledge of it is of the non-binary version) but I've noticed (as I'm sure you have as well) that there are occasionally plain-text names of parts in the file ("neck", "pelvis", around hex line D485) with a bunch of jumbled characters interspersed. As far as I can understand, there'd be no reason to break up these strings with other stuff unless it was some sort of secondary operation, like compression. Of course, the weird thing is that it's adding characters rather than actually compressing the file, so I'm not sure what it could be doing aside from deliberately obfuscating the file. Unless we sort of reverse-engineer the compression algorithm, I think our best attempt would be to try to identify what algorithm is being used, provided it's non-proprietary.

That's my take on it, anyway. I'll take a closer look at the extracted files when I have a chance. Thank you for making your extractor by the way; it's really helpful. Is there any chance you'd release the source so that I could work on it too, to try to "uncompress" the extracted files?

[JrMasterModelBuilder]

Yeah, it's really wierd. I still wonder if it's related to the mystery bytes associated with each file. At least whatever it's doing is most-likely consistent. hook.x and lhook.x are duplicates.

For source code. Sure. Let me clean it up first:

Version 1.1 (More robust and editable.)

Source code. Includes Flash CS5 FLA and XML descriptor file along with a text file of all the ActionScript.

Also, here's just the ActionScript:

//AUTHOR: JrMasterModelBuilder


import flash.filesystem.*;

import flash.events.Event;

import flash.net.FileFilter;

import flash.utils.ByteArray;


/*==================== NON-FUNCTION TERMINATING VARIABLES ====================*/

//File data.

var fileDataArchive:ByteArray = new ByteArray();

//Array of all file names, offsets, and sizes.

var fileListArchive:Array = new Array();

//Blank file data.

var blankBytes:ByteArray = new ByteArray();

//Remember the folder path from dialog to dialog.

var folderMemory:String = new String();

//Default text.

var defaultText:String = "Click EXTRACT to browse for a BLK file to extract, then select a folder where it will extract all the files.\n";

/*==================== END NON-FUNCTION TERMINATING VARIABLES ====================*/



/*==================== MAIN SETUP ====================*/

//Set the text on the dialog.

text_txt.text = defaultText;


//Add actions to the buttons.

extract_btn.addEventListener(MouseEvent.CLICK, extract_btn_f);

function extract_btn_f(e:MouseEvent):void

{

	erase();

	text_txt.text = defaultText;

	openFile_e();

}

build_btn.addEventListener(MouseEvent.CLICK, build_btn_f);

function build_btn_f(e:MouseEvent):void

{

	text_txt.text = "ERROR: Archive building not yet supported.";

}

/*==================== END MAIN SETUP ====================*/



/*==================== USEFUL FUNCTIONS ====================*/

//Erase memory.

function erase():void

{

	fileDataArchive.clear();

	fileListArchive = new Array();

}

//String find replace. Useful for removing the problematic backslash from Windows file paths.

function strRep(string:String, fnd:String, rpl:String):String

{

	return string.split(fnd).join(rpl);

}

//Just the file name.

function noPath(input:String):String

{

	return input.substring(input.lastIndexOf("/") + 1, input.length);

}

//Just the folder path.

function allPath(input:String):String

{

	return input.substring(0, input.lastIndexOf("/") + 1);

}

//Find the integer the 4 bytes at a certain offset represent.

function findNumber(offset:int):int

{

	return fileDataArchive[offset] + (fileDataArchive[offset+1] * 256) + (fileDataArchive[offset+2] * 65536) + (fileDataArchive[offset+3] * 16777216);

}

//Decimal to ascii converter.

function char(dec:int):String

{

	return String.fromCharCode(dec);

}

//Integer to 4 byte decimal converter.

function findDecimal(input:int):Array

{

	var iplace1:int = 0;

	var iplace2:int = 0;

	var iplace3:int = 0;

	var iplace4:int = 0;

	var returnArray:Array = new Array();


	//Roll it over to the next place until the end giving what's left to the first number.

	while(input >= 16777216)

	{

		iplace4 += 1;

		input -= 16777216;

	}

	while(input >= 65536)

	{

		iplace3 += 1;

		input -= 65536;

	}

	while(input >= 256)

	{

		iplace2 += 1;

		input -= 256;

	}

	iplace1 = input;


	returnArray.push(iplace1);

	returnArray.push(iplace2);

	returnArray.push(iplace3);

	returnArray.push(iplace4);


	return returnArray;

}

/*==================== END USEFUL FUNCTIONS ====================*/



/*==================== EXTRACTION FUNCTIONS ====================*/

//Browse for the file to extract.

function openFile_e():void

{

	//File object.

	var file:File = new File();

	if(folderMemory.toString() != "")

	{

		file.nativePath = folderMemory;

	}


	//File type filter object.

	var fileFilter:FileFilter = new FileFilter("All files", "*.*");

	//Open the file dialog with filter and wait for one to be chosen.

    file.browseForOpen("Open", [fileFilter]);

    file.addEventListener(Event.SELECT, fileSelected);

	//When selected.

	function fileSelected(e):void

	{

		folderMemory = allPath(strRep(e.target.nativePath, "\\", "/"));

		//New file stream object.

	    var stream:FileStream = new FileStream();

		//Open the file selected, load all the bytes, and close it back up.

	    stream.open(e.target, FileMode.READ);

		stream.readBytes(fileDataArchive);

		stream.close();

		//Process the file.

		process_e();

		//Check that there are file to save.

		if(fileDataArchive.length < 1)

		{

			return;

		}

		//Save the files.

		text_txt.text = "After a folder is selected, extraction will begin. Please be patient while the files are extracted. When complete, this message will change."

		saveExtract_e();

	}

}

//Write all the files to the selected folder.

function saveFiles_e(extractto:String):void

{

	for(var i:int = 0; i < fileListArchive[0].length; i++)

	{

//		trace(fileListArchive[0][i] + " || " + fileListArchive[1][i] + " || " + fileListArchive[2][i]);

		var fileOpen:File = new File(extractto + fileListArchive[0][i]);

		var stream:FileStream = new FileStream();

		stream.open(fileOpen, FileMode.WRITE);

		//If the file is blank, write blank bytes. Else, write the correct bytes.

		if(fileListArchive[1][i] == 0 || fileListArchive[2][i] == 0)

		{

			stream.writeBytes(blankBytes);				

		}

		else

		{

			stream.writeBytes(fileDataArchive, fileListArchive[1][i], fileListArchive[2][i]);

		}

		stream.close();

	}

	text_txt.text = "File extraction complete!";

}

//Browse for folder to extract to.

function saveExtract_e():void

{

	var file:File = new File();

	if(folderMemory.toString() != "")

	{

		file.nativePath = folderMemory;

	}

	file.addEventListener(Event.SELECT, dirSelect);

	file.browseForDirectory("Choose a folder.");

	function dirSelect(e:Event):void

	{

		folderMemory = strRep(e.target.nativePath, "\\", "/");

		saveFiles_e(strRep(e.target.nativePath, "\\", "/"));

	}

}

//Load all the file info into arrays.

function listFiles_e(offset:int, number:int):Array

{

	//2D array with file name, offset, and size.

	var fileList:Array = new Array(3);

	fileList[0] = new Array(number);

	fileList[1] = new Array(number);

	fileList[2] = new Array(number);


	for(var i:int = 0; i < number; i++)

	{

		var fileName:String = "";

		for(var i2:int = 0; i2 < 40; i2++)

		{

			if(fileDataArchive[offset+i2] != 0)

			{

				fileName += char(fileDataArchive[offset+i2]);

			}

		}

		//Add above to array.

		fileList[0][i] = "/" + fileName;

		fileList[1][i] = findNumber(offset + 52);

		fileList[2][i] = findNumber(offset + 40);

		//Skip to next file offset.

		offset += 56;

	}

	return fileList;

}

//Process the file.

function process_e():void

{

	//Check if file has BLKF 66 76 75 70 for a header.

	if(fileDataArchive[0] != 66 && fileDataArchive[1] != 76 && fileDataArchive[2] != 75 && fileDataArchive[3] != 70)

	{

//		trace("ERROR: Not a BLK file.");

		text_txt.text = "ERROR: Not a BLK file.";

		erase();

		return;

	}

	fileListArchive = listFiles_e(12, findNumber(8));

}

/*==================== END EXTRACTION FUNCTIONS ====================*/

[Dovahn]

Excellent; thanks a lot. That'll make my next plan a little easier.

I noticed that many of the files contain repeating sequences of data that have one or two differences. This kind of thing makes sense, as there are probably a lot of enumerated coordinates, for example, for 3D data. However, these sequences are often offset every few enumerations by one of the mysterious FF sequences. Other times (as I'm sure you've noticed) other English strings are interrupted, either by an FF or something else. I'm going to start by trying to find a pattern in the FF interjections in those repeating patterns, and then take a look at the interrupted strings. Whatever these extraneous characters are, they must be deterministic if the original code was able to decode them.

I'll keep you posted! I might have results in a few weeks, depending on how much time I have.

[JrMasterModelBuilder]

Excellent; thanks a lot. That'll make my next plan a little easier.

I noticed that many of the files contain repeating sequences of data that have one or two differences. This kind of thing makes sense, as there are probably a lot of enumerated coordinates, for example, for 3D data. However, these sequences are often offset every few enumerations by one of the mysterious FF sequences. Other times (as I'm sure you've noticed) other English strings are interrupted, either by an FF or something else. I'm going to start by trying to find a pattern in the FF interjections in those repeating patterns, and then take a look at the interrupted strings. Whatever these extraneous characters are, they must be deterministic if the original code was able to decode them.

I'll keep you posted! I might have results in a few weeks, depending on how much time I have.

Cool! Let me know if I can help.

[Dovahn]

Yeah, I could use some additional brain-power here.

The main problem that I've got is that, aside from the header, this looks very unlike a regular .x file. There are a few keywords here and there (often split by an inexplicable "FF") but an awful lot of information seems to be missing.

Just so I'm clear, here is "muds.x" (one of the smallest 3D files).

Muds.x:

ÿxof 0303ÿbin 0032ÿÿXSkinMesÿhHeader

ÿÎiñ<|ÿÿ«D“À÷Âb¯Ñrâ(ÿðn÷MaxWeigÿhtsPerVe¿rtex)¼.@FaceIþPBonesãþñ-CDupÿlicationIndicesÿIUÖ¸ÉוIÿ‰ÃS©¨°1ãe)ÿðP–±ýPOriginSalC½4µi—²yø5;

oÒÿºgA Ã€"Oï%ú»1ÄtraÿnsformNodeNamh±9

Ü

vD–ï9.ÿ4*áw:põÿð	Matrÿix4x4úm°OffseõtyFrîBRoo+t

ÿðÑT5¯õ

ã€?"*	//-#ÌÛÕ

nC½É3$C³¿,!·¿|<?F}Ëœl"‡÷ûh>u!C3?6V ??V l"ê»v?q „"à/‡Æ>.Ž u 2"i%Nû}¨!¿lâ€Ã>ï·|¼¾™!«‘?·Â¿¥!q!¹(¾u!â„¢!.!Ñ!Â!â„¢!Êx~'Ø Ç#£‘¿¡!xÂ!u!•(¿}ª2!Iی!14Ã!Ñ ?Â¥ €˜"©!q!,!I1 X20ƒ¼R2"042e$Ëœ ¼†Ä <?u=y?‹3¡ ¾e1©!m0l"u110l"Ã!À• Ø"…$ø!42Ã4€Ž„æ Ã4ۈ $1Ãœ51¾ M1u%.!i!m4€1!I…5A9OKFâ€?rO$FÂOŸGâ„¢5} è!à2¹ ¼2Ã$ˆ6…9ÉIÉIyIQ5uM0ÀE°JÉH¾A_SSEIEM®™5AW P3$W!ÂTW!“QÛ£UÛŸUŸQÛÓQoŸQäÃUõÃUµßU䵫ÓUä

h«	e	ea	ºÓUa_Ceº@hoa_e_oaÓU_§U_!€ƒe!§UW!¯eÃeÇÔhç×eçßeßa"çÂTÃOÂTOPq-#u-ÂTÂ¥ÂT«ÂT[qOq`xoq_u_qRoq ÂTº!"ÂT›qÂq# x¯qŸuŸq¯q©$ÂT%&ÂT'*ÃŒt(ÂT)*Ãœxßqûqëa“QqOßeOïUOÛÿi¯U“QOq[qA+`x«ïU[qµ,ÂT-ìTÃUÂq›q_u›qº€/e›q	y	eÃq.Å“x/Å’doe0ld¿yoaOeëaÇßuÇ“Q¯eÇP!ÿy!Âe12Q3ÂT[‘Oq4`Ëœ5Â¥6ÂT7,tu89Xâ€â€œQ›‘Âqou›‘oqŸ•([‘oqOu:;Ëœâ€â€œQÛ‘Ãq¯uÛ‘¯qß•›‘¯q

Âu<=Øâ€â€œQQëaHûq“Q2!@¨Û‘AÃŒxB„â€Qĥqïe‡‘ç@_Â¥QçëaN:ãalÅ“U#_ÂÂü('l"O–?DĽYl"ê[&>Ƥ?ÿ+ùd?<ù侞,!I-0? բb‡ƒÂ>Þ¡Æ 2$¼£3ûÄÙù¡¿z7ž>ï_–†¾ê¡½O?O>9¿Þ¡Â¡

¸Ô

²ê¤"¸¿Î¨¿>ê·¿ò¨¿æ¨¿yx)s2"6±hƒ·>"°Ã¡Ã>ê¡ú¡Â¡B´Âµ?y2"±')°¹²±ò ¾ž±ú¡¾°l"±‚°l"±æ =±2!Ù¡%±RŽ°ýµ¾î¸u±	Ã…i±åµ E¹¢¾¸¿Ö¸ D1#Π@ù¥-±é²–²£´zÌ¿ŠÈ¾‰_›_­_¿_Ñ_ã_õ_oo+o=oOoaoso…o—o©o»oÃoßoño'^ãPÃ¥Pá;y«GuCu?qOuGqOqÂÃ¥Ku[q“u‡uÆ’u[qÂu_q«™‹u›qÓuÇuÃu›qÃuŸqë™ËuÂ!Â3ÂEÂ?qGqPÃ¥Gqk‰Gqµp鋉[uÂÃ¥_q«‰Ç…°éˉ›uouŸq뉕ð韭w'ŸÞÓ0ùK™Ûuçq…?q/ußq;qÂçq‹™ÛqïqßuïqguÔOqÿyOq?uÛq÷qïy§uóu«qïuïq«qÀåÛqûq÷qÄÃ…ÃuûqÃqP÷qÃqõÛqçq/¥çqk©çqۍ/¥ûq‹¥G!‘O!TÿextureCo§ord£¦[#†,"Àî=ÂúÔá=ˆ˾2 €¿þÔ;¡¿ç}!ø"÷V¼ül ñô©£?ÌÌÃL&"43³öÂ?ßü©ƒ¿ü( ÃÌãL¾7!óI ê<öÿ€?!ÌÌÌ>ž÷Xñ>#Ÿ1ñ@C!w!+!}ê @à|ê<‡!"ˆ ;!ìw!3!>u:¢ÀþÔy1¢"Â$À©Ã¯¡ßÀd°b¾3!çS‹‡>#@2"×!+!õû…àP€¿øS°=Â×!{#á!ÃŒ>ç#w€9Öô iá>§#lxÃÂ!ÃŒ>¿#€?Ç!Å“c!Â#€?ú° c#zý¡ÀG$Àw¿ÂJ4I1€?W3=1I!x 1o¿ì‡¸ €¿/3L†23!Â9+!¯!+!w5€Ã!ó#y9Rz2Ç!C#1z"×5/5/59 

"ÄàÑþ™¡Materia_lList¤¡RÉÄÀÃ.O@OROdOvOˆOÅ¡O¬O¾OÃOâOôO__*_H<_N_][z‚E

gÃð•¡4¤ˆTxÃfff?^â„¢UÃÌÌ=Â¥UÖrþÃFilenamóe

ÃÀS¡mudâ€Â Ã“U

Here's a sample of a simple cube that I made, exported into a standard binary .x format:

xof 0303bin 0032VertexDuplicationIndices

IUÖ¸ÉוI‰ÃS©¨°1ã)nIndices)nOriginalVertices4)indicesnIndicesEffectParamDWord

¼c9áQ®]L°Ã£©Ù|Ã¥1	ParamName)ValueFVFData


ç¶ùŽƒNâ€Â­Ã¬ÃˆÂ°Ã€H—)dwFVF)nDWords4)datanDWordsEffectFloats

³âÃñã

(N¯¡Zu

(-)nFloats4*FloatsnFloatsEffectString

~	[Õ¶½RL°=`QÈÂB1ValueEffectDWord

Ã,bn•©MÂÅ *ùO<ç)ValueEffectInstance

ä÷1ãYÂLŽ™ìW’Â1EffectFilenameAnimTicksPerSecond

CZAž¦{sJ‡C·=Gè„v)AnimTicksPerSecond

VertexElement

FR÷#öH¹øƒP…3o)Type)Method)Usage)

UsageIndexDeclData

SÃ¥"¿,)ÂGŸêb½UKÓ)	nElements4

VertexElementElements	nElements)nDWords4)datanDWordsEffectParamFloats

 ¹0õbŒG›†ä¬ŸNA‹1	ParamName)nFloats4*FloatsnFloatsEffectParamString

ˆL¼Ãâ€Ã®FÂv,(ÂÅ’â€Â1	ParamName1ValueFrame	RootFrame

FrameTransformMatrix

€?€?€¿€?FrameCube

FrameTransformMatrix

€?€?€?€?Mesh

H€?€?€¿€?€¿€¿€¿€¿€¿€¿€?€¿€?€?€?€¿€?€?€¿€¿€?€?€¿€?€?€?€¿€?€?€?€?€¿€?€?€¿€¿€?€¿€¿€?€¿€?€¿€¿€?€¿€¿€¿€¿€¿€¿€¿€¿€?€¿€?€?€¿€?€¿€?€?€?€?€?€¿€¿€?€¿€¿€?€?1



MeshNormals

H%�%�%ÿ%�%ÿ%ÿ%ÿ%ÿ%ÿ%ÿ%�%ÿ%�%�%�%ÿ%�%�%ÿ%ÿ%�%�%ÿ%�%�%�%ÿ%�%�%�%�%ÿ%�%�%ÿ%ÿ%�%ÿ%ÿ%�%ÿ%�%ÿ%ÿ%�%ÿ%ÿ%ÿ%ÿ%ÿ%ÿ%ÿ%ÿ%�%ÿ%�%�%ÿ%�%ÿ%�%�%�%�%�%ÿ%ÿ%�%ÿ%ÿ%�%�1



MeshMaterialList

Material

ÃÃŒL?ÃÃŒL?ÃÃŒL?€??€?€?€?VertexDuplicationIndices

At least its got a bunch of unreadable characters. But the original file still has all of these strange ÿs, among other things. Even weirder, "muds.x" doesn't even have all of the plaintext keywords ("MeshNormals", "FrameTransformMatrix") that the sample file has. My suspicion was that the file was obfuscated somehow, but more than that, it seems as though either some information is omitted, or it was somehow compressed even more.

I ran a frequency analysis on the ÿ characters, and found that, not surprisingly as you observed, they continually show up around 8 characters apart. However, it's not consistent. Sometimes they drop out completely. Sometimes they're 7 characters apart. Sometimes its another sequence that interrupts an English string.

I'm going to keep poking around to see if I can correlate any more of the "muds.x" file to the sample file I posted here, but I'm a bit stuck. Any thoughts?

[JrMasterModelBuilder]

It just gets better and better. Do all Direct X .x files have those missing strings?

This might be a long shot, but could it maybe really be a "tzip" MSZip Compressed Text File or "bzip" MSZip Compressed Binary File .x file with the "bin" header?

I've been looking through the other files, and it's not just the .x files with extra bytes. Other files have characters breaking text strings, and it's not always "ÿ".

Maybe if they would have spent less time making crazy file formats and more time working on the game, it would have been released.

[Dovahn]

Heh, yeah.

My first suspicion was some sort of compression (and I still think that's the case), but it would have to be some sort of post-processing that also resulted in the formation of the .blk itself. If there's one thing that I'm fairly certain about, its that the first few header bytes should be unmodified in the file format itself. Plus the header would label it "bzip" or "tzip" if it were either of those.

So I'm thinking that it might be a larger compression algorithm, which I think gives two options: either its something documented, so we'd just have to figure out what compression it's using, or its proprietary/designed specifically to obfuscate. To be honest I wouldn't have suspected that deliberate obfuscation goes back as far as this game does, but I suppose people have been hacking games ever since they first started existing.

Either way, it may or may not be important to look at the whole .blk file when trying to figure this out. There might be a large-scale repeating pattern that I'm missing in the individual .x file. However, you were able to pretty successfully break down the .blk's structure, so I'm not sure if the entire thing is relevant. Still, it is an interesting thought to look for some larger scale repeating patterns. Maybe it's simpler than I thought.

But still, it can't just be obfuscation; it must be compression as well, because "muds.x" is much smaller than even the simplest compressed .x file. But almost all compression algorithms I vaguely understand require some place in the compressed file to store the compressed information, so it wouldn't be this easily parsable (as you parsed it). Hmm... time to do some research into compression algorithms.

[JrMasterModelBuilder]

Do you know roughly how much a compression algorithm can reduce a file size?

I think you're probably right about the compression. If the files are compressed, then maybe the unknown bytes represent the uncompressed file size.

[Dovahn]

That makes sense, it's just... Well, I don't know how much you know about compression (so forgive me if this is all old hat), but here's an example of what's perplexing me: I know of a format called LZW, which is named after a bunch of people that implemented a particular, index-based compression algorithm. Basically, the compressor scans the files for repeating patterns. Then it writes that pattern in an index somewhere and replaces it each time in the file with a smaller, key character. So that way, you get more in less space, by not having to rewrite all of those repeating patterns. But the problem with these files... There doesn't seem to be an index. Somewhere, the data need to be actually stored, but it seems that all I can see is the replaced key codes. Maybe it's in all of those numbered files. But if they all have standard .x headers as well, I'm just not sure where it could have gone. As you determined, the structure of the whole file is pretty straightforward.

As for the amount, I'd bet that it's doable to what we see. I think different algorithms can reduce different things differently.

[JrMasterModelBuilder]

Ok, that makes sense. Maybe it's not index-based compression?

[Minifig9292]

Are you positive that was never finished?

Because I have a bionicle game somewhere that I bought in 2002, that looks nearly identical to that one.

Let me go find it and I'll tell you about it.

EDIT:

No never mind, the game was this one:

http://www.gamespot.com/pc/adventure/bionicle/index.html

[lol username]

Are you positive that was never finished?

Because I have a bionicle game somewhere that I bought in 2002, that looks nearly identical to that one.

Let me go find it and I'll tell you about it.

... *epic, epic facepalm*

Totally different developers...

[Minifig9292]

Are you positive that was never finished?

Because I have a bionicle game somewhere that I bought in 2002, that looks nearly identical to that one.

Let me go find it and I'll tell you about it.

... *epic, epic facepalm*

Totally different developers...

Yeah I just clarified myself, my memory was VERY hazy about that game since I rarely played it.

[JrMasterModelBuilder]

I had another thought. Maybe those broken strings at the beginning are indexed bytes that are used later?

[Minifig9292]

Looking at this thread I keep hearing echos of my old 6 year long involvement with the Half-Life2 Leak community.

I LOVE fixing things like this and I really like toying with this kind of content.

I am greatly interested in unreleased game content and such.

I really wish I could help you guys with tracking this down and/or fixing it.

My extensive knowledge of the HL2 Leak engine, as well as my limited knowledge of the S.T.A.L.K.E.R 1935 alpha build, sadly, cannot aid anyone I don't think.

The scripting can't be the same, the model types can't be the same, nothing can be.

I may have a slight understanding of the engine, though not even close to JMMB due to myself modding and fixing for a variety of different engines.

I'm very knowledgeable of the Gamebro engine, Soruce,, Goldsource, Unreal, whatever the heck stalker runs on, and Cryengines 1 & 2.

I am extremely interested in this game and if you guys DO think I could do anything to possibly help with whatever, it'd be great. :satisfied:

[Dovahn]

Sure, Katatonic, we'd love some extra help! You know as much as we do -- right now we're trying to figure out what sort of compression/encryption they used on the game files, specifically the .x files. If you haven't followed (maybe you have, but I'll fill you in anyway!) we've been given a packaged file that contains a number of 3D files from the game. Thanks to JMMB's unpacker (on a previous page) we've pulled out the individual files, but we're still trying to figure out how to make those readable. They seem to have extra characters and are possibly compressed, but I don't have a lot of experience with compression, so I'm just looking for patterns.

Oh, JMMB, so you're suggesting that the readable stuff is in fact the index, and the actual filled in content is the jumbled stuff later in the file? That's really interesting! I'd be a little unsure as to why they need to repeat the header each time, if it's the same in all of them (wouldn't that be the *first* thing to be indexed?). If your theory is right, my guess would be that we'd see more readable strings in the earlier files (physically earlier in the compressed file) as they haven't been indexed yet. Later files along the byte stream could then just reference the beginning ones and therefore be smaller.

Going by the plausible assumption that the unknown number in each file is the uncompressed size, I compared the ratio of the actual size to the original size. I would expect that, if the strings are indices and later files therefore need fewer indices (they can just reference older ones) then there would be more compression in later files and the ratio would be smaller. The result isn't encouraging (see here), but there do seem to be some patterns. Just for fun, I also plotted it along the byte position in the file (here). It seems that the file size has something to do with the value and regularity of the ratio, but I'd have to think some more to figure out what it means. Plus, we're not sure of what the unknown number really does represent, but given the patterns I noticed, the ratio might be helpful.

[Minifig9292]

The only strange compressions I've ever dealt with is .GFC and .ESP files.

Unless you got one of those two I'm not sure what I could do.

I also know how to decompile and recompile models.

But I don't know if a 10 year old game engine needs that.

[JrMasterModelBuilder]

Oh, JMMB, so you're suggesting that the readable stuff is in fact the index, and the actual filled in content is the jumbled stuff later in the file? That's really interesting! I'd be a little unsure as to why they need to repeat the header each time, if it's the same in all of them (wouldn't that be the *first* thing to be indexed?). If your theory is right, my guess would be that we'd see more readable strings in the earlier files (physically earlier in the compressed file) as they haven't been indexed yet. Later files along the byte stream could then just reference the beginning ones and therefore be smaller.

I was figuring that each file was compressed individually with it's own index in each file which was then packaged in the blk file. But I'm not sure how to go about testing that theory.

Going by the plausible assumption that the unknown number in each file is the uncompressed size, I compared the ratio of the actual size to the original size. I would expect that, if the strings are indices and later files therefore need fewer indices (they can just reference older ones) then there would be more compression in later files and the ratio would be smaller. The result isn't encouraging (see here), but there do seem to be some patterns. Just for fun, I also plotted it along the byte position in the file (here). It seems that the file size has something to do with the value and regularity of the ratio, but I'd have to think some more to figure out what it means. Plus, we're not sure of what the unknown number really does represent, but given the patterns I noticed, the ratio might be helpful.

I do know that the unknown bytes associated with each file are always larger than the compressed(?) file size.

[JrMasterModelBuilder]

Since double-posting isn't against the rules:

It appears other games may use the same BLK file format. I found someone else who appears to have had the same problem with Bioshock: http://www.gamedev.net/topic/461627-opening-the-bioshock-blk-files/

After doing some more research into modding other games that may use the same file format, it looks like there may be other file(s) associated with the blk files to extract them. I think I'll ask Mark if he can give us a list of all files for the game.

[Lair]

Since double-posting isn't against the rules:

...did you honestly have to put this here?

Posted 26 May 2011 - 05:26 PM

this isn't even a double post...

Also who's Mark?

[JrMasterModelBuilder]

Since double-posting isn't against the rules:

...did you honestly have to put this here?

Posted 26 May 2011 - 05:26 PM

this isn't even a double post...

Also who's Mark?

*shrugs* On BZP, we were always told not to double-post, which was defined as two consecutive posts. Of course, the rules there are rather wacky. . .

Mark Durham knows the guy who owns the game. He's the one who gave us the files to the game.

[Lair]

*shrugs* On BZP

I gave a long exasperated sigh right there. Sorry. It's not you I have anything against... it's BZP.

On BZPower you're also told that Spoilers are ONLY to be used for actuall spoilers. No large image containing, sorry. You all suck and Hurgu Dume or whoever is the forum.

Except now Bionicle is over so spoilers are useless.

So Mark is your friend and his friend gave you both copies... or is this something about designers?

[Dovahn]

That's temptingly relevant. My guess would be that .blk is simply a common name for a proprietary format, but it certainly does look like there should be some external storage file for all the data. But good luck getting anything from RQ -- unless you know otherwise, I don't think he's communicated with anyone in months. Seriously, though, good luck; I'd be as happy to see more info as you would. And of course we have updated save games to try, if DB "got his old computer out of storage".

I haven't done any more work on this recently. But I was waiting for some additional communication with RQ because, like you, I don't think we have enough. But it's on my list. :)