MDL

[Holy_Diver]

I'm looking at the MDL files under a microscope. If anyone knows something they're holding back now is the time to share.

I'm working under the assumption the format starts to look very much like the popular .mdl format at one point or another.

I'd like to determine where the header ends and datablocks begin asap. I'm going to be in town(s) a bit this weekend / won't have very regular internet access from this point on. I will have my computer / dig into this a bit more myself if I end up stuck with nothing else to do at some point.


Off the top of my head, it looks like the following guesses might hold water...

First two octets: These are a "magic number" which seem to differ based on the folder where you'll find the file (different numbers may indicate different header layouts etc.)

Next octet: seems important, probably the number of discreet entities in the file.

Next 3 octets: I'm guessing this is a global additive RGB colour. It is always 1,1,1 which would makes sense because we don't see coloured models (that I know of) and 0,0,0 would probably make the entire file transparent.

Next 2 octets: Not sure, probably important.

Next 4 octets: These appear to be 0 in all the files I've examined. Possibly a reserved field, maybe a 32bit number or more than one number of unknown significance.

Next 4 octets: This appears to be two 16bit values (the headers may differentiate at this point)

Next 512 bits and onward: This looks like a string of 32bit values, either file offsets or meta data. This might be the point where the header begins to converge with the traditional mdl format. I will look into this possibility asap.


I could benefit from some simple mdl files if someone would be kind enough to prepare/share some. The non-discreet info is compressed via lookup tables however for simple planes and such the indices should overlap in a meaningful way. If possible please provide all the info you can about any such files. Things like a cube, square, triangle, etc.

EDITED: Nvm, we are only able to make MDO files yes? The Mdo and Mdl files appear to be fairly differently encoded. Does anyone have a guess that the Mdo format might be based on a popular format?

[Holy_Diver]

Man I don't think this format is at all based on Quake MDL. Even if it is, I know it has to have "bones" in it, and Quake mdl does not do bones (it just does frame interpolation) ...there are variants with bones apparently (something called Game Studio, possibly Half-Life) ...but I'm not sure these could've influenced Som.

The first 32bits are probably not magic numbers. They are probably flags of some form. Simple information not in the prf files. The colour theory may be in doubt also (@ML, which files had the field you could use to override colour?)

The only real positive lead I have is a weird file, data\item\model\BOX1.MDL. I'm not sure where it came from, but it should elucidate a lot of the basics. I have a feeling the format was influenced by MDL but technically speaking I'll be surprised if there is any bit to bit relationship.

[Holy_Diver]

After staring at mdl files compulsively all weekend I think I understand this well enough to start modeling it.

It definitely bears no resemblance to mdl files. It's basically a quite involved packet based encoding not unlike Playstation primitives.

I'd like to document it somehow but I'm afraid anything short of a diagram would be a waste of time. I'd be happy to spell it out if anyone good at diagrams would like to bang out something pretty.

I'm confident I can get most files rendering / rip the basic geometry, but it could be a while before I'm convinced I'm aware of all the different control codes. There are very well codes that were never used with any of the Som mdl files. Of course more advanced stuff like animation will come last. And of course you probably won't want to start popping out custom mdl files until everything is mapped out. There might very well be ways to modify the existing files to a large degree first.

I'm going to add Som mdl format to Assimp in one form or another so the files will open up in the assimpview tool. At that point it would be a lot of help if you all could dig thru all the mdl files to see if any don't match up visually (which will probably be a ‎  very iterative process)


EDITED: I think btw, given the nature of the format it's probably possible to extend the format in such a way that we could add information to the files which would not keep the same files from being used by Som. That would/could prove very handy down the line.

[Madison Lastrega]

Sory folks I have been busy , this week is my wifes birthday........Anyway, Alls I can say is that from my research, MDO files.............I made a few Primative base color models..A square and a cube...... WITHOUT textures......... the first sections of the files are the base color, However if you add a texture it's files file reference and a whole bunch of other stuff (UV Mapping and such I suppose) then the base color code...... then it gets confusing from there......... ........the less poly the MDO the smaller the file..... In MdL files the base texture IS baked into the actual model......(Hence, open the NPC select texture NONE and what do you got) So the first part of the file is the actual TEXTURE and it's uv mapping......TEHN sctroll down and then look you'll find the same color code setup, that's why I DO think my work with mdo will open doors to MDL files...Example..Take the MAN NPC there are three types replace the first section of one mdl with the same section from another version and you got easy texture switching...Roughly the first 45 or 60 K of Hex (depending on the size texture)when I did a custom texture with the man I embeded it using the tim method.......It added 192k to the MDL size proving that the textures are embedded and not linked like MDO files...................
Next..... As I said before .CP data is attached to ANY MDL that moves........ the files mainly contains the patterns of movement for both walking and magic.....Thats how I make my customs....... Upon looking way down any MDL file you will see simular patterns REsembleing cp patterns of movment...They are the models movement , Just dont know exactly where they end or begin exactly:)...... Take simple MDL items like boxes and levers...........they have simular paterns and are easier to work with .........My work has been halted due to recent issues but I still work a little each day trying understand , how the NODES (parts of SOM) work together.... I have switched to atempting to put another chest model in place of one the default ones........... Will keep you guys posted on the results............ Hope this helps a little................ML

[Madison Lastrega]

I thought my last post needed proof so here ya go, included in this post are the files that I used that PROVE som reads mdo files as raw data there is no header no little section the defines the number of verts or polys NO HEADER test these yourself..... swap colors all you like , the model data is the same on all three and as you can see the data is ONLY TWO sections 1 base color, and 2 Model data... If you add a texture to these you get a slightly bigger file with texture and uv references before color, that is referenced in the PRF file................ but then 3 sections............ som mdo files (at least) are read as raw data and that is why they need prf files to tell som what to do with the raw data .........(takes bow) thank you and goodnight:) ‎ 

[Holy_Diver]

I will read all of this asap but I think I'd like to keep the discussion on MDL and not MDO. I don't see any similarity between the two. MDL was a little confusing to stare at, but MDO just makes no sense to me. That said I'm sure if I looked at it deliberately it would start to unravel. At the least I can say there is no apparent relationship format wise. Do we have enough tools to work with MDO files already? If so I'd prefer to perfect our understanding of MDL first.

Speaking of which you all might be relieved to hear that I've a fairly strong grasp of the MDL format a to z and it's just a matter of time before you'll have your MDL tools. I expect more friction working with Assimp than with the tools itself however it's worth it so that there will be a long laundry list of import formats and counting right out of the box.

I actually think I'd rather work on deciphering the PSOne TOD format before MDO unless the From' MDO tools are really inadequate.

[Holy_Diver]

Thanks, that cube .mdo file will help a lot when I go there (if no one beats me to it)

If you can afford to prepare slightly dissimilar models and compare the outputs that can be a very effective way to figure out what's what. I'm completely useless when it comes to using the From' import tools. Not that it matters with .mdl (because there are none that I know of)

My approach ultimately was just to stare at a few different files in a hexeditor like a WW2 codebreaker, occasionally testing series by making changes and loading the MDL into Objedit to see what happened. That weird BOX1.MDL file I found really helped get the ball rolling but I mostly relied on the arrow, arm, fire/fake, and the wooden chest (O000.MDL)

I have to admit I'm a little curious where BOX1.MDL came from.

[code=The MDL header looks like this]struct Header_SOM
{
//! 16 bytes
uint8_t unknown00; //! perhaps bitwise flags
uint8_t num_anims; //! reference frame animation
uint8_t num_stops; //! stop-motion like animation
uint8_t num_skins; //! texture images
uint8_t num_parts; //! one or more required
uint8_t unknown03; //! perhaps always one
int32_t off_anims; //! dword anim/stop/skin block offset
int16_t unknown04; //! perhaps always zero
int16_t add_anims; //! dword sizeof anims block
int16_t add_stops; //! dword sizeof stops block

//! this is the actual order of the data blocks in the file
int32_t offset(int fourcc, int pt=0)const
{
switch(fourcc)
{ ‎ 
case 'base': return 4+7*num_parts;

case 'face': return 4+AI_BE(parts[pt].off_faces);
case 'vert': return 4+AI_BE(parts[pt].off_verts);
case 'norm': return 4+AI_BE(parts[pt].off_norms);

case 'anim': return 4+AI_BE(off_anims);
case 'stop': return 4+AI_BE(off_anims)
‎  ‎  +AI_BE(add_anims);
case 'skin': return 4+AI_BE(off_anims)
‎  ‎  +AI_BE(add_anims)
+AI_BE(add_stops);

default: return 0x7FFFFFFF;
}
}

struct //! 28 bytes (these may actually be 16bit)
{
int32_t off_verts; //! dword offset to vertex block
int32_t num_verts; //!
int32_t off_norms; //! dword offset to normal block
int32_t num_norms; //!
int32_t off_faces; //! dword offset to face block
int32_t num_faces; //!
int32_t reserve00; //! probably always zero
}parts[1] PACK_STRUCT;

} PACK_STRUCT;[/code]

[Madison Lastrega]

great work Holy.......few questions though? could you send me a link or tell me what hex editor your currently using? I use xix32 and it's view can be confusing......Next could you send me one of the MDL you have mapped, since I started this project I taught myself Hex to ansi, so if you send me one mapped I can make notes of the normal postion of the header parts......... (will help once we tackle TOUGH MDL's like NPC's and such...My test subject has been MDL o0004 ( the tomb looking box) that is the smallest MDL WITH animation......... one more thing, from what you shown, I don't see anything that would indicate "skeleton based" I feel and I stand by this, that som mdls use FRAME based movment like Quake mDL files...... The research I have done with .CP data indicates FRAME based movement....Even your example SHOWS that they move by frames..... and no Joint or skeleton points...... Now that would make sence BECAUSE in 1999......99.9.9.9.9.9.9% of all games were animation based....Only FF8 at the time had even incorperated the use of skeltons...... the tech was still evolving, and SoM being cheaply made would be anothere indication of FRAME based movement...... Play with the .cp files of NPC or emenies..... You can make them glide, teleport, MISS frames and such, So unless DEFINATIVE PROOF comes forth, I HAVE to say that The animation is stored in frames in the MDL, and would explain why some MDL's are huge, while ones with little movement (like the slamander) are not so big at all................. Less frames................ ‎  ‎  But Holy I must say again........GREAT JOB.........Thanks:)
NOW that being said Our work will be tough in mapping....... Now I want to quickly explain MY work with MDO files, I know that they are different,,,,,MDo Have No header........But the info we can gain from them is awesome......... I have even gotten simpilier in test, WITH BOMB results......... I made a single sided RED plane and a double sided plane.... Very small MDO...... And I have been able to isolate EVERY part...And can now Manipulate the actual verts of the MDO...... The point.........in the MDL files the data of each frame, and the actual MODLE will be MUCH like the MDO base as far as the atual model goes and it's frames....Once we can isolate WHERE the model data is , and it's frames of the model...We could instead of inserting new models like I want we could ALTER the existing Verts and lines of the file with out breaking it, changing the shape of the model in the files, then with clever reskinning, we could make ALTERED versions of existing models THAT WORK , so thats why my MDO research is important:).Keep up the good work, and I will continue on my end till a resolution is found that works efficiently:) ps will send my mapped results of my planes to you, believe me, MDO is one of the ways we will solve the MDL customs problem:) with your work as the forefront......thanks Holy, till next time................................Happy Modeling

[Holy_Diver]

I actually just use PSPad's "open in Hex editor" feature, because that's the text editor I tend to use. All hexeditors are pretty much the same deal. Some might layout the hex slightly different. I find I prefer groups of 16bit hex values 8 on each line.

I don't think there is any reason you'd want to manually hexedit a MDL file, other than replacing the TIM texture(s). It's all just very technical model data. Nothing like colours you can tweak in there.

You can see in the header there is num_anims and off_anims. That goes to the animation data block. The data blocks are packet based, so what's in there I can't actually say, because I have not attempted to decode the packets. It is a global block (not per object) so it could be like another file encapsulated within the file (like the TIM texture) for all I know. It's the last thing on my list to figure out. I will want to make sure we have the visuals right first.

I think Quake mdl file frames are very simple vertex morphing animation frames. They don't define a local coordinate system or anything. So that seems very inadequate to me. The MDL format is fairly sophisticated for it's day. It's clearly influenced by the PSOne file formats (it uses Tim textures after all... which was never meant to be used outside of Playstation libraries) so I suspect the MDL format existed so PSOne game models could be easily converted over to Som. The MDO format seems like it might be the more sophisticated of the two, even though obviously the enemies/npcs are dependent upon MDL for their action based animations.

When I'm done you will have a tool that converts very many model formats into a MDL file and other features. I will probably do the MDO format also around the same time just for completeness sake. Therefore I'm trying to say don't knock yourself out too much slaving over hacking the files. But whatever you can find out it might be very helpful if I get stuck once I start staring at MDO files.

I program these kinds of things, so I know what to look for in binary files. I can just stare at the files over some coffee and pretty much work it all out once I sit down to do it.

All in all the MDL format conforms closely to all my expectations and conjectures over the last year. I gotta admit I never expected to sit down and work it out from scratch at this point, but Todd telling me it might be a Quake MDL file seemed to at least get me to look at it. Since I found this PSOne file format bible on my harddrive last night, I'd say it's just a matter of time before I have all the Shadow Tower and PSOne KF stuff converted over for use with Som. It's too bad John is MIA. He really wanted to see this day come.


EDITED: If you have any simple MDO files you've prepared, if you attach them to a post like you did before I can find them later when I need them.

[Holy_Diver]

Can anyone think of a MDL file with semitransparent elements? The TIM texture modes don't support transparency more or less. The non "true color" mode has 1 bit for enabling transparency but it's supposed to be interpreted by the PSOne primitive mode which I'm not sure what the analog to this would be under Som. There could still be some other non-texture way to apply uniform transparency to faces in the MDL file so I ask.