Compression/Decompression

Does anyone have any info about the compression algorythm use on the unicode files. I am getting sick of launching ffedit everytime I need to process .s and .w files. Besides, by the time I know they are compressed, I already have them memory resident in a buffer, so there is a considerable efficiency gain by doing it myself, if it ain't too complicated!!!Martyn T. Griffin
griffinmt@mindspring.com

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

RE: Compression/Decompression, Engineer, Sep-28-02, 11:05 PM, (1) RE: Compression/Decompression, griffinmt, Sep-28-02, 11:08 PM, (2) RE: Compression/Decompression, sniper297, Sep-29-02, 00:33 AM, (3) RE: Compression/Decompression, kevarc, Sep-29-02, 00:40 AM, (4) RE: Compression/Decompression, griffinmt, Sep-29-02, 00:50 AM, (5) RE: Compression/Decompression, Hack, Sep-29-02, 05:24 AM, (7) RE: Compression/Decompression, davidbeach, Sep-29-02, 03:14 PM, (13) RE: Compression/Decompression, bobs, Sep-29-02, 07:03 PM, (14) RE: Compression/Decompression, bobs, Sep-29-02, 07:44 PM, (15) RE: Compression/Decompression, griffinmt, Sep-29-02, 09:47 PM, (17) RE: Compression/Decompression, sehenley, Sep-30-02, 01:20 AM, (19) RE: Compression/Decompression, griffinmt, Sep-30-02, 09:25 AM, (20) RE: Compression/Decompression, JohnCS, Sep-29-02, 01:27 AM, (6) RE: Compression/Decompression, griffinmt, Sep-29-02, 09:55 AM, (8) RE: Compression/Decompression, LawrenceKellie, Sep-29-02, 12:13 PM, (9) RE: Compression/Decompression, griffinmt, Sep-29-02, 12:59 PM, (10) RE: Compression/Decompression, JohnCS, Sep-29-02, 01:21 PM, (11) RE: Compression/Decompression, MikeSimpson, Sep-29-02, 08:11 PM, (16) RE: Compression/Decompression, griffinmt, Sep-29-02, 10:41 PM, (18) RE: Compression/Decompression, LawrenceKellie, Sep-29-02, 02:05 PM, (12) RE: Gaining on it, griffinmt, Sep-30-02, 01:53 PM, (21) RE: Gaining on it, griffinmt, Oct-01-02, 02:10 AM, (22) RE: Gaining on it, MikeSimpson, Oct-01-02, 02:40 AM, (23) RE: Gaining on it, griffinmt, Oct-01-02, 08:01 AM, (24)

Conferences | Topics | Previous Topic | Next Topic

Are you looking for an easier way to compress and decompress, like a windows version or windows wrapper?

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

I am looking for a way to do this within my code. 1) Would prefer to code it myself
2) Would be willing to pass a couple of buffer references to a dll base routine.Martyn T. Griffin
griffinmt@mindspring.com

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

"already have them memory resident in a buffer", I'd be curious to know exactly what program you're trying to hack them with. As far as uncompressing, I just use Route Riter's file utility these days, copy a bunch of S files into a folder, select them from Route Riter, then compress or decompress all at once.

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

You have to be carefull doing W files. There is a bug that will screw things up.Kevin Arceneaux
gdapia

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

I have an application that processes all of the .W files in a route, gathering information from it.If they were all decompressed already. I just open the file, issue a single read to get it into a dynamic buffer, do a Unicode to Ansi conversion, then process the contents. It takes a fraction of a second to do this, as opposed to reading and writing the file more than once, launching a dos box and waiting etc etc.It already works ok doing all that. I would like to make it run in seconds rather than minutes (+).As a result, I will not be touching the original files except once with a single read.The bug has to do with recompressing the W files I believe, using the supplied dos utilities that I am trying to avoid in the first place.This is code that will like run from the scheduler, and I won't be around to poke and pull at manual utilities.FWIW: the utility is one that, among other things, will plot a routes track layout (and optionally any road sections too), similar to what the activity editor does.Martyn T. Griffin
griffinmt@mindspring.com

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

Zlib. Cheers!
Marc - 3DTrains
http://www.3dtrains.comPro Series F-Units

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

>The bug has to do with recompressing the W files I believe, using
>the supplied dos utilities that I am trying to avoid in the first
>place.Martyn,My experience is that ffeditc screws things up while uncompressing the .W files. I've never taken the time to thoroughly test it out, but from what I have gathered, the RE can uncompress the files on the fly without corrupting them, but ffeditc seems to corrupt some files.David Beach.Palouse & Lewiston

1% Completed

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

David is right on the ffeditc bug. The result of the bug is objects that disappear or reappear, depending on the camera angle, as well as some objects floating in the air, upside down. It appears to be the loss the negative portion of a coordinate. Sometimes the problem is bad enough to crash the route editor and/or sim.Saving a corrupt tile with the route editor will fix the problem. Using ffeditc to compress an uncompressed file will also result in a good, packed file.Oddly enough, ffeditc doesn't seem to have the same problem with a shape file, although I haven't got very much experience unpacking those.Bob Siederer
Co-author of Marias Pass 3.0

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

I did an experiment on the uncompression problem. I took a good, packed .w file and used ffeditc to unpack it. I also had a good, unpacked copy of the same file that the route editor produced from an edit.I then compared the output.The problem with the ffeditc copy is not what I suspected, a missing minus sign, but a problem with missing right parenthesis. At the very top of a .w file are a series of ViewDBSphere entries. Some/most of these are missing the right parenthesis. At the end of the section, ffeditc seems to realize this and writes out enough right parenthesis to close out all the entries. There are cases, apparently valid, where a ViewDBSphere entry can exist within another one. Here's an example. The parenthesis that are missing in the ffeditc version are shown in red here.
Tr_Worldfile (
VDbIdCount ( 126 )
ViewDbSphere (
VDbId ( 0 )
Position ( 26.0642 1008.72 -297.116 )
Radius ( 1250.17 )
ViewDbSphere (
VDbId ( 1 )
Position ( 581.237 1007.44 -530.6 )
Radius ( 126.762 )
)
ViewDbSphere (
VDbId ( 2 )
Position ( 634.357 1009.92 -614.762 )
Radius ( 81.4319 )
ViewDbSphere (
VDbId ( 94 )
Position ( 634.36 1008.83 -601.722 )
Radius ( 81.3503 )
)
ViewDbSphere (
VDbId ( 95 )
Position ( 673.053 1008.12 -590.618 )
Radius ( 17.5801 )
)
)

This is from a valid, unpacked version of the file. Here's the same code section as produced by ffeditc's unpack option
Tr_Worldfile (
VDbIdCount ( 126 )
ViewDbSphere (
VDbId ( 0 )
Position ( 26.0642 1008.72 -297.116 )
Radius ( 1250.17 )
ViewDbSphere (
VDbId ( 1 )
Position ( 581.237 1007.44 -530.6 )
Radius ( 126.762 )
ViewDbSphere (
VDbId ( 2 )
Position ( 634.357 1009.92 -614.762 )
Radius ( 81.4319 )
ViewDbSphere (
VDbId ( 94 )
Position ( 634.36 1008.83 -601.722 )
Radius ( 81.3503 )
ViewDbSphere (
VDbId ( 95 )
Position ( 673.053 1008.12 -590.618 )
Radius ( 17.5801 )
I didn't include the closing parenthesis in the second example because they are at the bottom of the 126 entries! The entire rest of the file was identical, once you get past the ViewDBSphere entries.It is just incredible that MS couldn't fix a bug like this and include the revised utility in the first patch. Bob Siederer
Co-author of Marias Pass 3.0

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

Thanks for explaining the comp/decomp problem(s).During my testing so far I have only been uncompressing to a work file, deleting it when done. I also have been 'skipping' all the information upto the first TrackObj or DynTrack entry. I decided to do this with a memory scan so I wasn't reading though, balancing parens, and getting 'in too deep'. Glad I made that 'executive' decision All of this just strengthens my desire to expand the tokens etc myself. At least I could work on cleaning up my own bugs when (not if) needed.Looked at the .bnf file to see how the tokens are mapped. Really helps to read that from back to front. I suppose I should take a 'simple' .w tile as compressed and decompressed and 'try' to work out the token values if possible. Unless someone knows some more about how this should be done?? And this doesn't include disassembling ffedit!!!And anything I am able to find out that is usefull, I would be glad to share it.Martyn T. Griffin
griffinmt@mindspring.com

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

//Could be in ascii, unicode or binary format
//"SIMISA@@@@@@@@@@JINX0s1t______" == ascii
//"SIMISA@@@@@@@@@@JINX0s1t______" == unicode
//"SIMISA@" == compressed
mm_auto_ptr<> inflate( mm_auto_ptr<>& source )
{
unsigned char* p = source;
p += 8;
unsigned long size = *((unsigned long*) p );
p += 8;
mm_auto_ptr<> out( size );
unsigned char* q = out;
int result = 0;
result = zlib::uncompress( q, &size, p, source.Size() );
return out;
}
This was built for s and ace files – mm_auto_ptr is a memory mapped or disk file – byte 8-15 is the byte count for the Zlib uncompress routine.

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

Thanks a bundle. Was starting down this path last night, but needed to properly ascertain the beginning buffer area and how to allocate a large enough output buffer. And that was based only on a hunch.Now it is confirmed, I feel a lot better. I owe you a beer or two next time you are in my area!BTW: Any other tidbits of info are ALWAYS welcome.Martyn T. Griffin
griffinmt@mindspring.com

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

Its not actually a compression algorithm, its actually a compiler which tokenizes the files.

JohnCS
dem_ex@hotmail.com
visit the DEMEX website:
http://www.dem-ex.com

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

But who or where do you get the details from. You must have discussed this with someone to know at least this much.Thanks,
Martyn T. Griffin
griffinmt@mindspring.com

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

You must not be acquainted with JohnCS or his work. Lawrence

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

As it applies to DEMEX, I am. But not in any other aspect.Seems that there are lots of inuendos of information floating around without any actual facts. Rather than suggest that there is 'something' I am not aware of, could you rather be specific and tell me exactly what I 'should' be aware of.Martyn T. Griffin
griffinmt@mindspring.com

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

I can't recall who first made the observation - probably Jim Stevens but that is just a hunch.The presence of the tokenizing files in the utils/ffedit directory makes it pretty obvious however.

JohnCS
dem_ex@hotmail.com
visit the DEMEX website:
http://www.dem-ex.com

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

Also if you look at Bob's example of how the uncompressed version differs from the original, it is pretty obvious that a simple .zip type compression has not been used, otherwise they should have been identical.

Mike

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

LAST EDITED ON Sep-29-02 AT 10:42 PM (EDT)

The FFEDIT utility has the Deflate and Inflate routines linked into it. Those are some of the original routines used with zip. My assumption is that the data gets tokenized, then all but the header gets deflated.Will do some more poking around.Martyn T. Griffin
griffinmt@mindspring.com

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

>As it applies to DEMEX, I am. But not in any other aspect.
>
>Seems that there are lots of inuendos of information
>floating around without any actual facts. Rather than
>suggest that there is 'something' I am not aware of, could
>you rather be specific and tell me exactly what I 'should'
>be aware of.
>
>Martyn T. Griffin
>griffinmt@mindspring.com Take what you know about John's ability with Dem-ex and apply it to other aspects of MSTS, and you will know to what I refer. Lawrence

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

With some assistance from John and Sam, I have be able to at least, take a compressed .W file and inflate it back to its tokenized form. And I have been able to recognize 'some' structure to it, although it would be a ton easier if I new the token values to start with.Martyn T. Griffin
griffinmt@mindspring.com

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

Here is a sample .W file after expanding it with ffedit

SIMISA@@@@@@@@@@JINX0w0t______Tr_Worldfile (
Tr_Watermark ( 6 )
TrackObj (
UiD ( 1 )
SectionIdx ( 38077 )
Elevation ( 0 )
CollideFlags ( 7 )
FileName ( gr2l100mStrt.s )
StaticFlags ( 00200100 )
Position ( 749.491 184.519 963.799 )
QDirection ( 0 0.385251 0 0.922812 )
VDbId ( 4294967294 )
)
TrackObj (
UiD ( 2 )
SectionIdx ( 38077 )
Elevation ( 0 )
CollideFlags ( 23 )
FileName ( gr2l100mStrt.s )
StaticFlags ( 00200100 )
Position ( 820.594 184.519 893.483 )
QDirection ( 0 0.385254 0 0.922811 )
VDbId ( 4294967294 )
)
)
Now here is that same compressed file after processing with my application instead of ffedit.

SIMISA@@@@@@@@@@JINX0w0t______Tr_Worldfile (
Tr_Watermark ( 6 )
TrackObj (
UiD ( 1 )
SectionIdx ( 38077 )
Elevation ( 0 )
CollideFlags ( 7 )
FileName ( "gr2l100mStrt.s" )
StaticFlags ( 00200100 )
Position ( 749.491 184.519 963.799 )
QDirection ( 0 0.385251 0 0.922812 )
VDbId ( 4294967294 )
)
TrackObj (
UiD ( 2 )
SectionIdx ( 38077 )
Elevation ( 0 )
CollideFlags ( 23 )
FileName ( "gr2l100mStrt.s" )
StaticFlags ( 00200100 )
Position ( 820.594 184.519 893.483 )
QDirection ( 0 0.385254 0 0.922811 )
VDbId ( 4294967294 )
)
)

Both have an empty line after the SIMIS line, but it gets stripped in this output display.
There were only two differences.
1) Mine adds a CRLF after the last parens
2) Mine adds quotes around the filename string (not tested on real world yet)I have a long way to go, since it is a manual process to parse the token file for all the token values as well as any token header flags that need to be deciphered.
Hopefully, when I am done, we will have a friendlier version of FFEDIT as well.Martyn T. Griffin
griffinmt@mindspring.com

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

Good luck with that routine Martyn. MSTS does put quotes around the filename string in the .W files if it has a space in it, so if your routine puts quotes around the whole lot there should be no problem. I do not think an extra crlf after the last parens would make any difference either.When I wrote Route-Riter, I placed quotes around all the filenames in the .ref file and that works fine, easier to quote the whole lot than to miss one.One problem I found with ffeditc_unicode.exe is that it does not handle filenames with '&' in them, or certain foreign letters like the German umlaut. Uncompressing Horse&Cart provides you with an uncompressed file named Horse.If you can make a nice uncompress replacement for ffeditc which gives perfect results and fixes this filename problem, I would appreciate permission to use it with Route-Riter in future versions.Regards -

Mike

Alert | IP

Printer-friendly page | Edit | Reply | Reply With Quote | Top

If (when) I get this going, I would be happy to share it with you. You may not be able to use my application easily, but I plan on docing all of it so it would be somewhat straight forward to directly code it in, rather than launching an external piece of code as we do now with ffedit. At most you would need to do is externalize the token info and load it when needed.At this point, I am doing all of this with Winbatch since it is faster to develope it. The performance is still respectable for these type of memory manipulations, but when scanning all the world tiles and shape files in every route etc, the 'slowness' begins to show!Martyn T. Griffin
griffinmt@mindspring.com

Alert | IP

[ Home | MSTS Forum | MSTS FAQ | File Library ] Copyright © 2002 by Nels Anderson. All Rights Reserved.