Results 1 to 13 of 13

Thread: What Compressor Used?

  1. #1
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    19
    Thanks
    11
    Thanked 2 Times in 2 Posts

    What Compressor Used?

    Trying to determine what specific compressor was used in compressing the file images.dta?

    This file is part of the game "Nightmare Adventures - The Witch's Prison". Downloadable from here -

    Code:
    https://www.bigfishgames.com/games/5962/nightmare-adventures-the-witchs-prison/?pc
    The file images.dta is 212 190 501 bytes in size.

    The file seems to extract OK using 7zip. But on re-compressing file contents (as a zip file), the compressed file size does not match the original file size, despite trying different compression settings! The new compressed file size is usually smaller than the original. This is of some concern, as ideally the original and re-compressed file sizes should match. Although I understand a (slightly) smaller file size might still work, but this is dependent on the game software and how it reads this file.

    Using a hex editor, the file header reads -

    Click image for larger version. 

Name:	images.dta.PNG 
Views:	26 
Size:	13.6 KB 
ID:	6602

    This suggests the images.dta file was compressed using a compressor using (a variant of the) "zip" algorithm(?)

    Can anyone please confirm which specific compressor was used, and verify that re-compressed data (using the same compressor?) is the same as the original images.dta file?

    Thank you!

  2. #2
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    7-zip has a custom size-optimized algorithm for zip compression - better/slower than zlib, which is normally used for this.

    You can either process this file with precomp - which is a tool specifically made for a task like this.
    Or you can extract it, then try to compress with gnu zip - you can find it as zip.exe in UnxUtils.zip: https://sourceforge.net/projects/unx...utils/current/
    zip.exe -r -1 arch1 *
    ...
    zip.exe -r -9 arch9 *

    Then use a binary diff utility to see which zip level matches the data better (since some zip headers would be different in any case).

  3. #3
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    19
    Thanks
    11
    Thanked 2 Times in 2 Posts
    Thanks for the comments!

    I must be missing something(?) But how does precomp determine which compressor was used to create the file images.dta?

    I tried the zip.exe from the link, but this compressor doesn't produce the right file size despite trying different compression settings.

    The idea is to archive various programs (games and utilities), but to compress them as small as possible before archiving. Part of this process may require compressing parts of software which are already compressed as well.

    This is the case here for this particular game where the images.dta file is compressed. It is wanted to decompress this file. Then compress the file contents using precomp and/or other compressors to higher compression ratio. Then compress the entire game for archiving.

    Later, software (game) is de-compressed. Then the further compressed parts of the software are also de-compressed. Then, ideally, the de-compressed parts are re-compressed using the compressor which which used to compress software parts originally (by the developer). In this case here, the file images.dta must be re-compressed using the same original compressor (OR using another compressor, provided the file size output is the same and the content is the same as the original images.dta file).

    ?

  4. #4
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > I must be missing something(?)
    > But how does precomp determine which compressor was used to create the file images.dta?

    It doesn't.
    Previously precomp just looked for zlib options to match the detected zlib streams...
    like I described with zip.exe, but automatically.
    Now (v4.7) it uses this library: https://github.com/deus-libri/preflate
    Which in addition to actual content, which can be unpacked from a deflate stream,
    also additionally stores encoder choices that affect the compression output.

    > I tried the zip.exe from the link, but this compressor doesn't produce the
    > right file size despite trying different compression settings.

    As I said, you'd have to use diff on created zips and original file.
    Aside from actual compressed data, there're also multiple ways to write the zip headers -
    zlib is the library most frequently used for zip compression, but there're
    much more libraries to create the headers.

    https://github.com/sisong/HDiffPatch/releases

    > The idea is to archive various programs (games and utilities), but to
    > compress them as small as possible before archiving. Part of this process
    > may require compressing parts of software which are already compressed as well.

    Unfortunately you are not the first to invent this.
    This is called "repacking" and is quite popular.
    There're also many tools created specifically for this.
    For example: http://aluigi.altervista.org/quickbms.htm

    > In this case here, the file images.dta must be re-compressed using the same
    > original compressor (OR using another compressor, provided the file size
    > output is the same and the content is the same as the original images.dta file).

    Yes, but its not very simple.
    Basically a recompression tool has to be written for every codec specifically.

    Then, container headers can be easier to reproduce, but they are usually
    different in every game, so this requires some programming.

    Or you can use some binary diff tool to patch the recompressed file
    into actual game file.

  5. The Following User Says Thank You to Shelwien For This Useful Post:

    brispuss (10th May 2019)

  6. #5
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Quote Originally Posted by brispuss View Post
    The file images.dta is 212 190 501 bytes in size.

    The file seems to extract OK using 7zip. But on re-compressing file contents (as a zip file), the compressed file size does not match the original file size, despite trying different compression settings!
    The file is a ZIP archive indeed, but everything is stored (because the PNG files wouldn't compress much). You get a very similar file when using the option "ZIP" and "Store" in 7-Zip:

    Code:
    212.190.501 images.dta
    212.265.025 images.zip
    http://schnaader.info
    Damn kids. They're all alike.

  7. The Following User Says Thank You to schnaader For This Useful Post:

    brispuss (10th May 2019)

  8. #6
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    19
    Thanks
    11
    Thanked 2 Times in 2 Posts
    I can see there are some difficulties here!

    Apart from decompressing many compressed file formats, Quickbms can only update existing compressed files, it can't (re)create (new) compressed files. So this utility is not entirely useful in my case, since I need compressors that must be able to recreate original files (exactly) without the original files being present.

    The intention is to archive software. After archiving, the original software will be deleted.

    When decompressing archived software, the decompression and any re-compression must be such that it reproduces that same content as per the original content without having to compare the files to original files (which are no longer available as they had been deleted to save HDD space).

    So trying to use a binary difference patching utility is not going to work unless the original files are re-downloaded to use as a comparison to the decompressed/re-compressed files. This is not really the best option, as I'm trying to avoid extra steps in "re-creating" original software.

  9. #7
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Maybe you misunderstood something, but what I suggested is this:
    1) you have 1.zip and want to recompress it
    2) so you do something, but end up with 2.zip!=1.zip
    3) in this case, a patch can be created, like diff 2.zip 1.zip 2-to-1.patch
    4) so in your installer, you'd have the data necessary to make 2.zip and 2-to-1.patch,
    and after 2.zip is created, you can patch it back to 1.zip

    Yes, its a somewhat ugly method, but compression-wise its usually ok, if the patch has a reasonable size.

    Otherwise you'd have to reverse-engineer and write tools for working with specific game formats,
    basically new tools for every game.

    > Quickbms can only update existing compressed files

    Its also not an issue, since you can zero out the place used by extracted files in the container.

  10. #8
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    19
    Thanks
    11
    Thanked 2 Times in 2 Posts
    OK. Before deleting the original file and after creating a decompressed test file as a test, THEN do a comparison check between the original and the decompressed test files. For any differences between the two files, use a patching utility to apply a patch to the decompressed test file to make it match the original file.

    Any suggestions on a good reliable file patcher?

    Thank you.

  11. #9
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    https://github.com/sisong/HDiffPatch/releases as I said.
    It also has limited support for deflate recompression already integrated.

  12. The Following User Says Thank You to Shelwien For This Useful Post:

    brispuss (12th May 2019)

  13. #10
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    19
    Thanks
    11
    Thanked 2 Times in 2 Posts
    Thanks.

    Sorry, I missed the first post for this utility!

    Command line based, with five executables to choose from(!?).

    Although command syntax is listed when running any of the executables with no parameters, it is not entirely clear which executable I should use to create a patch based on differences between two files?

    Frankly I find these executables difficult to understand and use!? Are there no "driving instructions" for these executables?

    Are there easier to understand and to use alternative file differential patchers rather than this one? Preferably with GUI?

  14. #11
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > Command line based, with five executables to choose from(!?).

    Yes, with recompression and without.

    > Although command syntax is listed when running any of the executables with
    > no parameters, it is not entirely clear which executable I should use to
    > create a patch based on differences between two files?

    Just read the first line from each utility?
    Code:
    hdiffz  [options] oldPath newPath outDiffFile
    hpatchz [options] oldPath diffFile outNewPath
    > Frankly I find these executables difficult to understand and use!?
    > Are there no "driving instructions" for these executables?

    Just run hdiffz > hdiff.txt and read hdiff.txt

    > Are there easier to understand and to use alternative file differential patchers rather than this one?

    Not really. There're plenty of such utilities: https://en.wikipedia.org/wiki/Delta_encoding#VCDIFF
    But most have some problems with multi-GB files etc.

    > Preferably with GUI?

    Maybe "Beyond Compare"? I never needed one, but you can look.
    In any case, do you expect it to generate an sfx for you or what?

  15. #12
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    19
    Thanks
    11
    Thanked 2 Times in 2 Posts
    Thanks. But the nomenclature/syntax is slightly confusing (for me).

    No sfx required.

    But going back to an earlier reply -

    Quote Originally Posted by Shelwien View Post

    > Quickbms can only update existing compressed files

    Its also not an issue, since you can zero out the place used by extracted files in the container.
    What exactly do you mean by that? Can you explain in a bit more detail please?

  16. #13
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    quickbms only needs the container headers, both for export and reimport.
    So if we'd zero out the compressed data in the container file, this "empty" container can be compressed to a small enough size,
    and then you can unpack it and reimport recompressed files.
    As to specifically making this empty container, I don't have an universal solution - quickbms prints offset/size for exported files,
    so I'd make a script which would use these.
    For example, this is something similar: http://nishi.dreamhosters.com/u/lzmadump_v1.rar

    Afaik, there's an utility specifically for this, that takes a filelist, and removes the data of files in this list from
    a given container file, but I don't have it.

  17. The Following User Says Thank You to Shelwien For This Useful Post:

    brispuss (14th May 2019)

Similar Threads

  1. Kitty file compressor (Super small compressor)
    By snowcat in forum Data Compression
    Replies: 7
    Last Post: 26th April 2015, 16:46

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •