Page 1 of 2 12 LastLast
Results 1 to 30 of 32

Thread: Precomp 0.4.2

  1. #1
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts

    Precomp 0.4.2

    Precomp 0.4.2 is out. This mainly improves stability and the user interface. And there finally is a Linux version, but as it is the first one, I'd appreciate if people test it for any platform specific bugs or problems.

    List of changes:

    • Linux version!
    • Added CTRL-C detection and temporary files cleanup.
    • Added activity indicator.
    • Added optional recursion depth limit for intense and brute mode.
    • Added longhelp switch.
    • Renamed -l (recursion level) to -d (recursion depth).
    • Renamed -slow to -intense.
    • Changed percentage display to use 2 decimal places.
    • Streams that can be completely recompressed are processed about 30% faster now.
    • Various small bugs fixed that led to incorrect recompression.

    Have a look at http://schnaader.info/precomp.php
    http://schnaader.info
    Damn kids. They're all alike.

  2. #2
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    Superb a new version thanks!
    Instead of intense, maybe insane is a better word? or extreme. Intense is not quite the right word imo.

  3. #3
    Member
    Join Date
    Mar 2010
    Location
    Germany
    Posts
    116
    Thanks
    18
    Thanked 32 Times in 11 Posts
    Thanks for the new version.

    Wish: As far as i understand, the "packjpg_dll.dll" is only used to recompress jpg's (maybe pdf's too).
    Can you please add a way, so that this dll is not need by precomp, when there is e.g. "-t-j" option is giving?
    This would made precomp very handy (one file), when only zlib recompression is needed.

  4. #4
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Quote Originally Posted by Intrinsic View Post
    Superb a new version thanks!
    Instead of intense, maybe insane is a better word? or extreme. Intense is not quite the right word imo.
    Always keep in mind there is brute mode, too. I think it's important to be able to get an impression of the increasing effort and "intense" and "brute" work quite well for this. "insane" and "extreme" are - well, extremes and this would mean we had to rename brute mode to "very insane" or "more extreme" mode to still get this impression.

    Also note the "optional recursion depth limit" change. You can use "-intense0" now to apply intense mode only to the "base file" and it will be disabled in recursion. This makes the mode more usable. It still is slow, yes, but it also is very useful for some files as it'll find much more streams for those. So with the new version, there's an additional way to vary the mode, affecting its speed quite a lot.

    Quote Originally Posted by Biozynotiker View Post
    Wish: [...]
    Can you please add a way, so that this dll is not need by precomp, when there is e.g. "-t-j" option is giving?
    This would made precomp very handy (one file), when only zlib recompression is needed.
    There still will be zlib1.dll left, so it would be two files anyway. In version 0.4, I made an attempt to static link zLib which led to various crashes and errors, so I won't do further experiments with this one for now.

    As for using packjpg_dll.dll only when needed, I'll have a look at it. The idea to do so was there before, but I switched my programming environment to a new PC, so I set aside some minor changes like this one so I can be sure that everything runs fine as it is.
    Last edited by schnaader; 26th September 2011 at 14:14.
    http://schnaader.info
    Damn kids. They're all alike.

  5. #5
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    it's easy to dynamically load fucntions from dll:


    // Loads function from a unarc.dll
    FARPROC LoadFromDLL (char *funcname)
    {
    static HMODULE unarc_dll = 0;
    if (!unarc_dll)
    unarc_dll = LoadLibrary("unarc.dll");
    return GetProcAddress (unarc_dll, funcname);
    }

    main()
    {
    ...
    // Unload unarc.dll
    void (*UnloadDLL) (void) = (void (*) (void)) LoadFromDLL("UnloadDLL");
    if (!UnloadDLL) {printf("Cannot load UnloadDLL() from unarc.dll\n"); return 1;}
    (*UnloadDLL) ();
    }

  6. #6
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    it's easy to dynamically load functions from dll:
    [...]
    Thanks for the code. I'm looking forward to try it out, I found similar fragments before, but they often were confusing, inconsistent or simply didn't work.
    http://schnaader.info
    Damn kids. They're all alike.

  7. #7
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    http://nishi.dreamhosters.com/u/precomp042.exe
    http://nishi.dreamhosters.com/u/dllmerge-precomp042.rar

    With dynamic dll load, like what Bulat suggests, it would be impossible to automatically merge the dlls -
    I'd also have to patch the dll load code, so better don't.

  8. #8
    Member
    Join Date
    Jul 2011
    Location
    india
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts
    not able to recompress some .ff files from Call of duty

  9. #9
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    Why do you need to recompress them?

  10. #10
    Member
    Join Date
    Nov 2011
    Location
    Poland
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by lordvj View Post
    not able to recompress some .ff files from Call of duty
    Did you try

    Code:
    precomp042 -c- -brute

  11. #11
    Member
    Join Date
    Jul 2011
    Location
    india
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts
    brute will take ages

  12. #12
    Member
    Join Date
    Sep 2011
    Location
    Yan
    Posts
    10
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by lordvj View Post
    not able to recompress some .ff files from Call of duty
    I am game ripper/cracker so I have many experienced in compressing game into superb extreme compression..
    The first thing is too study how precomp work,and what type of data that precomp will pre-compress it..some of example is like this

    Example you have 1.ff..you compressed it with 7-zip..but the size not decreased at all isn't..you have to use precomp to "pre-compressed" the stream on that file and the truly the file should be bigger after you make pre-compression with it..but if you compressed the file that have been "pre-compressed" file with maximum fucking higher compression method..the size of file should be decrease compared than original file...i usually use CCM because the executable is small and the compression is fucking best than other compresser

    You can use my method here :-

    Original File -> Precomp -> CCM Compressor

    1.ff (100mb) -> 1.pcf (500mb) -> 1.ccm (50->70mb) [just theory]

    this is little example made by me

    I hope this is usefull

  13. #13
    Member
    Join Date
    Jul 2011
    Location
    india
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts
    thanks for the info, but that is nothing new to me , i'm aware of how it works and had used it on most cod .ff and it compressed really well , what i wasnt able to recompress using precomp were the multiplayer .ff files

    I use srep and 7z after precomp, cos ccm decompression is slow
    Last edited by lordvj; 19th November 2011 at 19:40.

  14. #14
    Member
    Join Date
    Sep 2011
    Location
    Yan
    Posts
    10
    Thanks
    0
    Thanked 0 Times in 0 Posts
    or you can use FreeARC using ccm option..for audio you can compress using Nero ACC codec..

  15. #15
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    872
    Thanks
    457
    Thanked 175 Times in 85 Posts
    Hi there

    precomp -c- crashes @ 98.45% of Installer.tar testset and is slow on Office.tar testset (2000 bytes written in 5 minutes -> MIME).

  16. #16
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Quote Originally Posted by Stephan Busch View Post
    Hi there

    precomp -c- crashes @ 98.45% of Installer.tar testset and is slow on Office.tar testset (2000 bytes written in 5 minutes -> MIME).
    The crash sounds like a PackJPG crash, so -t-j could help.

    There is a known bug (will be fixed in the next version) that makes Precomp freeze on some incomplete/invalid Base64 streams, so this could be the reason for Office.tar, you can try -t-m.
    http://schnaader.info
    Damn kids. They're all alike.

  17. #17
    Member
    Join Date
    Jul 2011
    Location
    india
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts
    is stdin stdout support planned for the next version?

  18. #18
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    It seems odd that an archive i successfully created previously using precomp(0.4.0) -> ccm now crashes at 0.13% unless i use -t-j, but when i do that the archive is a lot bigger. If i use -c- it doesn't crash and is the size i'd expect. I can provide what i'm compressing if you need it. It has no JPEGs in it, but does have libgfl282.dll and libgfle282.dll which contain the string JFIF.

    Edit: It crashes if i use -cb fyi at the same point, i guess -cb is default?
    Last edited by Intrinsic; 4th December 2011 at 19:48.

  19. #19
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Quote Originally Posted by Intrinsic View Post
    It seems odd that an archive i successfully created previously using precomp(0.4.0) -> ccm now crashes at 0.13% unless i use -t-j, but when i do that the archive is a lot bigger
    [...]
    -cb is default, yes. It uses bZip2 compression which will be much worse than using -c- (the "old" behaviour of Precomp 0.4) together with ccm. The PCF file created with "precomp -cb" will be compressed, so using CCM afterwards won't help and the result will be bigger than "precomp -c- + CCM". However, this doesn't explain the crash occuring with "(-cb) -t-j", but not with "-c-" - if PackJPG was the reason for the crash, only "-c- -t-j" would work.

    I'd be interested in a file to reproduce and fix the problem. You could try to test if the crash remains with an archive that only contains the two DLLs, but at the moment I suspect this to be a "silent" PackJPG error that somehow "confuses" the bZip2 part of Precomp afterwards, so it might as well disappear on a smaller testfile.
    http://schnaader.info
    Damn kids. They're all alike.

  20. #20
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    It seems to be PNG files, i can actually place any PNG file into this directory and it'll crash, if i remove them it works fine. If i just click the close button on the crash dialogue and then just leave it instead of forcing everything shut, sometimes it will actually end up compressing the files and they all unpack fine, other times the archive will be very small.

    Have attached a few images with -v on. PNGs.png is it with the original PNGs in there, if i put in one of my own i get the PNGs-2.png extra message about MultiPNG even if only 1 PNG is in there.

    Edit1: Oh yeah i have tried just compressing these files and it works fine. Have also tried putting these PNGs into other dirs with files that i know cause no problems and it works fine.

    Edit2: As a test i put the dll files mentioned above into the same dir as mentioned in edit1 and BOOM it crashed again...ODD further, if i remove either one of the dlls it works fine, but when they are together BOOM. On their own the dlls compress fine together, if i add any type of PNG to the dir then BOOM. So it seems some odd combination that's freaking something out. I have attached the offending DLLs with the pngs from the same dir, although any PNG makes it freak out it seems.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	No PNGs.png 
Views:	629 
Size:	10.1 KB 
ID:	1750   Click image for larger version. 

Name:	PNGs.png 
Views:	585 
Size:	11.1 KB 
ID:	1751   Click image for larger version. 

Name:	PNGs-2.png 
Views:	549 
Size:	11.0 KB 
ID:	1752  
    Attached Files Attached Files
    Last edited by Intrinsic; 5th December 2011 at 01:21.

  21. #21
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    I did some testing now, too, and it's definitely an error with PackJPG. I couldn't reproduce a crash using "-t-j", only crashes where JPEG recompression was enabled. As you can also see in your screenshots, PackJPG always gives an "unexpected end of data" error and seems to corrupt memory somehow. In the cases that are working, there isn't much additional data (like the PNGs), so Precomp is "lucky" and doesn't crash.

    Looking at the problems with PackJPG so far and strange bugs like this one, I'll definitely disable JPG recompression by default in the next version. Perhaps I'll have a look at the new version and the source code of PackJPG 2.5, it seems to handle invalid JPGs better and didn't crash with test files I tried so far.
    http://schnaader.info
    Damn kids. They're all alike.

  22. #22
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    Have you tried using the latest PackJPEG sources to build the dll? the one with precomp is from 2008.

    Edit1: I think this is actually a problem with precomp detecting what it thinks is a JPEG, as if i put different files into that directory i notice that the length that it thinks the JPEG is will change. Do you search for a certain series of bytes and guess that is the end? all normal JPEGs will end in FF D9 but of course that can appear anywhere in any file.

    And as an example here is a test just now that gives that "unexpected end of data" and works fine, and i think i have narrowed the problem down to "libgfl282.dll" as i can get it to crash with just that and another file in there...although just trying again now i can't seem to find which combo of files caused it to freakout with just this dll ;p

    (67.74%) Possible JPG found at position 1015788, length 98513
    packJPG error: unexpected end of data encountered
    No matches
    (69.37%) Possible GIF found at position 1040140
    (69.37%) Possible GIF found at position 1040148
    New size: 642202 instead of 1499478

    Done.
    Time: 312 milliseconds

    Recompressed streams: 0/0

    Errorlevel=2

    Edit2: I notice drwatson is creating a stack trace and dump, would you like a copy of those?
    And i mainly seem to get it to crash by including other dlls like libpng, libtiff. If i try other non-image releated dlls it doesn't crash like ones from gtk i took from the freearc directory.
    And one thing to point out with your comment about removing jpeg recompression, is that this same archive compresses without issue in precomp 0.4.0 so i'd say it's unfair to blame packjpg:

    Using precomp 0.4:
    Possible JPG (progressive) found at position 5940245, length 23363
    Best match: 23363 bytes, recompressed to 21093 bytes
    New size: 5962974 instead of 5963608

    Done.
    Time: 453 ms

    Recompressed streams: 2/2
    PNG streams: 1/1
    JPG streams (progressive): 1/1

    You can speed up Precomp for THIS FILE with these parameters:
    -c6 -m4 -l0

    Fast mode does exactly the same for this file, only faster.

    Errorlevel=0

    Using precomp 0.3.8:
    Possible JPG (progressive) found at position 5940245, length 23363
    Best match: 23363 bytes, recompressed to 21093 bytes
    New size: 5962974 instead of 5963608

    Done.
    Time: 421 ms

    Recompressed streams: 2/2
    PNG streams: 1/1
    JPG streams (progressive): 1/1

    You can speed up Precomp for THIS FILE with these parameters:
    -c6 -m4

    Fast mode does exactly the same for this file, only faster.



    I personally never had any issues with jpegs in precomp 0.3.8 or 0.4 from what i can remember. It's only since 0.4.1 and 0.4.2.
    Last edited by Intrinsic; 5th December 2011 at 18:33.

  23. #23
    Member
    Join Date
    Jul 2011
    Location
    india
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts
    just managed to extract the zlib content of the .ff files using offzip.exe. why doesnt precomp find those streams?

    edit : oh nevermind precomp could recompress them too ... its the MP.ff files that could'nt be recompressed offzip says the data is not zip . meh
    Last edited by lordvj; 5th December 2011 at 20:45.

  24. #24
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Quote Originally Posted by Intrinsic View Post
    Have you tried using the latest PackJPEG sources to build the dll? the one with precomp is from 2008.
    This was what I meant with "looking at the new version" - so far, Matthias Stirner sent me a built DLL and I just didn't have time to try building it myself from the source code that is available now.

    Quote Originally Posted by Intrinsic View Post
    I think this is actually a problem with precomp detecting what it thinks is a JPEG
    [..]
    And one thing to point out with your comment about removing jpeg recompression, is that this same archive compresses without issue in precomp 0.4.0 so i'd say it's unfair to blame packjpg:
    Well, this was my mistake - I made it sound like it's a pure PackJPG problem. Of course, Precomp doesn't have a full JPG detection and it could solve some of the crashes with an improved detection. But another part of the problem is PackJPG crashing or exiting with corrupt stack/memory for some corrupt JPGs (for an example, see http://encode.ru/threads/1243-Open-s...ll=1#post24398 - this file leads to a crash in all Precomp versions and even for official PackJPG versions before 2.5 and rejpeg). It seems PackJPG 2.5 handles this better, so building a new DLL and using it could indeed solve the problems.

    I experimented a bit with improved JPG detection before, but this also prevents valid JPEGs from being detected (f.e. thumbnails in JPEGs) that aren't 100% standard JPEGs but tolerated by most software.

    Also note that I don't want to remove JPG recompression, I just want to disable it by default and add a switch like "-jpg+" to enable it.

    Quote Originally Posted by Intrinsic View Post
    Edit2: I notice drwatson is creating a stack trace and dump, would you like a copy of those?
    Unfortunately, stack traces aren't useful here - I have a lot of testfiles for the "bad JPG" problem and can trace everything down to the exact line of code that goes wrong - but it always is some random line that usually works.
    http://schnaader.info
    Damn kids. They're all alike.

  25. #25
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    Yes i can imagine it being really hard to create one algo that will find a jpeg hidden in a file with 100% certainty and retain the speed. Have you thought about maybe using something like libjpeg-turbo to check if the jpeg is valid before passing it onto packjpg? or would that slow everything up too much?

    A switch to re-enable it would be good. I think that it'd be easy enough just to process jpegs before hand anyways if using freearc as there aren't likely to be *that* many jpegs hiding inside other files, apart from PSDs/PDFs.

  26. #26
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Good news: I just built the PackJPG 2.5 DLL and additionally to a minor speed anc compression gain, it also handles invalid input much better. Both the corrupt JPG part.jpg and your testfiles give "unexpected end of data encountered", but there's no crash anymore.

    The next thing to do will be to try compiling PackJPG for the Linux version so it can recompress JPEG files, too. The Linux version is another reason for disabling JPEG recompression by default because this way, the PCF files are exchangeable until the Linux version has PackJPG support, too - the current version shows an error if the Linux version stumbles upon a recompressed JPEG.
    Last edited by schnaader; 6th December 2011 at 22:38.
    http://schnaader.info
    Damn kids. They're all alike.

  27. #27
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    Sounds like great news, I look forward to a version to try out

  28. #28
    Member
    Join Date
    Jun 2012
    Location
    South Africa
    Posts
    15
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by schnaader View Post
    Good news: I just built the PackJPG 2.5 DLL and additionally to a minor speed anc compression gain, it also handles invalid input much better. Both the corrupt JPG part.jpg and your testfiles give "unexpected end of data encountered", but there's no crash anymore.

    The next thing to do will be to try compiling PackJPG for the Linux version so it can recompress JPEG files, too. The Linux version is another reason for disabling JPEG recompression by default because this way, the PCF files are exchangeable until the Linux version has PackJPG support, too - the current version shows an error if the Linux version stumbles upon a recompressed JPEG.
    Do you still have the PackJPG 2.5 DLL? The 2.4 version is too unstable.

  29. #29
    Member
    Join Date
    Jun 2012
    Location
    South Africa
    Posts
    15
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I've got a couple of zipped video filters that Precomp 0.4.2 doesn't work on.

    Here's one of the zipped files: http://www.mediafire.com/?8jg786a07tj9n0a

    The dialogue in the Precomf command window says the following:

    100% - New size: 340091 instead of 340060

    Done.
    Time: 1 seconds, 515 milliseconds

    Recompressed streams: 0/8
    ZIP streams: 0/6
    zLib streams (intense mode): 0/2

    None of the given compression and memory levels could be used.
    There will be no gain compressing the output file.

    Press any key to continue

    I checked to ensure that ZIP compression is enabled in the precomp.ini file.

    Anyway idea why Precomp won't decompress the zip files?

  30. #30
    Member
    Join Date
    Mar 2010
    Location
    Germany
    Posts
    116
    Thanks
    18
    Thanked 32 Times in 11 Posts
    Quote Originally Posted by MiY4Gi View Post
    Anyway idea why Precomp won't decompress the zip files?
    AFAIR, precomp can only handle zips, that is able to de/compress by zlib.dll.

Page 1 of 2 12 LastLast

Similar Threads

  1. Precomp (and Precomp Comfort) in 315 kb
    By Yuri Grille. in forum Data Compression
    Replies: 2
    Last Post: 1st April 2009, 19:40

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •