Page 1 of 2 12 LastLast
Results 1 to 30 of 38

Thread: xeloz

  1. #1
    Member
    Join Date
    Dec 2012
    Location
    japan
    Posts
    149
    Thanks
    30
    Thanked 59 Times in 35 Posts

    xeloz

    I made a new compressor.
    xeloz is byte-aligned LZSS with a 264KB window.
    This version is slow yet.
    Code:
    usage: xeloz command[switch] infile outfile
    command:
     c compress.
     d decompress.
    switch:
     1-100000 max chain length. default is 256. higher is better compression.
    
    
    result
    xeloz c enwik6 out : 377714
    xeloz c enwik7 out : 3877852
    xeloz c enwik8 out : 38443732
    xeloz c8 enwik8 out : 41845542
    Attached Files Attached Files

  2. The Following 3 Users Say Thank You to xezz For This Useful Post:

    Bulat Ziganshin (10th November 2015),Nania Francesco (16th June 2014),tnagylev (19th June 2014)

  3. #2
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    772
    Thanks
    63
    Thanked 270 Times in 190 Posts
    Nice job, very quick.

    enwik9 338,156,426 bytes, 311.884 sec. (xeloz c) but decompress crash.

  4. #3
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    The program go in crush in GAME1 test of WCC2014! Looks like a classic mistake from memory overrun!

  5. #4
    Member
    Join Date
    Dec 2012
    Location
    japan
    Posts
    149
    Thanks
    30
    Thanked 59 Times in 35 Posts
    Thanks Sportman, Nania.
    enwik9 338,156,426 bytes, 311.884 sec. (xeloz c) but decompress crash.
    Looks like a classic mistake from memory overrun!
    Please wait for fix.
    Attached Files Attached Files
    Last edited by xezz; 17th June 2014 at 16:10.

  6. #5
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    With new version the program go in crush in GAME1 test of WCC2014!

    If you have trouble you can put the part of code online to help!

  7. #6
    Member
    Join Date
    Dec 2012
    Location
    japan
    Posts
    149
    Thanks
    30
    Thanked 59 Times in 35 Posts
    OK, but code is not clean yet.
    Attached Files Attached Files
    Last edited by xezz; 18th June 2014 at 12:09.

  8. #7
    Member RichSelian's Avatar
    Join Date
    Aug 2011
    Location
    Shenzhen, China
    Posts
    156
    Thanks
    18
    Thanked 50 Times in 26 Posts
    Quote Originally Posted by xezz View Post
    OK, but code is not clean yet.

  9. #8
    Member
    Join Date
    Dec 2012
    Location
    japan
    Posts
    149
    Thanks
    30
    Thanked 59 Times in 35 Posts
    add comment, variable name is changed.
    Attached Files Attached Files

  10. #9
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Quote Originally Posted by xezz View Post
    add comment, variable name is changed.
    I spent one afternoon to change the parameters and gives me the same error in decompression. I hope to find the error soon!

  11. #10
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    I did other tests on the code and increasing the buffer you can see how formally there are no errors with both encoder and decoder. Trying to reduce the buffer you can see that is not passed some important data and there is not window buffer overrun. I will continue to experiment, but frankly, I suggest you create the code again making it more mathematically simple and effective.
    Last edited by Nania Francesco; 20th June 2014 at 12:25.

  12. #11
    Member
    Join Date
    Dec 2012
    Location
    japan
    Posts
    149
    Thanks
    30
    Thanked 59 Times in 35 Posts
    Thank you very much, Nania!
    Fixd bug in decoder.
    Attached Files Attached Files

  13. #12
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Thank you very much, Nania!
    Fixd bug in decoder.
    I'm glad you fixed the bug. The program looks interesting. With a greater recourse to memory and then increase the speed of decompression of 30-50%

  14. #13
    Member
    Join Date
    Dec 2012
    Location
    japan
    Posts
    149
    Thanks
    30
    Thanked 59 Times in 35 Posts
    Max window size is up to 17MB. default is about 519KB.
    Large window requires long chain(if short, result is worse).
    Code:
    xeloz c256:w0 in out // chain 256, window 72KB
    xeloz c4096:w8 in out // chain 4096, window 17MB
    Attached Files Attached Files

  15. #14
    Member
    Join Date
    Dec 2012
    Location
    japan
    Posts
    149
    Thanks
    30
    Thanked 59 Times in 35 Posts
    Released new version. Added some parameters.
    Code:
    usage
    to compress: xeloz c|C[#1][#2][#3] infile [outfile]
    c  slide window, better compression.
    C  fixed window, about 1.5 times faster than c.
    #1 0..8  window size. 2^(16-24) + X. default is 1031KB
    #2 0..9  select parser and finder. higher is better compression but slow. default is 6.
        0..2  greedy parser
        3..5  lazy parser
        6..8  optimal parser
    #3 0..9  if #2 isn't 0,3,6, it means encoding level. default is 2
    
    
    to decompress: xeloz d infile [outfile]
    Attached Files Attached Files

  16. The Following User Says Thank You to xezz For This Useful Post:

    Gonzalo (3rd September 2014)

  17. #15
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    323
    Thanks
    174
    Thanked 51 Times in 37 Posts
    Quote Originally Posted by xezz View Post
    Released new version. Added some parameters.
    Code:
    usage
    to compress: xeloz c|C[#1][#2][#3] infile [outfile]
    c  slide window, better compression.
    C  fixed window, about 1.5 times faster than c.
    #1 0..8  window size. 2^(16-24) + X. default is 1031KB
    #2 0..9  select parser and finder. higher is better compression but slow. default is 6.
        0..2  greedy parser
        3..5  lazy parser
        6..8  optimal parser
    #3 0..9  if #2 isn't 0,3,6, it means encoding level. default is 2
    
    
    to decompress: xeloz d infile [outfile]
    Downloaded and testing now...Thanks!

    EDIT This version requires libgcc_s_dw2-1.dll and libstdc++-6.dll. Xeloz crashes on every file on every setting. Version 0.2 works perfectly though. System is Win7 64-bit with 16GB RAM.
    Last edited by comp1; 3rd September 2014 at 21:41.

  18. #16
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    464
    Thanks
    202
    Thanked 81 Times in 61 Posts
    Thanks! But not working for me - XP SP3 1g ram 1.6 ghz amd single core processor.
    No error code, no 'usage' screen, no message at all... Just run then die

  19. #17
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    464
    Thanks
    202
    Thanked 81 Times in 61 Posts
    EDIT: never mind. My mistake. Just needed the DLLs

  20. #18
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    464
    Thanks
    202
    Thanked 81 Times in 61 Posts
    Well... 1st run: Just xeloz c in out:

    Exception Code: 0xc0000005, Parameters: , 0x00000000, 0x0000215e
    at memory 0x6fc862f3
    Last edited by Gonzalo; 3rd September 2014 at 22:28.

  21. #19
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    323
    Thanks
    174
    Thanked 51 Times in 37 Posts
    Update:

    As a matter of fact: I went to http://www.hyec.org/~da0ka/c/compress/ and tried all 0.3.x versions and they all crash.

    0.3.0 doesn't require the DLL files but the program still crashes. No error message is given.

    All tests were on Win7 x64, 16GB RAM.

  22. The Following User Says Thank You to comp1 For This Useful Post:

    Nania Francesco (4th September 2014)

  23. #20
    Member
    Join Date
    Dec 2012
    Location
    japan
    Posts
    149
    Thanks
    30
    Thanked 59 Times in 35 Posts
    Oh,no... Error is too many but my system reports no error.
    I tested on Windows XP SP3. compiler is gcc 4.8.1.
    Code:
    g++ -O2 -s divsufsort.c xeloz-0.3.5.3.cpp -o xeloz
    Well... 1st run: Just xeloz c in out:
    Exception Code: 0xc0000005, Parameters: , 0x00000000, 0x0000215e at memory 0x6fc862f3
    Memory overrun. Maybe check code is bad.
    How about "xeloz c41 in out" or "xeloz c47 in out" ?
    all 0.3.x versions and they all crash
    Those are still test versions. Hmm.. I tested but not crash...

  24. #21
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    464
    Thanks
    202
    Thanked 81 Times in 61 Posts
    > How about "xeloz c41 in out" or "xeloz c47 in out" ?

    Done. Same result. Identical info from debugger.

    EDIT:The output file is created. It always consist in one byte: "„" (84 hex)

    Any programmer? Dealing with tricky fixes or compiler option stills feeling like try to cure cancer to me...
    Last edited by Gonzalo; 4th September 2014 at 22:42.

  25. #22
    Member
    Join Date
    Dec 2012
    Location
    japan
    Posts
    149
    Thanks
    30
    Thanked 59 Times in 35 Posts
    @comp1, Gonzaro

    Does it work on your system? It computes the size of LZ77 parsing(greedy) only.
    Be careful, block is single.
    Code:
    usage: lz infile [kkp2|kkp3]
    If it crashes, hard to correct.
    Attached Files Attached Files
    • File Type: 7z lz.7z (34.7 KB, 89 views)

  26. #23
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    464
    Thanks
    202
    Thanked 81 Times in 61 Posts
    Yup. Sadly... Error:
    Event Time Event Type Thread ID Memory Address More Information
    00:00:00.046 Exception 3464 0x6fcaeeef Exception Code: 0xc0000005, Parameters: , 0x00000001, 0x00000004
    00:00:00.046 Exception 3464 0x6fcaeeef Exception Code: 0xc0000005, Parameters: , 0x00000001, 0x00000004
    00:00:00.062 Exit Process 3464 Exit Code: 0xc0000005
    Last edited by Gonzalo; 5th September 2014 at 16:33.

  27. #24
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    323
    Thanks
    174
    Thanked 51 Times in 37 Posts
    Still crash...

  28. #25
    Member
    Join Date
    Dec 2012
    Location
    japan
    Posts
    149
    Thanks
    30
    Thanked 59 Times in 35 Posts
    Maybe dll's problem, so added option '-static' to gcc.
    Attached Files Attached Files

  29. #26
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    323
    Thanks
    174
    Thanked 51 Times in 37 Posts
    Quote Originally Posted by xezz View Post
    Maybe dll's problem, so added option '-static' to gcc.
    YES IT WORKS!!!!!

    Thank you! That was what was needed

  30. #27
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    464
    Thanks
    202
    Thanked 81 Times in 61 Posts
    Yeah! worked for me too!
    Numbers incoming...

  31. #28
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    464
    Thanks
    202
    Thanked 81 Times in 61 Posts
    Well... First test.
    Data: Pseudo-random file, 1mb
    Method: All possible
    Ratio: from 112.441% to 112.455% - Almost the same.
    Time: Less than a half second - Exact timing in attachments (xeloz internal and Igors' timer)
    Crash in: Decompression with options C100, C101, C102 ... to C129; C160 to C199
    Decompresses wrong in: 702 to 899 - Both c and C (output = 0b)
    Decompresses wrong in: C100 to C129; C160 to C199 (output= 695035 bytes)
    The rest fine.

    PS: Why so many optionssssss!!!! Haha! Just kidding... But they are 1800 total permutations!
    Attached Files Attached Files
    Last edited by Gonzalo; 6th September 2014 at 00:32.

  32. #29
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    464
    Thanks
    202
    Thanked 81 Times in 61 Posts
    2nd test.
    Data: Windows executables - 342 files in a shelwien archive (.shar), 85.39m,
    no UPX-ed, pactically no resources like BITMAP, ICON or so on.
    Running full test.
    In the mean time, "C" crash in the first ten combinations in the following way:
    encoding
    895371 1 %
    ...
    17042351 19 %

    Exit code: -1073741819
    Last edited by Gonzalo; 6th September 2014 at 00:31.

  33. #30
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    464
    Thanks
    202
    Thanked 81 Times in 61 Posts
    Test 2 completed. Data set reduced in size: ~=5mb
    The ratio is too weak for the speed achieved. As for a comparison, Slugx compresses the same file slightly better than xeloz's strongest setting in the 0.67% of the time. (0.39s against 58.45s)
    *Really* impractical, at least for now... But it doesn't mean it's hopeless. It's a beta, after all.
    Happy packing!
    Attached Files Attached Files

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •