Results 1 to 8 of 8

Thread: Looking for standalone preprocessor for testing

  1. #1
    Member
    Join Date
    Feb 2013
    Location
    Canada
    Posts
    5
    Thanks
    3
    Thanked 0 Times in 0 Posts

    Looking for standalone preprocessor for testing

    I'm looking for a standalone preprocessor for testing my compressor - something that will transform the data to be more compressible; ie. for text etc.. I'm positive I've seen one here a couple of years ago, I just want one for quickly testing final results on top of my core compression. I'll end up focusing on that portion later, I need it purely for quick and dirty use without polluting my code at the moment.

    Thanks!

  2. #2
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 166 Times in 63 Posts

  3. #3
    Member
    Join Date
    Feb 2013
    Location
    Canada
    Posts
    5
    Thanks
    3
    Thanked 0 Times in 0 Posts
    Perfect it has a preprocessed and uncompressed output option!

  4. #4
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    bulat made delta which is a delta filtering preprocessor

    https://encode.ru/threads/1045-Delta...s-preprocessor
    Links are dead though

  5. The Following User Says Thank You to SvenBent For This Useful Post:

    chunick (9th February 2017)

  6. #5
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    216
    Thanks
    97
    Thanked 128 Times in 92 Posts
    Quote Originally Posted by SvenBent View Post
    bulat made delta which is a delta filtering preprocessor

    https://encode.ru/threads/1045-Delta...s-preprocessor
    Links are dead though
    Take a look here http://freearc.org/Research.aspx in "DELTA compression preprocessor", links work and newer version.

  7. The Following User Says Thank You to Mauro Vezzosi For This Useful Post:

    chunick (9th February 2017)

  8. #6
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    not sure what type of preprocessors you need, but now all my stuff is collected in the http://freearc.org/download/research/fazip03.zip - including exe, dispack070, mm, delta, and dict preprocessors

    f.e. "fazip delta <infile >outfile"

  9. #7
    Member
    Join Date
    Feb 2013
    Location
    Canada
    Posts
    5
    Thanks
    3
    Thanked 0 Times in 0 Posts
    very helpful thanks all! ... I was just looking to test my compressor with the benefits of different preprocessors, nothing particular but dict preprocessor is nice. I'm doing a lot of fine tuning and adjustments at the moment on my core code. I don't wan't to integrate the preprocessing stage at this point but would like to see if I get a significant boost as I'm adjusting - instead of just the raw data.

    Thanks!

  10. #8
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    and yeah, i forgot the rep filter. if you need parameters, this is a list of compression methods used in fa'next m1d..m9d modes:

    Compress with rep:512mb:c512+exe+4x4:zstd:2:8mb, $obj => rep:512mb:c512+4x4:zstd:2:8mb, $text => 4x4:zstd:2:8mb, $compressed => rep:512mb:c512

    Compress with rep:512mb:256:c256+exe+delta+4x4:zstd:7:16m:h8m, $obj => rep:512mb:256:c256+delta+4x4:zstd:7:16m:h8m, $text => delta+4x4:zstd:7:16m:h8m, $compressed => rep:512mb:256:c256+4x4:zstd:2:8mb, $wav => tta:m1, $bmp => mm:d1+4x4:zstd:2:8mb:t0

    Compress with rep:512mb+exe+delta+4x4:lzma:96mb:fast:32:mc4, $obj => rep:512mb+delta+4x4:lzma:96mb:fast:32:mc4, $text => dict:64mb:80%+4x4:lzma:96mb:fast:32:mc4:32m, $compressed => rep:512mb+4x4:zstd:2:8mb, $wav => tta:m1, $bmp => mm+4x4:lzma:96mb:fast:32:mc4

    Compress with rep:512mb+dispack070+delta+4x4:lzma:96mb:normal:16 :mc8, $obj => rep:512mb+delta+4x4:lzma:96mb:normal:16:mc8, $text => dict:64mb:75%+4x4:lzma:96mb:normal:16:mc8:32m, $compressed => rep:512mb+4x4:zstd:7:16m:h8m, $wav => tta:m1, $bmp => mm+4x4:lzma:96mb:normal:16:mc8

    Compress with rep:512mb+dispack070+delta+4x4:lzma:96mb:normal:12 8:mc32, $obj => rep:512mb+delta+4x4:lzma:96mb:normal:128:mc32, $text => dict:128mb:75%+4x4:lzma:96mb:normal:128:mc32, $wav => tta:m1, $bmp => mm+4x4:lzma:96mb:normal:128:mc32

    Compress with rep:2040mb+dispack070+delta+lzma:254mb:normal:bt4: 128, $obj => rep:2040mb+delta+lzma:254mb:normal:bt4:128, $text => dict:128mb:75%+lzma:254mb:normal:bt4:128, $wav => tta:m1, $bmp => mm+lzma:254mb:normal:bt4:128
    each "$group=method" describes compression method for some type of files. it seems that freearc/fazip implements all filters used in modern archivers in comination with LZ-based compressors, so you will not miss any
    Last edited by Bulat Ziganshin; 10th February 2017 at 05:54.

Similar Threads

  1. Testing for open-sources libraries on POWER8
    By nemequ in forum Data Compression
    Replies: 0
    Last Post: 2nd September 2016, 22:37
  2. Fuzz testing
    By m^2 in forum Data Compression
    Replies: 23
    Last Post: 10th January 2016, 10:01
  3. Dict preprocessor
    By pat357 in forum Data Compression
    Replies: 5
    Last Post: 2nd May 2014, 21:51
  4. Testing compressors with artificial data
    By Matt Mahoney in forum Data Compression
    Replies: 0
    Last Post: 25th April 2013, 08:22
  5. What's wrong with my testing script?
    By m^2 in forum Data Compression
    Replies: 20
    Last Post: 21st September 2008, 19:24

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •