Results 1 to 6 of 6

Thread: fast opensource multithread stidn packer

  1. #1
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    342
    Thanks
    12
    Thanked 34 Times in 28 Posts

    fast opensource multithread stidn packer

    i am searching for a freebsd compressor that con take data from pipe, with high parallelism (say 8 real threads) and high performances with low clock.
    currently i use pbzip2, but maybe there is something faster.
    data is zfs binary dump, so not much compressable.
    ideally i need 150/200MB/s with 6-8 threads at 2GHz

  2. #2
    Member
    Join Date
    Jun 2008
    Location
    G
    Posts
    372
    Thanks
    26
    Thanked 22 Times in 15 Posts
    7z with chunks in fastest mode. detects also when data is incompressable and just saves

    maybe zpipe ? but idont know if its multithreading

  3. #3
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    FreeArc doesn't build on FreeBSD.
    Plzip?

    The question is if you need a strong compressor at all. The best place to compress may (or may not) be zfs itself. Aside from that, if most of your data is already compressed, something like zstd may give you all the strength you need while using just 1 core for negligible strength difference.

  4. #4
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    342
    Thanks
    12
    Thanked 34 Times in 28 Posts
    Thanks to all answers.
    I do not need a strong compressor, because it's simply infeasible to have enough CPU power to process a stream at sustained 200MB/s.

    The data is a stream of a zfs filesystem (or better from a zfs snapshot), so it's almost all already compressed; in my case I am trying to gain say 10 or 20% when writing to the target system, because mechanical drive cannot write much more than 150-180 MB/s sustained.

    For those not familiar with zfs is possible to backup a "live" system (something like acronis etc) sending an entire disk (snapshot of a zpool, to be precise) to a file (or a pipe).

    Using very fast source (Xeon-SSD), high speed link (10Gb) the bottleneck became the target write bandwidth (aka the drive on the NAS), so every 1% of reduction will cut the time needed to complete a TB+ job.

    Xeon-servers tipically does not have much CPU power (low clock ~2GHz), but many physical cores (8 in my case), and plenty of RAM (128GB), and very fast disk (SSD or even better M2).

    I will try pigz and plzip

  5. #5
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    You may also try Bulat's rep/srep, either alone or in tandem with something else. Not sure if they support pipe IO though.

  6. #6
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    342
    Thanks
    12
    Thanked 34 Times in 28 Posts
    Quote Originally Posted by m^2 View Post
    You may also try Bulat's rep/srep, either alone or in tandem with something else. Not sure if they support pipe IO though.
    for now the winner is pigz. real fast indeed with many cored

Similar Threads

  1. Does Any Body Have Dir2pak packer ?
    By Manjunath in forum Download Area
    Replies: 14
    Last Post: 20th December 2014, 18:12
  2. Replies: 13
    Last Post: 7th November 2009, 01:02
  3. Replies: 4
    Last Post: 17th March 2008, 21:19
  4. rash - dummy EXE packer
    By encode in forum Forum Archive
    Replies: 17
    Last Post: 26th January 2008, 12:27
  5. TTA - very promising lossless WAV packer
    By Bulat Ziganshin in forum Forum Archive
    Replies: 12
    Last Post: 27th March 2007, 13:12

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •