Results 1 to 4 of 4

Thread: I have an idea for a multi-threaded compressor

  1. #1
    Member
    Join Date
    Mar 2010
    Location
    Canada
    Posts
    40
    Thanks
    0
    Thanked 2 Times in 2 Posts

    I have an idea for a multi-threaded compressor

    My question, if I publish the "C" code here, how much faster does it have to be than other compressors in it's class to be worthwhile looking at?

    The compressor is a simple byte oriented compressor/decompressor for a RAM DISK that is why I was looking at speed rather than compression ratio, but I also know multiple threads tend to clog up the caches and main memory interface so that is a major diminish in returns as you increase the number of threads.

    Or am I just wasting my time talking about it here?

    PS. If the idea of tacking meta data at the beginning of a compressed file already has been done could someone give me some pointers/URLs.

  2. The Following User Says Thank You to Earl Colby Pottinger For This Useful Post:

    snowcat (1st January 2019)

  3. #2
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    222
    Thanks
    89
    Thanked 46 Times in 30 Posts
    You should publish it (on GitLab or Bitbucket). We can't know anything about it until you publish it and benchmark it (or others benchmark it). It only has to be slightly faster to be worth looking at. Hell, it might be worth looking at even if it's slower than its peers, if it has other strengths like CPU/power efficiency, or compression ratio, or whatever. We can always learn something from any codec anyway. Multithreaded compression is still not mature or even common in some quarters, so a new multithreaded implementation is always nice. Anyway, fortune favors the brave – don't be afraid to put yourself out there.

    What do you mean by metadata at the beginning of the compressed file? That sounds like it could be a header, a model, a dictionary, or something else. What kind of metadata?

  4. The Following User Says Thank You to SolidComp For This Useful Post:

    snowcat (1st January 2019)

  5. #3
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    164
    Thanks
    115
    Thanked 10 Times in 9 Posts
    IMHO if you want it to be general purpose oriented it should compress slightly faster and tighter than 7z 1st of all.

  6. #4
    Member
    Join Date
    Dec 2011
    Location
    Cambridge, UK
    Posts
    437
    Thanks
    137
    Thanked 152 Times in 100 Posts
    If you're focusing on things like RAM disk and byte oriented packing, then likely comparing to LZ4 and/or Snappy are the obvious existing contenders out there.

    Give those a whirl on some data and see how your one compares.
    Good luck.

Similar Threads

  1. XORpressor. Yet another idea for a compressor
    By Cristo in forum Data Compression
    Replies: 5
    Last Post: 24th August 2016, 16:29
  2. Anyone thought of this yet? (Image Compressor Idea)
    By Lucas in forum Data Compression
    Replies: 12
    Last Post: 5th May 2015, 13:48
  3. Multi-way QuickSort
    By Piotr Tarsa in forum The Off-Topic Lounge
    Replies: 7
    Last Post: 9th May 2011, 18:59
  4. pxz: Multi threaded xz compressor
    By polemon in forum Data Compression
    Replies: 0
    Last Post: 21st March 2011, 02:55
  5. Multi-threaded compression
    By Cyan in forum Data Compression
    Replies: 34
    Last Post: 16th January 2011, 18:32

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •