Results 1 to 12 of 12

Thread: PPM Compressor Small datas

  1. #1
    Member
    Join Date
    Jul 2014
    Location
    Canada
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    PPM Compressor Small datas

    Hello,

    I'm new to the forum and I've been researching however not found.
    I wonder what the fastest ppm compressors for small data. 50 bytes for example.

    Thanks

  2. #2
    Member biject.bwts's Avatar
    Join Date
    Jun 2008
    Location
    texas
    Posts
    449
    Thanks
    23
    Thanked 14 Times in 10 Posts
    I am not sure why you are interested in speed. Most compressors will end up expanding a small file of 50 bytes or less. So to have a chance you would most likely need a "bijective ppm compressor" One of the first and still one of the best for small files is BICOM by Matt Timmermans.

    At http://www3.sympatico.ca/mt0000/bicom/bicom.html
    Last edited by biject.bwts; 4th July 2014 at 05:18. Reason: bad spelling

  3. #3
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,471
    Thanks
    26
    Thanked 120 Times in 94 Posts
    If the input strings have enough similarities then pre-trained static PPM model could be used to compress them.

  4. #4
    Member biject.bwts's Avatar
    Join Date
    Jun 2008
    Location
    texas
    Posts
    449
    Thanks
    23
    Thanked 14 Times in 10 Posts
    The reality is that for random data in files of 50 bytes or less. No general compressor exists to compress the data. Sure some of the files could be compressed smaller but most files would not be compressed to a smaller size. You are better off doing nothing. Unless you know the files of interest form some sort of statistical subset of all files and you use a compressor that takes advantages of the expected subset.

  5. #5
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,471
    Thanks
    26
    Thanked 120 Times in 94 Posts
    There's no compressor that can compress random data :] Maybe you've meant something else, like random, but skewed probability distributions.

    I think general compression algorithms won't work well on 50 bytes long independent chunks. It would be much more sensible to extract inter-block similarities either using pre-computed models from some reference input blocks or keeping a model between subsequent coding invocations.

  6. #6
    Member
    Join Date
    Jul 2014
    Location
    Canada
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I was wrong in saying that like speed. In the case no matter the speed, I need some compressor that compresses packets of 30 ~ 100 bytes in the smallest possible packets.

  7. #7
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,471
    Thanks
    26
    Thanked 120 Times in 94 Posts
    What's your usage scenario? If you're trying to reduce two-way communication overhead then I suggest keeping a compressor state on both ends. You can even use zlib with flushing after each packet and you'll probably achieve some noticeable reduction in size.

    Or at least provide some sample packets to analyze.

  8. #8
    Member biject.bwts's Avatar
    Join Date
    Jun 2008
    Location
    texas
    Posts
    449
    Thanks
    23
    Thanked 14 Times in 10 Posts
    Quote Originally Posted by nm_m View Post
    I was wrong in saying that like speed. In the case no matter the speed, I need some compressor that compresses packets of 30 ~ 100 bytes in the smallest possible packets.

    Your not giving enough information. Sometimes a fixed packet size is best. Don't forget if you are actually able to compress the packet smaller then somehow you have to be able to send the new packet length.

    To be able to compress at all you need to take advantage of the structure of the data. You also have to realize that many times the size of the packet will increase. You may have 30 to 100 bytes for input but then you may have to live with 20 to 110 bytes at output.

    It might even be better if several packets are combined to one big packet for compress and then broken apart after decompression. Tell us more.

    Look one could easily modify a 256 symbol bijective arithmetic coder to take a 100 byte file and if all bytes are different it would expand to its max of 110 bytes ( or what max your heart desires ) and then files with a lot of repeated characters would compress smaller than 100 bytes. The larger you allow for the maximum file expansion then the smaller a file with many repeats would compress.

  9. #9
    Member
    Join Date
    Jul 2014
    Location
    Canada
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Will be used to compress data TCP protocol. And packet size, I'm talking are between 10-100 bytes. I would like to reduce this size with a compressor.

  10. #10
    Member
    Join Date
    Feb 2013
    Location
    San Diego
    Posts
    1,057
    Thanks
    54
    Thanked 71 Times in 55 Posts
    Quote Originally Posted by nm_m View Post
    Will be used to compress data TCP protocol. And packet size, I'm talking are between 10-100 bytes. I would like to reduce this size with a compressor.
    A lot of the packets, especially ones that are minimum in length, are probably mostly canned data that's dictated by the protocol, like ACK messages. The best way to approach those is to replace them with shorter codes, such as with Huffman. The reason shorter messages aren't already built into the protocol, though, is likely because compression would slow down the performance of the network.

  11. #11
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,471
    Thanks
    26
    Thanked 120 Times in 94 Posts
    Compressing packets independently won't allow you to exploit similarities like repeated IP numbers, port numbers, etc between packets.

    You still didn't tell us whether you need to compress the packets indepenently or in blocks.

    Also efficient compressing of structured data requires some special treatment from the compressor. Parsing TCP packets shouldn't be hard but I haven't heard of any compressor that does that.

  12. #12
    Member
    Join Date
    Feb 2013
    Location
    San Diego
    Posts
    1,057
    Thanks
    54
    Thanked 71 Times in 55 Posts
    Quote Originally Posted by Piotr Tarsa View Post
    Compressing packets independently won't allow you to exploit similarities like repeated IP numbers, port numbers, etc between packets.

    You still didn't tell us whether you need to compress the packets indepenently or in blocks.

    Also efficient compressing of structured data requires some special treatment from the compressor. Parsing TCP packets shouldn't be hard but I haven't heard of any compressor that does that.
    True, even ACK packets would have parts that vary, such as IP addresses. So they couldn't be replaced 1:1 with static codes. But I think once you've separately dealt with things like addresses, you'd be left with a message saying "ACK." That's pretty much exactly the same concept as the kind of atomic symbol with associated probability that Huffman was invented to make codes for.

    Let's not get too deep into possible implementations, though, because the parts of this scheme that we know about don't add up to anything workable, as far as I can tell. You couldn't just compress TCP packets and send them back down the wire in compressed form and expect a functioning network. So, I think, either we have much too little information about the project to possibly start coming up with solutions, or perhaps the question was misinformed.

Similar Threads

  1. Compressing small packets
    By cbloom in forum Data Compression
    Replies: 4
    Last Post: 1st March 2013, 02:59
  2. Small just-for-fun challenge at codegolf
    By schnaader in forum Data Compression
    Replies: 13
    Last Post: 24th March 2012, 01:57
  3. compressing a really small 1k .COM file
    By Rugxulo in forum Data Compression
    Replies: 3
    Last Post: 28th November 2009, 01:32
  4. Compressing small bits of data
    By fredrp in forum Data Compression
    Replies: 9
    Last Post: 28th April 2009, 22:27
  5. A Small Warning
    By encode in forum The Off-Topic Lounge
    Replies: 1
    Last Post: 30th August 2008, 21:05

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •