Results 1 to 8 of 8

Thread: [Query] Regarding RAM compression

  1. #1
    Member
    Join Date
    Oct 2017
    Location
    India
    Posts
    2
    Thanks
    3
    Thanked 0 Times in 0 Posts

    Smile [Query] Regarding RAM compression

    Hello folks,

    I'm an absolute beginner in data compression. I was wondering an efficient way to compress the RAM contents.
    I have used ZSTD from Yann Collet's github (thanks Yann for the great work) but it is slower than LZ4 algorithm currently.

    I have following questions

    1. What is the best option for compressing RAM contents? Any specialized encoders? (open source will be great)
    2. Can I use integer compressing codecs to do the job on RAM data? Are they similar?
    3. Can anyone point me to the starting point for learning data compression? (Prerequisites etc.)

    Thank you sincerely.

  2. #2
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    1) "RAM contents" is not a good definition of a data type.
    Do you have samples? what do you actually try to compress?
    I suppose, you can try using paq8px/paq8pxd to see actual data types and entropy estimation for your samples.
    I can only expect that there'd be a large % of x64 code/COFF executables, compression of which can be improved by simple/fast preprocessing (E8 etc).
    Also maybe english text.

    2) Yes, but you'd have to detect blocks of integers first.
    Also I'm not aware of any good compressors for integers. Fast, yes, but compression ratio is not any better than universal codecs.

    3) https://en.wikipedia.org/wiki/Data_compression
    http://mattmahoney.net/dc/dce.html

    But there's no magic solution with multi-GB/s speed of both encoding and decoding,
    most compression algorithms just wouldn't be applicable.

  3. The Following User Says Thank You to Shelwien For This Useful Post:

    _Bluesman_ (6th November 2017)

  4. #3
    Member
    Join Date
    Oct 2017
    Location
    India
    Posts
    2
    Thanks
    3
    Thanked 0 Times in 0 Posts
    Shelwien,

    Thank you for the reply. By "RAM contents" I meant the portion of RAM can be compressed (like in Linux compcache, zswap etc). So, it will be a good mixture of code(string), numbers, string (messages, data etc). So, if I need to compress that "in-memory" data, where do I start?

    Any pointers please?


    Thank you.

  5. #4
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Well, you need samples first - find/make some swap files, crash dumps or something (without encryption).
    Maybe just record the data, if you can capture it.

    Then test some available codecs, like with https://github.com/inikep/lzbench

    Ideally, to reach the best compression, you'd have to port some preprocessors like delta/mm/E8/text,
    but if you want to keep it simple, just the codecs would be enough.

    Also, compression would be much better if you'd find a way to (de)compress larger chunks at once,
    instead of just processing 4k pages independently.
    One good option is to use LZ compression with a dictionary - zstd has such a mode, for example (-train etc).

  6. #5
    Member
    Join Date
    Dec 2011
    Location
    Cambridge, UK
    Posts
    437
    Thanks
    137
    Thanked 152 Times in 100 Posts
    I think people have tried LZ4 for "swap", as if you're about to have to swap due to lack of memory, then compression is still going to be faster than writing to disk. If LZ4 is too slow then more specialist deduplication methods such as RLE are perhaps preferable.

    Most entropy encoding methods are still likely too slow.

  7. #6
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    342
    Thanks
    12
    Thanked 34 Times in 28 Posts
    On FreeBSD 11.1 zfs' ARC can be compressed.

    It's the default "read cache" of the filesystem, sometimes with a ratio of 2:1 on "office" fileserver.
    AFAIK it's LZ4

    https://wiki.illumos.org/display/ill...RC+Compression

  8. #7
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    ram compression is a huge area on its own. look http://csl.skku.edu/papers/icce17.pdf which also mentions other ram compressors

    also https://encode.ru/threads/1936-Nakamichi
    https://github.com/centaurean/density (my own experiments with this algo end up with compression speed up to a few GB/s)

    you may find more fast compressors here:
    https://chiselapp.com/user/Justin_be...9d4b45c882a31e

  9. #8
    Member
    Join Date
    Nov 2015
    Location
    boot ROM
    Posts
    83
    Thanks
    25
    Thanked 15 Times in 13 Posts
    In Linux there is already "zram" thing, doing exactly that: it creates compressed "ram drive". The point is that one could put "swap" here, for example. Or maybe some volatile temporary data. Putting swap here basically brings you to state where you have paging, but paging happens to compressed ram area instead of HDD. So it kinda fast compared to accessing HDD. If only rarely used data are offloaded to zram, one could almost double RAM while getting only slight performance hit. What algos are here? LZ4 is the fastest one, LZO is slightly slower but compresses a bit better as well, and zlib being slow, but compresses even better. Though since it has got to deal with paging, and one really wants paging to be fast, zram device handles compressed pages some special way, like it puts 2 pages in place of one if it could get there or just one page in case it haven't managed to get this far. While it not most efficient way around, it speeds up paging, which must happen more or less real time, otherwise it gets slow and pointless. From practical standpoint most configurations are using LZ4 as it happens to be fastest. These days even some routers and Android phones are doing this trick right from factory, to improve overall user experience. Either way you want something blazing fast at compression and decompression. Paging is painful if it happens slow.

Similar Threads

  1. Super-Scalar RAM-CPU Cache Compression
    By Gonzalo in forum Data Compression
    Replies: 5
    Last Post: 16th February 2015, 02:06

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •