Results 1 to 8 of 8

Thread: Increase PAQ Memory

  1. #1
    Member
    Join Date
    Sep 2015
    Location
    Argentina
    Posts
    7
    Thanks
    4
    Thanked 0 Times in 0 Posts

    Increase PAQ Memory

    Hi. As I've told in other post, I'm actually doing some reasearch in data compression. I'm using PAQ8N and I need to compress large files, so I would like to try increasing memory to see new results. PAQ8n allows to use level 8 which means using aprox 1600MB of Memory, and my computer has 4Gb available to use.
    There's a line in the code:

    define
    MEM (0x10000<<level)

    I think that changing that number should solve the problem of increasing the memory but I had no success. I'd to reach 4Gb, can anyone help?

    Thanks!!

    Juan

  2. #2
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    A lot of the data structures use 32 bit indexes. It would take a bit more work. Or you could use zpaq.

  3. The Following User Says Thank You to Matt Mahoney For This Useful Post:

    juanandreslaura (17th September 2015)

  4. #3
    Member
    Join Date
    Sep 2015
    Location
    Argentina
    Posts
    7
    Thanks
    4
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Matt Mahoney View Post
    A lot of the data structures use 32 bit indexes. It would take a bit more work. Or you could use zpaq.
    Thanks for your answer. I've been reading your article about zPaq. I see that for 32 bits the memory is limited to 2Gb, but there's no specification about 64 bits (in fact, I have 64bits). Is there a memory usage limit? Level 5 is the highest level of compression and I would like to take advantage of not used memory (if posssible, I'd like to use all the available memory). What do you think?

    Thanks

    Juan

  5. #4
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    It depends on block size. zpaq groups file fragments into blocks, typically 16..64 MB to be compressed independently in parallel. You can choose larger block sizes like method 58, 59, 510, 511, which select 2^8..2^11 MB blocks instead of the default 5 = 56 = 2^6 MB. Memory usage is up to about 8x block size per thread for levels 1..4 and 16x for 5.

  6. The Following User Says Thank You to Matt Mahoney For This Useful Post:

    juanandreslaura (19th September 2015)

  7. #5
    Member
    Join Date
    Sep 2015
    Location
    Argentina
    Posts
    7
    Thanks
    4
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Matt Mahoney View Post
    It depends on block size. zpaq groups file fragments into blocks, typically 16..64 MB to be compressed independently in parallel. You can choose larger block sizes like method 58, 59, 510, 511, which select 2^8..2^11 MB blocks instead of the default 5 = 56 = 2^6 MB. Memory usage is up to about 8x block size per thread for levels 1..4 and 16x for 5.
    Thanks a lot!

  8. #6
    Member
    Join Date
    Sep 2015
    Location
    Arg
    Posts
    2
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hi, (At first sorry for my english)...


    I ran this command: zpaq64 a BGH E:\BGH -threads 1 -method 511 -summary -noattribute.


    So the memory usage will be aprox. => 16 * 1 * 2 ^ 11 MiB = 32 GB


    But i noted in Windows Task Manager, the memory used is about 15,6 gb (The total memory of the PC is 16GB). The rest of memory (32 - 15,6 aprox) will be taken of the virtual memory (Swap file)?


    Is usefull or convenient to use method=511 for this PC, or is better to use the method=510 to aviod the usage of virtual memory?

    Thanks.

  9. #7
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    If zpaq starts using virtual memory, it will be impractically slow. Stick with how much physical RAM you have.

  10. #8
    Member
    Join Date
    Sep 2015
    Location
    Arg
    Posts
    2
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Matt Mahoney View Post
    If zpaq starts using virtual memory, it will be impractically slow. Stick with how much physical RAM you have.
    I stopped the process, it was 1,76% and run by one day. I re-launch zpaq but with -method 510, y one day later i was at 10%.
    Thanks.

Similar Threads

  1. Memory Usage vs. Memory Requirements for (De)Compression?
    By comp1 in forum The Off-Topic Lounge
    Replies: 7
    Last Post: 1st June 2015, 04:53
  2. CPU to GPU memory bottleneck
    By boxerab in forum Data Compression
    Replies: 6
    Last Post: 12th June 2014, 19:41
  3. Optimization to increase speed
    By BetaTester in forum Data Compression
    Replies: 8
    Last Post: 11th November 2012, 20:31
  4. Memory Limit?
    By Earl Colby Pottinger in forum Data Compression
    Replies: 28
    Last Post: 11th April 2010, 03:59
  5. 2G+ memory blocks
    By Shelwien in forum Data Compression
    Replies: 0
    Last Post: 6th March 2009, 03:13

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •