Results 1 to 15 of 15

Thread: qpress

  1. #1
    Programmer
    Join Date
    May 2008
    Location
    denmark
    Posts
    94
    Thanks
    0
    Thanked 2 Times in 2 Posts

    qpress

    I've created a prototype of a new QuickLZ 1.40 based file archiver, named qpress.

    It's using asynchronous file I/O (using a pthread reader and writer) and is also bypassing file system cache so that the cache of other applications isn't flushed out.

    It should be pretty fast: www.quicklz.com/qpress.exe (using QuickLZ level 2 as default)

  2. #2
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks Lasse!

  3. #3
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Good Job ! Hi!

  4. #4
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Very quick (L1) test...

    Your prototype compresses ENWIK9 to 467,679,585 bytes in 39.35 seconds on my AMD Sempron 2400+, Windows XP SP2 machine.

  5. #5
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    http://shelwien.googlepages.com/qpress01.htm

    I'd say, that compression could be 1.5X better with that speed.

    Actually, what made me wonder here is ST2rc decoding speed.
    It was symmetric before... could it be that some weird caching
    details on Q9450 caused that?
    First I though that its due to unoptimized E8 in the ST2 loop,
    but it has almost the same speed without it.
    Mysterious.

  6. #6
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    @lasse reinhold

    i think the new qpress is a good program

    ---
    Current qpress 0.1 is using the QuickLZ 1.40 b9 library
    ---
    asks:
    1. would it be possible to have a version qpress
    with the QUICKLZ 1.31 - engine
    because for now it seems that
    the 1.31 version is faster then the 1.40 version

    2. maybe in the future for the 64-bit-version of quicklz 1.5
    will be too such a version of qpress ?

    best regards

  7. #7
    Programmer
    Join Date
    May 2008
    Location
    denmark
    Posts
    94
    Thanks
    0
    Thanked 2 Times in 2 Posts
    Hi Joerg,

    The QuickLZ 1.40 b9 library is slightly faster than the 1.30 library, just try the -m or -mem flags. Also note that qpress is using level 2 as default.

    So it must be a disk I/O issue. Maybe it's caching, since qpress use uncached disk I/O (use the B flag to use normal caching like all other programs).

    Have you emptied the system cache before testing (rebooting is the safest way)? If you are on Vista, also disable the Superfetch service, or else it will read recently used files into memory in the background, starting immediately after reboot.

    It could also be some bad design in qpress that makes it slower on some systems - I'm not done testing and tuning yet

    I'll certainly make a 64-bit version when 1.50 goes into beta. There's not much point in a 64-bit version for 1.31 or 1.40 because they are not much faster than 32-bit versions.
    Last edited by Lasse Reinhold; 30th June 2008 at 15:23.

  8. #8
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    thank you for your quick answer

    but if you look at http://www.winturtle.netsons.org/MOC/MOC.htm

    you can see: Ranked on Compression TIME

    program compression result compression time

    1 QUICKLZ 1.31 533.789.984 bytes 8,845 sec

    and

    15 QPRESS 0.1 (quicklz 1.40) 537.936.744 bytes 32,898 sec

    for me this means
    the quicklz 1.31 engine seems to be faster in several cases

  9. #9
    Member
    Join Date
    May 2008
    Location
    Earth
    Posts
    115
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by joerg View Post
    thank you for your quick answer

    but if you look at http://www.winturtle.netsons.org/MOC/MOC.htm

    you can see: Ranked on Compression TIME

    program compression result compression time

    1 QUICKLZ 1.31 533.789.984 bytes 8,845 sec

    and

    15 QPRESS 0.1 (quicklz 1.40) 537.936.744 bytes 32,898 sec

    for me this means
    the quicklz 1.31 engine seems to be faster in several cases
    There's some other quirks as well. Look at the "Ranked on Compression TIME " table,the first entry:
    Rank Compressor Size Time of Comp. Time of Dec. Compression
    Efficiency Decompression
    Efficiency Author Memory Option Algorithm
    1 QUICKLZ 1.31 533.789.984 8,845 23,486 1227,81 1493,03 Lasse Mikkel Reinhold 3 MB -1 LZ77
    I don't believe this! How can LZ77 compress faster than decompress?
    Last edited by IsName; 30th June 2008 at 17:21.

  10. #10
    Programmer
    Join Date
    May 2008
    Location
    denmark
    Posts
    94
    Thanks
    0
    Thanked 2 Times in 2 Posts
    Joerg: I still think it's an disk I/O issue. Either it's disk cache that hadn't been emptied before testing 1.31 (Nania?), or else it's a flaw in my new threaded I/O thing (1.31 uses fread/fwrite, qpress uses native API). Will look into it.

    IsName: This is probably a disk issue again. Disks write slower than they read, and decompression reads a "smaller" amount of data and write a "larger" amount. While compression is opposite. But again, would be nice with disk details of the MOC test - wether caches are flushed and if times are kernel time, user time, etc...

  11. #11
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts

    Hi Lasse!

    I use an automatic program GUI written with Free-Basic that is very precise with the millesimis in the tests! The API are excellent for compatibility but they require of put make up on for improving the speed!

  12. #12
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    www.quicklz.com
    ---
    here is a new release of qpress (based on quicklz 1.40)
    ---
    qpress.exe qpress-0.11-linux_386.tar.gz qpress_src.zip
    ---
    Latest news - 12-Aug-2008
    ---
    Just released qpress 0.11.
    Made I/O buffered as default (see -B flag) under Windows
    because unbuffered I/O seems to starve I/O of other processes
    (such requests are being translated directly into DMA transfers).
    Also reduced block size from 8 to 3 MiB to try helping it.
    Other minor changes such as printing compressed size during compression.
    Named pipes under Windows will be supported later.
    ---

  13. #13
    Member
    Join Date
    Jan 2007
    Location
    Moscow
    Posts
    239
    Thanks
    0
    Thanked 3 Times in 1 Post
    Why does "qpress.exe -v1 *.txt alltxt1" and "qpress.exe -v3 *.txt alltxt3" produce bit-identical results?

  14. #14
    Programmer
    Join Date
    May 2008
    Location
    denmark
    Posts
    94
    Thanks
    0
    Thanked 2 Times in 2 Posts
    Hi nimdamsk,

    You have to use -L1 or -L3 (or -vL1 or -vL3 in your case with verbose too) to change compression level.

    If you just use -v1 then the '1' will simply be ignored. qpress should acturly give an error message but I havn't implemented flag validation completely yet

  15. #15
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    new Version qpress 0.33 BETA - 09-Feb-2009

    win32 http://www.quicklz.com/qpress-033.exe
    source http://www.quicklz.com/qpress-033-source.zip

    on http://www.quicklz.com/
    are ports for several other Operating systems

    win64 / solaris 10 x86 / Linux-x64 / Linux-IBM-S390 / Linux-xscale-32

    ---
    the new version seems to work a little bit faster - 5 %
    especially under windows when using the cmdline flags L1 and T and K
    ---
    for example: qpress033 -rK4096T2L1 d:\orgdta\* c:\tdbq33s.qp03
    ---
    -Tn Use n threads/cores for compression
    where n = 1 to 256 (default = 2)
    Be aware of memory usage with large -T and -K
    -Kn Read from disk in n KiB chunks during compression
    where n = 64 to 32768 (default = 64)
    Be aware of memory usage with large -T and -K
    ---
    Last edited by joerg; 13th February 2009 at 13:13.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •