Page 3 of 14 FirstFirst 1234513 ... LastLast
Results 61 to 90 of 407

Thread: cmix

  1. #61
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    223
    Thanks
    106
    Thanked 102 Times in 63 Posts
    In case it is useful for making a Windows build, the cmix code at https://github.com/byronknoll/cmix has some bug fixes. The current github code will be the cmix v9 release (assuming all my benchmarks validate successfully). It will take a week or two until the official v9 release. enwik8 will have a compressed size of 15615723.

  2. The Following User Says Thank You to byronknoll For This Useful Post:

    Darek (30th March 2016)

  3. #62
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    202
    Thanks
    165
    Thanked 109 Times in 65 Posts
    the one from github still crashes under windows. maybe someone else will try.

  4. The Following User Says Thank You to load For This Useful Post:

    Darek (31st March 2016)

  5. #63
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    223
    Thanks
    106
    Thanked 102 Times in 63 Posts
    Quote Originally Posted by load View Post
    the one from github still crashes under windows. maybe someone else will try.
    @load, what did you use to compile it? I expect it should work in Windows compiled in cygwin with g++.

  6. #64
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    202
    Thanks
    165
    Thanked 109 Times in 65 Posts
    cygwin is what i used. but i also tried now some msys with the same result.

    i686-w64-mingw32-g++
    x86_64-w64-mingw32-g++

    the 32bit crashed with the same error message. it seems cmix wants to allocate an enormous amount of memory before it crashes... windows even needs to use the disk swap file, although the file i wanted to compress is only a few bytes.

  7. #65
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Skymmer has made very stable version of CMIX V6

  8. #66
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    202
    Thanks
    165
    Thanked 109 Times in 65 Posts
    i know. but he refuses to tell us what he changed in the source, what he obviously. did.

  9. #67
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Quote Originally Posted by load View Post
    i know. but he refuses to tell us what he changed in the source, what he obviously. did.
    Fact is that Skymmer version of CMIX6 has slightly better compression than last stable Byron CMIX6 build - however Byron tells us that this compile could have slightly worse compression ratio than Linux version.

  10. #68
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Quote Originally Posted by load View Post
    it seems cmix wants to allocate an enormous amount of memory before it crashes... windows even needs to use the disk swap file, although the file i wanted to compress is only a few bytes.
    Please make clear what "enormous" means here. Right from the cmix website:

    At least 32GB of RAM is recommended to run cmix.
    Code:
    Corpus	Original size (bytes)	Memory usage (KiB)	Compressed size (bytes)		Compression time (seconds)	
    
    enwik6	1000000	                19470840                181738                          561.83
    So even for a 1 MB file, you'll need 19 GB RAM.
    http://schnaader.info
    Damn kids. They're all alike.

  11. #69
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Quote Originally Posted by schnaader View Post
    Please make clear what "enormous" means here. Right from the cmix website:



    Code:
    Corpus	Original size (bytes)	Memory usage (KiB)	Compressed size (bytes)		Compression time (seconds)	
    
    enwik6	1000000	                19470840                181738                          561.83
    So even for a 1 MB file, you'll need 19 GB RAM.
    I've got laptop with 32GB of ram and cmix7 and cmix8 compiles crashes...
    From other side CMIX6 versions (both Byron and Skymmer) runs fine with my other laptop with "only" 16GB by heavily use of swap file.

  12. #70
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    223
    Thanks
    106
    Thanked 102 Times in 63 Posts
    Quote Originally Posted by load View Post
    i know. but he refuses to tell us what he changed in the source, what he obviously. did.
    Skymmer posted the steps he used to make the compile in post#58 of this thread. After my benchmarks finish running (1-2 weeks), I will try adding those changes into cmix to try to improve Windows compatibility. I will also try posting a cmix v9 executable to this thread to see if it works on other people's computers.

  13. The Following User Says Thank You to byronknoll For This Useful Post:

    avitar (1st April 2016)

  14. #71
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    202
    Thanks
    165
    Thanked 109 Times in 65 Posts
    Quote Originally Posted by byronknoll View Post
    Skymmer posted the steps he used to make the compile in post#58 of this thread. After my benchmarks finish running (1-2 weeks), I will try adding those changes into cmix to try to improve Windows compatibility. I will also try posting a cmix v9 executable to this thread to see if it works on other people's computers.
    i did read that, and applied this to cmix6, but still it crashed with exactly the same error.

    i rebuilt cmix8 again (msys/stable gcc 5.2) with "fixes" from #58 and tried to run it on my desktop system with 32gb ram and only 1.5gb in use, same crash.
    it can't be the lack of memory, as it eats only ~10gb ram when i get the crash dialog.


    maybe i'm doing something wrong (i'm not really good at this), but i don't see what could be done wrong, honestly. it's a straight forward "make", and no errors or warnings turn up during compiling process.
    the only thing i added is -static-libgcc -static-libstdc++ to the linker flags, for obvious reasons.

  15. #72
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    223
    Thanks
    106
    Thanked 102 Times in 63 Posts
    I added Skymmer's fixes to the code on github.

    I got a chance to try compiling cmix in Windows. cmix v9 seemed to work normally with cygwin/g++. I am using cygwin 1.7.30.

    I also tried compiling using mingw 13.4 from here: http://nuwen.net/mingw.html
    cmix v9 crashed with a segmentation fault. The crash seems to be coming from the PAQ8L model. When I change the memory parameter of PAQ8L from 11 to 10 (predictor.cpp line 97), the compile no longer crashes. However, this probably will hurt compression rate, so I will try to find a better fix.

    I am attaching the binaries that I built. Can someone try testing them? The cygwin executable might only work in cygwin 1.7.30.
    Attached Files Attached Files

  16. The Following User Says Thank You to byronknoll For This Useful Post:

    Darek (2nd April 2016)

  17. #73
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Quote Originally Posted by byronknoll View Post
    I added Skymmer's fixes to the code on github.

    I got a chance to try compiling cmix in Windows. cmix v9 seemed to work normally with cygwin/g++. I am using cygwin 1.7.30.

    I also tried compiling using mingw 13.4 from here: http://nuwen.net/mingw.html
    cmix v9 crashed with a segmentation fault. The crash seems to be coming from the PAQ8L model. When I change the memory parameter of PAQ8L from 11 to 10 (predictor.cpp line 97), the compile no longer crashes. However, this probably will hurt compression rate, so I will try to find a better fix.

    I am attaching the binaries that I built. Can someone try testing them? The cygwin executable might only work in cygwin 1.7.30.
    The mingw version runs fine on 32GB, I've started to test it. On 16GB doesn't work at all.
    The cygwin version gives error as on screen. That means "The application was improperly launched (0xc000007b). Click OK to close it.", however as you wrote my cygwin1.dll is 1.5.18.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	error.jpg 
Views:	157 
Size:	32.8 KB 
ID:	4250  
    Last edited by Darek; 2nd April 2016 at 19:52.

  18. #74
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    323
    Thanks
    174
    Thanked 51 Times in 37 Posts
    Quote Originally Posted by Darek View Post
    The minigw version runs fine on 32GB, I've started to test it. On 16GB doesn't work at all.
    The cygwin version gives error as on screen. That means "The application was improperly launched (0xc000007b). Click OK to close it.", however as you wrote my cygwin1.dll is 1.5.18. Where I can download 1.7.30 version?
    I also got the same results. Error with cygwin version and no problems with MinGW version on my 32GB machine.

  19. #75
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Quote Originally Posted by comp1 View Post
    I also got the same results. Error with cygwin version and no problems with MinGW version on my 32GB machine.
    Ok, in attached file you find three dll needed to run cygwin version of CMIX V9 on Win64. Just put it into Cmix directory. Now CygWin version also run!

    Darek

    EDIT:

    On my 16GB laptop with maximum swap file, and free memory b0th versions running too.

    But is something strange that I've found. Mingw version on 32GB laptop gives slightly better scores than on 16GB - couple of bytes (1 up to 9). I've checked names and are identical, theoretical memory usage are the same - both 32GB, but scores are different. Is possible that something depends on memory amounts allocated - which could be a little different?
    Attached Files Attached Files
    Last edited by Darek; 2nd April 2016 at 21:35.

  20. The Following User Says Thank You to Darek For This Useful Post:

    comp1 (2nd April 2016)

  21. #76
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    202
    Thanks
    165
    Thanked 109 Times in 65 Posts
    cygwin crashes here, no matter what dll i use:

    Exception: STATUS_ACCESS_VIOLATION at rip=00076F6F645
    rax=00000006B1834CA0 rbx=00000006B1834CA0 rcx=0000000771840020
    rdx=0000000000000000 rsi=00000000C0000000 rdi=0000000000000000
    r8 =0000000000000000 r9 =03FFFFFFFBFFFD33 r10=00000006B1834C90
    r11=0000000000000000 r12=00000006B17BEAB0 r13=0000000000008000
    r14=00000006B17BE8A0 r15=0000000000008000
    rbp=0000000000000000 rsp=0000000000229C88
    program=d:\\cmix.exe, pid 5568, thread main
    cs=0033 ds=002B es=002B fs=0053 gs=002B ss=002B
    Stack trace:
    Frame Function Args
    00000000000 00076F6F645 (00000000000, 000FFFFFFFF, 00000000000, 006B1834CA0)
    00000000000 0007759BA46 (00000000000, 00000000000, 006B17BEAB0, 006B1834CA0)
    00000000000 001800C4917 (00000000000, 00000000000, 001801360DF, 00000000000)
    00000000000 001801451FB (0000000001A, 00000000001, 00180111C9B, 0000000001A)
    00000000000 001800C00EF (0018013A039, 006000F70D0, 00180139FF8, 00600000020)
    00000000000 00180111C9B (006000F70D0, 00180139FF8, 00600000020, 00000000030)
    00000000000 001004136C0 (00180139FF8, 00600000020, 00000000030, 002000F491
    00000000000 0018013A039 (00180139FF8, 00600000020, 00000000030, 002000F491
    End of stack trace
    mingw runs fine.also when i compile it myself using the predictor.cpp changes.

  22. #77
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    223
    Thanks
    106
    Thanked 102 Times in 63 Posts
    Quote Originally Posted by Darek View Post
    But is something strange that I've found. Mingw version on 32GB laptop gives slightly better scores than on 16GB - couple of bytes (1 up to 9). I've checked names and are identical, theoretical memory usage are the same - both 32GB, but scores are different. Is possible that something depends on memory amounts allocated - which could be a little different?
    Hmm, that is strange. No, the amount of available memory shouldn't change the compression rate. Maybe this is due to compiling with -Ofast, which makes it faster but not standards compliant. Perhaps that makes it not portable between computers.

  23. #78
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Quote Originally Posted by byronknoll View Post
    Hmm, that is strange. No, the amount of available memory shouldn't change the compression rate. Maybe this is due to compiling with -Ofast, which makes it faster but not standards compliant. Perhaps that makes it not portable between computers.
    I confirm - I've tested some files with different memory amount on one computer and scores are identical. Something is curious between my two computers.

    @Load - did you try my "cmixV9_cygwin_dll.7z" set? It works for my both computers.

  24. #79
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    223
    Thanks
    106
    Thanked 102 Times in 63 Posts
    @Darek, I am attaching a mingw compile which is slower but hopefully should give the same output on any computer. Could you try testing the following executable on your two computers to see if they give the same output? Thanks!
    Attached Files Attached Files

  25. The Following User Says Thank You to byronknoll For This Useful Post:

    Darek (4th April 2016)

  26. #80
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Quote Originally Posted by byronknoll View Post
    @Darek, I am attaching a mingw compile which is slower but hopefully should give the same output on any computer. Could you try testing the following executable on your two computers to see if they give the same output? Thanks!
    Yes. I'm out of home now. I'll test It as I'll arrive Home tommorow.

  27. #81
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    772
    Thanks
    63
    Thanked 270 Times in 190 Posts
    Quote Originally Posted by byronknoll View Post
    I am attaching the binaries that I built. Can someone try testing them?
    cmix9 under Windows command line, no dictionary:
    enwik8 100,000,000 bytes to 16,270,942 bytes in 56825.94 sec.

  28. #82
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Quote Originally Posted by byronknoll View Post
    @Darek, I am attaching a mingw compile which is slower but hopefully should give the same output on any computer. Could you try testing the following executable on your two computers to see if they give the same output? Thanks!
    Now it's OK, both computers give the same results on MinGw version.
    CygWin version is OK too.

    Why the results are different on different machines on first compilation mingw?

  29. The Following User Says Thank You to Darek For This Useful Post:

    byronknoll (4th April 2016)

  30. #83
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    223
    Thanks
    106
    Thanked 102 Times in 63 Posts
    Linux, dictionary preprocessing, PAQ8L memory parameter set to 11: enwik8 compresses to 15,615,723 in 33,175.59 seconds with 26,313,052 KiB memory
    Linux, dictionary preprocessing, PAQ8L memory parameter set to 10: enwik8 compresses to 15,627,536 in 33,244.92 seconds with 23,046,324 KiB memory

    I haven't been able to figure out how to fix the mingw compile without setting PAQ8L to 10. So, I think I will just make 10 the default for cmix v9 to improve Windows compatibility. This also reduces memory usage by about 3GB (which I will probably make use of for cmix v10).

  31. The Following 2 Users Say Thank You to byronknoll For This Useful Post:

    Darek (4th April 2016),schnaader (4th April 2016)

  32. #84
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    223
    Thanks
    106
    Thanked 102 Times in 63 Posts
    Quote Originally Posted by Darek View Post
    Why the results are different on different machines on first compilation mingw?
    For the second mingw compile, I removed -march=native and set -O3 instead of -Ofast.

  33. #85
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Quote Originally Posted by byronknoll View Post
    For the second mingw compile, I removed -march=native and set -O3 instead of -Ofast.
    Yes, I understand it, but the question is how this change could made one executable which gives different scores on different machines.

    I'm still testing. It's going slowly due to:
    a) use swap file which slow down compression performance,
    b) using swap file makes CPU usage down which provides CPU booster to think that we aren't using much procesor power and didn't overclock procesor or even set clock down. Instead 3.8GHz I've running on 2.6GHz.

    One insight from testing and CMIX crashing because of memory usage: all crashes are on the same 14 files from my testbed - any version, even older gets crash on the same files. Size is not an factor because it's on small and bigger files, but it's always occurs when I run dictionary usage and CMIX started to pretraining. For Cygwin version it occurs also when I'm running on 32GB computer. Errors are similar: "0 [main] cmixX XXXX cygwin_exceptionen_stackdumpfile: Dumping stack trace to cmixX.exe.stackdump: . That means the error is showing on pretraining stage, not in compression phase at all.

    Darek
    Last edited by Darek; 5th April 2016 at 12:17.

  34. #86
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    223
    Thanks
    106
    Thanked 102 Times in 63 Posts
    Quote Originally Posted by Darek View Post
    Yes, I understand it, but the question is how this change could made one executable which gives different scores on different machines.
    I'm not exactly sure. When using -Ofast it is no longer standards compliant, so for example some floating point math can give different output than -O3. I think different computer architectures can also cause the output to change.

    Quote Originally Posted by Darek View Post
    One insight from testing and CMIX crashing because of memory usage: all crashes are on the same 14 files from my testbed - any version, even older gets crash on the same files. Size is not an factor because it's on small and bigger files, but it's always occurs when I run dictionary usage and CMIX started to pretraining. For Cygwin version it occurs also when I'm running on 32GB computer. Errors are similar: "0 [main] cmixX XXXX cygwin_exceptionen_stackdumpfile: Dumping stack trace to cmixX.exe.stackdump: . That means the error is showing on pretraining stage, not in compression phase at all.
    This sounds like a bug - can you give an example file which causes cmix v9 to crash?

  35. #87
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Quote Originally Posted by byronknoll View Post
    This sounds like a bug - can you give an example file which causes cmix v9 to crash?
    Please, example in attached zip file.
    Attached Files Attached Files
    • File Type: 7z R.7z (36.6 KB, 65 views)

  36. #88
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    223
    Thanks
    106
    Thanked 102 Times in 63 Posts
    Quote Originally Posted by Darek View Post
    Please, example in attached zip file.
    Thanks, I have fixed this bug. I am attaching a new executable with the fix. The fix is also submitted to Github.

    The bug only affected the mingw build. I was using "tmpfile()", which apparently needs administrator privileges to work in Windows. I changed the code to write temporary files to the output directory rather than using tmpfile().
    Attached Files Attached Files

  37. The Following User Says Thank You to byronknoll For This Useful Post:

    Darek (6th April 2016)

  38. #89
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Quote Originally Posted by byronknoll View Post
    The bug only affected the mingw build.
    Thanks for this version, now mingw is stable at all on 16GB and 32GB computer!

    But, this mentioned error ""0 [main] cmixX XXXX cygwin_exceptionen_stackdumpfile: Dumping stack trace to cmixX.exe.stackdump:" goes from cygwin version. Maybe it's different bug. For this version and these 14 files error ocurs both for 16GB and 32GB computer. As I mentioned only when CMIX start to pretraining.

  39. #90
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    223
    Thanks
    106
    Thanked 102 Times in 63 Posts
    Quote Originally Posted by Darek View Post
    But, this mentioned error ""0 [main] cmixX XXXX cygwin_exceptionen_stackdumpfile: Dumping stack trace to cmixX.exe.stackdump:" goes from cygwin version. Maybe it's different bug. For this version and these 14 files error ocurs both for 16GB and 32GB computer. As I mentioned only when CMIX start to pretraining.
    Ah, I guess the tmpfile problem can affect the cygwin version too. Potential fixes:
    1) Run cmix with administrator privileges.
    2) Run cmix within the cygwin environment.
    3) Compile a new version of cmix in cygwin, using the fixed code on github. I am running enwik9 in Linux (I have one computer with dual boot), so I won't be able to make a new cygwin compile for a while.

  40. The Following User Says Thank You to byronknoll For This Useful Post:

    Darek (6th April 2016)

Page 3 of 14 FirstFirst 1234513 ... LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •