Results 1 to 20 of 20

Thread: Paq8px building instructions

  1. #1
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    343
    Thanks
    235
    Thanked 226 Times in 123 Posts

    Paq8px building instructions

    Here is a guide to make a successful paq8px build from source...

    • ...on Windows with Visual Studio Community 2017 and/or MinGW-w64.
    • ...on Linux with GCC and/or Clang + LLVM.

    Extract the attached content, and open howto/index.html.

    Extra:
    If you have Windows and would like to have a Linux development/test platform as well, here you will find a chapter with instrunctions to set up a Virtual Machine with Linux (Ubuntu) equipped with the necessary development tools and shared folders.
    This chapter of the guide is a general one - it may be useful for anyone wishing to do some cross-platform C++ development.

    If you find any typos, errors or have any suggestions regarding the settings, the text, the layout or anything - let me know. I will update this first post.
    Attached Files Attached Files
    Last edited by Gotty; 27th May 2018 at 09:44. Reason: Update

  2. The Following 10 Users Say Thank You to Gotty For This Useful Post:

    Bulat Ziganshin (30th December 2017),Darek (30th December 2017),Gonzalo (6th June 2018),hexagone (30th December 2017),load (13th January 2018),Mauro Vezzosi (8th April 2018),pinguin2 (28th January 2018),samsat1024 (9th June 2018),schnaader (30th December 2017),Simorq (15th January 2018)

  3. #2
    Member Alexander Rhatushnyak's Avatar
    Join Date
    Oct 2007
    Location
    Canada
    Posts
    232
    Thanks
    38
    Thanked 80 Times in 43 Posts
    Quote Originally Posted by Gotty View Post
    • Linux (currently paq8px does not compile on Linux)


    If you find any typos, errors or have any suggestions
    I suggest making sure it runs correctly on Linux, after you make it compile on Linux. If it produces different compressed files on Linux and Windows, this is a serious issue and should be fixed. I guess cmix has this problem (but I can't run cmix on my computers: it needs much more RAM than they have)

    This newsgroup is dedicated to image compression:
    http://linkedin.com/groups/Image-Compression-3363256

  4. #3
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    520
    Thanks
    196
    Thanked 744 Times in 301 Posts
    @Gotty
    v127 doesn't compile on MSVC++ because of the new english stemmer, I'll fix it in the next release.

    @Alexander Rhatushnyak
    paq8px can use itself to pretrain the new exe model, so using that option will always produce different compressed files, even on Windows because each build is different.
    Since paq8px and cmix are just research projects, i.e., not intended for real world usage, I don't think that is a problem.

  5. #4
    Member Alexander Rhatushnyak's Avatar
    Join Date
    Oct 2007
    Location
    Canada
    Posts
    232
    Thanks
    38
    Thanked 80 Times in 43 Posts
    I started compressing LPCB images on 64-bit Ubuntu using Wine and the provided Windows executable of paq8px_v127, this works seemingly well, and hopefully compression will complete tomorrow, and decompression on Jan.2.

    Quote Originally Posted by mpais View Post
    so using that option will always produce different compressed files, even on Windows because each build is different
    If even without that option some compressed files on Linux and Windows are incompatible,
    this should be at least mentioned in the readme.txt, don't you think so?

    This newsgroup is dedicated to image compression:
    http://linkedin.com/groups/Image-Compression-3363256

  6. #5
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    520
    Thanks
    196
    Thanked 744 Times in 301 Posts
    Sure, I guess we can make a simple readme.txt, especially now that more people are getting interested in it.
    I had planned on making documentation explaining not only how to use paq8px, but also its various components and the design decisions that went into making them (the introduction text that you see in the code is very outdated (circa v75, I believe) and a lot has changed), but I don't really have much time for that.
    I've just been focusing on making improvements to it because those can then be ported to paq8pxd and cmix, so it seems to be the best use of my time.

  7. #6
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Quote Originally Posted by Alexander Rhatushnyak View Post
    I suggest making sure it runs correctly on Linux, after you make it compile on Linux. If it produces different compressed files on Linux and Windows, this is a serious issue and should be fixed. I guess cmix has this problem (but I can't run cmix on my computers: it needs much more RAM than they have)
    Quote Originally Posted by mpais View Post
    @Alexander Rhatushnyak
    paq8px can use itself to pretrain the new exe model, so using that option will always produce different compressed files, even on Windows because each build is different.
    Since paq8px and cmix are just research projects, i.e., not intended for real world usage, I don't think that is a problem.
    As soon as building instructions are available for Linux, I'll create a Dockerfile we can add to the GitHub repository and to Docker Hub (for an example, look at this repository and this Docker hub entry for FLIF). Wanted to do this anyway, but what you describe (being able to reproduce compressed files) even gives a perfect use case now.

    Since Docker virtualizises the whole environment, we get completely reproducable executables that can operate on files outside the Docker container on every platform that supports Docker (64-bit Windows 7 or higher, Linux, OS X). It also provides an alternative to the build instructions for all those platforms: install docker, then 1) run the image directly for an old version or 2) build a new image from the Dockerfile for the latest git version. If needed, we could even provide 3) check out a git version by tag or commit id and build this one.
    http://schnaader.info
    Damn kids. They're all alike.

  8. #7
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    343
    Thanks
    235
    Thanked 226 Times in 123 Posts
    Updated first post with:
    • Windows: How to set up the editor in Visual Studio 2017 to prevent the annoying autoformatting
    • Windows: How to make an optimal MinGW-w64 build

  9. #8
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    343
    Thanks
    235
    Thanked 226 Times in 123 Posts
    Updated first post with:

    • Suggested MinGW-w64 compiler version is changed from [Threads=posix] to [Threads=win32] saving ~23 KB in compiled executable
    • Modified MinGW-w64 build scripts (suppressed file not found errors on cleanup)
    • Supplied a quick build script (without PGO) for MinGW-w64 (see paq8px\mingw\_0_quick_build.cmd)
    • Updated MinGW-w64 build instructions

  10. The Following 2 Users Say Thank You to Gotty For This Useful Post:

    pinguin2 (28th January 2018),Simorq (15th January 2018)

  11. #9
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    216
    Thanks
    97
    Thanked 128 Times in 92 Posts
    I followed Paq8px building instructions to build a quick (_0_quick_build) and a profiled (_1_build_gen_profile + _2_run_paq + _3_build_use_profile) versions with g++ (x86_64-win32-sjlj-rev1, Built by MinGW-W64 project) 7.1.0.

    2 Maximum Compression files has different compressed size:
    Code:
    Quick build:
      857.915 AcroRd32.exe.paq8px141
    1.214.818 MSO97.DLL.paq8px141
    
    Profiled build:
      857.948 AcroRd32.exe.paq8px141
    1.214.793 MSO97.DLL.paq8px141
    "Building paq8px from source": in paq8px\howto\index2_zlib_source.html the link "MinGW-w64 setup" at the bottom of the page is broken, it should links to index41_mingw_install.html, not to paq8px\howto\index4_mingw.html.

    Running _2_run_paq.cmd gives "Specified input is a directory but should be a file: ..\profile-suite" and paq8px exits immediately.

  12. #10
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    47
    Thanks
    3
    Thanked 7 Times in 7 Posts
    Quote Originally Posted by Mauro Vezzosi View Post
    I followed Paq8px building instructions to build a quick (_0_quick_build) and a profiled (_1_build_gen_profile + _2_run_paq + _3_build_use_profile) versions with g++ (x86_64-win32-sjlj-rev1, Built by MinGW-W64 project) 7.1.0.

    2 Maximum Compression files has different compressed size:
    Code:
    Quick build:
      857.915 AcroRd32.exe.paq8px141
    1.214.818 MSO97.DLL.paq8px141
    
    Profiled build:
      857.948 AcroRd32.exe.paq8px141
    1.214.793 MSO97.DLL.paq8px141
    "Building paq8px from source": in paq8px\howto\index2_zlib_source.html the link "MinGW-w64 setup" at the bottom of the page is broken, it should links to index41_mingw_install.html, not to paq8px\howto\index4_mingw.html.

    Running _2_run_paq.cmd gives "Specified input is a directory but should be a file: ..\profile-suite" and paq8px exits immediately.
    Is prediction or probability counting in paq not stable ?

  13. #11
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    343
    Thanks
    235
    Thanked 226 Times in 123 Posts
    It is stable - as much as I have tested it so far. The result must be the same when compiling it with a different compiler (Visual C++ in my case) or with the same compiler with different optimization options (MinGW-w64). I'll look into it.
    I suspect a processor bug: it probably kicks in when the system (data registers, flags, memory) is in a specific state.

    @Mauro, did you encounter the problem on your i3-3240?

  14. #12
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    216
    Thanks
    97
    Thanked 128 Times in 92 Posts
    Quote Originally Posted by Gotty View Post
    @Mauro, did you encounter the problem on your i3-3240?
    I can't test it now because I haven't the quick build on i3-3240, tomorrow I'll run the quick and the profiled builds on it (my previous tests was on i7-4710HQ).

  15. #13
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    343
    Thanks
    235
    Thanked 226 Times in 123 Posts
    Oh, I thought the "difference" problem is on your i3-3240. So you have "difference" problems on both systems?

  16. #14
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    343
    Thanks
    235
    Thanked 226 Times in 123 Posts
    I could reproduce the problem. Thanx for all the info! I'm on it.
    It's not a cpu issue.

    Edit:
    Narrowed it down. The "difference" problem between a quick build and a profiled build is caused by the -ftree-vrp optimization switch present at -O2 and higher.
    Quick solution: if you include -fno-tree-vrp optimization switch the problem goes away.
    The problem is present only with the quick build.
    Something in the source code confuses/bothers GCC. I'll find it.
    Last edited by Gotty; 9th April 2018 at 22:53.

  17. #15
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    216
    Thanks
    97
    Thanked 128 Times in 92 Posts
    > Oh, I thought the "difference" problem is on your i3-3240. So you have "difference" problems on both systems?
    I found it on i7-4710HQ and I haven't tested it on i3-3240.
    I guess I can avoid to test it also on i3-3240, now we know where is the problem.

    > I'll find it.
    It seems to me that it's not easy to find the problem in paq8px.
    Could it be a problem in g++ instead? My version is 7.1.0, now there is 7.3.0.
    Thank you for everything you do!

  18. #16
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    343
    Thanks
    235
    Thanked 226 Times in 123 Posts
    Yes. You don't need more tests.
    The proplem exists starting from v140. I have the exact line of the code now (in FrenchStemmer::Stemmer()). But it does not make any sense, yet. The code looks all right to me.
    It is probably an uninitialized variable somewhere or some "light" memory corruption, or buffer overflow. I already found an uninitialized variable (Word::Language), fixed, but it was not that.
    My search goes on. I will come back soon.

    Edit: fixed in v141fix1. See the paq8px thread.
    Last edited by Gotty; 26th May 2018 at 19:07. Reason: Found the problem

  19. The Following 2 Users Say Thank You to Gotty For This Useful Post:

    Mauro Vezzosi (10th April 2018),Mike (10th April 2018)

  20. #17
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    343
    Thanks
    235
    Thanked 226 Times in 123 Posts
    Updated first post with:

    • Added instructions for Linux/GCC
    • Added instructions for Linux/Clang+LLVM
    • Modified the Windows/MinGW-w64 build scripts to make a faster executable in accordance with https://encode.ru/threads/342-paq8px...ll=1#post55981
    • Reorganized file names.
    • All pngs are now 8bit instead of 24bit + recompressed them to make the guide smaller.
    • Fixed problems reported by Mauro: fixed broken link and updated the pgo instructions for newer paq8px versions with the new command line interface (v137+)

    Extra:
    If you have Windows and would like to have a Linux development/test platform as well, here you will find a chapter with instrunctions to set up a Virtual Machine with Linux (Ubuntu) equipped with the necessary development tools and shared folders.
    This chapter of the guide is a general one - it may be useful for anyone wishing to do some cross-platform C++ development.

  21. The Following User Says Thank You to Gotty For This Useful Post:

    Darek (26th May 2018)

  22. #18
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    464
    Thanks
    202
    Thanked 81 Times in 61 Posts
    Hello Gotty! I just downloaded your guide. That is one comprehensive document! I'm impressed

    I have a quick question, though, if you don't mind: How much faster can profiling make the binary in your experience? Some 10-15% would be just great for me. Is it reasonable to expect something like that?

    As a suggestion, if you have this info, it could be included in the guide...

  23. #19
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    343
    Thanks
    235
    Thanked 226 Times in 123 Posts
    Hello Gonzalo! Thanx for the feedback!
    In case of paq8px the difference is just 1-2%. Sometimes they run at the same speed. But the profiled exe is much smaller.
    I have still some things on my to do list: an appendix with the useful gcc optimization switches with speed and size comparison to other compilers; and a "How to make a profiled build with Visual Studio". Maybe around Christmas.
    Last edited by Gotty; 7th June 2018 at 00:50.

  24. The Following User Says Thank You to Gotty For This Useful Post:

    Gonzalo (8th June 2018)

  25. #20
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    343
    Thanks
    235
    Thanked 226 Times in 123 Posts
    Warning:
    Avoid using -ffast-math (/fp:fast in vc++) when compiling paq8px.

    More: https://encode.ru/threads/342-paq8px...ll=1#post58227

Similar Threads

  1. paq8px
    By Jan Ondrus in forum Data Compression
    Replies: 1637
    Last Post: 15th July 2019, 17:58
  2. benchmark of mixed data types with SIMD instructions
    By just a worm in forum The Off-Topic Lounge
    Replies: 8
    Last Post: 10th June 2015, 22:28
  3. PAQ8PX for Mac OS X?
    By Shoal in forum Data Compression
    Replies: 11
    Last Post: 8th July 2011, 17:35
  4. Alternative paq8px builds
    By M4ST3R in forum Download Area
    Replies: 20
    Last Post: 25th June 2010, 16:19
  5. Replies: 0
    Last Post: 26th July 2007, 18:47

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •