Results 1 to 26 of 26

Thread: PPMd reformatting

  1. #1
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts

    PPMd reformatting

    http://ctxmodel.net/files/PPMd/ppmd_Jr1_sh1.rar
    + weird CLI replaced with simpler one
    + decode/encode functions merged
    + 30k source size instead of original 64k
    + one multiplication removed in binary rc,
    also precision slightly improved (and compression)
    - might be not portable to 64-bit systems atm

  2. #2
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 166 Times in 63 Posts
    thanks Eugene, a good work

  3. #3
    Member
    Join Date
    Jan 2007
    Location
    Moscow
    Posts
    239
    Thanks
    0
    Thanked 3 Times in 1 Post
    Thank you. Can't run on my AMD Turion, because it is not Pentium 4

  4. #4
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Its not really for running, but I patched it and reuploaded.
    I just always hope that people would remember how to
    patch these IntelC binaries

  5. #5
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    stops at 36mb of enwik

  6. #6
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    http://ctxmodel.net/files/PPMd/ppmd_Jr1_sh1.rar
    + GlueFreeBlocks() mistake fixed
    + PrintInfo() mistake fixed
    + -r1 support added (via c1/d1 options)

  7. #7
    Member
    Join Date
    Jul 2008
    Posts
    54
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I tried, but the file ppmd.exe crashes in winxp SP2 32bits using Quad core with 4gb's of ram
    The file PPMD0.exe works fine

  8. #8
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    There're just no validity checks.
    Anyway, the main point here is source.
    So, if you don't run the executable with supported commandline:
    (equivalent to ppmd e -o16 -m256)
    ppmd c input.file output.file
    ppmd d input.file output.file
    or
    (equivalent to ppmd e -o16 -m256 -r1)
    ppmd c1 input.file output.file
    ppmd d1 input.file output.file
    with input files existing and having a proper format,
    it would fall with exception.

  9. #9
    Member
    Join Date
    Jul 2008
    Posts
    54
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Well now runs, but crashes after few secons (25 secs aprox) encoding

  10. #10
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    http://ctxmodel.net/files/PPMd/ppmd_Jr1_sh2.rar
    - StartModel, UpdateModel and CreateSuccessors processed
    - some things tweaked, compression should be slightly better again

    As to crashes, I dunno.

  11. #11
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    http://ctxmodel.net/files/PPMd/ppmd_Jr1_sh3.rar
    - Reformatting complete
    - command line verification added
    - compression reversibility checked:
    Code:
    Jr1:     21562077 // ppmd e -o16 -m256 -r1 enwik8
    Jr1_sh3: 21520816 // ppmd c1 enwik8 enwik8.pmd

  12. #12
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    @Shelwien: Thank you so much for your great effort!

  13. #13
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Jr1_sh3 was the last "pure PPMd" version
    The rest would certainly introduce various "anti-optimizations".

    http://ctxmodel.net/files/PPMd/ppmd_Jr1_sh4.rar
    - "Model" class created, with ProcessByte function in it
    - processFile function implemented via ProcessByte and moved to ppmd.cpp
    - processSymbol functions moved from PPM_CONTEXT class into Model class.
    - stats printed with 0.5s interval, instead of every 64k
    - sh_v1m rangecoder attached
    Code:
    Jr1_sh3:  21520816
    Jr1_sh4:  21515922 // mostly due to carryless coding redundancy
    Jr1_sh4:  21515928 // using freq*(range/total) instead of 64-bit ops

  14. #14
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    http://ctxmodel.net/files/PPMd/ppmd_Jr1_sh5.rar
    - coding functions updated
    - rc calls in branches merged

    Code:
    Jr1:      21562077 // original
    Jr1_sh3:  21520816 // lost intervals in binary coding reclaimed + tweaks
    Jr1_sh4:  21515922 // full rangecoder (with carries)
    Jr1_sh5:  21511981 // binary SEE fix
    http://ctxmodel.net/files/PPMd/ppmd_Jr1_sh5a.rar
    - sh5 with additional statistics:

    Code:
    C:\>ppmd.exe c1 enwik8 1
      100000000 >21511975, 1.29 bpb, used:229.2MB, speed: 1342 KB/sec
    Processing time: 72.718s
    Total compressed = 21511981
      Binary::Escape = 1948713
      Binary::Symbol = 1570420
     Unmasked::rank0 = 1994736
    Unmasked::Escape = 1563951
    Unmasked::Symbol = 3944554
      Masked::Escape = 1823740
      Masked::Symbol = 8665859
         Counter Sum = 21511973

  15. #15
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    Interesting, is Dmitry Shkarin reading this forum? It might be that he will release PPMDvar.K!

  16. #16
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    The compression improvement from my tweaks is only around 0.2% for now, its insignificant.
    But I'd like to see the improved ppmonstr, i guess, with all his talks about TSE and such.

  17. #17
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    TSE? Who mentioned about it again?

  18. #18
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    TSE is some probability adjustment technique by Dimitry. There was a post on the old forum.

    http://encode.ru/forums/index.php?ac...page=0#msg8896

  19. #19
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Crashing on one of my files inc1 mode. I can mail the file.

  20. #20
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Thanks, that's probably the same bug i'd seen.
    The question is whether you really want me to fix it

  21. #21
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Not very much.

  22. #22
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    http://ctxmodel.net/files/PPMd/ppmd_Jr1_sh5.rar
    http://ctxmodel.net/files/PPMd/ppmd_Jr1_sh5a.rar
    - FIX: added missing line in ExpandTextArea()
    (Note: sh5a is sh5 with additional statistics and thus slower)

  23. #23
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    http://ctxmodel.net/files/PPMd/ppmd_Jr1_sh6.rar
    + File descriptions added
    + Version info added
    + version with model stats merged into main branch (enabled by macro)
    + real CLI added (ported from ash)
    + E8 included

  24. #24
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    http://ctxmodel.net/files/PPMd/ppmd_Jr1_sh7.rar
    + file header processing branches merged (encoding/decoding)
    + PrintInfo() replaced with simpler rewrite (with valid bpb)
    + main processing loop moved into file_process.inc
    + EOF symbol check removed (redundancy remains)
    + EscCount turned uint, CharMask flush removed
    + all global vars are incapsulated into Model class
    + final file stats fixed
    + UseE8 turned into template parameter

    Code:
    ppmd /e enwik8
     100000000 > 22774309, 1.82 bpb, 2719Kb/s, Encoding time: 35.938s
     22774309 > 100000000, 1.82 bpb, 2573Kb/s, Decoding time: 37.969s
    ppmd /r /e enwik8
     100000000 > 21511986, 1.72 bpb, 1302Kb/s, Encoding time: 75.032s
     21511986 > 100000000, 1.72 bpb, 1263Kb/s, Decoding time: 77.328s
    ppmd enwik8    // E8 enabled
     100000000 > 22774309, 1.82 bpb, 2687Kb/s, Encoding time: 36.360s
     22774309 > 100000000, 1.82 bpb, 2540Kb/s, Decoding time: 38.453s
    ppmd /r enwik8 // E8 enabled
     100000000 > 21511986, 1.72 bpb, 1296Kb/s, Encoding time: 75.343s
     21511986 > 100000000, 1.72 bpb, 1257Kb/s, Decoding time: 77.719s
    
    PPMd0.exe e -m256 -o16 -r0 enwik8
     100000000 >22825462,  1.40 bpb, 3335Kb/s, Encoding time: 29.312s
    PPMd0.exe e -m256 -o16 -r1 enwik8    
     100000000 >21562055,  1.30 bpb, 1428Kb/s, Encoding time: 68.360s
    (bpb is wrong here, as 1.30/8*100000000=16250000)
    So appears that E8 has some small but visible effect on speed.
    And original ppmd seems significantly faster, but that's probably
    because of too precise rangecoder... would be fixed later.

  25. #25
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    http://ctxmodel.net/files/PPMd/ppmd_Jr1_sh6.rar
    http://ctxmodel.net/files/PPMd/ppmd_Jr1_sh7.rar
    + FIX: mismatch in E8 filter near EOF
    + executable built with PGO for better benchmarking (5% speedup)

    Sami caught a bug in my filter class.
    Thanks, but now I have to repost a lot of stuff %).

  26. #26
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    http://ctxmodel.net/files/PPMd/ppmd_Jr1_sh8.rar
    + support for entropy stats removed
    + ppmd_process.inc renamed to model.inc
    + ProcessByte moved to ppmd_byte.inc
    + RestoreModel() (tree flush) moved from ProcessByte to ProcessFile
    + CutOff completely removed from the model
    + now ProcessFile selects RestoreModel or StartModel
    + file handles removed from model class, along with rc init
    + rangecoder removed from the model, now is external
    + flags (CutOff,UseE8,RCType) packed into byte
    + new, less precise rangecoder added, selectable by /c1
    + E8 filter updated: E9 handling and bigendian conversion disabled

    Code:
    cmp.size enc.tim dec.tim
    20784612 41.203s 42.516s  ppmd_sh7 /q /m1900 enwik8 
    20784611 39.297s 41.156s  ppmd_sh8 /q /m1900 enwik8 
    20784861 37.687s 38.046s  ppmd_sh8 /q /c1 /m1900 enwik8
    
    sh8/c1 encoding faster than sh7 by 8.53%
    sh8/c1 encoding faster than sh8/c0 by 4.09%
    sh8/c1 decoding faster than sh7 by 10.51%
    sh8/c1 decoding faster than sh8/c0 by 7.55%

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •