Page 1 of 2 12 LastLast
Results 1 to 30 of 44

Thread: Raising performance bar in arithmetic coding

  1. #1
    Member
    Join Date
    Feb 2011
    Location
    Moscow, Zelenograd
    Posts
    21
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Thumbs up Raising performance bar in arithmetic coding

    Raising performance bar in arithmetic coding

    We are proud to present to you the optimizied implementation of a new adaptive arithmetic coder FArCo, developed for the WMC (Wavelet Media Codec) video codec project. Coding performance is nearly the same as for such coders as Exponential-Golomb codes or Huffman coding, while its efficiency approaches that of the classical arithmetic coder. More detailed specification can be found on our website STBIT.RU in solutions section.




    Soon, the new arithmetic coder will be integrated into the WMC video codec, so its specifications will be updated. Lets subscribe for the news.

  2. #2
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Some quick thoughts:
    1 - We here traditionally expect working order-0 codec for a new proposed entropy coder to compare ours mostly used implementations. I couldn't find any information about how did you implemented it. So, I couldn't comment anything about it.
    2 - As to video codec, HuffYUV and Lagarith same in way (don't know anything about h264 lossless mode). Both HuffYUV and Lagarith use almost same static predictors over the video. Except that Lagarith uses other ways for entropy coding. And I really doubt that wavelet could be "really efficient" for lossless signal (audio/image/video) compression unless they are ex-lossy compressed signals. Wavelet is good at to correlate with quantized DCT coefficients, so no wonder it compresses better than static predictors (Am I wrong that you really used a really pure source for testing?).

    Overall, it's good to see some progress in compression world.
    BIT Archiver homepage: www.osmanturan.com

  3. #3
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts

  4. #4
    Member
    Join Date
    Feb 2011
    Location
    Moscow, Zelenograd
    Posts
    21
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Yes it`s only demonstration of performances

    It's commercial product and not potented yet. But i can give you a binary for testing...

  5. #5
    Member
    Join Date
    Feb 2011
    Location
    Moscow, Zelenograd
    Posts
    21
    Thanks
    0
    Thanked 0 Times in 0 Posts

    ... about testing WMC

    I've no time for global testing our video codec solution, but i do it for real testing sources when second version of WMC will be ready.

  6. #6
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > But i can give you a binary for testing...

    Then please send it to shelwien@gmail.com
    Also I really would like to know whether its derived from vecrc, or is an independent implementation.

  7. #7
    Member
    Join Date
    Feb 2011
    Location
    Moscow, Zelenograd
    Posts
    21
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Shelwien View Post
    > But i can give you a binary for testing...

    Then please send it to shelwien@gmail.com
    Ok, but i need a time to prepare it. (some days)

    Quote Originally Posted by Shelwien View Post
    Also I really would like to know whether its derived from vecrc, or is an independent implementation.
    I haven't any information about vecrc yet.

  8. #8
    Programmer Gribok's Avatar
    Join Date
    Apr 2007
    Location
    USA
    Posts
    159
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by stbit View Post
    Raising performance bar in arithmetic coding
    What's the point for this AD? We can't review your work unless you provide more details. From description We can see that the main idea is to use SIMD. This idea was originally discovered by Shelwien and implemented in vecrc.
    Enjoy coding, enjoy life!

  9. #9
    Member
    Join Date
    Feb 2011
    Location
    Moscow, Zelenograd
    Posts
    21
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Gribok View Post
    What's the point for this AD? We can't review your work unless you provide more details.
    You can't review, but you can check and try the performance, and try to do faster...

    Quote Originally Posted by Gribok View Post
    From description We can see that the main idea is to use SIMD. This idea was originally discovered by Shelwien and implemented in vecrc.
    SIMD is only hardware parallelism for x86 platform - i can implement it using others...
    look at the article - THE MAIN IDEA IS PARALLELISM (and some simplification on early stage)...

    ...
    The presented implementation of the arithmetic coder has the following features and
    limitations:
    ? input alphabet size is 4 bits [0 .. 15] or less which if motivated by the usage in the video
    codec
    ? adaptive model is periodic
    ? calculations are performed mostly using 16-bit integer arithmetic, except for a few
    additions and subtractions with 32-bit arguments
    ? divisions eliminated
    ? the minimum number of conditional jumps (only one for 16 symbols)
    ? simplified search on the decoder?s side (practically none)
    ? x86-based platform requires SSE2 instructions (future implementations will use SSSE3
    instructions).
    ...

    Do vec_rc have the same features??? - I don't know yet...

  10. #10
    Member
    Join Date
    Feb 2010
    Location
    Nordic
    Posts
    200
    Thanks
    41
    Thanked 36 Times in 12 Posts
    Quote Originally Posted by stbit View Post
    SIMD is only hardware parallelism for x86 platform - i can implement it using others...
    That is not true. SIMD literally means SINGLE INSTRUCTION MULTIPLE DATA and Shelwein's example could be implemented in a range of actual instruction sets e.g. altivec or SSE or such.

    can understand your discomfort with discussion about prior art when you mean to pursue a patent.

  11. #11
    Member
    Join Date
    Feb 2011
    Location
    Moscow, Zelenograd
    Posts
    21
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by willvarfar View Post
    That is not true. SIMD literally means SINGLE INSTRUCTION MULTIPLE DATA and Shelwein's example could be implemented in a range of actual instruction sets e.g. altivec or SSE or such.
    And i mean the same , the main idea is parallelism when we talk about SIMD, but i can implement the algorithm without parallel instructions at all and the benefit still be high.

  12. #12
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    I'd like to compare it to http://www.ctxmodel.net/files/fpaq0pv4b3.rar (specifically fpaq0pv4Bmm_O3_xN_ipo.exe there)
    on enwik8/9 - http://www.cs.fit.edu/~mmahoney/comp.../textdata.html
    (fpaq0p is an order0 coder).
    Also note that a fast adaptive entropy coder can be useful for BWT compression - see http://libbsc.com/downloads.aspx

  13. #13
    Member
    Join Date
    Feb 2011
    Location
    Moscow, Zelenograd
    Posts
    21
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Exclamation Test project available on site stbit.ru

    You can download and try it now .

    The test project available here. RAR contains the source code of the test project and the protected dll with the encoder FArCo.

    P.S. password for the rar is "stbit.ru", for getting the key send me your login name and hardware fingerprint.

  14. #14
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    What's the big idea for such protection for just a test program? You may sure that, if someone would really need your works, they can still reserve-engineer it.
    BIT Archiver homepage: www.osmanturan.com

  15. #15
    Member
    Join Date
    Feb 2011
    Location
    Moscow, Zelenograd
    Posts
    21
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by osmanturan View Post
    What's the big idea for such protection for just a test program? You may sure that, if someone would really need your works, they can still reserve-engineer it.
    Anything is better than nothing at all.
    I hope at least won the time, and the code there is not really optimized as compared with the current implementation - if the source will be opened, it will be out of date.

    Well, that it can be hacked- does not even mention it, I understand perfectly. But this is a conversation not quite to the topic.

  16. #16
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by stbit View Post
    Anything is better than nothing at all.
    No. You can be sure few will try it and even fewer will pass a word.
    As a user, tester and preacher I don't like to be treated this way. Goodbye.

  17. #17
    Member
    Join Date
    Feb 2011
    Location
    Moscow, Zelenograd
    Posts
    21
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by m^2 View Post
    No. You can be sure few will try it and even fewer will pass a word.
    As a user, tester and preacher I don't like to be treated this way. Goodbye.
    It's not the problem for me at all, it's even better for me . To provide a test of all is not my goal.

  18. #18
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    download & unrar the file with password=stbit.ru
    ---
    farco.exe does not start ... side-by-side-config ...
    ---
    any hints or commandline-parameters ?
    ---
    seems (if running) to have an interesting output on console:

    CS - compressed size
    ET - encode time (ms) , DT - decode time (ms)
    ES - encode speed (mbit/s) , DS - decode speed (mbit/s)
    CR - compression ratio

    i do not understand: why here is a sourcecode, if you do not want that anybody can look inside ?

    best regards
    Last edited by joerg; 13th April 2011 at 18:55.

  19. #19
    Member
    Join Date
    Feb 2011
    Location
    Moscow, Zelenograd
    Posts
    21
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Exclamation

    Quote Originally Posted by joerg View Post
    download & unrar the file with password=stbit.ru
    ---
    farco.exe does not start ... side-by-side-config ...
    ---
    any hints or commandline-parameters ?
    ...
    i do not understand: why here is a sourcecode, if you do not want that anybody can look inside ?

    best regards
    This is the source code for testing application, dll is protected (you can't start application in debug mode, and even if soft ice installed on your system) - for use the application firstly run the release version of farco in release folder. After that the windows with login and hardware fingerprint appears - send me you login name and hardware code only one time and I will try to answer you with the KEY as quickly as possible.

    It only takes a few seconds to email you info or leave it in the forum and after that you can try the FArCo

    Sorry, but this is a compulsory measure

  20. #20
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    856
    Thanks
    447
    Thanked 254 Times in 103 Posts
    send me you login name and hardware fingerprint code
    It only takes a few seconds
    Ah, sorry, i only have a retina infrared lock sensor.
    It's currently on maintenance....

  21. #21
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    run the release version of farco in release folder.
    After that the windows with login and hardware fingerprint appears

    on my computer (Windows Vista SP2) this program seems not working

    NO SUCH WINDOWS APPEARS ... sorry ...

  22. #22
    Member
    Join Date
    Feb 2011
    Location
    Moscow, Zelenograd
    Posts
    21
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Unhappy

    Quote Originally Posted by joerg View Post
    on my computer (Windows Vista SP2) this program seems not working

    NO SUCH WINDOWS APPEARS ... sorry ...
    This is the problem - I'll try to resolve it with Vista: (- I develope it on XP...

  23. #23
    Member
    Join Date
    Feb 2011
    Location
    Moscow, Zelenograd
    Posts
    21
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Cyan View Post
    Ah, sorry, i only have a retina infrared lock sensor.
    It's currently on maintenance....
    what is Retina infrared lock sensor?

  24. #24
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    It's also known as an "eye scanner"

    App seems to work on Win7.
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  25. #25
    Member
    Join Date
    Feb 2011
    Location
    Moscow, Zelenograd
    Posts
    21
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Question

    Quote Originally Posted by Cyan View Post
    Ah, sorry, i only have a retina infrared lock sensor.
    It's currently on maintenance....
    Quote Originally Posted by Black_Fox View Post
    It's also known as an "eye scanner"

    App seems to work on Win7.
    Does it the reason of problems to run the program?

  26. #26
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by stbit View Post
    Does it the reason of problems to run the program?
    Just read again that comment. It's technically equivalent.
    BIT Archiver homepage: www.osmanturan.com

  27. #27
    Member
    Join Date
    Feb 2011
    Location
    Moscow, Zelenograd
    Posts
    21
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by joerg View Post
    run the release version of farco in release folder.
    After that the windows with login and hardware fingerprint appears

    on my computer (Windows Vista SP2) this program seems not working

    NO SUCH WINDOWS APPEARS ... sorry ...
    Quote Originally Posted by osmanturan View Post
    Just read again that comment. It's technically equivalent.
    And you too . I ask whether the retina infrared lock sensor of a hindrance to run FArCo_test?

  28. #28
    Programmer Gribok's Avatar
    Join Date
    Apr 2007
    Location
    USA
    Posts
    159
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by stbit View Post
    It's not the problem for me at all, it's even better for me . To provide a test of all is not my goal.
    It is still not clear for me what you want. But if you want us to test your program please release it without fingerprint protection and don't waste our time.
    Enjoy coding, enjoy life!

  29. #29
    Member
    Join Date
    Feb 2011
    Location
    Moscow, Zelenograd
    Posts
    21
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Gribok View Post
    It is still not clear for me what you want. But if you want us to test your program please release it without fingerprint protection and don't waste our time.
    I've distribute the test only in order not to be unsubstantiated, if you do not want to make sure the results, then do not have anything to test (I already did).
    As for fingerprints, it is only two numbers that characterize your equipment, I do not understand what the problem is to exchange them for the key?

    I don't need to distribute the test. I am linked to specific machines for a little control the distribution and as a defense.

  30. #30
    Member Alexander Rhatushnyak's Avatar
    Join Date
    Oct 2007
    Location
    Canada
    Posts
    232
    Thanks
    38
    Thanked 80 Times in 43 Posts

    Question

    Quote Originally Posted by stbit View Post
    i can implement the algorithm without parallel instructions at all and the benefit still be high.
    AFAIU from this and from authors thread on forum.compression.ru, the code has either MMX or SSE instructions. Most probably SSE. Questions: any SSE or something more specific, like SSE2+ ? what about adding to the graph results of the code with no MMX and SSE?
    Last edited by Alexander Rhatushnyak; 14th April 2011 at 00:57.

    This newsgroup is dedicated to image compression:
    http://linkedin.com/groups/Image-Compression-3363256

Page 1 of 2 12 LastLast

Similar Threads

  1. Minimal Ashford arithmetic-coder termination
    By Ethatron in forum Data Compression
    Replies: 18
    Last Post: 15th January 2011, 15:38
  2. BCM 0.11 - A high performance BWT compressor
    By encode in forum Data Compression
    Replies: 44
    Last Post: 29th October 2010, 22:45
  3. Idea for raising compression efficiency on disk images
    By Mexxi in forum Data Compression
    Replies: 10
    Last Post: 18th February 2010, 05:56
  4. flzp_ac2 (flzp + an order-2 arithmetic coder)
    By inikep in forum Data Compression
    Replies: 4
    Last Post: 25th June 2008, 21:37
  5. Better compression performance across time?
    By Trixter in forum Data Compression
    Replies: 16
    Last Post: 16th June 2008, 23:35

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •