Results 1 to 9 of 9

Thread: simple low order encoders (java)

  1. #1
    Member
    Join Date
    Feb 2015
    Location
    United Kingdom
    Posts
    154
    Thanks
    20
    Thanked 66 Times in 37 Posts

    simple low order encoders (java)

    https://github.com/Pakams/simple-low-order-transform

    Here's some encoders I wrote, it only uses orders 1 and 2. On my machine I get about 20MB/s but that's mainly due to the converting loop at the input/output buffer, it should run faster if I clean up those few lines of code.

    Thoughts?

  2. #2
    Member RichSelian's Avatar
    Join Date
    Aug 2011
    Location
    Shenzhen, China
    Posts
    156
    Thanks
    18
    Thanked 50 Times in 26 Posts
    Quote Originally Posted by Lucas View Post
    https://github.com/Pakams/simple-low-order-transform

    Here's some encoders I wrote, it only uses orders 1 and 2. On my machine I get about 20MB/s but that's mainly due to the converting loop at the input/output buffer, it should run faster if I clean up those few lines of code.

    Thoughts?
    java is too slow to do such things.

  3. #3
    Member
    Join Date
    Dec 2011
    Location
    Cambridge, UK
    Posts
    437
    Thanks
    137
    Thanked 152 Times in 100 Posts
    There is a Java implementation of the 4-way unrolled order-0 and order-1 rANS codec here:

    https://github.com/samtools/htsjdk/b...rans/RANS.java

    This isn't my code, although it's based on my code (which is in turn based on Fabian Giesen's code). I think it's relatively fast, but I don't really do much with Java so I haven't benchmarked it.

  4. The Following 2 Users Say Thank You to JamesB For This Useful Post:

    Bulat Ziganshin (28th August 2015),RichSelian (29th August 2015)

  5. #4
    Member
    Join Date
    Feb 2015
    Location
    United Kingdom
    Posts
    154
    Thanks
    20
    Thanked 66 Times in 37 Posts
    Quote Originally Posted by RichSelian View Post
    java is too slow to do such things.
    Yeah... I plan on learning C in the next year, honestly I only use Java since it's easy to read.

  6. #5
    Member
    Join Date
    Nov 2014
    Location
    California
    Posts
    122
    Thanks
    36
    Thanked 33 Times in 24 Posts
    "java is too slow to do such things."
    Is it ? Except for specific cases (where the run time is so small that the JVM warm up has a significant impact), java is fast. I am surprised that most people point to speed while the major issue with java is more likely JVM memory usage.
    From this code
    https://github.com/flanglet/kanzi/blob/master/java/src/kanzi/transform/MTFT.java, the MoveToFront transform runs at 43 MB/s for enwik8 on my system. MTF run time is very sensitive to the statistics of the input data ...

    "There is a Java implementation of the 4-way unrolled order-0 and order-1 rANS codec here"

    My simple RANS implementation (also derived from Fabian Giesen's code) here https://github.com/flanglet/kanzi/blob/master/java/src/kanzi/entropy/ANSRangeEncoder.java gives 50+ MB/s encoding and 120+ MB/s decoding on my system. It could be made faster by multiplexing RANS bit streams and replacing the division/modulus by an inverse multiplication.


  7. #6
    Member
    Join Date
    Feb 2015
    Location
    United Kingdom
    Posts
    154
    Thanks
    20
    Thanked 66 Times in 37 Posts
    Quote Originally Posted by hexagone View Post
    the MoveToFront transform runs at 43 MB/s for enwik8 on my system
    My version currently uses arrays which I know are not the fastest system for updating the MTF table. But I tried to minimize the cost of computing the MTF by using different states to select the index. This is where the ordered models kind of take effect.

  8. #7
    Member
    Join Date
    Feb 2015
    Location
    United Kingdom
    Posts
    154
    Thanks
    20
    Thanked 66 Times in 37 Posts
    Anyway I'm going to try making a few (high-order bijective) statistical models in Java before I try my hand at C. I might try making something like sr2, except I'll not include any entropy encoder; it'll just be a transform for testing out different secondary encoding schemes like this release was.

  9. #8
    Member
    Join Date
    Dec 2012
    Location
    japan
    Posts
    149
    Thanks
    30
    Thanked 59 Times in 35 Posts
    I think C++ is better than C for you. Because Java is similar to C++

  10. The Following User Says Thank You to xezz For This Useful Post:

    Lucas (29th August 2015)

  11. #9
    Member
    Join Date
    Dec 2011
    Location
    Cambridge, UK
    Posts
    437
    Thanks
    137
    Thanked 152 Times in 100 Posts
    Quote Originally Posted by hexagone View Post
    My simple RANS implementation (also derived from Fabian Giesen's code) here https://github.com/flanglet/kanzi/blob/master/java/src/kanzi/entropy/ANSRangeEncoder.java gives 50+ MB/s encoding and 120+ MB/s decoding on my system. It could be made faster by multiplexing RANS bit streams and replacing the division/modulus by an inverse multiplication.
    Vadim, who converted to Java the version of rANS I was using, was impressed by the speed of the result. However IIRC he experiments with different multiplexing and found it didn't change speed much. This is hardly suprising as in C it is carefully trying to ensure that it has multiple identical operations to perform at the same time. The compiler changes some of these to SIMD (but not all), plus any stalls in memory fetches don't lead to bottlenecks as it has independent things to be getting on with. I'd be suprised if all of these still happen when going via the JVM, but then again modern JVMs are doing on-the-fly assembly generation.

    The division lookup table would still work, but only impacts on encoding time.

Similar Threads

  1. Kanzi: Java, Go and C++ compressors
    By hexagone in forum Data Compression
    Replies: 18
    Last Post: 10th July 2019, 02:20
  2. list of different jpeg encoders?
    By Mangix in forum Data Compression
    Replies: 4
    Last Post: 21st October 2013, 16:11
  3. Simple Variable Order Rangecoder
    By david_werecat in forum Data Compression
    Replies: 62
    Last Post: 4th January 2012, 18:59
  4. Java port of TarsaLZP
    By Piotr Tarsa in forum Data Compression
    Replies: 19
    Last Post: 8th July 2009, 06:46
  5. JAVA vs C++
    By Nania Francesco in forum Forum Archive
    Replies: 5
    Last Post: 20th January 2008, 22:07

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •