Results 1 to 16 of 16

Thread: Error recovery from archive?

  1. #1
    Member
    Join Date
    May 2008
    Location
    HK
    Posts
    160
    Thanks
    4
    Thanked 25 Times in 15 Posts

    Error recovery from archive?

    Besides RAR, which archive supports error recovery?
    Will you add such feature when you are designing archiving file format?

  2. #2
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    FreeArc, Sqx.

    Don't know about Sqx, but both FA and RAR are not reliable here, tiny damage in unfortunate place destroys all protection. I find it very sad, because this is a really useful function.

  3. #3
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    There's http://en.wikipedia.org/wiki/Par2

    As to rar, can you tell me where's that "unfortunate place"?
    Once I tried to break it, but failed - it even handled some deleted and inserted bytes.

  4. #4
    Member
    Join Date
    May 2008
    Location
    HK
    Posts
    160
    Thanks
    4
    Thanked 25 Times in 15 Posts
    Quote Originally Posted by Shelwien View Post
    There's http://en.wikipedia.org/wiki/Par2

    As to rar, can you tell me where's that "unfortunate place"?
    Once I tried to break it, but failed - it even handled some deleted and inserted bytes.
    IMO it is hard to say PAR(1/2/3) as "file archives".

  5. #5
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Shelwien View Post
    There's http://en.wikipedia.org/wiki/Par2

    As to rar, can you tell me where's that "unfortunate place"?
    Once I tried to break it, but failed - it even handled some deleted and inserted bytes.
    I got logs somewhere, but now I don't I guess that they got lost when my pendrive broke this summer. I did a quick and rough test to only find a bug where recovery failed to notice and fix a broken recovery record, not data breakage.
    Attached Files Attached Files

  6. #6
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Yes, but I guess they think that recovery is a method to fix an archive when you can't unpack it.
    Restoring lost recovery records probably requires rebuilding them from data, which is slow, so they don't do it.

  7. #7
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Shelwien View Post
    Yes, but I guess they think that recovery is a method to fix an archive when you can't unpack it.
    Restoring lost recovery records probably requires rebuilding them from data, which is slow, so they don't do it.
    Your recovery records might bit-rot and rar doesn't let you know even when you ask it to. This is very wrong even if it's faster.

  8. #8
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    Is this really a useful feature? I mean, how often do you see files lose just a few bits of data?

  9. #9
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    I left my bruteforce script running and got another bug:
    Code:
    Building fixed.tst1.rar
    Scanning...
    Data recovery record found 100%
    Sector 108 (offsets D800...DA00) damaged - cannot recover data
    Reconstruct archive structure ? [Y]es, [N]o
    Sounds bad and like said, cannot recover the original rar. Nevertheless data is intact.

    ADDED:
    Another kind of bug:
    Code:
    Building fixed.tst1.rar
    Scanning...
    RR- ??? - the file header is corrupt
    RR- ??? - the file header is corrupt
    Data recovery record not found
    Reconstructing tst1.rar
    Building rebuilt.tst1.rar
    Found  avutil-51.dll
    Done
    Again, data is intact.
    Attached Files Attached Files
    Last edited by m^2; 5th November 2011 at 20:37.

  10. #10
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    @Matt:
    Yes, its a very useful feature.
    In fact its a "must have" for large archives transferred via non-ECC channels (like http/ftp or usenet).
    And of course its not a matter of "a few bits". Archive recovery is based on methods similar to what's used in RAID, and is able to recover from loss of a few sectors.
    Anyway, I'd suggest to download rar and test something like
    - rar a -rr1% archive data
    - break the archive with a hexeditor (stuff like deleting 512 bytes is ok)
    - rar r archive
    - rar t archive

  11. #11
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Bingo. A single byte corruption that breaks an archive.

    Quote Originally Posted by Matt Mahoney View Post
    Is this really a useful feature? I mean, how often do you see files lose just a few bits of data?
    We posted at the same minute and I missed your question.
    There are many use cases:
    1. Optical backup. A little dated already, but discs loose readability over time, cheap disks often after few years. Initially internal error recovery algorithms work OK, but over time they become insufficient and storing some more of your own is helpful. And more importantly - it's free, your backup often don't take whole disks but only some portions of them and you can either leave the rest empty or fill it with something useful.
    2. Consumer HDDs have bit error ratio of 10e-15 - 10e-14. That's a lot and I've seen corrupted files on my disk. ADDED: CERN made a study on silent data corruption some time ago and found that....firmware bugs in their controllers caused much higher silent error ratio then they should get. So such things happen too. ZFS FTW. ADDED: SSDs have different rotting patterns than HDDs, flash itself is much worse, but controllers actively manage it and move data from time to time. But consumer SSDs are often flaky and you can guess whether producers put a lot of effort for such invisible features.
    3. Cyberlockers. I don't know why, but they are notoriously unreliable, if you use them a lot, you have to be prepared because sooner or later they will loose your data.

    These are only the uses that I had personally, there are likely others too.
    Attached Files Attached Files
    Last edited by m^2; 5th November 2011 at 22:17.

  12. #12
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Thanks, this is interesting.
    Apparently it fixed the actual broken byte, but then inserted a copy of 0xB8 bytes
    around the broken byte (0xD748-0xD7FF) at the start of the next sector (0xD800).
    This may be a bug, why don't you report it
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	tst1a.png 
Views:	333 
Size:	39.4 KB 
ID:	1708  

  13. #13
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Shelwien View Post
    This may be a bug, why don't you report it
    Because I don't care. :P

  14. #14
    Member Surfer's Avatar
    Join Date
    Mar 2009
    Location
    oren
    Posts
    203
    Thanks
    18
    Thanked 7 Times in 1 Post
    KuaiZip have recovery records, AFAIK it is LZMA + simple Reed-Solomon codes.

    Also, interesting video about new tool
    http://www.youtube.com/watch?v=Rf1c0BA-OXk

  15. #15
    Member
    Join Date
    May 2008
    Location
    HK
    Posts
    160
    Thanks
    4
    Thanked 25 Times in 15 Posts
    Quote Originally Posted by Surfer View Post
    KuaiZip have recovery records, AFAIK it is LZMA + simple Reed-Solomon codes.

    Also, interesting video about new tool
    http://www.youtube.com/watch?v=Rf1c0BA-OXk
    Try comparing with MultiPar as QuickPar has no further development in these years.

  16. #16
    Member Surfer's Avatar
    Join Date
    Mar 2009
    Location
    oren
    Posts
    203
    Thanks
    18
    Thanked 7 Times in 1 Post
    Quote Originally Posted by roytam1 View Post
    Try comparing with MultiPar as QuickPar has no further development in these years.
    You can ask author on youtube or try it for yourself.

Similar Threads

  1. Replies: 1
    Last Post: 12th June 2011, 02:01
  2. Unknown archive
    By Surfer in forum The Off-Topic Lounge
    Replies: 6
    Last Post: 15th July 2010, 00:59
  3. Forum archive
    By encode in forum Data Compression
    Replies: 5
    Last Post: 15th May 2008, 11:34
  4. Bit Archive Format
    By osmanturan in forum Forum Archive
    Replies: 39
    Last Post: 29th December 2007, 00:57
  5. New archive format
    By Matt Mahoney in forum Forum Archive
    Replies: 9
    Last Post: 25th December 2007, 12:22

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •