Results 1 to 13 of 13

Thread: precomp error: std::bad_alloc

  1. #1
    Member
    Join Date
    Nov 2011
    Location
    bobrov
    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Red face precomp error: std::bad_alloc

    Hi all,

    I ran precomp on 4GB archive, and it gave error:
    Code:
    94.21% \terminate called after throwing an instance of 'std::bad_alloc'
    what(): std::bad_alloc
    This application has requested the Runtime to terminate it in an unusual way.
    Please contact the application's support team for more information
    Can anyone help me resolve error?

  2. #2
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Using verbose mode (precomp -v ...) could help to get more information on what leads to that crash and which stream causes it.

    Looking at the error message, it looks like this is the Linux version, right? If so, do you have a Windows system available and can you try to run the Windows version of Precomp over that file?

    You can also try to split the file into smaller blocks (50 or 100 MB in size) with some archiver and run Precomp over each block, so that if one of the block still gives the error, it would be easier to upload somewhere so I can have a closer look at it.
    http://schnaader.info
    Damn kids. They're all alike.

  3. #3
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Its likely a failed new[] somewhere (out of memory).
    Could be packjpg, as usual

  4. #4
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    I can only suggest to use Valgrind.
    It's awesome for such bugs.

  5. #5
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Quote Originally Posted by Shelwien View Post
    Its likely a failed new[] somewhere (out of memory).
    Could be packjpg, as usual
    PackJPG isn't available in the Linux version, so I think it could be something different this time (if the bug really appears in the Linux version). Also had a similar bug directly after porting Precomp, it seems that Windows/MinGW doesn't care about fclose on an already closed file, but Linux/g++ does and throws a runtime error.

    If it really is an out of memory problem, though, it would be harder to reproduce and there either would be a big memory leak or a wrong allocation where e.g. a signed int is misinterpreted. My best guess atm are the GIF routines where space for the image data is allocated, all other allocations are for initialization of fixed size buffers and filenames.

    Quote Originally Posted by m^2 View Post
    I can only suggest to use Valgrind.
    It's awesome for such bugs.
    Yes, I already fixed some minor memory leaks with it. The downside is that it further slows down Precomp so I doubt it would be helpful on a 4 GB test file, but perhaps it can be reduced on a smaller testfile.
    Last edited by schnaader; 7th November 2011 at 17:43.
    http://schnaader.info
    Damn kids. They're all alike.

  6. #6
    Member
    Join Date
    Nov 2011
    Location
    bobrov
    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by schnaader View Post
    Using verbose mode (precomp -v ...) could help to get more information on what leads to that crash and which stream causes it.
    I will post back the results. (takes a lot of time even though i have added the -t-j option.)

    Quote Originally Posted by schnaader View Post
    Looking at the error message, it looks like this is the Linux version, right? If so, do you have a Windows system available and can you try to run the Windows version of Precomp over that file?
    Im using Windows.

  7. #7
    Member
    Join Date
    Nov 2011
    Location
    bobrov
    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Code:
    (94.21%) Possible GIF found at position 10629784
    terminate called after throwing an instance of 'std::bad_alloc'
    what(): std::bad_alloc
    will try with -t-fj

  8. #8
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Quote Originally Posted by lohem View Post
    Code:
    (94.21%) Possible GIF found at position 10629784
    terminate called after throwing an instance of 'std::bad_alloc'
    what(): std::bad_alloc
    will try with -t-fj
    Yes, this is what I suspected. There is some data that looks like a GIF header (which has a high chance of false positives, as the main marker is "GIF87a" or "GIF89a" and can appear e.g. in text files, too), but it seems there is no GIF. Some of the following bytes are interpreted as width/height of the image and this value gets too large, so new() tries to allocate a ridicolous large amount of memory (somewhere between 1 and 4 GB).

    "-t-fj" should work in this case, another possibility would be "-i10629784" to ignore this particular stream.

    It still would be nice if you could try to split the file into smaller parts (50-100 MB) and upload the crashing part somewhere so I could reproduce and fix the bug.
    Last edited by schnaader; 8th November 2011 at 18:59.
    http://schnaader.info
    Damn kids. They're all alike.

  9. #9
    Member
    Join Date
    Nov 2011
    Location
    bobrov
    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I am backing up my personal data, so i am not comfortable uploading part of it online. And, already a lot of days have been spent on this 4gb bundle. Yet, the job is not over.
    apart from the GIF bug, precomp is hanging on zlib-streams. (two times).
    It would be helpful if i could resume the process after adding the ignore position.
    Although, i wont be breaking down the bundle, I can run your fix on the whole thing and report back the results.

    Edit: It again hanged(at 67%) on a zlib-stream which it earlier was able process.
    Last edited by lohem; 10th November 2011 at 08:08.

  10. #10
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Quote Originally Posted by lohem View Post
    I am backing up my personal data, so i am not comfortable uploading part of it online. And, already a lot of days have been spent on this 4gb bundle. Yet, the job is not over.
    apart from the GIF bug, precomp is hanging on zlib-streams. (two times).
    OK, in this case I'll try to create an artificial GIF test case myself. By the way, if you use Precomp for backups, you should make a checksum of the original file (MD5, SHA1...) and compare it with the restored file. There have been some rare edge cases in the past that led to incorrect recompression e.g. for PNG files. At the moment, Precomp doesn't store any checksums (but will do so in later versions). Also make sure that you keep any tools needed to restore the backup, especially the Precomp binaries as it isn't backwards compatible yet.

    As for the problems with zLib streams, I can only guess. Perhaps there isn't enough free diskspace anymore, although Precomp should give a clear error message in this case. Also, did it only hang or did it crash? Some of the streams can take long to process. The activity indicator isn't visible in verbose mode and even if it is active, there are some actions that still block it so it won't move for a while.

    Another thing comes to my mind: You should be using "-c-" to disable the built-in bZip2 compression and compress the PCF file with your favorite tool afterwards (e.g. 7-Zip). This will generate a PCF file larger than the original, but make the precompression faster and lead to a smaller compressed file.
    http://schnaader.info
    Damn kids. They're all alike.

  11. #11
    Member
    Join Date
    Nov 2011
    Location
    bobrov
    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by schnaader View Post
    you should make a checksum of the original file (MD5, SHA1...) and compare it with the restored file.
    yes, i always do that.

    Quote Originally Posted by schnaader View Post
    especially the Precomp binaries as it isn't backwards compatible yet.
    good that you mentioned. I assumed it was backwards compatible.

    Quote Originally Posted by schnaader View Post
    As for the problems with zLib streams, I can only guess. Perhaps there isn't enough free diskspace anymore,
    There is more than enough disk space & RAM & CPU usage too.
    Is there a way to increase or specify precomp's RAM & CPU usage?

    Quote Originally Posted by schnaader View Post
    although Precomp should give a clear error message in this case. Also, did it only hang or did it crash?
    It gave mesg when the gif error occured. Other times(zlib) it just stops working. Then, I get windows mesg that precomp has stopped working.

    Quote Originally Posted by schnaader View Post
    Another thing comes to my mind: You should be using "-c-" to disable the built-in bZip2 compression
    Yes, i do that.

    The weird thing is why precomp stops-working on position that it was able to process on a previous run.

  12. #12
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Quote Originally Posted by lohem View Post
    It gave mesg when the gif error occured. Other times(zlib) it just stops working. Then, I get windows mesg that precomp has stopped working.
    That's strange. There were similar errors (zlib streams crashing although they worked fine in previous runs) with Precomp 0.4 that didn't use ZLIB1.DLL, which is why it was reintroduced and has to be in the same path as the Precomp binary. Since you're using the newest version, I can only imagine that there's some undetected bug with files larger than 2 GB or with rare zLib streams that causes this behaviour.
    http://schnaader.info
    Damn kids. They're all alike.

  13. #13
    Member
    Join Date
    Nov 2011
    Location
    bobrov
    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts
    yes, its strange. i did not have so many problems on one archive while using precomp before.
    Since, i am still not getting the complete output, i would just 7zip-it for the time.
    Hope bringing out this issue helps in making the next version better.

Similar Threads

  1. Error recovery from archive?
    By roytam1 in forum The Off-Topic Lounge
    Replies: 15
    Last Post: 9th November 2011, 11:52
  2. Replies: 1
    Last Post: 12th June 2011, 02:01
  3. Precomp (and Precomp Comfort) in 315 kb
    By Yuri Grille. in forum Data Compression
    Replies: 2
    Last Post: 1st April 2009, 19:40

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •