Results 1 to 8 of 8

Thread: nosso hack

  1. #1
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts

    nosso hack

    As some may remember, there was this thing - http://encode.ru/threads/140-Nosso-C...ion-technology

    And recently I stumbled on it again, and really looked into it this time.

    https://sites.google.com/site/shelwi...edirects=0&d=1

    This is a hacked version with removed encryption and archive size restrictions.

    And results are as follows:

    1. Nosso actually uses lzma for compression, although encrypts the code to avoid detection.
    And its even plain lzma with default parameters - they didn't even bother to optimize these.

    2. There's still a good thing in it - there's an .exe preprocessor with likely the best known results.
    Code:
    806,912 oodle213.dll  // sample 32-bit executable 
    267,433 oodle213.7z   // 7z ultra (bcj2)
    246,457 00000000.lzma // nosso payload (see test.bat)
    
    199,104 oodle213.dll.paq8px // paq8px69 -7 
    196,922 00000000.unp.paq8px // paq8px69 -7
    However it only supports _valid_ 32-bit PE/COFF files (ie no dumped/unpacked exes).
    And its not just an easy to hack off check - there's a PE header parser and everything.

    Thanks to kampaster for providing the original executable.

  2. The Following 14 Users Say Thank You to Shelwien For This Useful Post:

    Bulat Ziganshin (2nd June 2016),Christoph Diegelmann (2nd June 2016),comp1 (3rd June 2016),Cyan (2nd June 2016),encode (3rd June 2016),Intrinsic (2nd June 2016),lorents17 (2nd June 2016),Mike (2nd June 2016),mpais (3rd June 2016),ne0n (3rd June 2016),RamiroCruzo (4th June 2016),schnaader (3rd June 2016),Turtle (3rd June 2016),xinix (2nd June 2016)

  3. #2
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Ok, here's more

    https://sites.google.com/site/shelwi...edirects=0&d=1

    This is a durilca 0.5 patched with equivalent of old -l option, to be specific
    dur_flt.exe creates durilca.dmp file with all the preprocessed data, and
    dur_seg.exe creates a separate file for each preprocessor segment.
    Results:

    Code:
    1,005,425 durilca-t3.dmp        // dur_flt.exe e -o2 -t3 oodle213.dll
    1,002,941 durilca-t1.dmp        // dur_flt.exe e -o2 -t1 oodle213.dll
      807,678 dispack.bin           // dispack.exe e oodle213.dll dispack.bin
      198,025 dispack.bin.paq8px    // paq8px_v69_sse2.exe -7 dispack.bin
      193,266 durilca-t3.dmp.paq8px // paq8px_v69_sse2.exe -7 durilca-t3.dmp
      192,001 durilca-t1.dmp.paq8px // paq8px_v69_sse2.exe -7 durilca-t1.dmp
    So I guess nosso filter isn't really the best in the end.

  4. The Following 4 Users Say Thank You to Shelwien For This Useful Post:

    comp1 (3rd June 2016),Mike (3rd June 2016),mpais (3rd June 2016),xinix (3rd June 2016)

  5. #3
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    520
    Thanks
    196
    Thanked 744 Times in 301 Posts
    Thank you, some really nice results. I decided to run some tests with EMMA to see if it could benefit from such a preprocessor.

    Code:
    File: oodle213.dll, 806.912 bytes
    
    198.037 bytes, paq8px_v75 -8
    197.672 bytes, paq8pxd_v16 -s15
    186.772 bytes, EMMA 0.1.10 x86, Preset "Best"
    204.356 bytes, EMMA 0.1.10 x86, Preset "Best", x86/x64 model disabled
    
    File: durilca-t1.dmp, 1.002.941 bytes
    
    191.551 bytes, paq8px_v75 -8
    191.622 bytes, paq8pxd_v16 -s15
    193.253 bytes, EMMA 0.1.10 x86, Preset "Best"
    195.023 bytes, EMMA 0.1.10 x86, Preset "Best", x86/x64 model disabled
    
    File: 00000000.unp, 800.811 bytes
    
    196.584 bytes, paq8px_v75 -8
    197.287 bytes, paq8pxd_v16 -s15
    199.944 bytes, EMMA 0.1.10 x86, Preset "Best"
    200.451 bytes, EMMA 0.1.10 x86, Preset "Best", x86/x64 model disabled
    So, when going for maximum compression, both preprocessors actually hurt compression, since EMMA already has a model for x86/x64 code.
    But disable that model and Durilca's preprocessor gets a really good improvement. So it may be interesting to have some sort of fast
    preprocessor, currently EMMA's x86/x64 transform only does relative-to-absolute address conversions. I need to do more tests.

  6. #4
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts

  7. The Following User Says Thank You to Shelwien For This Useful Post:

    mpais (3rd June 2016)

  8. #5
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    dispack is also incorporated into freearc/fazip:
    M:\>fazip64.exe dispack070 z:\100m nul
    100%: 100,000,000 -> 98,276,089: 98.28%
    Cpu 142 mb/s (0.671 sec), real 135 mb/s (0.708 sec) = 95%

  9. #6
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    So https://github.com/ChromiumWebApps/c...ster/courgette

    Courgette is an exe diff utility used for chrome browser updates.
    In particular, it includes a disasm preprocessor.

    http://nishi.dreamhosters.com/u/courgette2016.rar (exe only)
    http://nishi.dreamhosters.com/u/courgette2016_v0.rar (exe+sources)
    Code:
    Must have exactly one of:
      -supported -asm, -dis, -disadj, -gen or -apply, -genbsdiff or -applybsdiff.
    Usage:
      courgette -supported <executable_file>
      courgette -dis <executable_file> <binary_assembly_file>
      courgette -asm <binary_assembly_file> <executable_file>
      courgette -disadj <executable_file> <reference> <binary_assembly_file>
      courgette -gen <v1> <v2> <patch>
      courgette -apply <v1> <patch> <v2>
    It supports pe/elf x86/x64/arm which is kinda cool.
    But results are a bit uncertain...

    Code:
    Effects of disasm filter on file compression:
    
    1,606,656 7z_dll 1,642,623 7z_dll_dis // courgette.exe -dis 7z_dll 7z_dll_dis (x64)
      537,691 1.plz    508,689 2.plz      // plzma c
      549,773 1.7z     521,212 2.7z       // 7z a -mx=9 -m0=lzma     
      497,590 1a.7z    521,212 2a.7z      // 7z a -mx=9 -myx=9 -m0=lzma     
      360,854 1.paq8p  384,534 1.paq8p    // paq8p -7
    As to actual diff performance, I guess it improved (see also http://encode.ru/threads/582-Executa...ration-methods )
    Code:
    Performance comparison of actual diff methods (bsdiff/bsdiff_sh/courgette-with-disasm)
    
    2,031,104 7z1.dll
    2,046,976 7z2.dll
    2,079,913 7z1_dis       // courgette.exe -dis 7z1.dll 7z1_dis
    2,096,071 7z2_dis       // courgette.exe -dis 7z2.dll 7z2_dis
      365,079 12_dll_bs     // courgette.exe -genbsdiff 7z1.dll 7z2.dll 12_dll_bs
      353,132 12_dll_bsh    // bsdiff_sh2 7z1_dis 7z2_dis 12_dis_bsh  
      231,263 12_dis_bs     // courgette.exe -genbsdiff 7z1_dis 7z2_dis 12_dis
      231,239 12_dis_bsh    // bsdiff_sh2 7z1.dll 7z2.dll 12_dll_bsh  
      116,738 12_dll        // courgette.exe -gen 7z1_dis 7z2_dis 12 (implies -dis?)
      159,811 12_dll_bs.7z  // 7z a -mx=9 -myx=9 12_dll_bs 12_dll_bs
      150,584 12_dll_bsh.7z // 7z a -mx=9 -myx=9 12_dis_bsh 12_dis_bsh
      121,732 12_dis_bsh.7z // 7z a -mx=9 -myx=9 12_dll_bsh 12_dll_bsh
      120,376 12_dis_bs.7z  // 7z a -mx=9 -myx=9 12_dis_bs 12_dis_bs
       75,730 12_dll.7z     // 7z a -mx=9 -myx=9 12_dll 12_dll

  10. The Following 3 Users Say Thank You to Shelwien For This Useful Post:

    Bulat Ziganshin (7th June 2016),comp1 (7th June 2016),schnaader (7th June 2016)

  11. #7
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Ah, thanks for posting, always wanted to try this out, but never found time for it.

    Might be useful for ooffice from silesia. Similar to your results, it only works for compressors without x86 transform. Also tried with the .so files from mozilla, but these are some unsupported outdated elf format.

    Code:
     6,152,192    ooffice
     6,429,813    courgette -dis
    
     2,860,954    Precomp v0.4.5 (similar to plain bzip2)
     2,509,785    courgette -dis | Precomp 0.4.5
    
     2,427,223    7-Zip Ultra LZMA2
     2,135,358    courgette -dis | 7-Zip Ultra LZMA2
    
     1,766,557    zpaq v7.05 -method 7
     1,856,912    courgette -dis | zpaq v7.05 -method 7
    
     1,440,341    paq8l -8
     1,576,733    courgette -dis | paq8l -8
    A detailed description of the process can be found here. I wondered if the relative->absolute transform could be combined with courgette, but it seems that it already does something similar:

    [...]
    • The Disassembler builds a list of addresses referenced by the machine code, numbering each one.
    • The Disassembler replaces and address used in machine instructions with its index number.
    [...]
    http://schnaader.info
    Damn kids. They're all alike.

  12. #8
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > replaces any address used in machine instructions with its index number

    Its a diff thing actually, and isn't very useful for exe preprocessing.
    As bcj2/flt32 shows, it would be better to move addresses to a different stream instead,
    then same code patterns with different addrs would match.

    But I think this is the first x64 disasm filter ever, so its still great :)

Similar Threads

  1. Hack Gzip to counter BREACH/CRIME ?
    By est in forum Data Compression
    Replies: 3
    Last Post: 9th August 2013, 06:04
  2. Nosso Compression technology..
    By maadjordan in forum Data Compression
    Replies: 50
    Last Post: 24th August 2009, 16:10
  3. Gzip 1.2.4 hack (OpenWatcom compiles)
    By Rugxulo in forum Data Compression
    Replies: 9
    Last Post: 22nd May 2009, 00:17
  4. Chasing NOSSO
    By nanoflooder in forum Data Compression
    Replies: 8
    Last Post: 12th April 2009, 10:56
  5. gzip-1.2.4-hack - a hacked version of gzip
    By encode in forum Forum Archive
    Replies: 63
    Last Post: 10th September 2007, 04:16

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •