Results 1 to 3 of 3

Thread: Creating A Compressor for 'JDLZ'

  1. #1
    Member
    Join Date
    Jan 2016
    Location
    Wisconsin, USA
    Posts
    2
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Creating A Compressor for 'JDLZ'

    I have recently delved into the deep dark hole that is data compression, and need some advice.

    Basically I am reverse-engineering an older video game I enjoy, trying to create some decent open-source modding tools for it. One big thing I am trying to accomplish is creating a compressor for one of the main compression formats the game uses, dubbed "JDLZ". For the past week or so I have been obsessed with this, and I feel like I'm getting somewhere with it finally. I am still fairly new to data compression though, so I'm hoping somebody can at least tell me if I'm on the right track with this.

    I was able to figure out how to decompress JDLZ, which was fairly simple. The code for that can be found on my GitHub here (see the decompress function specifically): https://github.com/MWisBest/OpenNFST...ession/JDLZ.cs
    It's in C#, but because of how simple it is you could probably just about run it through a C/C++ compiler or Java as-is lol

    My original plan to create a way to compress data back into this format was to try and translate the compression assembly code (that was for some reason left in the game's executable) to a higher-level language, but I'm not really getting anywhere with that so I'm now looking at the possibility of creating something that's compatible with the format and achieves at least some compression. My hope is that this format is just a derivative of something already out there, and I have a promising lead:
    To me, it seems that it may be derived from LZRW1(-a?). As far as I know, JDLZ first appeared in 2001, and if I was a video game developer at that time looking for something that decompressed quickly with decent ratios, LZRW would've been high on that list.

    I'm thinking it may be possible to make some modifications to LZRW1's compression code, and end up with something compatible with this JDLZ format. Problem is of course I'm new to this, so I'd greatly appreciate if somebody here could at least tell me if that's feasible before I spend too much time with it.

    Thanks!

  2. #2
    Member
    Join Date
    Feb 2013
    Location
    Internet
    Posts
    4
    Thanks
    0
    Thanked 3 Times in 3 Posts
    Hi MWisBest!

    The JDLZ decompressor you posted on GitHub is very simple, so I decided to write compatible compressor from scratch, as a programming exercise
    I hope this helps.


    Code:
    public static byte[] compress(byte[] input)
    {
        if (input == null) return null;
    
        const int HeaderSize = 16;
        const int MinMatchLength = 3;
        const int MaxSearchDepth = 16;
    
        int inputBytes = input.Length;
        byte[] output = new byte[inputBytes + ((inputBytes + 7) / 8 ) + HeaderSize + 1];
        int[] hashPos = new int[0x2000];
        int[] hashChain = new int[inputBytes];
    
        int outPos = 0;
        int inPos = 0;
        byte flags1bit = 1;
        byte flags2bit = 1;
        byte flags1 = 0;
        byte flags2 = 0;
    
        output[outPos++] = 0x4A; // 'J'
        output[outPos++] = 0x44; // 'D'
        output[outPos++] = 0x4C; // 'L'
        output[outPos++] = 0x5A; // 'Z'
        output[outPos++] = 0x02;
        output[outPos++] = 0x10;
        output[outPos++] = 0x00;
        output[outPos++] = 0x00;
        output[outPos++] = (byte)inputBytes;
        output[outPos++] = (byte)(inputBytes >> 8 );
        output[outPos++] = (byte)(inputBytes >> 16);
        output[outPos++] = (byte)(inputBytes >> 24);
        outPos += 4;
    
        int flags1Pos = outPos++;
        int flags2Pos = outPos++;
    
        flags1bit <<= 1;
        output[outPos++] = input[inPos++];
        inputBytes--;
    
        while (inputBytes > 0)
        {
            int bestMachLength = MinMatchLength - 1;
            int bestMatchDist = 0;
    
            if (inputBytes >= MinMatchLength)
            {
                int hash = (-0x1A1 * (input[inPos] ^ ((input[inPos + 1] ^ (input[inPos + 2] << 4)) << 4))) & 0x1FFF;
                int matchPos = hashPos[hash];
                hashPos[hash] = inPos;
                hashChain[inPos] = matchPos;
                int prevMatchPos = inPos;
    
                for (int i = 0; i < MaxSearchDepth; i++)
                {
                    int matchDist = inPos - matchPos;
                    if (matchDist > 2064 || matchPos >= prevMatchPos) break;
    
                    int matchLengthLimit = matchDist <= 16 ? 4098 : 34;
                    int maxMatchLength = inputBytes;
                    if (maxMatchLength > matchLengthLimit) maxMatchLength = matchLengthLimit;
                    if (bestMachLength >= maxMatchLength) break;
    
                    int matchLength = 0;
                    while (matchLength < maxMatchLength && input[inPos + matchLength] == input[matchPos + matchLength])
                        matchLength++;
    
                    if (matchLength > bestMachLength)
                    {
                        bestMachLength = matchLength;
                        bestMatchDist = matchDist;
                    }
    
                    prevMatchPos = matchPos;
                    matchPos = hashChain[matchPos];
                }
            }
    
            if (bestMachLength >= MinMatchLength)
            {
                flags1 |= flags1bit;
                inPos += bestMachLength;
                inputBytes -= bestMachLength;
                bestMachLength -= MinMatchLength;
    
                if (bestMatchDist < 17)
                {
                    flags2 |= flags2bit;
                    output[outPos++] = (byte)((bestMatchDist - 1) | ((bestMachLength >> 4) & 0xf0));
                    output[outPos++] = (byte)bestMachLength;
                }
                else
                {
                    bestMatchDist -= 17;
                    output[outPos++] = (byte)(bestMachLength | ((bestMatchDist >> 3) & 0xe0));
                    output[outPos++] = (byte)bestMatchDist;
                }
    
                flags2bit <<= 1;
            }
            else
            {
                output[outPos++] = input[inPos++];
                inputBytes--;
            }
    
            flags1bit <<= 1;
    
            if (flags1bit == 0)
            {
                output[flags1Pos] = flags1;
                flags1 = 0;
                flags1Pos = outPos++;
                flags1bit = 1;
            }
    
            if (flags2bit == 0)
            {
                output[flags2Pos] = flags2;
                flags2 = 0;
                flags2Pos = outPos++;
                flags2bit = 1;
            }
        }
    
        if (flags2bit > 1)
            output[flags2Pos] = flags2;
        else if (flags2Pos == outPos - 1)
            outPos = flags2Pos;
    
        if (flags1bit > 1)
            output[flags1Pos] = flags1;
        else if (flags1Pos == outPos - 1)
            outPos = flags1Pos;
    
        output[12] = (byte)outPos;
        output[13] = (byte)(outPos >> 8 );
        output[14] = (byte)(outPos >> 16);
        output[15] = (byte)(outPos >> 24);
    
        Array.Resize(ref output, outPos);
        return output;
    }

  3. The Following User Says Thank You to zombie28 For This Useful Post:

    MWisBest (17th January 2016)

  4. #3
    Member
    Join Date
    Jan 2016
    Location
    Wisconsin, USA
    Posts
    2
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by zombie28 View Post
    Hi MWisBest!

    The JDLZ decompressor you posted on GitHub is very simple, so I decided to write compatible compressor from scratch, as a programming exercise
    I hope this helps.
    OH MY GOD. THANK YOU!!
    I never expected this kind of help. You are lifesaver.
    Yesterday I had started to get a better idea of what the game's assembly code was doing, finally recognizing that ugly thing I couldn't figure out was the hashing thing. And you took less than a day to write a compatible compressor?! That's incredible.
    I gave it a quick test, and thought you might be interested to see how well it's working.

    Code:
    gameplay.bin (2,105,216 bytes): 815,800 vs 779,956
    GLOBALB.BUN (2,803,648 bytes): 1,610,028 vs 1,520,744
    InGameB.bun (946,264 bytes): 547,986 vs 522,637
    Just a smidge short of the original, but I could not be happier. There are some closed-source modding tools for these games which can compress JDLZ... except those idiots literally ripped the assembly code from the game, which I'm sure is illegal. With your compressor I can feel confident it's original.

    You are truly a blessing, this is incredible. I cannot thank you enough.

Similar Threads

  1. Kitty file compressor (Super small compressor)
    By snowcat in forum Data Compression
    Replies: 7
    Last Post: 26th April 2015, 16:46
  2. Creating our IRC channel
    By Bulat Ziganshin in forum The Off-Topic Lounge
    Replies: 15
    Last Post: 28th November 2009, 13:16

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •