Page 1 of 2 12 LastLast
Results 1 to 30 of 43

Thread: FreeArc 'Next license

  1. #1
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts

    FreeArc 'Next license

    (first post reserved for topic heading)

  2. #2
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Sometime ago i've started project of rewriting FreeArc from scratch. Now i'm approaching the point of first public alpha, so i will briefly describe its current state. Improvements over existing FreeArc:

    Main drawbacks:
    • incompatible archive format (it will be fixed only in the next version)
    • archive updating and lots of other features aren't yet implemented (help screen shows the features already implemented)

    The current agenda:
    • april: will be released soon
    • may: FreeArc archive fromat, automatic filetype detection based on contents
    • june: simple GUI (a-la HtmlArc)
    • july: not determied yet, but probably archive updating/generations

    Now you can stare at program docs and fix my dirty English Editing of the Wiki should be available for anyone with GitHub account, but if it doesn't work, i can add you to the list of project contributors.

  3. The Following 8 Users Say Thank You to Bulat Ziganshin For This Useful Post:

    avitar (11th April 2015),Euph0ria (22nd August 2015),load (11th April 2015),Mike (11th April 2015),Nania Francesco (11th April 2015),Stephan Busch (12th April 2015),Surfer (11th April 2015),surfersat (13th April 2015)

  4. #3
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Why rewrite?
    Still Haskell + C++?
    Any chance for a library?

  5. #4
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    342
    Thanks
    12
    Thanked 34 Times in 28 Posts
    i suggest a couple of features
    first: alphanumeric notes when adding. in future extract of version via name, not sequential number
    second: append only format. this make very easy and fast rsync
    third: stdio support

  6. #5
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    342
    Thanks
    12
    Thanked 34 Times in 28 Posts
    please do not use advanced hash but slower if not necessary.
    sha1 is much better than sha256 for example because twice as fast. md5 even batter
    and make a fast extraction of single file.
    finally a very fast list function
    in other words... zpaq ++
    Last edited by fcorbelli; 11th April 2015 at 14:53.

  7. #6
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    645
    Thanks
    205
    Thanked 196 Times in 119 Posts
    And, as you are changing the format, maybe replace the outdated Huffman to catch up with lzturbo: http://encode.ru/threads/2017-LzTurb...ll=1#post39815

  8. #7
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Code:
    C:\FB>timer fa.exe create m:\a c:\ d:\ e:\ z:\ --no-data --no-warnings -ds
    Scanning: 7,532,018,850,656 bytes in 328,525 folders and 2,700,795 files  (RAM 204 MiB, cpu 2.964 sec, real 20.065 sec)
    Archive directory: 153,917,518 bytes
    
    Kernel Time  =    16.692 = 00:00:16.692 =  82%
    User Time    =     3.120 = 00:00:03.120 =  15%
    Process Time =    19.812 = 00:00:19.812 =  97%
    Global Time  =    20.281 = 00:00:20.281 = 100%
    
    
    
    
    C:\FB>timer fa.exe l m:\a |tail
        172,586,496  2011-02-22 03:39:35  .A....  z:\vs2010\sp1\VS10sp1-KB983509.msp
    ===============================================================================
    7,531,763,218,119 => 0 bytes in 328,546 folders and 2,700,803 files
    
    Kernel Time  =     0.421 = 00:00:00.421 =   7%
    User Time    =     4.929 = 00:00:04.929 =  91%
    Process Time =     5.350 = 00:00:05.350 =  99%
    Global Time  =     5.383 = 00:00:05.383 = 100%
    
    
    
    C:\FB>timer fa.exe l m:\a -nNO-SUCH-FILE |tail
    ===============================================================================
    0 => 0 bytes in 328,546 folders and 0 files
    
    Kernel Time  =     0.312 = 00:00:00.312 =  13%
    User Time    =     1.887 = 00:00:01.887 =  84%
    Process Time =     2.199 = 00:00:02.199 =  97%
    Global Time  =     2.247 = 00:00:02.247 = 100%
    is it fast enough? but current version can extract only entire archive
    Last edited by Bulat Ziganshin; 11th April 2015 at 15:51.

  9. #8
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    Quote Originally Posted by Jarek View Post
    And, as you are changing the format, maybe replace the outdated Huffman to catch up with lzturbo: http://encode.ru/threads/2017-LzTurb...ll=1#post39815
    Thats epic.
    Jarek, nothing personal.

  10. #9
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    645
    Thanks
    205
    Thanked 196 Times in 119 Posts
    Skymmer, I have got used to the fact that data compression specialists are literally Huffman worshipers (see e.g. http://pages.cs.brandeis.edu/~dcc/Pr...rogram2015.pdf ) - because they have written dozens of papers/compressors on it ... but now it just means wasting both space and time ( http://encode.ru/threads/1920-In-mem...entropy-coders ).
    I see there is an issue understanding ANS - I could help, there are also descriptions of a few other persons, or one can just use e.g. FSE ... changing the format is the perfect moment for the upgrade.

  11. The Following User Says Thank You to Jarek For This Useful Post:

    avitar (11th April 2015)

  12. #10
    Member
    Join Date
    Jun 2008
    Location
    Germany
    Posts
    369
    Thanks
    5
    Thanked 6 Times in 4 Posts
    ..
    Last edited by JangoFatXL; 11th April 2015 at 21:45.

  13. #11
    Member
    Join Date
    Jun 2008
    Location
    Germany
    Posts
    369
    Thanks
    5
    Thanked 6 Times in 4 Posts
    Quote Originally Posted by Jarek View Post
    Skymmer, I have got used to the fact that data compression specialists are literally Huffman worshipers ...
    No. It is because you dropped a brick by suggesting Bulat using that program...

  14. #12
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Jarek, forum frequenters like me know about ANS very well. but freearc is an archiver, not a compressor program, and it relies on lzma, ppmd and other existing algorithms. i don't plan to change the archive format and don't have time to work on improving these compression algorithms

  15. #13
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    645
    Thanks
    205
    Thanked 196 Times in 119 Posts
    I apology, I was thinking about Tornado.

  16. #14
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Quote Originally Posted by m^2 View Post
    Why rewrite?
    Still Haskell + C++?
    Any chance for a library?
    technically, it's already a library. at least i taked into account all problems i had with old freearc code: all errors are handled with exceptions, no global vars, all user interaction is performed in the single, replaceable module. but looking at the lzturbo story, i've changed my mind about open-sourceness. the same story holds for dll - it will allow anyone from china incorporate freearс compression into his own program, name himself gmbh and start selling his program. if you have idea how to distribute dll without risks of being selled without my control - i'm all ears. if you just need a .lib library for your closed-source project - drop me a mail

    i rewrote program because it's 10 year old and there are lot of places that need rewrite. i believe that programs should be rewritten from scratch every few years in order to keep them tight, so it even was too late. It was started as FB, with haskell and C++ code, but finally i switched to C++14 and Boost. i believe it's more portable than haskell

  17. #15
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Quote Originally Posted by fcorbelli View Post
    please do not use advanced hash but slower if not necessary.
    sha1 is much better than sha256 for example because twice as fast. md5 even batter
    and make a fast extraction of single file.
    finally a very fast list function
    in other words... zpaq ++
    yes, my initial idea with FB was a zpaq++, but i changed it to implement freearc++ too hashing in FA is already pretty fast, 500 MB/s on i7-4770, and it can be further improved using better SHA implementation. SHA1 and MD5 aren't recommnded by NIST for new programs as current cyptoanalysis can break MD5 and close to break SHA1, but for those who don't believe in cryptography i can provide alternative hashing methods with speed of 20 GB/s

    about fast adding/listing/extraction - i especially care about scenaries with millions of files and gigabytes of data. in particular, i plan to provide alternative archive format that allows faster handling of archives with millions of files. but ATM i yet implement more basic things

  18. #16
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Quote Originally Posted by fcorbelli View Post
    i suggest a couple of features
    first: alphanumeric notes when adding. in future extract of version via name, not sequential number
    second: append only format. this make very easy and fast rsync
    third: stdio support
    1. can you explain deeper?
    2. yes, of course. i plan to implement zpaq-style backups since i like it very much
    3. yes, i should implement it sometime. although extraction from stdin will be impossible because archive directory is at the end of archive

  19. #17
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 166 Times in 63 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    i've changed my mind about open-sourceness.
    I assume that you are interested in getting some profits from your code. Several times I have been asked for recommendation of the best compressor for a specific type of data and given bandwidth requirements.
    Once the best compressors from my experiments were NanoZip, FreeArc, CCM, and CMM. The company resigned of closed-source programs, because they have bigger possibility of messy and buggy code and FreeArc won.

    Quote Originally Posted by Bulat Ziganshin View Post
    it will allow anyone from china incorporate freearс compression into his own program, name himself gmbh and start selling his program
    IMHO it's not a problem. If your code is not available they will take 7-zip or another compressor. Do you really lose anything? Maybe one day they will have to legalize their source codes, and will buy license from you.
    Last edited by inikep; 12th April 2015 at 12:01.

  20. #18
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    342
    Thanks
    12
    Thanked 34 Times in 28 Posts
    i know md5 and sha1 pretty well, and nist does not handle this case
    you do not need to take the fingertips of a ssl certificate, neither exchange DH.
    you need something with a very or virtual null collision rate
    do not use an hammer if you neeed a needle.
    turning on notes i suggest to add alphanumeric text to the version.
    notes is almoa useless for tradizional archives.
    but here you can use to extract the versions.


    suppose you have a codebase of freearc and you want to take all the versions in one single file.
    you can do pretty easily with zpaq

    but how to extract v27.2 from the archive?

    you cannot, because you need to know that say version 3 of the zpaq contain your source code 27.3

    as in a database i want a table to associate internal version numbers 1'2'3'4... to ASCII text initial'ok'deployed'newhash
    so i can restore say extract - version=deploy

    about working with very big file number, in future, i suggest to split data block file from metadata index
    so you can have your little database with say a binary tree or whatever

    and yes, i' m a senior or better mysql guru, so i like to think in db ways.

    about stdio: does not matter extrac, but ccompress.
    because is much common to dump a database to ascii and restore direct
    Last edited by fcorbelli; 12th April 2015 at 12:08.

  21. #19
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    342
    Thanks
    12
    Thanked 34 Times in 28 Posts
    sorry for my terrible language but sometimes i use a smartphone with italian keyboard that do not like very much english

  22. #20
    Member
    Join Date
    May 2009
    Location
    France
    Posts
    95
    Thanks
    13
    Thanked 72 Times in 42 Posts
    Hello,

    <off topic>Argh! I... Must... Resist...
    I can't.
    As I am a real bastard, dirty grammar nazi in my native language, could you please use next time "à la" instead of a-la? Thanks a lot.
    </off topic>
    Anyway, this is a nice job you try to accomplish here! Congratulations!

    AiZ

  23. #21
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Added Handling the compression method chapter to the docs. Now the docs approached 20kb for this single feature. mein got!

  24. #22
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    tbut looking at the lzturbo story, i've changed my mind about open-sourceness. the same story holds for dll - it will allow anyone from china incorporate freearс compression into his own program, name himself gmbh and start selling his program. if you have idea how to distribute dll without risks of being selled without my control - i'm all ears. if you just need a .lib library for your closed-source project - drop me a mail
    Could you tell use what you have on your mind when it comes to open-surceness? "it will allow" suggests that you want to make it more free. "if you have idea" - like you don't want it open at all.

  25. #23
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Quote Originally Posted by m^2 View Post
    Could you tell use what you have on your mind when it comes to open-surceness? "it will allow" suggests that you want to make it more free. "if you have idea" - like you don't want it open at all.
    it may be my bad english. i don't plan to make it open-sourced nor produce dll because i fear that source/dll will be stealed and distributed outside of my control

    Quote Originally Posted by inikep View Post
    I assume that you are interested in getting some profits from your code... The company resigned of closed-source programs, because they have bigger possibility of messy and buggy code and FreeArc won.
    i don't get any profit from that

    Quote Originally Posted by inikep View Post
    IMHO it's not a problem. If your code is not available they will take 7-zip or another compressor. Do you really lose anything? Maybe one day they will have to legalize their source codes, and will buy license from you.
    inikep, if you believe yourself, why don't you spread all your software as open-source and free?

  26. #24
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    342
    Thanks
    12
    Thanked 34 Times in 28 Posts
    open Source is a nonsense in a non USA/UE world.
    so when someone post the source code almost nothing can prevent a commercial closed source rebrand.
    this can be bad, but it's true.
    so open is about launch and forget, like a sidewinder
    someone cares, someone do not.

  27. #25
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    342
    Thanks
    12
    Thanked 34 Times in 28 Posts
    ps back in old days late 80 there is a gentleman agreement to report the original creator.
    today... not so common

  28. #26
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Quote Originally Posted by fcorbelli View Post
    you need something with a very or virtual null collision rate
    http://encode.ru/threads/1747-Extrem...ll=1#post34220

    > turning on notes i suggest to add alphanumeric text to the version.

    ok, i got it

    > about working with very big file number, in future, i suggest to split data block file from metadata index
    > so you can have your little database with say a binary tree or whatever

    yes, we have already discussed it. it may be even my own idea

  29. #27
    Member
    Join Date
    Mar 2009
    Location
    Prague, CZ
    Posts
    60
    Thanks
    27
    Thanked 6 Times in 6 Posts
    Well, open-source also means at least theoretical possibility, that if one day the original author disappears or abandon the project for various reasons, there is still a chance for users that someone will take over, not speaking about bugs etc... Of course the chance is bigger with smaller and simpler programs, but still... From a user point of view closed-sourceness is not good news.
    But nice to see u work on this nice/usefull program, and the plans. Good luck

  30. #28
    Member
    Join Date
    Jun 2008
    Location
    Germany
    Posts
    369
    Thanks
    5
    Thanked 6 Times in 4 Posts
    ps back in old days late 80 there is a gentleman agreement to report the original creator.
    today... not so common
    It's a shame to steal the work of others and claim it as own, I think it is a problem of mentality. Therefore I absolutely respect the decision to publish it as closed source.

    Furthermore there is still the possibilty to change the licence 15 years later and publish the sources if you like, F.e. like they did it with some games (Doom and Duke3D I think)
    Last edited by JangoFatXL; 12th April 2015 at 16:08.

  31. #29
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    it may be my bad english. i don't plan to make it open-sourced nor produce dll because i fear that source/dll will be stealed and distributed outside of my control
    Sad news.

  32. #30
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 166 Times in 63 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    inikep, if you believe yourself, why don't you spread all your software as open-source and free?
    All my significant programs are open-source and can be downloaded from http://pskibinski.pl/. Programs that I wrote later are fully owned by my employer with a transfer of copyrights.

    There is an additional advantage of open-source programs, people can help you find bugs and introduce some improvements. For example I was working with you on 4x4/tornado issues in 2009. I didn't know that the company didn't buy a license from you. I think they chose another compressor.


    Quote Originally Posted by Bulat Ziganshin View Post
    looking at the lzturbo story
    Maybe he stole your code and he build lzturbo, but I'm sure that currently lzturbo doesn't use your code almost at all. For example lzturbo -29 (bytewise encoding) gives much better compression than tornado. Actually I have working decompressor for lzturbo -29. This strength of compression can be achieved only with parser similar to lzma (which is better than tornado's).
    Last edited by inikep; 12th April 2015 at 20:44.

Page 1 of 2 12 LastLast

Similar Threads

  1. FreeArc
    By Bulat Ziganshin in forum Data Compression
    Replies: 3159
    Last Post: 13th October 2018, 03:23
  2. How freearc could be even better
    By calthax in forum Data Compression
    Replies: 4
    Last Post: 17th August 2014, 18:40
  3. Replies: 46
    Last Post: 1st October 2013, 19:14
  4. Rereleasing packJPG under a different license
    By packDEV in forum The Off-Topic Lounge
    Replies: 8
    Last Post: 28th February 2012, 21:01
  5. On OSS license
    By Cyan in forum Data Compression
    Replies: 12
    Last Post: 27th April 2011, 12:27

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •