New archive format

[Matt Mahoney]

I have been working on a new archive format that supports files over 2 GB without any nonportable code. As a demo, I created an archiver that stores both uncompressed files and files compressed with lpaq1 7 (needs 387 MB memory)
http://cs.fit.edu/~mmahoney/compression/#lpq1

To make the code portable without any Windows or Linux specific code, all files have to be accessed sequentially without using fseek(). Files are divided into blocks (about 64KB to 1MB) with the format:

"lPq" 1 [filename [0 mode oldsize newsize data]...]...

The first 4 bytes identify the archive and version number. The compressed files are concatenated together with a filename and a sequence of blocks. Each block has a mode ('c' for compressed or 's' for stored), the uncompressed size (4 bytes), compressed size (4 bytes) and the compressed data.

To list the contents, the program has to read the whole file, skipping over the data, but there is no easy other way to do this portably with large files. It is fast enough, though. The compressor doesn't know the file size until it is done, and it has to buffer the compressed data so it can write the block headers.

The archiver uses commands similar to 7zip and rar (just a, x, and l). To keep it simple, it only supports solid archives that can't be updated. (The compressor is not initialized between files, so compression is better). Also, it will never clobber files, just skip over existing files and keeps going. It lets you rename files when you extract them, which is an annoying problem with some archivers. It doesn't create directories (which would not be portable).

I will probably use this new format in my upcoming paq9.

[LovePimple]

Thanks Matt! I'm looking forward to updates of this, and the upcoming paq9.

[Simon Berger]

Yeah nice, lpaq with an archive format and also an extended one. It rocks, thank you

[Vacon]

Hello everyone,

quite an interesting attempt!
I'm wondering, if this format will make it possible to decrease the border between the different OS and make it easier to exchange data.
I had some really ugly discussions about something like that on OpenOffice.org's dev-list (german) regarding content and format of the files to be handed out to users (cab vs. 7z, about one year ago), so I'm really curious! Good luck!

Best regards!

[Bulat Ziganshin]

Im wondering, if this format will make it possible to decrease the border between the different OS and make it easier to exchange data.

this format will be used only in his own tools (and may be other with the same ideology) and just allows to omit all the complexities of OS-specific programming in the program which main goal is compression, not full-scale archives support

this format is inappropriate for general-purpose archivers like 7z/fa

[Vacon]

Hello everyone,

this format is inappropriate for general-purpose archivers like 7z/fa

This makes sense to me -> as I understand Matt, his main goal isnt strenght in compression (now), but *maybe* it can be enhanced someday to better serve some needs (for instance the lack of ability to create directories).
Concerning compression and comfort its *far* away from FreeArc and 7-zip, but TAR is usefull allthough its no compressor at all!
Nevertheless -> its an interesting idea.

Best regards!

[Simon Berger]

No, I think you understand it wrong. What the goal of this is to support all kind of files. It has nothing (not much) todo with compression. But for a real archiver it takes too long to step through the whole file without knowing sizes for example. No reasons on compression that will be very good here

[Vacon]

Hello everyone,

No, I think you understand it wrong.

Sh...t happens, so maybe you are right.
Matt?! Some more comments?

Best regards!

[Matt Mahoney]

Well, it really is an experimental format, not intended to replace archivers like zip or 7zip or rar. The goal is not lots of features or support for existing formats. The goal is a simple design where many files of different types can be compressed together to improve compression, but still be as useful as a file compressor that lets you name the output file.

Suppose I want to compress a folder with many different file types like text, .exe, .wav, .bmp, etc. The problem with single file compressors is I either have to compress the files one at a time, so I can't use redundancy between text files, or tar them together first, so I can't use special filters for the different file types. The lpq1 format will solve this problem by creating solid archives but allowing different file types to be compressed using different algorithms. Right now it supports -c (compress) and -s (store) but I plan to add other types in paq9. The default will be to let the program decide by using file name extensions or guessing by looking at the beginning of the file, but you can override that with options on a file by file basis.

Also, another reason to keep features to a minimum is if I decide to make self extracting archives, the decompressor needs to be small. So it doesn't make directories, search recursively, strip paths, ask if you want to overwrite files, update the archive, and all this other stuff you already know how to work around without learning a bunch of new commands. Wildcard handling is built into g++, so I don't plan to build that into the program to work with other compilers like VC++, Intel, Borland, etc. I would rather spend my effort improving speed and compression. Someone else can build a GUI and add all these features outside the engine if they want

[Vacon]

Hello everyone,

thank you Matt for making that clearer to us!

I would rather spend my effort improving speed and compression. Someone else can build a GUI and add all these features outside the engine if they want

Divide et impere.

Best regards!