DARK - a new BWT-based command-line archiver

[kvark]

Sometimes a little 'fallback' can be very useful

[LovePimple]

I'm sorry to hear about the HDD crash!

Did you back-up the source code?

[kvark]

Version 0.38 if i'm not mistaken was the last backuped, I will get this backup only tomorrow, so I don't know exactly. Right now I have only 0.36 But don't worry! New code will be more beautiful, fast and stable. It's time to give 'dark' a new birth

Theoreticaly I have an aweful improvement (up to 0.50), the one and only problem is my time...

[LovePimple]

Version 0.38 if i'm not mistaken was the last backuped, I will get this backup only tomorrow, so I don't know exactly. Right now I have only 0.36 But don't worry! New code will be more beautiful, fast and stable. It's time to give 'dark' a new birth

As this is a going to be a rebirth; are you going to add the dual (FAST and Super-High) compression levels we talked about earlier?

Maybe you could have three compression levels like THOR?


Theoreticaly I have an aweful improvement (up to 0.50), the one and only problem is my time...

I understand!

[kvark]

Currently (If not count that project is freezed for a while ) I'm working on the compression ratio AND compression fast/slow modes

Very possibly for high speed the ST6 (shindler Transform) will be used, but It's only one variant (decompression is slower).

It's Stephan Busch here (Squeeze Chart 2006)

On one of my large testsets (Encyclopedia) your Dark 0.41
fails; it keeps cpu load at 100% and works, but filesize of the
archive does not grow. I closed commandline window after 2 hours
and tried lower mem settings, but it does not finnish compression, neither the tar file nor the normal files in directory..
Dark's compression is amazing and better than your competitor YBS
in my benchmarks; and I would like to also see a wav and bmp filter, and maybe the jpg lossless recompressor of PAQ7 / PAQ8;
and some tricks for text preprocessing would also be nice.
I apreciate your work and if I finnish the last testset, I will
release my new Squeeze Chart with your programs results.


Yours,

Stephan Busch
squeezechart@gmx.de

[kvark]

Thanks, Stephan!

>On one of my large testsets (Encyclopedia) your Dark 0.41
>fails; it keeps cpu load at 100% and works, but filesize of the
>archive does not grow. I closed commandline window after 2
>hours and tried lower mem settings, but it does not finnish
> compression, neither the tar file nor the normal files in directory..

It has some bugs still... Thank you for testing!
As I sad before, 'dark' will reborn, soon. (Of course, without this bug)

> Dark's compression is amazing and better than your competitor
> YBS in my benchmarks;

Dark had never been better than ybs on texts, but can be in near future (without filters). May be it's the perfect fragmentor module's work.

> and I would like to also see a wav and bmp filter, and maybe the > jpg lossless recompressor of PAQ7 / PAQ8. and some tricks for
> text preprocessing would also be nice

Roger that

Thanks again!
Dmitry Malyshev

[LovePimple]

Dark's compression is amazing

Never underestimate the POWER of the DARK side!

[LovePimple]

I would like to also see a wav and bmp filter, and maybe the jpg lossless recompressor of PAQ7 / PAQ8;
and some tricks for text preprocessing would also be nice.
I apreciate your work

I agree!

[kvark]

To tell the truth, I don't satisfied with dark's compression engine at all... If you know what I know, you'd say it must be much better
Fragmentator is a powerful thing, especially in solid mode

BTW, Dark-0.50 will sort files before compressing and pack the path list, It will increase solid compression.
And what is amazing for me - I'm using dark myself! I've compressed some old games with dark-0.41 better then 7z and WinRAR in PPMD mode

[LovePimple]

To tell the truth, I don't satisfied with dark's compression engine at all... If you know what I know, you'd say it must be much better

I understand this but I have faith that you will eventually make it *MUCH* better.


BTW, Dark-0.50 will sort files before compressing and pack the path list, It will increase solid compression.
And what is amazing for me - I'm using dark myself! I've compressed some old games with dark-0.41 better then 7z and WinRAR in PPMD mode

Yes; it could be a very useful archiver if it had both FAST and Super-HIGH compression modes. It would be much better if it had selectable FAST, MEDIUM, and Super-HIGH compression though!

New super-archiver with GUI? Easier said than done... Anyway, lets wait and see!

[LovePimple]

New super-archiver with GUI? Easier said than done... Anyway, lets wait and see!

Don't really need the GUI myself. I'm quite happy with the command line version.

[kvark]

Dark0.45 - revision 9 (from 0.36)

1) Memory requiments - 5N (!!!!) (pack/unpack)
2) Archon lost 'Anchors module' (*)
3) Archon has got 'Tandem Repeats module' (*)
4) Reverse and Infolev flags
5) Solid archive: filesorting (+compression?)
6) Interface changed (c/x, %, ...)
7) OpenWatcom 1.0 ( was 1.4)

But! It hadn't tested carefully yet. Some sort of beta...
http://kvark.at.tut.by/dark-0.45r9.zip

(*) see Archon3 final structure,
http://darchiver.narod.ru/dark/Archon3fs.doc

[LovePimple]

Thanks for the update!

Compression is not as good on some files but its noticably faster. Using the reverse sorting order (-r) has given some interesting results with some files compressing better than with any previous version of Dark.

[kvark]

Faster?!

On my machine it's 1.5 times slower than 0.41b (I thought because of Watcom ver 1.0).
Also thanks for '-r' testing! I hadn't found such files myself

[kvark]

dark-0.45, the 'true' first version of new-born dark
http://kvark.at.tut.by/dark-0.45.zip
Homepage will be updated later.

Changes(since 0.45r9):
- Solved solid decompression problem. No more big temp files, increased speed
- Several fixes. This version seems to be stable (much more than dark-0.41b).
- compiled with OpenWatcomC 1.5

[kvark]

Do you ready for the next dark?
Look at it's homepage and welcome 'dark-0.46'!

book1: 768771 -> 215000 bytes )

[LovePimple]

Thanks for the updates!

I'll test them as soon as possible and let you know my results.

[LovePimple]

I've tested v0.46 pretty thoroughly and so far no problems found.

I'll continue to test it while we wait for Werner's benchmark results to be published.


Faster?!

On my machine it's 1.5 times slower than 0.41b (I thought because of Watcom ver 1.0).

I can't understand why its so much slower on your machine. Even when using the default settings it shows good speed on my tests.

Here are some timings using only the default settings for all versions of Dark.

Test files: 2,201 files: txt, bmp, exe, dll, html etc
Total uncompressed size: 397 MB (416,954,653 bytes)

Dark v0.41b

Compressed size: 169 MB (178,237,336 bytes)

Compression time:
Kernel Time = 6.562 = 00:00:06.562 = 2%
User Time = 271.078 = 00:04:31.078 = 93%
Process Time = 277.640 = 00:04:37.640 = 95%
Global Time = 289.234 = 00:04:49.234 = 100%


Dark v0.45r9

Compressed size: 169 MB (178,237,336 bytes)

Compression time:
Kernel Time = 6.750 = 00:00:06.750 = 2%
User Time = 271.046 = 00:04:31.046 = 93%
Process Time = 277.796 = 00:04:37.796 = 96%
Global Time = 288.843 = 00:04:48.843 = 100%


Dark v0.46

Compressed size: 169 MB (178,237,336 bytes)

Compression time:
Kernel Time = 6.203 = 00:00:06.203 = 2%
User Time = 272.812 = 00:04:32.812 = 91%
Process Time = 279.015 = 00:04:39.015 = 93%
Global Time = 298.672 = 00:04:58.672 = 100%

[LovePimple]

Here are my timings for compressing ENWIK8 with 20m block size.

Dark v0.46
Block: 20m, Frag: No, Solid: No
Compressed size: 21.9 MB (23,007,621 bytes)

Compression time:
Kernel Time = 0.984 = 00:00:00.984 = 1%
User Time = 88.750 = 00:01:28.750 = 93%
Process Time = 89.734 = 00:01:29.734 = 94%
Global Time = 94.828 = 00:01:34.828 = 100%

Good compression considering the 20m block size!


Compressed size with 30m block size (beats WinRAR 3.6).
21.5 MB (22,604,036 bytes)

Compressed size with 128m block size.
20.2 MB (21,231,325 bytes)

[kvark]

Are you sure posted results are correct? They cannot match byte to byte!

Anyway, thanks a lot, LovePimple!!!

You've done a great work! I'm satisfied with dark-0.46 stability
Will wait for Werner and Matt

Upd. I see the improvement is easy to miss here... I was tuning model for little files, book1 especially. May be I must check big files too...
BTW, what about 160Mb block?

[LovePimple]

Are you sure posted results are correct? They cannot match byte to byte!

Yes, posted results are correct. I still have the original compressed files and have double checked the results.

All timings were obtained by using this program.
URL


Upd. I see the improvement is easy to miss here... I was tuning model for little files, book1 especially. May be I must check big files too...
BTW, what about 160Mb block?

I don't have enough RAM to test with 160Mb block size.

[kvark]

It's impossible!
Can you send me at least 1 file were the compression matches byte-to-byte? I have never seen such situation myself (I mean dark-0.45r9 - dark-0.46).

[LovePimple]

Apologies, you are of course correct!

Due to an error in my batch script the incorrect version of Dark was being used during the test. The correct results are posted below.


Test files: 2,201 files: txt, bmp, exe, dll, html etc
Total uncompressed size: 397 MB (416,954,653 bytes)

Dark v0.41b

Compressed size: 170 MB (178,743,358 bytes)

Compression time:
Kernel Time = 6.640 = 00:00:06.640 = 2%
User Time = 266.296 = 00:04:26.296 = 92%
Process Time = 272.937 = 00:04:32.937 = 94%
Global Time = 288.766 = 00:04:48.766 = 100%


Dark v0.45r9

Compressed size: 170 MB (178,972,955 bytes)

Compression time:
Kernel Time = 6.562 = 00:00:06.562 = 2%
User Time = 286.921 = 00:04:46.921 = 92%
Process Time = 293.484 = 00:04:53.484 = 94%
Global Time = 309.500 = 00:05:09.500 = 100%


Dark v0.46

Compressed size: 169 MB (178,237,336 bytes)

Compression time:
Kernel Time = 6.390 = 00:00:06.390 = 2%
User Time = 270.828 = 00:04:30.828 = 92%
Process Time = 277.218 = 00:04:37.218 = 94%
Global Time = 293.500 = 00:04:53.500 = 100%

[LovePimple]

Test Files: Calgary Corpus (14 files)
Total uncompressed size: 2.99 MB (3,141,622 bytes)

Dark v0.41 (C)kvark, 2006
Block: 4m; Frag: auto; Solid: no
Allocated memory: 24mb
Total time: 1547ms

Compressed size: 766 KB (785,226 bytes)


Dark v0.45r9 (C)kvark, 2006
Block: 4m; Frag: Auto; Solid: No;
Memory usage: 20 Mb
Total time: 1594ms

Compressed size: 767 KB (785,658 bytes)


Dark v0.46 (C)kvark, 2006
Block: 4m, Frag: Auto, Solid: No
Memory usage: 20Mb
Total time: 1531ms

Compressed size: 765 KB (783,710 bytes)

[LovePimple]

Test Files: All 10 SFC files from maximumcompression.com
Total uncompressed size: 50.6 MB (53,134,726 bytes)

Dark v0.41 (C)kvark, 2006
Block: 4m; Frag: auto; Solid: no
Allocated memory: 24mb
Total time: 41219ms

Kernel Time = 0.406 = 00:00:00.406 = 0%
User Time = 40.312 = 00:00:40.312 = 97%
Process Time = 40.718 = 00:00:40.718 = 98%
Global Time = 41.250 = 00:00:41.250 = 100%

Compressed size: 12.1 MB (12,756,916 bytes)


Dark v0.45r9 (C)kvark, 2006
Block: 4m; Frag: Auto; Solid: No;
Memory usage: 20 Mb
Total time: 50219ms

Kernel Time = 0.437 = 00:00:00.437 = 0%
User Time = 49.171 = 00:00:49.171 = 97%
Process Time = 49.609 = 00:00:49.609 = 98%
Global Time = 50.218 = 00:00:50.218 = 100%

Compressed size: 12.1 MB (12,771,462 bytes)


Dark v0.46 (C)kvark, 2006
Block: 4m, Frag: Auto, Solid: No
Memory usage: 20Mb
Total time: 43922ms

Kernel Time = 0.359 = 00:00:00.359 = 0%
User Time = 43.078 = 00:00:43.078 = 98%
Process Time = 43.437 = 00:00:43.437 = 98%
Global Time = 43.938 = 00:00:43.938 = 100%

Compressed size: 12.1 MB (12,736,798 bytes)

[LovePimple]

Test files: 9,234 Files: txt, bmp, exe, dll, html etc, 459 Folders
Total uncompressed size: 1.62 GB (1,740,468,114 bytes)

Dark v0.46 (C)kvark, 2006
Block: 4m, Frag: Auto, Solid: No
Memory usage: 20Mb
Total time: 1417734ms

Kernel Time = 29.500 = 00:00:29.500 = 2%
User Time = 1319.453 = 00:21:59.453 = 93%
Process Time = 1348.953 = 00:22:28.953 = 95%
Global Time = 1417.813 = 00:23:37.813 = 100%

Compressed size: 739 MB (775,501,285 bytes)

[kvark]

Thanx a lot, again!
Now I see clearly 0.3% improvement and it's very good!

Werner updated his results. Now dark-0.46 is 48th... poor
I hope the enwik tests will show better results, especially with 160Mb block

[LovePimple]

Also thanks for '-r' testing! I hadn't found such files myself

It seems to work best on certain multimedia files. I've had good results using -r on .wav, .bmp, and even some .gif files.

One example is Werner's rafale.bmp which can be improved from 787197 to 785096 bytes by using the "r" (-b3600kr) switch.