4x4 - multithreaded compressor

[Bulat Ziganshin]

4x4 is a long-awaited multi-threading compression shell around tornado, lzma and grzip compression libraries. enjoy!

http://www.haskell.org/bz/4x4ver01.zip

the most exciting fact about 4x4 is that it's just 200 lines long and was developed just in 1 day. Haskell is amazing for making multi-threaded apps!

[encode]

Funny name, alternatives are 2+2=5, 2x2, 1+1=3... As far as I understand 4x4 is from car world (Four-wheel drive) - like 4x4 Hummer...

[Bulat Ziganshin]

4x4 is AMD initiative for multicore multiprocessor systems for this app this may mean "4 cores busy compressing and decompressing data using 4 algorithms" but actually there are only 3 algorithms and decompression is single-threaded. there is a room for future versions

added: i forsget about most important cause - it allows program to be placed before 7-zip in alphabetic listings

[LovePimple]

Thanks Bulat!

[Bulat Ziganshin]

my quick scanning of test1 results shows that now in PF 4x4 replaced lzturbo occurences. it's not because it's any better, but probably because its settings turns out to be better for this particular test:

<div class="jscript"><pre>
quicklz 1.4.0 quick2 619,945,569 21 00:11.980 5,836,800 29 00:08.096 5,832,704
thor 0.96 thor e2 587,980,752 22 00:12.136 3,305,472 30 00:11.466 3,293,184
tornado 0.3 tor -4 447,968,986 23 00:17.472 23,175,168 31 00:07.597 11,689,984
4x4 0.1 4x4 6 375,149,040 24 00:52.104 304,578,560 28 00:09.968 38,682,624
4x4 0.1 4x4 7 349,846,233 31 01:40.823 706,539,520 27 00:09.328 72,265,728
4x4 0.1 4x4 8 340,677,122 41 02:10.791 1,414,393,856 37 00:22.869 139,313,152
tornado 0.3 tor -7 334,050,397 66 01:02.322 111,325,184 31 00:08.970 69,476,352
tornado 0.3 tor -8 313,161,387 94 01:31.166 212,226,048 32 00:08.346 136,712,192
tornado 0.4a tor -9 295,759,477 104 01:40.527 414,216,192 39 00:08.002 271,220,736
tornado 0.4a tor -10 279,273,808 152 02:29.838 817,741,824 32 00:07.644 540,184,576
tornado 0.4a tor -11 267,109,958 240 03:57.293 1,624,543,232 33 00:08.377 1,078,104,064
</pre></div>

it will be great to find way to cut off last 3 seconds from 4x4 runtimes

unfortunately i had troubles sending program to testing. thanks to anonymous tester who done this work!

[encode]

(Bulat, please, don&#039;t post such *wide* results, like you see due to a forum bug the main table is widened due to such content. Or I will restrict the longest word (line)!)
EDIT: Fixed!

[Bulat Ziganshin]

4x4 version 0.2 - bigger, longer, uncut
* fast compression modes made even faster
* multi-threaded decompression
* fast & efficient text compression modes 1t..4t using grzip
* Win32 & Linux versions

On average, 4x4 is 12 times faster than lzturbo!!!

Below are detailed results from
http://www.metacompressor.com/uploads.aspx

<div class="jscript"><pre>file1 - big text 1,552,758,784

mode size compression/decompression time
4x4 0.2 1t 354,958,849 24 29
lzturbo 0.92 -59 322,372,806 219 21
4x4 0.2 4t 311,810,707 41 36
4x4 0.2 9 300,944,001 117 20

* 4x4 is 5 times faster than lzturbo!</pre></div>


<div class="jscript"><pre>file2 - windows 2008 server virtual disk 1,489,400,832

mode size compression/decompression time
lzturbo 0.92 -59 352,723,630 1343
4x4 0.2 6 344,522,813 25 24
4x4 0.2 9 268,639,166 97 19

* 4x4 is 53 times faster than lzturbo!!!</pre></div>


<div class="jscript"><pre>file3 - big apache log 2,072,612,014

mode size compression/decompression time
lzturbo 0.92 -59 103,156,939 313
4x4 0.2 4t 80,941,237 24 28

* 4x4 is 13 times faster than lzturbo!</pre></div>


<div class="jscript"><pre>file4 - enwik9 part wikipedia export 1,000,000,000

mode size compression/decompression time
4x4 0.2 1t 233,368,003 14 18
lzturbo 0.92 -59 232,701,587 155 14
4x4 0.2 4t 208,787,642 34 26

* 4x4 is 11 times faster than lzturbo!</pre></div>


<div class="jscript"><pre>file6 - source code linux, apache, mysql, php and firefox 729,287,225

mode size compression/decompression time
lzturbo 0.92 -59 156,185,652 109
4x4 0.2 1t 151,456,761 14 12
4x4 0.2 9 151,252,453 51 9

* 4x4 is 8 times faster than lzturbo!</pre></div>

[LovePimple]

Thanks Bulat!

[nimdamsk]

Hey! You spoil someone&#039;s business. Shame on you!

[Bulat Ziganshin]

finally, we got results for last file:

<div class="jscript"><pre>file7 - big db index 1,066,160,748

mode size compression/decompression time
lzturbo 0.92 -59 406,358,090 4405 20
4x4 0.2 1t 390,897,470 35 34
4x4 0.2 9 359,413,561 145 13

* 4x4 is 125 times faster than lzturbo! no comments </pre></div>

it seems that lzturbo optimal parsing is too slow on binary files to make it any usable, while on text files it easily outperformed by bwt/st4/ppmd algorithms. good for nothing

[Simon Berger]

Hmmmm. Didn?t you shake your head on reading those posts again later?

[Black_Fox]

Much more logical than hiding those results from others
EDIT: I mean, it means something when you&#039;re more than 10x faster than world&#039;s fastest compression library, think about it.

[Bulat Ziganshin]

Simon, i realize that i don&#039;t like Hamid and remember reasons to do it. But are you remember why you don&#039;t like me?

on technical grounds, there are not so many multithreaded compressors ATM, so comparison of two best ones seems logical, are you not agree? you definitely forgot that just one month ago you criticized me in exactly opposite direction - why i don&#039;t write lzturbo competitor if i think that it is so easy