-
-
Notifications
You must be signed in to change notification settings - Fork 186
/
lz4w.txt
73 lines (57 loc) · 3.65 KB
/
lz4w.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
LZ4W packer 1.43 (october 2020)
Copyright Stephane Dallongeville
https://github.com/Stephane-D/SGDK
LZ4W is part of SGDK (aka Sega Genesis Dev Kit) so refer to SGDK licence if needed.
LZ4W is a custom packer based on LZ4 compression scheme: https://github.com/lz4/lz4
The 'W' suffix stands for 'Word' as LZ4W only uses word memory accesses.
The main difference is that LZ4W is designed to take advantage of 16 bits CPU and achieve better decompression
performance on these CPU compared to classic LZ4.
The compression ratio offered by LZ4W is not as good to LZ4 but close enough depending the input data.
Note that LZ4W support using previous data buffer to improve compression if available.
Decompression speed (running on a 7.67 Mhz 68000 using optimized assembly code)
-------------------
LZ4 min=270KB/s max=390KB/s
LZ4W min=550KB/s max=950KB/s
As you can see the speed difference is very important, LZ4W can unpack at least twice as fast than classic LZ4 !
Compression ratio (LZ4HC vs custom LZ4W packer)
-----------------
linear tiles data original=12672 LZ4HC=6622 LZ4W=6582
planar tiles data original=1536 LZ4HC=1061 LZ4W=1038
highly compressible tiles original=346656 LZ4HC=24613 LZ4W=29308
single BMP sprite original=4798 LZ4HC=2202 LZ4W=2158
map data original=4576 LZ4HC=1367 LZ4W=1312
text file original=40992 LZ4HC=16088 LZ4W=21400
LZ4W performs nicely compared to LZ4 for small packet of data (it often does better than LZ4HC) but it becomes worst as the
data packet become larger or using lot of redudant byte pattern data (as TXT file).
Keep in mind that LZ4W has been developed for the Sega Megadrive target first which has very limited amount of RAM anyway (64 KB) so you're likely to unpack small data blocks almost of time.
Still for people looking for a very fast unpacker LZ4W is probably the best solution around for 68000 CPU based systems.
68000 unpacking code can be found in SGDK library (https://stephane-d.github.io/SGDK/)
LZ4W compressor is provided in JAR format (java code) in SGDK package so it requires a JVM (Java Virtual Machine) to work.
Use the following command to execute it: java -jar lz4w.jar
LZ4W format description
-----------------------
0000-xxxx: LZ4W compressed blocks
Each block consists of a 16 bit block header optionally followed by a literal data block and an optional
long offset word for long match block.
At this point you need to know the principe of the LZ77 compression format on which LZ4 and LZ4W are based.
block header format (16 bit)
----------------------------
LLLL MMMM OOOO OOOO
LLLL = literal size (in word) if > 0 then a literal data block follow the block header
MMMM = match size - 1 (in word) if (MMMM == 1) then we have a match size of 2 words
OOOOOOOO = match offset - 1 (in word) if (OOOOOOOO == 0) then we have a match offset of 1 word
* if (LLLL == 0) && (MMMM == 0) && (OOOOOOOO == 0) --> end (see ending block description)
* if (LLLL == 0) --> no literal data
* if (MMMM == 0)
- if (OOOOOOOO == 0) --> no match data
- if (OOOOOOOO != 0) --> long match block
OOOOOOOO = match length - 2 (in word) if (OOOOOOOO == 1) then we have a match length of 3 words
the next word *after* literal data block become the match offset - 1 (in word)
Match offset in encoded as follow: XOOOOOOO OOOOOOOO
- if (X == 0) the match comes from current data block (RAM)
- if (X == 1) the match comes from a previous data block (stored in ROM)
ending block
------------
Ending block consists of a last word data:
Dxxx xxxx YYYY YYYY
if (D == 0) YYYY YYYY = last byte of data