Nothing Special   »   [go: up one dir, main page]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

zstd: Add bigger default blocks #469

Merged
merged 1 commit into from
Jan 16, 2022
Merged

zstd: Add bigger default blocks #469

merged 1 commit into from
Jan 16, 2022

Conversation

klauspost
Copy link
Owner
@klauspost klauspost commented Jan 16, 2022 •
edited
Loading

Keep 64K blocks for fastest, but do max size for others.

Healthy compression increase for streams.

(ignore speed)

BEFORE:
github-june-2days-2019.json	zskp	1	6273951764	698989536	12717	470.49
github-june-2days-2019.json	zskp	2	6273951764	617881763	18084	330.86
github-june-2days-2019.json	zskp	3	6273951764	524340691	37059	161.45
github-june-2days-2019.json	zskp	4	6273951764	468052601	170093	35.18
AFTER:
github-june-2days-2019.json	zskp	1	6273951764	698768123	10316	580.00
github-june-2days-2019.json	zskp	2	6273951764	610876538	13942	429.13
github-june-2days-2019.json	zskp	3	6273951764	517662858	42937	139.35
github-june-2days-2019.json	zskp	4	6273951764	464617114	165806	36.09


BEFORE:
github-ranks-backup.bin	zskp	1	1862623243	454073018	4646	382.27
github-ranks-backup.bin	zskp	2	1862623243	418391187	7415	239.54
github-ranks-backup.bin	zskp	3	1862623243	409783212	22014	80.69
github-ranks-backup.bin	zskp	4	1862623243	383832658	124277	14.29
AFTER:
github-ranks-backup.bin	zskp	1	1862623243	454018498	4601	386.01
github-ranks-backup.bin	zskp	2	1862623243	417923331	9886	179.68
github-ranks-backup.bin	zskp	3	1862623243	409613246	24883	71.39
github-ranks-backup.bin	zskp	4	1862623243	383667998	126586	14.03

BEFORE:
nyc-taxi-data-10M.csv	zskp	1	3325605752	641401195	9706	326.73
nyc-taxi-data-10M.csv	zskp	2	3325605752	591748091	14125	224.53
nyc-taxi-data-10M.csv	zskp	3	3325605752	530289687	33988	93.31
nyc-taxi-data-10M.csv	zskp	4	3325605752	476000605	151369	20.95
AFTER:
nyc-taxi-data-10M.csv	zskp	1	3325605752	641305324	10939	289.91
nyc-taxi-data-10M.csv	zskp	2	3325605752	588976126	14511	218.56
nyc-taxi-data-10M.csv	zskp	3	3325605752	529329260	34664	91.49
nyc-taxi-data-10M.csv	zskp	4	3325605752	474949772	143660	22.08

BEFORE:
gob-stream	zskp	1	1911399616	234966570	3484	523.19
gob-stream	zskp	2	1911399616	205669791	4346	419.36
gob-stream	zskp	3	1911399616	175034659	12428	146.67
gob-stream	zskp	4	1911399616	163364634	50935	35.79
AFTER:
gob-stream	zskp	1	1911399616	234891609	3000	607.42
gob-stream	zskp	2	1911399616	203997694	5981	304.73
gob-stream	zskp	3	1911399616	173526523	12201	149.40
gob-stream	zskp	4	1911399616	162195235	49116	37.11

@klauspost
zstd: Add bigger default blocks
37f183b 
Keep 64K blocks for fastest, but do max size for others.

Healthy compression increase for streams.
@klauspost klauspost merged commit 22de983 into master Jan 16, 2022
@klauspost klauspost deleted the zstd-bigger-default-blocks branch January 16, 2022 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@klauspost