flate: Add limited window compression #843
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds a medium compressor that can operate with limited window size.
Exposed in gzip outside deflate for now.
Example sizes:
```
=== RUN TestFileWindow/32
gzip_test.go:349: size: 82504 bytes
=== RUN TestFileWindow/64
gzip_test.go:349: size: 75350 bytes
=== RUN TestFileWindow/128
gzip_test.go:349: size: 70668 bytes
=== RUN TestFileWindow/256
gzip_test.go:349: size: 69276 bytes
=== RUN TestFileWindow/512
gzip_test.go:349: size: 68327 bytes
=== RUN TestFileWindow/1024
gzip_test.go:349: size: 67876 bytes
=== RUN TestFileWindow/2048
gzip_test.go:349: size: 40900 bytes
=== RUN TestFileWindow/4096
gzip_test.go:349: size: 38684 bytes
=== RUN TestFileWindow/8192
gzip_test.go:349: size: 36263 bytes
=== RUN TestFileWindow/16384
gzip_test.go:349: size: 35434 bytes
=== RUN TestFileWindow/32768
gzip_test.go:349: size: 34654 bytes
--- PASS: TestFileWindow (0.03s)
```
Limited testing done.
|
@rojer - are you able to test if this does what you need? |
|
not immediately, but will do |
|
ok, i've run tests and happy to report that it works. |
|
@rojer Great! I will merge it then. It seems like the next version will be a feature release, so it may take a bit to get out. |
|
thank you @klauspost , much appreciated! |
kodiakhq bot
referenced
this pull request
in cloudquery/filetypes
Oct 1, 2023
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/klauspost/compress](https://togithub.com/klauspost/compress) | indirect | minor | `v1.16.7` -> `v1.17.0` | --- ### Release Notes <details> <summary>klauspost/compress (github.com/klauspost/compress)</summary> ### [`v1.17.0`](https://togithub.com/klauspost/compress/releases/tag/v1.17.0) [Compare Source](https://togithub.com/klauspost/compress/compare/v1.16.7...v1.17.0) #### What's Changed - Add dictionary builder by [@​klauspost](https://togithub.com/klauspost) in [https://github.com/klauspost/compress/pull/853](https://togithub.com/klauspost/compress/pull/853) - Add xerial snappy read/writer by [@​klauspost](https://togithub.com/klauspost) in [https://github.com/klauspost/compress/pull/838](https://togithub.com/klauspost/compress/pull/838) - flate: Add limited window compression by [@​klauspost](https://togithub.com/klauspost) in [https://github.com/klauspost/compress/pull/843](https://togithub.com/klauspost/compress/pull/843) - s2: Do 2 overlapping match checks by [@​klauspost](https://togithub.com/klauspost) in [https://github.com/klauspost/compress/pull/839](https://togithub.com/klauspost/compress/pull/839) - flate: Add amd64 assembly matchlen by [@​klauspost](https://togithub.com/klauspost) in [https://github.com/klauspost/compress/pull/837](https://togithub.com/klauspost/compress/pull/837) - gzip: Copy bufio.Reader on Reset by [@​thatguystone](https://togithub.com/thatguystone) in [https://github.com/klauspost/compress/pull/860](https://togithub.com/klauspost/compress/pull/860) - zstd: Remove offset from bitReader by [@​greatroar](https://togithub.com/greatroar) in [https://github.com/klauspost/compress/pull/854](https://togithub.com/klauspost/compress/pull/854) - fse, huff0, zstd: Remove always-nil error returns by [@​greatroar](https://togithub.com/greatroar) in [https://github.com/klauspost/compress/pull/857](https://togithub.com/klauspost/compress/pull/857) - tests: unnecessary use of fmt.Sprintf by [@​testwill](https://togithub.com/testwill) in [https://github.com/klauspost/compress/pull/836](https://togithub.com/klauspost/compress/pull/836) - tests: Fix OSS fuzzer t.Run by [@​klauspost](https://togithub.com/klauspost) in [https://github.com/klauspost/compress/pull/852](https://togithub.com/klauspost/compress/pull/852) - tests: Use Go 1.21.x by [@​klauspost](https://togithub.com/klauspost) in [https://github.com/klauspost/compress/pull/851](https://togithub.com/klauspost/compress/pull/851) #### New Contributors - [@​testwill](https://togithub.com/testwill) made their first contribution in [https://github.com/klauspost/compress/pull/836](https://togithub.com/klauspost/compress/pull/836) - [@​thatguystone](https://togithub.com/thatguystone) made their first contribution in [https://github.com/klauspost/compress/pull/860](https://togithub.com/klauspost/compress/pull/860) **Full Changelog**: klauspost/compress@v1.16.7...v1.17.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNi4xMDkuNCIsInVwZGF0ZWRJblZlciI6IjM2LjEwOS40IiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9-->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a medium compressor that can operate with limited window size.
Exposed in gzip outside deflate for now.
Example sizes:
Limited testing done.