Great idea to introduce this modern compression algorithm. My question is why you limit compression levels to just 4 options? I think it would be much more useful to support all standard levels 1 to 19. I do not see any reason why to use only 4 arbitrary selected levels. Or is it something enforced by zstd golang implementation?

For example myself I often use level 17 for online backup software as its speed on my computer nicely matches my max Internet uplink speed. I do not see why to remove option for fine tuning like this when it is available.

The Go zstd library github.com/klauspost/compress/zstd supports only four compression levels:

// EncoderLevel predefines encoder compression levels.
// Only use the constants made available, since the actual mapping
// of these values are very likely to change and your compression could change
// unpredictably when upgrading the library.
type EncoderLevel int

const (
	speedNotSet EncoderLevel = iota

	// SpeedFastest will choose the fastest reasonable compression.
	// This is roughly equivalent to the fastest Zstandard mode.
	SpeedFastest

	// SpeedDefault is the default "pretty fast" compression option.
	// This is roughly equivalent to the default Zstandard mode (level 3).
	SpeedDefault

	// SpeedBetterCompression will yield better compression than the default.
	// Currently it is about zstd level 7-8 with ~ 2x-3x the default CPU usage.
	// By using this, notice that CPU usage may go up in the future.
	SpeedBetterCompression

	// SpeedBestCompression will choose the best available compression option.
	// This will offer the best compression no matter the CPU cost.
	SpeedBestCompression

	// speedLast should be kept as the last actual compression option.
	// The is not for external usage, but is used to keep track of the valid options.
	speedLast
)

The EncoderLevelFromZstd function allows converting a standard zstd level to one of the levels supported by the library. However, there is practically no benefit from this conversion:

// EncoderLevelFromZstd will return an encoder level that closest matches the compression
// ratio of a specific zstd compression level.
// Many input values will provide the same compression level.
func EncoderLevelFromZstd(level int) EncoderLevel {
	switch {
	case level < 3:
		return SpeedFastest
	case level >= 3 && level < 6:
		return SpeedDefault
	case level >= 6 && level < 10:
		return SpeedBetterCompression
	default:
		return SpeedBestCompression
	}
}

The Go zstd library github.com/klauspost/compress/zstd supports only four compression levels:

Thank you for clarification. More granular levels would be nice but definitely are not critical. zstd with only 4 levels will make day and night difference compared to gzip anyway.

@ncw Thanks for the suggestion! I've refactored the compression handling - almost everything has been moved behind an interface, with separate implementations in gzip_handler.go, zstd_handler.go, uncompressed_handler.go and unknown_handler.go. The codebase should now be much cleaner.

The refactor looks great - thank you :-)

Can you add another target to https://github.com/rclone/rclone/blob/master/fstest/test_all/config.yaml

 - backend:  "compress"
   remote:   "TestCompressZstd:"
   fastlist: false

TestCompressZstd can be something like

[TestCompressZstd]
type = compress
remote = /tmp/rclone-compress-test-zstd
mode = zstd

And then check the integration tests all pass with test_all -remotes TestCompressZstd:,TestCompress:

Once we merge that will go into the integration tester and be run once per day.

Would it be possible to lose some dependencies? I think github.com/SaveTheRbtz/zstd-seekable-format-go uses go.uber.org/atomic which can be done in stdlib now I think and go.uber.org/zap - would it be possible to use the stdlib log/slog? That will integrate with rclone's logging system.

I will try to give the code another review in a day or two.

Can you add another target to https://github.com/rclone/rclone/blob/master/fstest/test_all/config.yaml

 - backend:  "compress"
   remote:   "TestCompressZstd:"
   fastlist: false

TestCompressZstd can be something like

[TestCompressZstd]
type = compress
remote = /tmp/rclone-compress-test-zstd
mode = zstd

And then check the integration tests all pass with test_all -remotes TestCompressZstd:,TestCompress:

@ncw Added a target in fstest/test_all/config.yaml and ran integration tests with the following configuration:

[TestCompress]
type = compress
remote = /tmp/rclone-compress-test-gzip
mode = gzip
level = -1

[TestCompressZstd]
type = compress
remote = /tmp/rclone-compress-test-zstd
mode = zstd
level = 2

Results:

I decided to fork github.com/SaveTheRbtz/zstd-seekable-format-go and removed logging, since it was mostly needed for debugging the library itself.
In addition:

• replaced go.uber.org/atomic with sync/atomic
• removed go.uber.org/* dependencies
• lowered the minimal Go version to 1.24.0

Now rclone uses github.com/a1ex3/zstd-seekable-format-go/pkg.

Additionally, fixed discovered issues with zstd in backend/compress.

Hi @A1ex3 sorry for the delay in looking at this. Your changes are great thank you :-) We ran the integration tests one last time locally and the bisync tests are failing - which they weren't for you earlier. Can you help?

Thanks

Nick

Hi @ncw,
I’ve just merged the latest changes from master locally into feat-compress-zstd branch and re-ran the integration tests for TestCompress and TestCompressZstd - both passed successfully.
Could you please share the test report or steps to reproduce the failing bisync tests?
Also, do they fail for a specific compression type or for all of them?

Is this likely to get merged imminently? I could really do with this feature.