Bug Report: builtin backup may create invalid file upon retry #18708
Description
The builtin backup engine has a retry mechanism. If a given file failed to be backed up, for any reason, we retry it at most once.
In some cases, if the failure happened while closing a given file, the retry mechanism will re-upload the file, but the file will never be restorable in the future.
Here is an example when restoring a given file:
can't restore from backup: failed to restore file 226 to fts_0000000000000460_000000000000013e_index_4.ibd: failed to copy file contents: open <path>: storage: object doesn't exist
That same file had failed once in the previous successful backup:
builtinbackupengine.go:873] Backing up file: platform/fts_0000000000000460_000000000000013e_index_4.ibd (attempt 1/2)
builtinbackupengine.go:818] Completed backing up "platform/fts_0000000000000460_000000000000013e_index_4.ibd" (attempt 1/2)
logutil.go:31] log: INFO file closed component=fs name=226 duration=30.108909814s partitions=1 bytes=114688 error="googleapi: got HTTP response code 503 with body: Service Unavailable"
failed to close file 226,platform/fts_0000000000000460_000000000000013e_index_4.ibd
builtinbackupengine.go:873] Backing up file: platform/fts_0000000000000460_000000000000013e_index_4.ibd (attempt 2/2)
builtinbackupengine.go:818] Completed backing up "platform/fts_0000000000000460_000000000000013e_index_4.ibd" (attempt 2/2)
logutil.go:31] log: INFO file closed component=fs name=226 duration=53.862171ms partitions=1 bytes=114688 error=<nil>
This makes me think that when retrying a failed file, we incorrectly overwrite the data. One solution could be to ensure we remove the given file from the remote storage before retrying it, so we always start from a clean slate.