📝

Go libraries to implement the lock mechanism

• https://github.com/gofrs/flock
• https://github.com/ipfs/go-fs-lock
• https://github.com/danjacques/gofslock
• https://github.com/jviney/go-filelock
• https://github.com/kawasin73/gointr
  • https://kawasin73.hatenablog.com/entry/2019/04/07/161330

Proposal of the lock mechanism

If we implement the lock mechanism, my idea is that we create a status file to AQUA_ROOT_DIR/statuses/<package path>.

e.g. AQUA_ROOT_DIR/statuses/github_release/github.com/cli/cli/v2.51.0/gh_2.51.0_macOS_arm64.zip/status.txt

1. Check the status of package AQUA_ROOT_DIR/locks/<package path>. Status is either completed or in progress
   2 If the status file doesn't exist, create a status file and set the status to in progress and start installing a package
2. After the installation completes, change the status to completed
3. If the status file exists but the status is in progress, wait for a while until the status becomes completed
4. If the status keeps in progress for over 10 minutes, the status file gets expired and is removed and the package directory is removed to reinstall the package
5. If the status file exists and the status is completed, but the package directory doesn't exists, change the status to in progress and install a package

The lock mechanism depends on OS level API, so maybe database such as SQLite is useful.
I'm not familiar with locking, so we need to look into further if we implement this.

I'd be happy to try hacking up a solution based on a file locking scheme or SQLite (if you're comfortable taking a dep to make it work).

But I think I see a couple of options that could be simpler. If we focus on correctness, and ignore the efficiency concerns of duplicate work (which is no worse than what is the case today), all we need is an atomic publish-directory operation. Two candidates spring to mind: renaming or symlinking. The way this would be used would go something like:

1. Check if the package is already installed (a directory exists under $(aqua root-dir)/pkgs in the expected location).
2. If so, we're done, otherwise...
3. Create a unique-named temporary work location to non-atomically install into. This should be a sibling to the location where we expect to find the package folder. For instance for shellcheck, the package folder is$(aqua root-dir)/pkgs/github_release/github.com/koalaman/shellcheck, so the temporary work location might be something like $(aqua root-dir)/pkgs/github_release/github.com/koalaman/shellcheck._aqua_install.123456. Name uniqueness could be ensured by using process ID in the name.
4. Install as normal, but into the temporary location.
5. Atomically "publish" the temporary location to the package's actual expected location.
6. If the publish operation succeeds we're done, otherwise...
7. If the publish operation failed, probably some other process installed the package while we were working. Delete our work dir. Then retry (goto 1).

There is a minor assumption with this process: I assume we can choose a tempdir naming scheme that will not conflict with any actual packages. For example, in the above example, if koalaman decided to publish a new tool named shellcheck._aqua_install.123456, and we wanted to package that up in Aqua, we'd be in a bad way. I don't think that's very likely to happen though.

Why specifically a sibling temp folder? Why not somewehre else? My understanding is that OS' guarantees about atomicity of operations often only extend to single filesystems. A temp folder anywhere else could be mounted on a different filesystem than the destination location, which might lead to weird results sometimes.

Which operation should we use for publication? As I see it, we've got two options, each with pros and cons.

• Renaming seems like a simple and obvious solution, but the catch is that directory renaming on Windows might not be atomic all the time. It is unclear if that non-atomicity manifests in ways we'd care about. And even if not atomic, the rename operation would still likely be faster than the other install operations, so the window of time and set of operations against which a race might occur would likely be reduced quite a bit, so its still an improvement for Windows (combined w/ a retry, it might even just straight up be a full fix, but I'm not a Win32 expert).
• Creating a symlink to an existing directory is atomic everywhere. But it means we need to leave behind the scratch dir the symlink points to around indefinitely, which is slightly aesthetically displeasing. Also it impacts the package-removal operation, which now needs to read through the symlink to fully clean the package (Go's os.RemoveAll will remove a symlink but not the directory it points to).

What do you think? If you'd like, I could send you a PR (just say which publication method you want). Or if you still prefer one of the other methods, I can try to build that instead. Or if you want to do it yourself, that's also fine with me. Just let me know.

Oh, I see. Your idea looks better than my idea.

renaming or symlinking

I think renaming is better than symlinking.

Why specifically a sibling temp folder? Why not somewehre else? My understanding is that OS' guarantees about atomicity of operations often only extend to single filesystems. A temp folder anywhere else could be mounted on a different filesystem than the destination location, which might lead to weird results sometimes.

I think it's reasonable. 👍

aqua installs a package into pkgPath here.

So I think we can replace pkgPath with a temporary path and move it to pkgPath.

Note that aqua installs go_build type package in checkFilesWrap.