Ever since we upgraded our Rails app to Rails 7.2, we have been experiencing a random raise ArgumentError, "'secret_key_base' for #{Rails.env} environment must be a type of String" error in our CI/CD build when running the app's tests. This issue doesn't always occur. When it does, sometimes just restarting the build works. Other times you have to restart the build several times before you get a successful build.

The stack trace in the build log shows that when the issue happens, it happens on this line of code in railties:

https://github.com/rails/rails/blob/v7.2.1.2/railties/lib/rails/application/configuration.rb#L517

Digging into this, I found that Rails 7.2 removed support for the legacy way that secrets were managed in a Rails app, e.g. via Rails.application.secrets, leaving Rails.application.credentials as the Rails-approved way to manage secrets in your Rails app.

Here is the PR in Rails 7.2 that removed support for Rails.application.secrets.

It turns out that this PR caused some regressions that were found during beta testing of Rails 7.2, and were corrected in this PR. Most importantly, this second PR introduced the behavior that when Rails.env.local? returns true, which it does when the rails environment is test (such as when running tests in a CI build), railties will now always generate a local secret_key_base stored in a file under tmp/local_secret.txt, and not use the one that is defined in config/credentials.yml.enc, which was the behavior in Rails 7.1 when we didn't have this CI issue.

Here is the code in railties that generates the secret for local Rails environments:

def generate_local_secret
  key_file = root.join("tmp/local_secret.txt")

  unless File.exist?(key_file)
    random_key = SecureRandom.hex(64)
    FileUtils.mkdir_p(key_file.dirname)
    File.binwrite(key_file, random_key)
  end

  File.binread(key_file)
end

I believe the reason why our CI build randomly experiences the raise ArgumentError, "'secret_key_base' for #{Rails.env} environment must be a type of String" error is that the generate_local_secret method above is not thread-safe, and our CI build runs the app's tests in multiple processes using the parallel_tests gem. See below:

# Run the tests with code coverage in parallel groups to speed up things
rake parallel:drop
rake parallel:setup
CI=1 COVERAGE=1 bundle exec parallel_test spec/ --type rspec --group-by runtime --runtime-log spec/parallel_runtime_rspec.log
rake coverage:merge_report

One process must be opening the tmp/local_secret.txt file and another process must be reading the file before the first process has written to the file and closed it.

To work around this issue, we added the following to our CI build process to force the creation of the tmp/local_secret.txt file prior to running the tests in parallel:

echo $(ruby -e "require 'securerandom'; puts SecureRandom.hex(64)") > tmp/local_secret.txt

Once we made this change we have not experienced the issue with any subsequent CI builds (there have been 25+ builds).

Steps to reproduce

Run a Rails app's tests in parallel using the parallel_tests gem, making sure that two or more processes are used. The issue doesn't always happen, but happens frequently enough that it becomes annoying.

Expected behavior

The generate_local_secret method in railties should work in a use case that is using multiple processes on a build machine to run the Rails app's tests.

Here's a thought - our Rails app defines a secret_key_base in config/credentials.yml.enc for all environments, including the test environment. The behavior in versions of Rails < 7.2 of using secret_key_base in config/credentials.yml.enc if it exists worked for our use case. Could that behavior be restored?

Actual behavior

Railties occasionally fails during app startup and throws an raise ArgumentError, "'secret_key_base' for #{Rails.env} environment must be a type of String" error.

System configuration

Rails version: We are using 7.2.1.2, but started noticing this issue when we first upgraded to 7.2

Ruby version: We are using 3.3.5, but I don't believe the Ruby version has any effect on this issue