I added two new methods to cache both installed and all taps.
All taps includes core taps no matter if they're installed locally
since they're always provided by the API anyway.
This makes it easier to cache `Tap.each` while making the code
easier to reason about. It also will be useful because we'll
be able to avoid the `Tap.select(&:installed?` pattern that has
recently invaded the codebase.
Note: I also stopped clearing all tap instance caches before
tests. Running `Tap.each` would cache existing taps which would
lead to unexpected behavior since the only existing tap before
each test is the core tap. This is the only tap whose directory
is not cleaned up between tests so we just clear it's cache directly.
We also now clear all tap instances after tests as well regardless
of whether the API was used that time.
The core taps exist outside of the normal cache busting cycle
so they need to clear explicitly at the instance level.
Just to be sure we should clear all of them each time.
This essentially reverts part of the change in this PR.
- https://github.com/Homebrew/brew/pull/16746
Now we only include the cachable registry when running tests.
We basically just load it first and add a bunch of methods to
it before loading the rest of the formula files when we require
global.rb. I added a check to make sure this require order is
preserved.
I also made a bunch of methods private, stop excluding classes
that inherit from casks since it's unnecessary and add more docs.
This adds a registry for all modules and classes that
cachable is included in. The registry allows us to
programmatically clear all caches in between tests
so that we don't forget to do that when adding a new
class or refactoring code. The goal here is to reduce
the number of flaky tests in the future.
- These are arbitrary length limits that had a load of disables in code.
- The limits were only increasing over time rather than decreasing.
- Fixing the problematic code to be shorter would take a long time for
questionable gain since the problem has been around so long.
- We're not going to make the really long things be any shorter any time soon.
- The instructions in issue 14685 say, pragmatically, "disable all the rubocop
rules we're never going to realistically fix e.g. Metrics/ClassLength". But
that felt like a slippery slope to more _really_ long modules/classes/blocks,
and the limits are here for a reason.
- Extract update tests into separate tests.
- Make `brew tests --online` run only once in its own job. This job
could be made non-required to fight flakiness.
- Split up the various macOS runs into several steps now that we have
more parallel macOS workers available.
- Cleanup some flaky tests.
- fix `check_for_config_scripts` test, no idea why this wasn't working
- don't autoretry tests if `focus` or `byebug` are enabled
- add a ARM64 Ventura bottle symlink
BuildPulse is trying to find flaky tests for us but, given the previous
model of using `rspec-retry`, it would rarely find them. Instead, let's
try to always rerun `brew tests` multiple times, report to BuildPulse
each time (by moving the reporting logic into `brew tests`) and disable
`rspec-retry` when using BuildPulse.
While we're here, let's enable `rspec-retry` locally so we don't have
flaky tests biting maintainers/contributors there.
The `VERBOSE_TESTS` variable was from cask and never gets set (and
is unset by `bin/brew`). Replace it with `HOMEBREW_VERBOSE_TESTS` and
set it by `--verbose` or `--debug`.
While we're here, remove an unneeded `VERBOSE` delete (as it's already
done by `bin/brew`).
- always retry each test at least once (confusingly this means a retry
count of 2 rather than 1)
- always wait at least 1 second between retries
- set a default retry metadata for integration tests rather than
overriding any specified values
- use `example.run` rather than `example.run_with_retry` for integration
tests because, confusingly, this avoids having the retry count be
half what it should be (because the attempts increases by one for
each `run_with_retry` call)
- use 4 retries for integration tests with 2**attempts*retry_wait
(retry wait now being 2) to actually have more retries than before
this commit (but with exponential backoff)
This should generally improve test flakiness in CI but particularly
improve the cleanup test flake we've seen recently.
