spec_helper: fix and improve retry logic.

- always retry each test at least once (confusingly this means a retry
  count of 2 rather than 1)
- always wait at least 1 second between retries
- set a default retry metadata for integration tests rather than
  overriding any specified values
- use `example.run` rather than `example.run_with_retry` for integration
  tests because, confusingly, this avoids having the retry count be
  half what it should be (because the attempts increases by one for
  each `run_with_retry` call)
- use 4 retries for integration tests with 2**attempts*retry_wait
  (retry wait now being 2) to actually have more retries than before
  this commit (but with exponential backoff)

This should generally improve test flakiness in CI but particularly
improve the cleanup test flake we've seen recently.
This commit is contained in:
Mike McQuaid 2021-01-27 12:43:28 +00:00
parent 6c68e4bcb9
commit f38707e92a
No known key found for this signature in database
GPG Key ID: 48A898132FD8EE70

View File

@ -80,6 +80,8 @@ RSpec.configure do |config|
if ENV["CI"]
config.verbose_retry = true
config.display_try_failure_messages = true
config.default_retry_count = 2
config.default_sleep_interval = 1
config.around(:each, :integration_test) do |example|
example.metadata[:timeout] ||= 120
@ -88,7 +90,10 @@ RSpec.configure do |config|
config.around(:each, :needs_network) do |example|
example.metadata[:timeout] ||= 120
example.run_with_retry retry: 5, retry_wait: 5
example.metadata[:retry] ||= 4
example.metadata[:retry_wait] ||= 2
example.metadata[:exponential_backoff] ||= true
example.run
end
end