Some servers will return an error response if a `Content-Length`
header isn't included in a `POST` request, so this adds it to the
`post_args` array when `post_form` or `post_json` are used.
This adds a `Livecheck::Options` class, which is intended to house
various configuration options that are set in `livecheck` blocks,
conditionally set by livecheck at runtime, etc. The general idea is
that when we add features involving configurations options (e.g., for
livecheck, strategies, curl, etc.), we can make changes to `Options`
without needing to modify parameters for strategy `find_versions`
methods, `Strategy` methods like `page_headers` and `page_content`,
etc. This is something that I've been trying to improve over the years
and `Options` should help to reduce maintenance overhead in this area
while also strengthening type signatures.
`Options` replaces the existing `homebrew_curl` option (which related
strategies pass to `Strategy` methods and on to `curl_args`) and the
new `url_options` (which contains `post_form` or `post_json` values
that are used to make `POST` requests). I recently added `url_options`
as a temporary way of enabling `POST` support without `Options` but
this restores the original `Options`-based implementation.
Along the way, I added a `homebrew_curl` parameter to the `url` DSL
method, allowing us to set an explicit value in `livecheck` blocks.
This is something that we've needed in some cases but I also intend
to replace implicit/inferred `homebrew_curl` usage with explicit
values in `livecheck` blocks once this is available for use. My
intention is to eventually remove the implicit behavior and only rely
on explicit values. That will align with how `homebrew_curl` options
work for other URLs and makes the behavior clear just from looking at
the `livecheck` block.
Lastly, this removes the `unused` rest parameter from `find_versions`
methods. I originally added `unused` as a way of handling parameters
that some `find_versions` methods have but others don't (e.g., `cask`
in `ExtractPlist`), as this allowed us to pass various arguments to
`find_versions` methods without worrying about whether a particular
parameter is available. This isn't an ideal solution and I originally
wanted to handle this situation by only passing expected arguments to
`find_versions` methods but there was a technical issue standing in
the way. I recently found an answer to the issue, so this also
replaces the existing `ExtractPlist` special case with generic logic
that checks the parameters for a strategy's `find_versions` method
and only passes expected arguments.
Replacing the aforementioned `find_versions` parameters with `Options`
ensures that the remaining parameters are fairly consistent across
strategies and any differences are handled by the aforementioned
logic. Outside of `ExtractPlist`, the only other difference is that
some `find_versions` methods have a `provided_content` parameter but
that's currently only used by tests (though it's intended for caching
support in the future). I will be renaming that parameter to `content`
in an upcoming PR and expanding it to the other strategies, which
should make them all consistent outside of `ExtractPlist`.
I initially set the type for livecheck's `post_form` and `post_json`
hashes to allow either a string or symbol key. I used string keys in
the documentation, as there will inevitably be some form field names
that would pose a problem for symbols (e.g., `E-mail` uses a hyphen,
`1twothree` starts with a digit, etc.). However, I remembered that we
can simply use quote symbols like `:"E-mail"` to handle these
situations, as they have the flexibility of a string while still being
a symbol.
With that in mind, this updates related type signatures to only allow
symbol keys and updates documentation and tests accordingly. The
documentation example contains a hyphenated form field, so it
demonstrates how to handle names that don't work as a bare symbol.
livecheck currently doesn't support `POST` requests but it wasn't
entirely clear how best to handle that. I initially approached it as
a `Post` strategy but unfortunately that would have required us to
handle response body parsing (e.g., JSON, XML, etc.) in some fashion.
We could borrow some of the logic from related strategies but we would
still be stuck having to update `Post` whenever we add a strategy for
a new format.
Instead, this implements `POST` support by borrowing ideas from the
`using: :post` and `data` `url` options found in formulae. This uses
a `post_form` option to handle form data and `post_json` to handle
JSON data, encoding the hash argument for each into the appropriate
format. The presence of either option means that curl will use a
`POST` request.
With this approach, we can make a `POST` request using any strategy
that calls `Strategy::page_headers` or `::page_content` (directly or
indirectly) and everything else works the same as usual. The only
change needed in related strategies was to pass the options through
to the `Strategy` methods.
For example, if we need to parse a JSON response from a `POST`
request, we add a `post_data` or `post_json` hash to the `livecheck`
block `url` and use `strategy :json` with a `strategy` block. This
leans on existing patterns that we're already familiar with and
shouldn't require any notable maintenance burden when adding new
strategies, so it seems like a better approach than a `Post` strategy.
The `url_provided` parameter of the `Strategy#from_url` method was
originally introduced in #9529 but I removed it in a later commit in
that PR in favor of a different approach. Unfortunately, I forgot to
remove the `url_provided` parameter, as it was no longer needed after
that change. This removes the parameter and updates `#from_url` calls
accordingly.
This improves the load time of most brew commands. For an example of
one of the simplest commands this speeds up:
Without Bootsnap:
```
$ hyperfine 'git checkout master; brew help' 'git checkout optimise_requires; brew help'
Benchmark 1: git checkout master; brew help
Time (mean ± σ): 525.0 ms ± 35.8 ms [User: 229.9 ms, System: 113.1 ms]
Range (min … max): 465.3 ms … 576.6 ms 10 runs
Benchmark 2: git checkout optimise_requires; brew help
Time (mean ± σ): 383.3 ms ± 25.1 ms [User: 133.0 ms, System: 72.1 ms]
Range (min … max): 353.0 ms … 443.6 ms 10 runs
Summary
git checkout optimise_requires; brew help ran
1.37 ± 0.13 times faster than git checkout master; brew help
```
With Bootsnap:
```
$ hyperfine 'git checkout master; brew help' 'git checkout optimise_requires; brew help'
Benchmark 1: git checkout master; brew help
Time (mean ± σ): 386.0 ms ± 30.9 ms [User: 130.2 ms, System: 93.8 ms]
Range (min … max): 359.5 ms … 469.3 ms 10 runs
Benchmark 2: git checkout optimise_requires; brew help
Time (mean ± σ): 330.2 ms ± 32.4 ms [User: 93.4 ms, System: 73.0 ms]
Range (min … max): 302.9 ms … 413.9 ms 10 runs
Summary
git checkout optimise_requires; brew help ran
1.17 ± 0.15 times faster than git checkout master; brew help
```
I previously expanded use of `typed: strict` in livecheck files but
the exception was `livecheck/strategy.rb`. This addresses the
`@strategies` type errors in that file and upgrades it to
`typed: strict`.
Co-authored-by: apainintheneck <apainintheneck@gmail.com>
This updates livecheck files to use `typed: script` where feasible.
The remaining exception is `livecheck/strategy.rb`, as I wasn't
able to figure out how to resolve the typing issues around the
`@strategies` variable (I tried a couple of approaches but couldn't
find a working solution). This includes changes to resolve the other
type errors in `strategy.rb` but leaves the file as `typed: true`
for now.
@samford is working on the ability to specify custom UA (among other things)
in livecheck blocks; "retrying" will cease to be relevant
Co-authored-by: Sam Ford <1584702+samford@users.noreply.github.com>
We discussed the idea of adding a livecheck strategy to check crate
versions years ago but decided to put it off because it would have
only applied to one formula at the time (and it wasn't clear that a
crate was necessary in that case). We now have a few formulae that
use a crate in the `stable` URL (`cargo-llvm-cov`, `pngquant`,
`oakc`) and another formula with a crate resource (`deno`), so
there's some value to the idea now.
I established a standard approach for checking crate versions in a
somewhat recent `pngquant` `livecheck` block update and this commit
reworks it into a strategy, so we won't have to duplicate that
`livecheck` block in these cases. With this strategy, we usually
won't even need a `livecheck` block at all.
Under normal circumstances, a regex and/or strategy block shouldn't
be necessary but the strategy supports them when needed. The response
from the crates.io API is a JSON object, so this uses
`Json#versions_from_content` internally and a `strategy` block will
receive the parsed `json` object and a regex (the strategy default or
the regex from the `livecheck` block).
The default curl args in `#curl_headers` cover most of
`Livecheck::Strategy::DEFAULT_CURL_ARGS` but `--max-redirs` was
overlooked. This adds an explicit `--max-redirs` argument in
the `#page_headers` `#curl_headers` call but it's worth mentioning
that this approach wouldn't benefit from any changes in
`DEFAULT_CURL_ARGS` and would need to be manually kept in parity.
`#curl_headers` was recently introduced into `Strategy#page_headers`
but only the call was modified and the method wasn't updated to
correctly work with the new return value, so all `HeaderMatch` checks
immediately started failing with an error.
This commit includes changes that return `#page_headers` to a working
state. I've removed the `result.assert_success!` call because it
prevents a few checks from being retried with `GET` (`firefox-cn`,
`krisp`, `prepros`).
This adds a generic `Yaml` strategy to livecheck that requires a
`strategy` block to operate. The YAML-parsing code is taken from the
existing approach in the `ElectronBuilder` strategy.
We don't currently have any `strategy` blocks in first-party taps
that manually parse YAML. However, creating a generic `Yaml` strategy
allows us to simplify `ElectronBuilder` (and any future strategy
that works with YAML) while making it easy to create custom `Yaml`
`strategy` blocks in formulae/casks as needed.
This adds a generic `Xml` strategy to livecheck that requires a
`strategy` block to operate. The XML-parsing code is taken from the
existing approach in the `Sparkle` strategy. As such, `Sparkle` has
been updated to use the `Xml#parse_xml` method instead.
Unlike the `Json` strategy, we don't currently have any `strategy`
blocks in first-party taps that manually parse XML. However, we had a
user request support for something like this and I was already working
on an `Xml` strategy (as a way of extracting the XML-parsing code
from `Sparkle` into something general-purpose), so here we are.
Future strategies that parse simple XML data can potentially use the
`Xml#find_versions` method (similar to how we have strategies that
leverage `PageMatch#find_versions`) instead of having to implement
something bespoke like `Sparkle`.
Passing the strategy symbol into the `#from_url` `select` block
means that we can also get rid of a `#from_symbol` call in a
different conditional branch, as we can directly compare the
`livecheck_strategy` symbol to `strategy_symbol`.
When the `Json` strategy was introduced, I forgot to also ensure
that it's only treated as usable (in `Strategy#from_url`) if a
`livecheck` block uses `strategy :json`. As a result, `Json` is
incorrectly treated as a usable strategy for all formulae/casks that
contain a `strategy` block.
Since all of these `livecheck` blocks specify a strategy, this bug
doesn't meaningfully impact livecheck's behavior (i.e., these checks
continue to use their explicitly-specified strategy). The only
practical difference is that `Json` incorrectly appears in the list
of usable strategies in livecheck's verbose JSON output.
This commit modifies `Strategy#from_url` to address this issue. The
easiest way to enforce this rule involved passing in the
`@strategies` key (a symbol) into the `select` block, so we can
compare it to `livecheck_strategy` (the strategy symbol specified in
the `livecheck` block).
This adds a generic `Json` strategy to livecheck that requires a
`strategy` block to operate. This is primarily intended as a
replacement for existing `strategy` blocks in formulae/casks that
use `JSON#parse`, as it allows us to internalize/standardize that
boilerplate while improving error-handling.
Additionally, future strategies that parse JSON data can use the
`Json#find_versions` method instead of having to reinvent the wheel
(similar to how we currently have a number of strategies that
leverage `PageMatch#find_versions`).
The default redirection maximum for `curl` is 50 but we should use
something more reasonable in livecheck. It's rare but a misconfigured
server with an endless redirection loop will hit the 50 redirection
limit. Unfortunately, we've encountered this in the wild (e.g., the
server for `getmail` and `memtester` endlessly redirects), so it's
not an idle concern. This commit basically adds `--max-redirs 5` to
`Livecheck::Strategy::DEFAULT_CURL_ARGS` to enforce a more reasonable
redirection maximum.
To be clear, the `max_iterations` logic in `#parse_curl_output`
(which was previously found in `Strategy#page_content`) doesn't
restrict the number of redirections that `curl` follows. At the point
the `curl` output is being parsed, the requests have already been
made and `max_iterations` simply restricts the number of responses
`#parse_curl_output` is willing to parse. If we use `--max-redirs`
and properly set `max_iterations` to `max-redirs + 1`, we shouldn't
encounter the "Too many redirects" error in `#parse_curl_output`.
Currently, only `Livecheck::Strategy::PAGE_HEADERS_CURL_ARGS` uses
the `--silent` option and `PAGE_CONTENT_CURL_ARGS` does not (though
there's no intention behind this omission). However, the
`#page_content` method should also use the `--silent` flag, to
prevent progress bar text (`#=#=#`, etc.) from appearing in output.
This is an issue because the regex that's used to identify `curl`
error messages in `stderr` (`^curl:.+$/`) will fail if leading
progress bar text is present. This leads to an ambiguous "cURL
failed without a detectable error" message instead of the actual
error message(s) from `curl`.
This commit addresses the issue by adding `--silent` to
`Livecheck::Strategy::DEFAULT_CURL_ARGS`, which both
`PAGE_HEADERS_CURL_ARGS` and `PAGE_CONTENT_CURL_ARGS` inherit.
The existing regex wasn't able to match errors like:
curl: option --something: is unknown
Additionally, the existing approach wouldn't capture multi-line
errors, whereas this captures all the `curl:` lines from `stderr`.
Valid `strategy` block return types currently vary between
strategies. Some only accept a string whereas others accept a string
or array of strings. [`strategy` blocks also accept a `nil` return
(to simplify early returns) but this was already standardized across
strategies.]
While some strategies only identify one version by default (where a
string is an appropriate return type), it could be that a strategy
block identifies more than one version. In this situation, the
strategy would need to be modified to accept (and work with) an
array from a `strategy` block.
Rather than waiting for this to become a problem, this modifies all
strategies to standardize on allowing `strategy` blocks to return a
string or array of strings (even if only one of these is currently
used in practice). Standardizing valid return types helps to further
simplify the mental model for `strategy` blocks and reduce cognitive
load.
This commit extracts related logic from `#find_versions` into
methods like `#versions_from_content`, which is conceptually similar
to `PageMatch#page_matches` (renamed to `#versions_from_content`
for consistency). This allows us to write tests for the related code
without having to make network requests (or stub them) at this point.
In general, this also helps to better align the structure of
strategies and how the various `#find_versions` methods work with
versions.
There's still more planned work to be done here but this is a step
in the right direction.
