The `Bitbucket` strategy checks download or tag pages but the content
is now fetched separately on page load, so the strategy is failing for
all related formulae. This updates the generated strategy URLs to
fetch the page content instead, which works as expected.
Some servers will return an error response if a `Content-Length`
header isn't included in a `POST` request, so this adds it to the
`post_args` array when `post_form` or `post_json` are used.
As suggested, this reworks `Options` to subclass `T::Struct`, which
simplifies the implementation and makes it easier to maintain.
One noteworthy difference in switching to `T::Struct` is that
`#serialize` omits `nil` values but I don't _think_ this should be a
problem for us. In terms of changes, I modified `#merge` to remove a
now-unnecessary `compact` call and updated related tests.
Co-authored-by: Douglas Eichelberger <697964+dduugg@users.noreply.github.com>
This adds a `Livecheck::Options` class, which is intended to house
various configuration options that are set in `livecheck` blocks,
conditionally set by livecheck at runtime, etc. The general idea is
that when we add features involving configurations options (e.g., for
livecheck, strategies, curl, etc.), we can make changes to `Options`
without needing to modify parameters for strategy `find_versions`
methods, `Strategy` methods like `page_headers` and `page_content`,
etc. This is something that I've been trying to improve over the years
and `Options` should help to reduce maintenance overhead in this area
while also strengthening type signatures.
`Options` replaces the existing `homebrew_curl` option (which related
strategies pass to `Strategy` methods and on to `curl_args`) and the
new `url_options` (which contains `post_form` or `post_json` values
that are used to make `POST` requests). I recently added `url_options`
as a temporary way of enabling `POST` support without `Options` but
this restores the original `Options`-based implementation.
Along the way, I added a `homebrew_curl` parameter to the `url` DSL
method, allowing us to set an explicit value in `livecheck` blocks.
This is something that we've needed in some cases but I also intend
to replace implicit/inferred `homebrew_curl` usage with explicit
values in `livecheck` blocks once this is available for use. My
intention is to eventually remove the implicit behavior and only rely
on explicit values. That will align with how `homebrew_curl` options
work for other URLs and makes the behavior clear just from looking at
the `livecheck` block.
Lastly, this removes the `unused` rest parameter from `find_versions`
methods. I originally added `unused` as a way of handling parameters
that some `find_versions` methods have but others don't (e.g., `cask`
in `ExtractPlist`), as this allowed us to pass various arguments to
`find_versions` methods without worrying about whether a particular
parameter is available. This isn't an ideal solution and I originally
wanted to handle this situation by only passing expected arguments to
`find_versions` methods but there was a technical issue standing in
the way. I recently found an answer to the issue, so this also
replaces the existing `ExtractPlist` special case with generic logic
that checks the parameters for a strategy's `find_versions` method
and only passes expected arguments.
Replacing the aforementioned `find_versions` parameters with `Options`
ensures that the remaining parameters are fairly consistent across
strategies and any differences are handled by the aforementioned
logic. Outside of `ExtractPlist`, the only other difference is that
some `find_versions` methods have a `provided_content` parameter but
that's currently only used by tests (though it's intended for caching
support in the future). I will be renaming that parameter to `content`
in an upcoming PR and expanding it to the other strategies, which
should make them all consistent outside of `ExtractPlist`.
I initially set the type for livecheck's `post_form` and `post_json`
hashes to allow either a string or symbol key. I used string keys in
the documentation, as there will inevitably be some form field names
that would pose a problem for symbols (e.g., `E-mail` uses a hyphen,
`1twothree` starts with a digit, etc.). However, I remembered that we
can simply use quote symbols like `:"E-mail"` to handle these
situations, as they have the flexibility of a string while still being
a symbol.
With that in mind, this updates related type signatures to only allow
symbol keys and updates documentation and tests accordingly. The
documentation example contains a hyphenated form field, so it
demonstrates how to handle names that don't work as a bare symbol.
Between this commit and the previous one, this brings test coverage
for `Livecheck::Strategy` up to 98.18% line coverage and 97.22%
branch coverage. The only uncovered areas are some Sorbet `params`
calls (which I'm not sure how to cover) and a conditional `break` in
`page_headers` that will be refactored away in the future.
The increased coverage is primarily in areas that weren't covered
before because they call methods that make network requests. I worked
around this with stubs and doubles, so we can test this code to some
degree. I plan to expand this approach to other areas in livecheck
that aren't covered for the same reason and that should significantly
increase test coverage (along with some other test improvements that
I have lined up).
livecheck currently doesn't support `POST` requests but it wasn't
entirely clear how best to handle that. I initially approached it as
a `Post` strategy but unfortunately that would have required us to
handle response body parsing (e.g., JSON, XML, etc.) in some fashion.
We could borrow some of the logic from related strategies but we would
still be stuck having to update `Post` whenever we add a strategy for
a new format.
Instead, this implements `POST` support by borrowing ideas from the
`using: :post` and `data` `url` options found in formulae. This uses
a `post_form` option to handle form data and `post_json` to handle
JSON data, encoding the hash argument for each into the appropriate
format. The presence of either option means that curl will use a
`POST` request.
With this approach, we can make a `POST` request using any strategy
that calls `Strategy::page_headers` or `::page_content` (directly or
indirectly) and everything else works the same as usual. The only
change needed in related strategies was to pass the options through
to the `Strategy` methods.
For example, if we need to parse a JSON response from a `POST`
request, we add a `post_data` or `post_json` hash to the `livecheck`
block `url` and use `strategy :json` with a `strategy` block. This
leans on existing patterns that we're already familiar with and
shouldn't require any notable maintenance burden when adding new
strategies, so it seems like a better approach than a `Post` strategy.
The `mesa` formula currently uses a mesa.freedesktop.org/archive/
`stable` URL but it redirects to archive.mesa3d.org. Upstream links
to archive.mesa3d.org as the location to find Mesa releases, so we
should update the formula URLs accordingly.
This updates the `Xorg` strategy to be able to handle
archive.mesa3d.org URLs, so livecheck will continue to be able to
check `mesa` without needing a one-off `livecheck` block. [This would
also work for `mesalib-glw` (which has an archive.mesa3d.org `stable`
URL) but that formula is deprecated.]
We recently updated the `Pypi` strategy to use the PyPI JSON API and
the default strategy behavior no longer relies on a regex, so the initial implementation didn't include regex handling. This restores
support for a `livecheck` block regex by updating the `DEFAULT_BLOCK`
logic to handle an optional regex. This allows us to use a regex to
omit parts of the `info.version` value without having to duplicate
the default block logic in a `strategy` block only to use a regex.
This isn't currently necessary for any existing formulae using the
`Pypi` strategy but we have a few that needed a custom regex with
the previous strategy approach, so they may need this functionality
in the future. Besides that, restoring regex support to `Pypi`
ensures that `livecheck`/`strategy` blocks work in a fairly
consistent manner across strategies.
This updates the block-handling logic in `Json::versions_from_content`
to naively pass the regex value when the block has two parameters. Up
to now, we have been ensuring that `regex` is not `nil` and this
makes sense with existing usage (the `Crate` strategy's default
block, formulae/cask `strategy` blocks). However, we need to allow a
`nil` `regex` value to make it possible to add an optional `regex`
parameter in the `Pypi::DEFAULT_BLOCK` Proc. This is necessary to
allow the `Pypi` strategy to work with an optional regex from a
`livecheck` block again [without creating an additional
`DEFAULT_BLOCK` variant with a regex parameter].
Among other things, the previous commit added a `provided_content`
paramter to `Pypi::find_versions`, so this takes advantage of that to
expand `Pypi` test coverage to 100%.
This reworks the new `Pypi` JSON API implementation to use
`Json::find_versions` in `Pypi::find_versions`, borrowing some of the
approach from the `Crate` strategy.
Besides that, this pares down the fields in the
`::generate_input_values` return hash to only `:url`, as we're not
using a generated regex to match version information in this setup.
This adds a `provided_content` parameter to `::find_versions` as part
of this process and I will expand the `Pypi` tests to increase
coverage (like the `Crates` tests) in a later PR. 75% of `Pypi` checks
are failing at the moment (with some returning inaccurate version
information), so the current priority is getting this fix merged in
the short-term.
Formulae, casks, and resources have a `#livecheckable?` method that
indicates whether they contain a `livecheck` block. This is intended
to be read as "has a livecheckable?", not "is livecheckable?" (as
livecheck can find versions for some packages/resources without a
`livecheck` block). Unfortunately, correct understanding of this
method's behavior [outside of documentation] relies on historical
knowledge that few people possess, so this is often confusing to
anyone who hasn't been working on livecheck since 2020.
In the olden days, a "livecheckable" was a Ruby file containing a
`livecheck` block (originally a hash) with a filename that
corresponded to a related formula. The `livecheck` blocks in
livecheckable files were integrated into their respective formulae in
August 2020, so [first-party] livecheckables ceased to exist at that
time. From that point forward, we simply referred to these as
`livecheck` blocks.
With that in mind, this clarifies the situation by replacing
"livecheckable" language. This includes renaming `#livecheckable?` to
`#livecheck_defined?`, replacing usage of "livecheckable" as a noun
with "`livecheck` block", replacing "livecheckable" as a boolean with
"livecheck_defined", and replacing incorrect usage of "livecheckable"
as an adjective with "checkable".
Up to now, we haven't been accounting for `#url` symbol arguments in
`livecheck` blocks that don't reference a checkable URL. This can
either be an invalid symbol (e.g., using the `:stable` formula symbol
in a cask) or a valid symbol where the referenced URL doesn't exist
(e.g., using `:head` when there's no `head` URL). [Almost all of the
valid symbols are required URLs but `head` is optional.]
Over the years, we've had a handful of slips where we've used `:url`
in formulae (when it's only valid in casks) and `:stable` in casks
(when it's only valid in formulae). In this scenario,
`livecheck_url_string` is `nil`, so livecheck falls back to
`#checkable_urls`. In this scenario, `stable` and `url` are the first
checkable URLs for formulae and casks (respectively), so the checks
ended up working as expected merely by chance. This isn't obvious in
the output and even the debug output looks normal. It only becomes
apparent that livecheck isn't working as expected if it iterates
through more than one checkable URL before reaching one that works
(not the case in those instances).
With that in mind, this adds an error when a `#url` symbol is used
but it doesn't correspond to a checkable URL. This will account for
both of the mentioned scenarios (invalid symbols and valid ones
referencing a non-existent URL) and prevent livecheck from quietly
proceeding in an unexpected manner.
`Livecheck#preprocess_url` only contains logic for rewriting Git URLs,
so it makes more sense for this code to be part of the `Git` strategy
instead. Outside of better code organization, this saves us from
having to maintain the list of strategies to skip processing (which
is sometimes forgotten when a new strategy is added) and makes it
easier to do something similar in other strategies as needed.
One thing to note is that `Livecheck#preprocess_url` was previously
called on the URL before each strategy's `#match?` method was called.
To maintain the existing behavior, this calls `Git#preprocess_url` in
`Git#match?`. However, we need the processed URL when we use the `Git`
strategy, so we have to call `Git#preprocess_url` again. To avoid
duplicating effort, I've added a `@processed_urls` hash to the `Git`
strategy and have set up `Git#preprocess_url` to cache processed
URLs, so we only do the work once. There may be a better way of
handling it but this seems to work as expected.
livecheck is returning an `Unable to get versions` error for the
`ansible-lint`, `aws-sam-cli`, and `pyqt-builder` formulae. These use
the `Pypi` strategy without a `livecheck` block, so they use the
generated regex from the strategy. The `Pypi` strategy matches the
version from the tarball link on the pypi.org package page but this
fails for these packages because the formula's `stable` tarball uses
hyphens in the filename (e.g., `ansible-lint-...`) but the current
tarball filename uses underscores (e.g., `ansible_lint-...`).
This addresses the issue by updating the strategy regex to replace
[escaped] `-` or `_` characters in the package name with `[_-]`, so
the regex will match regardless of the delimiter used in the formula
filename.
When the `--extract-plist` option was added to livecheck, conditions
were added in `#run_checks` to skip casks using `ExtractPlist` if the
`--extract-plist` isn't used and the run involves multiple
formulae/casks. This integrates the skip into the `SkipConditions`
class.
We discussed the idea of adding a livecheck strategy to check crate
versions years ago but decided to put it off because it would have
only applied to one formula at the time (and it wasn't clear that a
crate was necessary in that case). We now have a few formulae that
use a crate in the `stable` URL (`cargo-llvm-cov`, `pngquant`,
`oakc`) and another formula with a crate resource (`deno`), so
there's some value to the idea now.
I established a standard approach for checking crate versions in a
somewhat recent `pngquant` `livecheck` block update and this commit
reworks it into a strategy, so we won't have to duplicate that
`livecheck` block in these cases. With this strategy, we usually
won't even need a `livecheck` block at all.
Under normal circumstances, a regex and/or strategy block shouldn't
be necessary but the strategy supports them when needed. The response
from the crates.io API is a JSON object, so this uses
`Json#versions_from_content` internally and a `strategy` block will
receive the parsed `json` object and a regex (the strategy default or
the regex from the `livecheck` block).
This refactors verbose code in the `Sparkle` strategy where we access
element text into a reusable `Xml#element_text` method, replacing
chained calls like `item.elements["title"]&.text&.strip&.presence`
with `Xml.element_text(item, "title")`.
`#element_text` is only used to retrieve the text of a child element
in the `Sparkle` strategy but it can also retrieve the text from the
provided element if the `child_path` argument is omitted (i.e.,
`Xml.element_text(item)`). This will allow us to also avoid similar
calls like `item.text.strip.presence` in the future.
We need to be able to replicate the `Sparkle` strategy's sorting
and filtering behavior in a related cask audit, so this extracts
the logic into reusable methods.
This also stores `item.minimum_system_version` as a `MacOSVersion`
object (instead of a string), so we can do proper version comparison
(instead of naive string comparison) wherever needed.
Historically, the `Sparkle` strategy's `Item` struct has only
included basic values from the appcast that are commonly useful.
Over time we've selectively added/surfaced more values as we've
encountered outliers that require use of different values in a
`strategy` block.
We now need to use `minimumSystemValue`, so this expands the `Item`
struct to include any appcast value that we could conceivably want
to use in the future. This will hopefully save us from having to make
more modifications to the struct (and related tests) before we can
use a previously-unused value in a `strategy` block.
- Fix cask info output being incorrect
- Improve some code referring to casks as formulae
- Move livecheck cask fixtures to not shadow existing names
- Adjust the cask tap symlinking logic to make handling outdated
shadowed casks significantly easier
- Fix various flaky tests caused by casks sharding logic
- Prefer longer paths when there's multiple formulae or casks in a tap
with the same name rather than always using the first
The `Bitbucket` strategy currently matches versions from tag
tarball links on a project's `downloads/?tab=tags` page. It appears
that Bitbucket now uses a hash as the filename on this page instead
of the tag name, so the existing regex no longer matches.
This adds an alternative regex to match versions from the tag name
element (e.g., `<td class="name">example-1.2.3</td>`), which will fix
version matching in this scenario.
This adds tests to cover all of the strategy outside of the
`#find_versions` method, which we don't currently test because it
involves a network request.
`GithubLatest` was updated to use parts of `GithubRelease` and this
works fine within `brew livecheck` (since we `require` all the
strategies) but we need to explicitly `require` `GithubRelease` in
the `GithubLatest` test file now. Without this, we encounter errors
in parts of `GithubLatest` where `GithubRelease` is referenced.
