Skip to content

osx-clang: work around Homebrew's clang lacking REG_ENHANCED#6138

Closed
dscho wants to merge 1 commit intomainfrom
fix-ci-osx-clang-and-reftables-jobs
Closed

osx-clang: work around Homebrew's clang lacking REG_ENHANCED#6138
dscho wants to merge 1 commit intomainfrom
fix-ci-osx-clang-and-reftables-jobs

Conversation

@dscho
Copy link
Copy Markdown
Member

@dscho dscho commented Mar 20, 2026

The osx-clang and osx-reftable CI jobs on macOS started failing with:

compat/regcomp_enhanced.c:7:13: error: use of undeclared identifier
'REG_ENHANCED'

The failure coincides with the GitHub Actions macos-14-arm64 runner image being updated from 20260302.0147 to 20260317.0174. The key change in that image update is the Homebrew version bump from 5.0.15 to 5.1.0.

Homebrew 5.1.0 introduced automatic linking for versioned keg-only formulae when the unversioned sibling is absent (see Homebrew/brew#21676, announced at https://brew.sh/2026/03/10/homebrew-5.1.0/). The runner image installs llvm@15 (keg-only) but not unversioned llvm. Under Homebrew 5.0.x that formula stayed in its keg and its clang binary only lived at $(brew --prefix llvm@15)/bin/clang. Under 5.1.0, because unversioned llvm is absent, llvm@15 is now auto-linked into /opt/homebrew/bin/, which sits earlier in PATH than /usr/bin.

The net effect is that CC=clang in CI now silently resolves to Homebrew's LLVM 15.0.7 clang instead of Apple's system clang (Apple clang 15.0.0, bundled with Xcode 15.4). The runner image README confirms this: the reported "Clang/LLVM" version flipped from 15.0.0 to 15.0.7 between image releases, matching the Homebrew LLVM version exactly.

Homebrew's LLVM clang uses different include paths from Apple's clang. In particular, the regex.h it sees does not define REG_ENHANCED, which is an Apple-specific extension present in the macOS SDK headers since at least macOS 10.12. The Makefile unconditionally sets USE_ENHANCED_BASIC_REGULAR_EXPRESSIONS for all Darwin builds via config.mak.uname, which pulls in compat/regcomp_enhanced.c, which references REG_ENHANCED, hence the build failure.

The osx-gcc job (CC=gcc-13) is unaffected because Homebrew GCC is configured to use Apple's SDK sysroot, so it still picks up Apple's regex.h which defines REG_ENHANCED. The osx-meson job is unaffected because Meson does a compile-time test for REG_ENHANCED (via compiler.get_define) and simply skips the feature when it is absent.

Work around this by setting NO_REGEX when CC=clang on Darwin, which makes the build use Git's bundled regex implementation instead of the system one. This sidesteps the missing REG_ENHANCED define entirely.

This corresponds to https://lore.kernel.org/git/d340af9e-334c-4e81-e58a-fc3dea73ebdd@gmx.de/, which I proposed to upstream. Historically, upstream's process is not amenable to quick fixes like that, and besides, we'd have to wait to get the fix until the next Git release, which is due April 20th, and I'm not willing to wait that long.

The `osx-clang` and `osx-reftable` CI jobs on macOS started failing
with:

    compat/regcomp_enhanced.c:7:13: error: use of undeclared identifier
    'REG_ENHANCED'

The failure coincides with the GitHub Actions `macos-14-arm64` runner
image being updated from `20260302.0147` to `20260317.0174`.  The key
change in that image update is the Homebrew version bump from 5.0.15 to
5.1.0.

Homebrew 5.1.0 introduced automatic linking for versioned keg-only
formulae when the unversioned sibling is absent (see
Homebrew/brew#21676, announced at
https://brew.sh/2026/03/10/homebrew-5.1.0/).  The runner image installs
`llvm@15` (keg-only) but not unversioned `llvm`.  Under Homebrew 5.0.x
that formula stayed in its keg and its `clang` binary only lived at
`$(brew --prefix llvm@15)/bin/clang`.  Under 5.1.0, because unversioned
`llvm` is absent, `llvm@15` is now auto-linked into
`/opt/homebrew/bin/`, which sits earlier in PATH than `/usr/bin`.

The net effect is that `CC=clang` in CI now silently resolves to
Homebrew's LLVM 15.0.7 clang instead of Apple's system clang (Apple
clang 15.0.0, bundled with Xcode 15.4).  The runner image README
confirms this: the reported "Clang/LLVM" version flipped from 15.0.0 to
15.0.7 between image releases, matching the Homebrew LLVM version
exactly.

Homebrew's LLVM clang uses different include paths from Apple's clang.
In particular, the `regex.h` it sees does not define `REG_ENHANCED`,
which is an Apple-specific extension present in the macOS SDK headers
since at least macOS 10.12.  The Makefile unconditionally sets
`USE_ENHANCED_BASIC_REGULAR_EXPRESSIONS` for all Darwin builds via
`config.mak.uname`, which pulls in `compat/regcomp_enhanced.c`, which
references `REG_ENHANCED`, hence the build failure.

The `osx-gcc` job (CC=gcc-13) is unaffected because Homebrew GCC is
configured to use Apple's SDK sysroot, so it still picks up Apple's
`regex.h` which defines `REG_ENHANCED`.  The `osx-meson` job is
unaffected because Meson does a compile-time test for `REG_ENHANCED`
(via `compiler.get_define`) and simply skips the feature when it is
absent.

Work around this by setting `NO_REGEX` when `CC=clang` on Darwin, which
makes the build use Git's bundled regex implementation instead of the
system one.  This sidesteps the missing `REG_ENHANCED` define entirely.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
@dscho dscho requested review from mjcheetham and rimrul March 20, 2026 15:16
@dscho dscho self-assigned this Mar 20, 2026
@dscho
Copy link
Copy Markdown
Member Author

dscho commented Mar 24, 2026

As per the analysis I sent to the Git mailing list:

This seems to no longer be necessary -- the latest workflow run on
master succeeded (https://github.com/git/git/actions/runs/23392330338).

Yeah, I saw that too and was wondering if the glitch was reverted or
something at the platform side.

It was reverted at the platform side. The runner-images team rolled back
the broken image. Georgy Puzakov (runner-images team member) wrote on
2026-03-20 in
actions/runner-images#13827 (comment):

"We have some issues with new image version(macOS 15). We made a
rollback version yesterday, can you check if the issue still
exists?"

And indeed, the broken macos-14-arm64/20260317.0174 release has been
deleted from https://github.com/actions/runner-images/releases -- the
latest published macos-14-arm64 release as of time of writing is once
again (the last known good one):
https://github.com/actions/runner-images/releases/tag/macos-14-arm64%2F20260302.0147

Did some kind soul fix the runners?

If so, thanks. It certainly was not me ;-)

Not me either, but I guess nobody earnestly thought that either of us to
fix GitHub's Action runners 😁 But I did dig into the root cause, so here
is a summary for the record.

The culprit is Homebrew 5.1.0 (released 2026-03-10, announced at
https://brew.sh/2026/03/10/homebrew-5.1.0/), which introduced automatic
linking for versioned keg-only formulae when the unversioned sibling is
absent (Homebrew/brew#21676). The runner image
installs llvm@15 (keg-only) but not unversioned llvm. Under Homebrew
5.1.0, llvm@15 got auto-linked into /opt/homebrew/bin/, and CC=clang
silently resolved to Homebrew's LLVM clang instead of Apple's
/usr/bin/clang. That LLVM clang does not see the macOS SDK headers, so
REG_ENHANCED is undefined and the build fails.

Interestingly, this auto-linking feature was actually reverted within
Homebrew itself (Homebrew/brew#21682) and then
re-landed with fixes (Homebrew/brew#21684), all
before the runner image was even built. So the re-landed version is what
shipped in the image.

Which means that the image rollback fixes things for now, but the
underlying Homebrew behavior still exists. The next runner image rebuild
that picks up Homebrew 5.1.0+ could reintroduce the problem unless the
runner-images team pins llvm@15 as keg-only or explicitly unlinks it.

In light of that, the workaround in
js/macos-homebrew-forgets-reg-enhanced may still be worth keeping as
insurance, even though the immediate breakage has been mitigated by the
image rollback. We would be one image refresh away from the same failure
otherwise.

With dedicated engineering tools like Merge Requests/Pull Requests, Git for Windows has the luxury of simply closing this here PR with the option of reopening it later if the need arises. The Git project, of course, deprives itself of project management tools by limiting its each and every tool to be the Git mailing list, be it for bug tracking, code contributions, code review, feature requests, users' pleas for help with their Git woes, etc, therefore it has to resort to awkward links to external tools (lore.kernel.org is not part of the Git project, really).

I'll gladly accept having the luxury to close this PR with the option of reopening it later on as needed.

@dscho dscho closed this Mar 24, 2026
@dscho dscho deleted the fix-ci-osx-clang-and-reftables-jobs branch March 24, 2026 06:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants