Switch pipelining metadata action to hollow rlib (-Zno-codegen)#3870
Switch pipelining metadata action to hollow rlib (-Zno-codegen)#3870walter-zeromatter wants to merge 1 commit intobazelbuild:mainfrom
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
|
Hey @walter-zeromatter, we'll need a signed CLA to proceed with the review, could you let us know if that works for you? |
scentini
added
the
awaiting-response
|
It's Walter's work. We just merged it to my fork a while ago so the world can benefit from it via rules_rs. |
|
Thanks for the clarification! In that case we only need to wait for the CLA. |
|
Sorry, I've been waiting on our legal. Just got approval, should have the cla signed later today |
|
@scentini Sorry about the delay, should be all good now. |
walter-zeromatter
force-pushed
the
user/wgray/pipelining
branch
from
349e91c to
841c962
Compare
|
Apologies - This code was written some time ago while I was doing some early experimenting with LLMs, and while I do think replacing the -emit=metadata,link + kill rustc approach with -Zno-codegen, A number of the claimed benefits here simply aren't real - I've since learned better, and will be overhauling this PR to something more appropriate. Please hold. |
walter-zeromatter
force-pushed
the
user/wgray/pipelining
branch
from
841c962 to
b55be1b
Compare
|
@scentini This is now a proper fix. Sorry for prior confusion. I do think this is the right approach, and I'll be making a few follow-up PRs to adopt a few other improvements from buck2's rust rules including lightweight metadata for check passes & hopefully some kind of lint check people can run to identify if they have non-deterministic proc macro dependencies. |
walter-zeromatter
force-pushed
the
user/wgray/pipelining
branch
2 times, most recently
from
3e65472 to
e470d22
Compare
Replace the process wrapper's rmeta-interception approach (kill rustc after metadata emission) with rustc's -Zno-codegen flag, which produces a hollow rlib containing metadata and MIR but no object code. This is the same approach used by Buck2. Key changes: - Metadata action uses `rustc -Zno-codegen --emit=link=<path>` to produce a hollow rlib (_meta.rlib) instead of raw .rmeta files - Remove --rustc-quit-on-rmeta flag, LineOutput::Terminate, and all associated kill logic from the process wrapper - Full action emits only --emit=link (no longer includes metadata) - Set RUSTC_BOOTSTRAP=1 on both metadata and full actions for SVH compatibility (required for the unstable -Zno-codegen flag)
walter-zeromatter
force-pushed
the
user/wgray/pipelining
branch
from
e470d22 to
63ca376
Compare
3 participants
Replace the --rustc-quit-on-rmeta / .rmeta approach with Buck2-style hollow rlibs: the RustcMetadata action runs rustc to completion with -Zno-codegen, emitting a .rlib archive. This approach mirrors the one used by buck2 and avoids needing to kill rustc mid-output in order to produce metadata.
While not fixing problems with SVH mismatches when non-determinism, this does simplify the codepath and uses a production tested technique that doesn't have any of the dangers associated with killing the rustc process while it's still active.
Previous work on this branch & the claimed prevention of SVH mismatches proved to be from changes which effectively removed all the actual benefits of the pipelined compilation