Modularize orchestrator wiring: extract queue profiles, topic registry builder, and controller registration

[JamyDev]

Summary

Adds an RFC at doc/rfc/submitqueue/modular-queue-wiring.md proposing 5 extractions to make the orchestrator's queue setup more modular:

1. Promote queueExtensions/queueRegistry into submitqueue/core/queueprofile
2. Extract declarative topic-registry builder into submitqueue/core/topicregistry
3. Extract controller registration helpers into submitqueue/orchestrator/controller/wire
4. Add data-driven profile hints to QueueConfig (YAML-driven extension selection)
5. Simplify example orchestrator main.go from ~950 to ~300 lines

Also adds the RFC to doc/rfc/index.md. Doc-only change, no code modifications.

Test Plan

CI/Build validation

Revert Plan

git revert

API Changes

No API changes

Monitoring and Alerts

Standard monitoring via existing dashboards

🤖 Generated by minion
Co-Authored-By: minion-bot noreply@minion.uber.com
Co-Authored-By: jamy@uber.com
Agent: claude-code | Task: b069bea1-12a5-478a-b517-61ce0abd6749

View your task: https://minion.uberinternal.com/task/b069bea1-12a5-478a-b517-61ce0abd6749

AI Verification

Verified at 202618b on Jul 15 18:28 · 2 files analyzed · 791ms

Verifier	Status	Issues
ureview	completed	0

0 issues detected

_{Skipped verifiers: go-proto-lint, ios-lint, ios-test, uber-one, android-lint, android-coverage, java-lint, web, diff-template, go-lint, go-coverage, go-thrift-lint, java-coverage, claude · EngWiki}

Prior runs

Run at 0387fe7 on Jul 15 17:37 · 2 files · 913ms · 0 issues detected
Run at 3d8d47f on Jul 13 22:32 · 2 files · 1s · 0 issues detected
Run at fcb7447 on Jun 30 15:50 · 2 files · 1s · 0 issues detected

Issues

[behinddwalls]

maybe we call it Create? conflates with Build (CI)

[behinddwalls]

would it possible to run through an example for this...might make things a bit more clear..

I am chatting with Claude to see if we can land on some good sample

[behinddwalls]

Design counter-proposal: declare, don't assemble

Full end-to-end walkthrough (every construct defined, every file shown step by step): gist — declare, don't assemble. Summary below.

The test that drove it: could another repo import submitqueue as a library and build an fx / plain-main / test-harness service from it?

The core move: the RFC's extractions (and our first drafts of this feedback) still leave each service implementing assembly — four copies kept aligned by convention and review. The counter-proposal: each service declares itself as data, one shared engine assembles.

piece	owner	form
engine — construction, lifecycle, DLQ pairing, start/stop ordering	platform/pipeline.Construct + platform/lifecycle — written once, tested once	code
service definition — Deps (seams) · Stages (queue controllers, one row = controller + topic + DLQ reconciler) · Controllers (RPC, unbound to any wire contract)	each domain pkg	data + plain funcs
deployment policy — impl selection (profiles.go), creds, topic names, classifiers, transport, signals	each main (our service/ mains become reference hosts)	host code

Highlights, each shown concretely in the gist:

• Adding a stage = one table row. Today it takes 3 hand-coordinated edits (topic config, registration, registerDLQControllers); "every consumed stage has a DLQ pair + reconciler" becomes true by construction instead of by review — just in time for the speculation stages.
• One lifecycle.Component handle; the library exposes no server, reads no env, traps no signals. An fx host maps 1:1 (each block an fx.Provide, Construct one fx.Invoke, lifecycle two fx.Hook lines).
• Per-queue impl variation is the existing Factory contract ("identity in, resolve internally") — profiles stay host-private in profiles.go, so core/queueprofile stays trigger-gated exactly per the in-code TODO.
• Controllers are the portable Go surface (proto messages, no server interface — binding to a proto service/transport is ~10 lines of host glue), and the committed api/ protos are the cross-language surface: other languages regenerate stubs from the same sources, one contract, two codegens.
• Not the DI framework this RFC rightly rejects: no runtime graph, no reflection, no topo-sort — a typed engine over declarative data; one-offs enter via Options.

The problem statement and the QueueConfig A/B/C analysis survive nearly verbatim — this mostly changes the extraction plan's shape. Happy to pair on the rewrite.

[albertywu]

@JamyDev @behinddwalls can we have something fluent like this?

app, err := submitqueue.New().
    WithStorage(store).
    WithMessageQueue(mq).
    WithQueue(
        submitqueue.Queue("go-code").
            WithChangeProvider(changeProvider).
            WithBuildRunner(buildkiteBuildRunner).
            WithScorer(defaultScorer).
            WithConflictAnalyzer(tango),
    ).
    WithQueue(
        submitqueue.Queue("java-code").
            WithChangeProvider(changeProvider).
            WithBuildRunner(buildkiteBuildRunner).
            WithScorer(defaultScorer).
            WithConflictAnalyzer(tango),
    ).
    Build()
if err != nil {
    return fmt.Errorf("build SubmitQueue app: %w", err)
}

if err := app.Start(ctx); err != nil {
    return fmt.Errorf("start SubmitQueue app: %w", err)
}
defer app.Stop(context.Background())

return app.Wait(ctx)

or if some config needs to come from YAML for practical reasons (Uber infra):

cfg := loadDeploymentConfig()

app, err := submitqueue.New().
    WithStorage(newStorage(cfg.Database)).
    WithMessageQueue(newMessageQueue(cfg.MessageQueue)).
    WithQueue(newGoCodeQueue(cfg.GoCode)).
    WithQueue(newJavaCodeQueue(cfg.JavaCode)).
    Build()

func newGoCodeQueue(cfg MonorepoConfig) submitqueue.QueueBuilder {
    return submitqueue.Queue("go-code").
        WithBuildRunner(
            buildkite.New(buildkite.Config{
                Endpoint: cfg.BuildEndpoint,
                Timeout:  cfg.BuildTimeout,
            }),
        ).
        WithScorer(
            heuristic.New(goCodeScoringBuckets),
        ).
        WithConflictAnalyzer(
            targets.NewGoAnalyzer(cfg.Repository),
        ).
        WithPrioritizer(
            sticky.New(cfg.MaxBuilds),
        )
}

[JamyDev]

I think we can add something on top to allow for a fluent builder yea