docs(factories): document the control room - #531
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
7eb63f1 to
19b9aa9
Compare
|
/oz-review |
|
I'm re-reviewing this pull request in response to a review request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
![oz-for-oss[bot]](https://avatars.githubusercontent.com/in/3450139?s=60&v=4){kind=small}
There was a problem hiding this comment.
Overview
This PR replaces the Factory control-room placeholder with a full docs page that orients readers across Dashboard, Activity, Runs, Agents, Automations, Code, Scorers, Benchmarks, and Settings. The changed file is documentation-only; I did not find security issues, and spec_context.md contains no approved spec commitments to compare against.
Concerns
- The page is broadly structured and linkable, but a few sentences should be tightened so readers can scan the page and understand cost/metric caveats without ambiguity.
Verdict
Found: 0 critical, 0 important, 3 suggestions
Approve with nits
Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
| --- | ||
|
|
||
| Factory control room documentation will land in a follow-up PR. | ||
| The control room is the web surface for one factory. It shows what your agents are working on, what they produced, its recorded attributed cost, and how the factory is configured. Select a factory in the sidebar to expand its pages. |
There was a problem hiding this comment.
💡 [SUGGESTION] Clarify the pronoun here; its recorded attributed cost reads as if it refers to the factory rather than the work.
| The control room is the web surface for one factory. It shows what your agents are working on, what they produced, its recorded attributed cost, and how the factory is configured. Select a factory in the sidebar to expand its pages. | |
| The control room is the web surface for one factory. It shows what your agents are working on, what they produced, the attributed cost recorded for that work, and how the factory is configured. Select a factory in the sidebar to expand its pages. |
|
|
||
| ### Open and stop a work item | ||
|
|
||
| Click a row to open the detail pane beside the list. The pane shows the work item title, what the agent is currently doing, the stage, its origin as a Slack, Linear, or Jira ticket chip with any external references, pull request artifacts, and a collapsible details section with environment, cost, and run time. Click the clock control for **Event history**, a reverse-chronological timeline of the runs behind the work item. Click the short identifier in the pane header to copy the full work item ID. |
There was a problem hiding this comment.
💡 [SUGGESTION] Split this into shorter sentences or bullets; it packs row opening, pane contents, event history, and ID copying into one dense paragraph, which makes the task flow hard to scan.
|
|
||
| **Dashboard** covers a date range you choose with the presets or the date picker. Three cards head the page: **Autonomy** (the share of merged PRs that reached merge with no human touch beyond an approval review), **Time saved** (merged-PR line changes converted to approximate engineer-hours), and **PR latency** (median time from run kickoff through PR, review, and merge). | ||
|
|
||
| Below them, **Cost per PR** headlines the median cost of the PRs opened in the range, with **By complexity** and **By size** views and an expander for the most expensive PRs by attributed cost. It is a lower-bound estimate that can omit usage and does not match billing; see [Measure and improve a factory](/factories/measure-and-improve/) for its limitations. **Pull requests** charts opened alongside merged, and **Run breakdown** charts total runs with a drill-down by agent type, status, source, root versus subruns, model, and harness. |
There was a problem hiding this comment.
💡 [SUGGESTION] Specify what usage the estimate can omit so readers do not interpret this as usage in general being omitted from the Dashboard.
| Below them, **Cost per PR** headlines the median cost of the PRs opened in the range, with **By complexity** and **By size** views and an expander for the most expensive PRs by attributed cost. It is a lower-bound estimate that can omit usage and does not match billing; see [Measure and improve a factory](/factories/measure-and-improve/) for its limitations. **Pull requests** charts opened alongside merged, and **Run breakdown** charts total runs with a drill-down by agent type, status, source, root versus subruns, model, and harness. | |
| Below them, **Cost per PR** headlines the median cost of the PRs opened in the range, with **By complexity** and **By size** views and an expander for the most expensive PRs by attributed cost. It is a lower-bound estimate that can omit some run usage and does not match billing; see [Measure and improve a factory](/factories/measure-and-improve/) for its limitations. **Pull requests** charts opened alongside merged, and **Run breakdown** charts total runs with a drill-down by agent type, status, source, root versus subruns, model, and harness. |
There was a problem hiding this comment.
Overview
This PR replaces the Factory control-room placeholder with a full docs page that orients readers across Dashboard, Activity, Runs, Agents, Automations, Code, Scorers, Benchmarks, and Settings. The changed file is documentation-only; I did not find security issues, and spec_context.md contains no approved spec commitments to compare against.
Concerns
- The page is broadly structured and linkable, but a few sentences should be tightened so readers can scan the page and understand cost/metric caveats without ambiguity.
Verdict
Found: 0 critical, 0 important, 3 suggestions
Approve with nits
Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
| --- | ||
|
|
||
| Factory control room documentation will land in a follow-up PR. | ||
| The control room is the web surface for one factory. It shows what your agents are working on, what they produced, its recorded attributed cost, and how the factory is configured. Select a factory in the sidebar to expand its pages. |
There was a problem hiding this comment.
💡 [SUGGESTION] Clarify the pronoun here; its recorded attributed cost reads as if it refers to the factory rather than the work.
| The control room is the web surface for one factory. It shows what your agents are working on, what they produced, its recorded attributed cost, and how the factory is configured. Select a factory in the sidebar to expand its pages. | |
| The control room is the web surface for one factory. It shows what your agents are working on, what they produced, the attributed cost recorded for that work, and how the factory is configured. Select a factory in the sidebar to expand its pages. |
|
|
||
| ### Open and stop a work item | ||
|
|
||
| Click a row to open the detail pane beside the list. The pane shows the work item title, what the agent is currently doing, the stage, its origin as a Slack, Linear, or Jira ticket chip with any external references, pull request artifacts, and a collapsible details section with environment, cost, and run time. Click the clock control for **Event history**, a reverse-chronological timeline of the runs behind the work item. Click the short identifier in the pane header to copy the full work item ID. |
There was a problem hiding this comment.
💡 [SUGGESTION] Split this into shorter sentences or bullets; it packs row opening, pane contents, event history, and ID copying into one dense paragraph, which makes the task flow hard to scan.
|
|
||
| **Dashboard** covers a date range you choose with the presets or the date picker. Three cards head the page: **Autonomy** (the share of merged PRs that reached merge with no human touch beyond an approval review), **Time saved** (merged-PR line changes converted to approximate engineer-hours), and **PR latency** (median time from run kickoff through PR, review, and merge). | ||
|
|
||
| Below them, **Cost per PR** headlines the median cost of the PRs opened in the range, with **By complexity** and **By size** views and an expander for the most expensive PRs by attributed cost. It is a lower-bound estimate that can omit usage and does not match billing; see [Measure and improve a factory](/factories/measure-and-improve/) for its limitations. **Pull requests** charts opened alongside merged, and **Run breakdown** charts total runs with a drill-down by agent type, status, source, root versus subruns, model, and harness. |
There was a problem hiding this comment.
💡 [SUGGESTION] Specify what usage the estimate can omit so readers do not interpret this as usage in general being omitted from the Dashboard.
| Below them, **Cost per PR** headlines the median cost of the PRs opened in the range, with **By complexity** and **By size** views and an expander for the most expensive PRs by attributed cost. It is a lower-bound estimate that can omit usage and does not match billing; see [Measure and improve a factory](/factories/measure-and-improve/) for its limitations. **Pull requests** charts opened alongside merged, and **Run breakdown** charts total runs with a drill-down by agent type, status, source, root versus subruns, model, and harness. | |
| Below them, **Cost per PR** headlines the median cost of the PRs opened in the range, with **By complexity** and **By size** views and an expander for the most expensive PRs by attributed cost. It is a lower-bound estimate that can omit some run usage and does not match billing; see [Measure and improve a factory](/factories/measure-and-improve/) for its limitations. **Pull requests** charts opened alongside merged, and **Run breakdown** charts total runs with a drill-down by agent type, status, source, root versus subruns, model, and harness. |
|
/oz-review |
|
I'm re-reviewing this pull request in response to a review request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
There was a problem hiding this comment.
Overview
This PR replaces the Factory control-room placeholder with a full task-oriented guide to the factory-level Dashboard, Activity, Runs, Agents, Automations, Code, Scorers, Benchmarks, and Settings surfaces. I found no blocking correctness, security, or spec-drift concerns in the attached diff.
Concerns
- No blocking concerns.
- Spec comparison was not applicable because the attached spec context says no approved or repository spec context was found.
- Security pass found no findings for this docs-only change.
Verdict
Found: 0 critical, 0 important, 2 suggestions
Approve with nits
Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
| --- | ||
|
|
||
| Factory control room documentation will land in a follow-up PR. | ||
| The control room is the web surface for one factory. It shows what your agents are working on, what they produced, its recorded attributed cost, and how the factory is configured. Select a factory in the sidebar to expand its pages. |
There was a problem hiding this comment.
💡 [SUGGESTION] Tighten the opening so the cost reference clearly belongs to the factory work rather than to the control room itself.
| The control room is the web surface for one factory. It shows what your agents are working on, what they produced, its recorded attributed cost, and how the factory is configured. Select a factory in the sidebar to expand its pages. | |
| The control room is the web surface for one factory. It shows what your agents are working on, what they produced, the recorded attributed cost of that work, and how the factory is configured. Select a factory in the sidebar to expand its pages. |
|
|
||
| **Scorers** defines how an LLM judge classifies completed conversations: rubric and labels, threshold, agent scope, sampling rate, judge model, and whether the scorer is active or paused. Each scorer has a **Self-improvement** section you enable or disable. | ||
|
|
||
| **Benchmarks** suites define an agent and its tasks, each with success criteria. The launch dialog chooses the harness, model, and runner configurations to compare, the scorers, and how many repetitions to run. Results include cost and quality with a per-task comparison; the same classification caveat as scorers applies, so read them alongside [Measure and improve a factory](/factories/measure-and-improve/). |
There was a problem hiding this comment.
💡 [SUGGESTION] Fix the subject noun so this reads cleanly.
| **Benchmarks** suites define an agent and its tasks, each with success criteria. The launch dialog chooses the harness, model, and runner configurations to compare, the scorers, and how many repetitions to run. Results include cost and quality with a per-task comparison; the same classification caveat as scorers applies, so read them alongside [Measure and improve a factory](/factories/measure-and-improve/). | |
| **Benchmark** suites define an agent and its tasks, each with success criteria. The launch dialog chooses the harness, model, and runner configurations to compare, the scorers, and how many repetitions to run. Results include cost and quality with a per-task comparison; the same classification caveat as scorers applies, so read them alongside [Measure and improve a factory](/factories/measure-and-improve/). |
19b9aa9 to
0c8ed9b
Compare
|
/oz-review |
|
I'm re-reviewing this pull request in response to a review request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
There was a problem hiding this comment.
Overview
This PR replaces the placeholder Factory control-room page with a task-oriented overview of the Dashboard, Activity, Runs, Agents, Automations, Code, Scorers, Benchmarks, and Settings surfaces. The new page is structurally sound for MDX, follows the docs style guide overall, and I did not find security issues or spec drift because no approved spec context was provided.
Concerns
- One sentence in the dashboard metrics section is ambiguous and should be tightened before or after merge.
Verdict
Found: 0 critical, 0 important, 1 suggestions
Approve with nits
Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
|
|
||
| **Dashboard** covers a date range you choose with the presets or the date picker. Three cards head the page: **Autonomy** (the share of merged PRs that reached merge with no human touch beyond an approval review), **Time saved** (merged-PR line changes converted to approximate engineer-hours), and **PR latency** (median time from run kickoff through PR, review, and merge). | ||
|
|
||
| Below them, **Cost per PR** headlines the median cost of the PRs opened in the range, with **By complexity** and **By size** views and an expander for the most expensive PRs by attributed cost. It is a lower-bound estimate that can omit some run usage and does not match billing; see [Measure and improve a factory](/factories/measure-and-improve/) for its limitations. **Pull requests** charts opened alongside merged, and **Run breakdown** charts total runs with a drill-down by agent type, status, source, root versus subruns, model, and harness. |
There was a problem hiding this comment.
💡 [SUGGESTION] Clarify that the Pull requests view compares opened and merged PR counts.
| Below them, **Cost per PR** headlines the median cost of the PRs opened in the range, with **By complexity** and **By size** views and an expander for the most expensive PRs by attributed cost. It is a lower-bound estimate that can omit some run usage and does not match billing; see [Measure and improve a factory](/factories/measure-and-improve/) for its limitations. **Pull requests** charts opened alongside merged, and **Run breakdown** charts total runs with a drill-down by agent type, status, source, root versus subruns, model, and harness. | |
| Below them, **Cost per PR** headlines the median cost of the PRs opened in the range, with **By complexity** and **By size** views and an expander for the most expensive PRs by attributed cost. It is a lower-bound estimate that can omit some run usage and does not match billing; see [Measure and improve a factory](/factories/measure-and-improve/) for its limitations. The **Pull requests** chart compares opened and merged PRs, and **Run breakdown** charts total runs with a drill-down by agent type, status, source, root versus subruns, model, and harness. |
0c8ed9b to
29be740
Compare
|
/oz-review |
|
I'm re-reviewing this pull request in response to a review request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
There was a problem hiding this comment.
Overview
This PR replaces the Factory control-room placeholder with a task-oriented documentation page covering the Dashboard, Activity, Runs, Agents, Automations, Code, Scorers, Benchmarks, and Settings surfaces. I reviewed the attached annotated diff against the docs style guidance, checked the changed prose for comment/test quality obligations, and found no security-relevant changes or approved spec commitments to validate.
Concerns
- None.
Verdict
Found: 0 critical, 0 important, 0 suggestions
Approve
Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
|
I'm starting a first review of this pull request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
There was a problem hiding this comment.
Overview
This PR replaces the Factory control-room placeholder with a task-oriented page covering Activity, Runs, Dashboard metrics, agent and automation management, Code, Scorers, Benchmarks, and Settings. No approved spec context was available, and the docs-only diff did not introduce security-relevant code or configuration.
Concerns
- The Activity section describes the Event history control by icon shape instead of leading with the control label.
- The benchmark paragraph references a scorer caveat before making that caveat explicit for readers.
Verdict
Found: 0 critical, 0 important, 2 suggestions
Approve with nits
Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
|
|
||
| Click a row to open the detail pane beside the list. The pane shows the work item title, current agent activity, stage, origin, external references, pull request artifacts, environment, cost, and run time. | ||
|
|
||
| Click the clock control for **Event history**, a reverse-chronological timeline of the runs behind the work item. Click the short identifier in the pane header to copy the full work item ID. |
There was a problem hiding this comment.
💡 [SUGGESTION] Use the control label instead of the icon shape so readers who do not see the icon can find the same action.
| Click the clock control for **Event history**, a reverse-chronological timeline of the runs behind the work item. Click the short identifier in the pane header to copy the full work item ID. | |
| Click **Event history** to open a reverse-chronological timeline of the runs behind the work item. Click the short identifier in the pane header to copy the full work item ID. |
|
|
||
| **Scorers** defines how an LLM judge classifies completed conversations: rubric and labels, threshold, agent scope, sampling rate, judge model, and whether the scorer is active or paused. Each scorer has a **Self-improvement** section you enable or disable. | ||
|
|
||
| **Benchmark** suites define an agent and its tasks, each with success criteria. The launch dialog chooses the harness, model, and runner configurations to compare, the scorers, and how many repetitions to run. Results include cost and quality with a per-task comparison; the same classification caveat as scorers applies, so read them alongside [Measure and improve a factory](/factories/measure-and-improve/). |
There was a problem hiding this comment.
💡 [SUGGESTION] The sentence references a scorer caveat that this page has not explained yet, so make the benchmark caveat explicit here.
| **Benchmark** suites define an agent and its tasks, each with success criteria. The launch dialog chooses the harness, model, and runner configurations to compare, the scorers, and how many repetitions to run. Results include cost and quality with a per-task comparison; the same classification caveat as scorers applies, so read them alongside [Measure and improve a factory](/factories/measure-and-improve/). | |
| Benchmark suites define an agent and its tasks, each with success criteria. The launch dialog chooses the harness, model, and runner configurations to compare, the scorers, and how many repetitions to run. Results include cost and quality with per-task comparisons. Because benchmark quality depends on scorer classifications, read results alongside [Measure and improve a factory](/factories/measure-and-improve/). |
29be740 to
67729fe
Compare
|
/oz-review |
|
You've used all 5 |
Co-Authored-By: Warp Agent <agent@warp.dev>
67729fe to
b713648
Compare
|
/oz-review |
|
You've used all 5 |
Editorial pass on the control room page for external readers:
- Replace internal jargon (web surface, source mode, terminal state,
polling interval, atomic commit) with plain language
- State Activity's default filters directly instead of "two defaults
shape what you see"
- Restructure dense comma lists into scannable bullets (dashboard
metrics, run pane actions, Code tab source modes, editor boundaries)
- Explain what the scorer Self-improvement toggle does, per the UI copy
- Fix inverted phrasing ("the automation editor overrides no execution
settings") and cut duplicated sidebar orientation
No factual changes; all labels and behavior stay as verified.
Co-Authored-By: Warp <agent@warp.dev>
Cut prose that describes what UI elements do rather than documenting behavior: search/filter/sort control walkthroughs, filter chip and URL state mechanics, pane field inventories, refresh cadence, and editor form-field enumerations. Kept the semantics readers can't infer from the UI: default Activity filters, metric definitions and cost caveats, the Stop task warning, source-mode boundaries, and read-only rules. Co-Authored-By: Warp <agent@warp.dev>
…vement, code tab - Replace the 'no way to steer' note: View session opens the run's shared agent session (cloud agent session sharing) for real-time follow-ups. - Add the Self-improvement page to the pages table and scorer section; drop the stale claim that the Dashboard lists self-improvement PRs. - Link the Code tab section to Factory definitions as code, align its source modes with the current UI (GitHub links out; live-managed), and document the in-room branch review (Request changes / Approve & merge). - Refresh details: harness is editable in the agent editor, Scorers are read-only for file-managed factories, work-item pane shows origin and View agent, Cost per PR expands into most expensive PRs. Co-Authored-By: Warp <agent@warp.dev>
peicodes
left a comment
peicodes
left a comment
There was a problem hiding this comment.
Minor comments and potential gotchas but overall looks good
|
|
||
| ## Pages at a glance | ||
|
|
||
| Select a factory in the sidebar to open its pages. **Runs**, **MCPs and apps**, and **Secrets** sit above the factory list and cover your whole team, not a single factory. |
There was a problem hiding this comment.
Integrations is also a top level page
There was a problem hiding this comment.
Added — Integrations is now listed with the team-level pages. (ccb560d)
| | **Self-improvement** | Pull requests filed to fix scorer-detected failures | Review fix PRs and open the runs behind them | | ||
| | **Benchmarks** | Benchmark suites and their runs | Compare harness, model, and runner configurations | | ||
| | **Code** | The factory's definition files | Browse and edit a Warp-managed definition | | ||
| | **Settings** | Configuration the factory owns | Change identity, repos, runners, and integrations | |
There was a problem hiding this comment.
Not sure how to word this but "integrations accessible by this factory" because there's a top-level integrations page
There was a problem hiding this comment.
Reworded to "the integrations the factory can access". (ccb560d)
|
|
||
| ## Track work items on Activity | ||
|
|
||
| **Activity** shows the factory's work items grouped by stage: Triage, Planning, Building, and Reviewing. |
There was a problem hiding this comment.
Maybe worth mentioning terminal stages like completed and cancelled (which are default filtered out)
There was a problem hiding this comment.
Added a sentence covering the Complete and Cancelled terminal stages; the next paragraph already explains they're filtered out by default and how to surface them. (ccb560d)
|
|
||
| ## Inspect runs | ||
|
|
||
| A run is a single agent execution. The team-level **Runs** page lists every run you have access to; a factory's **Runs** page lists only runs from that factory's agents. |
There was a problem hiding this comment.
Maybe worth clarifying how runs differ from tasks from the activity page
There was a problem hiding this comment.
Added a sentence contrasting the two: a work item tracks one piece of work through the stages and can span several runs, while a run is a single agent execution. (ccb560d)
| Run pages don't include a chat input, but you can still steer a run: **View session** opens its [shared agent session](/platform/viewing-cloud-agent-runs/), where you follow the agent in real time and send follow-up instructions while the run's environment is active. After the environment shuts down, the same button opens the conversation transcript. | ||
| ::: | ||
|
|
||
| ## Read dashboard metrics |
|
|
||
| ## Change factory settings | ||
|
|
||
| **Settings** holds the configuration the factory owns: the factory's name and connected repos, whether pull requests are authored by the agent or the run creator, the **Analysis model** self-improvement uses to analyze failed runs, runners, integrations, and deletion. Deleting a factory cannot be undone. |
There was a problem hiding this comment.
"integrations accessible by this factory"
There was a problem hiding this comment.
Done — now reads "the integrations accessible to this factory". (ccb560d)
- List Integrations among the team-level sidebar pages - Clarify Settings manages the integrations the factory can access - Mention the Complete and Cancelled terminal stages on Activity - Contrast runs with Activity work items Co-Authored-By: Warp <agent@warp.dev>
3 participants
Summary
Adds a task-oriented Factory control-room guide covering the working Dashboard, Activity, Runs, Agents, Automations, Code, Scorers, Benchmarks, and Settings surfaces. It documents current search/filter/stop behavior, metrics and Self-improvement surfaces, source-aware editing, runner/host controls, and the boundaries between observation, configuration, and repository policy.
Broken, placeholder, spec-only, and unwired surfaces are deliberately excluded.
Dependencies
hyc/factories-shared-ialists Control room under Operate after dependencies land.Branch-local link CI can report missing sibling pages until those dependencies merge; the complete integrated build is green.
Foundation
Shared navigation, route placeholders, Early Access badge support, and guide migrations are merged in #537. This PR now contains only its feature-owned files and passes CI independently.
Validation
npm run typecheck: 0 errorsnpm run build: 377 pages built2c864b0b8404Proposed reviewers
Based on the Warp Factories Soft Launch (August 18th) tracker. For planning only; no review requests have been sent.
@peicodes@harryalbert@LegobenScreenshots
Not included. No approved Factory control-room assets exist yet; the page uses one compact surface/action table and verified UI labels.
Unverified claims
None — documented routes, labels, controls, availability boundaries, metrics, and source modes were verified against current source.
Conversation: https://staging.warp.dev/conversation/5ff89820-2d80-4518-981e-178845029de1
Plans: https://staging.warp.dev/drive/notebook/7ZPKWz7hM5I59o4Gg2ptYi and https://staging.warp.dev/drive/notebook/DpRWhMQ0DLCajPPMggXw5e
Co-Authored-By: Warp Agent agent@warp.dev