Rewrite the parser as hand-written recursive descent, removing goyacc - #35
Merged
Conversation
Lay out the architecture for replacing the goyacc-generated parser (parser.y, hintparser.y, and the goyacc toolchain) with hand-written recursive descent, following the same architecture as the sibling parsers zetajones, doubleclick, meyer, and teesql: oracle-generated golden corpus with todo metadata, next-test/check-parse dev loop, rule-attribution comments, differential testing, and fuzzing. The AST and public parser API are frozen; the goyacc parser at a pinned commit serves as the oracle so the new parser can be verified field-for-field against the trees produced today. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
The RD parser lexes on demand through a sliding window over the existing Scanner, parses statement lists with exact stmtText() semantics, and falls back to the goyacc parser for anything unimplemented. Under go test, every input the RD parser handles is re-parsed by goyacc and the two results compared field-for-field via the new internal/dump renderer, turning the whole existing suite into a differential corpus. Also: keyword-class tables extracted from parser.y (the future single source of truth), MARINO_RD_LOG fallback logging with cmd/next-test to rank unimplemented constructs, and internal/panics to cope with the generated token constants shadowing the any/recover builtins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
Ports the Expression/BoolPri/PredicateExpr/BitExpr/SimpleExpr grammar layers with the %left ladder as precedence climbing, all literal forms (string adjacency, charset introducers, hex/bit), identifiers and qualified generic calls, system/user variables, CASE/CAST/CONVERT, INTERVAL date arithmetic (with the LALR-style speculation), the keyword and non-keyword builtin function forms, aggregates with windowing clauses, window function calls, and frame specs. Expression origin offsets replicate the generated parser's per-reduction yySetOffset stamping. DO statements now route through the RD parser, putting expressions under the differential harness. The dump renderer reads OriginalText instead of Text() so rendering never memoizes converted text onto the tree the suite deep-compares. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
SELECT with all clause forms (options with merge semantics, field lists
with offset/text bookkeeping, FROM/joins with the LALR shift-greedy
cross-join rebalancing via ast.NewCrossJoin, index hints, partitions,
AS OF, TABLESAMPLE, LATERAL), set operations with the SetOprClauseList
folding and last-field text fixups, CTEs, TABLE/VALUES statements,
locking clauses, INTO OUTFILE, window definitions, and real SubSelect
parsing with statement-text spans.
The '(' ambiguities mirror the LALR resolutions: expression atoms and IN
predicates prefer the parenthesized-expression derivation and fall back
to the subquery one; table factors and statements resolve by scanning to
the first non-'(' token.
The differential dump no longer renders OriginTextPosition: the goyacc
runtime restamps it from stale expression values in reused parser-stack
slots (empty reductions point yyVAL one past the top), so its value
depends on stack layout rather than the statement. The RD parser stamps
deterministic production-start offsets — the value the explicit test
assertions expect — and those assertions remain the gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
INSERT/REPLACE (column lists, VALUES with row aliases, SET form, query forms with the braced-subquery action), UPDATE (single-table with ORDER BY/LIMIT vs multi-table), DELETE (single-table, USING, and tables-before-FROM multi-table forms with the TableNameOptWild disambiguation), LOAD DATA with all clauses, IMPORT INTO (file and SELECT sources), and the BATCH non-transactional wrapper. WITH-prefixed statements now dispatch to SELECT/UPDATE/DELETE after the clause. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
CLAUDE.md carries the family-convention dev guide, PLAN.md's testing section now reflects the in-process differential harness as built, and internal/rdconventions.md records the porting conventions for the remaining grammar families. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
GrantStmt/GrantProxyStmt/GrantRoleStmt and RevokeStmt/RevokeRoleStmt with the shared RoleOrPrivElemList parse and the ON/TO/FROM disambiguation matching the LALR resolution, all PrivType alternatives (including the MariaDB-gated BINLOG MONITOR error), privilege levels, REQUIRE clauses, user specs with auth options, and the REVOKE ALL PRIVILEGES, GRANT OPTION special case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
Ports every production of hintparser.y: all TableOptimizerHintOpt alternatives, LEADING with nested lists, READ_FROM_STORAGE, QB_NAME (both forms), SET_VAR, MEMORY_QUOTA with the overflow-warning path, TIME_RANGE, and the hint identifier/value helpers. Error behavior is byte-identical: hintparser.y has no error productions, so the first syntax error appends one Errorf and aborts, while unrecognized-hint warnings come from ordinary nil-hint productions. The RD statement parser now parses hints with rdParseHint while the goyacc oracle keeps the old hint parser, so the in-process differential cross-checks hint ASTs inside every hinted statement. A dedicated corpus + fixed-seed random differential test covers the hint parser directly. ParseHint stays on goyacc until removal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
Work-in-progress snapshot of the column-type system and the SET/SHOW/transaction/misc statement families; the tree builds and the full parser suite (including the rd/yacc differential) passes at this point. Completion commits follow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
SET in all its forms (variable assignments with scope handling and the ON/BINARY special-value guards, PASSWORD, TRANSACTION characteristics, CONFIG, SESSION_STATES, RESOURCE GROUP, ROLE and DEFAULT ROLE), the complete SHOW statement family, transactions (BEGIN/START TRANSACTION variants, COMMIT/ROLLBACK with completion types, savepoints), and USE/KILL/TRUNCATE/FLUSH/HELP/SHUTDOWN/RESTART/LOCK-UNLOCK TABLES/ BINLOG/PREPARE/EXECUTE/DEALLOCATE/EXPLAIN-DESC/TRACE with their exact grammar actions. SET BINDING, LOCK/UNLOCK STATS, and EXPLAIN/TRACE of still-unported statements fall back to goyacc deliberately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
Work-in-progress snapshot from the CREATE TABLE port; the tree builds and the full parser suite (including the rd/yacc differential) passes at this point. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
The full column type system (numeric with mode-dependent REAL/DOUBLE handling, string types in all spellings, ENUM/SET with binary-literal element flen math, blob/text, date-time, VECTOR), column definitions with every ColumnOption (defaults with the parenthesized-value algebra, generated columns, AUTO_RANDOM, references), constraints (primary/ unique/fulltext/foreign key/check, vector and columnar indexes, index options with exact merge semantics), and CREATE TABLE end to end: LIKE form, AS SELECT forms, every TableOption, and the complete partition grammar with its validation aborts. Warning-emitting actions force-lex the lookahead first (goyacc lexes its one-token lookahead before any reduce), keeping scanner positions inside warning strings byte-identical for the differential. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
CREATE DATABASE/INDEX/VIEW/USER/ROLE/SEQUENCE/STATISTICS/PLACEMENT POLICY/MASKING POLICY/RESOURCE GROUP (with the CREATE OR REPLACE disambiguation), every DROP form including DROP PREPARE and the HYPO index variant with the lock/algorithm validation quirks, RENAME TABLE/USER, FLASHBACK to timestamp/TSO in all six forms, and RECOVER TABLE. CREATE/DROP BINDING and procedures deliberately still fall back to goyacc for a later pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
AlterTableStmt in all seven alternatives and every AlterTableSpec: add/drop/modify/change columns, constraints (including vector and columnar indexes), all partition operations (add/drop/truncate/ coalesce/reorganize/exchange/rebuild/import/discard and the LESS THAN forms with their node-text spans), rename variants, charset conversion, lock/algorithm clauses, TiFlash replicas, attributes and stats options, plus ALTER DATABASE/SEQUENCE/INSTANCE/RANGE/POLICY/RESOURCE GROUP/USER. LALR subtleties reproduced and documented in place: the spec-level table-option list ends at commas (%prec higherThanComma) unlike CREATE, greedy comma consumption in partition-name and alter-order lists, the ALTER DATABASE no-name alternative excluding CHARSET/ENCRYPTION, and the MASKING/STATS_EXTENDED shift-over-identifier decisions. EXPLAIN of ALTER TABLE is wired through ExplainableStmt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
The full procedure grammar: parameters with modes, BEGIN/END blocks, DECLARE for variables/conditions/cursors/handlers with the complete handler-condition set, IF/CASE/WHILE/REPEAT, labeled statements with the label-mismatch action, OPEN/FETCH/CLOSE, CALL with expr-origin stamping, and DROP PROCEDURE — with the body and parameter source-text captures transcribed exactly. Bindings: CREATE/DROP/SET BINDING in all forms (FOR/USING, plan digests, FROM HISTORY) with their statement source-text slicing, wired into the CREATE/DROP/SET dispatchers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
ADMIN in every alternative (DDL job control with alter-job options, checks/checksums, index recovery/cleanup, binding admin, REPAIR TABLE, BDR roles, workload snapshots), ANALYZE TABLE in all ten forms with option validation, BACKUP/RESTORE with the full BRIE option set and stream/point-in-time forms, SPLIT/DISTRIBUTE region statements, CANCEL jobs, CALIBRATE RESOURCE, RECOMMEND INDEX, QUERY WATCH, TRAFFIC capture/replay, PLAN REPLAYER (with its statement source-text capture), REFRESH/LOAD/LOCK/UNLOCK STATS, and the SHOW-led BRIE/traffic forms. ANALYZE is now wired through procedure bodies and TRACE as well. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
The RD parser can now produce final errors with goyacc's exact semantics: position-only syntax errors formatted like Scanner.Errorf from the offending lookahead token's recorded position, action errors at reduce-time lookahead positions (actionErrorf), and farthest-failure tracking so backtracking reports the deepest token a viable parse reached, the way the single-pass automaton does. A 782-entry error corpus (testdata/errors.json, byte-exact fields) records every invalid input the suite exercises with goyacc's error and warning strings; TestRDErrorFidelity replays them through the RD error mode and now passes with zero mismatches. Detection-point fixes along the way: greedy '.'-qualified table names, NotSym shifting before non-predicate tokens, DEFAULT builtin-function shifting, LOAD DATA IGNORE...LINES, SHOW TABLE PARTITION, the CREATE dispatcher error position, and partition Validate deferring to FOLLOW-set syntax errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
Deletes parser.y (17.7k lines), hintparser.y, the generated parser.go and hintparser.go (55k lines), the vendored goyacc tool, and the genkeyword generator. Token constants, the lexer value types, and the yyLexer interface are snapshotted verbatim into token_kinds.go and hint_token_kinds.go, hand-maintained from here on. keywords.go stops being generated; TestKeywordConsistent now keeps it, the lexer's tokenMap, and the identifier keyword classes in exact three-way agreement, replacing the parser.y-scraping test. ParseSQL and ParseHint route only through the hand-written parser, with error semantics validated byte-for-byte against the 782-entry corpus recorded from goyacc before removal. Coverage gaps the silent fallback had been masking are fixed: OPTIMIZE TABLE, SQL_TSI_* time units, TIMESTAMPADD/DIFF as identifiers, TABLE/VALUES statement kinds in subquery position, and derived tables inside parenthesized join lists. The Parser reuses its scan buffer and token window across calls (as the goyacc parser reused its symbol stack); benchmarks against the pre-rewrite parser: SysbenchSelect 5972→4864 ns/op, ParseSimple 17741→14060 ns/op, ParseComplex 237400→209091 ns/op, with fewer allocations per parse across the board. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the goyacc-generated LALR parser (
parser.y, 17.7k lines / 713 productions, plushintparser.y) with a hand-written recursive-descent parser, following the architecture of sqlc's other parsers (zetajones, doubleclick, meyer, teesql). goyacc, the grammar files, the generated parsers, and the generator tooling are all removed.Contracts preserved
astpackage is untouched; every statement produces the same node graph as before.Parser,Scanner,New/Reset/ParseSQL/Parse/ParseOneStmt, the digester, andParseHintkeep their signatures and behavior.line N column M near "..."format is reproduced exactly, including action-error positions (reduce-time lookahead) and farthest-failure reporting. A recorded corpus of 782 invalid inputs with the goyacc parser's exact error and warning strings (parser/testdata/errors.json, captured before the removal) is replayed byTestRDErrorFidelity: 782/782 match.OriginTextPositionis now the deterministic production-start offset. goyacc restamped it from stale parser-stack slots (empty reductions read one past the stack top), producing context-dependent values; no test relied on those. Recorded in PLAN.md.How the rewrite was done
The migration ran statement family by statement family behind an in-process differential harness: while goyacc was still in-tree, every input handled by the RD parser was also parsed by goyacc and the two ASTs compared field-for-field with a reflective renderer (
internal/dump). LALR conflict resolutions (shift-greedy cross joins, paren disambiguation between expressions and subqueries,NOTshifting, partition-option FOLLOW sets) were replicated deliberately and are commented at the sites that implement them. The harness and fallback were removed in the final milestone once error fidelity was established.Architecture
parser/rd_parser.go— streaming token window over the existing lexer (O(1) memory, mark/rewind speculation), statement dispatch with one owner per leading token, and the historical statement-text bookkeeping.parser/parse_*.go— one file per statement family (~25 files); every nontrivial function names the grammar production it implements.parser/parse_expr.go/parse_func.go— the expression precedence ladder and atoms.parser/parse_hint.go— the optimizer-hint sub-parser, replacinghintparser.y.parser/rd_errors.go— error construction reproducing the goyacc format.parser/token_kinds.go/hint_token_kinds.go— token constants and lexer value types, snapshotted from the generated code and hand-maintained now;TestKeywordConsistentkeeps the keyword tables in exact agreement.Results
go test ./... -race), including the restore/roundtrip suites and the 512KB single-INSERT memory-budget test.PLAN.md records the milestone plan and the decisions that survive it; CLAUDE.md describes the new layout for future contributors.
🤖 Generated with Claude Code
https://claude.ai/code/session_01GXutbhBiY6MZWbdXRrM7CK
Generated by Claude Code