Skip to content

perf: eliminate N+1 queries in __make_cre_links and __get_all_nodes_and_cres#850

Open
shiwani42 wants to merge 1 commit intoOWASP:mainfrom
shiwani42:fix/n-plus-one-query-patterns
Open

perf: eliminate N+1 queries in __make_cre_links and __get_all_nodes_and_cres#850
shiwani42 wants to merge 1 commit intoOWASP:mainfrom
shiwani42:fix/n-plus-one-query-patterns

Conversation

@shiwani42
Copy link
Copy Markdown

Fixes #848

What changed

__make_cre_links

Replaced the per-link Node fetch with a single JOIN query. Previously, for a CRE with L links this method issued 1 + L queries. It now issues one query regardless of link count.

Before:

for link in self.session.query(Links).filter(Links.cre == cre.id).all():
    node = self.session.query(Node).filter(Node.id == link.node).first()

After:

rows = (
    self.session.query(Links, Node)
    .join(Node, Node.id == Links.node)
    .filter(Links.cre == cre.id)
    .all()
)
for link, node in rows:
    ...

__get_all_nodes_and_cres

Replaced the IDs-first fetch pattern with direct ORM object queries. For CREs, the previous code called get_cre_by_db_id per record, which added a SELECT external_id FROM cre WHERE id = ? round-trip before reaching get_CREs. Fetching full ORM objects and passing the already-known external_id directly cuts that out. The same fix is applied to the node branch.

Tests

All existing tests pass. The changed methods are exercised by test_get_CREs, test_export, test_get_root_cres, and test_get_cre_hierarchy.

…nd_cres

__make_cre_links was issuing a separate SELECT for each linked node —
one query to fetch all Links rows for a CRE, then one more per row to
fetch the corresponding Node. Replace with a single JOIN query so the
whole thing resolves in one round-trip regardless of how many links a
CRE has.

__get_all_nodes_and_cres was fetching IDs first (SELECT id FROM …)
and then re-querying each record individually. For CREs this chained
into get_cre_by_db_id, which added a further SELECT external_id per
CRE before finally calling get_CREs. Fetch full ORM objects directly
and call get_CREs(external_id=) using the already-known external_id,
cutting out that extra per-CRE round-trip. The same IDs-first pattern
is fixed for the node branch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

N+1 query patterns in __make_cre_links and __get_all_nodes_and_cres

1 participant