Skip to content

Conversation

@script3r
Copy link
Owner

Refactor the directory scanner to a parallel producer-consumer architecture using ignore and rayon to dramatically improve performance.

Inspired by ripgrep, this change replaces the naive single-threaded walk with a parallel directory producer and concurrent file processing consumers. This architecture achieves ~4.37 GiB/s throughput by respecting .gitignore files, pre-filtering, and saturating CPU cores.


Open in Cursor Open in Web

@cursor
Copy link

cursor bot commented Sep 15, 2025

Cursor Agent can help with this pull request. Just @cursor in comments and I'll start working on changes in this branch.
Learn more about Cursor Agents

@script3r script3r marked this pull request as ready for review September 15, 2025 15:12
@script3r script3r merged commit f1cec94 into main Sep 15, 2025
4 checks passed
@script3r script3r deleted the cursor/parallelize-directory-scanning-and-file-processing-0935 branch September 21, 2025 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants