Improve test summary output for rspec and minitest
#361
Conversation
shawn-shellenbarger
changed the title
rspec-queue --report output
shawn-shellenbarger
force-pushed
the
ss-rspec-report
branch
2 times, most recently
from
25ad547 to
505a7cb
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR improves the readability of RSpec test failure reports by adding a summary section that lists failing test files at the top, followed by detailed error information. This addresses the problem where parsing failed tests in CI was difficult due to verbose output without a clear overview.
- Adds a "FAILED TESTS SUMMARY" section showing files with failure counts
- Reorganizes output to show summary first, then detailed errors
- Formats detailed errors with numbered sections and clear separators
Reviewed Changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| ruby/lib/rspec/queue.rb | Main implementation that adds summary formatting and reorganizes error output display |
| ruby/test/integration/rspec_redis_test.rb | Updates test expectations to match the new formatted output structure |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
ruby/lib/rspec/queue.rb
Outdated
| errors.each do |error_output| | ||
| # Look for the rspec rerun command (should always be present thanks to BuildStatusRecorder) | ||
| # The rspec command appears on its own line, possibly after whitespace | ||
| if match = error_output.match(/^\s*rspec\s+([^\s]+)(?::\d+)?/m) |
There was a problem hiding this comment.
There was a problem hiding this comment.
There was a problem hiding this comment.
This was being used to grab the file name to use in the toplevel summary, not to actually filter anything from the backtrace.
E.g.
================================================================================
FAILED TESTS SUMMARY:
================================================================================
./test/fixtures/rspec_examples/calculator_spec.rb (3 failures)
./test/fixtures/rspec_examples/product_spec.rb (5 failures)
./test/fixtures/rspec_examples/user_spec.rb (4 failures)
================================================================================
I refactored the parsing to follow what is being done for minitest instead of relying on pattern matching.
After looking into the backtrace cleaner, I did go ahead and add some specific filtering to our backtraces because they are often truncated.
Warning: This log has exceeded the 2 MB display limit. Below we are only showing the last 2 MB of output.
Here is an example build for reference and the patterns I chose to filter
|
|
||
| private | ||
|
|
||
| def colorized_rerun_command(example, colorizer=::RSpec::Core::Formatters::ConsoleCodes) |
There was a problem hiding this comment.
Moved into failure_formatter.rb
| @notification.fully_formatted(nil), | ||
| colorized_rerun_command(@notification.example) | ||
| ].join("\n") | ||
| end |
There was a problem hiding this comment.
Here is a breakdown from an existing build on where these are located
shawn-shellenbarger
force-pushed
the
ss-rspec-report
branch
2 times, most recently
from
f4d2fcd to
8e8337a
Compare
shawn-shellenbarger
changed the title
rspec-queue --report outputrspec and minitest
shawn-shellenbarger
force-pushed
the
ss-rspec-report
branch
from
683e949 to
ba04ffb
Compare
2 participants
The output of the test summaries through
minitest-queue reportandrspec-queue reportmakes it difficult to quickly parse which test files are failing in CI. These changes include a summary at the top of each report of which tests failed before dumping the entire report.I've also added backtrace filtering for
rspecto remove noisy filepaths that bloat backtraces.The examples below are for
rspecand are intended to highlight the difference in the structure of the output. The backtrace filters weren't applied when I generated these.Output before changes
Output after changes