Skip to content

gh-146256: Add --jsonl collector to the profiling.sampling#146257

Draft
maurycy wants to merge 13 commits intopython:mainfrom
maurycy:tachyon-ndjson-kolektor
Draft

gh-146256: Add --jsonl collector to the profiling.sampling#146257
maurycy wants to merge 13 commits intopython:mainfrom
maurycy:tachyon-ndjson-kolektor

Conversation

@maurycy
Copy link
Copy Markdown
Contributor

@maurycy maurycy commented Mar 21, 2026 •
edited
Loading

This PR adds --jsonl discussed in #146256.

It's a draft - it does not contain tests yet. The class is below 2**8 lines of code and does not touch existing profiling.sampling code, so I took a leap. I'm already very uncomfortable with creating a PR for an issue that wasn't discussed. I don't want to clutter it with tests, until the format is solidified.

For example:

sudo -E \
  uv run \
    --python /Users/maurycy/src/github.com/maurycy/cpython/python.exe \
      python \
        -m profiling.sampling \
           run \
             --jsonl \
             -o /tmp/profile.jsonl /tmp/hello_world.py

Where /tmp/hello_world.py could be:

import time
print("Hello, World!")
time.sleep(0.1)

Given the similarity, the class just inherits from StackTraceCollector. (I think this is a good hint that the collectors should be separate from renderers.)

Visual Studio Code Extension

For the purpose of demonstrating the --jsonl usefulness, I have vibe-coded (with Claude Code) a simple VSCode Extension (only that) that leverages this play profile and displays it in the editor:

image

I think that, once we have --stream it will be much more exciting.

Apart from headless profilers: updating the real-time hot spots from the production in VSCode, or, well, making agents' life easier.

You can fetch the vibe-coded VSCode Extension here (no guarantees):

Or:

mkdir -p ~/.vscode/extensions/profiling-heatmap                                                                                                                                                                                   
curl -sL 'https://gist.githubusercontent.com/maurycy/50a80586636a90216efc86065bbfd6de/raw/719e46f4f086c819edbbee7b20c45c41eae15e8b/extension.js' -o ~/.vscode/extensions/profiling-heatmap/extension.js                           
curl -sL 'https://gist.githubusercontent.com/maurycy/fa4c3acd9e8e681a609bddc9ad04c4ae/raw/1edd3c2716a4a9e55aba8ed94850fd70f61fa606/package.json' -o ~/.vscode/extensions/profiling-heatmap/package.json                           
echo "Restart VSCode, then: Cmd+Shift+P -> 'Profiling: Load JSONL Profile'"

Please do not forget about removing ~/.vscode/extensions/profiling-heatmap/ after tests.

@maurycy maurycy requested a review from pablogsal as a code owner March 21, 2026 15:59
@bedevere-app bedevere-app bot mentioned this pull request Mar 21, 2026
Open
@maurycy maurycy marked this pull request as draft March 21, 2026 16:00
@maurycy maurycy changed the title gh-146256: Add --ndjson flag to the profiling.sampling gh-146256: Add --jsonl flag to the profiling.sampling Mar 21, 2026
@maurycy maurycy changed the title gh-146256: Add --jsonl flag to the profiling.sampling gh-146256: Add --jsonl collector to the profiling.sampling Mar 23, 2026
maurycy added 8 commits March 30, 2026 19:22
@maurycy
Merge remote-tracking branch 'upstream/main' into tachyon-ndjson-kole…
6a0ea81 
…ktor

* upstream/main: (119 commits)
  pythongh-144270: Make SubElement parent and tag positional-only (pythonGH-144845)
  pythongh-146558: JIT optimize dict access for objects with known hash (python#146559)
  pythongh-139922: always run MSVC 64-bit tail-calling CI (pythonGH-146570)
  pythongh-126835: Fix _PY_IS_SMALL_INT() macro (python#146631)
  pythongh-146587: fix type slot assignment incase of multiple slots for same name (python#146593)
  pythongh-138122: Add differential flame graph (python#145785)
  pythongh-146416: Emscripten: Improve standard stream handling in node_entry.mjs (python#146417)
  pythongh-146444: Don't package as part of iOS 'build hosts' target (python#146628)
  pythongh-138850: Add --disable-epoll to configure (pythonGH-145768)
  pythongh-146444: Make Platforms/Apple/ compatible with Python 3.9 (python#146624)
  pythongh-138577: Fix keyboard shortcuts in getpass with echo_char (python#141597)
  pythongh-146556: Fix infinite loop in annotationlib.get_annotations() on circular __wrapped__ (python#146557)
  pythongh-146579: _zstd: Fix decompression options dict error message (python#146577)
  pythongh-146083: Upgrade bundled Expat to 2.7.5 (python#146085)
  pythongh-146080: fix a crash in SNI callbacks when the SSL object is gone (python#146573)
  pythongh-146090: fix memory management of internal `sqlite3` callback contexts (python#146569)
  pythongh-145876: Do not mask KeyErrors raised during dictionary unpacking in call (pythonGH-146472)
  pythongh-146004: fix test_args_from_interpreter_flags on windows (python#146580)
  pythongh-139003: Use frozenset for module level attributes in _pyrepl.utils (python#139004)
  pythonGH-146527: Add more data to GC statistics and add it to PyDebugOffsets (python#146532)
  ...
@maurycy
even a basic test
cb27fc0
@maurycy
separate func for end record
59cbb4a
@maurycy
proper name
25c6922
@maurycy
test_jsonl_collector_with_location_info
67cd39a
@maurycy
test synthetic frames
7c85d47
@maurycy
too many new lines
3eddae8
@maurycy
BUG? confusing... two ways to set skip_idle?
f71252e
@pablogsal
Copy link
Copy Markdown
Member

pablogsal commented Mar 31, 2026

@ivonastojanovic can you take a look?

@maurycy
Copy link
Copy Markdown
Contributor Author

maurycy commented Mar 31, 2026 •
edited
Loading

@ivonastojanovic @pablogsal Thank you.

Please note that I've started adding test coverage, so it might be worth waiting a day with a proper review (it's already interesting: confused myself with skip_idle), if you find the direction promising.

I will mark it as Ready for review immediately.

Perhaps #146256 and #145464 are the best places to discuss the format and the ideas.

maurycy added 2 commits March 31, 2026 12:25
@maurycy
Merge branch 'main' into tachyon-ndjson-kolektor
9836ffa 
* main:
  pythongh-145458: use `self.skip_idle` consistently in the tachyon profiler (python#145459)
  pythongh-146615: Fix format specifiers in Objects/ directory (pythonGH-146620)
  pythongh-146615: Fix format specifiers in Python/ directory (pythonGH-146619)
  pythongh-146615: Fix format specifiers in test cextensions (pythonGH-146618)
  pythongh-146615: Fix format specifiers in extension modules (pythonGH-146617)
  pythongh-146615: Fix crash in __get__() for METH_METHOD descriptors with invalid type argument (pythonGH-146634)
  pythongh-146376: Reduce timeout in Emscripten GHA workflow (python#146378)
  pythongh-146442: Fix various bugs in compiler pipeline (python#146443)
  pythongh-146238: Support half-floats in the array module (python#146242)
  pythongh-145056: Add support for merging collections.UserDict and frozendict (pythonGH-146465)
  pythongh-145056: Fix merging of collections.OrderedDict and frozendict (pythonGH-146466)
  pythongh-139633: Run netrc file permission check only once per parse (pythonGH-139634)
@maurycy
ok, thx b4fac15
c183109
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pablogsal pablogsal Awaiting requested review from pablogsal pablogsal is a code owner

Assignees

@ivonastojanovic ivonastojanovic

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@maurycy @pablogsal @ivonastojanovic