Replace async service with anyio service#973
Conversation
|
@parth-soni07 : Thank you so much for sharing this PR. Appreciate your great efforts and contribution. We will jointly do a detailed review. Will share feedback soon. Wish if you could resolve the merge conflicts. Once resolved, please ping me. I will re-run the CI/CD pipeline and share test results soon. Also, CCing @acul71, @sumanjeet0012 and @pacrob. |
|
@parth-soni07 I have reviewed the PR and have a few observations: |
Hey @sumanjeet0012 , thank you for your insights, yes indeed this is supposed to be a anyio migration and not trio, I have made some changes in the implementation to align with the name of the PR and use anyio. I am facing some issues in the version of anyio that is used in the p2pclient so currently working on that, will definitely ping once I am done with that. |
|
Hello @parth-soni07 thanks for this PR. AI Pull Request Review: PR #973PR Title: Replace async service with aniyo service 1. Summary of ChangesThis PR implements a major refactoring that replaces the legacy Understanding the Service Framework EcosystemTo understand the significance of this migration, it's important to understand the role of service frameworks in py-libp2p and the broader async Python ecosystem: async_service (Legacy - Being Replaced)The original
However, the upstream library became unmaintained, leaving py-libp2p with a frozen codebase that lacked modern async patterns, bug fixes, and improvements. The code was tightly coupled to Trio's async primitives (nurseries, cancellation scopes) and didn't leverage newer structured concurrency patterns. Trio (The Async Runtime)Trio is a modern, production-grade async framework for Python that emphasizes:
py-libp2p has been built on Trio from the beginning because its structured concurrency model aligns perfectly with libp2p's needs:
All core py-libp2p services (Swarm, PubSub, KadDHT, Circuit Relay) inherit from the AnyIO (The Compatibility Layer)AnyIO is a compatibility library that provides a unified async API that works with multiple backends:
AnyIO's key benefits:
anyio_service (The New Implementation)The new
The Relationship in py-libp2pIn py-libp2p's architecture: Current State (After this PR):
Why This Matters:
What Changed:
Breaking Changes:
2. Strengths
3. Issues FoundCritical
Major
Minor
4. Security ReviewSecurity Impact: None / Low No security vulnerabilities identified. The changes are primarily architectural refactoring:
Potential Considerations:
5. Documentation and ExamplesStrengths:
Issues:
6. Newsfragment RequirementRequired Action:File: Content should be: Replaced legacy async_service implementation with modern anyio_service framework built on AnyIO for better structured concurrency, cross-platform compatibility, and improved maintainability.Requirements:
Type Selection:
Impact: PR cannot be approved without this newsfragment file. This is a mandatory requirement, not optional. 7. Tests and ValidationTest Coverage:New Tests Added:
Test Updates:
Issues:
Build and Linting:Status: ❌ FAILED - Both Results Summary:✅ Passed:
❌ Failed:
Type Checking Errors Found:The type checkers (mypy and pyrefly) are reporting multiple issues with the AnyIO integration:
Root Cause Analysis:
Recommendation:
Action Required: These type checking errors must be resolved before merge, as they may indicate actual runtime issues or will cause CI to fail. Runtime Import Errors (CRITICAL):Status: ❌ FAILED - Error: Impact:
Root Cause: from anyio.abc import ObjectReceiveStream, ObjectSendStreamThese classes either:
Files Affected:
Action Required:
8. Recommendations for Improvement
9. Questions for the Author
10. Overall AssessmentQuality Rating: Needs WorkThe PR represents a well-structured migration with comprehensive code updates, good modular design, and maintained backward compatibility. However:
The architectural design is sound, but the implementation has critical issues that must be resolved before merge. Security Impact: None / LowNo security concerns identified. The refactoring improves error handling and resource management. Merge Readiness: Needs fixesBlockers:
Before Merge:
After Merge Considerations:
Confidence: LowThe code changes are well-structured and comprehensive architecturally, but:
The import errors are a critical blocker - the code cannot be tested or used until these are fixed. This suggests the AnyIO integration may not have been fully tested or the API usage is incorrect for the AnyIO version being used. Additional Context from Discussion #1008The related discussion #1008 highlights a dependency conflict with
This context is important for understanding the dependency management strategy and ensuring the anyio version constraint works across all dependencies. Review completed: 2025-11-13
Next Steps: Author should address blockers, particularly the critical import errors (highest priority) and missing newsfragment, before requesting re-review. |
- Introduced hashlib for SHA-256 hashing in routing_table.py, utils.py, and transport.py. - Updated peer ID generation in id.py to use SHA-256 for larger keys and identity multihash for smaller keys. - Adjusted tests in test_peerid.py to reflect the new hashing approach. - Added missing import statements for hashlib where necessary.
Fix/replace async with anyio
…anyio_manager_stats.py
|
@pacrob, the tests are still hanging. We can skip this PR for the upcoming release. |
|
@sumanjeet0012 : Thank you so much for updating the PR. Wish if you could resolve the merge conflicts. We are planning to have another release in the next 2 weeks. Wish if we could arrive at a good conclusion on this PR. |
…e by keeping async_service removal Made-with: Cursor
Made-with: Cursor
|
Recent commits (merge resolution and cleanup) Two commits were added to this branch:
|
…ent 524 - Parametrize test_anyio_manager_stats with anyio_backend=trio to avoid asyncio/Trio conflict in same worker (was causing 1200s timeout on py310/Windows). - Add newsfragments/524.feature.rst for async_service -> anyio_service replacement. Made-with: Cursor
Replace lambda returning coroutine with proper async callable so root task count and test timing are correct; avoids timeout and wrong assertion. Made-with: Cursor
|
CI fix: test_anyio_manager_stats
Change: Replaced with |
…ssues - Add _anyio_manager_stats_runner.py: standalone script run via subprocess with a single anyio.run(..., backend='trio'), avoiding nested trio.run() when the same pytest-xdist worker ran other trio tests. - Replace inline async tests with sync tests that subprocess the runner; fixes 1200s timeout and 'Attempted to call run() from inside a run()' in CI. Made-with: Cursor
|
Hello @seetadev @pacrob @sumanjeet0012 @yashksaini-coder |
… 2s timeout - test_anyio_manager_stats: subprocess timeout 60->120s, runner fail_after(90), relax root_tasks assert to >=1 for timing-dependent AnyIOManager (fixes Windows/CI timeout and assertion). - test_trio_based_service: do_service_lifecycle_check 0.1s->2s timeout so wait_started/wait_finished pass on Windows CI (fixes TooSlowError). Made-with: Cursor
|
CI: latest commit (98fa178)
No new time-driven sleeps were added; tests remain event-driven (only timeout ceilings were relaxed). |
|
Hi @seetadev , from my point of view this PR is ready to be merged. |
|
Hello @parth-soni07 , I've made some mod to this PR, please review this PR and tell if it's ok |
Thanks @acul71, checking, also CCing @sumanjeet0012 |
|
@acul71 : Great, we will merge this PR very soon. This is super awesome. Wish to have a thumbs up from @sumanjeet0012 , @pacrob and you too before we merge. @parth-soni07 and @yashksaini-coder : Appreciate your contribution and team work with @sumanjeet0012 |
Hey @acul71, thanks for the mods, All changes look good to me & all tests are passing. |
|
@seetadev LGTM |
1 similar comment
|
@seetadev LGTM |
* Replace async service with aniyo service * Modularized the code to match the async service pattern * Minor changes to fix CI/CD * Minor changes to conf.py * revert conf.py to prev * Removed async service * Fixed the docs based CI/CD errors * Fixed the docs based CI/CD errors * Fixed the imports to now use anyio instead of trio * minor version changes * Skip obsolete dependency * Updated the p2pclient version * Added constraints file for anyio version issue * Updated the tox.in for tox tests * Updated readthedocs * Updated readthedocs * Reverted to prev * fix pyprojec.toml * Update to latest main branch commit * lint error fix * chore: Replace async to anyio * refactor: Update hashing implementation across DHT and peer ID modules - Introduced hashlib for SHA-256 hashing in routing_table.py, utils.py, and transport.py. - Updated peer ID generation in id.py to use SHA-256 for larger keys and identity multihash for smaller keys. - Adjusted tests in test_peerid.py to reflect the new hashing approach. - Added missing import statements for hashlib where necessary. * Refactor imports to replace async_service with anyio_service across multiple files and fix lint issues * Refactor FunctionTask to use anyio.open_cancel_scope for better cancellation handling * Update p2pclient dependency to use git URL for anyio dependency issue * Update p2pclient dependency to use git URL for anyio version conflict * Refactor to replace anyio.Event with anyio.create_event and update type hints for better clarity and compatibility * changed anyio version for compatibility * reverted changes pyproject.toml file * Add anyio dependency with version constraints for compatibility * updated codes for anyio v4 * Exclude TrioManager from libp2p.tools.anyio_service module documentation * Trigger CI CD * Updated p2pclient reference for py-multihash * resolve conflicts * Update p2pclient dependency to version 0.2.1 in dev and test requirements * length error fixed * fixed import path * remove constraits file * Refactor: Remove skip markers for AnyIO tests, and done some minor changes * Trigger CI * Refactor: Enhance exception handling and task management in AnyIOManager; add tests for service functionality * Refactor: Clean up whitespace and improve formatting in AnyIOManager and related tests * chore: Remove unnecessary development dependencies from pyproject.toml * refactor: Remove test file for AnyIOManager service * refactor: Replace AnyIOManager with background_anyio_service in stats tests * refactor: Remove unused AnyIOManager import from test_anyio_manager_stats.py * refactor: Replace AnyIOManager with background_anyio_service in test_anyio_manager_stats.py * Update logger names from async_service to anyio_service Made-with: Cursor * Fix CI: use trio backend for anyio_manager_stats tests; add newsfragment 524 - Parametrize test_anyio_manager_stats with anyio_backend=trio to avoid asyncio/Trio conflict in same worker (was causing 1200s timeout on py310/Windows). - Add newsfragments/524.feature.rst for async_service -> anyio_service replacement. Made-with: Cursor * Fix test_anyio_manager_stats: use anyio.sleep_forever instead of lambda Replace lambda returning coroutine with proper async callable so root task count and test timing are correct; avoids timeout and wrong assertion. Made-with: Cursor * CI: run anyio_manager_stats tests in subprocess to avoid xdist/trio issues - Add _anyio_manager_stats_runner.py: standalone script run via subprocess with a single anyio.run(..., backend='trio'), avoiding nested trio.run() when the same pytest-xdist worker ran other trio tests. - Replace inline async tests with sync tests that subprocess the runner; fixes 1200s timeout and 'Attempted to call run() from inside a run()' in CI. Made-with: Cursor * CI: anyio_manager_stats 120s timeout + relaxed assert; trio lifecycle 2s timeout - test_anyio_manager_stats: subprocess timeout 60->120s, runner fail_after(90), relax root_tasks assert to >=1 for timing-dependent AnyIOManager (fixes Windows/CI timeout and assertion). - test_trio_based_service: do_service_lifecycle_check 0.1s->2s timeout so wait_started/wait_finished pass on Windows CI (fixes TooSlowError). Made-with: Cursor --------- Co-authored-by: Manu Sheel Gupta <manusheel.edu@gmail.com> Co-authored-by: yashksaini-coder <yashksaini89@gmail.com> Co-authored-by: yashksaini-coder <115717039+yashksaini-coder@users.noreply.github.com> Co-authored-by: Sumanjeet <sumanjeet0012@gmail.com> Co-authored-by: acul71 <34693171+acul71@users.noreply.github.com>

What was wrong?
The codebase previously relied on a custom async_service implementation, which was no longer actively maintained and introduced unnecessary complexity. It made lifecycle management fragile and lacked support for modern async patterns and structured concurrency.
Issue #524
How was it fixed?
The legacy async_service implementation was fully replaced with a new design built using anyio, a well-supported, high-level async framework. This transition modernizes the service infrastructure and significantly improves reliability, maintainability, and task supervision.