Initial draft docs (#283)

Author: filiplajszczak

.gitignore

@@ -10,6 +10,7 @@
 bin/
 build/
 dist/
+docs/_build/
 include/
 lib/
 pip-selfcheck.json

README.rst

@@ -1,9 +1,13 @@
 pytest-rerunfailures
 ====================
 
+.. START-SHORT-DESCRIPTION
+
 pytest-rerunfailures is a plugin for `pytest <https://pytest.org>`_ that
 re-runs tests to eliminate intermittent failures.
 
+.. END-SHORT-DESCRIPTION
+
 .. image:: https://img.shields.io/badge/license-MPL%202.0-blue.svg
    :target: https://github.com/pytest-dev/pytest-rerunfailures/blob/master/LICENSE
    :alt: License
@@ -14,6 +18,8 @@ re-runs tests to eliminate intermittent failures.
    :target: https://github.com/pytest-dev/pytest-rerunfailures/actions
    :alt: GitHub Actions
 
+.. START-INSTALLATION
+
 Requirements
 ------------
 
@@ -40,6 +46,8 @@ To install pytest-rerunfailures:
 
   $ pip install pytest-rerunfailures
 
+.. END-INSTALLATION
+
 Recover from hard crashes
 -------------------------
 
@@ -212,6 +220,8 @@ Here's an example of the output provided by the plugin when run with
 Note that output will show all re-runs. Tests that fail on all the re-runs will
 be marked as failed.
 
+.. START-COMPATIBILITY
+
 Compatibility
 -------------
 
@@ -222,6 +232,10 @@ Compatibility
   `flaky <https://pypi.org/project/flaky/>`_, you can only have
   ``pytest-rerunfailures`` or ``flaky`` but not both.
 
+.. END-COMPATIBILITY
+
+.. START-CONTRIBUTING
+
 Resources
 ---------
 
@@ -239,3 +253,5 @@ Development
     @hookimpl(tryfirst=True)
     def pytest_runtest_makereport(item, call):
         print(item.execution_count)
+
+.. END-CONTRIBUTING

docs/changelog.rst

@@ -0,0 +1 @@
+.. include:: ../CHANGES.rst

docs/cli.rst

@@ -0,0 +1,82 @@
+Command Line Interface Options
+==============================
+
+.. option:: --only-rerun
+
+   **Description**:
+       Specify regex patterns for errors to rerun. Use this option multiple times to accumulate a list of regexes.
+
+   **Type**:
+       String (repeatable)
+
+   **Default**:
+       None
+
+   **Example**:
+       .. code-block:: bash
+
+           pytest --only-rerun AssertionError --only-rerun ValueError
+
+.. option:: --reruns
+
+   **Description**:
+       The number of times to rerun failing tests.
+
+   **Type**:
+       Integer
+
+   **Default**:
+       Not set (must be provided)
+
+   **Example**:
+       .. code-block:: bash
+
+           pytest --reruns 5
+
+.. option:: --reruns-delay
+
+   **Description**:
+       Delay in seconds between reruns.
+
+   **Type**:
+       Float
+
+   **Default**:
+       Not set (must be provided)
+
+   **Example**:
+       .. code-block:: bash
+
+           pytest --reruns 5 --reruns-delay 1
+
+.. option:: --rerun-except
+
+   **Description**:
+       Specify regex patterns for errors to exclude from reruns. Use this option multiple times to accumulate a list of regexes.
+
+   **Type**:
+       String (repeatable)
+
+   **Default**:
+       None
+
+   **Example**:
+       .. code-block:: bash
+
+           pytest --reruns 5 --rerun-except AssertionError --rerun-except OSError
+
+.. option:: --fail-on-flaky
+
+   **Description**:
+       If set, the test run will fail with exit code 7 if a flaky test passes on rerun.
+
+   **Type**:
+       Boolean flag
+
+   **Default**:
+       False
+
+   **Example**:
+       .. code-block:: bash
+
+           pytest --fail-on-flaky

docs/conf.py

@@ -0,0 +1,21 @@
+import os
+import sys
+from datetime import datetime, timezone
+
+sys.path.insert(0, os.path.abspath("../src"))
+
+project = "pytest-rerunfailures"
+copyright = (
+    f"2012-{datetime.now(tz=timezone.utc).year}, Leah Klearman and pytest-dev team"
+)
+author = "Leah Klearman"
+
+extensions = [
+    "sphinx.ext.autodoc",
+]
+
+templates_path = ["_templates"]
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+
+html_theme = "sphinx_rtd_theme"
+html_static_path = ["_static"]

docs/configuration.rst

@@ -0,0 +1,63 @@
+Configuration
+=============
+
+The ``pytest.ini`` configuration file allows you to set default values for the plugin's options,
+enabling a consistent test execution environment without the need to specify :doc:`command-line options </cli>` every time.
+
+Available ``pytest.ini`` Options
+--------------------------------
+
+Below are the ``pytest.ini`` options supported by the plugin:
+
+``reruns``
+^^^^^^^^^^
+
+- **Description**: Sets the default number of times to rerun failed tests. If not set, you must provide the :option:`--reruns` option on the command line.
+- **Type**: String
+- **Default**: Not set (must be provided as a CLI argument if not configured).
+- **Example**:
+
+  .. code-block:: ini
+
+     [pytest]
+     reruns = 3
+
+``reruns_delay``
+^^^^^^^^^^^^^^^^
+
+- **Description**: Sets the default delay (in seconds) between reruns of failed tests.
+- **Type**: String
+- **Default**: Not set (optional).
+- **Example**:
+
+  .. code-block:: ini
+
+     [pytest]
+     reruns_delay = 2.5
+
+Example
+-------
+
+To configure your test environment for consistent retries and delays, add the following options to your ``pytest.ini`` file:
+
+.. code-block:: ini
+
+   [pytest]
+   reruns = 3
+   reruns_delay = 2.0
+
+This setup ensures that:
+
+- Failed tests will be retried up to 3 times.
+- There will be a 2-second delay between each retry.
+
+Overriding ``pytest.ini`` Options
+---------------------------------
+
+Command-line arguments always override ``pytest.ini`` settings. For example:
+
+.. code-block:: bash
+
+   pytest --reruns 5 --reruns-delay 1.5
+
+This will retry tests 5 times with a 1.5-second delay, regardless of the values set in ``pytest.ini``.

docs/contributing.rst

@@ -0,0 +1,5 @@
+.. include:: ../CONTRIBUTING.rst
+
+.. include:: ../README.rst
+    :start-after: .. START-CONTRIBUTING
+    :end-before: .. END-CONTRIBUTING

docs/index.rst

@@ -0,0 +1,18 @@
+pytest-rerunfailures
+====================
+
+.. include:: ../README.rst
+    :start-after: .. START-SHORT-DESCRIPTION
+    :end-before: .. END-SHORT-DESCRIPTION
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+
+   installation
+   quickstart
+   cli
+   configuration
+   mark
+   contributing
+   changelog

docs/installation.rst

@@ -0,0 +1,10 @@
+Requirements and Installation
+=============================
+
+.. include:: ../README.rst
+   :start-after: .. START-INSTALLATION
+   :end-before: .. END-INSTALLATION
+
+.. include:: ../README.rst
+   :start-after: .. START-COMPATIBILITY
+   :end-before: .. END-COMPATIBILITY

docs/mark.rst

@@ -0,0 +1,82 @@
+Mark Specific Tests as Flaky
+============================
+
+The ``@pytest.mark.flaky`` decorator allows you to mark individual tests as flaky and configure them to
+automatically re-run a specified number of times upon failure. This is particularly useful for specific tests
+that are intermittently failing due to non-deterministic conditions (e.g., network latency, race conditions).
+That mark also allows to override global settings specified via :doc:`command-line options </cli>`.
+
+Basic Usage
+-----------
+
+To use the ``@pytest.mark.flaky`` decorator, include it in your test function and specify the number of retries using the ``reruns`` argument:
+
+.. code-block:: python
+
+   @pytest.mark.flaky(reruns=3)
+   def test_example():
+       import random
+       assert random.choice([True, False])
+
+In this example, ``test_example`` will automatically re-run up to 3 times if it fails.
+
+Additional Options
+------------------
+
+The ``@pytest.mark.flaky`` decorator supports the following optional arguments:
+
+``reruns_delay``
+^^^^^^^^^^^^^^^^
+
+Specify a delay (in seconds) between re-runs.
+
+.. code-block:: python
+
+   @pytest.mark.flaky(reruns=5, reruns_delay=2)
+   def test_example():
+       import random
+       assert random.choice([True, False])
+
+This will retry the test 5 times with a 2-second pause between attempts.
+
+``condition``
+^^^^^^^^^^^^^
+
+Re-run the test only if a specified condition is met.
+The condition can be any expression that evaluates to ``True`` or ``False``.
+
+.. code-block:: python
+
+   import sys
+
+   @pytest.mark.flaky(reruns=3, condition=sys.platform.startswith("win32"))
+   def test_example():
+       import random
+       assert random.choice([True, False])
+
+In this example, the test will only be re-run if the operating system is Windows.
+
+
+``only_rerun``
+^^^^^^^^^^^^^^
+
+Re-run the test only for specific exception types or patterns.
+That overrides the :option:`--only-rerun` command-line option.
+
+.. code-block:: python
+
+   @pytest.mark.flaky(reruns=5, only_rerun=["AssertionError", "ValueError"])
+   def test_example():
+       raise AssertionError()
+
+``rerun_except``
+^^^^^^^^^^^^^^^^
+
+Exclude specific exception types or patterns from triggering a re-run.
+That overrides the :option:`--rerun-except` command-line option.
+
+.. code-block:: python
+
+   @pytest.mark.flaky(reruns=5, rerun_except="AssertionError")
+   def test_example():
+       raise ValueError()