Multiproc doesn't capture all metrics

[matt0x6F]

Hey folks,

I have a similar setup to what is described here. The only difference is that I have nginx in front talking to gunicorn over sockets, but I don't think that matters as far as I can tell.

I wasn't able to use this code:

# Using multiprocess collector for registry
def make_metrics_app():
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)
    return make_asgi_app(registry=registry)


metrics_app = make_metrics_app()
app.mount("/metrics", metrics_app)

It results in a page that gets a 307, so I implemented this:

@router.get("/metrics", response_class=PlainTextResponse, include_in_schema=False)
async def get_metrics() -> str:
    """
    Get prometheus metrics
    """
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)
    data = generate_latest(registry)
    res = Response(content=data)
    res.headers["Content-Type"] = CONTENT_TYPE_LATEST
    res.headers["Content-Length"] = str(len(data))
    return res

Apart from that my setup is pretty typical, I have some counters and a histogram:

AUTOMATIONS_TOTAL: Final = Gauge(
    name=prefix_("automations_total"),
    documentation="Number of automations run",
    labelnames=["platform", "action", "kind", "status"],
    multiprocess_mode="sum",
)

REQUEST_TIME_SECONDS: Final = Histogram(
    name=prefix_("request_time_seconds"),
    documentation="Time spent processing request",
    labelnames=["method", "url_rule", "status_code"],
)

REQUESTS_IN_PROGRESS_TOTAL: Final = Gauge(
    name=prefix_("requests_in_progress_total"),
    documentation="Number of concurrent requests",
    # See Metrics Tuning (Gauge)
    # https://github.com/prometheus/client_python#multiprocess-mode-gunicorn
    multiprocess_mode="sum",
)

My in progress counter always remains at zero, but I'm hoping that's because requests finish faster than I can see them. The histogram, however, never shows up - I see no record of it in my PROMETHEUS_MULTIPROC_DIR and it's never rendered in the output. Any ideas for what I could troubleshoot?

[matt0x6F]

Some extra context: most all of these get called in async middleware. Here's a rather benign one:

async def add_request_id(
    request: Request, call_next: Callable[[Request], Awaitable[Response]]
) -> Response:
    if not is_prometheus_endpoint(request):
        REQUESTS_IN_PROGRESS_TOTAL.inc()
        request.state.request_id = correlation_id.get()
        start_time = time.time()

    response = await call_next(request)

    if not is_prometheus_endpoint(request):
        time_taken = time.time() - start_time

        # we replace UUIDs with <uuid> so that we flatten the curve of cardinality with unique URLs across all urls
        REQUEST_TIME_SECONDS.labels(
            method=request.method,
            url_rule=replace_uuids(request.url.path),
            status_code=response.status_code,
        ).observe(time_taken)
        REQUESTS_IN_PROGRESS_TOTAL.dec()

    return response

[csmarchbanks]

Hmm, I am not seeing anything obvious, does REQUESTS_IN_PROGRESS show up in the multiproc dir?

[gg-mmill]

I have similar issues, with some histogram metrics specifically not showing up, only in certain contexts ( threaded celery workers e.g. ). Working on having reproducible code.

[danobi]

You are probably not setting PROMETHEUS_MULTIPROC_DIR before importing prometheus_client. Note that even a transitive import of prometheus_client from some downstream dependency can trigger this issue.

[gg-mmill]

Does adding a labelnames argument (with a dummy label name) in REQUESTS_IN_PROGRESS_TOTAL solves the issue ?

I had similar troubles, and this solved it for me (see #1123)