Loglevel not reflecting for health check

[danielkalotai]

Report

We have a problem, that we have set the log level to error but we can still see info logs in the k8s pod.
We are using the promitor-agent-resource-discovery chart.

Here is a part of our values.yaml file

telemetry:
  defaultLogLevel: "Error"
  applicationInsights:
    enabled: false
    key: ""
    logLevel: "Error"
  containerLogs:
    enabled: true
    logLevel: "Error"

As long as I understood the coding correctly, there should now only be Error logs displayed from the pod. But we still see INF logs.

prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Executed action Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController.Get (Promitor.Agents.ResourceDiscovery) in 0.3546ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Executed action Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController.Get (Promitor.Agents.ResourceDiscovery) in 0.4904ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Executed action Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController.Get (Promitor.Agents.ResourceDiscovery) in 0.4547ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Request finished HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - - - 200 2344 application/json 1.0588ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Request finished HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - - - 200 2344 application/json 1.3891ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Request finished HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - - - 200 2344 application/json 1.2806ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Request starting HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - -
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Request starting HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - -
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Request starting HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - -
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Route matched with {action = "Get", controller = "Discovery"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(System.String) on controller Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController (Promitor.Agents.ResourceDiscovery).
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Route matched with {action = "Get", controller = "Discovery"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(System.String) on controller Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController (Promitor.Agents.ResourceDiscovery).
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Route matched with {action = "Get", controller = "Discovery"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(System.String) on controller Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController (Promitor.Agents.ResourceDiscovery).

In the deployed configmap which is coming from the helm chart, I can see in the runtime.yaml key, that the telemetry is set to Error and is reflecting our values.yaml loglevel.

Expected Behavior

Don't see INF logs of the healthcheck if loglevel is set to Error

Actual Behavior

We see the following log lines

prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Executed action Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController.Get (Promitor.Agents.ResourceDiscovery) in 0.3546ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Executed action Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController.Get (Promitor.Agents.ResourceDiscovery) in 0.4904ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Executed action Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController.Get (Promitor.Agents.ResourceDiscovery) in 0.4547ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Request finished HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - - - 200 2344 application/json 1.0588ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Request finished HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - - - 200 2344 application/json 1.3891ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Request finished HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - - - 200 2344 application/json 1.2806ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Request starting HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - -
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Request starting HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - -
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Request starting HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - -
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Route matched with {action = "Get", controller = "Discovery"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(System.String) on controller Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController (Promitor.Agents.ResourceDiscovery).
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Route matched with {action = "Get", controller = "Discovery"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(System.String) on controller Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController (Promitor.Agents.ResourceDiscovery).
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Route matched with {action = "Get", controller = "Discovery"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(System.String) on controller Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController (Promitor.Agents.ResourceDiscovery).

Steps to Reproduce the Problem

1. use log level error
2. check logs
3. there will be info logs from healthcheck
   ...

Component

Resource Discovery

Version

0.9.0

Configuration

Configuration:
ConfigMap of the K8s deployment:

data:
  resource-discovery-declaration.yaml: |-
     ...
      name: postgres-databases
      type: PostgreSql
  runtime.yaml: |-
    server:
      httpPort: "88"
    authentication:
      mode: ServicePrincipal
      identityId: XXX
    cache:
      enabled: "true"
      durationInMinutes: "5"
    telemetry:
      applicationInsights:
        isEnabled: "false"
        verbosity: "Error"
      containerLogs:
        isEnabled: "true"
        verbosity: "Error"
      defaultVerbosity: "Error"

Logs

prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Executed action Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController.Get (Promitor.Agents.ResourceDiscovery) in 0.3546ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Executed action Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController.Get (Promitor.Agents.ResourceDiscovery) in 0.4904ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Executed action Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController.Get (Promitor.Agents.ResourceDiscovery) in 0.4547ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Request finished HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - - - 200 2344 application/json 1.0588ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Request finished HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - - - 200 2344 application/json 1.3891ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:20:00 INF] Request finished HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - - - 200 2344 application/json 1.2806ms
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Request starting HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - -
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Request starting HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - -
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Request starting HTTP/1.1 GET http://prometheus-promitor-agent-resource-discovery:8889/api/v1/resources/groups/postgres-databases/discover - -
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Route matched with {action = "Get", controller = "Discovery"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(System.String) on controller Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController (Promitor.Agents.ResourceDiscovery).
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Route matched with {action = "Get", controller = "Discovery"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(System.String) on controller Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController (Promitor.Agents.ResourceDiscovery).
prometheus-promitor-agent-resource-discovery-676d5fdc78-n7qjm prometheus-promitor-agent-resource-discovery [07:21:00 INF] Route matched with {action = "Get", controller = "Discovery"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(System.String) on controller Promitor.Agents.ResourceDiscovery.Controllers.DiscoveryController (Promitor.Agents.ResourceDiscovery).

Platform

Other

Contact Details

No response

[github-actions]

Thank you for opening an issue! We rely on the community to maintain Promitor. (Learn more)

Is this something you want to contribute?

[danielkalotai]

Another addition to this is, that the metrics endpoint is showing the same behaviour of printing out logs, even if the log level is set to error.

[nemobis]

I have the same issue. I'm not sure how to check whether the configuration is applied, as I don't see this line in the configmap where it seems supposed to be and the image does not seem to allow executing any command so I can't "ssh" into it.