Add task-specific log capturing to task diagnostics mechanism #7217
Conversation
|
|
||
| - logs: | ||
| Dumps all logs specific to the task, including DEBUG logs, regardless of whether DEBUG logging | ||
| is otherwise enabled |
There was a problem hiding this comment.
TODO: should this be the default behavior, or should there be a separate modifier to unfilter the logs?
Are there any concerns around data leakage? System logs were not previously thought of as potentially exposable via the API.
I filed #7214 before thinking of this specific idea - the idea there was to unfilter the logs for that task specifically in the log handler. I'm not actually sure if that is possible or desirable though. Maybe that setting could control the log level collected here, but maybe that's unnecessarily configurable.
There was a problem hiding this comment.
After figuring out how to implement this (see the last commit marked WIP), I kinda do think that it should be at most opt-in (if we do it at all), because it requires you to change the log level of the root logger, otherwise debug logs get filtered out completely.
There was a problem hiding this comment.
You could probably implement it as two separate features. logs and debug-logs, where the latter modifies the behavior of the former
There was a problem hiding this comment.
If the question is should task-level logging be off by default, I think the answer is yes. After users upgrade into this, they should get the same they had before.
There was a problem hiding this comment.
I'm not concerned with users being able to see their own logs from a data leakage perspective.
There was a problem hiding this comment.
@bmbouter I mean should task-specific logs be DEBUG-by-default, or should that be opt-in, or should it be allowed in the first place.
The implementation of it is necessarily a bit hacky (changing the worker's root logger's level - at runtime - for the duration of the task before switching back). The less-hacky alternatives also have downsides.
There was a problem hiding this comment.
I think it should be debug and only debug. It keeps it simple, and this is exclusively for debugging what's going on in a task... That's my take at least. If we want it more/better/different later, we can do more on it later.
There was a problem hiding this comment.
So you're fine with this? 5532ddc#diff-1340f0b578800854ca335c83c0d5aafb9e7be9e9b3776e6171b2966e6d4af161R239
There was a problem hiding this comment.
I don't see DEBUG named there, but overall yes that looks fine.
There was a problem hiding this comment.
For the root log handler NOTSET and DEBUG are equivalent. The difference is that NOTSET looks at the log level of the parent log handler. But for the root handler there is none.
| '%(asctime)s - %(name)s - %(levelname)s - %(message)s', | ||
| datefmt='%Y-%m-%d %H:%M:%S' | ||
| ) | ||
| file_handler.setFormatter(formatter) |
There was a problem hiding this comment.
TODO: this is not the same formatter as the standard Pulp one. Should we use the same formatter? Maybe not? Correlation ID is a bit extraneous when all logs relate to the same task, the pulp prefix is irrelevant, removing some of that stuff is maybe useful.
dralley
force-pushed
the
task-logging
branch
2 times, most recently
from
5d9ca0f to
21eb4d6
Compare
TASK_DIAGNOSTICS and X-Task-Diagnostics can now capture all logs for specific tasks, including DEBUG logs, regardless of whether DEBUG logs are otherwise enabled for the system logger. Because the logs are specific to the task, the logs are linear and not broken up by unrelated logs from other services/workers/tasks. Assisted By: Claude closes pulp#7215
These logs just aren't very helpful, and they're extremely verbose, to the point where they inflate log size by orders of magnitude and meaningfully slow execution.
Temporarily change the root logger's log level, but make sure that the console logger is still only logging at INFO level. closes pulp#7214
2 participants
TASK_DIAGNOSTICS and X-Task-Diagnostics can now capture all logs for specific tasks, including DEBUG logs, regardless of whether DEBUG logs are otherwise enabled for the system logger. Because the logs are specific to the task, the logs are linear and not broken up by unrelated logs from other services/workers/tasks.
Assisted By: Claude
closes #7214