feat(async): Set priorities to async tasks#14195
feat(async): Set priorities to async tasks#14195kiblik wants to merge 5 commits intoDefectDojo:devfrom
Conversation
github-actions
bot
added
the
settings_changes
kiblik
force-pushed
the
celery_set_prio
branch
3 times, most recently
from
95561b9 to
1f8e468
Compare
Signed-off-by: kiblik <5609770+kiblik@users.noreply.github.com>
Co-authored-by: valentijnscholten <valentijnscholten@gmail.com>
kiblik
force-pushed
the
celery_set_prio
branch
from
53a8e1e to
e882d0f
Compare
kiblik
force-pushed
the
celery_set_prio
branch
2 times, most recently
from
3dd4cdb to
6eb4791
Compare
Signed-off-by: kiblik <5609770+kiblik@users.noreply.github.com>
kiblik
force-pushed
the
celery_set_prio
branch
from
6eb4791 to
161b919
Compare
valentijnscholten
added
the
affects_pro
|
I've converted it to draft as we can only test/merge this after 2.55.2 has been released and merged back into |
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
Resolved conflicts by: - Removing @dojo_async_task decorators that were removed in upstream - Keeping priority parameters from the PR - Adapting to refactored notification system (standalone task functions) - Using correct priority values (3 for notifications/jira, 1 for webhooks, 4 for cleanup tasks, 0 for status checks)
|
Conflicts have been resolved. A maintainer will review the pull request shortly. |
valentijnscholten
removed
the
affects_pro
|
@kiblik I resolved conflicts and published the PR so we can merge it and test it with Pro. The PR does not affect Pro in the sense that Pro will keep working and the Pro celery tasks will get priority 3 which is fine. But we can/should do a Pro PR together or after this one to move some of the Pro celery tasks to other priority levels. |
…tasks - post_process_finding_save: priority=3 (user-triggered, regular task) - post_process_findings_batch: priority=4 (background batch processing) - calculate_grade: priority=4 (background processing) - All JIRA tasks (push_finding_to_jira, push_finding_group_to_jira, push_engagement_to_jira, close_epic, update_epic, add_epic): priority=4 (integrations tier, should not compete with imports for worker slots)
|
@kiblik At first I thought this was a nice way to implement separate queues without having to actually create separate queues in Valkey. However with this approach there's a risk of starvation on the prio 4/5 tasks if an instance is extremely busy (dupe_delete for example). It also may make the task ordering unexpected/unpredictable making it harder to analyse performance problems. What do you think? Since we've never had any reports on the need for priorities or separate queues I have become hesitant to merge this. A compromise could be to have only two priorities 0 (critical) stuff that is needed to keep the instance alive/healthy (1) everything else. |
4 participants
This PR is adding support for priority queues. More important tasks will be processed sooner and vice versa.
The status of each queue is listed in the system settings overview:
Queue with id
0is automatically marked without a suffix (all other are using naming conventioncelery:1,celery:2...). Thanks to this fact, we are keeping backward compatibility (if there are any tasks in the queue during migration to the new version, we will still process those tasks after the upgrade).It is a bit harder to write tests for this. But I tested manually (stop worker, play around, start worker, observe logs) and it was working as expected.
The role of each queue is described in
settings.dist.py(and reflected in the overview in system settings). If task do not have a priority assigned, priority is set to the default value:3I'm open to feedback if priority of any task should be changed.