Skip to content

Conversation

rauchy
Copy link
Contributor

@rauchy rauchy commented Nov 4, 2019

Currently, all RQ jobs are distributed between 3 queues:

  1. periodic - handles periodic jobs stated in schedule.py.
  2. schemas - handles refresh_schemas jobs.
  3. default - for all other jobs.

A couple of issues that this causes:

  1. Since different types of jobs go to default, it's hard to get an understanding of the current distribution of jobs. A solution to this might be to split default into job-specific queues, then we could just observe the queue size.
  2. Within periodic, the refresh_queries job needs to have the highest priority.
  3. Within default, the send_email jobs needs to have the highest priority.

@rauchy rauchy requested a review from arikfr November 4, 2019 12:55
@rauchy rauchy merged commit 6f791a0 into master Nov 6, 2019
@rauchy rauchy deleted the rq-job-priorities branch November 6, 2019 11:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants