Delayed Tasks and Actions #105
Description
We need to have the ability to delay tasks for a period of time in order to accommodate orders that must be fulfilled at a future time e.g. call me in the morning. To have this capability we will need to schedule tasks for a specific period of time before alerting a delegator about it (not necessarily the one who created the task). We will have to store these messages in DynamoDB and create a thread on every API endpoint that polls the table for "ripe" tasks.
IMPORTANT NOTE ON CONSISTENCY: In order to ensure that there aren't any race conditions the thread that is polling the table will have to change the items "taken" field to True. (if it fails due to a consistency error we will simply skip to the next iteration of the thread) Once, we have "acquired the lock" by modifying the item the thread can go ahead and complete the task. After completion, the thread can delete the item from the table. It should be noted that the we will have "zombie" items in the delayed_tasks table if the execution of that tasks fails. This can be handled at a later date and should be noted in the source code.
NOTE ON TIME: This feature requires clock-drift to be negligible (within +/- one minute) and that our servers are synced to NTP. Since we deploy with docker containers we can't modify the system time on the host machine, thus putting us at the mercy of Amazon's EC2 machines.