Upload more than one file per FTPS connection

[jeremyestein]

Description

We currently upload one file per snakemake job send_ftps. The DSH emails us once for each FTPS connection we make. When we implement #45 , this is going to make that Slack channel very noisy each night.

It will also be unnecessarily slow.

Hopefully snakemake has a feature to group together similar jobs into a batch.

IIRC the PIXL FTPS code has a hardcoded assumption that only one file will need uploading.

Implementation challenges:

• get snakemake to group together jobs
• might need to copy and modify PIXL FTPS code so that it can do more than one file per connection

Definition of done

• We end up with a reasonable number of notifications after each nightly batch of processing
• We understand what happens if a connection gets interrupted halfway through, and can recover from this