Setting up your cluster and workers¶
Run vs. Run-cloud¶
Scheduling a workflow is straight-forward. Simply change all your scripts and commands from the
run method to the
run_cloud method. For example, if you are using the command line:
simmate workflows run-cloud my_settings.yaml
This schedules your workflow, but it won't run yet. It is simply sitting in the queue and waiting for a worker to pick it up. Once we start a worker, then it will actually run.
Start a "single-flow" worker¶
Whereever you'd like to run the workflow, start a worker with:
simmate engine start-singleflow-worker
If you are on a cluster, start-worker should be called within your submit script (e.g. inside
submit.sh for SLURM). Don't run workers on the head node!
Start a "many-flow" worker¶
When you run this "singleflow" worker, you'll notice that the Worker will start, run 1 workflow, then shutdown. This is the recommended approach for HPC clusters because it follow best practices for sharing resources. You don't want a worker hogging computational resources if there aren't any workflows scheduled to run!
However, if you would like more control over how many workflows are ran or even run a worker endlessly, you can use the command:
simmate engine start-worker
If your team runs many mini workflows that are under 5 minutes, starting/stopping workers could be a pain (sometimes it can take simmate up to 10 seconds to set everything up). That's a significant overhead and wasted computation time.
To overcome this, we would run a worker that shuts down after 10 workflows or if the queue is empty:
simmate engine start-worker --nitems-max 10 --close-on-empty-queue
Starting many workers at once¶
If you need to start many workers at once, you can use the
start-cluster command as well.
# starts 5 local workers simmate engine start-cluster 5
Controlling what workflows are ran by each worker¶
The full guide for custom workers is still being written. See workflow "tags" in the full guides for more information.
Connecting others to your scheduler¶
If they are connected to your database, then they're good to go! Other schedulers like Prefect or Dask require extra setup.