Monitoring & Cleanup¶

Simmate provides several CLI commands to monitor the status of your queue and manage WorkItems. These commands are essential for keeping track of large batches of jobs and maintaining a healthy database.

Checking Queue Stats¶

To get a quick overview of how many jobs are in each state (Pending, Running, Completed, Failed, Cancelled), use:

simmate compute workitems stats

For a more detailed breakdown, including stats for each specific workflow (categorized by tags), use:

simmate compute workitems stats --detail

You can also filter these stats by time to see recent activity:

# Show stats for jobs updated in the last 24 hours
simmate compute workitems stats --detail --recent 24

Inspecting WorkItems¶

To see a list of individual jobs and their current status in a table format, use the workitems list command:

simmate compute workitems list

You can filter the list by status, tags, or recency:

# Show only errored jobs
simmate compute workitems list --status E

# Show jobs with a specific tag
simmate compute workitems list --tag my-tag

# Show jobs updated in the last 12 hours
simmate compute workitems list --recent 12

Error Summaries¶

When jobs fail, they are marked as Failed. To see a quick summary of the error messages for all failed jobs without digging into individual log files, run:

simmate compute workitems errors

This command will print the WorkItem ID followed by the exception message, making it easy to identify common failure modes across your cluster.

Stale & Crashed Workers¶

Workers check in with the database regularly (a "heartbeat"). If a worker is killed without a clean shutdown (e.g. by SLURM, an OOM error, or a node failure), it stops checking in:

• After 5 minutes without a check-in, the worker is marked as Stale Heartbeat. If it starts checking in again, it recovers on its own.
• After 1 hour without a check-in, the worker is marked as Crashed. Any WorkItems it was running are marked as INTERRUPTED.

This check runs every 5 minutes as a schedule, so you'll need the scheduler running (simmate compute scheduler start, see Schedules). You can also run it yourself:

from simmate.compute import SimmateWorker

SimmateWorker.update_stale_workers()

Interrupted items are not retried automatically. To put them back in the queue so they start over from scratch:

from simmate.compute import WorkItem

WorkItem.requeue_interrupted()

Cleaning Up the Queue¶

Over time, the WorkItem table can grow very large, which may eventually slow down database queries. It's a good practice to periodically delete old or unnecessary entries.

Automatic Cleanup¶

When the scheduler is running (simmate compute scheduler start, see Schedules), a nightly task deletes entries that haven't been updated in over 30 days:

• WorkItems that are finished (Completed, Failed, Cancelled, or Interrupted). This includes interrupted items that were never requeued.
• Workers that are no longer active (Stopped, Crashed, etc.).

Pending and Running items, and active workers, are never removed. You can also run the cleanup yourself with a different cutoff:

from simmate.compute import SimmateWorker, WorkItem

WorkItem.delete_inactive(days=7)
SimmateWorker.delete_inactive(days=7)

Deleting Completed Jobs¶

This is the most common cleanup task. It removes entries for jobs that completed successfully.

simmate compute workitems delete --completed --confirm

Deleting by Tag¶

If you ran a test batch or a specific project that is now complete, you can delete those specific entries:

simmate compute workitems delete --tag test-runs --confirm

Deleting Everything¶

simmate compute workitems delete --all --confirm

Programmatic Monitoring¶

If you are using Python, you can monitor and manage jobs directly using the WorkItem object returned by run_cloud():

• workitem.is_done(): Returns True if the job finished or was canceled.
• workitem.is_running(): Returns True if a worker is currently processing the job.
• workitem.is_pending(): Returns True if the job is still in the queue.
• workitem.cancel(): Attempts to cancel a Pending job.
• workitem.result(): A blocking call that waits for the job to finish and returns the result.

Database Maintenance¶

If you are using a shared Postgres database, the WorkItem table is often the most frequently updated table. In addition to deleting entries via Simmate, you may occasionally need to run standard database maintenance commands like VACUUM (in Postgres) to reclaim disk space after large deletions.