Submitit is a Python 3.8+ toolbox developed by Facebook Incubator for submitting jobs to Slurm clusters, as well as providing a local executor for testing. It simplifies the process of dispatching Python functions to compute nodes, managing job states, and retrieving results. The current version is 1.5.4, and it sees active maintenance with occasional releases.
pip install submititVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates how to use `submitit.AutoExecutor` to submit Python functions as jobs. It showcases configuring an executor, defining a simple function, submitting multiple jobs in a batch, and retrieving their results. The `AutoExecutor` intelligently switches between Slurm and local execution based on the environment.
Review job logs and `submitit`'s internal state for jobs that are expected to be pre-empted or timed out. Adapt your application's logic if it relied on the previous detection mechanism. Ensure your Slurm environment is consistent.
Inspect any file paths or directory paths passed to `submitit` or configured in Slurm scripts. Ensure they are correctly formatted and do not contain characters that might have been misinterpreted before the quoting fixes.
For production or critical workloads, explicitly use `submitit.SlurmExecutor` or `submitit.LocalExecutor` to guarantee the desired execution environment. Always test `AutoExecutor` behavior in your target environments to understand its fallback logic.
Ensure all arguments passed to `executor.submit()` are standard Python types or custom classes that are fully `cloudpickle`-compatible. For complex setups, consider passing file paths to data or configuration instead of directly passing objects, allowing the job to load resources within its own context.
Ensure all functions and their arguments passed to `executor.submit()` are defined at the top-level of a module, are static methods, or are otherwise robustly pickleable. Avoid using local functions or lambdas directly.
Import executor classes directly from the top-level `submitit` package: `from submitit import SlurmExecutor, AutoExecutor, LocalExecutor`.
Either change the log folder to a path where the user has write permissions, or modify the permissions of the existing log folder to allow writing.
Ensure the Slurm job's execution environment includes all necessary packages. This can be done by activating a Conda or virtual environment within the Slurm job (e.g., using `pre_cmd` in `slurm_options`) or by installing dependencies directly on the compute nodes.
No dependency data recorded yet.