Georg Heiler

dagster-slurm is an open-source Python integration that allows data scientists and research software engineers to run Dagster pipeline assets on both a laptop and Slurm-managed HPC supercomputers without making any code changes. It automatically handles SSH transport, environment packaging via pixi-pack, and Slurm job submission, while streaming logs and scheduler metrics back to the Dagster...

Magenta Telekom ingests many terabytes of new data every day, and every downstream consumer wants it immediately. The real bottleneck turned out not to be hardware but humans wrestling with hidden, hard-wired dependencies in hundreds of heterogeneous pipelines and sometimes tool silos.