Automate away friction
- What do I mean
- Automation tools and patterns
Automate away friction
- What do I mean
Friction: Manual tasks and workflows that take time and mental energy but are just a means to an end
Automate: Minimize the needed involvement of a human
Trade-off: The time and mental strain spent doing a task vs the cost of automating it
Automate away friction
- What do I mean
Pick your battles!
- Data pipelines and workflows
- Testing
- Deployment
-
Building/compiling
(code/webpages/documents/...) - Installations and environments
- Publishing and uploading
- ...
Some examples of good targets for automation
Automate away friction
- What do I mean
Pick your battles!
Knightmare: A DevOps Cautionary Tale - An interesting read, see handout
Takeaway: humans are not reliable for critical, long, and repetitive tasks! (if it cannot be automated - checklists do help)
Automate away friction
- What do I mean
- Automation tools and patterns
Automate away friction
- Automation tools and patterns
Environments
- uv
- conda
- containers
- ...
Automate away friction
- Automation tools and patterns
Environments
- uv
# automate recreating exact environment
uv lock
uv sync
Automate away friction
- Automation tools and patterns
Environments
- uv
- conda
# automate recreating exact environment
conda env export > environment.yml
conda env create -f environment.yml
Automate away friction
- Automation tools and patterns
Environments
- uv
- conda
- containers
# Containerfile
FROM alpine:3.21
MAINTAINER Daniel daniel.kastinen@irf.se
USER root
RUN apk add --no-cache python3
ADD test_code.py /root/
CMD python /root/test_code.py
Automate away friction
- Automation tools and patterns
Environments
Automate away friction
- Automation tools and patterns
Environments
Managing many system environments (not just Python!)
Lmod - an environment module system
$ module avail
$ module load package1 package2 ...
$ module unload package1 package2 ...
What is used by HPC2N
Automate away friction
- Automation tools and patterns
Environments
Managing many system environments (not just Python!)
It modifies the shell environment
$ env | grep -i ddt
$ module load ddt
$ env | grep -i ddt
DDTPATH=/opt/apps/ddt/5.0.1/bin
LD_LIBRARY_PATH=/opt/apps/ddt/5.0.1/lib:...
PATH=/opt/apps/ddt/5.0.1/bin:...
$ module unload ddt
$ env | grep -i ddt
Automate away friction
- Automation tools and patterns
Building
pip - automates python dependencies and build
system
Automate away friction
- Automation tools and patterns
Building
pip - automates python dependencies and build
system
# setup.py
import setuptools
from distutils.core import Extension
clib_beam = Extension(
name='pyant.clibbeam',
sources=[
'src/clibbeam/array.c',
'src/clibbeam/beam.c',
],
include_dirs=['src/clibbeam/'],
)
setuptools.setup(
ext_modules=[clib_beam],
)
Automate away friction
- Automation tools and patterns
Building
Does your Python code have external dependencies?
List and check for them somehow! For example
import setuptools
import shutil
class DependencyNotFound(Exception):
pass
if shutil.which("ffmpeg") is None:
raise DependencyNotFound("ffmpeg is not installed - aaaaaah!")
setuptools.setup()
Automate away friction
- Automation tools and patterns
Building
Other Python build systems
Automate away friction
- Automation tools and patterns
Building
make - automate building and whatever else rule-based file-dependant
workflow you have
# Makefile
source_files := $(wildcard diagrams/*.gv)
output_files_tmp := $(source_files:.gv=.png)
output_files := $(addprefix static/,$(output_files_tmp))
all: $(output_files)
static/diagrams/%.png : diagrams/%.gv
dot -Tpng -o$@ $<
Automate away friction
- Automation tools and patterns
Building
Automate away friction
- Automation tools and patterns
Building
Automate away friction
- Automation tools and patterns
Building
meson - generic open source build system (written in Python)
# meson.build
project('tutorial', 'c')
gtkdep = dependency('gtk+-3.0')
executable('demo', 'main.c', dependencies : gtkdep)
what scipy uses
More links in the handouts
Automate away friction
- Automation tools and patterns
Installing
Most software follow something like:
$ ./configure
$ make
$ make install
So how does package managers work?
Automate away friction
- Automation tools and patterns
Installing
PKGBUILD example (used by Arch)
_name=urwid
pkgname=python-urwid
pkgver=2.6.16
pkgrel=1
pkgdesc='Curses-based user interface library'
arch=('any')
url='https://urwid.org/'
license=('LGPL-2.1-only')
depends=(
'glib2'
'python'
'python-wcwidth'
)
makedepends=(
'python-build'
'python-installer'
'python-setuptools'
'python-setuptools-scm'
'python-wheel'
)
checkdepends=(
'python-gobject'
'python-pytest'
'python-pyzmq'
'python-tornado'
'python-trio'
'python-twisted'
)
optdepends=(
'python-gobject: for gobject integration'
'python-pyserial: for LCD and serial integration'
'python-pyzmq: for zmq integration'
'python-tornado: for tornado integration'
'python-trio: for trio integration'
'python-twisted: for twisted integration'
)
source=("https://github.com/urwid/urwid/archive/$pkgver/${pkgname#python-}-$pkgver.tar.gz")
sha512sums=('7a928fe6e35714b351f99bce025fa2d50873052dac8b33d8b0dc76ffb8784c623cd83dd6314df5c61c11165734919803bd90f17e7aef1dbe032aa30615830ecd')
b2sums=('76304a463491e433c5cd1551c6c94996046a81a0d50e1dcd44a4ad84eaba52358bc352c1e72599e88749c6ac7e3dccfe582d050b34be69a930bcfe5b9df6b932')
build() {
cd ${pkgname#python-}-$pkgver
export SETUPTOOLS_SCM_PRETEND_VERSION=$pkgver
python -m build --wheel --no-isolation
}
check() {
cd ${pkgname#python-}-$pkgver
# Override addopts as they invoke coverage testing
pytest --override-ini="addopts=-vv --doctest-modules -s" tests
}
package() {
cd ${pkgname#python-}-$pkgver
python -m installer --destdir="$pkgdir" dist/*.whl
}
# vim: ts=2 sw=2 et:
Automate away friction
- Automation tools and patterns
Installing
PKGBUILD example (used by Arch)
$ makepkg -si
Then this will install onto your system!
Links to similar systems for Ubuntu in handouts
Automate away friction
- Automation tools and patterns
Workflow automation
- snakemake
- make
- just scripts!
Automate away friction
- Automation tools and patterns
Workflow automation
- snakemake
Automate away friction
- Automation tools and patterns
Workflow automation
- snakemake
# make_data.py
from pathlib import Path
import random
import string
from snakemake.script import snakemake
def make_file(path, ind, ext):
with open(path / f"{ind}{ext}", "w") as fh:
fh.write("".join(random.choices(
string.ascii_uppercase + string.digits,
k=random.randint(1, 257),
)))
path = Path(snakemake.input[0])
make_file(path, "data", ".dat")
make_file(path, "data", ".txt")
Automate away friction
- Automation tools and patterns
Workflow automation
- snakemake
# sum_files.py
from pathlib import Path
from snakemake.script import snakemake
total = 0
for file in snakemake.input:
total += Path(file).stat().st_size
with open(snakemake.output[0], "wb") as fh:
fh.write(total.to_bytes())
snakemake --dag dot | dot -Tpng > dag.png
Automate away friction
- Automation tools and patterns
Workflow automation
- snakemake
snakemake --dag dot | dot -Tpng > dag.png
automate away friction
- automation tools and patterns
workflow automation
Automate away friction
- Automation tools and patterns
Workflow automation
Links to more info in the handout
Automate away friction
- Automation tools and patterns
Workflow automation
Next up: ansible "system admin" automation tool
- hosts
- tasks
How it works
Automate away friction
- Automation tools and patterns
Workflow automation
Next up: ansible "system admin" automation tool
- hosts
How it works
# hosts.ini
[computers]
this_machine ansible_host=127.0.0.1
Automate away friction
- Automation tools and patterns
Workflow automation
- tasks
# tasks.yml
---
- name: This is a set of tasks
hosts: computers
gather_facts: True
become: no
tasks:
- name: save info
ansible.builtin.copy:
content: "{{ hostvars[inventory_hostname] | to_nice_json }}"
dest: ./setup-dump.json
- name: check file
ansible.builtin.file:
path: ./setup-dump.json
state: file
Automate away friction
- Automation tools and patterns
Workflow automation
ansible-playbook -i hosts.ini tasks.yml -c local
Automate away friction
- Automation tools and patterns
Workflow automation
Automate away friction
- Automation tools and patterns
Workflow automation
- GitLab CI
- Github Actions
- Jenkins
- Ambient
- ...
CI - tools
Automate away friction
- Automation tools and patterns
Workflow automation
- GitLab CI
(IRF hosts this!)
A system for automatically running tasks in containers on worker machines when triggered
This is what builds this website on every git push!
Automate away friction
- Automation tools and patterns
Workflow automation
# .gitlab-ci.yml
image: registry.gitlab.com/pages/hugo/hugo_extended:0.139.3
stages:
- deploy
pages:
stage: deploy
script:
- apk add --no-cache bash wget iputils tar
- ./install-revealjs
- hugo -b "https://danielk.developer.irf.se/phd-course-software-development"
artifacts:
expire_in: 5 minutes
paths:
- public
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
Automate away friction
- Automation tools and patterns
Workflow automation
Automate away friction
- Automation tools and patterns
Workflow automation
Automate away friction
The takeaway:
Automate away friction so you can concentrate on the important bits
Lets have a look at the handout