GitXplorerGitXplorer
o

airflow-dokku

public
10 stars
3 forks
0 issues

Commits

List of commits on branch main.
Unverified
960d04c31be06970a351c5fac040fbe1e84cb0ed

refactor template

oohld committed 4 years ago
Unverified
01c1031e06f03951f05df8d82a4f708d98281f1b

ignore PID

oohld committed 4 years ago
Unverified
2a38881816575b71c86e4cb527d7ee2ca9c649ff

init

oohld committed 4 years ago
Verified
e3fbe56550f42bc83e98ffa61a87996abf4497d8

Initial commit

oohld committed 4 years ago

README

The README file for this repository.

airflow-dokku

My attempt to run Airflow with Dokku PaaS. In production-ready mode (with Postgres, Redis). This code should also work with Heroku (so this can be called airflow-heroku) but I have not tested it.

Production

I really love to book a small droplet on Digital Ocean, install there a Dokku instance and then to try to run everything as Dokku apps.

Create Dokku app for Airflow

dokku apps:create airflow

Plugins for production-ready Airflow

You need to install & attach Postgres and Redis (this is my random choice, you can try to replace these to Mysql & RabbitMQ).

dokku postgres:create airflow
dokku postgres:link airflow airflow --alias "AIRFLOW__CORE__SQL_ALCHEMY_CONN"

I don't need to scale Airflow's Celery now so I'll skip the Redis installation and attachment (but it is almost the same as for Postgres).

Setup the env variables

dokku config:set airflow \
AIRFLOW__CORE__EXECUTOR=LocalExecutor \
AIRFLOW__CORE__FERNET_KEY='<any large random secret string>' \
AIRFLOW__CORE__LOAD_EXAMPLES=False \
AIRFLOW__CORE__SQL_ALCHEMY_CONN=<copy of DATABASE_URL value from Postgres>

Create Airflow superuser

Open Dokku app's shell:

dokku enter airflow web

And then call the proper airflow CLI command:

airflow users create \
    --username admin \
    --firstname Peter \
    --lastname Parker \
    --role Admin \
    --email spiderman@superhero.org

Activate Dokku auto deploys with Github Actions CI

Open the file at .github/workflows/dokku.yml and add there the URL to the machine where your Dokku runs. Then open Github repo's Settings tab -> Secrets and add SHH_PRIVATE_KEY variable. GitHub Action CI will use this key to connect to your machine via ssh.

HTTPs with Letsencrypt

If you attach a domain to a Dokku app you can add a HTTPs security to your service. This can be done with Dokku Letsencrypt plugin.

dokku letsencrypt:enable airflow

Contact me if you have any questions.