GitXplorerGitXplorer
c

luigi-kubernetes

public
27 stars
3 forks
2 issues

Commits

List of commits on branch master.
Verified
a17d4b684bf682e80dd85a7f44aa8b5d6539b658

Add link to slides

cchhantyal committed 5 years ago
Unverified
1fbd59d3bd77c2cb24874fa8c0f7ed8936d60967

fix readme text

committed 6 years ago
Unverified
59483dffee16c881ccef6d3ecef2b9ce0f471e8f

bump version

committed 6 years ago
Unverified
636339fac250a27bde3258e7008f010330730f13

move luigid resources to luigid dir

committed 6 years ago
Unverified
9b9826226d764ffd01d729a4c60d13f8a8c52cc0

bump version

committed 6 years ago
Unverified
e9146b78fcb968f49ad50208241d8ec930f9db02

add sleep option

committed 6 years ago

README

The README file for this repository.

Luigi examples with Kubernetes

This is Luigi and Kubernetes demo presented at Europython 2019 in Basel.

Luigi pipeline can be easily triggered from Kubernetes cronjob. By doing so, one can get all the benefits of container orchestration system like flexible deployment, horizontal scaling and easier maintenance.

Slides: https://speakerdeck.com/chhantyal/building-data-workflows-with-luigi-and-kubernetes

Run

Luigi example on local

pipenv run python -m luigi --module example SalesReport --date=2019-07-11 --local-scheduler

Luigi example on Kubernetes (minikube)

Minikube:

  • Start minikube: minikube start --memory 4096
  • Open K8S dashboard: minikube dashboard

Luigid:

  • Deploy luigid: mkctl apply -f luigid/deployment.yaml && mkctl apply -f luigid/service.yaml
  • Open Luigi UI: minikube service luigid-service

Luigi task:

  • Deploy as cronjob: mkctl apply -f cronjob.yaml
  • Run: kubecron run luigi-example

Docker

Docker images used in above deployments are available in Docker Hub https://hub.docker.com/u/chhantyal