Bitnami/charts
2
0
Fork 0
mirror of https://github.com/bitnami/charts.git synced 2026-03-05 06:47:25 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1161 from tompizmor/add_airflow_chart

Browse Source 
[bitnami/airflow] Add airflow chart
This commit is contained in:
Tomas Pizarro
2019-05-07 17:38:45 +02:00
committed by GitHub
parent 2848909005 1056190ffa
commit 12a19ea941
24 changed files with 2142 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
README.md
View File

@@ -44,6 +44,7 @@ $ helm search bitnami
## Bitnami charts
- [Apache](https://github.com/bitnami/charts/tree/master/bitnami/apache)
- [Apache Airflow](https://github.com/bitnami/charts/tree/master/bitnami/airflow)
- [Cassandra](https://github.com/bitnami/charts/tree/master/bitnami/cassandra)
- [Elasticsearch](https://github.com/bitnami/charts/tree/master/bitnami/elasticsearch)
- [ExternalDNS](https://github.com/bitnami/charts/tree/master/bitnami/external-dns)

1
bitnami/airflow/.helmignore Normal file
View File

@@ -0,0 +1 @@
.git

18
bitnami/airflow/Chart.yaml Normal file
View File

@@ -0,0 +1,18 @@
apiVersion: v1
name: airflow
version: 0.0.1
appVersion: 1.10.3
description: Apache Airflow is a platform to programmatically author, schedule and monitor workflows.
keywords:
- apache
- airflow
- workflow
- dag
home: https://airflow.apache.org/
sources:
- https://github.com/bitnami/bitnami-docker-airflow
maintainers:
- name: Bitnami
email: containers@bitnami.com
engine: gotpl
icon: https://bitnami.com/assets/stacks/airflow/img/airflow-stack-110x117.png

199
bitnami/airflow/README.md Normal file
View File

@@ -0,0 +1,199 @@
# Apache Airflow
[Apache Airflow]() is a platform to programmatically author, schedule and monitor workflows.
## TL;DR;
```console
$ helm install bitnami/airflow
```
## Introduction
This chart bootstraps an [Apache Airflow](https://github.com/bitnami/bitnami-docker-airflow) deployment on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager.
Bitnami charts can be used with [Kubeapps](https://kubeapps.com/) for deployment and management of Helm Charts in clusters. This Helm chart has been tested on top of [Bitnami Kubernetes Production Runtime](https://kubeprod.io/) (BKPR). Deploy BKPR to get automated TLS certificates, logging and monitoring for your applications.
## Prerequisites
- Kubernetes 1.4+ with Beta APIs enabled
- PV provisioner support in the underlying infrastructure
## Installing the Chart
To install the chart with the release name `my-release`:
```console
$ helm install --name my-release bitnami/airflow
```
The command deploys Airflow on the Kubernetes cluster in the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation.
> **Tip**: List all releases using `helm list`
## Uninstalling the Chart
To uninstall/delete the `my-release` deployment:
```console
$ helm delete my-release
```
The command removes all the Kubernetes components associated with the chart and deletes the release.
## Configuration
The following tables lists the configurable parameters of the Kafka chart and their default values.
| Parameter | Description | Default |
| ----------------------------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| `global.imageRegistry` | Global Docker image registry | `nil` |
| `global.imagePullSecrets` | Global Docker registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `image.registry` | Airflow image registry | `docker.io` |
| `image.repository` | Airflow image name | `bitnami/airflow` |
| `image.tag` | Airflow image tag | `{VERSION}` |
| `image.pullPolicy` | Airflow image pull policy | `Always` |
| `image.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `image.debug` | Specify if debug values should be set | `false` |
| `schedulerImage.registry` | Airflow Scheduler image registry | `docker.io` |
| `schedulerImage.repository` | Airflow Scheduler image name | `bitnami/airflow-shceduler` |
| `schedulerImage.tag` | Airflow Scheduler image tag | `{VERSION}` |
| `schedulerImage.pullPolicy` | Airflow Scheduler image pull policy | `Always` |
| `schedulerImage.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `schedulerImage.debug` | Specify if debug values should be set | `false` |
| `workerImage.registry` | Airflow Worker image registry | `docker.io` |
| `workerImage.repository` | Airflow Worker image name | `bitnami/airflow-worker` |
| `workerImage.tag` | Airflow Worker image tag | `{VERSION}` |
| `workerImage.pullPolicy` | Airflow Worker image pull policy | `Always` |
| `workerImage.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `workerImage.debug` | Specify if debug values should be set | `false` |
| `gitImage.registry` | Git image registry | `docker.io` |
| `gitImage.repository` | Git image name | `bitnami/git` |
| `gitImage.tag` | Git image tag | `{VERSION}` |
| `gitImage.pullPolicy` | Git image pull policy | `Always` |
| `gitImage.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `updateStrategy` | Update strategy for the stateful set | `RollingUpdate` |
| `rollingUpdatePartition` | Partition update strategy | `nil` |
| `airflow.configurationConfigMap` | Name of an existing config map containing the Airflow config file | `nil` |
| `airflow.dagsConfigMap` | Name of an existing config map containing all the DAGs files you want to load in Airflow. | `nil` |
| `airflow.loadExamples` | Switch to load some Airflow examples | `true` |
| `airflow.cloneDagFilesFromGit.enabled` | Enable in order to download DAG files from git repository. | `false` |
| `airflow.cloneDagFilesFromGit.repository` | Repository where download DAG files from | `nil` |
| `airflow.cloneDagFilesFromGit.revision` | Revision from repository to checkout | `nil` |
| `airflow.baseUrl` | URL used to access to airflow web ui | `nil` |
| `airflow.worker.port` | Airflow Worker port | `8793` |
| `airflow.worker.replicas` | Number of Airflow Worker replicas | `2` |
| `airflow.auth.forcePassword` | Force users to specify a password | `false` |
| `airflow.auth.username` | Username to access web UI | `user` |
| `airflow.auth.password` | Password to access web UI | `nil` |
| `airflow.auth.fernetKey` | Fernet key to secure connections | `nil` |
| `airflow.auth.existingSecret` | Name of an existing secret containing airflow password and fernet key | `nil` |
| `airflow.extraEnvVars` | Extra environment variables to add to airflow web, worker and scheduler pods | `nil` |
| `securityContext.enabled` | Enable security context | `true` |
| `securityContext.fsGroup` | Group ID for the container | `1001` |
| `securityContext.runAsUser` | User ID for the container | `1001` |
| `service.type` | Kubernetes Service type | `ClusterIP` |
| `service.port` | Airflow Web port | `8080` |
| `service.nodePort` | Kubernetes Service nodePort | `nil` |
| `service.loadBalancerIP` | loadBalancerIP for Kafka Service | `nil` |
| `service.annotations` | Service annotations | `` |
| `ingress.enabled` | Enable ingress controller resource | `false` |
| `ingress.certManager` | Add annotations for cert-manager | `false` |
| `ingress.annotations` | Ingress annotations | `[]` |
| `ingress.hosts[0].name` | Hostname to your Wordpress installation | `airflow.local` |
| `ingress.hosts[0].path` | Path within the url structure | `/` |
| `ingress.tls[0].hosts[0]` | TLS hosts | `airflow.local` |
| `ingress.tls[0].secretName` | TLS Secret (certificates) | `airflow.local-tls` |
| `ingress.secrets[0].name` | TLS Secret Name | `nil` |
| `ingress.secrets[0].certificate` | TLS Secret Certificate | `nil` |
| `ingress.secrets[0].key` | TLS Secret Key | `nil` |
| `nodeSelector` | Node labels for pod assignment | `{}` |
| `tolerations` | Toleration labels for pod assignment | `[]` |
| `affinity` | Map of node/pod affinities | `{}` |
| `resources` | CPU/Memory resource requests/limits | Memory: `256Mi`, CPU: `250m` |
| `livenessProbe.enabled` | would you like a livessProbed to be enabled | `true` |
| `livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | 180 |
| `livenessProbe.periodSeconds` | How often to perform the probe | 20 |
| `livenessProbe.timeoutSeconds` | When the probe times out | 5 |
| `livenessProbe.failureThreshold` | Minimum consecutive failures for the probe to be considered failed after having succeeded. | 6 |
| `livenessProbe.successThreshold` | Minimum consecutive successes for the probe to be considered successful after having failed | 1 |
| `readinessProbe.enabled` | would you like a readinessProbe to be enabled | `true` |
| `readinessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | 30 |
| `readinessProbe.periodSeconds` | How often to perform the probe | 10 |
| `readinessProbe.timeoutSeconds` | When the probe times out | 5 |
| `readinessProbe.failureThreshold` | Minimum consecutive failures for the probe to be considered failed after having succeeded. | 6 |
| `readinessProbe.successThreshold` | Minimum consecutive successes for the probe to be considered successful after having failed | 1 |
| `postgresql.enabled` | Switch to enable or disable the PostgreSQL helm chart | `true` |
| `postgresql.postgresqlUsername` | Airflow Postgresql username | `bn_airflow` |
| `postgresql.postgresqlPassword` | Airflow Postgresql password | `nil` |
| `postgresql.postgresqlDatabase` | Airflow Postgresql database | `bitnami_airflow` |
| `externalDatabase.host` | External PostgreSQL host | `nil` |
| `externalDatabase.user` | External PostgreSQL user | `nil` |
| `externalDatabase.password` | External PostgreSQL password | `nil` |
| `externalDatabase.database` | External PostgreSQL database name | `nil` |
| `externalDatabase.port` | External PostgreSQL port | `nil` |
| `redis.enabled` | Switch to enable or disable the Redis helm chart | `true` |
| `externalRedis.host` | External Redis host | `nil` |
| `externalRedis.port` | External Redis port | `nil` |
| `externalRedis.password` | External Redis password | `nil` |
| `metrics.enabled` | Start a side-car prometheus exporter | `false` |
| `metrics.image.registry` | Airflow exporter image registry | `docker.io` |
| `metrics.image.repository` | Airflow exporter image name | `pbweb/airflow-prometheus-exporter` |
| `metrics.image.tag` | Airflow exporter image tag | `latest` |
| `metrics.image.pullPolicy` | Image pull policy | `IfNotPresent` |
| `metrics.image.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |
| `metrics.podAnnotations` | Additional annotations for Metrics exporter | `{prometheus.io/scrape: "true", prometheus.io/port: "9112"}` |
Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example,
```console
$ helm install --name my-release \
--set airflow.auth.username=my-user \
--set airflow.auth.password=my-passsword \
--set airflow.auth.fernetKey=my-fernet-key \
bitnami/airflow
```
The above command sets the credentials to access the Airflow web UI.
Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,
```console
$ helm install --name my-release -f values.yaml bitnami/airflow
```
> **Tip**: You can use the default [values.yaml](values.yaml)
## Persistence
The Bitnami Airflow chart relies on the PostgreSQL chart persistence. This means that Airflow does not persist anything.
## Generate a Fernet key
A Fernet key is required in order to encrypt password within connections. The Fernet key must be a base64-encoded 32-byte key.
Learn how to generate one [here](https://bcb.github.io/airflow/fernet-key)
## Load DAG files
There are three different ways to load your custom DAG files into the Airflow chart. All of them are compatible so you can use more than one at the same time.
### Option 1: Load locally from the `files` folder
If you plan to deploy the chart from your filesystem, you can copy your DAG files inside the `files/dags` directory. A config map will be created with those files and it will be mounted in all airflow nodes..
### Option 2: Specify an existing config map
You can manually create a config map containing all your DAG files and then pass the name when deploying Airflow chart. For that, you can pass the option `--set airflow.dagsConfigMap`.
### Option 3: Get your DAG files from a git repository
You can store all your DAG files on a GitHub repository and then clone to the Airflow pods with an initContainer. In order to do that, you can deploy airflow with the following options:
```console
helm install --name my-release bitnami/airflow \
--set airflow.cloneDagFilesFromGit.enabled=true \
--set airflow.cloneDagFilesFromGit.repository=https://github.com/USERNAME/REPOSITORY \
--set airflow.cloneDagFilesFromGit.revision=master
```

1
bitnami/airflow/files/dags/REAMDE.md Normal file
View File

@@ -0,0 +1 @@
You can copy here your DAGs files so they are mounted at "/opt/bitnami/airflow/dags" inside the docker image.

9
bitnami/airflow/requirements.lock Normal file
View File

@@ -0,0 +1,9 @@
dependencies:
- name: postgresql
repository: https://charts.bitnami.com/bitnami
version: 3.18.3
- name: redis
repository: https://charts.bitnami.com/bitnami
version: 7.0.1
digest: sha256:cf902ebdc6a871b1414ad64c95abb2fecf188b6cb0209bb91995620f84ca35ca
generated: 2019-05-02T11:16:02.72367469+02:00

9
bitnami/airflow/requirements.yaml Normal file
View File

@@ -0,0 +1,9 @@
dependencies:
- name: postgresql
version: 3.x.x
repository: https://charts.bitnami.com/bitnami
condition: postgresql.enabled
- name: redis
version: 7.x.x
repository: https://charts.bitnami.com/bitnami
condition: redis.enabled

78
bitnami/airflow/templates/NOTES.txt Normal file
View File

@@ -0,0 +1,78 @@
{{- if not .Values.airflow.baseUrl -}}
###############################################################################
### ERROR: You did not provide an external URL in your 'helm install' call ###
###############################################################################
This deployment will be incomplete until you configure Airflow with a resolvable
host. To configure Airflow with the URL of your service:
{{- if .Values.ingress.enabled }}
1. Get the Airflow URL indicated on the Ingress Rule and associate it to your cluster external IP:
export CLUSTER_IP=$(minikube ip) # On Minikube. Use: `kubectl cluster-info` on others K8s clusters
export HOSTNAME=$(kubectl get ingress --namespace {{ .Release.Namespace }} {{ template "airflow.fullname" . }} -o jsonpath='{.spec.rules[0].host}')
echo "Airflow URL: http://$HOSTNAME/"
echo "$CLUSTER_IP $HOSTNAME" | sudo tee -a /etc/hosts
{{- else }}
1. Get the Airflow URL by running:
{{- if contains "NodePort" .Values.service.type }}
export APP_HOST=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
export APP_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ template "airflow.fullname" . }})
{{- else if contains "ClusterIP" .Values.service.type }}
export APP_HOST=127.0.0.1
export APP_PORT=8080
{{- else if contains "LoadBalancer" .Values.service.type }}
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
Watch the status with: 'kubectl get svc --namespace {{ .Release.Namespace }} -w {{ template "airflow.fullname" . }}'
export APP_HOST=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ template "airflow.fullname" . }} --template "{{ "{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}" }}")
export APP_PORT=80
{{- end }}
export APP_PASSWORD=$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "airflow.fullname" . }} -o jsonpath="{.data.airflow-password}" | base64 --decode)
export APP_DATABASE_PASSWORD=$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "airflow.postgresql.fullname" . }} -o jsonpath="{.data.postgresql-password}" | base64 --decode)
export APP_REDIS_PASSWORD=$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "airflow.redis.secretName" . }} -o jsonpath="{.data.redis-password}" | base64 --decode)
{{- end }}
2. Complete your Airflow deployment by running:
helm upgrade {{ .Release.Name }} bitnami/{{ .Chart.Name }} \
--set service.type={{ .Values.service.type }},airflow.baseUrl=http://$APP_HOST:$APP_PORT,airflow.auth.password=$APP_PASSWORD,postgresql.postgresqlPassword=$APP_DATABASE_PASSWORD,redis.password=$APP_REDIS_PASSWORD{{- if .Values.global }}{{- if .Values.global.imagePullSecrets }},global.imagePullSecrets={{ .Values.global.imagePullSecrets }}{{- end }}{{- end }}
{{- else -}}
1. Get the Airflow URL by running:
{{- if .Values.ingress.enabled }}
echo URL : {{ .Values.airflow.baseUrl }}
{{- else if eq .Values.service.type "ClusterIP" }}
echo URL : http://127.0.0.1:{{ default "8080" .Values.service.port }}
kubectl port-forward --namespace {{ .Release.Namespace }} svc/{{ template "airflow.fullname" . }} {{ default "8080" .Values.service.port }}:8080
{{- else if eq .Values.servicePort "nodePort" }}
export APP_HOST=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
export APP_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ template "airflow.fullname" . }})
echo URL : http://$APP_HOST:$APP_PORT
{{- else }}
echo URL : {{ .Values.airflow.baseUrl }}
{{- end }}
2. Get your Airflow login credentials by running:
echo User: {{ .Values.airflow.auth.username }}
echo Password: $(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "airflow.fullname" . }} -o jsonpath="{.data.airflow-password}" | base64 --decode)
{{- end }}

236
bitnami/airflow/templates/_helpers.tpl Normal file
View File

@@ -0,0 +1,236 @@
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "airflow.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 24 -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "airflow.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "airflow.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Return the proper Airflow image name
*/}}
{{- define "airflow.image" -}}
{{- $registryName := .Values.image.registry -}}
{{- $repositoryName := .Values.image.repository -}}
{{- $tag := .Values.image.tag | toString -}}
{{/*
Helm 2.11 supports the assignment of a value to a variable defined in a different scope,
but Helm 2.9 and 2.10 doesn't support it, so we need to implement this if-else logic.
Also, we can't use a single if because lazy evaluation is not an option
*/}}
{{- if .Values.global }}
{{- if .Values.global.imageRegistry }}
{{- printf "%s/%s:%s" .Values.global.imageRegistry $repositoryName $tag -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- end -}}
{{/*
Return the proper Airflow Scheduler image name
*/}}
{{- define "airflow.schedulerImage" -}}
{{- $registryName := .Values.schedulerImage.registry -}}
{{- $repositoryName := .Values.schedulerImage.repository -}}
{{- $tag := .Values.schedulerImage.tag | toString -}}
{{/*
Helm 2.11 supports the assignment of a value to a variable defined in a different scope,
but Helm 2.9 and 2.10 doesn't support it, so we need to implement this if-else logic.
Also, we can't use a single if because lazy evaluation is not an option
*/}}
{{- if .Values.global }}
{{- if .Values.global.imageRegistry }}
{{- printf "%s/%s:%s" .Values.global.imageRegistry $repositoryName $tag -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- end -}}
{{/*
Return the proper Airflow Worker image name
*/}}
{{- define "airflow.workerImage" -}}
{{- $registryName := .Values.workerImage.registry -}}
{{- $repositoryName := .Values.workerImage.repository -}}
{{- $tag := .Values.workerImage.tag | toString -}}
{{/*
Helm 2.11 supports the assignment of a value to a variable defined in a different scope,
but Helm 2.9 and 2.10 doesn't support it, so we need to implement this if-else logic.
Also, we can't use a single if because lazy evaluation is not an option
*/}}
{{- if .Values.global }}
{{- if .Values.global.imageRegistry }}
{{- printf "%s/%s:%s" .Values.global.imageRegistry $repositoryName $tag -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- end -}}
{{/*
Return the proper git image name
*/}}
{{- define "git.image" -}}
{{- $registryName := .Values.git.registry -}}
{{- $repositoryName := .Values.git.repository -}}
{{- $tag := .Values.git.tag | toString -}}
{{/*
Helm 2.11 supports the assignment of a value to a variable defined in a different scope,
but Helm 2.9 and 2.10 doesn't support it, so we need to implement this if-else logic.
Also, we can't use a single if because lazy evaluation is not an option
*/}}
{{- if .Values.global }}
{{- if .Values.global.imageRegistry }}
{{- printf "%s/%s:%s" .Values.global.imageRegistry $repositoryName $tag -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- end -}}
{{/*
Return the proper Airflow Metrics image name
*/}}
{{- define "airflow.metrics.image" -}}
{{- $registryName := .Values.metrics.image.registry -}}
{{- $repositoryName := .Values.metrics.image.repository -}}
{{- $tag := .Values.metrics.image.tag | toString -}}
{{/*
Helm 2.11 supports the assignment of a value to a variable defined in a different scope,
but Helm 2.9 and 2.10 doesn't support it, so we need to implement this if-else logic.
Also, we can't use a single if because lazy evaluation is not an option
*/}}
{{- if .Values.global }}
{{- if .Values.global.imageRegistry }}
{{- printf "%s/%s:%s" .Values.global.imageRegistry $repositoryName $tag -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- else -}}
{{- printf "%s/%s:%s" $registryName $repositoryName $tag -}}
{{- end -}}
{{- end -}}
{{/*
Return the proper Docker Image Registry Secret Names
*/}}
{{- define "airflow.imagePullSecrets" -}}
{{/*
Helm 2.11 supports the assignment of a value to a variable defined in a different scope,
but Helm 2.9 and 2.10 does not support it, so we need to implement this if-else logic.
Also, we can not use a single if because lazy evaluation is not an option
*/}}
{{- if .Values.global }}
{{- if .Values.global.imagePullSecrets }}
imagePullSecrets:
{{- range .Values.global.imagePullSecrets }}
- name: {{ . }}
{{- end }}
{{- else if or .Values.image.pullSecrets .Values.schedulerImage.pullSecrets .Values.workerImage.pullSecrets .Values.git.pullSecrets .Values.metrics.image.pullSecrets }}
imagePullSecrets:
{{- range .Values.image.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- range .Values.schedulerImage.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- range .Values.workerImage.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- range .Values.git.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- range .Values.metrics.image.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- end -}}
{{- else if or .Values.image.pullSecrets .Values.schedulerImage.pullSecrets .Values.workerImage.pullSecrets .Values.git.pullSecrets .Values.metrics.image.pullSecrets }}
imagePullSecrets:
{{- range .Values.image.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- range .Values.schedulerImage.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- range .Values.workerImage.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- range .Values.git.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- range .Values.metrics.image.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- end -}}
{{- end -}}
Create a default fully qualified postgresql name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "airflow.postgresql.fullname" -}}
{{- $name := default "postgresql" .Values.postgresql.nameOverride -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
Create a default fully qualified redis name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "airflow.redis.fullname" -}}
{{- $name := default "redis" .Values.redis.nameOverride -}}
{{- printf "%s-%s-master" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
Create a template for the redis secret
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "airflow.redis.secretName" -}}
{{- $name := default "redis" .Values.redis.nameOverride -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Get the secret name
*/}}
{{- define "airflow.secretName" -}}
{{- if .Values.airflow.auth.existingSecret -}}
{{- printf "%s" .Values.airflow.auth.existingSecret -}}
{{- else -}}
{{- printf "%s" (include "airflow.fullname" .) -}}
{{- end -}}
{{- end -}}

13
bitnami/airflow/templates/configmap-dags.yaml Normal file
View File

@@ -0,0 +1,13 @@
{{- if .Files.Glob "files/dags/*.py" }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "airflow.fullname" . }}-dag-files
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
data:
{{ (.Files.Glob "files/dags/*.py").AsConfig | indent 2 }}
{{ end }}

14
bitnami/airflow/templates/configmap.yaml Normal file
View File

@@ -0,0 +1,14 @@
{{- if .Values.config }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "airflow.fullname" . }}-configuration
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
data:
airflow.cfg: |-
{{ .Values.config | indent 4 }}
{{- end -}}

190
bitnami/airflow/templates/deployment-scheduler.yaml Normal file
View File

@@ -0,0 +1,190 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ template "airflow.fullname" . }}-scheduler
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/component: scheduler
spec:
selector:
matchLabels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: scheduler
replicas: {{ .Values.replicaCount }}
template:
metadata:
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: scheduler
{{- if .Values.podAnnotations }}
annotations:
{{ toYaml .Values.podAnnotations | nindent 8 }}
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector:
{{ toYaml .Values.nodeSelector | nindent 8 }}
{{- end -}}
{{- with .Values.affinity }}
affinity:
{{ toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{ toYaml . | nindent 8 }}
{{- end }}
spec:
{{- include "airflow.imagePullSecrets" . | nindent 6 }}
{{- if .Values.securityContext.enabled }}
securityContext:
fsGroup: {{ .Values.securityContext.fsGroup }}
runAsUser: {{ .Values.securityContext.runAsUser }}
{{- end }}
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
initContainers:
- name: git-clone-repository
image: "{{ template "git.image" . }}"
imagePullPolicy: {{ .Values.git.pullPolicy | quote }}
command: [ '/bin/sh', '-c' , 'git clone {{ .Values.airflow.cloneDagFilesFromGit.repository }} /dags && cd /dags && git checkout {{ .Values.airflow.cloneDagFilesFromGit.revision }}']
volumeMounts:
- name: git-cloned-dag-files
mountPath: /dags
{{- end }}
containers:
- name: airflow-scheduler
image: {{ template "airflow.schedulerImage" . }}
imagePullPolicy: {{ .Values.schedulerImage.pullPolicy | quote }}
env:
{{- if .Values.schedulerImage.debug }}
- name: BASH_DEBUG
value: "1"
- name: NAMI_DEBUG
value: "1"
- name: NAMI_LOG_LEVEL
value: "trace8"
{{- end }}
- name: AIRFLOW_DATABASE_NAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlDatabase }}
{{- else }}
value: {{ .Values.externalDatabase.database | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_USERNAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlUsername }}
{{- else }}
value: {{ .Values.externalDatabase.user | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_PASSWORD
valueFrom:
secretKeyRef:
{{- if .Values.postgresql.enabled }}
name: {{ template "airflow.postgresql.fullname" . }}
key: postgresql-password
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
key: db-password
{{- end }}
- name: AIRFLOW_DATABASE_HOST
{{- if .Values.postgresql.enabled }}
value: {{ template "airflow.postgresql.fullname" . }}
{{- else }}
value: {{ .Values.externalDatabase.host | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_PORT_NUMBER
{{- if .Values.postgresql.enabled }}
value: "5432"
{{- else }}
value: {{ .Values.externalDatabase.port | quote }}
{{- end }}
- name: REDIS_HOST
{{- if .Values.redis.enabled }}
value: {{ template "airflow.redis.fullname" . }}
{{- else }}
value: {{ .Values.externalRedis.host | quote }}
{{- end }}
- name: REDIS_PORT_NUMBER
{{- if .Values.redis.enabled }}
value: "6379"
{{- else }}
value: {{ .Values.externalRedis.port | quote }}
{{- end }}
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
{{- if .Values.redis.enabled }}
name: {{ template "airflow.redis.secretName" . }}
key: redis-password
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externalredis" }}
key: redis-password
{{- end }}
- name: AIRFLOW_EXECUTOR
value: "CeleryExecutor"
- name: AIRFLOW_FERNET_KEY
valueFrom:
secretKeyRef:
name: {{ if .Values.airflow.auth.existingSecret }}{{ .Values.airflow.auth.existingSecret }}{{ else }}{{ template "airflow.fullname" . }}{{ end }}
key: airflow-fernetKey
- name: AIRFLOW_WEBSERVER_HOST
value: {{ template "airflow.fullname" . }}
- name: AIRFLOW_LOAD_EXAMPLES
{{- if .Values.airflow.loadExamples }}
value: "yes"
{{- else }}
value: "no"
{{- end }}
{{- if .Values.airflow.extraEnvVars }}
{{ toYaml .Values.airflow.extraEnvVars | nindent 8 }}
{{- end }}
{{- if .Values.livenessProbe.enabled }}
livenessProbe:
{{- end }}
{{- if .Values.readinessProbe.enabled }}
readinessProbe:
{{- end }}
volumeMounts:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
mountPath: /opt/bitnami/airflow/dags/local
{{- end }}
{{- if .Values.airflow.dagsConfigMap }}
- name: external-dag-files
mountPath: /opt/bitnami/airflow/dags/external
{{- end }}
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
- name: git-cloned-dag-files
mountPath: /opt/bitnami/airflow/dags/git
{{- end }}
{{- if .Values.airflow.configurationConfigMap }}
- name: custom-configuration-file
mountPath: /opt/bitnami/airflow/airflow.cfg
subPath: airflow.cfg
{{- end }}
resources:
{{ toYaml .Values.resources | indent 10 }}
volumes:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
configMap:
name: {{ template "airflow.fullname" . }}-dag-files
{{- end }}
{{- if .Values.airflow.dagsConfigMap }}
- name: external-dag-files
configMap:
name: {{ .Values.airflow.dagsConfigMap }}
{{- end }}
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
- name: git-cloned-dag-files
emptyDir: {}
{{- end }}
{{- if .Values.airflow.configurationConfigMap }}
- name: custom-configuration-file
configMap:
name: {{ .Values.airflow.configurationConfigMap }}
{{- end }}

220
bitnami/airflow/templates/deployment-web.yaml Normal file
View File

@@ -0,0 +1,220 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ template "airflow.fullname" . }}-web
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/component: web
spec:
selector:
matchLabels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: web
replicas: {{ .Values.replicaCount }}
template:
metadata:
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: web
{{- if .Values.podAnnotations }}
annotations:
{{ toYaml .Values.podAnnotations | nindent 8 }}
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector:
{{ toYaml .Values.nodeSelector | nindent 8 }}
{{- end -}}
{{- with .Values.affinity }}
affinity:
{{ toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{ toYaml . | nindent 8 }}
{{- end }}
spec:
{{- include "airflow.imagePullSecrets" . | nindent 6 }}
{{- if .Values.securityContext.enabled }}
securityContext:
fsGroup: {{ .Values.securityContext.fsGroup }}
runAsUser: {{ .Values.securityContext.runAsUser }}
{{- end }}
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
initContainers:
- name: git-clone-repository
image: "{{ template "git.image" . }}"
imagePullPolicy: {{ .Values.git.pullPolicy | quote }}
command: [ '/bin/sh', '-c' , 'git clone {{ .Values.airflow.cloneDagFilesFromGit.repository }} /dags && cd /dags && git checkout {{ .Values.airflow.cloneDagFilesFromGit.revision }}']
volumeMounts:
- name: git-cloned-dag-files
mountPath: /dags
{{- end }}
containers:
- name: airflow-web
image: {{ template "airflow.image" . }}
imagePullPolicy: {{ .Values.image.pullPolicy | quote }}
env:
{{- if .Values.image.debug }}
- name: BASH_DEBUG
value: "1"
- name: NAMI_DEBUG
value: "1"
- name: NAMI_LOG_LEVEL
value: "trace8"
{{- end }}
- name: AIRFLOW_DATABASE_NAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlDatabase }}
{{- else }}
value: {{ .Values.externalDatabase.database | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_USERNAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlUsername }}
{{- else }}
value: {{ .Values.externalDatabase.user | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_PASSWORD
valueFrom:
secretKeyRef:
{{- if .Values.postgresql.enabled }}
name: {{ template "airflow.postgresql.fullname" . }}
key: postgresql-password
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
key: db-password
{{- end }}
- name: AIRFLOW_DATABASE_HOST
{{- if .Values.postgresql.enabled }}
value: {{ template "airflow.postgresql.fullname" . }}
{{- else }}
value: {{ .Values.externalDatabase.host | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_PORT_NUMBER
{{- if .Values.postgresql.enabled }}
value: "5432"
{{- else }}
value: {{ .Values.externalDatabase.port | quote }}
{{- end }}
- name: REDIS_HOST
{{- if .Values.redis.enabled }}
value: {{ template "airflow.redis.fullname" . }}
{{- else }}
value: {{ .Values.externalRedis.host | quote }}
{{- end }}
- name: REDIS_PORT_NUMBER
{{- if .Values.redis.enabled }}
value: "6379"
{{- else }}
value: {{ .Values.externalRedis.port | quote }}
{{- end }}
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
{{- if .Values.redis.enabled }}
name: {{ template "airflow.redis.secretName" . }}
key: redis-password
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externalredis" }}
key: redis-password
{{- end }}
- name: AIRFLOW_EXECUTOR
value: "CeleryExecutor"
- name: AIRFLOW_USERNAME
value: {{ .Values.airflow.auth.username }}
- name: AIRFLOW_PASSWORD
valueFrom:
secretKeyRef:
name: {{ if .Values.airflow.auth.existingSecret }}{{ .Values.airflow.auth.existingSecret }}{{ else }}{{ template "airflow.fullname" . }}{{ end }}
key: airflow-password
- name: AIRFLOW_FERNET_KEY
valueFrom:
secretKeyRef:
name: {{ if .Values.airflow.auth.existingSecret }}{{ .Values.airflow.auth.existingSecret }}{{ else }}{{ template "airflow.fullname" . }}{{ end }}
key: airflow-fernetKey
- name: AIRFLOW_WEBSERVER_HOST
value: {{ template "airflow.fullname" . }}
{{- if .Values.airflow.baseUrl }}
- name: AIRFLOW_BASE_URL
value: {{ .Values.airflow.baseUrl }}
{{- end }}
- name: AIRFLOW_LOAD_EXAMPLES
{{- if .Values.airflow.loadExamples }}
value: "yes"
{{- else }}
value: "no"
{{- end }}
{{- if .Values.airflow.extraEnvVars }}
{{ toYaml .Values.airflow.extraEnvVars | nindent 8 }}
{{- end }}
ports:
- name: http
containerPort: 8080
{{- if .Values.livenessProbe.enabled }}
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
successThreshold: {{ .Values.livenessProbe.successThreshold }}
failureThreshold: {{ .Values.livenessProbe.failureThreshold }}
{{- end }}
{{- if .Values.readinessProbe.enabled }}
readinessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
successThreshold: {{ .Values.readinessProbe.successThreshold }}
failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
{{- end }}
volumeMounts:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
mountPath: /opt/bitnami/airflow/dags/local
{{- end }}
{{- if .Values.airflow.dagsConfigMap }}
- name: external-dag-files
mountPath: /opt/bitnami/airflow/dags/external
{{- end }}
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
- name: git-cloned-dag-files
mountPath: /opt/bitnami/airflow/dags/git
{{- end }}
{{- if .Values.airflow.configurationConfigMap }}
- name: custom-configuration-file
mountPath: /opt/bitnami/airflow/airflow.cfg
subPath: airflow.cfg
{{- end }}
resources:
{{ toYaml .Values.resources | indent 10 }}
volumes:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
configMap:
name: {{ template "airflow.fullname" . }}-dag-files
{{- end }}
{{- if .Values.airflow.dagsConfigMap }}
- name: external-dag-files
configMap:
name: {{ .Values.airflow.dagsConfigMap }}
{{- end }}
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
- name: git-cloned-dag-files
emptyDir: {}
{{- end }}
{{- if .Values.airflow.configurationConfigMap }}
- name: custom-configuration-file
configMap:
name: {{ .Values.airflow.configurationConfigMap }}
{{- end }}

33
bitnami/airflow/templates/ingress.yaml Normal file
View File

@@ -0,0 +1,33 @@
{{- if .Values.ingress.enabled }}
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: {{ template "airflow.fullname" . }}
labels:
app: "{{ template "airflow.fullname" . }}"
chart: "{{ template "airflow.chart" . }}"
release: {{ .Release.Name | quote }}
heritage: {{ .Release.Service | quote }}
annotations:
{{- if .Values.ingress.certManager }}
kubernetes.io/tls-acme: "true"
{{- end }}
{{- range $key, $value := .Values.ingress.annotations }}
{{ $key }}: {{ $value | quote }}
{{- end }}
spec:
rules:
{{- range .Values.ingress.hosts }}
- host: {{ .name }}
http:
paths:
- path: {{ default "/" .path }}
backend:
serviceName: "{{ template "airflow.fullname" $ }}"
servicePort: http
{{- end }}
{{- if .Values.ingress.tls }}
tls:
{{ toYaml .Values.ingress.tls | indent 4 }}
{{- end }}
{{- end }}

73
bitnami/airflow/templates/metrics-deployment.yaml Normal file
View File

@@ -0,0 +1,73 @@
{{- if .Values.metrics.enabled }}
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: {{ template "airflow.fullname" . }}-exporter
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/component: metrics
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
template:
metadata:
annotations:
{{ toYaml .Values.metrics.podAnnotations | indent 8 }}
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/component: metrics
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
{{- include "airflow.imagePullSecrets" . | indent 6 }}
containers:
- image: {{ template "airflow.metrics.image" . }}
name: airflow-exporter
env:
- name: AIRFLOW_PROMETHEUS_DATABASE_BACKEND
value: "postgres"
- name: AIRFLOW_PROMETHEUS_DATABASE_HOST
{{- if .Values.postgresql.enabled }}
value: {{ template "airflow.postgresql.fullname" . }}
{{- else }}
value: {{ .Values.externalDatabase.host | quote }}
{{- end }}
- name: AIRFLOW_PROMETHEUS_DATABASE_PORT
{{- if .Values.postgresql.enabled }}
value: "5432"
{{- else }}
value: {{ .Values.externalDatabase.port | quote }}
{{- end }}
- name: AIRFLOW_PROMETHEUS_DATABASE_USER
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlUsername }}
{{- else }}
value: {{ .Values.externalDatabase.user | quote }}
{{- end }}
- name: AIRFLOW_PROMETHEUS_DATABASE_PASSWORD
valueFrom:
secretKeyRef:
{{- if .Values.postgresql.enabled }}
name: {{ template "airflow.postgresql.fullname" . }}
key: postgresql-password
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
key: db-password
{{- end }}
- name: AIRFLOW_PROMETHEUS_DATABASE_NAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlDatabase }}
{{- else }}
value: {{ .Values.externalDatabase.database | quote }}
{{- end }}
ports:
- name: metrics
containerPort: 9112
resources:
{{ toYaml .Values.metrics.resources | indent 10 }}
{{- end }}

22
bitnami/airflow/templates/metrics-svc.yaml Normal file
View File

@@ -0,0 +1,22 @@
{{- if .Values.metrics.enabled }}
apiVersion: v1
kind: Service
metadata:
name: {{ template "airflow.fullname" . }}-exporter
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/component: metrics
spec:
type: ClusterIP
ports:
- name: metrics
port: 9112
targetPort: metrics
selector:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: metrics
{{- end }}

14
bitnami/airflow/templates/secret-externaldb.yaml Normal file
View File

@@ -0,0 +1,14 @@
{{- if not .Values.postgresql.enabled }}
apiVersion: v1
kind: Secret
metadata:
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
type: Opaque
data:
db-password: {{ .Values.externalDatabase.password | b64enc | quote }}
{{- end }}

14
bitnami/airflow/templates/secret-externalredis.yaml Normal file
View File

@@ -0,0 +1,14 @@
{{- if not .Values.redis.enabled }}
apiVersion: v1
kind: Secret
metadata:
name: {{ printf "%s-%s" .Release.Name "externalredis" }}
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
type: Opaque
data:
redis-password: {{ .Values.externalRedis.password | b64enc | quote }}
{{- end }}

27
bitnami/airflow/templates/secrets.yaml Normal file
View File

@@ -0,0 +1,27 @@
{{- if (not .Values.airflow.auth.existingSecret) -}}
apiVersion: v1
kind: Secret
metadata:
name: {{ template "airflow.fullname" . }}
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
type: Opaque
data:
{{- if .Values.airflow.auth.password }}
airflow-password: {{ .Values.airflow.auth.password | b64enc | quote }}
{{ else if (not .Values.airflow.auth.forcePassword) }}
airflow-password: {{ randAlphaNum 10 | b64enc | quote }}
{{ else }}
airflow-password: {{ required "An Airflow Password is required!" .Values.airflow.auth.password }}
{{- end }}
{{- if .Values.airflow.auth.fernetKey }}
airflow-fernetKey: {{ .Values.airflow.auth.fernetKey | b64enc | quote }}
{{ else if (not .Values.airflow.auth.forcePassword) }}
airflow-fernetKey: {{ randAlphaNum 32 | b64enc | b64enc | quote }}
{{ else }}
airflow-fernetKey: {{ required "An Airflow Fernet Key is required!" .Values.airflow.auth.fernetKey }}
{{- end }}
{{- end }}

205
bitnami/airflow/templates/statefulset-worker.yaml Normal file
View File

@@ -0,0 +1,205 @@
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
name: {{ template "airflow.fullname" . }}-worker
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/component: worker
spec:
selector:
matchLabels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: worker
serviceName: {{ template "airflow.fullname" . }}-headless
replicas: {{ .Values.airflow.worker.replicas }}
updateStrategy:
type: {{ .Values.updateStrategy | quote }}
{{- if (eq "Recreate" .Values.updateStrategy) }}
rollingUpdate: null
{{- else if .Values.rollingUpdatePartition }}
rollingUpdate:
partition: {{ .Values.rollingUpdatePartition }}
{{- end }}
template:
metadata:
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: worker
spec:
{{- include "airflow.imagePullSecrets" . | indent 6 }}
{{- if .Values.securityContext.enabled }}
securityContext:
fsGroup: {{ .Values.securityContext.fsGroup }}
runAsUser: {{ .Values.securityContext.runAsUser }}
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
{{- end }}
{{- if .Values.tolerations }}
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
{{- end }}
{{- if .Values.affinity }}
affinity:
{{ toYaml .Values.affinity | indent 8 }}
{{- end }}
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
initContainers:
- name: git-clone-repository
image: "{{ template "git.image" . }}"
imagePullPolicy: {{ .Values.git.pullPolicy | quote }}
command: [ '/bin/sh', '-c' , 'git clone {{ .Values.airflow.cloneDagFilesFromGit.repository }} /dags && cd /dags && git checkout {{ .Values.airflow.cloneDagFilesFromGit.revision }}']
volumeMounts:
- name: git-cloned-dag-files
mountPath: /dags
{{- end }}
containers:
- name: airflow-worker
image: "{{ template "airflow.workerImage" . }}"
imagePullPolicy: "{{ .Values.workerImage.pullPolicy }}"
resources:
{{ toYaml .Values.resources | indent 10 }}
env:
{{- if .Values.workerImage.debug }}
- name: BASH_DEBUG
value: "1"
- name: NAMI_DEBUG
value: "1"
- name: NAMI_LOG_LEVEL
value: "trace8"
{{- end }}
- name: AIRFLOW_DATABASE_NAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlDatabase }}
{{- else }}
value: {{ .Values.externalDatabase.database | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_USERNAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlUsername }}
{{- else }}
value: {{ .Values.externalDatabase.user | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_PASSWORD
valueFrom:
secretKeyRef:
{{- if .Values.postgresql.enabled }}
name: {{ template "airflow.postgresql.fullname" . }}
key: postgresql-password
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
key: db-password
{{- end }}
- name: AIRFLOW_DATABASE_HOST
{{- if .Values.postgresql.enabled }}
value: {{ template "airflow.postgresql.fullname" . }}
{{- else }}
value: {{ .Values.externalDatabase.host | quote }}
{{- end }}
- name: AIRFLOW_DATABASE_PORT_NUMBER
{{- if .Values.postgresql.enabled }}
value: "5432"
{{- else }}
value: {{ .Values.externalDatabase.port | quote }}
{{- end }}
- name: REDIS_HOST
{{- if .Values.redis.enabled }}
value: {{ template "airflow.redis.fullname" . }}
{{- else }}
value: {{ .Values.externalRedis.host | quote }}
{{- end }}
- name: REDIS_PORT_NUMBER
{{- if .Values.redis.enabled }}
value: "6379"
{{- else }}
value: {{ .Values.externalRedis.port | quote }}
{{- end }}
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
{{- if .Values.redis.enabled }}
name: {{ template "airflow.redis.secretName" . }}
key: redis-password
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externalredis" }}
key: redis-password
{{- end }}
- name: AIRFLOW_EXECUTOR
value: "CeleryExecutor"
- name: AIRFLOW_FERNET_KEY
valueFrom:
secretKeyRef:
name: {{ if .Values.airflow.auth.existingSecret }}{{ .Values.airflow.auth.existingSecret }}{{ else }}{{ template "airflow.fullname" . }}{{ end }}
key: airflow-fernetKey
- name: AIRFLOW_WEBSERVER_HOST
value: {{ template "airflow.fullname" . }}
{{- if .Values.airflow.extraEnvVars }}
{{ toYaml .Values.airflow.extraEnvVars | nindent 8 }}
{{- end }}
ports:
- name: worker
containerPort: {{ .Values.airflow.worker.port }}
{{- if .Values.livenessProbe.enabled }}
livenessProbe:
tcpSocket:
port: worker
initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
successThreshold: {{ .Values.livenessProbe.successThreshold }}
failureThreshold: {{ .Values.livenessProbe.failureThreshold }}
{{- end }}
{{- if .Values.readinessProbe.enabled }}
readinessProbe:
tcpSocket:
port: worker
initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
successThreshold: {{ .Values.readinessProbe.successThreshold }}
failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
{{- end }}
volumeMounts:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
mountPath: /opt/bitnami/airflow/dags/local
{{- end }}
{{- if .Values.airflow.dagsConfigMap }}
- name: external-dag-files
mountPath: /opt/bitnami/airflow/dags/external
{{- end }}
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
- name: git-cloned-dag-files
mountPath: /opt/bitnami/airflow/dags/git
{{- end }}
{{- if .Values.airflow.configurationConfigMap }}
- name: custom-configuration-file
mountPath: /opt/bitnami/airflow/airflow.cfg
subPath: airflow.cfg
{{- end }}
volumes:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
configMap:
name: {{ template "airflow.fullname" . }}-dag-files
{{- end }}
{{- if .Values.airflow.dagsConfigMap }}
- name: external-dag-files
configMap:
name: {{ .Values.airflow.dagsConfigMap }}
{{- end }}
{{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
- name: git-cloned-dag-files
emptyDir: {}
{{- end }}
{{- if .Values.airflow.configurationConfigMap }}
- name: custom-configuration-file
configMap:
name: {{ .Values.airflow.configurationConfigMap }}
{{- end }}

20
bitnami/airflow/templates/svc-headless-worker.yaml Normal file
View File

@@ -0,0 +1,20 @@
apiVersion: v1
kind: Service
metadata:
name: {{ template "airflow.fullname" . }}-headless
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
spec:
type: ClusterIP
clusterIP: None
ports:
- name: worker
port: 8793
targetPort: worker
selector:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: worker

31
bitnami/airflow/templates/svc.yaml Normal file
View File

@@ -0,0 +1,31 @@
apiVersion: v1
kind: Service
metadata:
name: {{ template "airflow.fullname" . }}
labels:
app.kubernetes.io/name: {{ include "airflow.name" . }}
helm.sh/chart: {{ include "airflow.chart" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
annotations:
{{- range $key, $value := .Values.service.annotations }}
{{ $key }}: {{ $value | quote }}
{{- end }}
spec:
type: {{ .Values.service.type }}
{{- if eq .Values.service.type "LoadBalancer" }}
{{- if .Values.service.loadBalancerIP }}
loadBalancerIP: {{ .Values.service.loadBalancerIP }}
{{- end }}
{{- end }}
ports:
- name: http
port: 8080
{{- if and .Values.service.nodePort (or (eq .Values.service.type "NodePort") (eq .Values.service.type "LoadBalancer")) }}
nodePort: {{ .Values.service.nodePort }}
{{- end }}
targetPort: http
selector:
app.kubernetes.io/name: {{ include "airflow.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: web

357
bitnami/airflow/values-production.yaml Normal file
View File

@@ -0,0 +1,357 @@
## Global Docker image parameters
## Please, note that this will override the image parameters, including dependencies, configured to use the global value
## Current available global Docker image parameters: imageRegistry and imagePullSecrets
##
# global:
# imageRegistry: myRegistryName
# imagePullSecrets:
# - myRegistryKeySecretName
## Bitnami Airflow image version
## ref: https://hub.docker.com/r/bitnami/airflow/tags/
##
image:
registry: docker.io
repository: bitnami/airflow
tag: 1.10.3
## Specify a imagePullPolicy
## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
##
pullPolicy: Always
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# pullSecrets:
# - myRegistryKeySecretName
## Set to true if you would like to see extra information on logs
## It turns BASH and NAMI debugging in minideb
## ref: https://github.com/bitnami/minideb-extras/#turn-on-bash-debugging
debug: false
## Bitnami Airflow Scheduler image version
## ref: https://hub.docker.com/r/bitnami/airflow-scheduler/tags/
##
schedulerImage:
registry: docker.io
repository: bitnami/airflow-scheduler
tag: 1.10.3
## Specify a imagePullPolicy
## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
##
pullPolicy: Always
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# pullSecrets:
# - myRegistryKeySecretName
## Set to true if you would like to see extra information on logs
## It turns BASH and NAMI debugging in minideb
## ref: https://github.com/bitnami/minideb-extras/#turn-on-bash-debugging
debug: false
## Bitnami Airflow Worker image version
## ref: https://hub.docker.com/r/bitnami/airflow-worker/tags/
##
workerImage:
registry: docker.io
repository: bitnami/airflow-worker
tag: 1.10.3
## Specify a imagePullPolicy
## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
##
pullPolicy: Always
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# pullSecrets:
# - myRegistryKeySecretName
## Set to true if you would like to see extra information on logs
## It turns BASH and NAMI debugging in minideb
## ref: https://github.com/bitnami/minideb-extras/#turn-on-bash-debugging
debug: false
## Bitnami git image version
## ref: https://hub.docker.com/r/bitnami/git/tags/
##
git:
registry: docker.io
repository: bitnami/git
tag: latest
pullPolicy: IfNotPresent
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# pullSecrets:
# - myRegistryKeySecretName
## StatefulSet controller supports automated updates. There are two valid update strategies: RollingUpdate and OnDelete
## ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets
##
updateStrategy: RollingUpdate
## Partition update strategy
## https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#partitions
##
# rollingUpdatePartition:
## Airflow Components configuration
##
airflow:
## Name of an existing config map containing the Airflow config file
##
# configurationConfigMap:
## Name of an existing config map containing all the DAGs files you want to load in Airflow.
##
# dagsConfigMap:
## Airflow generic configuration
##
loadExamples: false
## Enable in order to download DAG files from git repository.
##
cloneDagFilesFromGit:
enabled: false
repository:
revision:
## URL used to access to airflow web ui
##
baseUrl: http://airflow.local
## Airflow worker component configuration
worker:
port: 8793
replicas: 3
auth:
## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.
## If it is not forced, a random password will be generated.
##
forcePassword: true
## Specify username to acccess web UI
##
username: user
## Specify password to acccess web UI
##
# password:
## Specify a fernet key to secure airflow connections
## Fernet key must be 32 url-safe base64-encoded bytes.
## More info at:
## - https://airflow.readthedocs.io/en/stable/howto/secure-connections.html
## - https://bcb.github.io/airflow/fernet-key
##
# fernetKey:
## Use existing secret (ignores previous password and fernetKey)
##
# existingSecret:
## All the parameters from the configuration file can be overwritten by using environment variables with this format:
## AIRFLOW__{SECTION}__{KEY}. Note the double underscores. More info at https://airflow.readthedocs.io/en/stable/howto/set-config.html
##
# extraEnvVars:
# - name: AIRFLOW__SMTP__SMTP_HOST
# value: "localhost"
# - name: AIRFLOW__SMTP__SMTP_PORT
# value: "25"
# - name: AIRFLOW__SMTP__USER
# value: ""
## Kubernetes Security Context
## https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
##
securityContext:
enabled: true
fsGroup: 1001
runAsUser: 1001
## Kubernetes configuration
## For minikube, set this to NodePort, elsewhere use LoadBalancer
##
service:
type: ClusterIP
port: 8080
## Specify the NodePort value for the LoadBalancer and NodePort service types.
## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport
##
# nodePort:
## Use loadBalancerIP to request a specific static IP,
# loadBalancerIP:
## Service annotations done as key:value pairs
annotations:
## Configure the ingress resource that allows you to access the
## Airflow installation. Set up the URL
## ref: http://kubernetes.io/docs/user-guide/ingress/
##
ingress:
## Set to true to enable ingress record generation
enabled: true
## Set this to true in order to add the corresponding annotations for cert-manager
certManager: false
## Ingress annotations done as key:value pairs
## For a full list of possible ingress annotations, please see
## ref: https://github.com/kubernetes/ingress-nginx/blob/master/docs/annotations.md
##
## If tls is set to true, annotation ingress.kubernetes.io/secure-backends: "true" will automatically be set
## If certManager is set to true, annotation kubernetes.io/tls-acme: "true" will automatically be set
annotations:
# kubernetes.io/ingress.class: nginx
## The list of hostnames to be covered with this ingress record.
## Most likely this will be just one host, but in the event more hosts are needed, this is an array
hosts:
- name: airflow.local
path: /
## The tls configuration for the ingress
## see: https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
tls:
- hosts:
- airflow.local
secretName: airflow.local-tls
secrets:
## If you're providing your own certificates, please use this to add the certificates as secrets
## key and certificate should start with -----BEGIN CERTIFICATE----- or
## -----BEGIN RSA PRIVATE KEY-----
##
## name should line up with a tlsSecret set further up
## If you're using cert-manager, this is unneeded, as it will create the secret for you if it is not set
##
## It is also possible to create and manage the certificates outside of this helm chart
## Please see README.md for more information
# - name: airflow.local-tls
# key:
# certificate:
## Node labels for pod assignment
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}
## Tolerations for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
tolerations: []
## Affinity for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
##
affinity: {}
## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
# limits:
# cpu: 200m
# memory: 1Gi
# requests:
# memory: 256Mi
# cpu: 250m
## Configure extra options for liveness and readiness probes
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes)
livenessProbe:
enabled: true
initialDelaySeconds: 180
periodSeconds: 20
timeoutSeconds: 5
failureThreshold: 6
successThreshold: 1
readinessProbe:
enabled: true
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 6
successThreshold: 1
##
## PostgreSQL chart configuration
##
## https://github.com/helm/charts/blob/master/stable/postgresql/values.yaml
##
postgresql:
enabled: true
postgresqlUsername: bn_airflow
#postgresqlPassword:
postgresqlDatabase: bitnami_airflow
externalDatabase:
## All of these values are only used when postgresql.enabled is set to false
## Database host
host: localhost
## non-root Username for Airflow Database
user: bn_airflow
## Database password
password: ""
## Database name
database: bitnami_airflow
## Database port number
port: 5432
##
## Redis chart configuration
##
## https://github.com/helm/charts/blob/master/stable/redis/values.yaml
##
redis:
enabled: true
# password: ""
externalRedis:
## All of these values are only used when redis.enabled is set to false
## Redis host
host: localhost
## Redis port number
port: 6379
## Redis password
password: ""
## Prometheus Exporter / Metrics
##
metrics:
enabled: false
image:
registry: docker.io
repository: pbweb/airflow-prometheus-exporter
tag: latest
pullPolicy: IfNotPresent
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# pullSecrets:
# - myRegistryKeySecretName
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9112"
## Metrics exporter resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
# resources: {}
## Metrics exporter labels and tolerations for pod assignment
# nodeSelector: {"beta.kubernetes.io/arch": "amd64"}
# tolerations: []
## Metrics exporter pod Annotation and Labels
# podAnnotations: {}
# podLabels: {}

357
bitnami/airflow/values.yaml Normal file
View File

@@ -0,0 +1,357 @@
## Global Docker image parameters
## Please, note that this will override the image parameters, including dependencies, configured to use the global value
## Current available global Docker image parameters: imageRegistry and imagePullSecrets
##
# global:
# imageRegistry: myRegistryName
# imagePullSecrets:
# - myRegistryKeySecretName
## Bitnami Airflow image version
## ref: https://hub.docker.com/r/bitnami/airflow/tags/
##
image:
registry: docker.io
repository: bitnami/airflow
tag: 1.10.3
## Specify a imagePullPolicy
## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
##
pullPolicy: Always
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# pullSecrets:
# - myRegistryKeySecretName
## Set to true if you would like to see extra information on logs
## It turns BASH and NAMI debugging in minideb
## ref: https://github.com/bitnami/minideb-extras/#turn-on-bash-debugging
debug: false
## Bitnami Airflow Scheduler image version
## ref: https://hub.docker.com/r/bitnami/airflow-scheduler/tags/
##
schedulerImage:
registry: docker.io
repository: bitnami/airflow-scheduler
tag: 1.10.3
## Specify a imagePullPolicy
## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
##
pullPolicy: Always
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# pullSecrets:
# - myRegistryKeySecretName
## Set to true if you would like to see extra information on logs
## It turns BASH and NAMI debugging in minideb
## ref: https://github.com/bitnami/minideb-extras/#turn-on-bash-debugging
debug: false
## Bitnami Airflow Worker image version
## ref: https://hub.docker.com/r/bitnami/airflow-worker/tags/
##
workerImage:
registry: docker.io
repository: bitnami/airflow-worker
tag: 1.10.3
## Specify a imagePullPolicy
## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
##
pullPolicy: Always
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# pullSecrets:
# - myRegistryKeySecretName
## Set to true if you would like to see extra information on logs
## It turns BASH and NAMI debugging in minideb
## ref: https://github.com/bitnami/minideb-extras/#turn-on-bash-debugging
debug: false
## Bitnami git image version
## ref: https://hub.docker.com/r/bitnami/git/tags/
##
git:
registry: docker.io
repository: bitnami/git
tag: latest
pullPolicy: IfNotPresent
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# pullSecrets:
# - myRegistryKeySecretName
## StatefulSet controller supports automated updates. There are two valid update strategies: RollingUpdate and OnDelete
## ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets
##
updateStrategy: RollingUpdate
## Partition update strategy
## https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#partitions
##
# rollingUpdatePartition:
## Airflow Components configuration
##
airflow:
## Name of an existing config map containing the Airflow config file
##
# configurationConfigMap:
## Name of an existing config map containing all the DAGs files you want to load in Airflow.
##
dagsConfigMap: patata
## Airflow generic configuration
##
loadExamples: false
## Enable in order to download DAG files from git repository.
##
cloneDagFilesFromGit:
enabled: true
repository: https://github.com/tompizmor/sample-airflow-dag
revision: master
## URL used to access to airflow web ui
##
# baseUrl:
## Airflow worker component configuration
worker:
port: 8793
replicas: 1
auth:
## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.
## If it is not forced, a random password will be generated.
##
forcePassword: false
## Specify username to acccess web UI
##
username: user
## Specify password to acccess web UI
##
# password:
## Specify a fernet key to secure airflow connections
## Fernet key must be 32 url-safe base64-encoded bytes.
## More info at:
## - https://airflow.readthedocs.io/en/stable/howto/secure-connections.html
## - https://bcb.github.io/airflow/fernet-key
##
# fernetKey:
## Use existing secret (ignores previous password and fernetKey)
##
# existingSecret:
## All the parameters from the configuration file can be overwritten by using environment variables with this format:
## AIRFLOW__{SECTION}__{KEY}. Note the double underscores. More info at https://airflow.readthedocs.io/en/stable/howto/set-config.html
##
# extraEnvVars:
# - name: AIRFLOW__SMTP__SMTP_HOST
# value: "localhost"
# - name: AIRFLOW__SMTP__SMTP_PORT
# value: "25"
# - name: AIRFLOW__SMTP__USER
# value: ""
## Kubernetes Security Context
## https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
##
securityContext:
enabled: true
fsGroup: 1001
runAsUser: 1001
## Kubernetes configuration
## For minikube, set this to NodePort, elsewhere use LoadBalancer
##
service:
type: ClusterIP
port: 8080
## Specify the NodePort value for the LoadBalancer and NodePort service types.
## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport
##
# nodePort:
## Use loadBalancerIP to request a specific static IP,
# loadBalancerIP:
## Service annotations done as key:value pairs
annotations:
## Configure the ingress resource that allows you to access the
## Airflow installation. Set up the URL
## ref: http://kubernetes.io/docs/user-guide/ingress/
##
ingress:
## Set to true to enable ingress record generation
enabled: false
## Set this to true in order to add the corresponding annotations for cert-manager
certManager: false
## Ingress annotations done as key:value pairs
## For a full list of possible ingress annotations, please see
## ref: https://github.com/kubernetes/ingress-nginx/blob/master/docs/annotations.md
##
## If tls is set to true, annotation ingress.kubernetes.io/secure-backends: "true" will automatically be set
## If certManager is set to true, annotation kubernetes.io/tls-acme: "true" will automatically be set
annotations:
# kubernetes.io/ingress.class: nginx
## The list of hostnames to be covered with this ingress record.
## Most likely this will be just one host, but in the event more hosts are needed, this is an array
hosts:
- name: airflow.local
path: /
## The tls configuration for the ingress
## see: https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
tls:
- hosts:
- airflow.local
secretName: airflow.local-tls
secrets:
## If you're providing your own certificates, please use this to add the certificates as secrets
## key and certificate should start with -----BEGIN CERTIFICATE----- or
## -----BEGIN RSA PRIVATE KEY-----
##
## name should line up with a tlsSecret set further up
## If you're using cert-manager, this is unneeded, as it will create the secret for you if it is not set
##
## It is also possible to create and manage the certificates outside of this helm chart
## Please see README.md for more information
# - name: airflow.local-tls
# key:
# certificate:
## Node labels for pod assignment
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}
## Tolerations for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
tolerations: []
## Affinity for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
##
affinity: {}
## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
# limits:
# cpu: 200m
# memory: 1Gi
# requests:
# memory: 256Mi
# cpu: 250m
## Configure extra options for liveness and readiness probes
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes)
livenessProbe:
enabled: true
initialDelaySeconds: 180
periodSeconds: 20
timeoutSeconds: 5
failureThreshold: 6
successThreshold: 1
readinessProbe:
enabled: true
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 6
successThreshold: 1
##
## PostgreSQL chart configuration
##
## https://github.com/helm/charts/blob/master/stable/postgresql/values.yaml
##
postgresql:
enabled: true
postgresqlUsername: bn_airflow
#postgresqlPassword:
postgresqlDatabase: bitnami_airflow
externalDatabase:
## All of these values are only used when postgresql.enabled is set to false
## Database host
host: localhost
## non-root Username for Airflow Database
user: bn_airflow
## Database password
password: ""
## Database name
database: bitnami_airflow
## Database port number
port: 5432
##
## Redis chart configuration
##
## https://github.com/helm/charts/blob/master/stable/redis/values.yaml
##
redis:
enabled: true
# password: ""
externalRedis:
## All of these values are only used when redis.enabled is set to false
## Redis host
host: localhost
## Redis port number
port: 6379
## Redis password
password: ""
## Prometheus Exporter / Metrics
##
metrics:
enabled: false
image:
registry: docker.io
repository: pbweb/airflow-prometheus-exporter
tag: latest
pullPolicy: IfNotPresent
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# pullSecrets:
# - myRegistryKeySecretName
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9112"
## Metrics exporter resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
# resources: {}
## Metrics exporter labels and tolerations for pod assignment
# nodeSelector: {"beta.kubernetes.io/arch": "amd64"}
# tolerations: []
## Metrics exporter pod Annotation and Labels
# podAnnotations: {}
# podLabels: {}