Bitnami/charts
2
0
Fork 0
mirror of https://github.com/bitnami/charts.git synced 2026-03-04 06:47:57 +08:00
Code Issues Packages Projects Releases Wiki Activity

[bitnami/airflow] Fix cannot fetch logs in worker nodes (#7603)

Browse Source 
* Airflow containers requires to pass a new Airflow configuration param called AIRFLOW_SECRET_KEY

* This value is used by Airflow's task logs fetching from workers

* Update image values to the worker, scheduler and web containers

* These new container images fix a problem that prevented users from
  dowloading logs from workers through the UI

* Fix scheduler,worker and web images value

* Last README.md updates about the new value auth.secretKey and images

* Bump up chart version

* fix fernet and secret key env var name
This commit is contained in:
Roberto Andradas
2021-09-23 16:19:49 +02:00
committed by GitHub
parent 79eac4490b
commit ca00baab7a
6 changed files with 55 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
bitnami/airflow/Chart.yaml
View File

@@ -32,4 +32,4 @@ name: airflow
sources:
- https://github.com/bitnami/bitnami-docker-airflow
- https://airflow.apache.org/
version: 11.0.3
version: 11.0.4

41
bitnami/airflow/README.md
View File

@@ -89,17 +89,18 @@ The command removes all the Kubernetes components associated with the chart and
### Airflow common parameters
| Name | Description | Value |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- |
| `auth.existingSecret` | Name of an existing secret containing password and fernet key ('airflow-password and 'airflow-fernetKey' keys) | `""` |
| `auth.fernetKey` | Fernet key to secure connections | `""` |
| `auth.forcePassword` | Force users to specify a password | `false` |
| `auth.password` | Password to access web UI | `""` |
| `auth.username` | Username to access web UI | `user` |
| `configurationConfigMap` | Name of an existing config map containing the Airflow config file | `""` |
| `executor` | Airflow executor, it should be one of 'SequentialExecutor', 'LocalExecutor', 'CeleryExecutor', 'KubernetesExecutor', 'CeleryKubernetesExecutor' | `CeleryExecutor` |
| `dagsConfigMap` | Name of an existing config map containing all the DAGs files you want to load in Airflow | `""` |
| `loadExamples` | Switch to load some Airflow examples | `false` |
| Name | Description | Value |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------- |
| `auth.existingSecret` | Name of an existing secret containing password, fernet key and secret key ('airflow-password', 'airflow-fernetKey' and 'airflow-secretKey' keys) | `""` |
| `auth.fernetKey` | Fernet key to secure connections | `""` |
| `auth.forcePassword` | Force users to specify a password | `false` |
| `auth.password` | Password to access web UI | `""` |
| `auth.username` | Username to access web UI | `user` |
| `auth.secretKey` | Secret key to run your flask app | `""` |
| `configurationConfigMap` | Name of an existing config map containing the Airflow config file | `""` |
| `executor` | Airflow executor, it should be one of 'SequentialExecutor', 'LocalExecutor', 'CeleryExecutor', 'KubernetesExecutor', 'CeleryKubernetesExecutor' | `CeleryExecutor` |
| `dagsConfigMap` | Name of an existing config map containing all the DAGs files you want to load in Airflow | `""` |
| `loadExamples` | Switch to load some Airflow examples | `false` |
### Airflow web parameters
@@ -108,7 +109,7 @@ The command removes all the Kubernetes components associated with the chart and
| ---------------------------------------- | --------------------------------------------------------------------------- | -------------------- |
| `web.image.registry` | Airflow image registry | `docker.io` |
| `web.image.repository` | Airflow image repository | `bitnami/airflow` |
| `web.image.tag` | Airflow image tag (immutable tags are recommended) | `2.1.4-debian-10-r0` |
| `web.image.tag` | Airflow image tag (immutable tags are recommended) | `2.1.4-debian-10-r4` |
| `web.image.pullPolicy` | Airflow image pull policy | `IfNotPresent` |
| `web.image.pullSecrets` | Airflow image pull secrets | `[]` |
| `web.image.debug` | Enable image debug mode | `false` |
@@ -162,7 +163,7 @@ The command removes all the Kubernetes components associated with the chart and
| -------------------------------------------- | ----------------------------------------------------------------------- | --------------------------- |
| `scheduler.image.registry` | Airflow Scheduler image registry | `docker.io` |
| `scheduler.image.repository` | Airflow Scheduler image repository | `bitnami/airflow-scheduler` |
| `scheduler.image.tag` | Airflow Scheduler image tag (immutable tags are recommended) | `2.1.3-debian-10-r23` |
| `scheduler.image.tag` | Airflow Scheduler image tag (immutable tags are recommended) | `2.1.4-debian-10-r4` |
| `scheduler.image.pullPolicy` | Airflow Scheduler image pull policy | `IfNotPresent` |
| `scheduler.image.pullSecrets` | Airflow Scheduler image pull secrets | `[]` |
| `scheduler.image.debug` | Enable image debug mode | `false` |
@@ -195,7 +196,7 @@ The command removes all the Kubernetes components associated with the chart and
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ------------------------ |
| `worker.image.registry` | Airflow Worker image registry | `docker.io` |
| `worker.image.repository` | Airflow Worker image repository | `bitnami/airflow-worker` |
| `worker.image.tag` | Airflow Worker image tag (immutable tags are recommended) | `2.1.3-debian-10-r23` |
| `worker.image.tag` | Airflow Worker image tag (immutable tags are recommended) | `2.1.4-debian-10-r4` |
| `worker.image.pullPolicy` | Airflow Worker image pull policy | `IfNotPresent` |
| `worker.image.pullSecrets` | Airflow Worker image pull secrets | `[]` |
| `worker.image.debug` | Enable image debug mode | `false` |
@@ -370,6 +371,7 @@ $ helm install my-release \
--set auth.username=my-user \
--set auth.password=my-passsword \
--set auth.fernetKey=my-fernet-key \
--set auth.secretKey=my-secret-key \
bitnami/airflow
```
@@ -399,6 +401,10 @@ A Fernet key is required in order to encrypt password within connections. The Fe
Learn how to generate one [here](https://bcb.github.io/airflow/fernet-key)
### Generate a Secret key
Secret key used to run your flask app. It should be as random as possible. However, when running more than 1 instances of webserver, make sure all of them use the same secret_key otherwise one of them will error with “CSRF session token is missing”.
### Load DAG files
There are two different ways to load your custom DAG files into the Airflow chart. All of them are compatible so you can use more than one at the same time.
@@ -464,6 +470,7 @@ type: Opaque
data:
airflow-password: "Smo1QTJLdGxXMg=="
airflow-fernetKey: "YVRZeVJVWnlXbU4wY1dOalVrdE1SV3cxWWtKeFIzWkVRVTVrVjNaTFR6WT0="
airflow-secretKey: "a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08="
postgresql-password: "cG9zdGdyZXMK"
redis-password: "cmVkaXMK"
```
@@ -605,7 +612,8 @@ $ helm install airflow bitnami/airflow \
```console
$ export AIRFLOW_PASSWORD=$(kubectl get secret --namespace default airflow -o jsonpath="{.data.airflow-password}" | base64 --decode)
$ export AIRFLOW_FERNETKEY=$(kubectl get secret --namespace default airflow -o jsonpath="{.data.airflow-fernetKey}" | base64 --decode)
$ export AIRFLOW_FERNET_KEY=$(kubectl get secret --namespace default airflow -o jsonpath="{.data.airflow-fernetKey}" | base64 --decode)
$ export AIRFLOW_SECRET_KEY=$(kubectl get secret --namespace default airflow -o jsonpath="{.data.airflow-secretKey}" | base64 --decode)
$ export POSTGRESQL_PASSWORD=$(kubectl get secret --namespace default airflow-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
$ export REDIS_PASSWORD=$(kubectl get secret --namespace default airflow-redis -o jsonpath="{.data.redis-password}" | base64 --decode)
$ export POSTGRESQL_PVC=$(kubectl get pvc -l app.kubernetes.io/instance=airflow,app.kubernetes.io/name=postgresql,role=primary -o jsonpath="{.items[0].metadata.name}")
@@ -631,7 +639,8 @@ $ helm upgrade airflow bitnami/airflow \
--set loadExamples=true \
--set web.baseUrl=http://127.0.0.1:8080 \
--set auth.password=$AIRFLOW_PASSWORD \
--set auth.fernetKey=$AIRFLOW_FERNETKEY \
--set auth.fernetKey=$AIRFLOW_FERNET_KEY \
--set auth.secretKey=$AIRFLOW_SECRET_KEY \
--set postgresql.postgresqlPassword=$POSTGRESQL_PASSWORD \
--set postgresql.persistence.existingClaim=$POSTGRESQL_PVC \
--set redis.password=$REDIS_PASSWORD \

9
bitnami/airflow/templates/NOTES.txt
View File

@@ -43,6 +43,7 @@ host. To configure Airflow with the URL of your service:
{{- end }}
{{ include "common.utils.secret.getvalue" (dict "secret" $airflowSecretName "field" "airflow-password" "context" $) }}
{{ include "common.utils.secret.getvalue" (dict "secret" $airflowSecretName "field" "airflow-fernetKey" "context" $) }}
{{ include "common.utils.secret.getvalue" (dict "secret" $airflowSecretName "field" "airflow-secretKey" "context" $) }}
{{ include "common.utils.secret.getvalue" (dict "secret" $postgresqlSecretName "field" "postgresql-password" "context" $) }}
{{ include "common.utils.secret.getvalue" (dict "secret" $redisSecretName "field" "redis-password" "context" $) }}
@@ -52,7 +53,8 @@ host. To configure Airflow with the URL of your service:
--set service.type={{ .Values.service.type }} \
--set web.baseUrl=http://$AIRFLOW_HOST:$AIRFLOW_PORT \
--set auth.password=$AIRFLOW_PASSWORD \
--set auth.fernetKey=$AIRFLOW_FERNETKEY \
--set auth.fernetKey=$AIRFLOW_FERNET_KEY \
--set auth.secretKey=$AIRFLOW_SECRET_KEY \
--set postgresql.postgresqlPassword=$POSTGRESQL_PASSWORD \
--set redis.auth.password=$REDIS_PASSWORD
@@ -99,6 +101,11 @@ host. To configure Airflow with the URL of your service:
{{- $requiredFernetKeyPassword := dict "valueKey" "auth.fernetKey" "secret" $airflowSecretName "field" "airflow-fernetKey" "context" $ -}}
{{- $requiredFernetKeyPasswordError := include "common.validations.values.single.empty" $requiredFernetKeyPassword -}}
{{- $passwordValidationErrors = append $passwordValidationErrors $requiredFernetKeyPasswordError -}}
{{- $requiredSecretKeyPassword := dict "valueKey" "auth.secretKey" "secret" $airflowSecretName "field" "airflow-secretKey" "context" $ -}}
{{- $requiredSecretKeyPasswordError := include "common.validations.values.single.empty" $requiredSecretKeyPassword -}}
{{- $passwordValidationErrors = append $passwordValidationErrors $requiredSecretKeyPasswordError -}}
{{- end -}}
{{- $postgresqlPasswordValidationErrors := include "common.validations.values.postgresql.passwords" (dict "secret" $postgresqlSecretName "subchart" true "context" $) -}}

5
bitnami/airflow/templates/_helpers.tpl
View File

@@ -293,6 +293,11 @@ Add environment variables to configure airflow common values
secretKeyRef:
name: {{ include "airflow.secretName" . }}
key: airflow-fernetKey
- name: AIRFLOW_SECRET_KEY
valueFrom:
secretKeyRef:
name: {{ include "airflow.secretName" . }}
key: airflow-secretKey
- name: AIRFLOW_LOAD_EXAMPLES
value: {{ ternary "yes" "no" .Values.loadExamples | quote }}
{{- if .Values.web.image.debug }}

7
bitnami/airflow/templates/config/secret.yaml
View File

@@ -26,4 +26,11 @@ data:
{{ else }}
airflow-fernetKey: {{ required "An Airflow Fernet Key is required!" .Values.auth.fernetKey }}
{{- end }}
{{- if .Values.auth.secretKey }}
airflow-secretKey: {{ .Values.auth.secretKey | b64enc | quote }}
{{ else if (not .Values.auth.forcePassword) }}
airflow-secretKey: {{ randAlphaNum 32 | b64enc | b64enc | quote }}
{{ else }}
airflow-secretKey: {{ required "An Airflow Secret Key is required!" .Values.auth.secretKey }}
{{- end }}
{{- end }}

13
bitnami/airflow/values.yaml
View File

@@ -157,11 +157,12 @@ tolerations: []
## @section Airflow common parameters
## Authentication
## @param auth.existingSecret Name of an existing secret containing password and fernet key ('airflow-password and 'airflow-fernetKey' keys)
## @param auth.existingSecret Name of an existing secret containing password, fernet key and secret key ('airflow-password', 'airflow-fernetKey' and 'airflow-secretKey' keys)
## @param auth.fernetKey Fernet key to secure connections
## @param auth.forcePassword Force users to specify a password
## @param auth.password Password to access web UI
## @param auth.username Username to access web UI
## @param auth.secretKey Secret key to run your flask app
##
auth:
existingSecret: ""
@@ -176,6 +177,10 @@ auth:
forcePassword: false
password: ""
username: user
## More info about the secret key at:
## - https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key
##
secretKey: ""
## @param configurationConfigMap Name of an existing config map containing the Airflow config file
##
configurationConfigMap: ""
@@ -204,7 +209,7 @@ web:
image:
registry: docker.io
repository: bitnami/airflow
tag: 2.1.4-debian-10-r0
tag: 2.1.4-debian-10-r4
## Specify a imagePullPolicy
## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
@@ -393,7 +398,7 @@ scheduler:
image:
registry: docker.io
repository: bitnami/airflow-scheduler
tag: 2.1.3-debian-10-r23
tag: 2.1.4-debian-10-r4
## Specify a imagePullPolicy
## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
@@ -525,7 +530,7 @@ worker:
image:
registry: docker.io
repository: bitnami/airflow-worker
tag: 2.1.3-debian-10-r23
tag: 2.1.4-debian-10-r4
## Specify a imagePullPolicy
## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images