Add javsalgar feedback

bitnami/airflow/README.md

@@ -74,7 +74,7 @@ The following tables lists the configurable parameters of the Kafka chart and th
 | `gitImage.pullSecrets`                    | Specify docker-registry secret names as an array                                            | `[]` (does not add image pull secrets to deployed pods)      |
 | `updateStrategy`                          | Update strategy for the stateful set                                                        | `RollingUpdate`                                              |
 | `rollingUpdatePartition`                  | Partition update strategy                                                                   | `nil`                                                        |
-| `airflow.airflowConfigurationConfigMap`   | Name of an existing config map containing the Airflow config file                           | `nil`                                                        |
+| `airflow.configurationConfigMap`          | Name of an existing config map containing the Airflow config file                           | `nil`                                                        |
 | `airflow.dagsConfigMap`                   | Name of an existing config map containing all the DAGs files you want to load in Airflow.   | `nil`                                                        |
 | `airflow.loadExamples`                    | Switch to load some Airflow examples                                                        | `true`                                                       |
 | `airflow.cloneDagFilesFromGit.enabled`    | Enable in order to download DAG files from git repository.                                  | `false`                                                      |
@@ -84,8 +84,8 @@ The following tables lists the configurable parameters of the Kafka chart and th
 | `airflow.worker.port`                     | Airflow Worker port                                                                         | `8793`                                                       |
 | `airflow.worker.replicas`                 | Number of Airflow Worker replicas                                                           | `2`                                                          |
 | `airflow.auth.forcePassword`              | Force users to specify a password                                                           | `false`                                                      |
-| `airflow.auth.airflowUsername`            | Username to access web UI                                                                   | `user`                                                       |
-| `airflow.auth.airflowPassword`            | Password to access web UI                                                                   | `nil`                                                        |
+| `airflow.auth.username`                   | Username to access web UI                                                                   | `user`                                                       |
+| `airflow.auth.password`                   | Password to access web UI                                                                   | `nil`                                                        |
 | `airflow.auth.fernetKey`                  | Fernet key to secure connections                                                            | `nil`                                                        |
 | `airflow.auth.existingSecret`             | Name of an existing secret containing airflow password and fernet key                       | `nil`                                                        |
 | `airflow.extraEnvVars`                    | Extra environment variables to add to airflow web, worker and scheduler pods                | `nil`                                                        |
@@ -148,8 +148,10 @@ Specify each parameter using the `--set key=value[,key=value]` argument to `helm
 
 ```console
 $ helm install --name my-release \
-  --set airflow.auth.airflowUsername=my-user,airflow.auth.airflowPassword=my-passsword \
-    bitnami/airflow
+               --set airflow.auth.username=my-user \
+               --set airflow.auth.password=my-passsword \
+               --set airflow.auth.fernetKey=my-fernet-key \
+               bitnami/airflow
 ```
 
 The above command sets the credentials to access the Airflow web UI.
@@ -166,9 +168,15 @@ $ helm install --name my-release -f values.yaml bitnami/airflow
 
 The Bitnami Airflow chart relies on the PostgreSQL chart persistence. This means that Airflow does not persist anything.
 
+## Generate a Fernet key
+
+A Fernet key is required in order to encrypt password within connections. The Fernet key must be a base64-encoded 32-byte key.
+
+Learn how to generate one [here](https://bcb.github.io/airflow/fernet-key)
+
 ## Load DAG files
 
-There are three different ways to load your custom DAG files into the Airflow chart:
+There are three different ways to load your custom DAG files into the Airflow chart. All of them are compatible so you can use more than one at the same time.
 
 ### Option 1: Load locally from the `files` folder
 

bitnami/airflow/templates/NOTES.txt

@@ -44,7 +44,7 @@ host. To configure Airflow with the URL of your service:
 2. Complete your Airflow deployment by running:
 
   helm upgrade {{ .Release.Name }} bitnami/{{ .Chart.Name }} \
-    --set service.type={{ .Values.service.type }},airflow.baseUrl=http://$APP_HOST:$APP_PORT,airflow.auth.airflowPassword=$APP_PASSWORD,postgresql.postgresqlPassword=$APP_DATABASE_PASSWORD,redis.password=$APP_REDIS_PASSWORD{{- if .Values.global }}{{- if .Values.global.imagePullSecrets }},global.imagePullSecrets={{ .Values.global.imagePullSecrets }}{{- end }}{{- end }}
+    --set service.type={{ .Values.service.type }},airflow.baseUrl=http://$APP_HOST:$APP_PORT,airflow.auth.password=$APP_PASSWORD,postgresql.postgresqlPassword=$APP_DATABASE_PASSWORD,redis.password=$APP_REDIS_PASSWORD{{- if .Values.global }}{{- if .Values.global.imagePullSecrets }},global.imagePullSecrets={{ .Values.global.imagePullSecrets }}{{- end }}{{- end }}
 
 {{- else -}}
 
@@ -72,7 +72,7 @@ host. To configure Airflow with the URL of your service:
 
 2. Get your Airflow login credentials by running:
 
-  echo User:    {{ .Values.airflow.auth.airflowUsername }}
+  echo User:    {{ .Values.airflow.auth.username }}
   echo Password: $(kubectl get secret --namespace {{ .Release.Namespace }} {{ template "airflow.fullname" . }} -o jsonpath="{.data.airflow-password}" | base64 --decode)
 
   {{- end  }}

bitnami/airflow/templates/_helpers.tpl

@@ -224,14 +224,14 @@ We truncate at 63 chars because some Kubernetes name fields are limited to this
 {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
 {{- end -}}
 
-{{ template "airflow.dagFilesConfigMap" . }}
+
 {{/*
-Get the DAG files ConfigMap name.
+Get the secret name
 */}}
-{{- define "airflow.dagFilesConfigMap" -}}
-{{- if .Values.airflow.dagsConfigMap -}}
-{{- printf "%s" .Values.airflow.dagsConfigMap -}}
+{{- define "airflow.secretName" -}}
+{{- if .Values.airflow.auth.existingSecret -}}
+{{- printf "%s" .Values.airflow.auth.existingSecret -}}
 {{- else -}}
-{{- printf "%s-dag-files" (include "airflow.fullname" .) -}}
+{{- printf "%s" (include "airflow.fullname" .) -}}
 {{- end -}}
 {{- end -}}

bitnami/airflow/templates/configmap-dags.yaml

@@ -1,4 +1,4 @@
-{{- if and (.Files.Glob "files/dags/*.py") (not .Values.airflow.dagsConfigMap) }}
+{{- if .Files.Glob "files/dags/*.py" }}
 apiVersion: v1
 kind: ConfigMap
 metadata:
@@ -9,5 +9,5 @@ metadata:
     app.kubernetes.io/instance: {{ .Release.Name }}
     app.kubernetes.io/managed-by: {{ .Release.Service }}
 data:
-{{ (.Files.Glob "files/dags/*.{py}").AsConfig | indent 2 }}
+{{ (.Files.Glob "files/dags/*.py").AsConfig | indent 2 }}
 {{ end }}

bitnami/airflow/templates/deployment-scheduler.yaml

@@ -45,14 +45,14 @@ spec:
         fsGroup: {{ .Values.securityContext.fsGroup }}
         runAsUser: {{ .Values.securityContext.runAsUser }}
       {{- end }}
-      {{- if and .Values.airflow.cloneDagFilesFromGit.enabled (not .Values.airflow.dagsConfigMap) }}
+      {{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
       initContainers:
       - name: git-clone-repository
         image: "{{ template "git.image" . }}"
         imagePullPolicy: {{ .Values.git.pullPolicy | quote }}
         command: [ '/bin/sh', '-c' , 'git clone {{ .Values.airflow.cloneDagFilesFromGit.repository }} /dags && cd /dags && git checkout {{ .Values.airflow.cloneDagFilesFromGit.revision }}']
         volumeMounts:
-        - name: custom-dag-files
+        - name: github-cloned-dag-files
           mountPath: /dags
       {{- end }}
       containers:
@@ -149,11 +149,19 @@ spec:
         readinessProbe:
         {{- end }}
         volumeMounts:
-        {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap .Values.airflow.cloneDagFilesFromGit.enabled }}
-        - name: custom-dag-files
-          mountPath: /opt/bitnami/airflow/dags
+        {{- if .Files.Glob "files/dags/*.py" }}
+        - name: local-dag-files
+          mountPath: /opt/bitnami/airflow/dags/local
         {{- end }}
-        {{- if .Values.airflow.airflowConfigurationConfigMap }}
+        {{- if .Values.airflow.dagsConfigMap }}
+        - name: external-dag-files
+          mountPath: /opt/bitnami/airflow/dags/external
+        {{- end }}
+        {{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
+        - name: github-cloned-dag-files
+          mountPath: /opt/bitnami/airflow/dags/github
+        {{- end }}
+        {{- if .Values.airflow.configurationConfigMap }}
         - name: custom-configuration-file
           mountPath: /opt/bitnami/airflow/airflow.cfg
           subPath: airflow.cfg
@@ -161,17 +169,22 @@ spec:
         resources:
 {{ toYaml .Values.resources | indent 10 }}
       volumes:
-      {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap .Values.airflow.cloneDagFilesFromGit.enabled }}
-      - name: custom-dag-files
-        {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap }}
+      {{- if .Files.Glob "files/dags/*.py" }}
+      - name: local-dag-files
         configMap:
-          name: {{ template "airflow.dagFilesConfigMap" . }}
-        {{- else if .Values.airflow.cloneDagFilesFromGit.enabled }}
-        emptyDir: {}
-        {{- end }}
+          name: {{ template "airflow.fullname" . }}-dag-files
       {{- end }}
-      {{- if .Values.airflow.airflowConfigurationConfigMap }}
+      {{- if .Values.airflow.dagsConfigMap }}
+      - name: external-dag-files
+        configMap:
+          name: {{ .Values.airflow.dagsConfigMap }}
+      {{- end }}
+      {{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
+      - name: github-cloned-dag-files
+        emptyDir: {}
+      {{- end }}
+      {{- if .Values.airflow.configurationConfigMap }}
       - name: custom-configuration-file
         configMap:
-          name: {{ .Values.airflow.airflowConfigurationConfigMap }}
+          name: {{ .Values.airflow.configurationConfigMap }}
       {{- end }}

bitnami/airflow/templates/deployment-web.yaml

@@ -45,14 +45,14 @@ spec:
         fsGroup: {{ .Values.securityContext.fsGroup }}
         runAsUser: {{ .Values.securityContext.runAsUser }}
       {{- end }}
-      {{- if and .Values.airflow.cloneDagFilesFromGit.enabled (not .Values.airflow.dagsConfigMap) }}
+      {{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
       initContainers:
       - name: git-clone-repository
         image: "{{ template "git.image" . }}"
         imagePullPolicy: {{ .Values.git.pullPolicy | quote }}
         command: [ '/bin/sh', '-c' , 'git clone {{ .Values.airflow.cloneDagFilesFromGit.repository }} /dags && cd /dags && git checkout {{ .Values.airflow.cloneDagFilesFromGit.revision }}']
         volumeMounts:
-        - name: custom-dag-files
+        - name: github-cloned-dag-files
           mountPath: /dags
       {{- end }}
       containers:
@@ -127,7 +127,7 @@ spec:
         - name: AIRFLOW_EXECUTOR
           value: "CeleryExecutor"
         - name: AIRFLOW_USERNAME
-          value: {{ .Values.airflow.auth.airflowUsername }}
+          value: {{ .Values.airflow.auth.username }}
         - name: AIRFLOW_PASSWORD
           valueFrom:
             secretKeyRef:
@@ -179,11 +179,19 @@ spec:
           failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
         {{- end }}
         volumeMounts:
-        {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap .Values.airflow.cloneDagFilesFromGit.enabled }}
-        - name: custom-dag-files
-          mountPath: /opt/bitnami/airflow/dags
+        {{- if .Files.Glob "files/dags/*.py" }}
+        - name: local-dag-files
+          mountPath: /opt/bitnami/airflow/dags/local
         {{- end }}
-        {{- if .Values.airflow.airflowConfigurationConfigMap }}
+        {{- if .Values.airflow.dagsConfigMap }}
+        - name: external-dag-files
+          mountPath: /opt/bitnami/airflow/dags/external
+        {{- end }}
+        {{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
+        - name: github-cloned-dag-files
+          mountPath: /opt/bitnami/airflow/dags/github
+        {{- end }}
+        {{- if .Values.airflow.configurationConfigMap }}
         - name: custom-configuration-file
           mountPath: /opt/bitnami/airflow/airflow.cfg
           subPath: airflow.cfg
@@ -191,17 +199,22 @@ spec:
         resources:
 {{ toYaml .Values.resources | indent 10 }}
       volumes:
-      {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap .Values.airflow.cloneDagFilesFromGit.enabled }}
-      - name: custom-dag-files
-        {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap }}
+      {{- if .Files.Glob "files/dags/*.py" }}
+      - name: local-dag-files
         configMap:
-          name: {{ template "airflow.dagFilesConfigMap" . }}
-        {{- else if .Values.airflow.cloneDagFilesFromGit.enabled }}
-        emptyDir: {}
-        {{- end }}
+          name: {{ template "airflow.fullname" . }}-dag-files
       {{- end }}
-      {{- if .Values.airflow.airflowConfigurationConfigMap }}
+      {{- if .Values.airflow.dagsConfigMap }}
+      - name: external-dag-files
+        configMap:
+          name: {{ .Values.airflow.dagsConfigMap }}
+      {{- end }}
+      {{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
+      - name: github-cloned-dag-files
+        emptyDir: {}
+      {{- end }}
+      {{- if .Values.airflow.configurationConfigMap }}
       - name: custom-configuration-file
         configMap:
-          name: {{ .Values.airflow.airflowConfigurationConfigMap }}
+          name: {{ .Values.airflow.configurationConfigMap }}
       {{- end }}

bitnami/airflow/templates/secrets.yaml

@@ -10,12 +10,12 @@ metadata:
     app.kubernetes.io/managed-by: {{ .Release.Service }}
 type: Opaque
 data:
-  {{- if .Values.airflow.auth.airflowPassword }}
-  airflow-password: {{ .Values.airflow.auth.airflowPassword | b64enc | quote }}
+  {{- if .Values.airflow.auth.password }}
+  airflow-password: {{ .Values.airflow.auth.password | b64enc | quote }}
   {{ else if (not .Values.airflow.auth.forcePassword) }}
   airflow-password: {{ randAlphaNum 10 | b64enc | quote }}
   {{ else }}
-  airflow-password: {{ required "An Airflow Password is required!" .Values.airflow.auth.airflowPassword }}
+  airflow-password: {{ required "An Airflow Password is required!" .Values.airflow.auth.password }}
   {{- end }}
   {{- if .Values.airflow.auth.fernetKey }}
   airflow-fernetKey: {{ .Values.airflow.auth.fernetKey | b64enc | quote }}

bitnami/airflow/templates/statefulset-worker.yaml

@@ -49,14 +49,14 @@ spec:
       affinity:
 {{ toYaml .Values.affinity | indent 8 }}
       {{- end }}
-      {{- if and .Values.airflow.cloneDagFilesFromGit.enabled (not .Values.airflow.dagsConfigMap) }}
+      {{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
       initContainers:
       - name: git-clone-repository
         image: "{{ template "git.image" . }}"
         imagePullPolicy: {{ .Values.git.pullPolicy | quote }}
         command: [ '/bin/sh', '-c' , 'git clone {{ .Values.airflow.cloneDagFilesFromGit.repository }} /dags && cd /dags && git checkout {{ .Values.airflow.cloneDagFilesFromGit.revision }}']
         volumeMounts:
-        - name: custom-dag-files
+        - name: github-cloned-dag-files
           mountPath: /dags
       {{- end }}
       containers:
@@ -166,27 +166,40 @@ spec:
           failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
         {{- end }}
         volumeMounts:
-        {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap .Values.airflow.cloneDagFilesFromGit.enabled }}
-        - name: custom-dag-files
-          mountPath: /opt/bitnami/airflow/dags
+        {{- if .Files.Glob "files/dags/*.py" }}
+        - name: local-dag-files
+          mountPath: /opt/bitnami/airflow/dags/local
         {{- end }}
-        {{- if .Values.airflow.airflowConfigurationConfigMap }}
+        {{- if .Values.airflow.dagsConfigMap }}
+        - name: external-dag-files
+          mountPath: /opt/bitnami/airflow/dags/external
+        {{- end }}
+        {{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
+        - name: github-cloned-dag-files
+          mountPath: /opt/bitnami/airflow/dags/github
+        {{- end }}
+        {{- if .Values.airflow.configurationConfigMap }}
         - name: custom-configuration-file
           mountPath: /opt/bitnami/airflow/airflow.cfg
           subPath: airflow.cfg
         {{- end }}
       volumes:
-      {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap .Values.airflow.cloneDagFilesFromGit.enabled }}
-      - name: custom-dag-files
-        {{- if or (.Files.Glob "files/dags/*.py") .Values.airflow.dagsConfigMap }}
+      {{- if .Files.Glob "files/dags/*.py" }}
+      - name: local-dag-files
         configMap:
-          name: {{ template "airflow.dagFilesConfigMap" . }}
-        {{- else if .Values.airflow.cloneDagFilesFromGit.enabled }}
-        emptyDir: {}
-        {{- end }}
+          name: {{ template "airflow.fullname" . }}-dag-files
       {{- end }}
-      {{- if .Values.airflow.airflowConfigurationConfigMap }}
+      {{- if .Values.airflow.dagsConfigMap }}
+      - name: external-dag-files
+        configMap:
+          name: {{ .Values.airflow.dagsConfigMap }}
+      {{- end }}
+      {{- if .Values.airflow.cloneDagFilesFromGit.enabled }}
+      - name: github-cloned-dag-files
+        emptyDir: {}
+      {{- end }}
+      {{- if .Values.airflow.configurationConfigMap }}
       - name: custom-configuration-file
         configMap:
-          name: {{ .Values.airflow.airflowConfigurationConfigMap }}
+          name: {{ .Values.airflow.configurationConfigMap }}
       {{- end }}

bitnami/airflow/values-production.yaml

@@ -109,7 +109,7 @@ updateStrategy: RollingUpdate
 airflow:
   ## Name of an existing config map containing the Airflow config file
   ##
-  # airflowConfigurationConfigMap:
+  # configurationConfigMap:
   ## Name of an existing config map containing all the DAGs files you want to load in Airflow.
   ##
   # dagsConfigMap:
@@ -137,10 +137,10 @@ airflow:
     forcePassword: true
     ## Specify username to acccess web UI
     ##
-    airflowUsername: user
+    username: user
     ## Specify password to acccess web UI
     ##
-    # airflowPassword:
+    # password:
     ## Specify a fernet key to secure airflow connections
     ## Fernet key must be 32 url-safe base64-encoded bytes.
     ## More info at:
@@ -148,7 +148,7 @@ airflow:
     ## - https://bcb.github.io/airflow/fernet-key
     ##
     # fernetKey:
-    ## Use existing secret (ignores previous airflowPassword and fernetKey)
+    ## Use existing secret (ignores previous password and fernetKey)
     ##
     # existingSecret:
 

bitnami/airflow/values.yaml

@@ -109,19 +109,19 @@ updateStrategy: RollingUpdate
 airflow:
   ## Name of an existing config map containing the Airflow config file
   ##
-  # airflowConfigurationConfigMap:
+  # configurationConfigMap:
   ## Name of an existing config map containing all the DAGs files you want to load in Airflow.
   ##
-  # dagsConfigMap:
+  dagsConfigMap: patata
   ## Airflow generic configuration
   ##
-  loadExamples: true
+  loadExamples: false
   ## Enable in order to download DAG files from git repository.
   ##
   cloneDagFilesFromGit:
-    enabled: false
-    repository:
-    revision:
+    enabled: true
+    repository: https://github.com/tompizmor/sample-airflow-dag
+    revision: master
   ## URL used to access to airflow web ui
   ##
   # baseUrl:
@@ -137,10 +137,10 @@ airflow:
     forcePassword: false
     ## Specify username to acccess web UI
     ##
-    airflowUsername: user
+    username: user
     ## Specify password to acccess web UI
     ##
-    # airflowPassword:
+    # password:
     ## Specify a fernet key to secure airflow connections
     ## Fernet key must be 32 url-safe base64-encoded bytes.
     ## More info at:
@@ -148,7 +148,7 @@ airflow:
     ## - https://bcb.github.io/airflow/fernet-key
     ##
     # fernetKey:
-    ## Use existing secret (ignores previous airflowPassword and fernetKey)
+    ## Use existing secret (ignores previous password and fernetKey)
     ##
     # existingSecret: