mirror of
https://github.com/bitnami/charts.git
synced 2026-08-10 14:15:55 +08:00
[bitnami/airflow] Metrics based on StatsD (#30459)
This commit is contained in:
Juan Ariza Toledano
@@ -14,3 +14,9 @@ file:
|
||||
/opt/bitnami/airflow/dags:
|
||||
filetype: directory
|
||||
exists: true
|
||||
{{- if .Vars.metrics.enabled }}
|
||||
http:
|
||||
http://airflow-statsd-metrics:{{ .Vars.metrics.service.ports.metrics }}/metrics:
|
||||
status: 200
|
||||
timeout: 30000
|
||||
{{- end }}
|
||||
|
||||
@@ -16,3 +16,8 @@ dags:
|
||||
branch: main
|
||||
name: toys
|
||||
path: dags/toys
|
||||
metrics:
|
||||
enabled: true
|
||||
service:
|
||||
ports:
|
||||
metrics: 9102
|
||||
|
||||
@@ -37,6 +37,7 @@
|
||||
"path": "/.vib"
|
||||
},
|
||||
"tests_file": "airflow/goss/goss.yaml",
|
||||
"vars_file": "airflow/runtime-parameters.yaml",
|
||||
"remote": {
|
||||
"pod": {
|
||||
"workload": "deploy-airflow-web"
|
||||
|
||||
@@ -1,8 +1,12 @@
|
||||
# Changelog
|
||||
|
||||
## 22.0.0 (2024-11-15)
|
||||
|
||||
* [bitnami/airflow] Metrics based on StatsD ([#30459](https://github.com/bitnami/charts/pull/30459))
|
||||
|
||||
## 21.9.0 (2024-11-14)
|
||||
|
||||
* [bitnami/airflow] Add support to setup db on k8s job ([#30440](https://github.com/bitnami/charts/pull/30440))
|
||||
* [bitnami/airflow] Add support to setup db on k8s job (#30440) ([1a7d623](https://github.com/bitnami/charts/commit/1a7d623709ba680e7d049e81bddc44d9115f5f1f)), closes [#30440](https://github.com/bitnami/charts/issues/30440)
|
||||
|
||||
## 21.8.0 (2024-11-13)
|
||||
|
||||
|
||||
@@ -7,8 +7,8 @@ annotations:
|
||||
images: |
|
||||
- name: airflow
|
||||
image: docker.io/bitnami/airflow:2.10.3-debian-12-r5
|
||||
- name: airflow-exporter
|
||||
image: docker.io/bitnami/airflow-exporter:0.20220314.0-debian-12-r51
|
||||
- name: statsd-exporter
|
||||
image: docker.io/bitnami/airflow-statsd:0.28.0-debian-12-r0
|
||||
apiVersion: v2
|
||||
appVersion: 2.10.3
|
||||
dependencies:
|
||||
@@ -39,4 +39,4 @@ maintainers:
|
||||
name: airflow
|
||||
sources:
|
||||
- https://github.com/bitnami/charts/tree/main/bitnami/airflow
|
||||
version: 21.9.0
|
||||
version: 22.0.0
|
||||
|
||||
+99
-70
@@ -230,7 +230,7 @@ To make this process easier, the chart contains the `resourcesPreset` values, wh
|
||||
|
||||
### Prometheus metrics
|
||||
|
||||
This chart can be integrated with Prometheus by setting `metrics.enabled` true. This will deploy a sidecar container with [airflow-exporter](https://github.com/PBWebMedia/airflow-prometheus-exporter) in all Airflow pods and a `metrics` service, which can be configured under the `metrics.service` section. This `metrics` service will be have the necessary annotations to be automatically scraped by Prometheus.
|
||||
This chart can be integrated with Prometheus by setting `metrics.enabled` true. This will configure Airflow components to send StatsD metrics to the [StatsD exporter](https://github.com/prometheus/statsd_exporter) that transforms them into Prometheus metrics. The StatsD exporter is deployed as a standalone deployment and service in the same namespace as the Airflow deployment.
|
||||
|
||||
#### Prometheus requirements
|
||||
|
||||
@@ -1079,76 +1079,99 @@ The Bitnami Airflow chart relies on the PostgreSQL chart persistence. This means
|
||||
| `rbac.create` | Create Role and RoleBinding | `false` |
|
||||
| `rbac.rules` | Custom RBAC rules to set | `[]` |
|
||||
|
||||
### Airflow metrics parameters
|
||||
### StatsD metrics parameters
|
||||
|
||||
| Name | Description | Value |
|
||||
| ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- |
|
||||
| `metrics.enabled` | Whether or not to create a standalone Airflow exporter to expose Airflow metrics | `false` |
|
||||
| `metrics.image.registry` | Airflow exporter image registry | `REGISTRY_NAME` |
|
||||
| `metrics.image.repository` | Airflow exporter image repository | `REPOSITORY_NAME/airflow-exporter` |
|
||||
| `metrics.image.digest` | Airflow exporter image digest in the way sha256:aa.... Please note this parameter, if set, will override the tag | `""` |
|
||||
| `metrics.image.pullPolicy` | Airflow exporter image pull policy | `IfNotPresent` |
|
||||
| `metrics.image.pullSecrets` | Airflow exporter image pull secrets | `[]` |
|
||||
| `metrics.extraEnvVars` | Array with extra environment variables to add Airflow exporter pods | `[]` |
|
||||
| `metrics.extraEnvVarsCM` | ConfigMap containing extra environment variables for Airflow exporter pods | `""` |
|
||||
| `metrics.extraEnvVarsSecret` | Secret containing extra environment variables (in case of sensitive data) for Airflow exporter pods | `""` |
|
||||
| `metrics.containerPorts.http` | Airflow exporter metrics container port | `9112` |
|
||||
| `metrics.resourcesPreset` | Set container resources according to one common preset (allowed values: none, nano, micro, small, medium, large, xlarge, 2xlarge). This is ignored if metrics.resources is set (metrics.resources is recommended for production). | `nano` |
|
||||
| `metrics.resources` | Set container requests and limits for different resources like CPU or memory (essential for production workloads) | `{}` |
|
||||
| `metrics.podSecurityContext.enabled` | Enable security context for the pods | `true` |
|
||||
| `metrics.podSecurityContext.fsGroupChangePolicy` | Set filesystem group change policy | `Always` |
|
||||
| `metrics.podSecurityContext.sysctls` | Set kernel settings using the sysctl interface | `[]` |
|
||||
| `metrics.podSecurityContext.supplementalGroups` | Set filesystem extra groups | `[]` |
|
||||
| `metrics.podSecurityContext.fsGroup` | Set Airflow exporter pod's Security Context fsGroup | `1001` |
|
||||
| `metrics.containerSecurityContext.enabled` | Enable Airflow exporter containers' Security Context | `true` |
|
||||
| `metrics.containerSecurityContext.seLinuxOptions` | Set SELinux options in container | `{}` |
|
||||
| `metrics.containerSecurityContext.runAsUser` | Set Airflow exporter containers' Security Context runAsUser | `1001` |
|
||||
| `metrics.containerSecurityContext.runAsGroup` | Set Airflow exporter containers' Security Context runAsGroup | `1001` |
|
||||
| `metrics.containerSecurityContext.runAsNonRoot` | Set Airflow exporter containers' Security Context runAsNonRoot | `true` |
|
||||
| `metrics.containerSecurityContext.privileged` | Set metrics container's Security Context privileged | `false` |
|
||||
| `metrics.containerSecurityContext.allowPrivilegeEscalation` | Set metrics container's Security Context allowPrivilegeEscalation | `false` |
|
||||
| `metrics.containerSecurityContext.readOnlyRootFilesystem` | Set metrics container's Security Context readOnlyRootFilesystem | `true` |
|
||||
| `metrics.containerSecurityContext.capabilities.drop` | List of capabilities to be dropped | `["ALL"]` |
|
||||
| `metrics.containerSecurityContext.seccompProfile.type` | Set container's Security Context seccomp profile | `RuntimeDefault` |
|
||||
| `metrics.lifecycleHooks` | for the Airflow exporter container(s) to automate configuration before or after startup | `{}` |
|
||||
| `metrics.automountServiceAccountToken` | Mount Service Account token in pod | `false` |
|
||||
| `metrics.hostAliases` | Airflow exporter pods host aliases | `[]` |
|
||||
| `metrics.podLabels` | Extra labels for Airflow exporter pods | `{}` |
|
||||
| `metrics.podAnnotations` | Extra annotations for Airflow exporter pods | `{}` |
|
||||
| `metrics.podAffinityPreset` | Pod affinity preset. Ignored if `metrics.affinity` is set. Allowed values: `soft` or `hard` | `""` |
|
||||
| `metrics.podAntiAffinityPreset` | Pod anti-affinity preset. Ignored if `metrics.affinity` is set. Allowed values: `soft` or `hard` | `soft` |
|
||||
| `metrics.nodeAffinityPreset.type` | Node affinity preset type. Ignored if `metrics.affinity` is set. Allowed values: `soft` or `hard` | `""` |
|
||||
| `metrics.nodeAffinityPreset.key` | Node label key to match Ignored if `metrics.affinity` is set. | `""` |
|
||||
| `metrics.nodeAffinityPreset.values` | Node label values to match. Ignored if `metrics.affinity` is set. | `[]` |
|
||||
| `metrics.affinity` | Affinity for pod assignment | `{}` |
|
||||
| `metrics.nodeSelector` | Node labels for pod assignment | `{}` |
|
||||
| `metrics.priorityClassName` | Airflow exporter pods' priorityClassName | `""` |
|
||||
| `metrics.tolerations` | Tolerations for pod assignment | `[]` |
|
||||
| `metrics.topologySpreadConstraints` | Topology Spread Constraints for pod assignment spread across your cluster among failure-domains. Evaluated as a template | `[]` |
|
||||
| `metrics.schedulerName` | Name of the k8s scheduler (other than default) for Airflow exporter | `""` |
|
||||
| `metrics.extraVolumes` | Optionally specify extra list of additional volumes for the Airflow exporter pods | `[]` |
|
||||
| `metrics.extraVolumeMounts` | Optionally specify extra list of additional volumeMounts for the Airflow exporter containers | `[]` |
|
||||
| `metrics.service.ports.http` | Airflow exporter metrics service port | `9112` |
|
||||
| `metrics.service.clusterIP` | Static clusterIP or None for headless services | `""` |
|
||||
| `metrics.service.sessionAffinity` | Control where client requests go, to the same pod or round-robin | `None` |
|
||||
| `metrics.service.annotations` | Annotations for the Airflow exporter service | `{}` |
|
||||
| `metrics.serviceMonitor.enabled` | if `true`, creates a Prometheus Operator ServiceMonitor (requires `metrics.enabled` to be `true`) | `false` |
|
||||
| `metrics.serviceMonitor.namespace` | Namespace in which Prometheus is running | `""` |
|
||||
| `metrics.serviceMonitor.interval` | Interval at which metrics should be scraped | `""` |
|
||||
| `metrics.serviceMonitor.scrapeTimeout` | Timeout after which the scrape is ended | `""` |
|
||||
| `metrics.serviceMonitor.labels` | Additional labels that can be used so ServiceMonitor will be discovered by Prometheus | `{}` |
|
||||
| `metrics.serviceMonitor.selector` | Prometheus instance selector labels | `{}` |
|
||||
| `metrics.serviceMonitor.relabelings` | RelabelConfigs to apply to samples before scraping | `[]` |
|
||||
| `metrics.serviceMonitor.metricRelabelings` | MetricRelabelConfigs to apply to samples before ingestion | `[]` |
|
||||
| `metrics.serviceMonitor.honorLabels` | Specify honorLabels parameter to add the scrape endpoint | `false` |
|
||||
| `metrics.serviceMonitor.jobLabel` | The name of the label on the target service to use as the job name in prometheus. | `""` |
|
||||
| `metrics.networkPolicy.enabled` | Specifies whether a NetworkPolicy should be created | `true` |
|
||||
| `metrics.networkPolicy.allowExternal` | Don't require client label for connections | `true` |
|
||||
| `metrics.networkPolicy.allowExternalEgress` | Allow the pod to access any range of port and all destinations. | `true` |
|
||||
| `metrics.networkPolicy.extraIngress` | Add extra ingress rules to the NetworkPolicy | `[]` |
|
||||
| `metrics.networkPolicy.extraEgress` | Add extra ingress rules to the NetworkPolicy | `[]` |
|
||||
| `metrics.networkPolicy.ingressNSMatchLabels` | Labels to match to allow traffic from other namespaces | `{}` |
|
||||
| `metrics.networkPolicy.ingressNSPodMatchLabels` | Pod labels to match to allow traffic from other namespaces | `{}` |
|
||||
| Name | Description | Value |
|
||||
| ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
|
||||
| `metrics.enabled` | Enable a StatsD exporter that collects StatsD metrics from Airflow components and expose them as Prometheus metrics | `false` |
|
||||
| `metrics.image.registry` | StatsD exporter image registry | `REGISTRY_NAME` |
|
||||
| `metrics.image.repository` | StatsD exporter image repository | `REPOSITORY_NAME/statsd-exporter` |
|
||||
| `metrics.image.digest` | StatsD exporter image digest in the way sha256:aa.... Please note this parameter, if set, will override the tag | `""` |
|
||||
| `metrics.image.pullPolicy` | StatsD exporter image pull policy | `IfNotPresent` |
|
||||
| `metrics.image.pullSecrets` | StatsD exporter image pull secrets | `[]` |
|
||||
| `metrics.configuration` | Specify content for StatsD exporter's mappings.yml | `""` |
|
||||
| `metrics.existingConfigmap` | Name of an existing config map containing the StatsD exporter's mappings.yml | `""` |
|
||||
| `metrics.containerPorts.ingest` | StatsD exporter ingest container port (used for the metrics ingestion from Airflow components) | `9125` |
|
||||
| `metrics.containerPorts.metrics` | StatsD exporter metrics container port (used to expose Prometheus metrics) | `9102` |
|
||||
| `metrics.resourcesPreset` | Set container resources according to one common preset (allowed values: none, nano, micro, small, medium, large, xlarge, 2xlarge). This is ignored if metrics.resources is set (metrics.resources is recommended for production). | `nano` |
|
||||
| `metrics.resources` | Set container requests and limits for different resources like CPU or memory (essential for production workloads) | `{}` |
|
||||
| `metrics.podSecurityContext.enabled` | Enable security context for the pods | `true` |
|
||||
| `metrics.podSecurityContext.fsGroupChangePolicy` | Set filesystem group change policy | `Always` |
|
||||
| `metrics.podSecurityContext.sysctls` | Set kernel settings using the sysctl interface | `[]` |
|
||||
| `metrics.podSecurityContext.supplementalGroups` | Set filesystem extra groups | `[]` |
|
||||
| `metrics.podSecurityContext.fsGroup` | Set StatsD exporter pod's Security Context fsGroup | `1001` |
|
||||
| `metrics.containerSecurityContext.enabled` | Enable StatsD exporter containers' Security Context | `true` |
|
||||
| `metrics.containerSecurityContext.seLinuxOptions` | Set SELinux options in container | `{}` |
|
||||
| `metrics.containerSecurityContext.runAsUser` | Set StatsD exporter containers' Security Context runAsUser | `1001` |
|
||||
| `metrics.containerSecurityContext.runAsGroup` | Set StatsD exporter containers' Security Context runAsGroup | `1001` |
|
||||
| `metrics.containerSecurityContext.runAsNonRoot` | Set StatsD exporter containers' Security Context runAsNonRoot | `true` |
|
||||
| `metrics.containerSecurityContext.privileged` | Set StatsD exporter containers' Security Context privileged | `false` |
|
||||
| `metrics.containerSecurityContext.allowPrivilegeEscalation` | Set StatsD exporter containers' Security Context allowPrivilegeEscalation | `false` |
|
||||
| `metrics.containerSecurityContext.readOnlyRootFilesystem` | Set StatsD exporter containers' Security Context readOnlyRootFilesystem | `true` |
|
||||
| `metrics.containerSecurityContext.capabilities.drop` | List of capabilities to be dropped | `["ALL"]` |
|
||||
| `metrics.containerSecurityContext.seccompProfile.type` | Set containers' Security Context seccomp profile | `RuntimeDefault` |
|
||||
| `metrics.livenessProbe.enabled` | Enable livenessProbe on StatsD exporter containers | `true` |
|
||||
| `metrics.livenessProbe.initialDelaySeconds` | Initial delay seconds for livenessProbe | `180` |
|
||||
| `metrics.livenessProbe.periodSeconds` | Period seconds for livenessProbe | `20` |
|
||||
| `metrics.livenessProbe.timeoutSeconds` | Timeout seconds for livenessProbe | `5` |
|
||||
| `metrics.livenessProbe.failureThreshold` | Failure threshold for livenessProbe | `6` |
|
||||
| `metrics.livenessProbe.successThreshold` | Success threshold for livenessProbe | `1` |
|
||||
| `metrics.readinessProbe.enabled` | Enable readinessProbe on StatsD exporter containers | `true` |
|
||||
| `metrics.readinessProbe.initialDelaySeconds` | Initial delay seconds for readinessProbe | `30` |
|
||||
| `metrics.readinessProbe.periodSeconds` | Period seconds for readinessProbe | `10` |
|
||||
| `metrics.readinessProbe.timeoutSeconds` | Timeout seconds for readinessProbe | `5` |
|
||||
| `metrics.readinessProbe.failureThreshold` | Failure threshold for readinessProbe | `6` |
|
||||
| `metrics.readinessProbe.successThreshold` | Success threshold for readinessProbe | `1` |
|
||||
| `metrics.startupProbe.enabled` | Enable startupProbe on StatsD exporter containers | `false` |
|
||||
| `metrics.startupProbe.initialDelaySeconds` | Initial delay seconds for startupProbe | `60` |
|
||||
| `metrics.startupProbe.periodSeconds` | Period seconds for startupProbe | `10` |
|
||||
| `metrics.startupProbe.timeoutSeconds` | Timeout seconds for startupProbe | `1` |
|
||||
| `metrics.startupProbe.failureThreshold` | Failure threshold for startupProbe | `15` |
|
||||
| `metrics.startupProbe.successThreshold` | Success threshold for startupProbe | `1` |
|
||||
| `metrics.customLivenessProbe` | Custom livenessProbe that overrides the default one | `{}` |
|
||||
| `metrics.customReadinessProbe` | Custom readinessProbe that overrides the default one | `{}` |
|
||||
| `metrics.customStartupProbe` | Custom startupProbe that overrides the default one | `{}` |
|
||||
| `metrics.lifecycleHooks` | for the StatsD exporter containers' to automate configuration before or after startup | `{}` |
|
||||
| `metrics.automountServiceAccountToken` | Mount Service Account token in pod | `false` |
|
||||
| `metrics.hostAliases` | StatsD exporter pods host aliases | `[]` |
|
||||
| `metrics.podLabels` | Extra labels for StatsD exporter pods | `{}` |
|
||||
| `metrics.podAnnotations` | Extra annotations for StatsD exporter pods | `{}` |
|
||||
| `metrics.podAffinityPreset` | Pod affinity preset. Ignored if `metrics.affinity` is set. Allowed values: `soft` or `hard` | `""` |
|
||||
| `metrics.podAntiAffinityPreset` | Pod anti-affinity preset. Ignored if `metrics.affinity` is set. Allowed values: `soft` or `hard` | `soft` |
|
||||
| `metrics.nodeAffinityPreset.type` | Node affinity preset type. Ignored if `metrics.affinity` is set. Allowed values: `soft` or `hard` | `""` |
|
||||
| `metrics.nodeAffinityPreset.key` | Node label key to match Ignored if `metrics.affinity` is set. | `""` |
|
||||
| `metrics.nodeAffinityPreset.values` | Node label values to match. Ignored if `metrics.affinity` is set. | `[]` |
|
||||
| `metrics.affinity` | Affinity for StatsD exporter pods assignment | `{}` |
|
||||
| `metrics.nodeSelector` | Node labels for StatsD exporter pods assignment | `{}` |
|
||||
| `metrics.priorityClassName` | StatsD exporter pods' priorityClassName | `""` |
|
||||
| `metrics.tolerations` | Tolerations for pod assignment | `[]` |
|
||||
| `metrics.topologySpreadConstraints` | Topology Spread Constraints for pod assignment spread across your cluster among failure-domains. Evaluated as a template | `[]` |
|
||||
| `metrics.schedulerName` | Name of the k8s scheduler (other than default) for StatsD exporter | `""` |
|
||||
| `metrics.terminationGracePeriodSeconds` | Seconds StatsD exporter pod needs to terminate gracefully | `""` |
|
||||
| `metrics.extraVolumes` | Optionally specify extra list of additional volumes for the StatsD exporter pods | `[]` |
|
||||
| `metrics.extraVolumeMounts` | Optionally specify extra list of additional volumeMounts for the StatsD exporter containers | `[]` |
|
||||
| `metrics.service.ports.ingest` | StatsD exporter ingest service port (used for the metrics ingestion from Airflow components) | `9125` |
|
||||
| `metrics.service.ports.metrics` | StatsD exporter metrics service port (used to expose Prometheus metrics) | `9102` |
|
||||
| `metrics.service.clusterIP` | Static clusterIP or None for headless services | `""` |
|
||||
| `metrics.service.sessionAffinity` | Control where client requests go, to the same pod or round-robin | `None` |
|
||||
| `metrics.service.annotations` | Annotations for the StatsD metrics service | `{}` |
|
||||
| `metrics.serviceMonitor.enabled` | if `true`, creates a Prometheus Operator ServiceMonitor (requires `metrics.enabled` to be `true`) | `false` |
|
||||
| `metrics.serviceMonitor.namespace` | Namespace in which Prometheus is running | `""` |
|
||||
| `metrics.serviceMonitor.interval` | Interval at which metrics should be scraped | `""` |
|
||||
| `metrics.serviceMonitor.scrapeTimeout` | Timeout after which the scrape is ended | `""` |
|
||||
| `metrics.serviceMonitor.labels` | Additional labels that can be used so ServiceMonitor will be discovered by Prometheus | `{}` |
|
||||
| `metrics.serviceMonitor.selector` | Prometheus instance selector labels | `{}` |
|
||||
| `metrics.serviceMonitor.relabelings` | RelabelConfigs to apply to samples before scraping | `[]` |
|
||||
| `metrics.serviceMonitor.metricRelabelings` | MetricRelabelConfigs to apply to samples before ingestion | `[]` |
|
||||
| `metrics.serviceMonitor.honorLabels` | Specify honorLabels parameter to add the scrape endpoint | `false` |
|
||||
| `metrics.serviceMonitor.jobLabel` | The name of the label on the target service to use as the job name in prometheus. | `""` |
|
||||
| `metrics.networkPolicy.enabled` | Specifies whether a NetworkPolicy should be created | `true` |
|
||||
| `metrics.networkPolicy.allowExternal` | Don't require client label for connections | `true` |
|
||||
| `metrics.networkPolicy.allowExternalEgress` | Allow the pod to access any range of port and all destinations. | `true` |
|
||||
| `metrics.networkPolicy.extraIngress` | Add extra ingress rules to the NetworkPolicy | `[]` |
|
||||
| `metrics.networkPolicy.extraEgress` | Add extra ingress rules to the NetworkPolicy | `[]` |
|
||||
| `metrics.networkPolicy.ingressNSMatchLabels` | Labels to match to allow traffic from other namespaces | `{}` |
|
||||
| `metrics.networkPolicy.ingressNSPodMatchLabels` | Pod labels to match to allow traffic from other namespaces | `{}` |
|
||||
|
||||
### Airflow database parameters
|
||||
|
||||
@@ -1219,6 +1242,12 @@ Find more information about how to deal with common errors related to Bitnami's
|
||||
|
||||
## Upgrading
|
||||
|
||||
### To 22.0.0
|
||||
|
||||
This major version replaces exposing Prometheus metrics using the [Airflow prometheus exporter](https://github.com/PBWebMedia/airflow-prometheus-exporter), that exposes metrics based on the data retrieved from the database, by configuring Airflow components to send StatsD metrics to the [StatsD exporter](https://github.com/prometheus/statsd_exporter) that transforms them into Prometheus metrics. Find more information about this approach in the [Apache Airflow official documentation](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html#setup-statsd).
|
||||
|
||||
No upgrades issues are expected when upgrading from `21.x.x` but existing dashboards and alerts based on the previous metrics should be adapted to the new ones.
|
||||
|
||||
### To 21.0.0
|
||||
|
||||
This major version uses a single container image ([`bitnami/airflow`](https://github.com/bitnami/containers/tree/main/bitnami/airflow) by default) to run every Airflow component (Webserver, Scheduler and Worker) so `bitnami/airflow-scheduler` and `bitnami/airflow-worker` images are no longer necessary. Also, operations to load custom DAGs and plugins via init containers also use this same image so `bitnami/git` and `bitnami/os-shell` are no longer necessary either. These changes implies several simplifications in the chart values:
|
||||
|
||||
@@ -40,6 +40,13 @@ Return the proper Airflow Worker fullname
|
||||
{{- printf "%s-worker" (include "common.names.fullname" .) | trunc 63 | trimSuffix "-" -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/*
|
||||
Return the proper Airflow metrics fullname
|
||||
*/}}
|
||||
{{- define "airflow.metrics.fullname" -}}
|
||||
{{- printf "%s-statsd-metrics" (include "common.names.fullname" .) | trunc 63 | trimSuffix "-" -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/*
|
||||
Return the proper Airflow image name
|
||||
*/}}
|
||||
@@ -167,6 +174,17 @@ Get the configmap name for Airflow Webserver
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/*
|
||||
Get the configmap name for StatsD exporter
|
||||
*/}}
|
||||
{{- define "airflow.metrics.configMapName" -}}
|
||||
{{- if .Values.metrics.existingConfigmap -}}
|
||||
{{- print (tpl .Values.metrics.existingConfigmap .) -}}
|
||||
{{- else -}}
|
||||
{{- print (include "airflow.metrics.fullname" .) -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/*
|
||||
Return the name of the secret containing the TLS certificates for Airflow webserver
|
||||
*/}}
|
||||
|
||||
@@ -25,6 +25,12 @@ data:
|
||||
[logging]
|
||||
colored_console_log=False
|
||||
|
||||
[metrics]
|
||||
statsd_on={{ ternary "True" "False" (.Values.metrics.enabled) }}
|
||||
statsd_port={{ .Values.metrics.service.ports.ingest }}
|
||||
statsd_prefix=airflow
|
||||
statsd_host={{ include "airflow.metrics.fullname" . }}
|
||||
|
||||
[scheduler]
|
||||
standalone_dag_processor={{ ternary "True" "False" .Values.dagProcessor.enabled }}
|
||||
|
||||
@@ -34,7 +40,7 @@ data:
|
||||
[webserver]
|
||||
base_url={{ include "airflow.baseUrl" . }}
|
||||
enable_proxy_fix={{ ternary "True" "False" (and .Values.ingress.enabled .Values.ingress.tls) }}
|
||||
web_server_port={{ .Values.web.containerPorts.http}}
|
||||
web_server_port={{ .Values.web.containerPorts.http }}
|
||||
{{- if .Values.web.tls.enabled }}
|
||||
web_server_ssl_cert="/opt/bitnami/airflow/certs/tls.crt"
|
||||
web_server_ssl_key="/opt/bitnami/airflow/certs/tls.key"
|
||||
|
||||
@@ -36,6 +36,9 @@ spec:
|
||||
# Allow outbound connections to other cluster pods
|
||||
- ports:
|
||||
- port: {{ .Values.service.ports.http }}
|
||||
{{- if .Values.metrics.enabled }}
|
||||
- port: {{ .Values.metrics.service.ports.ingest }}
|
||||
{{- end }}
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels: {{- include "common.labels.matchLabels" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 14 }}
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
{{- /*
|
||||
Copyright Broadcom, Inc. All Rights Reserved.
|
||||
SPDX-License-Identifier: APACHE-2.0
|
||||
*/}}
|
||||
|
||||
{{- if not .Values.metrics.existingConfigmap }}
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: {{ template "airflow.metrics.fullname" . }}
|
||||
namespace: {{ include "common.names.namespace" . | quote }}
|
||||
labels: {{- include "common.labels.standard" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 4 }}
|
||||
app.kubernetes.io/component: metrics
|
||||
{{- if .Values.commonAnnotations }}
|
||||
annotations: {{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
|
||||
{{- end }}
|
||||
data:
|
||||
mappings.yml: |-
|
||||
{{- if .Values.metrics.configuration }}
|
||||
{{- include "common.tplvalues.render" (dict "value" .Values.metrics.configuration "context" .) | indent 4 }}
|
||||
{{- else }}
|
||||
mappings:
|
||||
- match: airflow.dagrun.dependency-check.*.*
|
||||
name: "airflow_dagrun_dependency_check"
|
||||
labels:
|
||||
dag_id: "$1"
|
||||
|
||||
- match: airflow.operator_successes_(.*)
|
||||
match_type: regex
|
||||
name: "airflow_operator_successes"
|
||||
labels:
|
||||
operator: "$1"
|
||||
|
||||
- match: airflow.operator_failures_(.*)
|
||||
match_type: regex
|
||||
name: "airflow_operator_failures"
|
||||
labels:
|
||||
operator: "$1"
|
||||
|
||||
- match: airflow.scheduler_heartbeat
|
||||
match_type: regex
|
||||
name: "airflow_scheduler_heartbeat"
|
||||
labels:
|
||||
type: counter
|
||||
|
||||
- match: airflow.dag_processor_heartbeat
|
||||
match_type: regex
|
||||
name: "airflow_dag_processor_heartbeat"
|
||||
labels:
|
||||
type: counter
|
||||
|
||||
- match: airflow.dag.*.*.duration
|
||||
name: "airflow_task_duration"
|
||||
labels:
|
||||
dag_id: "$1"
|
||||
task_id: "$2"
|
||||
|
||||
- match: airflow.dagrun.duration.success.*
|
||||
name: "airflow_dagrun_duration"
|
||||
labels:
|
||||
dag_id: "$1"
|
||||
|
||||
- match: airflow.dagrun.duration.failed.*
|
||||
name: "airflow_dagrun_failed"
|
||||
labels:
|
||||
dag_id: "$1"
|
||||
|
||||
- match: airflow.dagrun.schedule_delay.*
|
||||
name: "airflow_dagrun_schedule_delay"
|
||||
labels:
|
||||
dag_id: "$1"
|
||||
|
||||
- match: airflow.dag_processing.last_runtime.*
|
||||
name: "airflow_dag_processing_last_runtime"
|
||||
labels:
|
||||
dag_file: "$1"
|
||||
|
||||
- match: airflow.dag_processing.last_run.seconds_ago.*
|
||||
name: "airflow_dag_processing_last_run_seconds_ago"
|
||||
labels:
|
||||
dag_file: "$1"
|
||||
|
||||
- match: airflow.pool.open_slots.*
|
||||
name: "airflow_pool_open_slots"
|
||||
labels:
|
||||
pool: "$1"
|
||||
|
||||
- match: airflow.pool.used_slots.*
|
||||
name: "airflow_pool_used_slots"
|
||||
labels:
|
||||
pool: "$1"
|
||||
|
||||
- match: airflow.pool.starving_tasks.*
|
||||
name: "airflow_pool_starving_tasks"
|
||||
labels:
|
||||
pool: "$1"
|
||||
|
||||
- match: airflow.executor.open_slots.*
|
||||
name: "airflow_executor_open_slots"
|
||||
labels:
|
||||
executor: "$1"
|
||||
|
||||
- match: airflow.executor.queued_tasks.*
|
||||
name: "airflow_executor_queued_tasks"
|
||||
labels:
|
||||
executor: "$1"
|
||||
|
||||
- match: airflow.executor.running_tasks.*
|
||||
name: "airflow_executor_running_tasks"
|
||||
labels:
|
||||
executor: "$1"
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
@@ -7,7 +7,7 @@ SPDX-License-Identifier: APACHE-2.0
|
||||
apiVersion: {{ include "common.capabilities.deployment.apiVersion" . }}
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: {{ printf "%s-exporter" (include "common.names.fullname" .) }}
|
||||
name: {{ template "airflow.metrics.fullname" . }}
|
||||
namespace: {{ include "common.names.namespace" . | quote }}
|
||||
{{- $versionLabel := dict "app.kubernetes.io/version" ( include "common.images.version" ( dict "imageRoot" .Values.metrics.image "chart" .Chart ) ) }}
|
||||
{{- $labels := include "common.tplvalues.merge" ( dict "values" ( list .Values.commonLabels $versionLabel ) "context" . ) }}
|
||||
@@ -25,6 +25,7 @@ spec:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
checksum/configmap: {{ include (print $.Template.BasePath "/metrics/configmap.yaml") . | sha256sum }}
|
||||
{{- if .Values.metrics.podAnnotations }}
|
||||
{{- include "common.tplvalues.render" (dict "value" .Values.metrics.podAnnotations "context" $) | nindent 8 }}
|
||||
{{- end }}
|
||||
@@ -47,6 +48,9 @@ spec:
|
||||
{{- if .Values.metrics.nodeSelector }}
|
||||
nodeSelector: {{- include "common.tplvalues.render" ( dict "value" .Values.metrics.nodeSelector "context" $) | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- if .Values.metrics.terminationGracePeriodSeconds }}
|
||||
terminationGracePeriodSeconds: {{ .Values.metrics.terminationGracePeriodSeconds }}
|
||||
{{- end }}
|
||||
{{- if .Values.metrics.tolerations }}
|
||||
tolerations: {{- include "common.tplvalues.render" (dict "value" .Values.metrics.tolerations "context" .) | nindent 8 }}
|
||||
{{- end }}
|
||||
@@ -59,12 +63,14 @@ spec:
|
||||
{{- if .Values.metrics.schedulerName }}
|
||||
schedulerName: {{ .Values.metrics.schedulerName }}
|
||||
{{- end }}
|
||||
serviceAccountName: {{ include "airflow.serviceAccountName" . }}
|
||||
{{- if .Values.metrics.podSecurityContext.enabled }}
|
||||
securityContext: {{- include "common.compatibility.renderSecurityContext" (dict "secContext" .Values.metrics.podSecurityContext "context" $) | nindent 8 }}
|
||||
{{- end }}
|
||||
containers:
|
||||
- image: {{ include "airflow.metrics.image" . | quote }}
|
||||
name: airflow-exporter
|
||||
- name: statsd-exporter
|
||||
image: {{ include "airflow.metrics.image" . | quote }}
|
||||
imagePullPolicy: {{ .Values.metrics.image.pullPolicy }}
|
||||
{{- if .Values.metrics.containerSecurityContext.enabled }}
|
||||
securityContext: {{- include "common.compatibility.renderSecurityContext" (dict "secContext" .Values.metrics.containerSecurityContext "context" $) | nindent 12 }}
|
||||
{{- end }}
|
||||
@@ -73,46 +79,43 @@ spec:
|
||||
{{- end }}
|
||||
{{- if .Values.diagnosticMode.enabled }}
|
||||
args: {{- include "common.tplvalues.render" (dict "value" .Values.diagnosticMode.args "context" $) | nindent 12 }}
|
||||
{{- else }}
|
||||
args:
|
||||
- "--statsd.mapping-config=/config/mappings.yml"
|
||||
{{- end }}
|
||||
env:
|
||||
- name: AIRFLOW_PROMETHEUS_DATABASE_BACKEND
|
||||
value: "postgres"
|
||||
- name: AIRFLOW_PROMETHEUS_DATABASE_HOST
|
||||
value: {{ include "airflow.database.host" . }}
|
||||
- name: AIRFLOW_PROMETHEUS_DATABASE_PORT
|
||||
value: {{ include "airflow.database.port" . }}
|
||||
- name: AIRFLOW_PROMETHEUS_DATABASE_USER
|
||||
value: {{ include "airflow.database.user" . }}
|
||||
{{- if or (not .Values.postgresql.enabled) .Values.postgresql.auth.enablePostgresUser }}
|
||||
- name: AIRFLOW_PROMETHEUS_DATABASE_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: {{ include "airflow.database.secretName" . }}
|
||||
key: {{ include "airflow.database.secretKey" . }}
|
||||
{{- else }}
|
||||
- name: ALLOW_EMPTY_PASSWORD
|
||||
value: "true"
|
||||
{{- end }}
|
||||
- name: AIRFLOW_PROMETHEUS_DATABASE_NAME
|
||||
value: {{ include "airflow.database.name" . }}
|
||||
{{- if .Values.metrics.extraEnvVars }}
|
||||
{{- include "common.tplvalues.render" (dict "value" .Values.metrics.extraEnvVars "context" $) | nindent 12 }}
|
||||
{{- end }}
|
||||
envFrom:
|
||||
{{- if .Values.metrics.extraEnvVarsCM }}
|
||||
- configMapRef:
|
||||
name: {{ .Values.metrics.extraEnvVarsCM }}
|
||||
{{- end }}
|
||||
{{- if .Values.metrics.extraEnvVarsSecret }}
|
||||
- secretRef:
|
||||
name: {{ .Values.metrics.extraEnvVarsSecret }}
|
||||
{{- end }}
|
||||
{{- if .Values.metrics.lifecycleHooks }}
|
||||
lifecycle: {{- include "common.tplvalues.render" (dict "value" .Values.metrics.lifecycleHooks "context" $) | nindent 12 }}
|
||||
{{- end }}
|
||||
{{- if not .Values.diagnosticMode.enabled }}
|
||||
{{- if .Values.metrics.customLivenessProbe }}
|
||||
livenessProbe: {{- include "common.tplvalues.render" (dict "value" .Values.metrics.customLivenessProbe "context" $) | trim | nindent 12 }}
|
||||
{{- else if .Values.metrics.livenessProbe.enabled }}
|
||||
livenessProbe: {{- include "common.tplvalues.render" (dict "value" (omit .Values.metrics.livenessProbe "enabled") "context" $) | nindent 12 }}
|
||||
tcpSocket:
|
||||
port: metrics
|
||||
{{- end }}
|
||||
{{- if .Values.metrics.customReadinessProbe }}
|
||||
readinessProbe: {{- include "common.tplvalues.render" (dict "value" .Values.metrics.customReadinessProbe "context" $) | nindent 12 }}
|
||||
{{- else if .Values.metrics.readinessProbe.enabled }}
|
||||
readinessProbe: {{- include "common.tplvalues.render" (dict "value" (omit .Values.metrics.readinessProbe "enabled") "context" $) | nindent 12 }}
|
||||
httpGet:
|
||||
path: /metrics
|
||||
port: metrics
|
||||
{{- end }}
|
||||
{{- if .Values.metrics.customStartupProbe }}
|
||||
startupProbe: {{- include "common.tplvalues.render" (dict "value" .Values.metrics.customStartupProbe "context" $) | nindent 12 }}
|
||||
{{- else if .Values.metrics.startupProbe.enabled }}
|
||||
startupProbe: {{- include "common.tplvalues.render" (dict "value" (omit .Values.metrics.startupProbe "enabled") "context" $) | nindent 12 }}
|
||||
tcpSocket:
|
||||
port: metrics
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
ports:
|
||||
- name: metrics
|
||||
containerPort: {{ .Values.metrics.containerPorts.http }}
|
||||
containerPort: {{ .Values.metrics.containerPorts.metrics }}
|
||||
- name: ingest
|
||||
containerPort: {{ .Values.metrics.containerPorts.ingest }}
|
||||
protocol: UDP
|
||||
{{- if .Values.metrics.resources }}
|
||||
resources: {{- toYaml .Values.metrics.resources | nindent 12 }}
|
||||
{{- else if ne .Values.metrics.resourcesPreset "none" }}
|
||||
@@ -122,12 +125,18 @@ spec:
|
||||
- name: empty-dir
|
||||
mountPath: /tmp
|
||||
subPath: tmp-dir
|
||||
- name: configuration
|
||||
mountPath: /config/mappings.yml
|
||||
subPath: mappings.yml
|
||||
{{- if .Values.metrics.extraVolumeMounts }}
|
||||
{{- include "common.tplvalues.render" (dict "value" .Values.metrics.extraVolumeMounts "context" $) | nindent 12 }}
|
||||
{{- end }}
|
||||
volumes:
|
||||
- name: empty-dir
|
||||
emptyDir: {}
|
||||
- name: configuration
|
||||
configMap:
|
||||
name: {{ include "airflow.metrics.configMapName" . }}
|
||||
{{- if .Values.metrics.extraVolumes }}
|
||||
{{- include "common.tplvalues.render" (dict "value" .Values.metrics.extraVolumes "context" $) | nindent 8 }}
|
||||
{{- end }}
|
||||
|
||||
@@ -3,11 +3,11 @@ Copyright Broadcom, Inc. All Rights Reserved.
|
||||
SPDX-License-Identifier: APACHE-2.0
|
||||
*/}}
|
||||
|
||||
{{- if .Values.metrics.networkPolicy.enabled }}
|
||||
{{- if and .Values.metrics.enabled .Values.metrics.networkPolicy.enabled }}
|
||||
kind: NetworkPolicy
|
||||
apiVersion: {{ include "common.capabilities.networkPolicy.apiVersion" . }}
|
||||
metadata:
|
||||
name: {{ printf "%s-exporter" (include "common.names.fullname" .) }}
|
||||
name: {{ template "airflow.metrics.fullname" . }}
|
||||
namespace: {{ include "common.names.namespace" . | quote }}
|
||||
labels: {{- include "common.labels.standard" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 4 }}
|
||||
app.kubernetes.io/component: metrics
|
||||
@@ -33,27 +33,19 @@ spec:
|
||||
protocol: UDP
|
||||
- port: 53
|
||||
protocol: TCP
|
||||
# Allow outbound connections to other cluster pods
|
||||
- ports:
|
||||
- port: {{ include "airflow.database.port" . | replace "\"" "" }}
|
||||
- port: {{ .Values.metrics.service.ports.http }}
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels: {{- include "common.labels.matchLabels" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 14 }}
|
||||
{{- if .Values.metrics.networkPolicy.extraEgress }}
|
||||
{{- include "common.tplvalues.render" ( dict "value" .Values.metrics.networkPolicy.extraEgress "context" $ ) | nindent 4 }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
ingress:
|
||||
- ports:
|
||||
- port: {{ .Values.metrics.service.ports.http }}
|
||||
- port: {{ include "airflow.database.port" . | replace "\"" "" }}
|
||||
- port: {{ .Values.metrics.containerPorts.http }}
|
||||
- port: {{ .Values.metrics.containerPorts.ingest }}
|
||||
- port: {{ .Values.metrics.containerPorts.metrics }}
|
||||
{{- if not .Values.metrics.networkPolicy.allowExternal }}
|
||||
from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
{{ printf "%s-exporter" (include "common.names.fullname" .) }}-client: "true"
|
||||
{{ printf "%s-client" (include "airflow.metrics.fullname" .) }}: "true"
|
||||
{{- if .Values.metrics.networkPolicy.ingressNSMatchLabels }}
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
|
||||
@@ -7,7 +7,7 @@ SPDX-License-Identifier: APACHE-2.0
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ printf "%s-exporter" (include "common.names.fullname" .) }}
|
||||
name: {{ template "airflow.metrics.fullname" . }}
|
||||
namespace: {{ include "common.names.namespace" . | quote }}
|
||||
{{- $versionLabel := dict "app.kubernetes.io/version" ( include "common.images.version" ( dict "imageRoot" .Values.metrics.image "chart" .Chart ) ) }}
|
||||
{{- $labels := include "common.tplvalues.merge" ( dict "values" ( list .Values.commonLabels $versionLabel ) "context" . ) }}
|
||||
@@ -24,8 +24,12 @@ spec:
|
||||
clusterIP: {{ .Values.metrics.service.clusterIP }}
|
||||
{{- end }}
|
||||
ports:
|
||||
- name: udp-ingest
|
||||
port: {{ .Values.metrics.service.ports.ingest }}
|
||||
protocol: UDP
|
||||
targetPort: ingest
|
||||
- name: http-metrics
|
||||
port: {{ .Values.metrics.service.ports.http }}
|
||||
port: {{ .Values.metrics.service.ports.metrics }}
|
||||
protocol: TCP
|
||||
targetPort: metrics
|
||||
{{- $podLabels := include "common.tplvalues.merge" ( dict "values" ( list .Values.metrics.podLabels .Values.commonLabels ) "context" . ) }}
|
||||
|
||||
@@ -7,7 +7,7 @@ SPDX-License-Identifier: APACHE-2.0
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: {{ printf "%s-exporter" (include "common.names.fullname" .) }}
|
||||
name: {{ template "airflow.metrics.fullname" . }}
|
||||
namespace: {{ default (include "common.names.namespace" .) .Values.metrics.serviceMonitor.namespace | quote }}
|
||||
{{- $versionLabel := dict "app.kubernetes.io/version" ( include "common.images.version" ( dict "imageRoot" .Values.metrics.image "chart" .Chart ) ) }}
|
||||
{{- $labels := include "common.tplvalues.merge" ( dict "values" ( list .Values.metrics.serviceMonitor.labels .Values.commonLabels $versionLabel ) "context" . ) }}
|
||||
|
||||
@@ -37,6 +37,9 @@ spec:
|
||||
- ports:
|
||||
- port: {{ .Values.service.ports.http }}
|
||||
- port: {{ .Values.worker.containerPorts.http }}
|
||||
{{- if .Values.metrics.enabled }}
|
||||
- port: {{ .Values.metrics.service.ports.ingest }}
|
||||
{{- end }}
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels: {{- include "common.labels.matchLabels" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 14 }}
|
||||
|
||||
@@ -36,6 +36,9 @@ spec:
|
||||
# Allow outbound connections to other cluster pods
|
||||
- ports:
|
||||
- port: {{ .Values.service.ports.http }}
|
||||
{{- if .Values.metrics.enabled }}
|
||||
- port: {{ .Values.metrics.service.ports.ingest }}
|
||||
{{- end }}
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels: {{- include "common.labels.matchLabels" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 14 }}
|
||||
|
||||
@@ -10,6 +10,7 @@ metadata:
|
||||
name: {{ template "airflow.web.fullname" . }}
|
||||
namespace: {{ include "common.names.namespace" . | quote }}
|
||||
labels: {{- include "common.labels.standard" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 4 }}
|
||||
app.kubernetes.io/component: web
|
||||
{{- if .Values.commonAnnotations }}
|
||||
annotations: {{- include "common.tplvalues.render" (dict "value" .Values.commonAnnotations "context" $) | nindent 4 }}
|
||||
{{- end }}
|
||||
|
||||
@@ -37,6 +37,9 @@ spec:
|
||||
- ports:
|
||||
- port: {{ .Values.web.containerPorts.http }}
|
||||
- port: {{ .Values.service.ports.http }}
|
||||
{{- if .Values.metrics.enabled }}
|
||||
- port: {{ .Values.metrics.service.ports.ingest }}
|
||||
{{- end }}
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels: {{- include "common.labels.matchLabels" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 14 }}
|
||||
|
||||
@@ -37,6 +37,9 @@ spec:
|
||||
- ports:
|
||||
- port: {{ .Values.service.ports.http }}
|
||||
- port: {{ .Values.worker.containerPorts.http }}
|
||||
{{- if .Values.metrics.enabled }}
|
||||
- port: {{ .Values.metrics.service.ports.ingest }}
|
||||
{{- end }}
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels: {{- include "common.labels.matchLabels" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 14 }}
|
||||
|
||||
+109
-48
@@ -2684,24 +2684,26 @@ rbac:
|
||||
## - list
|
||||
##
|
||||
rules: []
|
||||
## @section Airflow metrics parameters
|
||||
|
||||
## @section StatsD metrics parameters
|
||||
##
|
||||
metrics:
|
||||
## @param metrics.enabled Whether or not to create a standalone Airflow exporter to expose Airflow metrics
|
||||
## @param metrics.enabled Enable a StatsD exporter that collects StatsD metrics from Airflow components and expose them as Prometheus metrics
|
||||
##
|
||||
enabled: false
|
||||
## Bitnami Airflow exporter image
|
||||
## ref: https://hub.docker.com/r/bitnami/airflow-exporter/tags/
|
||||
## @param metrics.image.registry [default: REGISTRY_NAME] Airflow exporter image registry
|
||||
## @param metrics.image.repository [default: REPOSITORY_NAME/airflow-exporter] Airflow exporter image repository
|
||||
## @skip metrics.image.tag Airflow exporter image tag (immutable tags are recommended)
|
||||
## @param metrics.image.digest Airflow exporter image digest in the way sha256:aa.... Please note this parameter, if set, will override the tag
|
||||
## @param metrics.image.pullPolicy Airflow exporter image pull policy
|
||||
## @param metrics.image.pullSecrets Airflow exporter image pull secrets
|
||||
## Bitnami StatsD exporter image
|
||||
## ref: https://hub.docker.com/r/bitnami/statsd-exporter/tags/
|
||||
## @param metrics.image.registry [default: REGISTRY_NAME] StatsD exporter image registry
|
||||
## @param metrics.image.repository [default: REPOSITORY_NAME/statsd-exporter] StatsD exporter image repository
|
||||
## @skip metrics.image.tag StatsD exporter image tag (immutable tags are recommended)
|
||||
## @param metrics.image.digest StatsD exporter image digest in the way sha256:aa.... Please note this parameter, if set, will override the tag
|
||||
## @param metrics.image.pullPolicy StatsD exporter image pull policy
|
||||
## @param metrics.image.pullSecrets StatsD exporter image pull secrets
|
||||
##
|
||||
image:
|
||||
registry: docker.io
|
||||
repository: bitnami/airflow-exporter
|
||||
tag: 0.20220314.0-debian-12-r51
|
||||
repository: bitnami/statsd-exporter
|
||||
tag: 0.28.0-debian-12-r0
|
||||
digest: ""
|
||||
pullPolicy: IfNotPresent
|
||||
## Optionally specify an array of imagePullSecrets.
|
||||
@@ -2712,20 +2714,19 @@ metrics:
|
||||
## - myRegistryKeySecretName
|
||||
##
|
||||
pullSecrets: []
|
||||
## @param metrics.extraEnvVars Array with extra environment variables to add Airflow exporter pods
|
||||
## @param metrics.configuration Specify content for StatsD exporter's mappings.yml
|
||||
##
|
||||
extraEnvVars: []
|
||||
## @param metrics.extraEnvVarsCM ConfigMap containing extra environment variables for Airflow exporter pods
|
||||
configuration: ""
|
||||
## @param metrics.existingConfigmap Name of an existing config map containing the StatsD exporter's mappings.yml
|
||||
##
|
||||
extraEnvVarsCM: ""
|
||||
## @param metrics.extraEnvVarsSecret Secret containing extra environment variables (in case of sensitive data) for Airflow exporter pods
|
||||
##
|
||||
extraEnvVarsSecret: ""
|
||||
## @param metrics.containerPorts.http Airflow exporter metrics container port
|
||||
existingConfigmap: ""
|
||||
## @param metrics.containerPorts.ingest StatsD exporter ingest container port (used for the metrics ingestion from Airflow components)
|
||||
## @param metrics.containerPorts.metrics StatsD exporter metrics container port (used to expose Prometheus metrics)
|
||||
##
|
||||
containerPorts:
|
||||
http: 9112
|
||||
## Airflow exporter resource requests and limits
|
||||
ingest: 9125
|
||||
metrics: 9102
|
||||
## StatsD exporter resource requests and limits
|
||||
## ref: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
|
||||
## @param metrics.resourcesPreset Set container resources according to one common preset (allowed values: none, nano, micro, small, medium, large, xlarge, 2xlarge). This is ignored if metrics.resources is set (metrics.resources is recommended for production).
|
||||
## More information: https://github.com/bitnami/charts/blob/main/bitnami/common/templates/_resources.tpl#L15
|
||||
@@ -2742,13 +2743,13 @@ metrics:
|
||||
## memory: 1024Mi
|
||||
##
|
||||
resources: {}
|
||||
## Airflow exporter pods' Security Context
|
||||
## StatsD exporter pods' Security Context
|
||||
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod
|
||||
## @param metrics.podSecurityContext.enabled Enable security context for the pods
|
||||
## @param metrics.podSecurityContext.fsGroupChangePolicy Set filesystem group change policy
|
||||
## @param metrics.podSecurityContext.sysctls Set kernel settings using the sysctl interface
|
||||
## @param metrics.podSecurityContext.supplementalGroups Set filesystem extra groups
|
||||
## @param metrics.podSecurityContext.fsGroup Set Airflow exporter pod's Security Context fsGroup
|
||||
## @param metrics.podSecurityContext.fsGroup Set StatsD exporter pod's Security Context fsGroup
|
||||
##
|
||||
podSecurityContext:
|
||||
enabled: true
|
||||
@@ -2756,18 +2757,18 @@ metrics:
|
||||
sysctls: []
|
||||
supplementalGroups: []
|
||||
fsGroup: 1001
|
||||
## Airflow exporter containers' Security Context
|
||||
## StatsD exporter containers' Security Context
|
||||
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container
|
||||
## @param metrics.containerSecurityContext.enabled Enable Airflow exporter containers' Security Context
|
||||
## @param metrics.containerSecurityContext.enabled Enable StatsD exporter containers' Security Context
|
||||
## @param metrics.containerSecurityContext.seLinuxOptions [object,nullable] Set SELinux options in container
|
||||
## @param metrics.containerSecurityContext.runAsUser Set Airflow exporter containers' Security Context runAsUser
|
||||
## @param metrics.containerSecurityContext.runAsGroup Set Airflow exporter containers' Security Context runAsGroup
|
||||
## @param metrics.containerSecurityContext.runAsNonRoot Set Airflow exporter containers' Security Context runAsNonRoot
|
||||
## @param metrics.containerSecurityContext.privileged Set metrics container's Security Context privileged
|
||||
## @param metrics.containerSecurityContext.allowPrivilegeEscalation Set metrics container's Security Context allowPrivilegeEscalation
|
||||
## @param metrics.containerSecurityContext.readOnlyRootFilesystem Set metrics container's Security Context readOnlyRootFilesystem
|
||||
## @param metrics.containerSecurityContext.runAsUser Set StatsD exporter containers' Security Context runAsUser
|
||||
## @param metrics.containerSecurityContext.runAsGroup Set StatsD exporter containers' Security Context runAsGroup
|
||||
## @param metrics.containerSecurityContext.runAsNonRoot Set StatsD exporter containers' Security Context runAsNonRoot
|
||||
## @param metrics.containerSecurityContext.privileged Set StatsD exporter containers' Security Context privileged
|
||||
## @param metrics.containerSecurityContext.allowPrivilegeEscalation Set StatsD exporter containers' Security Context allowPrivilegeEscalation
|
||||
## @param metrics.containerSecurityContext.readOnlyRootFilesystem Set StatsD exporter containers' Security Context readOnlyRootFilesystem
|
||||
## @param metrics.containerSecurityContext.capabilities.drop List of capabilities to be dropped
|
||||
## @param metrics.containerSecurityContext.seccompProfile.type Set container's Security Context seccomp profile
|
||||
## @param metrics.containerSecurityContext.seccompProfile.type Set containers' Security Context seccomp profile
|
||||
## e.g:
|
||||
## containerSecurityContext:
|
||||
## enabled: true
|
||||
@@ -2788,21 +2789,74 @@ metrics:
|
||||
drop: ["ALL"]
|
||||
seccompProfile:
|
||||
type: "RuntimeDefault"
|
||||
## @param metrics.lifecycleHooks for the Airflow exporter container(s) to automate configuration before or after startup
|
||||
## Configure extra options for StatsD exporter containers' liveness, readiness and startup probes
|
||||
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes
|
||||
## @param metrics.livenessProbe.enabled Enable livenessProbe on StatsD exporter containers
|
||||
## @param metrics.livenessProbe.initialDelaySeconds Initial delay seconds for livenessProbe
|
||||
## @param metrics.livenessProbe.periodSeconds Period seconds for livenessProbe
|
||||
## @param metrics.livenessProbe.timeoutSeconds Timeout seconds for livenessProbe
|
||||
## @param metrics.livenessProbe.failureThreshold Failure threshold for livenessProbe
|
||||
## @param metrics.livenessProbe.successThreshold Success threshold for livenessProbe
|
||||
##
|
||||
livenessProbe:
|
||||
enabled: true
|
||||
initialDelaySeconds: 180
|
||||
periodSeconds: 20
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 6
|
||||
successThreshold: 1
|
||||
## @param metrics.readinessProbe.enabled Enable readinessProbe on StatsD exporter containers
|
||||
## @param metrics.readinessProbe.initialDelaySeconds Initial delay seconds for readinessProbe
|
||||
## @param metrics.readinessProbe.periodSeconds Period seconds for readinessProbe
|
||||
## @param metrics.readinessProbe.timeoutSeconds Timeout seconds for readinessProbe
|
||||
## @param metrics.readinessProbe.failureThreshold Failure threshold for readinessProbe
|
||||
## @param metrics.readinessProbe.successThreshold Success threshold for readinessProbe
|
||||
##
|
||||
readinessProbe:
|
||||
enabled: true
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 6
|
||||
successThreshold: 1
|
||||
## @param metrics.startupProbe.enabled Enable startupProbe on StatsD exporter containers
|
||||
## @param metrics.startupProbe.initialDelaySeconds Initial delay seconds for startupProbe
|
||||
## @param metrics.startupProbe.periodSeconds Period seconds for startupProbe
|
||||
## @param metrics.startupProbe.timeoutSeconds Timeout seconds for startupProbe
|
||||
## @param metrics.startupProbe.failureThreshold Failure threshold for startupProbe
|
||||
## @param metrics.startupProbe.successThreshold Success threshold for startupProbe
|
||||
##
|
||||
startupProbe:
|
||||
enabled: false
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 1
|
||||
failureThreshold: 15
|
||||
successThreshold: 1
|
||||
## @param metrics.customLivenessProbe Custom livenessProbe that overrides the default one
|
||||
##
|
||||
customLivenessProbe: {}
|
||||
## @param metrics.customReadinessProbe Custom readinessProbe that overrides the default one
|
||||
##
|
||||
customReadinessProbe: {}
|
||||
## @param metrics.customStartupProbe Custom startupProbe that overrides the default one
|
||||
##
|
||||
customStartupProbe: {}
|
||||
## @param metrics.lifecycleHooks for the StatsD exporter containers' to automate configuration before or after startup
|
||||
##
|
||||
lifecycleHooks: {}
|
||||
## @param metrics.automountServiceAccountToken Mount Service Account token in pod
|
||||
##
|
||||
automountServiceAccountToken: false
|
||||
## @param metrics.hostAliases Airflow exporter pods host aliases
|
||||
## @param metrics.hostAliases StatsD exporter pods host aliases
|
||||
## https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/
|
||||
##
|
||||
hostAliases: []
|
||||
## @param metrics.podLabels Extra labels for Airflow exporter pods
|
||||
## @param metrics.podLabels Extra labels for StatsD exporter pods
|
||||
## Ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
|
||||
##
|
||||
podLabels: {}
|
||||
## @param metrics.podAnnotations Extra annotations for Airflow exporter pods
|
||||
## @param metrics.podAnnotations Extra annotations for StatsD exporter pods
|
||||
## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
|
||||
##
|
||||
podAnnotations: {}
|
||||
@@ -2833,16 +2887,16 @@ metrics:
|
||||
## - e2e-az2
|
||||
##
|
||||
values: []
|
||||
## @param metrics.affinity Affinity for pod assignment
|
||||
## @param metrics.affinity Affinity for StatsD exporter pods assignment
|
||||
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
|
||||
## Note: metrics.podAffinityPreset, metrics.podAntiAffinityPreset, and metrics.nodeAffinityPreset will be ignored when it's set
|
||||
##
|
||||
affinity: {}
|
||||
## @param metrics.nodeSelector Node labels for pod assignment
|
||||
## @param metrics.nodeSelector Node labels for StatsD exporter pods assignment
|
||||
## Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
|
||||
##
|
||||
nodeSelector: {}
|
||||
## @param metrics.priorityClassName Airflow exporter pods' priorityClassName
|
||||
## @param metrics.priorityClassName StatsD exporter pods' priorityClassName
|
||||
##
|
||||
priorityClassName: ""
|
||||
## @param metrics.tolerations Tolerations for pod assignment
|
||||
@@ -2853,23 +2907,29 @@ metrics:
|
||||
## Ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/#spread-constraints-for-pods
|
||||
##
|
||||
topologySpreadConstraints: []
|
||||
## @param metrics.schedulerName Name of the k8s scheduler (other than default) for Airflow exporter
|
||||
## @param metrics.schedulerName Name of the k8s scheduler (other than default) for StatsD exporter
|
||||
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
|
||||
##
|
||||
schedulerName: ""
|
||||
## @param metrics.extraVolumes Optionally specify extra list of additional volumes for the Airflow exporter pods
|
||||
## @param metrics.terminationGracePeriodSeconds Seconds StatsD exporter pod needs to terminate gracefully
|
||||
## ref: https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
|
||||
##
|
||||
terminationGracePeriodSeconds: ""
|
||||
## @param metrics.extraVolumes Optionally specify extra list of additional volumes for the StatsD exporter pods
|
||||
##
|
||||
extraVolumes: []
|
||||
## @param metrics.extraVolumeMounts Optionally specify extra list of additional volumeMounts for the Airflow exporter containers
|
||||
## @param metrics.extraVolumeMounts Optionally specify extra list of additional volumeMounts for the StatsD exporter containers
|
||||
##
|
||||
extraVolumeMounts: []
|
||||
## Airflow exporter service configuration
|
||||
## StatsD metrics service configuration
|
||||
##
|
||||
service:
|
||||
## @param metrics.service.ports.http Airflow exporter metrics service port
|
||||
## @param metrics.service.ports.ingest StatsD exporter ingest service port (used for the metrics ingestion from Airflow components)
|
||||
## @param metrics.service.ports.metrics StatsD exporter metrics service port (used to expose Prometheus metrics)
|
||||
##
|
||||
ports:
|
||||
http: 9112
|
||||
ingest: 9125
|
||||
metrics: 9102
|
||||
## @param metrics.service.clusterIP Static clusterIP or None for headless services
|
||||
## ref: https://kubernetes.io/docs/concepts/services-networking/service/#choosing-your-own-ip-address
|
||||
##
|
||||
@@ -2879,11 +2939,11 @@ metrics:
|
||||
## ref: https://kubernetes.io/docs/concepts/services-networking/service/
|
||||
##
|
||||
sessionAffinity: None
|
||||
## @param metrics.service.annotations [object] Annotations for the Airflow exporter service
|
||||
## @param metrics.service.annotations [object] Annotations for the StatsD metrics service
|
||||
##
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "{{ .Values.metrics.service.ports.http }}"
|
||||
prometheus.io/port: "{{ .Values.metrics.service.ports.metrics }}"
|
||||
## Prometheus Operator ServiceMonitor configuration
|
||||
##
|
||||
serviceMonitor:
|
||||
@@ -2975,6 +3035,7 @@ metrics:
|
||||
##
|
||||
ingressNSMatchLabels: {}
|
||||
ingressNSPodMatchLabels: {}
|
||||
|
||||
## @section Airflow database parameters
|
||||
|
||||
## PostgreSQL chart configuration
|
||||
|
||||
Reference in New Issue
Block a user