Bitnami/charts
2
0
Fork 0
mirror of https://github.com/bitnami/charts.git synced 2026-03-08 00:37:41 +08:00
Code Issues Packages Projects Releases Wiki Activity

[bitnami/kafka] Include new parameter to allow setting external names for load balanced external listeners (#8983)

Browse Source 
Changes about PR bitnami/charts#8983
* Include new variable to allow set external names for external listeners. This fix bitnami/charts#8875
* Remove loadbalancersIPs validations from NOTES.txt and include loadBalancerAnnotations parameter to allow users set different annotations for each external load balancer

Co-authored-by: Juan Ariza Toledano <juanariza@vmware.com>
Signed-off-by: Fran Mulero <fmulero@vmware.com>
This commit is contained in:
Fran Mulero
2022-02-15 08:40:41 +01:00
committed by GitHub
parent 6d39fc7d04
commit a8028ded2e
8 changed files with 97 additions and 87 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
bitnami/kafka/Chart.yaml
View File

@@ -29,4 +29,4 @@ name: kafka
sources:
- https://github.com/bitnami/bitnami-docker-kafka
- https://kafka.apache.org/
version: 15.1.1
version: 15.2.0

82
bitnami/kafka/README.md
View File

@@ -7,7 +7,7 @@ Apache Kafka is a distributed streaming platform designed to build real-time pip
[Overview of Apache Kafka](http://kafka.apache.org/)
Trademarks: This software listing is packaged by Bitnami. The respective trademarks mentioned in the offering are owned by the respective companies, and use of them does not imply any affiliation or endorsement.
## TL;DR
```console
@@ -217,45 +217,47 @@ The command removes all the Kubernetes components associated with the chart and
### Traffic Exposure parameters
| Name | Description | Value |
| ------------------------------------------------- | ---------------------------------------------------------------------------------------------- | --------------------- |
| `service.type` | Kubernetes Service type | `ClusterIP` |
| `service.ports.client` | Kafka svc port for client connections | `9092` |
| `service.ports.internal` | Kafka svc port for inter-broker connections | `9093` |
| `service.ports.external` | Kafka svc port for external connections | `9094` |
| `service.nodePorts.client` | Node port for the Kafka client connections | `""` |
| `service.nodePorts.external` | Node port for the Kafka external connections | `""` |
| `service.sessionAffinity` | Control where client requests go, to the same pod or round-robin | `None` |
| `service.clusterIP` | Kafka service Cluster IP | `""` |
| `service.loadBalancerIP` | Kafka service Load Balancer IP | `""` |
| `service.loadBalancerSourceRanges` | Kafka service Load Balancer sources | `[]` |
| `service.externalTrafficPolicy` | Kafka service external traffic policy | `Cluster` |
| `service.annotations` | Additional custom annotations for Kafka service | `{}` |
| `service.extraPorts` | Extra ports to expose in the Kafka service (normally used with the `sidecar` value) | `[]` |
| `externalAccess.enabled` | Enable Kubernetes external cluster access to Kafka brokers | `false` |
| `externalAccess.autoDiscovery.enabled` | Enable using an init container to auto-detect external IPs/ports by querying the K8s API | `false` |
| `externalAccess.autoDiscovery.image.registry` | Init container auto-discovery image registry | `docker.io` |
| `externalAccess.autoDiscovery.image.repository` | Init container auto-discovery image repository | `bitnami/kubectl` |
| `externalAccess.autoDiscovery.image.tag` | Init container auto-discovery image tag (immutable tags are recommended) | `1.23.3-debian-10-r7` |
| `externalAccess.autoDiscovery.image.pullPolicy` | Init container auto-discovery image pull policy | `IfNotPresent` |
| `externalAccess.autoDiscovery.image.pullSecrets` | Init container auto-discovery image pull secrets | `[]` |
| `externalAccess.autoDiscovery.resources.limits` | The resources limits for the auto-discovery init container | `{}` |
| `externalAccess.autoDiscovery.resources.requests` | The requested resources for the auto-discovery init container | `{}` |
| `externalAccess.service.type` | Kubernetes Service type for external access. It can be NodePort or LoadBalancer | `LoadBalancer` |
| `externalAccess.service.ports.external` | Kafka port used for external access when service type is LoadBalancer | `9094` |
| `externalAccess.service.loadBalancerIPs` | Array of load balancer IPs for each Kafka broker. Length must be the same as replicaCount | `[]` |
| `externalAccess.service.loadBalancerSourceRanges` | Address(es) that are allowed when service is LoadBalancer | `[]` |
| `externalAccess.service.nodePorts` | Array of node ports used for each Kafka broker. Length must be the same as replicaCount | `[]` |
| `externalAccess.service.useHostIPs` | Use service host IPs to configure Kafka external listener when service type is NodePort | `false` |
| `externalAccess.service.usePodIPs` | using the MY_POD_IP address for external access. | `false` |
| `externalAccess.service.domain` | Domain or external ip used to configure Kafka external listener when service type is NodePort | `""` |
| `externalAccess.service.annotations` | Service annotations for external access | `{}` |
| `externalAccess.service.extraPorts` | Extra ports to expose in the Kafka external service | `[]` |
| `networkPolicy.enabled` | Specifies whether a NetworkPolicy should be created | `false` |
| `networkPolicy.allowExternal` | Don't require client label for connections | `true` |
| `networkPolicy.explicitNamespacesSelector` | A Kubernetes LabelSelector to explicitly select namespaces from which traffic could be allowed | `{}` |
| `networkPolicy.externalAccess.from` | customize the from section for External Access on tcp-external port | `[]` |
| `networkPolicy.egressRules.customRules` | Custom network policy rule | `{}` |
| Name | Description | Value |
| ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- |
| `service.type` | Kubernetes Service type | `ClusterIP` |
| `service.ports.client` | Kafka svc port for client connections | `9092` |
| `service.ports.internal` | Kafka svc port for inter-broker connections | `9093` |
| `service.ports.external` | Kafka svc port for external connections | `9094` |
| `service.nodePorts.client` | Node port for the Kafka client connections | `""` |
| `service.nodePorts.external` | Node port for the Kafka external connections | `""` |
| `service.sessionAffinity` | Control where client requests go, to the same pod or round-robin | `None` |
| `service.clusterIP` | Kafka service Cluster IP | `""` |
| `service.loadBalancerIP` | Kafka service Load Balancer IP | `""` |
| `service.loadBalancerSourceRanges` | Kafka service Load Balancer sources | `[]` |
| `service.externalTrafficPolicy` | Kafka service external traffic policy | `Cluster` |
| `service.annotations` | Additional custom annotations for Kafka service | `{}` |
| `service.extraPorts` | Extra ports to expose in the Kafka service (normally used with the `sidecar` value) | `[]` |
| `externalAccess.enabled` | Enable Kubernetes external cluster access to Kafka brokers | `false` |
| `externalAccess.autoDiscovery.enabled` | Enable using an init container to auto-detect external IPs/ports by querying the K8s API | `false` |
| `externalAccess.autoDiscovery.image.registry` | Init container auto-discovery image registry | `docker.io` |
| `externalAccess.autoDiscovery.image.repository` | Init container auto-discovery image repository | `bitnami/kubectl` |
| `externalAccess.autoDiscovery.image.tag` | Init container auto-discovery image tag (immutable tags are recommended) | `1.23.3-debian-10-r7` |
| `externalAccess.autoDiscovery.image.pullPolicy` | Init container auto-discovery image pull policy | `IfNotPresent` |
| `externalAccess.autoDiscovery.image.pullSecrets` | Init container auto-discovery image pull secrets | `[]` |
| `externalAccess.autoDiscovery.resources.limits` | The resources limits for the auto-discovery init container | `{}` |
| `externalAccess.autoDiscovery.resources.requests` | The requested resources for the auto-discovery init container | `{}` |
| `externalAccess.service.type` | Kubernetes Service type for external access. It can be NodePort or LoadBalancer | `LoadBalancer` |
| `externalAccess.service.ports.external` | Kafka port used for external access when service type is LoadBalancer | `9094` |
| `externalAccess.service.loadBalancerAnnotations` | Array of load balancer annotations for external services. Useful to apply different annotations to each external load balancer. Length must be the same as replicaCount | `[]` |
| `externalAccess.service.loadBalancerIPs` | Array of load balancer IPs for each Kafka broker. Length must be the same as replicaCount | `[]` |
| `externalAccess.service.loadBalancerNames` | Array of load balancer names for each Kafka broker. It takes precedence over load balancer IPs, if loadBalancerNames is not defined then IP addresss will be used for external access in broker configuration. Length must be the same as replicaCount | `[]` |
| `externalAccess.service.loadBalancerSourceRanges` | Address(es) that are allowed when service is LoadBalancer | `[]` |
| `externalAccess.service.nodePorts` | Array of node ports used for each Kafka broker. Length must be the same as replicaCount | `[]` |
| `externalAccess.service.useHostIPs` | Use service host IPs to configure Kafka external listener when service type is NodePort | `false` |
| `externalAccess.service.usePodIPs` | using the MY_POD_IP address for external access. | `false` |
| `externalAccess.service.domain` | Domain or external ip used to configure Kafka external listener when service type is NodePort | `""` |
| `externalAccess.service.annotations` | Service annotations for external access | `{}` |
| `externalAccess.service.extraPorts` | Extra ports to expose in the Kafka external service | `[]` |
| `networkPolicy.enabled` | Specifies whether a NetworkPolicy should be created | `false` |
| `networkPolicy.allowExternal` | Don't require client label for connections | `true` |
| `networkPolicy.explicitNamespacesSelector` | A Kubernetes LabelSelector to explicitly select namespaces from which traffic could be allowed | `{}` |
| `networkPolicy.externalAccess.from` | customize the from section for External Access on tcp-external port | `[]` |
| `networkPolicy.egressRules.customRules` | Custom network policy rule | `{}` |
### Persistence parameters

33
bitnami/kafka/templates/NOTES.txt
View File

@@ -32,38 +32,6 @@ In order to replicate the container startup scripts execute this command:
{{- $tlsEndpointIdentificationAlgorithm := default "" .Values.auth.tls.endpointIdentificationAlgorithm -}}
{{- $tlsPassword := .Values.auth.tls.password -}}
{{- $servicePort := int .Values.service.ports.client -}}
{{- $loadBalancerIPListLength := len .Values.externalAccess.service.loadBalancerIPs -}}
{{- if and .Values.externalAccess.enabled (not .Values.externalAccess.autoDiscovery.enabled) (not (eq $replicaCount $loadBalancerIPListLength )) (eq .Values.externalAccess.service.type "LoadBalancer") }}
###############################################################################
### ERROR: You enabled external access to Kafka brokers without specifying ###
### the array of load balancer IPs for Kafka brokers. ###
###############################################################################
This deployment will be incomplete until you configure the array of load balancer
IPs for Kafka brokers. To complete your deployment follow the steps below:
1. Wait for the load balancer IPs (it may take a few minutes for them to be available):
kubectl get svc --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ template "kafka.name" . }},app.kubernetes.io/instance={{ .Release.Name }},app.kubernetes.io/component=kafka,pod" -w
2. Obtain the load balancer IPs and upgrade your chart:
{{- range $i, $e := until $replicaCount }}
LOAD_BALANCER_IP_{{ add $i 1 }}="$(kubectl get svc --namespace {{ $releaseNamespace }} {{ $fullname }}-{{ $i }}-external -o jsonpath='{.status.loadBalancer.ingress[0].ip}')"
{{- end }}
3. Upgrade you chart:
helm upgrade --namespace {{ .Release.Namespace }} {{ .Release.Name }} bitnami/{{ .Chart.Name }} \
--set replicaCount={{ $replicaCount }} \
--set externalAccess.enabled=true \
{{- range $i, $e := until $replicaCount }}
--set externalAccess.service.loadBalancerIPs[{{ $i }}]=$LOAD_BALANCER_IP_{{ add $i 1 }} \
{{- end }}
--set externalAccess.service.type=LoadBalancer
{{- else }}
{{- if and (or (eq .Values.service.type "LoadBalancer") .Values.externalAccess.enabled) (eq $externalClientProtocol "PLAINTEXT") }}
---------------------------------------------------------------------------------------------
@@ -330,7 +298,6 @@ ssl.endpoint.identification.algorithm=
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- include "kafka.checkRollingTags" . }}
{{- include "kafka.validateValues" . }}

29
bitnami/kafka/templates/_helpers.tpl
View File

@@ -336,6 +336,10 @@ Compile all warnings into a single message, and call fail.
{{- $messages := append $messages (include "kafka.validateValues.nodePortListLength" .) -}}
{{- $messages := append $messages (include "kafka.validateValues.externalAccessServiceType" .) -}}
{{- $messages := append $messages (include "kafka.validateValues.externalAccessAutoDiscoveryRBAC" .) -}}
{{- $messages := append $messages (include "kafka.validateValues.externalAccessAutoDiscoveryIPsOrNames" .) -}}
{{- $messages := append $messages (include "kafka.validateValues.externalAccessServiceList" (dict "element" "loadBalancerIPs" "context" .)) -}}
{{- $messages := append $messages (include "kafka.validateValues.externalAccessServiceList" (dict "element" "loadBalancerNames" "context" .)) -}}
{{- $messages := append $messages (include "kafka.validateValues.externalAccessServiceList" (dict "element" "loadBalancerAnnotations" "context" . )) -}}
{{- $messages := append $messages (include "kafka.validateValues.saslMechanisms" .) -}}
{{- $messages := append $messages (include "kafka.validateValues.tlsSecrets" .) -}}
{{- $messages := append $messages (include "kafka.validateValues.tlsSecrets.length" .) -}}
@@ -356,7 +360,7 @@ kafka: auth.clientProtocol auth.externalClientProtocol auth.interBrokerProtocol
{{- end -}}
{{- end -}}
{{/* Validate values of Kafka - number of replicas must be the same than NodePort list */}}
{{/* Validate values of Kafka - number of replicas must be the same as NodePort list */}}
{{- define "kafka.validateValues.nodePortListLength" -}}
{{- $replicaCount := int .Values.replicaCount }}
{{- $nodePortListLength := len .Values.externalAccess.service.nodePorts }}
@@ -385,6 +389,29 @@ kafka: rbac.create
{{- end -}}
{{- end -}}
{{/* Validate values of Kafka - LoadBalancerIPs or LoadBalancerNames should be set when autoDiscovery is disabled */}}
{{- define "kafka.validateValues.externalAccessAutoDiscoveryIPsOrNames" -}}
{{- $loadBalancerNameListLength := len .Values.externalAccess.service.loadBalancerNames -}}
{{- $loadBalancerIPListLength := len .Values.externalAccess.service.loadBalancerIPs -}}
{{- if and .Values.externalAccess.enabled (eq .Values.externalAccess.service.type "LoadBalancer") (not .Values.externalAccess.autoDiscovery.enabled) (eq $loadBalancerNameListLength 0) (eq $loadBalancerIPListLength 0) }}
kafka: externalAccess.service.loadBalancerNames or externalAccess.service.loadBalancerIPs
By specifying "externalAccess.enabled=true", "externalAccess.autoDiscovery.enabled=false" and
"externalAccess.service.type=LoadBalancer" at least one of externalAccess.service.loadBalancerNames
or externalAccess.service.loadBalancerIPs must be set and the length of those arrays must be equal
to the number of replicas.
{{- end -}}
{{- end -}}
{{/* Validate values of Kafka - number of replicas must be the same as loadBalancerIPs list */}}
{{- define "kafka.validateValues.externalAccessServiceList" -}}
{{- $replicaCount := int .context.Values.replicaCount }}
{{- $listLength := len (get .context.Values.externalAccess.service .element) -}}
{{- if and .context.Values.externalAccess.enabled (not .context.Values.externalAccess.autoDiscovery.enabled) (eq .context.Values.externalAccess.service.type "LoadBalancer") (gt $listLength 0) (not (eq $replicaCount $listLength)) }}
kafka: externalAccess.service.{{ .element }}
Number of replicas and {{ .element }} array length must be the same. Currently: replicaCount = {{ $replicaCount }} and {{ .element }} = {{ $listLength }}
{{- end -}}
{{- end -}}
{{/* Validate values of Kafka - SASL mechanisms must be provided when using SASL */}}
{{- define "kafka.validateValues.saslMechanisms" -}}
{{- if and (or (.Values.auth.clientProtocol | regexFind "sasl") (.Values.auth.interBrokerProtocol | regexFind "sasl") (and .Values.zookeeper.auth.enabled .Values.auth.sasl.jaas.zookeeperUser)) (not .Values.auth.sasl.mechanisms) }}

14
bitnami/kafka/templates/scripts-configmap.yaml
View File

@@ -86,20 +86,20 @@ data:
# Configure external ip and port
{{- if eq .Values.externalAccess.service.type "LoadBalancer" }}
{{- if .Values.externalAccess.autoDiscovery.enabled }}
export EXTERNAL_ACCESS_IP="$(<${SHARED_FILE})"
export EXTERNAL_ACCESS_HOST="$(<${SHARED_FILE})"
{{- else }}
export EXTERNAL_ACCESS_IP=$(echo '{{ .Values.externalAccess.service.loadBalancerIPs }}' | tr -d '[]' | cut -d ' ' -f "$(($ID + 1))")
export EXTERNAL_ACCESS_HOST=$(echo '{{ .Values.externalAccess.service.loadBalancerNames | default .Values.externalAccess.service.loadBalancerIPs }}' | tr -d '[]' | cut -d ' ' -f "$(($ID + 1))")
{{- end }}
export EXTERNAL_ACCESS_PORT={{ .Values.externalAccess.service.ports.external }}
{{- else if eq .Values.externalAccess.service.type "NodePort" }}
{{- if and .Values.externalAccess.service.usePodIPs .Values.externalAccess.autoDiscovery.enabled }}
export EXTERNAL_ACCESS_IP="${MY_POD_IP}"
export EXTERNAL_ACCESS_HOST="${MY_POD_IP}"
{{- else if or .Values.externalAccess.service.useHostIPs .Values.externalAccess.autoDiscovery.enabled }}
export EXTERNAL_ACCESS_IP="${HOST_IP}"
export EXTERNAL_ACCESS_HOST="${HOST_IP}"
{{- else if .Values.externalAccess.service.domain }}
export EXTERNAL_ACCESS_IP={{ .Values.externalAccess.service.domain }}
export EXTERNAL_ACCESS_HOST={{ .Values.externalAccess.service.domain }}
{{- else }}
export EXTERNAL_ACCESS_IP=$(curl -s https://ipinfo.io/ip)
export EXTERNAL_ACCESS_HOST=$(curl -s https://ipinfo.io/ip)
{{- end }}
{{- if .Values.externalAccess.autoDiscovery.enabled }}
export EXTERNAL_ACCESS_PORT="$(<${SHARED_FILE})"
@@ -112,7 +112,7 @@ data:
{{- if .Values.advertisedListeners }}
export KAFKA_CFG_ADVERTISED_LISTENERS={{ join "," .Values.advertisedListeners }}
{{- else }}
export KAFKA_CFG_ADVERTISED_LISTENERS="INTERNAL://${MY_POD_NAME}.{{ $fullname }}-headless.{{ $releaseNamespace }}.svc.{{ $clusterDomain }}:{{ $interBrokerPort }},CLIENT://${MY_POD_NAME}.{{ $fullname }}-headless.{{ $releaseNamespace }}.svc.{{ $clusterDomain }}:{{ $clientPort }},EXTERNAL://${EXTERNAL_ACCESS_IP}:${EXTERNAL_ACCESS_PORT}"
export KAFKA_CFG_ADVERTISED_LISTENERS="INTERNAL://${MY_POD_NAME}.{{ $fullname }}-headless.{{ $releaseNamespace }}.svc.{{ $clusterDomain }}:{{ $interBrokerPort }},CLIENT://${MY_POD_NAME}.{{ $fullname }}-headless.{{ $releaseNamespace }}.svc.{{ $clusterDomain }}:{{ $clientPort }},EXTERNAL://${EXTERNAL_ACCESS_HOST}:${EXTERNAL_ACCESS_PORT}"
{{- end }}
{{- end }}

3
bitnami/kafka/templates/statefulset.yaml
View File

@@ -7,8 +7,6 @@
{{- $interBrokerProtocol := include "kafka.listenerType" (dict "protocol" .Values.auth.interBrokerProtocol) -}}
{{- $clientProtocol := include "kafka.listenerType" (dict "protocol" .Values.auth.clientProtocol) -}}
{{- $externalClientProtocol := include "kafka.listenerType" (dict "protocol" (include "kafka.externalClientProtocol" . )) -}}
{{- $loadBalancerIPListLength := len .Values.externalAccess.service.loadBalancerIPs }}
{{- if not (and .Values.externalAccess.enabled (not .Values.externalAccess.autoDiscovery.enabled) (not (eq $replicaCount $loadBalancerIPListLength )) (eq .Values.externalAccess.service.type "LoadBalancer")) }}
apiVersion: {{ include "common.capabilities.statefulset.apiVersion" . }}
kind: StatefulSet
metadata:
@@ -571,4 +569,3 @@ spec:
selector: {{- include "common.tplvalues.render" (dict "value" .Values.logPersistence.selector "context" $) | nindent 10 }}
{{- end -}}
{{- end }}
{{- end }}

7
bitnami/kafka/templates/svc-external-access.yaml
View File

@@ -18,8 +18,11 @@ metadata:
{{- if $root.Values.commonLabels }}
{{- include "common.tplvalues.render" ( dict "value" $root.Values.commonLabels "context" $ ) | nindent 4 }}
{{- end }}
{{- if or $root.Values.externalAccess.service.annotations $root.Values.commonAnnotations }}
{{- if or $root.Values.externalAccess.service.annotations $root.Values.commonAnnotations $root.Values.externalAccess.service.loadBalancerAnnotations }}
annotations:
{{- if and (not (empty $root.Values.externalAccess.service.loadBalancerAnnotations)) (eq (len $root.Values.externalAccess.service.loadBalancerAnnotations) $replicaCount) }}
{{ include "common.tplvalues.render" ( dict "value" (index $root.Values.externalAccess.service.loadBalancerAnnotations $i) "context" $) | nindent 4 }}
{{- end }}
{{- if $root.Values.externalAccess.service.annotations }}
{{ include "common.tplvalues.render" ( dict "value" $root.Values.externalAccess.service.annotations "context" $) | nindent 4 }}
{{- end }}
@@ -30,7 +33,7 @@ metadata:
spec:
type: {{ $root.Values.externalAccess.service.type }}
{{- if eq $root.Values.externalAccess.service.type "LoadBalancer" }}
{{- if not (empty $root.Values.externalAccess.service.loadBalancerIPs) }}
{{- if and (not (empty $root.Values.externalAccess.service.loadBalancerIPs)) (eq (len $root.Values.externalAccess.service.loadBalancerIPs) $replicaCount) }}
loadBalancerIP: {{ index $root.Values.externalAccess.service.loadBalancerIPs $i }}
{{- end }}
{{- if $root.Values.externalAccess.service.loadBalancerSourceRanges }}

14
bitnami/kafka/values.yaml
View File

@@ -727,6 +727,20 @@ externalAccess:
## - Y.Y.Y.Y
##
loadBalancerIPs: []
## @param externalAccess.service.loadBalancerNames Array of load balancer Names for each Kafka broker. Length must be the same as replicaCount
## e.g:
## loadBalancerNames:
## - broker1.external.example.com
## - broker2.external.example.com
##
loadBalancerNames: []
## @param externalAccess.service.loadBalancerAnnotations Array of load balancer annotations for each Kafka broker. Length must be the same as replicaCount
## e.g:
## loadBalancerAnnotations:
## - external-dns.alpha.kubernetes.io/hostname: broker1.external.example.com.
## - external-dns.alpha.kubernetes.io/hostname: broker2.external.example.com.
##
loadBalancerAnnotations: []
## @param externalAccess.service.loadBalancerSourceRanges Address(es) that are allowed when service is LoadBalancer
## ref: https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service
## e.g: