[bitnami/spring-cloud-data-flow] Fixes and improvements (#2945)

* Ignore .DS_Store

* Spring Cloud Data Flow chart fixes and improvements:
1) added ability to configure Hibernate dialect
2) fixed issue with locating application.yaml in kubernetes cluster
3) added JDWP support
4) removed MariaDB from database comments, because external database can be PostgreSQL as well
5) fixed some wrong properties (externalDatabase.server -> externalDatabase.dataflow)

* Added changes to values-production.yaml.

* Bumped spring-cloud-dataflow to 0.1.3 version.

* Added hibernateDialect and jdwp variables to README.md.

* Fixed "missing starting space in comment"

* Fixed "java.sql.SQLInvalidAuthorizationSpecException: Access denied for user 'dataflow'@'10.244.0.5' (using password: YES)".

* Bump version to 0.2.0.

* Added support for external RabbitMQ.

* Added externalRabbitmq config to README.
Fixed typo: rabbitmq.rabbitmq -> rabbitmq.auth.

* Fixed config location, use /opt/bitnami/**/conf.
This commit is contained in:
Alexey Zhokhov
2020-07-06 20:52:47 +08:00
committed by GitHub
parent 73fbb1c676
commit f49b5a1ade
13 changed files with 229 additions and 32 deletions

1
.gitignore vendored
View File

@@ -1,3 +1,4 @@
*.tgz
/.idea/*
.vscode
.DS_Store

View File

@@ -1,6 +1,6 @@
apiVersion: v1
name: spring-cloud-dataflow
version: 0.1.4
version: 0.2.0
appVersion: 2.5.2
description: Spring Cloud Data Flow is a microservices-based toolkit for building streaming and batch data processing pipelines in Cloud Foundry and Kubernetes.
keywords:

View File

@@ -128,6 +128,8 @@ The following tables lists the configurable parameters of the Spring Cloud Data
| `server.autoscaling.maxReplicas` | Maximum number of Dataflow server replicas | `nil` |
| `server.autoscaling.targetCPU` | Target CPU utilization percentage | `nil` |
| `server.autoscaling.targetMemory` | Target Memory utilization percentage | `nil` |
| `server.jdwp.enabled` | Enable Java Debug Wire Protocol (JDWP) | `false` |
| `server.jdwp.port` | JDWP TCP port | `5005` |
### Dataflow Skipper parameters
@@ -178,6 +180,8 @@ The following tables lists the configurable parameters of the Spring Cloud Data
| `skipper.autoscaling.maxReplicas` | Maximum number of Skipper server replicas | `nil` |
| `skipper.autoscaling.targetCPU` | Target CPU utilization percentage | `nil` |
| `skipper.autoscaling.targetMemory` | Target Memory utilization percentage | `nil` |
| `skipper.jdwp.enabled` | Enable Java Debug Wire Protocol (JDWP) | `false` |
| `skipper.jdwp.port` | JDWP TCP port | `5005` |
| `externalSkipper.host` | Host of a external Skipper Server | `localhost` |
| `externalSkipper.port` | External Skipper Server port number | `7577` |
@@ -237,14 +241,20 @@ The following tables lists the configurable parameters of the Spring Cloud Data
| `externalDatabase.dataflow.database` | Name of the existing database to be used by Dataflow server | `dataflow` |
| `externalDatabase.skipper.user` | Existing username in the external db to be used by Skipper server | `skipper` |
| `externalDatabase.skipper.database` | Name of the existing database to be used by Skipper server | `skipper` |
| `externalDatabase.hibernateDialect` | Hibernate Dialect used by Dataflow/Skipper servers | `""` |
### RabbitMQ chart parameters
| Parameter | Description | Default |
|-------------------------------------------------|-------------------------------------------------------------------------|---------------------------------------------------------|
| `rabbitmq.enabled` | Enable/disable RabbitMQ chart installation | `true` |
| `rabbitmq.auth.user` | RabbitMQ username | `user` |
| `rabbitmq.auth.username` | RabbitMQ username | `user` |
| `rabbitmq.auth.password` | RabbitMQ password | _random 40 character alphanumeric string_ |
| `externalRabbitmq.enabled` | Enable/disable external RabbitMQ | `false` |
| `externalRabbitmq.host` | Host of the external RabbitMQ | `localhost` |
| `externalRabbitmq.port` | External RabbitMQ port number | `5672` |
| `externalRabbitmq.username` | External RabbitMQ username | `guest` |
| `externalRabbitmq.password` | External RabbitMQ password | `guest` |
### Kafka chart parameters

View File

@@ -150,7 +150,7 @@ Return true if a configmap object should be created for Spring Cloud Skipper
{{- end -}}
{{/*
Return the MariaDB database Hostname
Return the database Hostname
*/}}
{{- define "scdf.database.host" -}}
{{- if .Values.mariadb.enabled }}
@@ -161,7 +161,7 @@ Return the MariaDB database Hostname
{{- end -}}
{{/*
Return the MariaDB database Port
Return the database Port
*/}}
{{- define "scdf.database.port" -}}
{{- if .Values.mariadb.enabled }}
@@ -172,24 +172,35 @@ Return the MariaDB database Port
{{- end -}}
{{/*
Return the MariaDB database driver
Return the database driver
*/}}
{{- define "scdf.database.driver" -}}
{{- if .Values.mariadb.enabled -}}
{{- printf "org.mariadb.jdbc.Driver" -}}
{{- else -}}
{{- .Values.database.driver -}}
{{- .Values.externalDatabase.driver -}}
{{- end -}}
{{- end -}}
{{/*
Return the MariaDB database scheme
Return the database scheme
*/}}
{{- define "scdf.database.scheme" -}}
{{- if .Values.mariadb.enabled -}}
{{- printf "mariadb" -}}
{{- else -}}
{{- .Values.database.scheme -}}
{{- .Values.externalDatabase.scheme -}}
{{- end -}}
{{- end -}}
{{/*
Return the JDBC URL parameters
*/}}
{{- define "scdf.database.jdbc.parameters" -}}
{{- if .Values.mariadb.enabled -}}
{{- printf "?useMysqlMetadata=true" -}}
{{- else -}}
{{- printf "" -}}
{{- end -}}
{{- end -}}
@@ -200,7 +211,7 @@ Return the Data Flow Database Name
{{- if .Values.mariadb.enabled }}
{{- printf "dataflow" -}}
{{- else -}}
{{- printf "%s" .Values.externalDatabase.server.database -}}
{{- printf "%s" .Values.externalDatabase.dataflow.database -}}
{{- end -}}
{{- end -}}
@@ -211,7 +222,7 @@ Return the Data Flow Database User
{{- if .Values.mariadb.enabled }}
{{- printf "dataflow" -}}
{{- else -}}
{{- printf "%s" .Values.externalDatabase.server.user -}}
{{- printf "%s" .Values.externalDatabase.dataflow.user -}}
{{- end -}}
{{- end -}}
@@ -238,7 +249,7 @@ Return the Skipper Database User
{{- end -}}
{{/*
Return the MariaDB secret name
Return the Database secret name
*/}}
{{- define "scdf.database.secretName" -}}
{{- if .Values.mariadb.enabled }}
@@ -248,6 +259,61 @@ Return the MariaDB secret name
{{- end -}}
{{- end -}}
{{/*
Return the RabbitMQ host
*/}}
{{- define "scdf.rabbitmq.host" -}}
{{- if .Values.rabbitmq.enabled }}
{{- printf "%s" (include "scdf.rabbitmq.fullname" .) -}}
{{- else -}}
{{- printf "%s" .Values.externalRabbitmq.host -}}
{{- end -}}
{{- end -}}
{{/*
Return the RabbitMQ Port
*/}}
{{- define "scdf.rabbitmq.port" -}}
{{- if .Values.rabbitmq.enabled }}
{{- printf "%d" (.Values.rabbitmq.service.port | int ) -}}
{{- else -}}
{{- printf "%d" (.Values.externalRabbitmq.port | int ) -}}
{{- end -}}
{{- end -}}
{{/*
Return the RabbitMQ username
*/}}
{{- define "scdf.rabbitmq.user" -}}
{{- if .Values.rabbitmq.enabled }}
{{- printf "%s" .Values.rabbitmq.auth.username -}}
{{- else -}}
{{- printf "%s" .Values.externalRabbitmq.username -}}
{{- end -}}
{{- end -}}
{{/*
Return the RabbitMQ secret name
*/}}
{{- define "scdf.rabbitmq.secretName" -}}
{{- if .Values.rabbitmq.enabled }}
{{- printf "%s" (include "scdf.rabbitmq.fullname" .) -}}
{{- else -}}
{{- printf "%s-%s" (include "scdf.fullname" .) "externalrabbitmq" -}}
{{- end -}}
{{- end -}}
{{/*
Return the Hibernate dialect
*/}}
{{- define "scdf.database.hibernate.dialect" -}}
{{- if .Values.mariadb.enabled -}}
{{- printf "org.hibernate.dialect.MariaDB102Dialect" -}}
{{- else -}}
{{- .Values.externalDatabase.hibernateDialect -}}
{{- end -}}
{{- end -}}
{{/*
Compile all warnings into a single message, and call fail.
*/}}

View File

@@ -2,7 +2,7 @@
apiVersion: v1
kind: Secret
metadata:
name: {{ printf "%s-%s" (include "scdf.fullname" .) "externaldb" -}}
name: {{ printf "%s-%s" (include "scdf.fullname" .) "externaldb" }}
labels: {{- include "common.labels.standard" . | nindent 4 }}
type: Opaque
data:

View File

@@ -0,0 +1,10 @@
{{- if and (.Values.externalRabbitmq.enabled) (not .Values.rabbitmq.enabled) }}
apiVersion: v1
kind: Secret
metadata:
name: {{ printf "%s-%s" (include "scdf.fullname" .) "externalrabbitmq" }}
labels: {{- include "common.labels.standard" . | nindent 4 }}
type: Opaque
data:
rabbitmq-password: {{ .Values.externalRabbitmq.password | b64enc | quote }}
{{- end }}

View File

@@ -23,7 +23,9 @@ data:
echo "Rollout exit code: '${return_code}'"
return $return_code
}
{{- if .Values.mariadb.enabled }}
k8s_wait_for_statefulset {{ $releaseNamespace }} {{ $mariadbFullname }}
{{- end }}
{{- if or .Values.skipper.enabled .Values.server.configuration.streamingEnabled }}
{{- if .Values.rabbitmq.enabled }}
k8s_wait_for_statefulset {{ $releaseNamespace }} {{ $rabbitmqFullname }}

View File

@@ -56,16 +56,20 @@ data:
port: {{ $rsocketPort }}
{{- end }}
{{- end }}
{{- $hibernateDialect := include "scdf.database.hibernate.dialect" . }}
{{- if $hibernateDialect }}
jpa:
properties:
hibernate:
dialect: org.hibernate.dialect.MariaDB102Dialect
dialect: {{ $hibernateDialect }}
{{- end }}
datasource:
{{- $databaseScheme := include "scdf.database.scheme" . }}
{{- $databaseHost := include "scdf.database.host" . }}
{{- $databasePort := include "scdf.database.port" . }}
{{- $databaseName := include "scdf.database.server.name" . }}
url: 'jdbc:{{ $databaseScheme }}://{{ $databaseHost }}:{{ $databasePort }}/{{ $databaseName }}?useMysqlMetadata=true'
{{- $jdbcParameters := include "scdf.database.jdbc.parameters" . }}
url: 'jdbc:{{ $databaseScheme }}://{{ $databaseHost }}:{{ $databasePort }}/{{ $databaseName }}{{ $jdbcParameters }}'
driverClassName: {{ include "scdf.database.driver" . }}
username: {{ include "scdf.database.server.user" . }}
password: ${mariadb-password}

View File

@@ -84,12 +84,12 @@ spec:
value: "8080"
- name: SPRING_CLOUD_CONFIG_ENABLED
value: "false"
- name: SPRING_CLOUD_KUBERNETES_CONFIG_ENABLE_API
value: "false"
- name: SPRING_CLOUD_KUBERNETES_SECRETS_ENABLE_API
value: "true"
value: "false"
- name: SPRING_CLOUD_KUBERNETES_SECRETS_PATHS
value: "/etc/secrets"
- name: SPRING_CLOUD_KUBERNETES_CONFIG_NAME
value: {{ (include "scdf.server.configmapName" .) | quote }}
- name: SPRING_CLOUD_DATAFLOW_SERVER_URI
{{- if ne $serverServicePort 80 }}
value: {{ printf "http://%s-server.%s.svc.%s:%d" $fullname $releaseNamespace $clusterDomain $serverServicePort | quote }}
@@ -122,6 +122,10 @@ spec:
fieldPath: metadata.namespace
- name: KUBERNETES_TRUST_CERTIFICATES
value: {{ ternary "true" "false" .Values.server.configuration.trustK8sCerts | quote }}
{{- if .Values.server.jdwp.enabled }}
- name: JAVA_TOOL_OPTIONS
value: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address={{ .Values.server.jdwp.port }}"
{{- end }}
{{- range $key, $value := .Values.server.extraEnvVars }}
- name: {{ $key }}
value: "{{ $value }}"
@@ -141,6 +145,11 @@ spec:
- name: http
containerPort: 8080
protocol: TCP
{{- if .Values.server.jdwp.enabled }}
- name: jdwp
containerPort: {{ .Values.server.jdwp.port }}
protocol: TCP
{{- end }}
{{- if .Values.server.livenessProbe.enabled }}
livenessProbe:
httpGet:
@@ -174,6 +183,9 @@ spec:
- name: database
mountPath: /etc/secrets/database
readOnly: true
- name: config
mountPath: /opt/bitnami/spring-cloud-dataflow/conf
readOnly: true
{{- if .Values.server.sidecars }}
{{- include "common.tplvalues.render" ( dict "value" .Values.server.sidecars "context" $) | nindent 8 }}
{{- end }}
@@ -181,10 +193,15 @@ spec:
- name: database
secret:
secretName: {{ include "scdf.database.secretName" . }}
- name: config
configMap:
name: {{ include "scdf.server.configmapName" . }}
items:
- key: application.yaml
path: application.yml
{{- if .Values.waitForBackends.enabled }}
- name: scripts
configMap:
name: {{ include "scdf.fullname" . }}-scripts
defaultMode: 0755
{{- end }}

View File

@@ -15,10 +15,10 @@ data:
kubernetes:
accounts:
{{ .Values.skipper.configuration.accountName }}:
{{- if .Values.rabbitmq.enabled }}
{{- $rabbitmqHost := include "scdf.rabbitmq.fullname" . }}
{{- $rabbitmqPort := int .Values.rabbitmq.service.port }}
{{- $rabbitmqUser := .Values.rabbitmq.auth.username }}
{{- if or (.Values.rabbitmq.enabled) (.Values.externalRabbitmq.enabled) }}
{{- $rabbitmqHost := include "scdf.rabbitmq.host" . }}
{{- $rabbitmqPort := include "scdf.rabbitmq.port" . }}
{{- $rabbitmqUser := include "scdf.rabbitmq.user" . }}
environmentVariables: 'SPRING_RABBITMQ_HOST={{ $rabbitmqHost }},SPRING_RABBITMQ_PORT={{ $rabbitmqPort }},SPRING_RABBITMQ_USERNAME={{ $rabbitmqUser }},SPRING_RABBITMQ_PASSWORD=${rabbitmq-password}'
{{- else if .Values.kafka.enabled }}
environmentVariables: 'SPRING_CLOUD_STREAM_KAFKA_BINDER_BROKERS=${{ printf "{" }}{{ template "scdf.envrelease" . }}_KAFKA_SERVICE_HOST}:${{ printf "{" }}{{ template "scdf.envrelease" . }}_KAFKA_SERVICE_PORT},SPRING_CLOUD_STREAM_KAFKA_BINDER_ZK_NODES=${{ printf "{" }}{{ template "scdf.envrelease" . }}_ZOOKEEPER_SERVICE_HOST}:${{ printf "{" }}{{ template "scdf.envrelease" . }}_ZOOKEEPER_SERVICE_PORT}'
@@ -31,16 +31,20 @@ data:
{{- end }}
readinessProbeDelay: {{ .Values.deployer.readinessProbe.initialDelaySeconds }}
livenessProbeDelay: {{ .Values.deployer.livenessProbe.initialDelaySeconds }}
{{- $hibernateDialect := include "scdf.database.hibernate.dialect" . }}
{{- if $hibernateDialect }}
jpa:
properties:
hibernate:
dialect: org.hibernate.dialect.MariaDB102Dialect
dialect: {{ $hibernateDialect }}
{{- end }}
datasource:
{{- $databaseScheme := include "scdf.database.scheme" . }}
{{- $databaseHost := include "scdf.database.host" . }}
{{- $databasePort := include "scdf.database.port" . }}
{{- $databaseName := include "scdf.database.skipper.name" . }}
url: 'jdbc:{{ $databaseScheme }}://{{ $databaseHost }}:{{ $databasePort }}/{{ $databaseName }}?useMysqlMetadata=true'
{{- $jdbcParameters := include "scdf.database.jdbc.parameters" . }}
url: 'jdbc:{{ $databaseScheme }}://{{ $databaseHost }}:{{ $databasePort }}/{{ $databaseName }}{{ $jdbcParameters }}'
driverClassName: {{ include "scdf.database.driver" . }}
username: {{ include "scdf.database.skipper.user" . }}
password: ${mariadb-password}

View File

@@ -80,18 +80,22 @@ spec:
value: "7577"
- name: SPRING_CLOUD_CONFIG_ENABLED
value: "false"
- name: SPRING_CLOUD_KUBERNETES_CONFIG_ENABLE_API
value: "false"
- name: SPRING_CLOUD_KUBERNETES_SECRETS_ENABLE_API
value: "true"
value: "false"
- name: SPRING_CLOUD_KUBERNETES_SECRETS_PATHS
value: "/etc/secrets"
- name: SPRING_CLOUD_KUBERNETES_CONFIG_NAME
value: {{ (include "scdf.skipper.configmapName" .) | quote }}
- name: KUBERNETES_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: KUBERNETES_TRUST_CERTIFICATES
value: {{ ternary "true" "false" .Values.skipper.configuration.trustK8sCerts | quote }}
{{- if .Values.skipper.jdwp.enabled }}
- name: JAVA_TOOL_OPTIONS
value: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address={{ .Values.skipper.jdwp.port }}"
{{- end }}
{{- range $key, $value := .Values.skipper.extraEnvVars }}
- name: {{ $key }}
value: "{{ $value }}"
@@ -111,6 +115,11 @@ spec:
- name: http
containerPort: 7577
protocol: TCP
{{- if .Values.skipper.jdwp.enabled }}
- name: jdwp
containerPort: {{ .Values.skipper.jdwp.port }}
protocol: TCP
{{- end }}
{{- if .Values.skipper.livenessProbe.enabled }}
livenessProbe:
httpGet:
@@ -144,7 +153,10 @@ spec:
- name: database
mountPath: /etc/secrets/database
readOnly: true
{{- if .Values.rabbitmq.enabled }}
- name: config
mountPath: /opt/bitnami/spring-cloud-skipper/conf
readOnly: true
{{- if or (.Values.rabbitmq.enabled) (.Values.externalRabbitmq.enabled) }}
- name: rabbitmq
mountPath: /etc/secrets/rabbitmq
readOnly: true
@@ -156,10 +168,16 @@ spec:
- name: database
secret:
secretName: {{ include "scdf.database.secretName" . }}
{{- if .Values.rabbitmq.enabled }}
- name: config
configMap:
name: {{ include "scdf.skipper.configmapName" . }}
items:
- key: application.yaml
path: application.yml
{{- if or (.Values.rabbitmq.enabled) (.Values.externalRabbitmq.enabled) }}
- name: rabbitmq
secret:
secretName: {{ include "scdf.rabbitmq.fullname" . }}
secretName: {{ include "scdf.rabbitmq.secretName" . }}
{{- end }}
{{- if .Values.waitForBackends.enabled }}
- name: scripts

View File

@@ -311,6 +311,16 @@ server:
# targetCPU: 50
# targetMemory: 50
## Java Debug Wire Protocol (JDWP) parameters.
##
jdwp:
## Set to true to enable Java debugger.
##
enabled: false
## Specify port for remote debugging.
##
port: 5005
##
## Spring Cloud Skipper parameters.
##
@@ -536,6 +546,16 @@ skipper:
# targetCPU: 50
# targetMemory: 50
## Java Debug Wire Protocol (JDWP) parameters.
##
jdwp:
## Set to true to enable Java debugger.
##
enabled: false
## Specify port for remote debugging.
##
port: 5005
##
## External Skipper Configuration
##
@@ -752,6 +772,10 @@ externalDatabase:
skipper:
database: skipper
user: skipper
## Hibernate Dialect
##
# hibernateDialect: org.hibernate.dialect.MariaDB102Dialect
##
## RabbitMQ chart configuration
##

View File

@@ -311,6 +311,16 @@ server:
# targetCPU: 50
# targetMemory: 50
## Java Debug Wire Protocol (JDWP) parameters.
##
jdwp:
## Set to true to enable Java debugger.
##
enabled: false
## Specify port for remote debugging.
##
port: 5005
##
## Spring Cloud Skipper parameters.
##
@@ -536,6 +546,16 @@ skipper:
# targetCPU: 50
# targetMemory: 50
## Java Debug Wire Protocol (JDWP) parameters.
##
jdwp:
## Set to true to enable Java debugger.
##
enabled: false
## Specify port for remote debugging.
##
port: 5005
##
## External Skipper Configuration
##
@@ -735,6 +755,10 @@ externalDatabase:
skipper:
database: skipper
user: skipper
## Hibernate Dialect
##
# hibernateDialect: org.hibernate.dialect.MariaDB102Dialect
##
## RabbitMQ chart configuration
##
@@ -742,9 +766,26 @@ externalDatabase:
##
rabbitmq:
enabled: true
rabbitmq:
auth:
username: user
password: ""
##
## External RabbitMQ Configuration
##
## All of these values are ignored when rabbitmq.enabled is set to true
##
externalRabbitmq:
## Enables or disables external RabbitMQ, can be disabled when Kafka is using
##
enabled: false
## RabbitMQ host and port
##
host: localhost
port: 5672
## RabbitMQ username and password, password will be saved in a kubernetes secret
##
username: guest
password: guest
##
## Kafka chart configuration