[bitnami/dremio] Make Dremio work with AWS S3 storage (#36365)

* [bitnami/dremio] Make Dremio work with AWS S3 storage

Signed-off-by: Jota Martos <jota.martos@broadcom.com>

* Update CHANGELOG.md

Signed-off-by: Bitnami Bot <bitnami.bot@broadcom.com>

* [bitnami/dremio] Include importMinioCert parameters

Signed-off-by: Jota Martos <jota.martos@broadcom.com>

---------

Signed-off-by: Jota Martos <jota.martos@broadcom.com>
Signed-off-by: Bitnami Bot <bitnami.bot@broadcom.com>
Co-authored-by: Bitnami Bot <bitnami.bot@broadcom.com>
This commit is contained in:
Juan José Martos
2025-10-22 15:01:41 +02:00
committed by GitHub
parent a08e2cdcf1
commit 0705a02b25
10 changed files with 176 additions and 16 deletions

View File

@@ -1,8 +1,12 @@
# Changelog
## 3.0.13 (2025-08-26)
## 3.1.0 (2025-10-22)
* [bitnami/dremio] :zap: :arrow_up: Update dependency references ([#36188](https://github.com/bitnami/charts/pull/36188))
* [bitnami/dremio] Make Dremio work with AWS S3 storage ([#36365](https://github.com/bitnami/charts/pull/36365))
## <small>3.0.13 (2025-08-26)</small>
* [bitnami/dremio] :zap: :arrow_up: Update dependency references (#36188) ([c543356](https://github.com/bitnami/charts/commit/c54335638371cf2c3a259a9ab43f0fc017b085bb)), closes [#36188](https://github.com/bitnami/charts/issues/36188)
## <small>3.0.12 (2025-08-07)</small>

View File

@@ -42,4 +42,4 @@ sources:
- https://github.com/bitnami/charts/tree/main/bitnami/dremio
- https://github.com/bitnami/containers/tree/main/bitnami/dremio
- https://github.com/dremio/dremio-oss
version: 3.0.14
version: 3.1.0

View File

@@ -226,7 +226,7 @@ For configuring AWS as distributed storage, use the `externalS3` section (replac
```yaml
dremio:
distStorageType: minio
distStorageType: aws
minio:
enabled: false
@@ -235,7 +235,19 @@ externalS3:
accessKeyID: DREMIO_ACCESS_KEY_ID
accessKeySecret: DREMIO_ACCESS_KEY_SECRET
bucket: DREMIO_BUCKET
regien: DREMIO_REGION
region: DREMIO_REGION
```
If a role needs to be assumed to access s3, append this configuration to your deployment
```yaml
dremio:
coreSite:
appendConfiguration: |
<property>
<name>fs.s3a.assumed.role.arn</name>
<value>ROLE_TO_ASSUME</value>
</property>
```
#### Azure Storage as distributed storage
@@ -401,7 +413,7 @@ It is also possible to rely on the chart certificate auto-generation capabilitie
### Ingress
This chart provides support for Ingress resources. If you have an ingress controller installed on your cluster, such as [nginx-ingress-controller](https://github.com/bitnami/charts/tree/main/bitnami/nginx-ingress-controller) or [contour](https://github.com/bitnami/charts/tree/main/bitnami/contour) you can utilize it to serve your application.To enable Ingress integration, set `ingress.enabled` to `true`.
This chart provides support for Ingress resources. If you have an ingress controller installed on your cluster, such as [nginx-ingress-controller](https://github.com/bitnami/charts/tree/main/bitnami/nginx-ingress-controller) or [contour](https://github.com/bitnami/charts/tree/main/bitnami/contour) you can utilize it to serve your application. To enable Ingress integration, set `ingress.enabled` to `true`.
The most common scenario is to have one host name mapped to the deployment. In this case, the `ingress.hostname` property can be used to set the host name. The `ingress.tls` parameter can be used to add the TLS configuration for this host.
@@ -409,7 +421,7 @@ However, it is also possible to have more than one host. To facilitate this, the
> NOTE: For each host specified in the `ingress.extraHosts` parameter, it is necessary to set a name, path, and any annotations that the Ingress controller should know about. Not all annotations are supported by all Ingress controllers, but [this annotation reference document](https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md) lists the annotations supported by many popular Ingress controllers.
Adding the TLS parameter (where available) will cause the chart to generate HTTPS URLs, and the application will be available on port 443. The actual TLS secrets do not have to be generated by this chart. However, if TLS is enabled, the Ingress record will not work until the TLS secret exists.
Adding the TLS parameter (where available) will cause the chart to generate HTTPS URLs, and the application will be available on port 443. The actual TLS secrets do not have to be generated by this chart. However, if TLS is enabled, the Ingress record will not work until the TLS secret exists.
[Learn more about Ingress controllers](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/).
@@ -1174,6 +1186,18 @@ There are cases where you may want to deploy extra objects, such a ConfigMap con
| `defaultInitContainers.volumePermissions.containerSecurityContext.enabled` | Enabled init container' Security Context | `true` |
| `defaultInitContainers.volumePermissions.containerSecurityContext.seLinuxOptions` | Set SELinux options in init container | `{}` |
| `defaultInitContainers.volumePermissions.containerSecurityContext.runAsUser` | Set init container's Security Context runAsUser | `0` |
| `defaultInitContainers.importMinIOCert.resourcesPreset` | Set init container resources according to one common preset (allowed values: none, nano, small, medium, large, xlarge, 2xlarge). This is ignored if volumePermissions.resources is set (volumePermissions.resources is recommended for production). | `nano` |
| `defaultInitContainers.importMinIOCert.resources` | Set init container requests and limits for different resources like CPU or memory (essential for production workloads) | `{}` |
| `defaultInitContainers.importMinIOCert.containerSecurityContext.enabled` | Enabled Init container' Security Context | `true` |
| `defaultInitContainers.importMinIOCert.containerSecurityContext.seLinuxOptions` | Set SELinux options in Init container | `{}` |
| `defaultInitContainers.importMinIOCert.containerSecurityContext.runAsUser` | Set runAsUser in Init container' Security Context | `1001` |
| `defaultInitContainers.importMinIOCert.containerSecurityContext.runAsGroup` | Set runAsGroup in Init container' Security Context | `1001` |
| `defaultInitContainers.importMinIOCert.containerSecurityContext.runAsNonRoot` | Set runAsNonRoot in Init container' Security Context | `true` |
| `defaultInitContainers.importMinIOCert.containerSecurityContext.readOnlyRootFilesystem` | Set readOnlyRootFilesystem in Init container' Security Context | `true` |
| `defaultInitContainers.importMinIOCert.containerSecurityContext.privileged` | Set privileged in Init container' Security Context | `false` |
| `defaultInitContainers.importMinIOCert.containerSecurityContext.allowPrivilegeEscalation` | Set allowPrivilegeEscalation in Init container' Security Context | `false` |
| `defaultInitContainers.importMinIOCert.containerSecurityContext.capabilities.drop` | List of capabilities to be dropped in Init container | `["ALL"]` |
| `defaultInitContainers.importMinIOCert.containerSecurityContext.seccompProfile.type` | Set seccomp profile in Init container | `RuntimeDefault` |
### MinIO&reg; chart parameters

View File

@@ -217,7 +217,7 @@ Return true if a TLS credentials secret object should be created
{{- end -}}
{{/*
Return the Jenkins JKS password secret name
Return the Dremio password secret name
*/}}
{{- define "dremio.tls.passwordSecretName" -}}
{{- $secretName := .Values.dremio.tls.passwordSecret -}}
@@ -229,7 +229,7 @@ Return the Jenkins JKS password secret name
{{- end -}}
{{/*
Return the Scylladb TLS credentials secret
Return the Dremio TLS credentials secret
*/}}
{{- define "dremio.tls.secretName" -}}
{{- if .Values.dremio.tls.existingSecret -}}
@@ -276,7 +276,7 @@ Return the Dremio auth credentials secret
{{- define "dremio.dremio-conf.common.default" -}}
paths.local: {{ .Values.masterCoordinator.persistence.mountPath | quote }}
{{- if or (eq .Values.dremio.distStorageType "minio") (eq .Values.dremio.distStorageType "aws") }}
paths.dist: {{ printf "dremioS3://%s%s" (include "dremio.s3.bucket" .) (include "dremio.s3.path" .) | quote }}
paths.dist: {{ printf "dremioS3:///%s%s" (include "dremio.s3.bucket" .) (include "dremio.s3.path" .) | quote }}
{{- end }}
zookeeper: {{ include "dremio.zookeeper.hosts-with-port" . | quote }}
{{- /* Container ports */}}
@@ -342,6 +342,8 @@ Return the S3 backend host
{{- define "dremio.s3.host" -}}
{{- if .Values.minio.enabled -}}
{{- include "dremio.minio.fullname" . -}}
{{- else if and (eq .Values.dremio.distStorageType "aws") (not .Values.externalS3.host) -}}
{{- printf "s3.%s.amazonaws.com" .Values.externalS3.region -}}
{{- else -}}
{{- print .Values.externalS3.host -}}
{{- end -}}

View File

@@ -325,6 +325,44 @@ Init container definition for waiting for the database to be ready
subPath: app-conf-dir
{{- end -}}
{{/*
Returns an init-container that prepares CA for accessing MinIO
*/}}
{{- define "dremio.init-containers.importMinIOCert" -}}
- name: import-minio-cert
image: {{ include "dremio.image" . }}
imagePullPolicy: {{ .Values.dremio.image.pullPolicy }}
{{- if .Values.defaultInitContainers.importMinIOCert.containerSecurityContext.enabled }}
securityContext: {{- include "common.compatibility.renderSecurityContext" (dict "secContext" .Values.defaultInitContainers.importMinIOCert.containerSecurityContext "context" .) | nindent 4 }}
{{- end }}
{{- if .Values.defaultInitContainers.importMinIOCert.resources }}
resources: {{- toYaml .Values.defaultInitContainers.importMinIOCert.resources | nindent 4 }}
{{- else if ne .Values.defaultInitContainers.importMinIOCert.resourcesPreset "none" }}
resources: {{- include "common.resources.preset" (dict "type" .Values.defaultInitContainers.importMinIOCert.resourcesPreset) | nindent 4 }}
{{- end }}
command:
- /bin/bash
args:
- -ec
- |
echo "Importing MinIO public certificate"
# Copy original cacerts
cp /opt/bitnami/java/lib/security/cacerts /bitnami/java/cacerts/
keytool -importcert -file /certs/public.crt -keystore /bitnami/java/cacerts/cacerts -alias "minio" -noprompt
echo "Public certificate imported"
env:
{{- if .Values.extraEnvVars }}
{{- include "common.tplvalues.render" (dict "value" .Values.extraEnvVars "context" $) | nindent 4 }}
{{- end }}
volumeMounts:
- name: empty-dir
mountPath: /bitnami/java/cacerts
subPath: java-cacerts-dir
- name: minio-public-cert
mountPath: /certs
readOnly: true
{{- end -}}
{{/*
Init container definition for waiting for the database to be ready
*/}}
@@ -415,7 +453,7 @@ Init container definition for waiting for the database to be ready
check_s3() {
local -r s3_host="${1:-?missing s3}"
if curl -k --max-time 5 "${s3_host}" | grep "RequestId"; then
if curl -Ik --silent --max-time 5 "${s3_host}" | grep 'Server: \(MinIO\|AmazonS3\)'; then
return 0
else
return 1

View File

@@ -102,6 +102,9 @@ spec:
{{- if and .Values.dremio.tls.enabled .Values.defaultInitContainers.initCerts.enabled }}
{{- include "dremio.init-containers.init-certs" . | nindent 8 }}
{{- end }}
{{- if and (eq .Values.dremio.distStorageType "minio") .Values.minio.enabled .Values.minio.tls.enabled }}
{{- include "dremio.init-containers.importMinIOCert" . | nindent 8 }}
{{- end }}
{{- if .Values.coordinator.initContainers }}
{{- include "common.tplvalues.render" (dict "value" .Values.coordinator.initContainers "context" $) | nindent 8 }}
{{- end }}
@@ -127,7 +130,7 @@ spec:
value: {{ ternary "true" "false" (or .Values.dremio.image.debug .Values.diagnosticMode.enabled) | quote }}
- name: DREMIO_JAVA_SERVER_EXTRA_OPTS
value: |
{{- if and (eq .Values.dremio.distStorageType "minio") (or (and .Values.minio.enabled (((.Values.minio).tls).enabled) (((.Values.minio).tls).autoGenerated)) .Values.externalS3.disableCertChecking) }}
{{- if and (not .Values.minio.enabled) .Values.externalS3.disableCertChecking }}
-Dcom.amazonaws.sdk.disableCertChecking=true
{{- end }}
{{- if .Values.metrics.enabled }}
@@ -235,6 +238,11 @@ spec:
mountPath: /opt/bitnami/dremio/certs
subPath: app-processed-certs-dir
{{- end }}
{{- if and (eq .Values.dremio.distStorageType "minio") .Values.minio.enabled .Values.minio.tls.enabled }}
- name: empty-dir
mountPath: /opt/bitnami/java/lib/security/cacerts
subPath: java-cacerts-dir/cacerts
{{- end }}
{{- if .Values.metrics.enabled }}
- name: metrics-configuration
mountPath: /opt/bitnami/dremio/conf/dremio-telemetry.yaml
@@ -364,6 +372,14 @@ spec:
- name: data
emptyDir: {}
{{- end }}
{{- if and (eq .Values.dremio.distStorageType "minio") .Values.minio.enabled .Values.minio.tls.enabled }}
- name: minio-public-cert
secret:
secretName: {{ printf "%s-crt" ( include "dremio.minio.fullname" . ) }}
items:
- key: {{ ternary "tls.crt" .Values.minio.tls.server.certFilename .Values.minio.tls.autoGenerated.enabled }}
path: public.crt
{{- end }}
{{- if .Values.coordinator.extraVolumes }}
{{- include "common.tplvalues.render" (dict "value" .Values.coordinator.extraVolumes "context" $) | nindent 8 }}
{{- end }}

View File

@@ -114,6 +114,9 @@ spec:
{{- if and $.Values.dremio.tls.enabled $.Values.defaultInitContainers.initCerts.enabled }}
{{- include "dremio.init-containers.init-certs" $ | nindent 8 }}
{{- end }}
{{- if and (eq $.Values.dremio.distStorageType "minio") $.Values.minio.enabled $.Values.minio.tls.enabled }}
{{- include "dremio.init-containers.importMinIOCert" $ | nindent 8 }}
{{- end }}
{{- if $executorValues.initContainers }}
{{- include "common.tplvalues.render" (dict "value" $executorValues.initContainers "context" $) | nindent 8 }}
{{- end }}
@@ -139,7 +142,7 @@ spec:
value: {{ ternary "true" "false" (or $.Values.dremio.image.debug $.Values.diagnosticMode.enabled) | quote }}
- name: DREMIO_JAVA_SERVER_EXTRA_OPTS
value: |
{{- if and (eq $.Values.dremio.distStorageType "minio") (or (and $.Values.minio.enabled ((($.Values.minio).tls).enabled) ((($.Values.minio).tls).autoGenerated)) $.Values.externalS3.disableCertChecking) }}
{{- if and (not $.Values.minio.enabled) $.Values.externalS3.disableCertChecking }}
-Dcom.amazonaws.sdk.disableCertChecking=true
{{- end }}
{{- if $.Values.metrics.enabled }}
@@ -235,6 +238,11 @@ spec:
- name: empty-dir
mountPath: /opt/bitnami/dremio/conf
subPath: app-conf-dir
{{- if and (eq $.Values.dremio.distStorageType "minio") $.Values.minio.enabled $.Values.minio.tls.enabled }}
- name: empty-dir
mountPath: /opt/bitnami/java/lib/security/cacerts
subPath: java-cacerts-dir/cacerts
{{- end }}
{{- if $.Values.metrics.enabled }}
- name: metrics-configuration
mountPath: /opt/bitnami/dremio/conf/dremio-telemetry.yaml
@@ -366,6 +374,14 @@ spec:
path: DREMIO_AWS_SECRET_ACCESS_KEY
{{- end }}
{{- end }}
{{- if and (eq $.Values.dremio.distStorageType "minio") $.Values.minio.enabled $.Values.minio.tls.enabled }}
- name: minio-public-cert
secret:
secretName: {{ printf "%s-crt" ( include "dremio.minio.fullname" $ ) }}
items:
- key: {{ ternary "tls.crt" $.Values.minio.tls.server.certFilename $.Values.minio.tls.autoGenerated.enabled }}
path: public.crt
{{- end }}
{{- if $executorValues.extraVolumes }}
{{- include "common.tplvalues.render" (dict "value" $executorValues.extraVolumes "context" $) | nindent 8 }}
{{- end }}

View File

@@ -104,6 +104,9 @@ spec:
{{- if .Values.defaultInitContainers.upgradeKeystore.enabled }}
{{- include "dremio.init-containers.upgrade-keystore" . | nindent 8 }}
{{- end }}
{{- if and (eq .Values.dremio.distStorageType "minio") .Values.minio.enabled .Values.minio.tls.enabled }}
{{- include "dremio.init-containers.importMinIOCert" . | nindent 8 }}
{{- end }}
{{- if .Values.masterCoordinator.initContainers }}
{{- include "common.tplvalues.render" (dict "value" .Values.masterCoordinator.initContainers "context" $) | nindent 8 }}
{{- end }}
@@ -129,7 +132,7 @@ spec:
value: {{ ternary "true" "false" (or .Values.dremio.image.debug .Values.diagnosticMode.enabled) | quote }}
- name: DREMIO_JAVA_SERVER_EXTRA_OPTS
value: |
{{- if and (eq .Values.dremio.distStorageType "minio") (or (and .Values.minio.enabled (((.Values.minio).tls).enabled) (((.Values.minio).tls).autoGenerated)) .Values.externalS3.disableCertChecking) }}
{{- if and (not .Values.minio.enabled) .Values.externalS3.disableCertChecking }}
-Dcom.amazonaws.sdk.disableCertChecking=true
{{- end }}
{{- if .Values.metrics.enabled }}
@@ -234,6 +237,11 @@ spec:
- name: empty-dir
mountPath: /opt/bitnami/dremio/conf
subPath: app-conf-dir
{{- if and (eq .Values.dremio.distStorageType "minio") .Values.minio.enabled .Values.minio.tls.enabled }}
- name: empty-dir
mountPath: /opt/bitnami/java/lib/security/cacerts
subPath: java-cacerts-dir/cacerts
{{- end }}
{{- if .Values.metrics.enabled }}
- name: metrics-configuration
mountPath: /opt/bitnami/dremio/conf/dremio-telemetry.yaml
@@ -368,6 +376,14 @@ spec:
{{- else }}
emptyDir: {}
{{- end }}
{{- if and (eq .Values.dremio.distStorageType "minio") .Values.minio.enabled .Values.minio.tls.enabled }}
- name: minio-public-cert
secret:
secretName: {{ printf "%s-crt" ( include "dremio.minio.fullname" . ) }}
items:
- key: {{ ternary "tls.crt" .Values.minio.tls.server.certFilename .Values.minio.tls.autoGenerated.enabled }}
path: public.crt
{{- end }}
{{- if .Values.masterCoordinator.extraVolumes }}
{{- include "common.tplvalues.render" (dict "value" .Values.masterCoordinator.extraVolumes "context" $) | nindent 8 }}
{{- end }}

View File

@@ -63,12 +63,10 @@ stringData:
</property>
{{- else }}
{{- /* AWS specific settings */}}
{{- if .Values.externalS3.host }}
<property>
<name>fs.s3a.endpoint</name>
<value>{{ include "dremio.s3.host" . }}</value>
</property>
{{- end }}
<property>
<name>fs.s3a.endpoint.region</name>
<value>{{ include "dremio.s3.region" . }}</value>

View File

@@ -2263,6 +2263,52 @@ defaultInitContainers:
enabled: true
seLinuxOptions: {}
runAsUser: 0
## Init container that imports the MinIO&reg; TLS certificates into the Java keystore
##
importMinIOCert:
## Init container's resource requests and limits
## ref: http://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
## @param defaultInitContainers.importMinIOCert.resourcesPreset Set init container resources according to one common preset (allowed values: none, nano, small, medium, large, xlarge, 2xlarge). This is ignored if volumePermissions.resources is set (volumePermissions.resources is recommended for production).
## More information: https://github.com/bitnami/charts/blob/main/bitnami/common/templates/_resources.tpl#L15
##
resourcesPreset: "nano"
## @param defaultInitContainers.importMinIOCert.resources Set init container requests and limits for different resources like CPU or memory (essential for production workloads)
## Example:
## resources:
## requests:
## cpu: 2
## memory: 512Mi
## limits:
## cpu: 3
## memory: 1024Mi
##
resources: {}
## Init container Container Security Context
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container
## @param defaultInitContainers.importMinIOCert.containerSecurityContext.enabled Enabled Init container' Security Context
## @param defaultInitContainers.importMinIOCert.containerSecurityContext.seLinuxOptions [object,nullable] Set SELinux options in Init container
## @param defaultInitContainers.importMinIOCert.containerSecurityContext.runAsUser Set runAsUser in Init container' Security Context
## @param defaultInitContainers.importMinIOCert.containerSecurityContext.runAsGroup Set runAsGroup in Init container' Security Context
## @param defaultInitContainers.importMinIOCert.containerSecurityContext.runAsNonRoot Set runAsNonRoot in Init container' Security Context
## @param defaultInitContainers.importMinIOCert.containerSecurityContext.readOnlyRootFilesystem Set readOnlyRootFilesystem in Init container' Security Context
## @param defaultInitContainers.importMinIOCert.containerSecurityContext.privileged Set privileged in Init container' Security Context
## @param defaultInitContainers.importMinIOCert.containerSecurityContext.allowPrivilegeEscalation Set allowPrivilegeEscalation in Init container' Security Context
## @param defaultInitContainers.importMinIOCert.containerSecurityContext.capabilities.drop List of capabilities to be dropped in Init container
## @param defaultInitContainers.importMinIOCert.containerSecurityContext.seccompProfile.type Set seccomp profile in Init container
##
containerSecurityContext:
enabled: true
seLinuxOptions: {}
runAsUser: 1001
runAsGroup: 1001
runAsNonRoot: true
readOnlyRootFilesystem: true
privileged: false
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
seccompProfile:
type: "RuntimeDefault"
## @section MinIO&reg; chart parameters
## @extra minio For full list of MinIO&reg; values configurations please refere [here](https://github.com/bitnami/charts/tree/main/bitnami/minio)