Bitnami/charts
2
0
Fork 0
mirror of https://github.com/bitnami/charts.git synced 2026-03-16 14:57:08 +08:00
Code Issues Packages Projects Releases Wiki Activity

RabbitMQ: explain how to safely avoid a deployment deadlock (#25931)

Browse Source 
Explain the problem and a widely used solution
instead of recommending force booting nodes.

These practices are generally used by existing
RabbitMQ K8S Cluster Operators but this PR
intentionally does not recommend any of them.





Commit a suggested edit to bitnami/rabbitmq/README.md

Signed-off-by: Michael Klishin <michaelklishin@icloud.com>
Co-authored-by: Carlos Rodríguez Hernández <carlosrh@vmware.com>
This commit is contained in:
Michael Klishin
2024-06-07 04:38:20 -04:00
committed by GitHub
parent f2ed42df98
commit 3117b57314
1 changed files with 16 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files

44
bitnami/rabbitmq/README.md
View File

@@ -263,41 +263,29 @@ extraConfiguration: |
log.console.formatter = json
```
### Recover the cluster from complete shutdown
### How to Avoid Deadlocked Deployments After a Cluster-Wide Restart
> IMPORTANT: Some of these procedures can lead to data loss. Always make a backup beforehand.
RabbitMQ nodes assume their peers come back online within five minutes (by default). When the `OrderedReady` pod management policy is used with a readiness probe that implicitly requires a fully booted node, the deployment can deadlock:
The RabbitMQ cluster is able to support multiple node failures but, in a situation in which all the nodes are brought down at the same time, the cluster might not be able to self-recover.
- Kubernetes will expect the first node to pass a readiness probe
- The readiness probe may require a fully booted node
- The node will fully boot after it detects that its peers have come online
- Kubernetes will not start any more pods until the first one boots
This happens if the pod management policy of the statefulset is not `Parallel` and the last pod to be running wasn't the first pod of the statefulset. If that happens, update the pod management policy to recover a healthy state:
The following combination of deployment settings avoids the problem:
```console
$ kubectl delete statefulset STATEFULSET_NAME --cascade=false
helm upgrade RELEASE_NAME oci://REGISTRY_NAME/REPOSITORY_NAME/rabbitmq \
--set podManagementPolicy=Parallel \
--set replicaCount=NUMBER_OF_REPLICAS \
--set auth.password=PASSWORD \
--set auth.erlangCookie=ERLANG_COOKIE
```
- Use `podManagementPolicy: "Parallel"` to boot multiple cluster nodes in parallel
- Use `rabbitmq-diagnostics ping` for readiness probe
> Note: You need to substitute the placeholders `REGISTRY_NAME` and `REPOSITORY_NAME` with a reference to your Helm chart registry and repository. For example, in the case of Bitnami, you need to use `REGISTRY_NAME=registry-1.docker.io` and `REPOSITORY_NAME=bitnamicharts`.
To learn more, please consult RabbitMQ documentation guides:
For a faster resyncronization of the nodes, you can temporarily disable the readiness probe by setting `readinessProbe.enabled=false`. Bear in mind that the pods will be exposed before they are actually ready to process requests.
- [RabbitMQ Clustering guide: Node Restarts](https://www.rabbitmq.com/docs/clustering#restarting)
- [RabbitMQ Clustering guide: Restarts and Readiness Probes](https://www.rabbitmq.com/docs/clustering#restarting-readiness-probes)
- [Recommendations](https://www.rabbitmq.com/docs/cluster-formation#peer-discovery-k8s) for Operator-less (DIY) deployments to Kubernetes
If the steps above don't bring the cluster to a healthy state, it could be possible that none of the RabbitMQ nodes think they were the last node to be up during the shutdown. In those cases, you can force the boot of the nodes by specifying the `clustering.forceBoot=true` parameter (which will execute [`rabbitmqctl force_boot`](https://www.rabbitmq.com/rabbitmqctl.8.html#force_boot) in each pod):
#### Do Not Force Boot Nodes on a Regular Basis
```console
helm upgrade RELEASE_NAME oci://REGISTRY_NAME/REPOSITORY_NAME/rabbitmq \
--set podManagementPolicy=Parallel \
--set clustering.forceBoot=true \
--set replicaCount=NUMBER_OF_REPLICAS \
--set auth.password=PASSWORD \
--set auth.erlangCookie=ERLANG_COOKIE
```
> Note: You need to substitute the placeholders `REGISTRY_NAME` and `REPOSITORY_NAME` with a reference to your Helm chart registry and repository. For example, in the case of Bitnami, you need to use `REGISTRY_NAME=registry-1.docker.io` and `REPOSITORY_NAME=bitnamicharts`.
More information: [Clustering Guide: Restarting](https://www.rabbitmq.com/clustering.html#restarting).
Note that forcing nodes to boot is **not a solution** and doing so **can be dangerous**. Forced booting is a last resort mechanism in RabbitMQ that helps make remaining cluster nodes recover and rejoin each other after a permanent loss of some of their former peers. In other words, forced booting a node is an emergency event recovery procedure.
### Known issues
@@ -878,4 +866,4 @@ Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
limitations under the License.