Reindex Jira Data Center without downtime on Kubernetes

There are two ways to reindex Jira Data Center without downtime on Kubernetes. My Jira cluster runs a statefulSet in kubernetes. The configuration details:

DON’T use the following methods if you have livenessProbe to monitor the /status. As your pod may get killed when it is doing the re-indexing. You should monitor / for livenessProbe instead. Why? Check this out: Configured livenessProbe and readinessProbe for Jira pod in Kubernetes.

Method one: Temporarily remove one pod from the Jira service, so there will be no traffics directed to it. It is safer, but requires a bit more steps.

Kubernete service use label to select the backend pod. So the idea is to temporarily change the label of one pod.

# Check all pods that servers Jira service traffics
$ kubectl get pods -l app=jira
NAME     READY   STATUS    RESTARTS   AGE
jira-0   1/1     Running   0          155m
jira-1   1/1     Running   0          152m
jira-2   1/1     Running   0          150m

# Remove pod jira-2 by changing the label
$ kubectl label --overwrite pods jira-2 app=jira-index
pod/jira-2 labeled

# Confirm jira-2 has been removed
$ kubectl get pods -l app=jira
NAME     READY   STATUS    RESTARTS   AGE
jira-0   1/1     Running   0          159m
jira-1   1/1     Running   0          157m

# Forward remote pod port 8080 to local 9999
$ kubectl port-forward jira-2 9999:8080

# Go to http://localhost:9999, then run the Jira forground index. Once it is completed, add the pod back, so it can replicate the latest index file to other nodes.
$ kubectl label --overwrite pods jira-2 app=jira

Method two: Utilize the readinessProbe.

When Jira is doing the foreground reindexing, the /status return 503. So if you have the readnessProbe configured, then it will detect that the pod (the one that is doing reindexing) is not ready so no traffics will be directed to it. Well, user may hit this pod during the check gap. It depends on how often the readiness probe runs.

        readinessProbe:
          httpGet:
            path: /status
            port: 8080
          initialDelaySeconds: 120
          periodSeconds: 10

Reference: https://confluence.atlassian.com/jirakb/reindex-jira-data-center-without-downtime-757467576.html