Splitting a replica set across multiple data centers

Splitting a replica set over multiple Kubernetes clusters gives you a fault-tolerant system where members live in different data centers.

The Operator cannot place MongoDB Pods in other data centers by itself. You achieve this by deploying an Operator on each site: the Operator on the Main site manages the replica set configuration through cross-site replication. Operators on other sites bootstrap the unmanaged clusters whose members you register on the Main site as externalNodes. You configure hostnames, TLS, and membership manually.

This solution has the following limitations:

• Setup and scaling require manual operations.
• Backups are supported on the Main site only, not on the Replica sites.

Choose your topology

Goal	Topology	Members	Go to
Distribute a full data copy across data centers	1 Main + 2 Replica sites, all data-bearing	3 voting data members	Distribute data-bearing members across sites
You have (or want) only two data-bearing sites and need an odd vote count with a third site that stores no data	Primary-Secondary-Arbiter (PSA) across sites	2 data members + 1 external arbiter	Deploy Primary-Secondary-Arbiter across sites

For either options, you must do these Common steps first:

1. Generate TLS certificates
2. Deploy the main site
3. Export Secrets from the main site

In the following example setups we deploy sites in separate namespaces within a single Kubernetes cluster to emulate isolation and communication boundaries between data centers. In production, you will use separate Kubernetes clusters or physically distinct data centers to achieve true fault isolation, scalability, and resilience.

Common steps

Initial preparation

1. Clone the repository with all manifests and source code:

   git clone -b v1.22.0 https://github.com/percona/percona-server-mongodb-operator
   cd percona-server-mongodb-operator
   

2. Export namespace variables. Use the set that matches your topology:

   Three data-bearing sitesPSA architecture

   export MAIN=main
   export REPLICA1=replica1
   export REPLICA2=replica2
   

   export MAIN=main
   export REPLICA=replica
   export ARBITER=arbiter
   

3. Create the namespaces for your topology:

   Three data-bearing sitesPSA architecture

   kubectl create namespace $MAIN
   kubectl create namespace $REPLICA1
   kubectl create namespace $REPLICA2
   

   kubectl create namespace $MAIN
   kubectl create namespace $REPLICA
   kubectl create namespace $ARBITER
   

4. Deploy the Operator in each namespace. Replace <namespace> with each value you exported:

   kubectl apply --server-side -f deploy/bundle.yaml -n <namespace>
   

Deploy the Main site

TLS certificates generated by the Operator are not enough for multi-site layouts. You need to generate certificates manually on the Main site before you create the database cluster, and include the Main cluster name and the hostname of every replica set member you will connect.

1. Prepare TLS certificates with the Main site cluster name, the hostnames of all members you will add later, and the $MAIN namespace. See TLS certificates manual generation for the steps.

2. Rename deploy/cr.yaml to deploy/cr-main.yaml. Edit it and set:

   • metadata.name to a distinct cluster name (for example, main-cluster). It must match the name you used when generating certificates.
   • spec.secrets.ssl and spec.secrets.sslInternal to the certificates you created
   • spec.unsafeFlags.replsetSize to true to create a replica set with fewer than three instances
   • spec.replsets.size to 1
   • spec.sharding.enabled to false
   • Expose the replica set Pod so other sites can reach it. For example, via MCS svc.clusterset.local, or by using an externally reachable Service type such as LoadBalancer/NodePort and matching hostnames in TLS certificates.

   The sample configuration looks like this:

   cr-main.yaml

   apiVersion: psmdb.percona.com/v1
   kind: PerconaServerMongoDB
   metadata:
     name: main-cluster
     finalizers:
       - percona.com/delete-psmdb-pods-in-order
   spec:
     unsafeFlags:
       replsetSize: true
     updateStrategy: SmartUpdate
     secrets:
       users: main-cluster-secrets
       encryptionKey: main-cluster-mongodb-encryption-key
       ssl: main-cluster-ssl
       sslInternal: main-cluster-ssl-internal
     replsets:
     - name: rs0
       size: 1
       expose:
         enabled: true
         type: ClusterIP
     sharding:
       enabled: false
     # The rest of your configuration
   

3. Deploy the Main site:

   kubectl apply -f deploy/cr-main.yaml -n $MAIN
   

4. Wait until the cluster reports Ready:

   kubectl get psmdb main-cluster -n $MAIN -o yaml | grep state:
   

   Expected output: state: Ready.

Export Secrets from the Main site

All replica set sites must share the same user credentials and TLS certificates. For this reason, export them from the main site and create the corresponding Secrets on the replica sites. Follow the steps in the Export the Secrets section.

Also, update the namespace in each Secret to point to the target namespace. Depending on the topology you chose, the target namespaces are: $REPLICA1/$REPLICA2 for the replica clusters, or $REPLICA/ $ARBITER for the PSA.

Replace the <target-namespace>, <cluster-name>-<secret-name> placeholders as needed in the following command:

```bash
sed -i.bak "s/namespace: $MAIN/namespace: $REPLICA/" "<cluster-name>-<secret-name>.yaml"
```

As a result you should have the full set of Secret files for each unmanaged site you will deploy:

• <cluster-name>-ssl.yaml - SSL certificates for client connections,
• <cluster-name>-ssl-internal.yaml - SSL certificates for replication,
• <cluster-name>-secrets.yaml - user credentials,
• <cluster-name>-mongodb-encryption-key.yaml - encryption key file.

Continue with Distribute data-bearing members across sites or Deploy Primary-Secondary-Arbiter across sites for further configuration.

Distribute data-bearing members across sites

In this site you deploy one Main site and two Replica sites. Each site contains a single data-bearing member. Together they form a three-member replica set with an odd number of voters and a full data copy in every data center.

Site	Role	Members	Name	Namespace
Main (data center 1)	Data-bearing, Operator-managed	1 Pod (size: 1)	main-cluster	main
Replica 1 (data center 2)	Data-bearing, unmanaged	1 Pod (unmanaged: true, size: 1)	replica1-cluster	replica1
Replica 2 (data center 3)	Data-bearing, unmanaged	1 Pod (unmanaged: true, size: 1)	replica2-cluster	replica2

Deploy the Replica sites

Make sure you have completed the Common steps before you proceed.

Repeat the following for each Replica site (replica1-cluster and replica2-cluster). The steps differ only by cluster name and namespace.

The commands below show the configuration of the replica1-cluster cluster.

1. Apply the Secret files you prepared for that site:

   kubectl apply -f replica1-cluster-mongodb-encryption-key.yaml -n $REPLICA1
   kubectl apply -f replica1-cluster-secrets.yaml -n $REPLICA1
   kubectl apply -f replica1-cluster-ssl-internal.yaml -n $REPLICA1
   kubectl apply -f replica1-cluster-ssl.yaml -n $REPLICA1
   

2. Create a Custom Resource manifest for the site. Rename the deploy/cr.yaml to deploy/cr-replica1.yaml. Set the following keys:

   • metadata.name to a distinct name (replica1-cluster / replica2-cluster)
   • spec.secrets to the Secrets you created for that site in users, encryptionKey, ssl, sslInternal
   • unmanaged: true — only the Main site Operator may change the replica set configuration
   • backup.enabled: false - backups are available only on the Main site
   • updateStrategy to RollingUpdate or OnDelete
   • unsafeFlags.replsetSize: true and replsets.size: 1
   • Expose the Pod the same way you exposed the Main site member

   Here’s the example configuration for the replica1-cluster:

   cr-replica1.yaml

   apiVersion: psmdb.percona.com/v1
   kind: PerconaServerMongoDB
   metadata:
     name: replica1-cluster
     finalizers:
       - percona.com/delete-psmdb-pods-in-order
   spec:
     unmanaged: true
     updateStrategy: RollingUpdate
     secrets:
       users: replica1-cluster-secrets
       encryptionKey: replica1-cluster-mongodb-encryption-key
       ssl: replica1-cluster-ssl
       sslInternal: replica1-cluster-ssl-internal
     unsafeFlags:
       replsetSize: true
     replsets:
     - name: rs0
       size: 1
       expose:
         enabled: true
         type: ClusterIP
     sharding:
       enabled: false
     backup:
       enabled: false
     # The rest of your configuration
   

3. Deploy the site:

   kubectl apply -f deploy/cr-replica1.yaml -n $REPLICA1
   

4. Repeat steps 2 and 3 and deploy the replica2-cluster. Replace the cluster name and the namespace.

Add replica sites to the Main site

On the Main site, register both Replica members under replsets.externalNodes as voting members (votes: 1, priority: 1).

Specify endpoints as <pod-name>.<service-name>.<namespace>.svc.cluster.local:27017 (the service name is typically <cluster-name>-<replset-name>). Confirm endpoint names with kubectl get pods and kubectl get svc on each site. If you use MCS or external hostnames, use those reachable names instead and include them in TLS certificates.

cr-main.yaml

replsets:
- name: rs0
  size: 1
  externalNodes:
  - host: replica1-cluster-rs0-0.replica1-cluster-rs0.replica1.svc.cluster.local:27017
    votes: 1
    priority: 1
  - host: replica2-cluster-rs0-0.replica2-cluster-rs0.replica2.svc.cluster.local:27017
    votes: 1
    priority: 1

Apply the updated Main site Custom Resource:

kubectl apply -f deploy/cr-main.yaml -n $MAIN

Deploy Primary-Secondary-Arbiter across sites

Use this layout when you want two data-bearing sites and a third site that only participates in elections. This third site hosts the arbiter node that stores no data and cannot become primary. The arbiter breaks election ties so the replica set keeps an odd number of voters.

This topology uses replsets.externalNodes.arbiterOnly and requires Percona Operator for MongoDB 1.23.0 or later.

Site	Role	Members	Name	Namespace
Main (data center 1)	Data-bearing, Operator-managed	1 Pod (size: 1)	main-cluster	main
Replica (data center 2)	Data-bearing, unmanaged	1 Pod (unmanaged: true, size: 1)	replica-cluster	replica
Arbiter (data center 3)	Arbiter only, unmanaged	1 arbiter Pod	arbiter-cluster	arbiter

Deploy the Replica site (data-bearing)

Make sure you have completed the Common steps before you proceed.

Deploy one unmanaged data-bearing Replica site following the steps from Deploy the replica site. Create the Secrets from the files you created for the replica-cluster.

Use the $REPLICA namespace and the cluster name replica-cluster. Apply the corresponding Secret files, set unmanaged: true, backup.enabled: false, replsets.size: 1, and expose the Pod.

The example configuration is:

cr-replica.yaml

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
  name: replica-cluster
  finalizers:
    - percona.com/delete-psmdb-pods-in-order
spec:
  unmanaged: true
  updateStrategy: RollingUpdate
  secrets:
    users: replica-cluster-secrets
    encryptionKey: replica-cluster-mongodb-encryption-key
    ssl: replica-cluster-ssl
    sslInternal: replica-cluster-ssl-internal
  unsafeFlags:
    replsetSize: true
  replsets:
    - name: rs0
      size: 1
      expose:
        enabled: true
        type: ClusterIP
  sharding:
    enabled: false
  backup:
    enabled: false
  #The rest of your configuration

Deploy the arbiter site

Deploy a third cluster that hosts the arbiter node. You can run this site on Kubernetes (shown here) or as a standalone mongod arbiter outside Kubernetes.

1. Apply the arbiter site Secrets:

   kubectl apply -f arbiter-cluster-mongodb-encryption-key.yaml -n $ARBITER
   kubectl apply -f arbiter-cluster-secrets.yaml -n $ARBITER
   kubectl apply -f arbiter-cluster-ssl-internal.yaml -n $ARBITER
   kubectl apply -f arbiter-cluster-ssl.yaml -n $ARBITER
   

2. Configure the Custom Resource so this site runs an arbiter. Rename the file to deploy/cr-arbiter.yaml.

   Important settings:

   • A distinct cluster name (for example, arbiter-cluster)
   • unmanaged: true
   • updateStrategy: RollingUpdate or OnDelete
   • unsafeFlags.replsetSize: true and replsets.size: 0
   • replsets.arbiter.enabled: true and arbiter.size: 1
   • Pod exposure enabled so other sites can reach the arbiter
   • Secrets created in the previous step

   Example configuration:

   cr-arbiter.yaml

   apiVersion: psmdb.percona.com/v1
   kind: PerconaServerMongoDB
   metadata:
     name: arbiter-cluster
   spec:
     unmanaged: true
     updateStrategy: RollingUpdate
     secrets:
       users: arbiter-cluster-secrets
       encryptionKey: arbiter-cluster-mongodb-encryption-key
       ssl: arbiter-cluster-ssl
       sslInternal: arbiter-cluster-ssl-internal
     unsafeFlags:
       replsetSize: true
     replsets:
     - name: rs0
       size: 0
       expose:
         enabled: true
         type: ClusterIP
       arbiter:
         enabled: true
         size: 1
     sharding:
       enabled: false
   

3. Apply the configuration:

   kubectl apply -f deploy/cr-arbiter.yaml -n $ARBITER
   

   The arbiter Pod is named {cluster}-{replset}-arbiter-0 (for example, arbiter-cluster-rs0-arbiter-0).

Add the Replica member and external arbiter to the Main site

On the Main site, register both remote members under replsets.externalNodes:

• The Replica site data member as a normal voting member
• The arbiter site’s arbiter Pod (not its data Pod) with arbiterOnly: true, votes: 1, and priority: 0.

Specify the member’s endpoints in the format <pod-name>.<service-name>.<namespace>.svc.cluster.local:27017 (the service name is typically <cluster-name>-<replset-name>). To get the endpoint, run kubectl get psmdb on each site and check the output.

cr-main.yaml

replsets:
- name: rs0
  size: 1
  externalNodes:
  - host: replica-cluster-rs0-0.replica-cluster-rs0.replica.svc.cluster.local:27017
    votes: 1
    priority: 1
  - host: arbiter-cluster-rs0-arbiter-0.arbiter-cluster-rs0.arbiter.svc.cluster.local:27017
    votes: 1
    priority: 0
    arbiterOnly: true

If the arbiter runs outside Kubernetes, use its externally reachable hostname instead.

Verify replica set

1. Get the databaseAdmin password from the Main site:

   kubectl get secret main-cluster-secrets -n $MAIN -o jsonpath="{.data.MONGODB_DATABASE_ADMIN_PASSWORD}" | base64 --decode
   

2. Connect to the Main site Pod and list members:

   kubectl -n $MAIN exec -it main-cluster-rs0-0 -- mongosh -u databaseAdmin -p <dbAdminPassword>
   

   rs.status().members
   

The output differs per deployed topology:

Three data-bearing sitesPSA architecture

You should see three data-bearing voting members (Main, Replica 1, Replica 2), all with health: 1 when network and TLS configuration are correct.

You should see three voting members: the Main data Pod, the Replica data Pod, and the external arbiter (ARBITER / arbiterOnly), all with health: 1 when network and TLS configuration are correct.

Using more than one data-bearing member per site

You can scale the same idea beyond single-member sites. For example, deploy two data-bearing members on the Main site and two on the Replica site (size: 2 on each), then add a single external arbiter so the total number of voters stays odd (2 + 2 + 1). Use unsafeFlags.replsetSize: true when a site has an even number of local members without a local arbiter. The interconnect pattern is the same: list remote data members and the shared arbiter under externalNodes, with arbiterOnly: true on the arbiter entry.

---

Last update: July 17, 2026
Created: July 17, 2026