Skip to content

[BUG] mongodb dump restore-post-ready failed #10775

Closed
apecloud/kubeblocks-addons
#3428
Closed
[BUG] mongodb dump restore-post-ready failed#10775
apecloud/kubeblocks-addons
#3428
Assignees
gnolong
Labels
kind/bugSomething isn't working
Milestone
Release 1.1.0

Description

@JashBook
JashBook
opened on Aug 13, 2026

Describe the bug
A clear and concise description of what the bug is.

kbcli version 
heKubernetes: v1.30.4-vke.4
KubeBlocks: 1.1.0-beta.9
kbcli: 1.0.2
WARNING: version difference between kbcli (1.0.2) and kubeblocks (1.1.0-beta.9) 
➜  ~ helm get notes -n kb-system kb-addon-mongodb
NOTES:
--------------------------------------------------------------------------------
ENTERPRISE SOFTWARE — LICENSE NOTICE

This Helm chart is part of KubeBlocks Enterprise Addons, proprietary software
owned by ApeCloud Co., Ltd.

A valid KubeBlocks Enterprise License Agreement is required to install and use
this software. Unauthorized use, copying, or distribution is prohibited.

  License terms : see the LICENSE file included with this chart
  Licensing info : support@apecloud.com
  Website        : https://www.apecloud.com
--------------------------------------------------------------------------------


Release Information:
  Commit ID: "a64edbdf67b27556a745e5945a9cbf690da53ceb"
  Commit Time: "2026-07-29 10:28:46 +0800"
  Release Branch: "v1.1.0-beta.9"
  Release Time:  "2026-08-07 11:50:11 +0800"
  Enterprise: "true"

To Reproduce
Steps to reproduce the behavior:

  1. create cluster
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: mongodbs-ucdaah
  namespace: default
spec:
  clusterDef: mongodb
  topology: sharding
  terminationPolicy: Delete
  shardings:
    - name: shard
      shards: 3
      template:
        name: mongodb
        serviceVersion: 8.0.17
        replicas: 3
        disableExporter: false
        env:
          - name: SHARDING_NAME
            value: shard
        resources:
          requests:
            cpu: 200m
            memory: 0.5Gi
          limits:
            cpu: 200m
            memory: 0.5Gi
        persistentVolumeClaimRetentionPolicy:
          whenDeleted: Delete
        volumeClaimTemplates:
          - name: data
            spec:
              storageClassName: 
              accessModes:
                - ReadWriteOnce
              resources:
                requests:
                  storage: 20Gi
  componentSpecs:
    - name: config-server
      serviceVersion: 8.0.17
      replicas: 3
      disableExporter: false
      systemAccounts:
      - name: root
        passwordConfig:
          length: 16
          numDigits: 8
          numSymbols: 0
          letterCase: MixedCases
          seed: mongodbs-ucdaah
      env:
        - name: MONGODB_BALANCER_ENABLED
          value: "false"
      resources:
        limits:
          cpu: 200m
          memory: 0.5Gi
        requests:
          cpu: 200m
          memory: 0.5Gi
      persistentVolumeClaimRetentionPolicy:
        whenDeleted: Delete
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: 
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
    - name: mongos
      serviceVersion: 8.0.17
      replicas: 3
      disableExporter: false
      env:
        - name: MONGODB_BALANCER_ENABLED
          value: "false"
      resources:
        limits:
          cpu: 200m
          memory: 0.5Gi
        requests:
          cpu: 200m
          memory: 0.5Gi

      `kbcli cluster list mongodbs-ucdaah --show-labels  --namespace default `
    
NAME              NAMESPACE   CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS    CREATED-TIME                 LABELS                                         
mongodbs-ucdaah   default     mongodb              WipeOut              Running   Aug 13,2026 15:40 UTC+0800   

      `kbcli cluster list-instances mongodbs-ucdaah --namespace default `
    
NAME                              NAMESPACE   CLUSTER           COMPONENT          STATUS    ROLE        ACCESSMODE   AZ               CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE     NODE                          CREATED-TIME                 
mongodbs-ucdaah-config-server-0   default     mongodbs-ucdaah   config-server      Running   primary                  cn-guangzhou-b   200m / 200m          512Mi / 512Mi           data:20Gi   192.168.0.225/192.168.0.225   Aug 13,2026 15:40 UTC+0800   
mongodbs-ucdaah-config-server-1   default     mongodbs-ucdaah   config-server      Running   secondary                cn-guangzhou-b   200m / 200m          512Mi / 512Mi           data:20Gi   192.168.0.9/192.168.0.9       Aug 13,2026 15:40 UTC+0800   
mongodbs-ucdaah-config-server-2   default     mongodbs-ucdaah   config-server      Running   secondary                cn-guangzhou-b   200m / 200m          512Mi / 512Mi           data:20Gi   192.168.0.117/192.168.0.117   Aug 13,2026 15:40 UTC+0800   
mongodbs-ucdaah-mongos-0          default     mongodbs-ucdaah   mongos             Running   <none>                   cn-guangzhou-b   200m / 200m          512Mi / 512Mi           <none>      192.168.0.225/192.168.0.225   Aug 13,2026 15:40 UTC+0800   
mongodbs-ucdaah-mongos-1          default     mongodbs-ucdaah   mongos             Running   <none>                   cn-guangzhou-b   200m / 200m          512Mi / 512Mi           <none>      192.168.0.99/192.168.0.99     Aug 13,2026 15:40 UTC+0800   
mongodbs-ucdaah-mongos-2          default     mongodbs-ucdaah   mongos             Running   <none>                   cn-guangzhou-b   200m / 200m          512Mi / 512Mi           <none>      192.168.0.117/192.168.0.117   Aug 13,2026 15:40 UTC+0800   
mongodbs-ucdaah-shard-fvd-0       default     mongodbs-ucdaah   shard(shard-fvd)   Running   primary                  cn-guangzhou-b   200m / 200m          512Mi / 512Mi           data:20Gi   192.168.0.99/192.168.0.99     Aug 13,2026 15:40 UTC+0800   
mongodbs-ucdaah-shard-fvd-1       default     mongodbs-ucdaah   shard(shard-fvd)   Running   secondary                cn-guangzhou-b   200m / 200m          512Mi / 512Mi           data:20Gi   192.168.0.117/192.168.0.117   Aug 13,2026 15:40 UTC+0800   
mongodbs-ucdaah-shard-fvd-2       default     mongodbs-ucdaah   shard(shard-fvd)   Running   secondary                cn-guangzhou-b   200m / 200m          512Mi / 512Mi           data:20Gi   192.168.0.9/192.168.0.9       Aug 13,2026 15:40 UTC+0800   
mongodbs-ucdaah-shard-rqn-0       default     mongodbs-ucdaah   shard(shard-rqn)   Running   primary                  cn-guangzhou-b   200m / 200m          512Mi / 512Mi           data:20Gi   192.168.0.117/192.168.0.117   Aug 13,2026 15:40 UTC+0800   
mongodbs-ucdaah-shard-rqn-1       default     mongodbs-ucdaah   shard(shard-rqn)   Running   secondary                cn-guangzhou-b   200m / 200m          512Mi / 512Mi           data:20Gi   192.168.0.99/192.168.0.99     Aug 13,2026 15:40 UTC+0800   
mongodbs-ucdaah-shard-rqn-2       default     mongodbs-ucdaah   shard(shard-rqn)   Running   secondary                cn-guangzhou-b   200m / 200m          512Mi / 512Mi           data:20Gi   192.168.0.225/192.168.0.225   Aug 13,2026 15:40 UTC+0800   
mongodbs-ucdaah-shard-scs-0       default     mongodbs-ucdaah   shard(shard-scs)   Running   primary                  cn-guangzhou-b   200m / 200m          512Mi / 512Mi           data:20Gi   192.168.0.9/192.168.0.9       Aug 13,2026 15:40 UTC+0800   
mongodbs-ucdaah-shard-scs-1       default     mongodbs-ucdaah   shard(shard-scs)   Running   secondary                cn-guangzhou-b   200m / 200m          512Mi / 512Mi           data:20Gi   192.168.0.225/192.168.0.225   Aug 13,2026 15:40 UTC+0800   
mongodbs-ucdaah-shard-scs-2       default     mongodbs-ucdaah   shard(shard-scs)   Running   secondary                cn-guangzhou-b   200m / 200m          512Mi / 512Mi           data:20Gi   192.168.0.99/192.168.0.99     Aug 13,2026 15:40 UTC+0800
  1. dump backup
kbcli cluster backup mongodbs-ucdaah --method dump --namespace default

kbcli cluster list-backups mongodbs-ucdaah --namespace default
NAME                                            NAMESPACE   SOURCE-CLUSTER    METHOD   STATUS      TOTAL-SIZE   DURATION   DELETION-POLICY   CREATE-TIME                  COMPLETION-TIME              EXPIRATION   
backup-default-mongodbs-ucdaah-20260813160145   default     mongodbs-ucdaah   dump     Completed   147109       41s        Delete            Aug 13,2026 16:01 UTC+0800   Aug 13,2026 16:02 UTC+0800

describe backup

kubectl describe backup backup-default-mongodbs-ucdaah-20260813160145 
Name:         backup-default-mongodbs-ucdaah-20260813160145
Namespace:    default
Labels:       app.kubernetes.io/instance=mongodbs-ucdaah
              app.kubernetes.io/managed-by=kubeblocks-dataprotection
              apps.kubeblocks.io/component-name=config-server
              dataprotection.kubeblocks.io/backup-policy=mongodbs-ucdaah-config-server-backup-policy
              dataprotection.kubeblocks.io/backup-repo-name=backuprepo-kbcli-test
              dataprotection.kubeblocks.io/backup-type=Full
              dataprotection.kubeblocks.io/cluster-uid=e6fed72a-c6eb-4cdf-8f77-574bb2238780
              operations.kubeblocks.io/ops-name=backup-default-mongodbs-ucdaah-20260813160145
              operations.kubeblocks.io/ops-type=Backup
Annotations:  kubeblocks.io/cluster-snapshot:
                {"metadata":{"name":"mongodbs-ucdaah","namespace":"default","creationTimestamp":null,"annotations":{"kubeblocks.io/crd-api-version":"apps....
              kubeblocks.io/encrypted-system-accounts: {"config-server":{"root":"N2rOv7HwoRDzs63dWA4vmMdWS8bRiC4vIbJ8thn5t2t3WcdXS9dTCAN0o3I="}}
API Version:  dataprotection.kubeblocks.io/v1alpha1
Kind:         Backup
Metadata:
  Creation Timestamp:  2026-08-13T08:01:46Z
  Finalizers:
    dataprotection.kubeblocks.io/finalizer
  Generation:        1
  Resource Version:  267067665
  UID:               c7e021a4-eafe-4b35-be3d-15d38ac2f55b
Spec:
  Backup Method:       dump
  Backup Policy Name:  mongodbs-ucdaah-config-server-backup-policy
  Deletion Policy:     Delete
Status:
  Actions:
    Action Type:           Job
    Completion Timestamp:  2026-08-13T08:02:27Z
    Name:                  dp-prebackup-0-0
    Object Ref:
      API Version:         batch/v1
      Kind:                Job
      Name:                dp-prebackup-0-0-backup-default-mongodbs-ucdaah-20260813160145
      Namespace:           default
      Resource Version:    267067088
      UID:                 dc11501b-a9b9-40a6-bd79-0908db54a978
    Phase:                 Completed
    Start Timestamp:       2026-08-13T08:01:46Z
    Target Pod Name:       mongodbs-ucdaah-config-server-1
    Action Type:           Job
    Completion Timestamp:  2026-08-13T08:02:27Z
    Name:                  dp-backup-0
    Object Ref:
      API Version:       batch/v1
      Kind:              Job
      Name:              dp-backup-0-backup-default-mongodbs-ucdaah-20260813160145-c7e02
      Namespace:         default
      Resource Version:  267067661
      UID:               96bbfb83-a16d-4c28-b252-0b23ed23f99f
    Phase:               Completed
    Start Timestamp:     2026-08-13T08:01:52Z
    Target Pod Name:     mongodbs-ucdaah-config-server-1
  Backup Method:
    Action Set Name:  mongodb-shard-pbm-logical
    Env:
      Name:            PBM_IMAGE_TAG
      Value:           2.12.0
      Name:            PSM_IMAGE_TAG
      Value:           8.0.17
    Name:              dump
    Snapshot Volumes:  false
    Target Volumes:
      Volume Mounts:
        Mount Path:      /data/mongodb
        Name:            data
  Backup Repo Name:      backuprepo-kbcli-test
  Completion Timestamp:  2026-08-13T08:02:27Z
  Duration:              41s
  Extras:
    backup_name:      2026-08-13T08:01:54Z
    backup_type:      logical
    last_write_time:  2026-08-13T08:01:58Z
  Format Version:     0.1.0
  Path:               /default/mongodbs-ucdaah-e6fed72a-c6eb-4cdf-8f77-574bb2238780/config-server/backup-default-mongodbs-ucdaah-20260813160145
  Phase:              Completed
  Start Timestamp:    2026-08-13T08:01:46Z
  Target:
    Connection Credential:
      Password Key:  password
      Secret Name:   mongodbs-ucdaah-config-server-account-root
      Username Key:  username
    Pod Selector:
      Fallback Label Selector:
        Match Labels:
          app.kubernetes.io/instance:         mongodbs-ucdaah
          app.kubernetes.io/managed-by:       kubeblocks
          apps.kubeblocks.io/component-name:  config-server
          kubeblocks.io/role:                 primary
      Match Labels:
        app.kubernetes.io/instance:         mongodbs-ucdaah
        app.kubernetes.io/managed-by:       kubeblocks
        apps.kubeblocks.io/component-name:  config-server
        kubeblocks.io/role:                 secondary
      Strategy:                             Any
    Selected Target Pods:
      mongodbs-ucdaah-config-server-1
  Time Range:
    End:       2026-08-13T08:02:00Z
    Start:     2026-08-13T08:01:54Z
  Total Size:  147109
Events:
  Type    Reason         Age    From               Message
  ----    ------         ----   ----               -------
  Normal  CreatingJob    9m4s   backup-controller  creating job default/dp-prebackup-0-0-backup-default-mongodbs-ucdaah-20260813160145
  Normal  CreatingJob    8m58s  backup-controller  creating job default/dp-backup-0-backup-default-mongodbs-ucdaah-20260813160145-c7e02
  Normal  CreatedBackup  8m23s  backup-controller  Completed backup
  1. restore
kbcli cluster restore mongodbs-ucdaah-wihpk --backup backup-default-mongodbs-ucdaah-20260813160145
  1. See error
kubectl get restore -l app.kubernetes.io/instance=mongodbs-hovpkc-htenq
NAME                                                       BACKUP                                          RESTORE-TIME   STATUS      DURATION   CREATION-TIME          COMPLETION-TIME
mongodbs-hovpkc-htenq-config-server-cf323037-postready     backup-default-mongodbs-hovpkc-20260813160428                  Failed      3m15s      2026-08-13T08:06:12Z   2026-08-13T08:09:27Z
mongodbs-hovpkc-htenq-config-server-cf323037-preparedata   backup-default-mongodbs-hovpkc-20260813160428                  Completed   0s         2026-08-13T08:05:19Z   2026-08-13T08:05:19Z

kubectl describe restore mongodbs-hovpkc-htenq-config-server-cf323037-postready
Name:         mongodbs-hovpkc-htenq-config-server-cf323037-postready
Namespace:    default
Labels:       app.kubernetes.io/instance=mongodbs-hovpkc-htenq
              app.kubernetes.io/managed-by=kubeblocks
              apps.kubeblocks.io/component-name=config-server
Annotations:  <none>
API Version:  dataprotection.kubeblocks.io/v1alpha1
Kind:         Restore
Metadata:
  Creation Timestamp:  2026-08-13T08:06:12Z
  Finalizers:
    dataprotection.kubeblocks.io/finalizer
  Generation:  1
  Owner References:
    API Version:           apps.kubeblocks.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Component
    Name:                  mongodbs-hovpkc-htenq-config-server
    UID:                   6009e111-2482-434e-abe9-c4a06399acaa
  Resource Version:        267085025
  UID:                     c4994c00-6b5e-41c8-80a7-e8bab64e2465
Spec:
  Backup:
    Name:       backup-default-mongodbs-hovpkc-20260813160428
    Namespace:  default
  Container Resources:
  Ready Config:
    Connection Credential:
      Password Key:  password
      Secret Name:   mongodbs-hovpkc-htenq-config-server-account-root
      Username Key:  username
    Exec Action:
      Target:
        Pod Selector:
          Match Labels:
            app.kubernetes.io/instance:         mongodbs-hovpkc-htenq
            app.kubernetes.io/managed-by:       kubeblocks
            apps.kubeblocks.io/component-name:  config-server
    Job Action:
      Target:
        Pod Selector:
          Match Labels:
            app.kubernetes.io/instance:         mongodbs-hovpkc-htenq
            app.kubernetes.io/managed-by:       kubeblocks
            apps.kubeblocks.io/component-name:  config-server
            kubeblocks.io/role:                 primary
          Strategy:                             Any
        Volume Mounts:
          Mount Path:  /data/mongodb
          Name:        data
Status:
  Actions:
    Post Ready:
      Backup Name:  backup-default-mongodbs-hovpkc-20260813160428
      End Time:     2026-08-13T08:06:38Z
      Message:      successfully processed the "Job/restore-post-ready-c4994c00-backup-default-mongodbs-hovpk-8-0-0"
      Name:         postReady-0
      Object Key:   Job/restore-post-ready-c4994c00-backup-default-mongodbs-hovpk-8-0-0
      Start Time:   2026-08-13T08:06:12Z
      Status:       Completed
      Backup Name:  backup-default-mongodbs-hovpkc-20260813160428
      End Time:     2026-08-13T08:09:27Z
      Message:      Log Collector Output: -hovpkc-htenq-shard-5xn-0.mongodbs-hovpkc-htenq-shard-5xn-headless.default.svc.cluster.local:27017] [restore/2026-08-13T08:08:18.317735048Z] restore: waiting for dumpDone: restore stuck, last beat ts: 1786608503
2026-08-13T08:08:54Z E [mongodbs-hovpkc-htenq-shard-ljc/mongodbs-hovpkc-htenq-shard-ljc-0.mongodbs-hovpkc-htenq-shard-ljc-headless.default.svc.cluster.local:27017] [restore/2026-08-13T08:08:18.317735048Z] restore: waiting for dumpDone: restore stuck, last beat ts: 1786608503
failed with exit code 1

      Name:              postReady-1
      Object Key:        Job/restore-post-ready-c4994c00-backup-default-mongodbs-hovpk-8-1-0
      Start Time:        2026-08-13T08:06:38Z
      Status:            Failed
  Completion Timestamp:  2026-08-13T08:09:27Z
  Conditions:
    Last Transition Time:  2026-08-13T08:06:12Z
    Message:               
    Reason:                CheckBackupRepoSuccessfully
    Status:                True
    Type:                  CheckBackupRepo
    Last Transition Time:  2026-08-13T08:06:12Z
    Message:               validate restore spec successfully
    Reason:                ValidateSuccessfully
    Status:                True
    Type:                  ValidationPassed
    Last Transition Time:  2026-08-13T08:06:12Z
    Message:               restore failed for backup "backup-default-mongodbs-hovpkc-20260813160428", more information can be found in status.actions.postReady
    Reason:                Failed
    Status:                False
    Type:                  PostReady
  Duration:                3m15s
  Phase:                   Failed
  Start Timestamp:         2026-08-13T08:06:12Z
Events:
  Type     Reason            Age    From                Message
  ----     ------            ----   ----                -------
  Normal   RestoreStarting   7m23s  restore-controller  start to restore
  Normal   CreateRestoreJob  7m23s  restore-controller  created job default/restore-post-ready-c4994c00-backup-default-mongodbs-hovpk-8-0-0
  Normal   CreateRestoreJob  6m57s  restore-controller  created job default/restore-post-ready-c4994c00-backup-default-mongodbs-hovpk-8-1-0
  Warning  RestoreFailed     4m8s   restore-controller  restore failed for backup "backup-default-mongodbs-hovpkc-20260813160428", more information can be found in status.actions.postReady

kubectl get pod | grep post-ready
restore-post-ready-15eb249d-backup-default-mongodbs-ucdaa-4647l   0/2     Error         0          11m
restore-post-ready-15eb249d-backup-default-mongodbs-ucdaa-bksnz   0/2     Completed     0          12m
restore-post-ready-15eb249d-backup-default-mongodbs-ucdaa-gvll7   0/2     Error         0          9m46s

logs error pod

kubectl logs restore-post-ready-15eb249d-backup-default-mongodbs-ucdaa-4647l
Defaulted container "restore" out of: restore, restore-manager, dp-copy-datasafed (init)
2026-08-13 08:04:04 INFO: storage config have been extracted.
Error: get status of cluster: get cluster status: get config: get: mongo: no documents in result
INFO: PBM is not configured.
INFO: Checking if PBM storage config exists
Error: get: mongo: no documents in result
INFO: PBM storage config does not exist.
INFO: PBM storage configuration completed.
INFO: Syncing PBM config from storage...
Storage resync started
INFO: PBM config synced from storage.
INFO: Config server replica set name: mongodbs-ucdaah-config-server
INFO: Shard replica set names: mongodbs-ucdaah-shard-scs,mongodbs-ucdaah-shard-fvd,mongodbs-ucdaah-shard-rqn
INFO: Mapping shard mongodbs-ucdaah-shard-scs to mongodbs-ucdaah-wihpk-shard-2fh
INFO: Mapping shard mongodbs-ucdaah-shard-fvd to mongodbs-ucdaah-wihpk-shard-cn6
INFO: Mapping shard mongodbs-ucdaah-shard-rqn to mongodbs-ucdaah-wihpk-shard-lc8
INFO: Mapping config server mongodbs-ucdaah-config-server to mongodbs-ucdaah-wihpk-config-server
INFO: Shard mappings: mongodbs-ucdaah-wihpk-shard-2fh=mongodbs-ucdaah-shard-scs,mongodbs-ucdaah-wihpk-shard-cn6=mongodbs-ucdaah-shard-fvd,mongodbs-ucdaah-wihpk-shard-lc8=mongodbs-ucdaah-shard-rqn,mongodbs-ucdaah-wihpk-config-server=mongodbs-ucdaah-config-server
INFO: Waiting for prepare restore start signal...
configmap/mongodbs-ucdaah-wihpk-restore-signal created
INFO: Prepare restore start signal completed.
INFO: Restore 2026-08-13T08:04:16.364785854Z status: , retrying in 5s...
INFO: Restore 2026-08-13T08:04:16.364785854Z status: , retrying in 5s...
INFO: Restore 2026-08-13T08:04:16.364785854Z status: , retrying in 5s...
INFO: Restore 2026-08-13T08:04:16.364785854Z status: , retrying in 5s...
INFO: Restore 2026-08-13T08:04:16.364785854Z status: , retrying in 5s...
INFO: Restore 2026-08-13T08:04:16.364785854Z status: , retrying in 5s...
INFO: Restore 2026-08-13T08:04:16.364785854Z status: , retrying in 5s...
INFO: Restore 2026-08-13T08:04:16.364785854Z status: , retrying in 5s...
INFO: Restore 2026-08-13T08:04:16.364785854Z status: , retrying in 5s...
INFO: Restore 2026-08-13T08:04:16.364785854Z status: , retrying in 5s...
INFO: Restore 2026-08-13T08:04:16.364785854Z status: , retrying in 5s...
INFO: Restore 2026-08-13T08:04:16.364785854Z status: , retrying in 5s...
INFO: Restore 2026-08-13T08:04:16.364785854Z status: , retrying in 5s...
ERROR: Restore 2026-08-13T08:04:16.364785854Z status is still empty after 12 retries
INFO: Printing PBM tail logs
2026-08-13T08:04:19Z I [mongodbs-ucdaah-wihpk-shard-2fh/mongodbs-ucdaah-wihpk-shard-2fh-0.mongodbs-ucdaah-wihpk-shard-2fh-headless.default.svc.cluster.local:27017] [restore/2026-08-13T08:04:16.364785854Z] moving to state running
2026-08-13T08:04:20Z I [mongodbs-ucdaah-wihpk-config-server/mongodbs-ucdaah-wihpk-config-server-0.mongodbs-ucdaah-wihpk-config-server-headless.default.svc.cluster.local:27017] [restore/2026-08-13T08:04:16.364785854Z] moving to state running
2026-08-13T08:04:22Z I [mongodbs-ucdaah-wihpk-shard-lc8/mongodbs-ucdaah-wihpk-shard-lc8-0.mongodbs-ucdaah-wihpk-shard-lc8-headless.default.svc.cluster.local:27017] [restore/2026-08-13T08:04:16.364785854Z] restoring users and roles
2026-08-13T08:04:22Z I [mongodbs-ucdaah-wihpk-shard-cn6/mongodbs-ucdaah-wihpk-shard-cn6-0.mongodbs-ucdaah-wihpk-shard-cn6-headless.default.svc.cluster.local:27017] [restore/2026-08-13T08:04:16.364785854Z] restoring users and roles
2026-08-13T08:04:22Z I [mongodbs-ucdaah-wihpk-shard-lc8/mongodbs-ucdaah-wihpk-shard-lc8-0.mongodbs-ucdaah-wihpk-shard-lc8-headless.default.svc.cluster.local:27017] [restore/2026-08-13T08:04:16.364785854Z] moving to state dumpDone
2026-08-13T08:04:22Z I [mongodbs-ucdaah-wihpk-shard-cn6/mongodbs-ucdaah-wihpk-shard-cn6-0.mongodbs-ucdaah-wihpk-shard-cn6-headless.default.svc.cluster.local:27017] [restore/2026-08-13T08:04:16.364785854Z] moving to state dumpDone
2026-08-13T08:04:22Z I [mongodbs-ucdaah-wihpk-shard-2fh/mongodbs-ucdaah-wihpk-shard-2fh-0.mongodbs-ucdaah-wihpk-shard-2fh-headless.default.svc.cluster.local:27017] [restore/2026-08-13T08:04:16.364785854Z] restoring users and roles
2026-08-13T08:04:22Z I [mongodbs-ucdaah-wihpk-shard-2fh/mongodbs-ucdaah-wihpk-shard-2fh-0.mongodbs-ucdaah-wihpk-shard-2fh-headless.default.svc.cluster.local:27017] [restore/2026-08-13T08:04:16.364785854Z] moving to state dumpDone
2026-08-13T08:04:24Z I [mongodbs-ucdaah-wihpk-config-server/mongodbs-ucdaah-wihpk-config-server-0.mongodbs-ucdaah-wihpk-config-server-headless.default.svc.cluster.local:27017] [restore/2026-08-13T08:04:16.364785854Z] restoring users and roles
2026-08-13T08:04:24Z I [mongodbs-ucdaah-wihpk-config-server/mongodbs-ucdaah-wihpk-config-server-0.mongodbs-ucdaah-wihpk-config-server-headless.default.svc.cluster.local:27017] [restore/2026-08-13T08:04:16.364785854Z] moving to state dumpDone
2026-08-13T08:04:30Z I [mongodbs-ucdaah-wihpk-config-server/mongodbs-ucdaah-wihpk-config-server-0.mongodbs-ucdaah-wihpk-config-server-headless.default.svc.cluster.local:27017] 
                    %                        _____
                   %%%                      |  __ \
                 ###%%%%%%%%%%%%*           | |__) |__ _ __ ___ ___  _ __   __ _
                ###  ##%%      %%%%         |  ___/ _ \ '__/ __/ _ \| '_ \ / _` |
              ####     ##%       %%%%       | |  |  __/ | | (_| (_) | | | | (_| |
             ###        ####      %%%       |_|   \___|_|  \___\___/|_| |_|\__,_|
           ,((###         ###     %%%         _____                       _
          (((( (###        ####  %%%%        / ____|                     | |
         (((     ((#         ######         | (___   __ _ _   _  __ _  __| |
       ((((       (((#        ####           \___ \ / _` | | | |/ _` |/ _` |
      /((          ,(((        *###          ____) | (_| | |_| | (_| | (_| |
    ////             (((         ####       |_____/ \__, |\__,_|\__,_|\__,_|
   ///                ((((        ####                 | |
 /////////////(((((((((((((((((########                |_|   Join @ squad.percona.com/mongodb

** Join Percona Squad! **
Participate in monthly SWAG raffles, get early access to new product features,
invite-only ”ask me anything” sessions with database performance experts.

Interested? Fill in the form at squad.percona.com/mongodb


2026-08-13T08:04:30Z I [mongodbs-ucdaah-wihpk-config-server/mongodbs-ucdaah-wihpk-config-server-0.mongodbs-ucdaah-wihpk-config-server-headless.default.svc.cluster.local:27017] log options: log-path=/dev/stderr, log-level:D, log-json:false
2026-08-13T08:04:30Z I [mongodbs-ucdaah-wihpk-config-server/mongodbs-ucdaah-wihpk-config-server-0.mongodbs-ucdaah-wihpk-config-server-headless.default.svc.cluster.local:27017] pbm-agent:
Version:   2.12.0
Platform:  linux/amd64
GitCommit: 2f7100a2a1da2bb7074e15d757693999cac7574e
GitBranch: release-2.12.0
BuildTime: 2025-10-31_09:21_UTC
GoVersion: go1.25.3
2026-08-13T08:04:30Z I [mongodbs-ucdaah-wihpk-config-server/mongodbs-ucdaah-wihpk-config-server-0.mongodbs-ucdaah-wihpk-config-server-headless.default.svc.cluster.local:27017] starting PITR routine
2026-08-13T08:04:30Z I [mongodbs-ucdaah-wihpk-config-server/mongodbs-ucdaah-wihpk-config-server-0.mongodbs-ucdaah-wihpk-config-server-headless.default.svc.cluster.local:27017] node: mongodbs-ucdaah-wihpk-config-server/mongodbs-ucdaah-wihpk-config-server-0.mongodbs-ucdaah-wihpk-config-server-headless.default.svc.cluster.local:27017
2026-08-13T08:04:30Z I [mongodbs-ucdaah-wihpk-config-server/mongodbs-ucdaah-wihpk-config-server-0.mongodbs-ucdaah-wihpk-config-server-headless.default.svc.cluster.local:27017] conn level ReadConcern: majority; WriteConcern: majority
2026-08-13T08:04:30Z I [mongodbs-ucdaah-wihpk-config-server/mongodbs-ucdaah-wihpk-config-server-0.mongodbs-ucdaah-wihpk-config-server-headless.default.svc.cluster.local:27017] listening for the commands
2026-08-13T08:04:52Z E [mongodbs-ucdaah-wihpk-shard-cn6/mongodbs-ucdaah-wihpk-shard-cn6-0.mongodbs-ucdaah-wihpk-shard-cn6-headless.default.svc.cluster.local:27017] [restore/2026-08-13T08:04:16.364785854Z] restore: waiting for dumpDone: restore stuck, last beat ts: 1786608261
2026-08-13T08:04:52Z E [mongodbs-ucdaah-wihpk-shard-lc8/mongodbs-ucdaah-wihpk-shard-lc8-0.mongodbs-ucdaah-wihpk-shard-lc8-headless.default.svc.cluster.local:27017] [restore/2026-08-13T08:04:16.364785854Z] restore: waiting for dumpDone: restore stuck, last beat ts: 1786608261
2026-08-13T08:04:52Z E [mongodbs-ucdaah-wihpk-shard-2fh/mongodbs-ucdaah-wihpk-shard-2fh-0.mongodbs-ucdaah-wihpk-shard-2fh-headless.default.svc.cluster.local:27017] [restore/2026-08-13T08:04:16.364785854Z] restore: waiting for dumpDone: restore stuck, last beat ts: 1786608261
failed with exit code 1

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind/bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions