branch-operator wipes user-set ObjectStore fields (e.g. wal.maxParallel) on reconcile — can't customize barman-cloud config

[acrogenesis]

Summary

The Xata branch-operator reconciles its child ObjectStore CR with the Update verb (not Apply), which strips any field not present in its own template. This means fields the upstream barman-cloud-plugin CRD supports — like spec.configuration.wal.maxParallel — can't be customized externally. Every set attempt (SSA, JSON-Patch replace, merge patch) is reverted within milliseconds.

This bites hard during bulk data migrations: barman-cloud-wal-archive defaults to serial uploads (maxParallel: 1), and pg_restore of a multi-tens-of-GB DB generates WAL much faster than serial S3 uploads can ship it. WAL piles up in pg_wal/, fills the PVC, and crashes the postgres pod.

Repro

Verified on ghcr.io/xataio/xata/branch-operator:e1005c161244:

# 1. Try any of: SSA, merge patch, JSON-Patch replace
kubectl -n xata-clusters apply -f - --server-side --field-manager=ops-test --force-conflicts <<'YAML'
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: main-<branch>
  namespace: xata-clusters
spec:
  configuration:
    wal:
      maxParallel: 8
YAML

# kubectl: "serverside-applied"

# 2. Immediately read
kubectl -n xata-clusters get objectstore main-<branch> -o jsonpath='{.spec.configuration.wal}{"\n"}'
# {"compression":"gzip"}        ← maxParallel already gone

managedFields shows the offending operation:

manager: manager
operation: Update                ← branch-operator

Update semantics ignore field ownership entirely and replace the whole object server-side. SSA-aware controllers cooperate; Update-based ones can't.

Concrete impact

Real production migration we hit yesterday (~80 GB DB → fresh Xata cluster on g6-dedicated-8, Linode Object Storage):

• maxParallel: 1 (the unconfigurable default): pg_restore generated WAL at ~80 MB/s; archive shipped at ~5 MB/s; ~9,500 WAL segments piled up (≈ 151 GB) inside ~30 min and crashed the postgres pod (filesystem full).
• maxParallel: 8 (forced via scale-branch-op-to-0 + patch + restart): 8 concurrent uploads, archive kept pace, migration completed cleanly with no WAL backlog.

So this knob materially affects whether bulk migrations succeed on first try. Defaulting to 1 is hostile to the migration-to-Xata workflow that's a primary onboarding case.

Operational workaround (currently the only thing that works)

kubectl -n xata scale deploy branch-operator --replicas=0
kubectl -n xata-clusters patch objectstore main-<branch> --type=merge \
  -p '{\"spec\":{\"configuration\":{\"wal\":{\"maxParallel\":8}}}}'
kubectl -n xata-clusters delete pod main-<branch>-1     # so postgres sidecar re-reads config
# … run migration …
kubectl -n xata scale deploy branch-operator --replicas=1

Unergonomic and requires HA-degrading the cluster during the migration window.

Proposed fix

Either of the two would work; (1) is the smaller change:

1. Plumb Branch.spec.backup.wal through to ObjectStore.spec.configuration.wal. Today Branch.spec.backup exposes retention, walArchiving, dbBackup, serverName but not the underlying barman-cloud wal.* knobs (maxParallel, compression, encryption, archiveAdditionalCommandArgs, restoreAdditionalCommandArgs). Forwarding them would let users configure barman without bypassing the operator.
2. Switch branch-operator's ObjectStore reconciliation from Update to SSA (server-side Apply with a field-manager: branch-operator). This is the more general fix — it'd also let third-party tools (Pulumi, ArgoCD, kustomize overlays) own specific fields without getting stomped. Plus it's the modern controller-runtime pattern.

Environment

• branch-operator: ghcr.io/xataio/xata/branch-operator:e1005c161244
• CNPG operator: ghcr.io/xataio/xata-cnpg/cloudnative-pg:g4586f2942 (v1.28.0)
• barman-cloud-plugin: barman-cloud.cloudnative-pg.io v0.12.0
• Kubernetes: LKE 1.35.3

[tsg]

hi @acrogenesis,

we're generally working towards replacing the barman-cloud-plugin with an in-image pgBackRest because of these types of issues. We didn't expose maxParallel because that used to cause crashes during restores. I have to check if that's still an issue.

That said, we can do your proposed fix 1 for now, especially if the upstream maxParallel issue is solved in the meantime. Let me look into it.

While not really a solution, I have a couple of tips to deal with these situations:

• you can disable reconciliation for individual branches via an annotation:

metadata:
  name: example-branch
  annotations:
    xata.io/reconciliation-paused: "true"

• if you remove the whole backups section in the branch spec, it will disable the barman plugin. You can add it back after you have done the initial data load. This is because even with maxParallel you run the risk of WALs filling up the disk.

[burnerlee]

hey @tsg is it okay if I take a shot at this issue?