Streaming Replication, Recovery and FTS in Cloudberry

What actually happens between a segment host dying and the cluster serving queries again — traced end to end.

Streaming Replication

20.3.1 walsender / walreceiver and replication slots

The same PostgreSQL streaming replication drives both segment primary→mirror and coordinator→standby. The mirror runs a walreceiver; the primary spawns a walsender per connection. WalSndLoop() (src/backend/replication/walsender.c:2521) ships flushed WAL via XLogSendPhysical(), and a sender in WALSNDSTATE_CATCHUP only promotes to WALSNDSTATE_STREAMING once fully drained — a point the comment explicitly ties to no-data-loss failover. Cloudberry tags the segment-replication sender: walsnd->is_for_gp_walreceiver = (strcmp(application_name, GP_WALRECEIVER_APPNAME) == 0) (walsender.c:2743).

WalReceiverMain() (src/backend/replication/walreceiver.c:189) loops: walrcv_receive() → XLogWalRcvProcessMsg() writes WAL → XLogWalRcvSendReply() → XLogWalRcvFlush() (:525) advances the mirror’s flush LSN, which the primary’s synchronous-replication logic waits on. A physical replication slot pins the WAL a lagging mirror still needs by holding back restart_lsn; Cloudberry’s internal segment slot is INTERNAL_WAL_REPLICATION_SLOT_NAME = "internal_wal_replication_slot" (src/include/replication/slot.h:191).

20.3.2 Connection establishment and execution flow

Streaming on a standby is driven by two cooperating processes: the startup process (replays WAL, decides when streaming is needed) and the walreceiver (connects to the primary’s walsender and pulls WAL). They communicate only through the WalRcv shared struct, whose walRcvState field is a small state machine: WALRCV_STOPPED → WALRCV_STARTING → WALRCV_STREAMING, plus WALRCV_WAITING / WALRCV_RESTARTING / WALRCV_STOPPING.

Step 1 — startup requests streaming. When the startup process needs WAL it cannot find locally, it calls RequestXLogStreaming(tli, recptr, conninfo, slotname, ...) (walreceiverfuncs.c): under the spinlock it stores conninfo / slot / start LSN+timeline into WalRcv and flips the state, signalling the postmaster to fork a walreceiver (or setting the latch if one is alive):

src/backend/replication/walreceiverfuncs.c:312

if (walrcv->walRcvState == WALRCV_STOPPED)
{ launch = true; walrcv->walRcvState = WALRCV_STARTING; }
else
    walrcv->walRcvState = WALRCV_RESTARTING;
walrcv->receiveStart = recptr;
walrcv->receiveStartTLI = tli;
if (launch)      SendPostmasterSignal(PMSIGNAL_START_WALRECEIVER);
else if (latch)  SetLatch(latch);

Step 2–3 — walreceiver reads its orders and connects. WalReceiverMain() (walreceiver.c:189) moves the shared state to WALRCV_STREAMING, copies out the conninfo / slot / startpoint, then calls walrcv_connect(). The appname argument becomes the connection’s application_name — in CBDB this is cluster_name when set (deployed as gp_walreceiver):

src/backend/replication/walreceiver.c:304

/* Establish the connection to the primary for XLOG streaming */
wrconn = walrcv_connect(conninfo, false, false,
                        cluster_name[0] ? cluster_name : "walreceiver", &err);
if (!wrconn)
    ereport(ERROR, (errcode(ERRCODE_CONNECTION_FAILURE),
        errmsg("could not connect to the primary server: %s", err)));

walrcv_connect dispatches to libpqrcv_connect() (libpqwalreceiver.c:138), which builds a libpq connection with the crucial replication=true (a physical replication connection — not "database", which is logical) and the appname as fallback_application_name:

src/backend/replication/libpqwalreceiver/libpqwalreceiver.c:161

keys[i] = "dbname";        vals[i] = conninfo;
keys[++i] = "replication"; vals[i] = logical ? "database" : "true";
keys[++i] = "fallback_application_name"; vals[i] = appname;
conn->streamConn = PQconnectStartParams(keys, vals, true);

Because the standby connects with application_name = gp_walreceiver, the primary’s walsender sets is_for_gp_walreceiver when it sees this name (§20.3.1) — wiring the connection into Greenplum/Cloudberry FTS replication tracking.

Step 4–5 — identify & START_REPLICATION. The walreceiver issues IDENTIFY_SYSTEM (cross-checking the system id and that the primary’s timeline is at/ahead of its startpointTLI), then walrcv_startstreaming() → libpqrcv_startstreaming() sends the START_REPLICATION [SLOT name] <LSN> TIMELINE <tli> command. A physical start returns PGRES_COPY_BOTH:

src/backend/replication/libpqwalreceiver/libpqwalreceiver.c:499

appendStringInfoString(&cmd, "START_REPLICATION");
if (options->slotname != NULL)
    appendStringInfo(&cmd, " SLOT "%s"", options->slotname);
appendStringInfo(&cmd, " %X/%X", LSN_FORMAT_ARGS(options->startpoint));
appendStringInfo(&cmd, " TIMELINE %u", options->proto.physical.startpointTLI);
res = libpqrcv_PQexec(conn->streamConn, cmd.data);   /* expect PGRES_COPY_BOTH */

Step 6 — the receive loop. Once streaming, the walreceiver loops on walrcv_receive() → XLogWalRcvProcessMsg (writes WAL for 'w' data, handles 'k' keepalives), then XLogWalRcvSendReply (tells the primary the received LSN) and XLogWalRcvFlush (fsync, advancing the standby’s flushed LSN). When no data is ready it naps on WaitLatchOrSocket:

src/backend/replication/walreceiver.c:480

len = walrcv_receive(wrconn, &buf, &wait_fd);
if (len != 0)
{
    for (;;) {
        if (len > 0)  XLogWalRcvProcessMsg(buf[0], &buf[1], len - 1, startpointTLI);
        else if (len == 0) break;
        else if (len < 0) { endofwal = true; break; }   /* timeline ended */
        len = walrcv_receive(wrconn, &buf, &wait_fd);
    }
    XLogWalRcvSendReply(false, false);
    XLogWalRcvFlush(false, startpointTLI);     /* advance flushedUpto */
}

The flushed LSN written here is exactly the value CreateRestartPoint (§20.2.6) reads via GetWalRcvFlushRecPtr to decide which WAL segments are still needed — closing the loop between replication and checkpointing. A len < 0 means the primary ended the COPY stream (end of timeline); the loop returns control to await fresh orders from the startup process.

20.3.3 Synchronous replication — waiting for the mirror’s flush LSN

A synchronous commit blocks in SyncRepWaitForLSN(lsn, commit) (src/backend/replication/syncrep.c:187): the backend enqueues itself, sets syncRepState = SYNC_REP_WAITING, and sleeps on its latch until a walsender confirms the LSN. The wakeup side is SyncRepReleaseWaiters() (syncrep.c:647), which publishes the confirmed flush LSN and wakes every waiter at or below it:

src/backend/replication/syncrep.c:385 (wait) / :724 (release)

/* in SyncRepWaitForLSN: */
for (;;) {
    ResetLatch(MyLatch);
    if (MyProc->syncRepState == SYNC_REP_WAIT_COMPLETE) break;
    ...
}
/* in SyncRepReleaseWaiters, when the standby's flush advances: */
if (walsndctl->lsn[SYNC_REP_WAIT_FLUSH] < flushPtr) {
    walsndctl->lsn[SYNC_REP_WAIT_FLUSH] = flushPtr;
    numflush = SyncRepWakeQueue(false, SYNC_REP_WAIT_FLUSH);
}

At the default flush level this means the commit is durable on both primary and mirror before the client sees success — an in-sync mirror loses no acknowledged commit. Cloudberry enables synchronous replication for segments by setting synchronous_standby_names = "*", and special-cases the coordinator: under IS_QUERY_DISPATCHER() it does not block when there is no active is_for_gp_walreceiver standby (syncrep.c:248), so a missing standby coordinator never stalls commits.

20.3.4 The tie to FTS — disabling syncrep when a mirror is down

This is the key availability mechanism: if a mirror dies, synchronous commits would block forever, so FTS must turn replication synchronicity off. SyncRepWaitForLSN’s fast-exit keys on SYNC_STANDBY_DEFINED, which tracks synchronous_standby_names. Cloudberry flips it programmatically — SetSyncStandbysDefined() sets it to "*", UnsetSyncStandbysDefined() to "" (src/backend/replication/gp_replication.c:598,612). When FTS sends the SYNCREP_OFF message, the primary handles it by unsetting the GUC so commits proceed immediately:

src/backend/fts/ftsmessagehandler.c:307 (HandleFtsWalRepSyncRepOff)

ereport(LOG, (errmsg("turning off synchronous wal replication due to FTS request")));
UnsetSyncStandbysDefined();              /* synchronous_standby_names = "" */
GetMirrorStatus(&response, NULL);
SendFtsResponse(&response, FTS_MSG_SYNCREP_OFF);

Bridge to §20.4. Replication guarantees a second, byte-identical copy of the WAL exists. What it does not do is make that copy usable: a mirror’s data files are mid-stream and inconsistent until the log is replayed and any in-doubt transactions resolved. Turning a pile of streamed WAL back into a consistent, openable database is system recovery.

System Recovery

20.4.1 StartupXLOG and the control-file DBState switch

Single-node recovery is driven by StartupXLOG() (src/backend/access/transam/xlog.c:5320). It first branches on ControlFile->state to learn how the previous instance terminated, then reads the checkpoint and seeds shared memory. Only DB_SHUTDOWNED and DB_SHUTDOWNED_IN_RECOVERY count as a clean stop; everything else sets didCrash = true:

src/backend/access/transam/xlog.c:5354 (DBState switch, condensed)

switch (ControlFile->state)
{
    case DB_SHUTDOWNED:             /* clean — redo usually skipped       */
    case DB_SHUTDOWNED_IN_RECOVERY: /* standby cleanly stopped            */
    case DB_SHUTDOWNING:            /* shutdown interrupted -> crash       */
    case DB_IN_CRASH_RECOVERY:      /* crashed *while* recovering          */
    case DB_IN_ARCHIVE_RECOVERY:    /* crashed during archive/standby redo */
    case DB_IN_PRODUCTION:          /* crashed while live — common case    */
    default:                        /* FATAL: invalid cluster state        */
}

The checkpoint then seeds ShmemVariableCache, including the Cloudberry-specific distributed xid counter — without it the coordinator could re-issue gxids already used before the crash:

src/backend/access/transam/xlog.c:5485

ShmemVariableCache->nextXid  = checkPoint.nextXid;
ShmemVariableCache->nextGxid = checkPoint.nextGxid;   /* DistributedTransactionId */
...
StartupCLOG(); StartupMultiXact();                    /* SLRUs, after nextXid set */

pg_subtrans and the segment-only pg_distributedlog SLRU are started once the oldest active xid is known; DistributedLog_Startup() is a no-op on the coordinator (distributedlog.c:766).

20.4.2 The redo loop and invalid-page tracking

PerformWalRecovery() (src/backend/access/transam/xlogrecovery.c:1684) positions at the checkpoint REDO point and applies records, dispatching each to its resource manager: GetRmgr(record->xl_rmid).rm_redo(xlogreader) (xlogrecovery.c:2023). If full_page_writes is off, a record may reference a page whose relation was later dropped/truncated; rather than failing, log_invalid_page() (src/backend/access/transam/xlogutils.c:107) records the reference, forget_invalid_pages() removes it when the covering drop/truncate replays, and XLogCheckInvalidPages() (xlogutils.c:251) PANICs at the consistency point if any reference is still unresolved — that means the WAL was incomplete or corrupt.

20.4.3 Crash vs archive/standby recovery; the crash-fsync deviation

The mode is chosen by two empty signal files (recovery.conf is now fatal): standby.signal → standby mode (how a standby coordinator and segment mirrors run); recovery.signal → archive/PITR; neither → plain crash recovery (xlogrecovery.c:1062). Cloudberry deliberately diverges from upstream on crash fsync — instead of fsyncing the whole data directory (which can hold millions of AO/heap files) it syncs only the WAL files, relying on full-page-write images to repair torn data pages during redo:

src/backend/access/transam/xlog.c:5441 (GPDB crash path)

if (ControlFile->state != DB_SHUTDOWNED &&
    ControlFile->state != DB_SHUTDOWNED_IN_RECOVERY)
{
    RemoveTempXlogFiles();
    if (access(BACKUP_LABEL_FILE, F_OK) != 0)
        SyncAllXLogFiles();                 /* WAL only, NOT full pgdata */
    if (Gp_role == GP_ROLE_DISPATCH)
        *shmCleanupBackends = true;         /* tell DTX recovery to clean QEs */
    didCrash = true;
}

20.4.4 Distributed recovery — the second layer

Local replay makes each node self-consistent, but a distributed transaction commits across many nodes. Where §12.2 drove the 2PC protocol at runtime, recovery must reconstruct and finish whatever was in flight when the cluster stopped. Cloudberry resolves in-doubt 2PC in two layers. Layer 1 (capture & seed): every checkpoint snapshots in-flight DTX via getDtxCheckPointInfo() (src/backend/storage/ipc/procarray.c:2541) — the committed gxids plus any in-progress dtx flagged includeInCkpt — into the checkpoint record (call site xlog.c:7274/:7314). On replay, redoDtxCheckPoint() feeds them to redoDistributedCommitRecord(), seeding shmCommittedGxidArray.

Layer 2 (reconcile): a dedicated dtx recovery background worker (registered src/backend/postmaster/postmaster.c:428; coordinator-only via DtxRecoveryStartRule) runs DtxRecoveryMain() → recoverTM() → recoverInDoubtTransactions() (src/backend/cdb/cdbdtxrecovery.c:181), which makes two passes:

src/backend/cdb/cdbdtxrecovery.c:203 (recoverInDoubtTransactions, condensed)

/* Pass 1: finish the ones we KNOW committed (seeded from replay) */
for (i = 0; i < *shmNumCommittedGxacts; i++) {
    dtxFormGid(gid, shmCommittedGxidArray[i]);
    doNotifyCommittedInDoubt(gid);            /* re-broadcast COMMIT PREPARED */
    RecordDistributedForgetCommitted(shmCommittedGxidArray[i]);
}
*shmNumCommittedGxacts = 0;

/* Pass 2: anything else still prepared on a segment must be aborted */
htab = gatherRMInDoubtTransactions(0, true);  /* select gid from pg_prepared_xacts */
abortRMInDoubtTransactions(htab);             /* ROLLBACK PREPARED */

After the one-time pass, the worker loops periodically calling AbortOrphanedPreparedTransactions(), which aborts genuinely orphaned prepared transactions while skipping any gid still IsDtxInProgress() — catching stragglers from later failures.

20.4.5 How prepared (phase-1) transactions survive a crash

A segment’s 2PC state is an ordinary PostgreSQL prepared transaction, durable via the XLOG_XACT_PREPARE record and the pg_twophase/<xid> state file. restoreTwoPhaseData() scans the on-disk files before replay; PrepareRedoAdd() reconstructs entries during replay; RecoverPreparedTransactions() (src/backend/access/transam/twophase.c:2200) does the full reload at end of recovery — and crucially cracks the Cloudberry GID to recover the distributed xid and mark the local distributed-xact active:

src/backend/access/transam/twophase.c:2256 (excerpt)

dtxDeformGid(gid, &distribXid);
localDistribXactData.state     = LOCALDISTRIBXACT_STATE_ACTIVE;
localDistribXactData.distribXid = distribXid;
MarkAsPreparingGuts(gxact, xid, gid, &localDistribXactData, ...);

This is exactly the state a segment exposes through pg_prepared_xacts, which the coordinator’s gatherRMInDoubtTransactions() queries in §20.4.4 to decide commit vs. abort.

20.4.6 DistributedLog’s role in recovery

pg_distributedlog (a segment-side SLRU) maps each local xid to the distributed xid under which it committed — the basis of distributed-snapshot visibility after restart. Heap visibility uses pg_clog (local commit); distributed visibility uses pg_distributedlog (DistributedLog_CommittedCheck(), distributedlog.c:491) to translate a local xid into a distributed xid the snapshot can test. Recovery rebuilds both before any query runs; on a hot-standby coordinator the forget redo also advances latestCompletedGxid to keep distributed snapshots correct during continuous replay.

20.4.7 Standby promotion and segment recovery

When a standby coordinator finishes replaying and is promoted, StartupXLOG (after marking DB_IN_PRODUCTION) calls UpdateCatalogForStandbyPromotion() (xlog.c:5162) → gp_activate_standby() (src/backend/utils/gp/segadmin.c:739), which swaps the dbids of the old and new coordinator in gp_segment_configuration (idempotent, so it survives a crash mid-promotion). A segment mirror, by contrast, is promoted by FTS (§20.5).

A failed segment is rebuilt with gprecoverseg: incremental recovery runs pg_rewind (copies only diverged blocks; the default when the old data dir is intact), and full recovery runs pg_basebackup to clone the whole directory. After file-level recovery the segment starts in standby.signal mode and catches up via streaming replication; FTS then re-marks it up.

Bridge to §20.5. Local and distributed recovery can make a failed node consistent again — but only once something restarts it, and only after something decided it had failed and, if necessary, promoted its mirror in the meantime. That decision-maker is the cluster’s fault detector: FTS, the pillar that turns a detected failure into a role change.

Cluster High Availability — FTS

20.5.1 Process model, triggering, and key GUCs

FTS (Fault Tolerance Service) runs as a single background worker started only on the coordinator — FtsProbeStartRule returns Gp_role == GP_ROLE_DISPATCH (src/backend/fts/fts.c:108). FtsProbeMain publishes its PID, installs handlers, and enters FtsLoop() (fts.c:279). Each cycle reads gp_segment_configuration into a CdbComponentDatabases and, if mirrors exist, calls FtsWalRepMessageSegments().

Triggering is timer + on-demand: the loop sleeps gp_fts_probe_interval on a latch, but a backend that hits a fault calls FtsNotifyProber() (sends PMSIGNAL_WAKEN_FTS), forcing an immediate cycle. The key GUCs (defaults from src/backend/utils/misc/guc_gp.c):

• gp_fts_probe_interval — 60s (10–3600): period of a full probe cycle.
• gp_fts_probe_timeout — 20s (0–3600): max wait for one segment’s response before it’s treated as failed.
• gp_fts_probe_retries — 5 (0–100): retries before a failure is acted on.
• gp_fts_mark_mirror_down_grace_period — 30s: grace window after a mirror disconnect before it may be marked down.
• gp_fts_replication_attempt_count — 10: walsender reconnect attempts after which the disconnect timestamp is ignored (so a crash-looping walsender can’t extend the grace window forever).

20.5.2 The probe protocol and the response fields

There are exactly three message types, sent as a libpq query string, with one fixed 5-boolean response shape (src/include/postmaster/fts_comm.h):

src/include/postmaster/fts_comm.h:44 / :103

#define FTS_MSG_PROBE        "PROBE"
#define FTS_MSG_SYNCREP_OFF  "SYNCREP_OFF"
#define FTS_MSG_PROMOTE      "PROMOTE"

#define Natts_fts_message_response 5
#define Anum_fts_message_response_is_mirror_up        0
#define Anum_fts_message_response_is_in_sync          1
#define Anum_fts_message_response_is_syncrep_enabled  2
#define Anum_fts_message_response_is_role_mirror      3
#define Anum_fts_message_response_request_retry       4

The segment fills these in GetMirrorStatus (src/backend/replication/gp_replication.c): is_mirror_up (the GP walsender is catching-up with a valid write LSN, or streaming), is_in_sync (up and streaming → catalog mode='s'), is_syncrep_enabled (SYNC_STANDBY_DEFINED set), is_role_mirror (the segment thinks it is a mirror — used to detect a lost PROMOTE), and request_retry (mirror down but still inside the grace period → don’t mark it down yet). The probe also does an O_DIRECT read+write disk check (checkIODataDirectory), so a wedged disk fails the PROBE and triggers failover.

20.5.3 The non-blocking probe cycle

FtsWalRepMessageSegments() builds one fts_segment_info per primary-mirror pair and drives all of them through a single poll() set — one async libpq connection per segment, not a thread or blocking call per segment — so one process probes N segments concurrently with bounded latency:

src/backend/fts/ftsprobe.c:1305

while (!allDone(&context) && FtsIsActive())
{
    ftsConnect(&context);    /* PQconnectStart / PQconnectPoll (async)   */
    ftsPoll(&context);       /* ONE poll(PollFds, nfds, 50ms) for all     */
    ftsSend(&context);       /* PQsendQuery the message when POLLOUT      */
    ftsReceive(&context);    /* parse 5x1 response, advance state         */
    processRetry(&context);
    is_updated |= processResponse(&context);  /* may flip roles in catalog */
}

ftsCheckTimeout is a wall-clock guard independent of poll: if a segment hasn’t reached success and now - startTime > gp_fts_probe_timeout, it is forced to the failed state — so a TCP-level hang can’t stall the whole cycle.

20.5.4 The per-segment state machine

Each segment is a small state machine (states in src/include/postmaster/ftsprobe.h): the three action states FTS_{PROBE,SYNCREP_OFF,PROMOTE}_SEGMENT, their _RETRY_WAIT, _SUCCESS, _FAILED siblings, and the terminal FTS_RESPONSE_PROCESSED. A failure is never acted on before retries are exhausted — processResponse asserts AssertImply(IsFtsMessageStateFailed(state), retry_count == gp_fts_probe_retries) (ftsprobe.c:999). The interesting escalations:

• Probe succeeds, mirror down, syncrep enabled, not a grace-period retry → mark the mirror down in the catalog, then FTS_SYNCREP_OFF_SEGMENT to unblock commits stuck in SyncRepWaitForLSN.
• Probe succeeds but reports isRoleMirror → a previous PROMOTE never took effect → re-send FTS_PROMOTE_SEGMENT.
• Probe fails, not a normal restart, mirror is in sync → flip roles in the catalog (dead primary → m/down, mirror → p), swap the pointers, FTS_PROMOTE_SEGMENT.
• Probe fails, mirror NOT in sync → double fault: log FTS double fault detected, do not promote, content goes unavailable.
• Soft retry: a successful probe with request_retry && mirror alive loops through RETRY_WAIT — the grace period is enforced segment-side (is_probe_retry_needed), not in the prober.

20.5.5 What gets written to the catalog, and the version bump

The catalog write is probeWalRepUpdateConfig() (src/backend/fts/fts.c:176): in one transaction it inserts an audit row into gp_configuration_history and flips three columns of the segment’s gp_segment_configuration tuple — role (p/m), status (u/d), mode (s/n). It also updates the shared-memory status bitmap ftsProbeInfo->status[dbid] read on the dispatch hot path. When anything changed, FtsLoop does ftsProbeInfo->status_version++; every transaction start compares the cached fts_version against it (getFtsVersion()), and on mismatch cdbcomponent_updateCdbComponents() rebuilds the component cache so QD/QE re-read roles and route around the dead primary on their next statement.

20.5.6 Mirror promotion, end to end

Crucially, the catalog is updated before the PROMOTE message is sent (ftsprobe.c:1134), so the dispatcher stops using the dead primary even while promotion is in flight. The mirror handles FTS_MSG_PROMOTE in HandleFtsWalRepPromote() (src/backend/fts/ftsmessagehandler.c:376) — idempotent, acting only if it is in DB_IN_ARCHIVE_RECOVERY: it UnsetSyncStandbysDefined(), creates the internal replication slot (so a later pg_rewind of the old primary works), and calls SignalPromote(), which writes PROMOTE_SIGNAL_FILE and signals the postmaster; the startup process finishes recovery and the mirror comes up writable.

This promote-on-mirror-down path is exactly what the regression fix “ensure FTS detects mirror down” hardened — the probe must reliably observe is_mirror_up = false for the sync-rep-off / failover logic to fire deterministically.

20.5.7 Standalone gpfts + etcd

Cloudberry also ships a standalone FTS daemon (src/bin/gpfts/) that runs the same probe state machine and message protocol. The differences are where leadership comes from and where config lives: gpfts must win a distributed etcd lock before probing (and a lease-renewal thread exits the process if renewal fails, so a partitioned FTS yields leadership), and it serializes the whole configuration into etcd instead of writing gp_segment_configuration + bumping an in-SHMEM version. Consumers read config back from etcd. The decision logic is shared; only the deployment and storage differ.

A Failure, End to End

20.6.1 The scenario: a primary segment’s host fails under load

With the four pillars in place, here is how they interlock when one segment’s host dies in the middle of a workload. The seconds in the timeline are illustrative, but every transition and mechanism is real and cited to the section that established it.

20.6.2 Phase by phase

t0 — the fault — The host of primary seg2 stops responding — power loss, kernel panic, or a wedged disk that will now fail the probe’s O_DIRECT read/write check (§20.5.2). In-flight queries touching seg2 error out; new commits there cannot complete.

t0–t5s — detection — FTS’s next probe to seg2 gets no answer, so the per-segment state machine goes PROBE → FTS_PROBE_FAILED and then loops through RETRY_WAIT up to gp_fts_probe_retries times with ~1s waits — a transient blip is never acted on (§20.5.3, §20.5.4). A wall-clock gp_fts_probe_timeout guard stops a TCP hang from stalling the whole cycle.

decision — promote, not double-fault — Retries are exhausted, this is not a normal restart, and seg2’s mirror reports in-sync. FTS therefore chooses failover rather than declaring a double fault (§20.5.4).

catalog first — FTS writes the catalog before sending PROMOTE: the dead primary becomes role m / status down, the mirror becomes role p, and status_version is bumped (§20.5.5, §20.5.6). So the dispatcher stops using the dead primary even while promotion is still in flight.

promotion — FTS sends FTS_MSG_PROMOTE; the mirror’s HandleFtsWalRepPromote unsets synchronous standbys, creates the internal replication slot (so the old primary can later be pg_rewind-ed), and calls SignalPromote. Its startup process finishes recovery and the mirror opens writable (§20.5.6).

recovery of the new primary — Finishing recovery runs RecoverPreparedTransactions, reloading any phase-1 prepared 2PC the segment held and re-cracking the distributed xid from the GID (§20.4.5). Cluster-wide, the coordinator’s dtx recovery worker reconciles in-doubt distributed transactions — committing those known committed, aborting the rest (§20.4.4).

reroute — On its next statement each QD/QE compares its cached fts_version to the bumped value, sees the mismatch, and cdbcomponent_updateCdbComponents() rebuilds the component cache so work routes to the promoted segment (§20.5.5).

repair — Later an operator runs gprecoverseg: incremental (pg_rewind, copying only diverged blocks) or full (pg_basebackup). The rebuilt seg2 comes up as the new mirror in standby.signal mode, catches up by streaming replication, and FTS marks it up (§20.4.7).

20.6.3 Why no committed transaction is lost

The failover loses no acknowledged commit because synchronous replication (§20.3.3) had already advanced the mirror’s flush LSN past every committed transaction before the client was told “committed.” The promoted mirror therefore already holds every acknowledged change; recovery only has to finish replaying and resolve in-doubt 2PC (§20.4.4, §20.4.5). What the cluster trades away is availability during the detection window — bounded by gp_fts_probe_timeout × gp_fts_probe_retries — not durability.

Had the mirror not been in sync at the instant of failure, FTS would log a double fault and leave that content unavailable rather than promote a stale copy (§20.5.4) — availability is deliberately sacrificed so the cluster never serves data that was lost.

Quick Reference

20.7.1 FTS and recovery GUCs

Defaults from src/backend/utils/misc/guc_gp.c; see §20.5.1 for how the probe loop uses them.

Fault-detection and WAL/recovery tunables.

GUC	Default	Range	Role
gp_fts_probe_interval	60s	10–3600	Period of a full probe cycle (§20.5.1).
gp_fts_probe_timeout	20s	0–3600	Max wait for one segment’s response before it is treated as failed.
gp_fts_probe_retries	5	0–100	Retries before a failure is acted on.
gp_fts_mark_mirror_down_grace_period	30s	—	Grace window after a mirror disconnect before it may be marked down.
gp_fts_replication_attempt_count	10	—	Walsender reconnect attempts after which a stale disconnect timestamp is ignored.
wal_segment_size	64MB	1MB–1GB	WAL segment file size — CBDB default vs PostgreSQL’s 16MB; fixed at initdb (§20.2.4).
synchronous_standby_names	* on segments	—	Which standbys must ack; FTS clears it (to an empty string) to unblock commits when a mirror dies (§20.3.4).

20.7.2 Transaction WAL opcodes (RM_XACT_ID)

The nine RM_XACT_ID info-byte opcodes (§20.2.7).

Opcode	Value	Meaning
XLOG_XACT_COMMIT	0x00	Local commit (§20.2.5).
XLOG_XACT_PREPARE	0x10	2PC phase-1 prepare.
XLOG_XACT_ABORT	0x20	Local abort.
XLOG_XACT_COMMIT_PREPARED	0x30	Commit of a prepared transaction.
XLOG_XACT_ABORT_PREPARED	0x40	Abort of a prepared transaction.
XLOG_XACT_ASSIGNMENT	0x50	Sub-xid → top-xid assignment.
XLOG_XACT_INVALIDATIONS	0x60	Catalog cache-invalidation messages.
XLOG_XACT_DISTRIBUTED_COMMIT	0x70	Cloudberry distributed commit; carries the gxid (§20.2.7).
XLOG_XACT_DISTRIBUTED_FORGET	0x80	Cloudberry distributed forget; 2PC phase-2 complete (§20.2.8).

20.7.3 FTS probe response fields

The fixed 5-boolean probe response, filled by GetMirrorStatus (§20.5.2).

Field (Anum)	Set when
is_mirror_up (0)	The GP walsender is catching-up with a valid write LSN, or streaming.
is_in_sync (1)	Mirror up and streaming → catalog mode='s'.
is_syncrep_enabled (2)	SYNC_STANDBY_DEFINED is set on the primary.
is_role_mirror (3)	The segment believes it is a mirror — used to detect a lost PROMOTE.
request_retry (4)	Mirror down but still inside the grace period — do not mark it down yet.

20.7.4 Control-file database states (DBState)

ControlFile->state values branched on by StartupXLOG (§20.4.1).

State	Meaning at startup
DB_SHUTDOWNED	Clean stop — redo usually skipped.
DB_SHUTDOWNED_IN_RECOVERY	Standby cleanly stopped.
DB_SHUTDOWNING	Shutdown was interrupted → treated as crash.
DB_IN_CRASH_RECOVERY	Crashed while already recovering.
DB_IN_ARCHIVE_RECOVERY	Crashed during archive / standby redo.
DB_IN_PRODUCTION	Crashed while live — the common case.