WAL, Replication Slots & AWS DMS: A PostgreSQL Deep Dive

If you run change data capture (CDC) out of PostgreSQL with AWS DMS, one component quietly holds the fate of your source database in its hands: the logical replication slot. Understand how the write-ahead log and slots interact, and you understand why disks fill, why CDC stalls, and how to fix it.

TL;DR

• The WAL is PostgreSQL’s cluster-wide, physical redo log. Every change is written there first.
• A replication slot is passive state — a bookmark (restart_lsn) that forces PostgreSQL to keep WAL until the consumer has processed it.
• DMS is the consumer. A walsender + output plugin on the source do the actual decoding; DMS pulls the changes and acknowledges them, which advances the slot.
• If a slot stops advancing (task stopped, or a quiet table in a busy DB), WAL piles up on the source — worst case, the source database runs out of disk and stops accepting writes.
• Multiple slots share one WAL — they don’t duplicate it. Retention is set by the oldest restart_lsn across all slots.

In this article

1. How the write-ahead log works
2. What exactly is retained in WAL
3. Replication slots: bookmarks that pin WAL
4. Who actually does the decoding
5. How AWS DMS uses all of this
6. The impact of retained WAL
7. Multiple slots: is the WAL deduplicated?
8. The quiet-table trap & the heartbeat
9. Hands-on: inspecting slots
10. Dropping a slot (and a puzzling warning)
11. Cheat sheet

1. How the write-ahead log works

WAL (Write-Ahead Log) is PostgreSQL’s durability mechanism. Before any change touches the actual data files, it is first written as a record to the WAL. That is the “write-ahead” rule: log first, apply later. If the server crashes, PostgreSQL replays the WAL to recover every committed transaction.

Mechanically:

• WAL is an append-only sequence of 16 MB segment files in the pg_wal/ directory.
• Every byte position in that sequence has an address called an LSN (Log Sequence Number), for example B/507B9CE8. LSNs only ever move forward.
• WAL isn’t only for crash recovery — it also feeds physical replication (standbys), archiving (point-in-time backups), and logical decoding (which is what DMS uses).

Why WAL normally stays bounded

WAL would grow forever if nothing cleaned it up. Normally PostgreSQL recycles (reuses) or removes old segments once they are no longer needed. A segment is “no longer needed” only once all of these are past it:

• a checkpoint has flushed the corresponding data pages to disk,
• any physical replicas have received it,
• it has been archived (if archiving is enabled), and
• every replication slot has confirmed it no longer needs it.

That last condition is the one your CDC setup controls — and the one that causes trouble.

2. What exactly is retained in WAL

An important distinction first: a slot does not store your table’s changes anywhere. It only holds a pointer that forces PostgreSQL to keep the raw WAL files on disk. So “what is retained” is the entire cluster’s WAL byte-stream from the slot’s restart_lsn up to the current write position.

WAL is a low-level, physical redo log of everything that changes on-disk state. It is not table-organized and it is not SQL — it is an ordered sequence of records, each stamped with an LSN:

• Heap tuple changes — the actual row data: INSERT → the new row image; UPDATE → the new row (plus, depending on the table’s REPLICA IDENTITY, the old key or full old row); DELETE → enough to identify the deleted row
• Index changes caused by those row changes (b-tree, GIN, etc.).
• TOAST data — out-of-line storage for large column values.
• Sequence advances, multixact/CLOG updates, visibility-map and free-space-map changes.
• Transaction control records — commit and abort records (with XID and commit timestamp). Logical decoding uses these to know when and how to emit a transaction.
• Checkpoint records and other cluster-wide bookkeeping.
• DDL effects — schema changes land as system-catalog row changes, which are themselves WAL-logged.

💡 Why WAL is bigger than “just the data”

Full-page images (FPIs): after each checkpoint, the first modification to any 8 KB page writes the entire page into WAL (protection against torn writes), not just the changed bytes. On a busy database this can be the majority of WAL volume — so retained WAL is often far larger than the logical size of what changed.

Two points matter most:

1. It is the whole cluster, not your table. WAL is instance-wide (shared across all databases in the instance). Logical decoding later filters it down to your table, but retention happens on the raw stream. A stalled slot therefore keeps WAL for every table, index, and database in that LSN range. This is why a quiet captured table can still cause huge retention — the bytes being held are mostly other tables’ activity.
2. Long-running transactions extend it too. A slot’s position cannot advance past the start of the oldest transaction that was still open, because decoding must be able to reconstruct that whole transaction. A single long-lived transaction — even one unrelated to your table — can pin WAL back to when it began.

3. Replication slots: bookmarks that pin WAL

A logical replication slot is passive state on disk. It doesn’t run or decode anything by itself. It stores a small set of pointers plus the name of an output plugin:

• restart_lsn — the oldest LSN the slot still needs; PostgreSQL will not delete WAL at or after this. This is what pins WAL.
• confirmed_flush_lsn — how far the consumer has acknowledged.
• catalog_xmin — the oldest transaction id whose catalog rows must be preserved, so decoding can interpret the schema of changes.
• the output plugin (e.g. test_decoding, pgoutput, pglogical) chosen when the slot was created.

The consumer advances restart_lsn by confirming “I’ve safely processed everything up to LSN X.” When the consumer keeps consuming, restart_lsn chases the current write position and old WAL is recycled normally. When the consumer stops confirming — the task is stopped, or the captured table is silent — restart_lsn freezes and every WAL segment generated since then is held.

4. Who actually does the decoding

The slot is a bookmark. The real work is done by a walsender process on the source, using the slot’s output plugin. The division of labor:

How a decode actually happens

1. The consumer connects and starts streaming → the slot becomes active, and a walsender starts.
2. The walsender reads WAL forward from restart_lsn.
3. Because a transaction’s changes are scattered through WAL and interleaved with others, the walsender buffers changes per transaction in a reorder buffer (in memory, spilling to disk past logical_decoding_work_mem) and only emits a transaction once it sees the COMMIT record — in commit order. Aborted transactions are discarded.
4. For each committed change it calls the output plugin, which produces the logical row event, filtered to the tables the consumer wants.
5. The consumer acknowledges; the slot’s pointers advance; WAL behind restart_lsn becomes eligible for recycling.

⚠️ Key consequence

Decoding runs on the source instance and consumes its CPU and memory. It only happens while a consumer is connected (active = t). If the task is stopped, no walsender runs, nothing decodes, restart_lsn does not move, and WAL just piles up.

5. How AWS DMS uses all of this

AWS DMS is the consumer. When a DMS task with a CDC component runs against a PostgreSQL source, it:

1. Uses (or creates) a logical replication slot on the source. DMS uses the test_decoding or pglogical output plugin depending on configuration.
2. Connects on the streaming replication protocol, so a walsender begins decoding WAL from the slot’s position.
3. Receives decoded row changes, writes them to the target (S3, another database, etc.), and sends feedback (“flushed up to LSN X”), which advances confirmed_flush_lsn and eventually restart_lsn.

ℹ️ Slots live on the writer

Logical replication slots exist only on the primary / writer. You cannot create or read a logical slot on a standard read replica. So CDC must connect to the writer — even if you offload a full load to a reader, the CDC stream comes from the writer’s slot.

Full-load-and-CDC vs. CDC-only, and why it matters for WAL

A single full-load-and-cdc DMS task caches changes during the full load and only applies them — advancing the slot — after the full load completes. On a very large table whose full load runs for many hours, the slot’s restart_lsn effectively stalls for that entire window, and WAL accumulates the whole time.

Splitting into a dedicated CDC-only task that consumes continuously from the start keeps restart_lsn advancing, so the source can recycle WAL throughout. That single change is often the difference between a healthy source and a full disk.

6. The impact of retained WAL

The core problem is unbounded disk growth on the source. What that leads to, in order of severity:

1. Source storage fills up. The pinned WAL accumulates in pg_wal/. On a busy database this can be gigabytes per hour.
2. The worst outcome: the source stops accepting writes. PostgreSQL must be able to write WAL to commit anything. If the volume fills, writes fail and the instance can crash or become unrecoverable. This takes down the source, not just DMS — a production outage caused by a downstream consumer.
3. RDS vs. Aurora flavor of the damage: RDS PostgreSQL — pg_wal sits on the instance’s EBS volume; it eats free storage until you hit a DiskFull / storage-full state. Aurora PostgreSQL — no traditional local pg_wal disk, but an unconsumed logical slot still forces the cluster to retain log data, so the cluster volume grows (and you are billed for it), and it can impact the writer.
4. Secondary effects even before it is full: larger backups/snapshots, longer restart/recovery times, checkpoint and I/O pressure, and catalog bloat (via catalog_xmin holding back VACUUM on system catalogs).
5. It does not self-heal. Space is reclaimed only when restart_lsn advances (the consumer resumes) or the slot is dropped. Dropping a slot means CDC cannot resume from where it left off — you would need a re-sync.

7. Multiple slots: is the WAL deduplicated?

Yes — there is exactly one physical WAL stream per instance, shared by everything. Each slot only holds its own restart_lsn pointer into that single stream; the segment files on disk exist once. Slots do not each keep their own copy.

PostgreSQL keeps WAL back to the minimum (oldest) restart_lsn across all slots. Every slot reads from those same shared files. So:

• Two slots at the same restart_lsn cost the same disk as one slot — no multiplication.
• The most-behind slot dictates retention for everyone. WAL that faster slots have already passed still cannot be recycled if one lagging slot still needs it.

Concretely, with:

slot_fast    restart_lsn = B/900   (advancing, near current)
slot_slow    restart_lsn = B/100   (stalled)
current WAL              = B/950

PostgreSQL retains everything from B/100 to B/950 — the slow slot’s position — even though slot_fast only needs from B/900. It is a single retained range governed by the minimum, not the sum.

⚠️ So “more slots = more WAL” is imprecise

Multiple slots are not N copies of the WAL. The real costs of more slots are: (1) more independent pointers → higher chance that some slot stalls and drags min(restart_lsn) backward; (2) each logical slot runs its own walsender decoding the same WAL, so CPU/memory on the source multiplies even though storage does not; (3) each slot’s catalog_xmin can hold back catalog VACUUM.

8. The quiet-table trap & the heartbeat

Here is the subtle failure mode. Because a slot decodes the whole database’s WAL but advances only when the consumer processes changes for its tables, a problem arises when:

your captured table is quiet, while the rest of the database is busy.

Sequence of events:

1. The database’s current WAL position races forward because of other tables’ activity.
2. DMS decodes that whole stream but finds nothing for your table, so it has nothing to write and no new checkpoint to confirm.
3. restart_lsn stays pinned at the last change to your table — possibly hours ago.
4. PostgreSQL cannot recycle any WAL generated since then — including all the busy tables’ WAL — and it accumulates.

So the risk is specifically “my table is silent while the database around it is not.” A useful way to phrase it: the heartbeat matters when your captured set is a low-traffic subset of a high-traffic database.

The DMS WAL heartbeat

DMS offers a heartbeat feature (the heartbeatEnable=Y connection attribute). It periodically writes a tiny transaction to the source so there is always something for DMS to capture and acknowledge, which keeps the slot advancing and lets WAL recycle even when your table is idle.

ℹ️ When is the heartbeat actually needed?

Only for the quiet-table-in-a-busy-DB case:

• Captured table busy → not needed; its own changes advance the slot.
• Whole DB quiet → not needed; little WAL is generated at all.
• Captured table quiet + rest of DB busy → this is the case it exists for.

Two caveats: the heartbeat writes to the source (it needs write permission and creates a heartbeat table), and it only helps while the task is running.

A neat trick: busy tables as a built-in heartbeat

Because the slot advances on any captured change across the task’s table set, putting a busy table in the same CDC task as quiet tables makes the busy table act as a natural heartbeat — its steady stream of changes keeps restart_lsn moving, and the quiet tables ride along. No extra permissions required.

But mixing has trade-offs: one CDC task is a single apply pipeline, so a hot table can add latency (head-of-line) for the quiet ones, and a table-level error can affect the whole task. And it only works while the busy table is actually busy — a shared quiet window (overnight, maintenance) stalls the slot again. Isolating one big table into its own slot is great for throughput and blast radius, but it gives up this free heartbeat.

Other mitigations

• Keep the slot consuming — the real fix (a continuous CDC-only task).
• Drop truly unused slots — an orphaned/inactive slot pins WAL forever.
• max_slot_wal_keep_size (PG13+) — caps how much WAL a slot may pin; past the cap PostgreSQL invalidates the slot to protect itself. A source-side safety valve — but the CDC task then fails and needs a fresh start.
• Monitor pg_replication_slots retained bytes and the storage / TransactionLogsDiskUsage metrics.

9. Hands-on: inspecting slots

Everything you need is in the pg_replication_slots view. The one query to keep handy:

SELECT slot_name, restart_lsn, confirmed_flush_lsn, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
FROM pg_replication_slots;

The retained column is the distance backward from the current write position to each slot’s restart_lsn. So a larger retained means an older restart_lsn (further behind “now”).

Reading an example

                           slot_name                            | active | retained
----------------------------------------------------------------+--------+----------
 cdc_ledger_slot                                                | f      | 772 MB
 ck6bhogpx5g33a5r_00016401_ca6b50c5_9469_4408_bb7d_8280a5e3fe01 | f      | 534 MB
 kecvi2i4qbc2xlxt_00016401_e4db94ab_ee18_4019_a990_d0dbd4c23b65 | f      | 534 MB
 ld5kwyx7mre33keo_00016401_a9052b4d_0a1f_4b1f_ba76_810b3a51b1ae | f      | 534 MB
(4 rows)

How to read this:

• Oldest slot: cdc_ledger_slot at 772 MB — the largest gap, so the oldest restart_lsn. Since PostgreSQL retains WAL back to the minimum restart_lsn across all slots, this slot is currently governing retention for the whole instance.
• The three hashed names are DMS auto-generated slots (created when a task started without an explicit slotName).
• All are active = f (inactive) — no consumer is connected to any of them, so none is advancing. Inactive slots that never get consumed are the classic silent WAL leak.

Why do the three show the same size?

pg_current_wal_lsn() is evaluated once for the whole query, so it is the same “now” for every row. With that fixed, equal retained implies equal restart_lsn — those three slots are pinned at the same WAL position. That typically means they were created at essentially the same moment and none has consumed anything since (their restart_lsn is frozen at their creation LSN).

✅ Tip

pg_size_pretty rounds, so “534 MB” could hide a few MB of difference. To see exact positions, look at the raw LSNs:

SELECT slot_name, restart_lsn, confirmed_flush_lsn, active,
       pg_current_wal_lsn() AS current_lsn
FROM pg_replication_slots
ORDER BY restart_lsn;   -- oldest restart_lsn first

Just the restart_lsn

restart_lsn is a plain column (a pg_lsn value like B/507B9CE8):

-- One specific slot
SELECT slot_name, restart_lsn, confirmed_flush_lsn, active, wal_status
FROM pg_replication_slots
WHERE slot_name = 'cdc_ledger_slot';
-- The instance-wide oldest restart_lsn (the one governing retention)
SELECT min(restart_lsn) FROM pg_replication_slots;

wal_status (PG13+) is worth watching: reserved / extended / unreserved / lost. A value of lost means WAL the slot needed was already removed — the slot is now unusable.

ℹ️ Where to run these

Run on the writer/primary (logical slots live there). Reading pg_replication_slots needs sufficient privileges (superuser / rds_superuser / a role with pg_read_all_stats or replication).

10. Dropping a slot (and a puzzling warning)

An inactive slot (active = f) can be dropped directly. Run on the writer, as a user with replication privilege (or rds_replication):

SELECT pg_drop_replication_slot('ld5kwyx7mre33keo_00016401_a9052b4d_0a1f_4b1f_ba76_810b3a51b1ae');

⚠️ This is destructive and irreversible

Dropping a slot discards its restart_lsn, so any DMS task meant to resume from it no longer can — it would need a fresh slot and a new start point (typically a full reload). Only drop slots whose tasks are genuinely abandoned. Dropping the oldest slot is what actually releases the most WAL.

If a slot is active

pg_drop_replication_slot errors with replication slot "..." is active for PID N. Stop the consuming task first; if it still holds on, terminate the backend, then retry:

SELECT pg_terminate_backend(active_pid)
FROM pg_replication_slots
WHERE slot_name = 'your_slot_name';

The “could not open directory pg_replslot/…” warning

You may see this when dropping a slot:

WARNING:  could not open directory "pg_replslot/some_slot": No such file or directory
WARNING:  could not remove directory "pg_replslot/some_slot"
 pg_drop_replication_slot
--------------------------

The drop still succeeded. pg_drop_replication_slot returns void, so the blank value under the header is the normal “success, nothing to return” output. There was no ERROR — only WARNINGs, which do not abort the operation.

Every slot has an on-disk directory pg_replslot/<slot_name>/ holding its state and any reorder-buffer spill files. Dropping a slot (a) removes its in-memory/registered entry — the thing that shows in pg_replication_slots — and (b) deletes that directory. The warnings say step (b) found the directory already gone, so there was nothing to remove; PostgreSQL logs a warning but still completes step (a), which is what actually drops the slot.

Why would the directory be missing while the slot record existed? Common causes:

• Aurora / RDS managed environment. On Aurora PostgreSQL especially, slot storage is handled differently from community PostgreSQL, and these benign “could not open/remove directory” warnings on drop are a known artifact.
• A prior failover or restart. The on-disk directory can be cleaned while the slot metadata persists, leaving exactly this mismatch — plausible whenever the instance has failed over or restarted (and consistent with the slot being inactive).

Either way it is cosmetic. Confirm the slot is gone:

SELECT slot_name, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
FROM pg_replication_slots;

Once the oldest slot is gone, the new instance-wide min(restart_lsn) takes over, and WAL below it becomes eligible for recycling at the next checkpoint (space is reclaimed on checkpoint, not instantly).

11. Cheat sheet

The one-paragraph mental model

WAL is one shared, cluster-wide redo log. A replication slot is a bookmark that says “don’t delete WAL past here until I’ve processed it.” A walsender decodes that WAL on demand and streams changes to DMS, which acknowledges them to move the bookmark forward. Keep the bookmark moving and everything is healthy; let it freeze — a stopped task, a quiet table in a busy database, an orphaned slot — and the source’s disk fills with WAL it can never reclaim on its own. Watch restart_lsn, and you watch the health of the whole pipeline.