Every organization with a large file archive faces the same problem: detecting what changed and acting on it. New documents need OCR. New images need classification. Moved folders need their metadata updated. The traditional answers — inotify, polling, watchdog scripts — all break down at scale, especially on network-attached storage where inotify doesn’t function at all and polling costs O(total files) every cycle regardless of how little actually changed. Meanwhile, database Change Data Capture has been a solved problem for years. Tools like Debezium tail transaction logs and emit structured change events with guaranteed delivery, crash recovery, and zero missed writes. Filesystems have had no equivalent — until you realize that ZFS has been a CDC system all along. Its copy-on-write block pointer tree records the birth transaction of every block, making zfs diff between two snapshots an O(changes) operation that survives reboots, catches offline modifications, and costs nothing to set up beyond the snapshots you should already be taking.
The Real Problem: AI Processing Without Disrupting Production
Before diving into the technical mechanism, it helps to understand the use case that drove us to this architecture.
We manage a document archiving system for a media production company. Thousands of project folders spread across hundreds of terabytes of ZFS-backed NAS storage. Each project folder contains compliance documents, contracts, identification photos, release forms — the kind of content that benefits enormously from deep AI processing. OCR extracts searchable text from scanned PDFs. Vision models classify document types and extract structured fields. Face detection matches ID photos to known individuals. Embedding models make everything semantically searchable.
The problem is that this AI processing is computationally expensive. We have GPU-equipped workstations and a Mac Studio with 64GB of unified memory that can run 32-billion-parameter models locally, but during production hours those machines are busy. Editors are cutting video. Producers are ingesting new footage. The network is saturated with 10-gigabit transfers between workstations and storage.
The processing has to happen during off-peak hours — nights and weekends — when the GPUs are idle and the network is quiet. But we also can’t lock files or check documents in and out during the workday. The production team needs uninterrupted access to everything on the NAS at all times. And we certainly can’t afford to rescan 500,000+ files every night just to find the 200 that someone added or moved during the day.
What we needed was a way to know exactly what changed since the last processing run, generate a precise work list, and hand it to our AI pipeline to chew through overnight. No file locking. No full rescans. No disruption to daytime operations.
ZFS snapshot diffing turned out to be the perfect answer.
Why Traditional Change Detection Fails on Network Storage
If your files live on a local ext4 or NTFS volume, inotify (Linux) or ReadDirectoryChangesW (Windows) can give you real-time file change notifications through kernel-level hooks. But this falls apart in several ways that matter for production environments.
First, inotify doesn’t work on network-mounted filesystems. The Linux kernel’s CIFS module has never completed inotify support for SMB shares. If your storage is a NAS accessed over SMB or NFS — and in any media, legal, or healthcare environment, it almost certainly is — inotify simply doesn’t fire. Python’s watchdog library falls back to a polling observer on network mounts, which brings its own problems.
Second, inotify is lossy. It maintains a fixed-size kernel queue. If changes arrive faster than your application drains the queue, events are silently dropped. On a busy production NAS where multiple editors are saving files simultaneously, this happens more often than you’d like. There’s no replay mechanism — once an event is lost, it’s gone.
Third, inotify requires an active daemon. If your watcher crashes, reboots, or simply wasn’t running when changes occurred, you have no way to recover the missed events short of a full rescan. In our environment, the monitoring machine reboots for kernel updates, loses network connectivity during switch maintenance, and occasionally gets commandeered for other tasks. Any approach that requires continuous uptime to guarantee completeness is fragile.
Polling — periodically walking the entire directory tree and comparing file metadata against a database — works universally but scales terribly. A stat() call over SMB requires a network round-trip. Even on 10-gigabit ethernet, you’re looking at 5,000 to 20,000 stat calls per second, putting a 500,000-file scan at 25 to 100 seconds best-case. That’s tolerable if you poll every 30 minutes, but it means you’re doing the same expensive work regardless of whether 10,000 files changed or zero did. On a quiet Sunday, that’s pure waste.
ZFS's Hidden Superpower: Birth Transaction Groups
ZFS was designed from the ground up as a copy-on-write filesystem. When you modify a file, ZFS never overwrites existing data. Instead, it writes new blocks to fresh locations on disk, then atomically updates parent pointers all the way up to the root of the storage pool — a structure called the uberblock. Old blocks remain intact until explicitly freed.
This architecture makes snapshots essentially free. Creating a snapshot is just preserving a reference to the current root block pointer. No data is copied. No I/O is performed beyond writing a small metadata record. You can take snapshots every 15 minutes for years with negligible overhead.
The key insight is in how ZFS tracks when each block was written. Every block pointer in the tree carries a birth transaction group number (birth TXG) — the identifier of the transaction group during which that block was allocated. A snapshot corresponds to a specific TXG. This means ZFS has, embedded in its own data structure, a complete record of which blocks existed at any point in time.
The zfs diff command exploits this. Given two snapshots, it walks the block pointer tree from the root, examining birth TXGs at each level. If an indirect (parent) block’s birth TXG predates the earlier snapshot, then every leaf block beneath it is guaranteed unchanged, and the entire subtree is skipped. The algorithm only descends into branches where at least one block was born between the two snapshot TXGs.
This makes zfs diff an O(changes) operation, not O(total files). If 200 files changed among 500,000, ZFS examines metadata for roughly 200 files worth of tree traversal plus some overhead for the intermediate nodes. The other 499,800 files are never touched.
The output is clean and structured:
M F /tank/projects/PRJ-2847/Documents/release_form.pdf
+ F /tank/projects/PRJ-2847/Documents/id_scan_front.jpg
+ F /tank/projects/PRJ-2847/Documents/id_scan_back.jpg
- F /tank/projects/PRJ-2491/old_contract.pdf
R F /tank/projects/PRJ-2491/draft.pdf -> /tank/projects/PRJ-2491/final.pdfM for modifications, + for additions, - for deletions, R for renames. The -F flag adds file type indicators. The -H flag produces tab-separated output that’s trivial to parse programmatically.
ZFS also provides an even faster pre-check. The written property reports the amount of referenced space written since the most recent snapshot. A quick zfs get -H -o value written tank/projects returns 0 instantly if nothing changed at all, allowing you to skip the diff entirely on quiet periods. During a typical weeknight, many of our dataset volumes show zero writes, and we skip them in milliseconds.
Comparing ZFS Diff to Database CDC
The structural parallel to database Change Data Capture is precise, and recognizing it changes how you think about filesystem monitoring.
In a database, CDC systems like Debezium tail the write-ahead log (WAL). The WAL records every insert, update, and delete with a Log Sequence Number (LSN). A CDC consumer tracks its position in the WAL and can resume from any point, replaying changes it missed. This provides guaranteed delivery, crash recovery, and change-proportional processing.
ZFS’s birth TXGs are the filesystem equivalent of LSNs. Snapshots are the equivalent of database checkpoints. zfs diff is the equivalent of reading the WAL between two checkpoints. And the written property is the equivalent of checking whether the WAL position has advanced at all.
| Database CDC | ZFS Equivalent |
|---|---|
| Log Sequence Number | Birth Transaction Group |
| WAL / Binlog | Block Pointer Tree |
| Checkpoint | Snapshot |
| Debezium connector | zfs diff |
| WAL position check | written property |
| Guaranteed delivery | Atomic snapshot comparison |
| Crash recovery | Retroactive diff between any snapshots |
The one difference is latency. Database CDC can stream changes in near-real-time as transactions commit. ZFS diff is periodic, bounded by your snapshot interval. But for AI processing pipelines that run in batch during off-peak hours, this is not a limitation — it’s a feature. You don’t want real-time change streaming triggering GPU-intensive OCR jobs during production hours. You want a clean, complete changeset ready to process when the workday ends.
The Architecture: Snapshot, Diff, Classify, Process
The pipeline we built has four stages, and the beauty is in how simple each one is.
Stage 1: Snapshot Management. A cron job on the NAS takes snapshots of each dataset on a fixed schedule — every 30 minutes during business hours, hourly overnight. We retain the most recent snapshot from each processing run as our baseline. Old snapshots are destroyed automatically after 48 hours. Total overhead: effectively zero.
Stage 2: Change Extraction. When the off-peak processing window opens (typically 8 PM), a Python script SSHs to the NAS and runs zfs diff -FHt between the baseline snapshot and the current state. The output is parsed into structured change events: file path, change type, file extension, parent project folder, and timestamp. These events are inserted into a PostgreSQL staging table.
Stage 3: Classification. A lightweight classifier examines each change event and determines what processing it needs. This is mostly deterministic — file extension and folder path tell you 90% of what you need to know. A new .pdf in a Documents/ folder gets queued for OCR. A new .jpg in a compliance folder gets queued for ID document detection. A renamed folder triggers a metadata update. Only ambiguous cases get routed to a vision model for classification, and even that is cheap (CLIP-based zero-shot classification runs at 130+ images per second on a mid-range GPU).
Stage 4: Off-Peak Processing. The classified work queue feeds into the AI pipeline. The RTX GPU that spent the day idle at an editor’s desk now runs Whisper for transcription, CLIP for image classification, and a vision-language model for structured OCR on identified documents. The Mac Studio that served chat queries during the day now runs a 7-billion-parameter model for document understanding. Processing continues until the work queue is empty or the next business day begins, whichever comes first.
The critical property is that none of this touches the NAS during production hours in any way that affects users. ZFS snapshots are invisible to SMB clients. The diff runs against snapshot metadata, not live files. When the AI pipeline reads files for processing overnight, it reads from the live filesystem, but there’s no contention because the production team has gone home. And because we know exactly which files changed, we never waste GPU time reprocessing files that haven’t been touched.
What You Skip by Not Rescanning
The efficiency gains are dramatic enough to be worth quantifying.
Our archive contains roughly 500,000 files across 3,200 project folders. A full stat-walk over SMB takes approximately 90 seconds on a good day. The subsequent processing to determine which files are new or modified — comparing against a database of previously processed file metadata — adds another 30 seconds of database queries.
On a typical business day, the production team modifies or adds 100 to 500 files. The zfs diff that captures those changes completes in under 2 seconds. The written property check that determines whether a diff is even necessary takes under 50 milliseconds.
That’s a 60x improvement on active days and an effectively infinite improvement on quiet days (50ms vs 120 seconds for the same result: “nothing changed”).
But the bigger win is in the processing pipeline itself. Without change detection, you have two bad options: reprocess everything (impossibly expensive with AI models), or maintain complex bookkeeping to track what you’ve already processed and what’s changed (fragile, error-prone, and a maintenance burden). ZFS diff gives you a third option: ask the filesystem itself what changed, get a precise answer in seconds, and process only that.
The Agentic Pattern
This architecture fits naturally into the autonomous agent pattern that has gained significant traction in 2026. The core pattern is the same: an always-on system that watches for events, classifies them, and dispatches specialized handlers without human intervention.
In our implementation, the “agent” is simpler than a full conversational AI assistant — it’s a Python service with a PostgreSQL work queue, a set of processing modules, and a scheduling system. But the architectural principles are identical. Events arrive (via ZFS diff). A classifier routes them to appropriate handlers based on content type. Specialized processors (OCR, face detection, embedding generation) execute against the work queue. Results are written back to the database. The system maintains memory of what it has processed and skips duplicates.
The skills-based decomposition is what makes it maintainable. Adding support for a new document type means writing a new processor module and adding a classification rule. The change detection layer, the work queue, and the scheduling system don’t change. When we added vision-language OCR for handwritten forms last month, it was a single new Python file and a two-line addition to the classifier. The ZFS diff pipeline didn’t know or care.
Practical Considerations
A few things we learned deploying this in production.
Snapshot naming matters. Use a consistent naming convention like @cdc-YYYYMMDD-HHMMSS so your scripts can find the most recent baseline snapshot programmatically. Avoid names that conflict with your backup snapshot schedule.
The written property isn’t perfect. It can report non-zero values from internal ZFS metadata changes (encryption operations, extended attribute updates) even when no user-visible files changed. Treat it as a fast negative test: if written is zero, skip the diff. If it’s non-zero, run the diff, but be prepared for it to sometimes return an empty changeset.
QNAP QuTS hero (and TrueNAS, and any ZFS-based NAS) supports this natively. QuTS hero is built on ZFS, so zfs diff works out of the box via SSH. You don’t need any special software or configuration beyond enabling SSH access and setting up snapshot schedules, which you should have for backup purposes anyway.
Handle renames gracefully. ZFS diff reports renames as a single R event with both old and new paths, which is much cleaner than the correlated delete-plus-create events you’d get from inotify. Your processing pipeline should treat renames as metadata updates rather than triggering a full reprocess of the file.
Deleted files need attention too. When a file disappears from the NAS, your search index, compliance database, and embedding store need to be updated. ZFS diff’s - events give you an explicit, reliable signal for cleanup that polling-based approaches often miss or handle poorly.
The Unfilled Gap
Nobody has built a “filesystem Debezium.” There’s no widely adopted framework that publishes ZFS diffs as structured events to a message bus, provides consumer group semantics for multiple downstream processors, and handles offset tracking and replay. The closest analogs — BeeGFS modification events, NTFS USN Journal consumers, Apache NiFi’s ListFile processor — are either proprietary, platform-specific, or crude approximations.
This is a gap worth filling. ZFS is deployed everywhere that large file archives live: media production houses, research universities running petabyte storage pools, hospitals with PACS imaging archives, legal firms with document management systems. AWS offers managed ZFS through FSx for OpenZFS. TrueNAS markets to media, research, and AI workloads explicitly. QNAP and Synology ship ZFS-based operating systems. The storage layer is already there.
What’s missing is the connective tissue between ZFS’s native change detection capability and the AI processing pipelines that organizations are building on top of their archives. The architecture isn’t complex — we built ours in a few hundred lines of Python — but formalizing it as a pattern and eventually as a reusable framework would save a lot of people from reinventing the same inotify-based solutions that will inevitably fail them at scale.
Conclusion
ZFS’s block pointer tree with birth transaction groups is not an implementation detail. It’s the structural equivalent of a database transaction log, and it makes filesystem Change Data Capture a first-class concept rather than a hack bolted onto the side. For any organization running AI processing against a large file archive — and that includes most media companies, law firms, healthcare providers, and research institutions — this changes the calculus of what’s practical.
You don’t need real-time filesystem watchers that crash and lose events. You don’t need polling scripts that burn cycles checking files that haven’t changed. You don’t need file locking or check-in/check-out systems that slow down your production team. You need two snapshots and a diff, and you probably already have the infrastructure to do it.
The GPUs sitting idle at your editors’ desks tonight are waiting for a work list. ZFS already knows what’s on it.
At Agave IS we build AI-integrated infrastructure for media production and document archiving environments — the kind of systems described here. If you’re running AI processing against a large archive and want to stop rescanning everything every night, let’s talk.