DO-NOT-MERGE: RFC: NVMe-oF orchestrator coexistence — nvme-discoverd, ownership registry, and exclusion list

[martin-belanger]

This PR is for review only and will not be merged. It contains four RFC documents proposing a coordinated set of features for NVMe-oF orchestrator coexistence in nvme-cli 3.0. Please use inline comments to provide feedback on specific sections.

Tip: Markdown files are shown as raw text by default in the diff view. Click the "Display the rich diff" button (the document icon at the top-right of each file) to render them as formatted text — much easier to read.

---

Background

As NVMe-oF deployments grow, hosts increasingly run multiple tools that manage NVMe-oF connections: nvme-stas, dracut/initramfs scripts, and soon nvme-discoverd. Without coordination, these orchestrators can conflict — one may disconnect a controller that another is actively managing, or connect to a controller that another has deliberately excluded.

This RFC set proposes two libnvme building blocks to solve the coexistence problem (ownership registry and exclusion list), and a new daemon orchestrator (nvme-discoverd) that puts them to use.

---

The four RFCs

rfc-nvme-orchestrator-coexistence.md — Start here

The top-level document. Frames the two distinct conflict scenarios (accidental disconnect, accidental connect), introduces the two prevention mechanisms (ownership registry and exclusion list), defines a three-tier orchestrator hierarchy (raw commands → manual orchestrators → daemon orchestrators), and shows how nvme-discoverd and nvme-stas naturally partition work without requiring IPC coupling between them.

rfc-nvme-registry.md — Ownership registry

A lightweight, cooperative registry under /run/nvme/registry/ that lets orchestrators declare ownership of connected controllers. nvme disconnect-all consults the registry before acting so it never disconnects a controller managed by a running daemon. The registry is a libnvme building block: a C API (libnvmf_registry_*), a Python binding, and a nvme registry CLI command family. An implementation PR already exists: #3425.

rfc-nvme-exclusion.md — Exclusion list

A human-administered exclusion list at /etc/nvme/exclusions/ that prevents orchestrators from auto-connecting to controllers the administrator wants excluded. Its design center is auto-discovered controllers (mDNS, CDC DLP) where there is no configuration entry to remove. Managed via nvme exclusion CRUD commands. Enforcement is cooperative — each orchestrator reads the list and skips matching controllers; libnvme does not enforce it.

rfc-nvme-discoverd.md — nvme-discoverd

A new daemon proposed for nvme-cli 3.0. It manages NVMe-oF connections for statically configured controllers, NBFT boot controllers, FC-discovered controllers, and (in a future release) mDNS-discovered controllers. Key design choices:

• Connect-only by design — never issues disconnects; eliminates the entire connect/disconnect ordering complexity (CtrlTerminator problem). TP8010 fabric zoning is out of scope; that belongs to nvme-stas.
• One systemd transient unit per controller — nvme-discoverd never calls libnvmf_connect() directly; blocking /dev/nvme-fabrics writes happen in child processes managed by systemd. Achieves Daniel's no-threading goal.
• Registers ownership via --owner discoverd on each generated unit; NBFT reconnects use --owner nbft to preserve the lifetime invariant set at boot.
• Respects the exclusion list before each connect and reconnect; NBFT controllers bypass the check unconditionally.
• Coexists with nvme-stas through natural discovery partitioning: nvme-stas owns mDNS/TCP, nvme-discoverd owns NBFT/FC/RDMA and manual config. The registry-check rule in nvme-stas eliminates bounce loops without any IPC coupling.

---

What feedback is sought

• Architecture: Does the three-tier orchestrator model match how you see these tools being used in practice?
• Ownership registry: Any concerns with the directory layout, API shape, or cooperative-only enforcement model?
• Exclusion list: File format, use cases, nvme exclusion command design?
• nvme-discoverd: Transient unit design, state file layout, startup reconciliation logic, retry policy, NBFT handling?
• Coexistence rules: The owner=nbft lifetime invariant, NBFT exclusion bypass, unowned-connections-are-fair-game semantics — do these cover the cases you care about?

---

Related

• Implementation PR for the ownership registry: libnvmf: NVMe Controller Ownership Registry (V2) #3425 (branch registry-v2)
• Previous registry discussion: Prevent NVMe-oF boot devices from being removed by disconnect-all #2913
• Feature tracking issue: libnvme/nvme: add NVMe controller ownership registry #3433

[tbzatek]

Well written design documents, I appreciate the level of detail!

Question about the legacy nvme connect-all --nbft: still being used by dracut, so I guess it stays, yet discoverd reimplements some logic. Would be nice to unify the codebase to avoid duplication. My concern is interpretation of various NBFT flags and processes... although certain best practices got added in the Boot Spec, there are still gaps. There have always been disparity between various UEFI implementations and the OS, even though we generally tend to avoid adding quirks in the nvme-cli NBFT code for firmware bugs.

[tbzatek]

The plan is to always supply owner information, e.g. 'udisks', with a possibility to override this via an optional argument to any arbitrary string.

Since UDisks is rather high-level layer, we're able to respect exclusions actively and refuse connection, unless forced. At this point only simple connect & disconnect commands are provided, falling into Tier 1. However there were plans to provide simple discovery incl. mDNS support browser functionality in the future, leaning towards Tier 2 (as there will likely always be an external consumer to trigger any action first).

[martin-belanger]

The plan is to always supply owner information, e.g. 'udisks', ...

Thanks — I'll update the UDisks entry accordingly: supplies owner='udisks' (overridable to an arbitrary string), actively respects exclusions and refuses unless forced, Tier 1 today. I'll add this to the RFC.

However there were plans to provide simple discovery incl. mDNS support browser functionality in the future...

On the future mDNS direction, though, I'd like to clarify before we pencil in a Tier 2 / mDNS-browser role for UDisks — because mDNS/DNS-SD discovery of Discovery Controllers is a genuine can of worms, and a third independent browser would compound it. We're already wrestling with how to coordinate mDNS browsing between just two orchestrators, nvme-discoverd and nvme-stas: having both browse at once is a misconfiguration we have to actively detect and arbitrate (the "zeroconf-conflict" problem, still unsolved in the general case). If UDisks adds its own mDNS browser, that's a third layer independently discovering the same DCs and potentially connecting, with no shared owner of "who browses mDNS on this host." Could you say more about what UDisks intends — a full mDNS DC-discovery browser, or something narrower (e.g. surfacing controllers another component already discovered)? If it's the former, I think we need a cross-orchestrator story for mDNS-browser ownership before multiple layers start doing it independently — two is already hard, and three concurrent browsers on one host would be painful to reason about. I'd rather flag it now than debug three browsers fighting later.

[tbzatek]

Generally speaking, any new party can come to the game anytime.

The plan with UDisks is to provide underlying services for gvfs - it might actually be gvfs simply adding another protocol to its existing avahi-based network browser. UDisks would then provide high-level discovery and connect services, guarded by polkit rules. This is all on demand and it's tiered - i.e. the base gvfsd-network browser would just report machines in the network exposing _nvme-disc._tcp and discovery is only made upon user intention - i.e. discovery is not supposed to be done proactively. The target audience are end users with expectation that neither nvme-discoverd or nvme-stas are going to be configured for zeroconf on such systems. Neither gvfs or UDisks can use discoverd as that's systemd-only, this will need to be separate reimplementation, reusing as many general parts from libnvme as possible.

[tbzatek]

We need this for NBFT testing... although the design is well suited for real-world scenarios, we'd like to have option to manually disconnect a particular nbft controller and prevent it to be reconnected.

Further thoughts:

• explicitly stop nvme-discoverd for such tests?
• what about a multipath scenario where a particular network interface is intentionally going down: perform disconnect first and then tear down the interface to avoid the kernel host nvme driver reconnection mechanism?

[martin-belanger]

Good — there's a clean answer to all three.

Manually disconnect a particular NBFT controller, and keep it disconnected. There's a single command for exactly this: nvme disconnect --exclude <device>. It derives an exclusion entry from the controller's sysfs attributes (transport, traddr, trsvcid, subsystem NQN — plus host-iface when the connection is pinned to one), writes that entry to the exclusion list first, and only then disconnects. Writing the exclusion before the disconnect is deliberate: it guarantees the entry is in place before the device-removal event ever reaches discoverd, so discoverd gets no window to reconnect. That's the surgical, race-free option — it suppresses just that one controller while discoverd keeps managing everything else.

This works because we're relaxing the previous "NBFT bypasses exclusion" rule: discoverd now honours the exclusion list for NBFT controllers too. (owner=nbft still protects the controller from other orchestrators — only discoverd's own reconnect yields to your explicit exclusion. And since excluding a boot device is a foot-cannon, the nvme exclusion add path warns when an entry would match an owner=nbft controller, skippable with --force.) If you instead want to disconnect without excluding — i.e. let discoverd bring it straight back — plain nvme disconnect <device> is the no-checks single-device escape hatch with no exclusion side effect.

Stopping discoverd for tests? You can, and for a quick "stop managing everything" it's fine. But note it's the blunt instrument: discoverd is the single monitor for every connection on every interface, so stopping it suspends management of all paths, not just the controller under test. If you want to isolate one controller while the rest stay managed, the exclusion list above is the better lever.

The multipath interface-down case. Here I'm less sure there's a problem to solve, so let me lay out what actually happens and you can tell me if you're after something more. When a local interface is brought down under a live controller, the kernel doesn't special-case it — it enters its normal reconnect loop, which is slow (one attempt per reconnect-delay, default 10 s) and, when the connection is pinned with --host-iface, fails fast each time. It doesn't flood anything, and multipath has already failed I/O over to the live paths; when the interface returns, the path reconnects. discoverd, being connect-only, keeps that controller in its desired set and reconnects on return — the intended behaviour. So "disconnect first, then ifdown" is a reasonable manual hygiene step to avoid the (harmless, slow) reconnect churn, but it isn't something discoverd needs to orchestrate. If your goal is to permanently retire a path, that's a desired-set/config change (discoverd should stop wanting it) — and again the exclusion list is the tool. If you're seeing an actual failure beyond the cosmetic reconnect churn, tell me what it is and I'll dig in; otherwise I'd rather not build orderly single-path drain into a connect-only daemon without a concrete problem that needs it.

[tbzatek]

discoverd now honours the exclusion list for NBFT controllers too.``

That's great to hear, thanks.

The multipath interface-down case.

This was mentiond as just another test case we've been performing, to test the link-up reconnection hooks. I.e. avoiding the kernel connection recovery mechanism and have a clean plate. This is obviously supposed to be handled by discoverd now, we'll align our tests accordingly. There are some more extreme use cases like in-place upgrades with intermediary userspace that is performing dark magic... unimportant for now I'd say.

[tbzatek]

Not sure whether this ever served its purpose... @mwilck ?

[martin-belanger]

I suspect you're right that it never really served a purpose, and the reason fits the rest of the picture. This rule does one thing — pin nbft* interface names so udev doesn't rename them — and the only thing that ever consumed that name was the NM dispatcher (80-nvmf-connect-nbft.sh), which matches nbft* to trigger the late-NBFT connect — i.e. the very path you just described as never really working. So with no working consumer, the naming rule has nothing to serve. I currently list it as "Kept" (orthogonal naming), but I'm inclined to move it to "Remove" alongside the NM-dispatcher path: if the late-NBFT machinery goes, this goes with it. I'll defer the definitive history to @mwilck — he'd know if anything else ever relied on the nbft* name — and tie the final disposition to the broader question of whether discoverd carries NBFT handling at all.