User-space storage I/O frameworks

Research date: 2026-10-09

This report selects 21 GitHub repositories that provide reusable mechanisms for issuing, scheduling, translating, or serving storage I/O from user space. It covers direct NVMe access, asynchronous file I/O, block-device targets and caching, filesystem interfaces, and remote-storage client engines. “User-space” includes frameworks that cooperate with kernel drivers; it does not imply kernel bypass. Seastar is included specifically for its storage subsystem, and BaM for its GPU-initiated storage framework. Complete databases and distributed filesystems, benchmark-only tools, and generic networking runtimes are outside this selection.

The criteria identify engineering material worth studying, not a guarantee that every component is correct or equally exemplary:

C1 — Difficult correctness: meaningful invariants involving concurrency, resource ownership, protocol semantics, adversarial inputs, or failure recovery.
C2 — Reusable abstractions: substantial interfaces or components supporting multiple storage backends, applications, or execution models.
C3 — Performance with structure: explicit handling of storage costs through understandable queueing, batching, scheduling, caching, or memory-management architecture.
C4 — Sustained evolution: evidence of development across years together with compatibility work, testing, or complexity management. Repository age and recent activity alone do not establish this criterion.

Direct device access and NVMe interfaces

1. spdk/spdk

Language/role: C; user-space storage drivers and composable block-storage services.

SPDK is a particularly rich study of how device access, execution ownership, and virtual block devices fit together. Its abstraction boundaries remain visible even when the data path avoids conventional kernel I/O.

C1: An SPDK thread can move between operating-system threads, but must never execute concurrently on two of them. Pollers and per-thread I/O channels impose lifetime and teardown constraints; message passing replaces many shared-state operations. These are explicit correctness contracts in the concurrency design.
C2: The block-device layer gives physical drivers and stacked virtual devices a common interface, supporting combinations such as logical volumes, RAID, and encryption rather than requiring applications to understand each device directly.
C3: Per-thread channels, polling, and message queues explain where contention and scheduling overhead are removed. Study the concurrency design alongside the bdev documentation to connect that execution model to actual storage composition.

2. xnvme/xnvme

Language/role: C; a common NVMe-oriented I/O API over multiple operating-system and user-space backends.

xNVMe is useful for understanding portability without hiding completion-oriented device behavior. Its repository documents backends including Linux interfaces and SPDK, allowing an application to retain a common command and queue vocabulary while changing how commands reach storage.

C1: The C examples show command-context ownership, separate submission and completion error checks, queue-full handling through completion processing and retry, and draining outstanding work before releasing buffers and the queue.
C2: Device, command-context, buffer, and queue abstractions separate application logic from backend selection. This is more substantial than a single-system-call wrapper: memory allocation and asynchronous completion also participate in the interface.
C3: The examples expose queue capacity, explicit completion polling, and asynchronous batching. An engineer can study exactly how backpressure and completion progress affect a portable storage loop.

The examples are versioned published documentation; check the repository when matching them to a newer checkout.

3. SamsungDS/libvfn

Language/role: C; VFIO-based user-space PCIe/NVMe access, including DMA mapping and command-queue primitives.

libvfn is a smaller, more direct counterpart to a full storage service framework. It exposes the machinery needed to build a custom device engine, while also providing conveniences for issuing and polling requests. Use the canonical SamsungDS repository rather than historical organization names.

C1: The performance example implementation makes DMA address mapping, request acquisition/release, outstanding-request accounting, and draining visible. Buffer mappings and requests must remain valid across submission and completion.
C2: The repository separates low-level register/queue access from higher-level command submission utilities, leaving room for applications with different polling and execution arrangements.
C3: The example batches submission-tail doorbell updates and advances the completion head after reaping a group of completions. This gives a concrete study of amortizing device notifications without concealing queue state behind a large runtime.

4. linux-nvme/libnvme

Language/role: C; Linux NVMe command, device-discovery, and management library.

libnvme belongs here as a kernel-mediated user-space device interface. It is valuable when studying specification-level command construction and completion semantics, rather than replacing the kernel NVMe driver.

C1: The I/O API documentation distinguishes NVMe completion status from negative operating-system errors, controller handles from namespace handles, and synchronous execution from asynchronous queue use. It explicitly warns about mixing synchronous and asynchronous operations on a shared handle.
C2: The repository provides reusable NVMe structures and command APIs together with device discovery and management. Applications need not independently reproduce command layouts, namespace handling, or result interpretation.
C3: The asynchronous interface exposes submission cookies, reaping, and draining, making pipelined commands and batching an explicit alternative to one-command-at-a-time execution.

Start with the I/O API and follow its handle and completion types into the source. Published documentation can advance independently of a distribution's packaged version.

5. ZaidQureshi/bam

Language/role: C++/CUDA with a supporting kernel component; research framework for GPU-initiated access to NVMe storage.

BaM extends earlier NVMe-access code with concurrent GPU queues, a software-managed GPU cache, and array abstractions. Those additions constitute a substantive framework rather than a renamed fork. The repository identifies it as the ASPLOS 2023 research implementation.

C1: The page-cache implementation coordinates valid, busy, and dirty states, reference counts, and device-scope atomic operations. Page acquisition, release, and dirty-data handling expose difficult GPU concurrency invariants.
C2: Array and range abstractions let GPU algorithms address storage-backed data without constructing every NVMe transaction themselves.
C3: Warp-level request coalescing, caching, and mappings across devices address both redundant requests and storage parallelism. These mechanisms can be followed in the same implementation.

Scope caveat: This is hardware-specific research software. The repository documents PCIe peer-to-peer and GPU requirements and warns about newer-kernel compatibility. Its prerequisites should be read before treating it as a deployable general-purpose library.

User-space block targets and caching

6. ublk-org/ublksrv

Language/role: C/C++; Linux user-space block-device server library and target implementations.

ublksrv connects the kernel ublk interface to user-space storage implementations using io_uring commands and per-queue, tag-indexed request buffers. It is useful for studying the boundary between a kernel-visible block device and a replaceable storage backend.

C1: The command manual source distinguishes recovery modes that fail outstanding requests from modes that reissue them, and whether new I/O waits while a server is absent. These choices directly affect application-visible failure and replay semantics.
C2: The library supports different target implementations; documented examples include loop, null, NBD, iSCSI, and NFS targets. Queue and request plumbing can be reused independently of the backing storage protocol.
C3: Queue-depth control, preallocated request buffers, and registered-buffer/zero-copy options make memory copying and outstanding concurrency explicit architectural choices.

Read the recovery and buffer sections together: restarting a server and managing memory ownership are linked concerns.

7. ublk-org/libublk-rs

Language/role: Rust; user-space ublk device, queue, and request-processing library.

This is a separate Rust framework for implementing ublk devices, with its own device/queue types, asynchronous processing, buffer descriptors, and validation. It offers useful comparisons with the C/C++ server design above.

C1: The I/O implementation checks queue/tag partitioning, validates buffer modes against device capabilities, and includes layout assertions and field-offset tests for kernel-facing structures. Raw-address descriptors remain an explicit responsibility boundary rather than being made safe merely by the language.
C2: Device and queue objects support multiple serving styles, while buffer descriptors represent ordinary slices, automatically registered buffers, raw addresses, and special completion data. This separates target logic from transport details.
C3: Thread-local io_uring state, batched operations, and alternatives to copying give a concrete account of how the Rust API preserves the underlying queue model.

The I/O module is the strongest entry point for studying where type-level structure ends and kernel protocol obligations begin.

8. open-iscsi/tcmu-runner

Language/role: C; user-space SCSI backstore framework for Linux LIO/TCMU.

TCMU-runner lets backend authors implement storage operations while reusing SCSI command processing and the kernel/user-space transport. The project guide describes both daemon plugins and embedding libtcmu in another application's event loop.

C1: The guide requires single-threaded handling of the master event source while permitting per-device execution choices. The library header specifies command-processing/completion sequencing and command-private memory lifetime. These are meaningful ownership and ordering constraints for asynchronous backends.
C2: A handler can process a whole SCSI command or provide narrower read/write/flush callbacks, with fallback handling. Applications can reuse either the runner's event loop or the lower-level library, making the abstraction useful beyond one daemon arrangement.

Activity caveat: The GitHub API reported the latest repository push as 2024-05-27 when checked. The repository was not archived, but that does not establish ongoing maintenance; evaluate it as an established implementation with limited recent activity evidence.

9. Open-CAS/ocf

Language/role: C; reusable block-cache engine with environment adaptation, usable in kernel and user-space integrations.

OCF belongs here through its user-space embedding model and SPDK integration, rather than because the entire project runs exclusively in user space. The architecture overview describes the common cache engine used by different surrounding environments.

C1: The I/O engine design explains difficult cases such as partially cached dirty reads, cleaning data back to the core before rereading, and redirects caused by cache errors or unsuitable requests. Cache correctness depends on those transitions, not just hit/miss lookup.
C2: Cache modes select reusable I/O interfaces and read/write engines; environment wrappers separate caching policy from operating-system and allocation machinery.
C3: Asynchronous cache filling, shared engines across modes, sequential-request cutoffs, and specialized request paths make cache admission and background work explicit.

The repository also documents C unit tests and Python-driven functional testing, useful entry context for evaluating changes to the state machines.

Asynchronous file I/O and storage runtimes

10. axboe/liburing

Language/role: C; user-space library, documentation, and regression tests for Linux io_uring.

liburing is foundational for understanding completion-driven storage APIs. Its relevance goes beyond convenient syscall wrappers: the repository documents the shared-memory protocol and contains substantial kernel/userspace regression coverage.

C1: The io_uring manual explains submission/completion rings, memory ordering, correlation through user data, and the requirement to preserve data buffers until completion. Submission order is not a substitute for understanding completion dependencies.
C2: Queue setup, request preparation, submission, and completion helpers support many file and block I/O arrangements while leaving application scheduling policy open.
C3: Shared rings, batched submissions, and polling modes provide explicit ways to reduce repeated syscall and notification costs, with the associated ordering responsibilities still visible.

Use the manual together with the changelog, which records regression fixes, test changes, and handling of differing kernel capabilities. Available operations depend on the running kernel as well as the library version.

11. scylladb/seastar

Language/role: C++; asynchronous application framework, included for its file I/O and disk scheduling subsystem.

Seastar is broader than storage, but its disk scheduler is a substantial reusable implementation in its own right. Study how asynchronous operation lifetime interacts with fairness and admission, rather than treating the entire application framework as the category match.

C1: The I/O queue implementation tracks queued and executing work through dispatch, completion, failure, and cancellation. It also preserves vectored-I/O state and splits requests around supported limits. Accounting and object lifetime must agree across each exit path.
C2: Priority classes, request intents, futures, and queue/group objects separate callers' storage operations from scheduling policy and device limits.
C3: Shared capacity accounting, bandwidth throttling, and fair scheduling make interference between workloads an explicit design problem. The implementation provides a useful bridge from scheduling abstractions to actual read/write dispatch.

The I/O queue source is the primary entry point; follow its descriptors and group types when investigating a particular scheduling invariant.

12. DataDog/glommio

Language/role: Rust; thread-per-core asynchronous runtime with a substantial direct-file-I/O subsystem.

Glommio is particularly useful for studying how a runtime exposes the costs and restrictions of direct I/O. Its I/O module guide distinguishes buffered files from DMA-oriented files instead of assigning them identical performance semantics.

C1: DmaFile implementation and API contracts explain alignment, legal short writes, and ownership of buffers while the kernel uses them. Aligned and convenience read operations have different responsibilities.
C2: File objects and higher-level I/O facilities let applications choose direct or buffered access while sharing runtime integration and error handling.
C3: Multi-read processing can merge adjacent or overlapping requests and return individual results. Explicit limits on merged-buffer size and read amplification make the tradeoff between fewer device operations and excess reads inspectable.

Start with the I/O guide, then trace DmaFile's batched-read path to connect API ergonomics with the physical requests it generates.

13. tokio-rs/tokio-uring

Language/role: Rust; io_uring-backed asynchronous file I/O integrated with a Tokio runtime.

The project is a compact study of adapting Rust futures to completion-based I/O. Its published API documentation describes a current-thread runtime and operations that return buffer ownership together with their result.

C1: Reads and writes retain owned buffers until completion. Explicit asynchronous close addresses a lifetime problem that ordinary destructor-based cleanup cannot always make predictable; implicit drop may defer closing to background work.
C2: The runtime and file API integrate completion-based storage with Tokio tasks and other runtime facilities, providing reusable operation and buffer interfaces.
C3: The design document explains queue affinity, per-thread execution, and cancellation/resource-lifetime tradeoffs. It is valuable architectural material, but is a design proposal and should not be mistaken for a guarantee that every proposed feature shipped.

Activity caveat: The repository was not archived; the API reported its latest push as 2025-07-07. Check implementation and ecosystem fit before adopting it for new work.

14. ned14/llfio

Language/role: C++17; cross-platform low-level file I/O with typed native-handle abstractions.

LLFIO offers a different perspective from queue runtimes: it concentrates on file, path, and memory-mapping semantics while exposing operating-system behavior and costs. The project guide describes its handle hierarchy, scatter/gather I/O, and allocation-conscious path representation.

C1: The path_handle interface uses open directory handles as stable anchors even when other processes rename ancestors. It also explicitly documents remaining time-of-check/time-of-use hazards in existence checks, making the boundary of the abstraction clear.
C2: Path, I/O, file, and mapped-file handles form reusable layers rather than a collection of unrelated convenience functions. Explicit cloning and handle ownership expose resource semantics.
C3: Lightweight path handles, non-owning path views, scatter/gather operations, and memory mapping connect interface design to allocation and syscall costs.

Use the develop branch entry points above. The guide records historical standardization work and its abandonment in 2025; it should not be described as an API currently destined for the C++ standard.

User-space filesystem interfaces

15. libfuse/libfuse

Language/role: C; Linux user-space filesystem protocol library and mounting support.

libfuse exposes both path-oriented synchronous callbacks and a lower-level inode-oriented interface with asynchronous replies. It is foundational material for understanding the obligations between a filesystem server and the kernel's VFS/cache machinery.

C1: The low-level operations contract specifies lookup-reference accounting, forgotten inodes, and unlinked-but-open objects. It also distinguishes flush from durability and explains why flush can occur multiple times for an open file.
C2: The two API levels let implementers choose between a simpler filesystem model and explicit request/inode control while reusing protocol transport and dispatch.
C4: The dated changelog spans releases from 2016 onward and documents callback/API transitions, deprecations, regression fixes, and hardening. That is direct evidence of sustained compatibility and complexity management.

Maintenance caveat: The repository explicitly reports no active regular contributors and limited maintainer capacity, while accepting contributions and producing releases. Recent activity should not obscure that staffing statement.

16. hanwen/go-fuse

Language/role: Go; native FUSE library and inode-based filesystem framework.

go-fuse is useful for studying how filesystem semantics fit a garbage-collected, concurrently scheduled language. The filesystem package guide goes well beyond mounting examples and describes the contracts implementers must preserve.

C1: Inode identity, hard links, deleted-but-open files, parallel callbacks, and cancellation through Go contexts require careful state ownership. The guide also explains deadlock hazards when a process accesses the filesystem it serves.
C2: Inode objects and operation-specific node interfaces let an implementation provide only the operations it needs, with a reusable bridge to lower-level FUSE requests.
C3: Kernel content, attribute, and directory-entry caches are controlled through timeouts and explicit invalidation notifications. The documentation connects those choices to request traffic and context-switch costs, while explaining the consistency obligations when data changes outside the current request.

Start with the package guide's inode, caching, and concurrency sections; they make a strong checklist for reviewing a concrete Go filesystem implementation.

17. cberner/fuser

Language/role: Rust; FUSE protocol implementation and low-level filesystem callback interface.

fuser is a substantive Rust implementation descended from the fuse-rs lineage, with its own request types, replies, capability handling, and filesystem trait. It is useful for studying how much of a kernel/userspace protocol can be represented through types, and which obligations remain with the filesystem author.

C1: The core library source documents lookup-reference/forget behavior and differences caused by writeback caching. Kernel capability negotiation explicitly rejects unsupported combinations rather than silently promising behavior the library cannot provide.
C2: The Filesystem trait, inode/file-handle types, operation-specific replies, and kernel-configuration API provide reusable structure for diverse filesystem backends. The trait's concurrency bounds make shared access part of the interface.

Study the capability negotiation and callback documentation together. Rust ownership helps manage library resources, but it does not automatically establish correct inode lifetime, cache coherence, or writeback semantics for an application built on this API.

18. winfsp/winfsp

Language/role: Primarily C; Windows filesystem proxy driver and user-space filesystem library.

WinFsp extends the selection beyond Linux and exposes the complexity of implementing user-space filesystems while preserving Windows file-sharing, caching, and memory-mapping behavior.

C1: The design document follows requests through preprocessing, pending queues, preparation in the filesystem process, processing tables, and completion. It explains matching out-of-order replies to requests and managing device state when the user-space process exits.
C2: A common driver/library boundary lets different filesystem implementations supply storage behavior while sharing the Windows filesystem integration and request transport.
C3: Preparing mapped buffers in the proper process context and combining reply submission with request retrieval address copying and transition costs. The request-state architecture makes those optimizations understandable rather than incidental.

Read the design's request lifecycle alongside its discussion of Windows semantics. A successful read/write callback alone is insufficient to reproduce a native filesystem's sharing and cache behavior.

19. dokan-dev/dokany

Language/role: C/C++; Windows kernel proxy and user-space filesystem callback framework.

Dokany is an explicitly evolved fork of legacy Dokan, with documented API changes and later threading/memory-management redesigns. It counts here as a separate implementation with substantive evolution, not as an additional copy of the original project. The project guide explains that lineage and its native and FUSE-compatible interfaces.

C1: The callback header distinguishes cleanup from final close, requires thread-safe callbacks, and warns that memory-mapped reads or writes may arrive after cleanup. Delete and timeout semantics add further failure-sensitive contracts.
C2: A shared operations structure and per-file context connect diverse user-space storage implementations to the same Windows driver and dispatch machinery.
C3: The interface documents optional IPC batching and its latency/CPU tradeoff. The repository's evolution of worker and memory-pool behavior offers concrete performance architecture to investigate.

The callback header is the best starting point for comparing Dokany's resource lifecycle with WinFsp or FUSE.

Remote-storage client engines

20. sahlberg/libiscsi

Language/role: C; asynchronous iSCSI initiator and SCSI task library, with synchronous convenience interfaces.

libiscsi lets applications access remote block storage without requiring a kernel initiator for their data path. The project guide documents TCP and iSER transports, asynchronous operation, and an accompanying SCSI conformance test tool.

C1: The public interface specifies how reconnect backoff changes the events a caller should poll, and why periodic service calls are necessary for timeouts to advance. Timeout settings attach to requests at creation; changing a setting does not retroactively rewrite in-flight requests.
C2: Applications can integrate the library's file descriptor and requested events into their own event loop or use the synchronous layer. SCSI tasks and transport selection are reusable beyond one storage utility.
C3: Explicit queue state and asynchronous progress allow multiple commands to remain outstanding while the application controls scheduling, rather than imposing a blocking request loop.

The public header is especially useful for studying the contract between a protocol engine and an external event loop.

21. sahlberg/libnfs

Language/role: C; user-space NFS client and RPC engine with raw asynchronous, POSIX-like asynchronous, and synchronous APIs.

libnfs provides a remote-file counterpart to libiscsi's remote-block interface. Its project guide explains the three API levels, zero-copy work, protocol support, and the incompatible API-v2 transition.

C1: The public header requires event-loop service for timeout processing, including when no descriptor event occurs, and distinguishes fatal context errors. Request/response identifiers and protocol session-slot state introduce additional ordering and lifetime obligations.
C2: The three API layers let applications choose between low-level RPC control and familiar file operations without implementing the NFS transport anew.
C3: Hashed transaction lookup supports large sets of outstanding RPCs, while NFS session slots bound how many protocol requests may proceed. The guide explains how queued work is released as slots become available.

The API-v2 incompatibility is material when selecting examples or integrating an existing application; do not assume older call signatures or cancellation behavior remain valid.

Coverage, search method, and limitations

Discovery used live web search with more than six distinct formulations, followed by reading official GitHub pages/API metadata and primary implementation or architecture material for every retained repository. Search angles included:

User-space storage frameworks, SPDK, xNVMe, and portable block-I/O libraries.
VFIO/NVMe drivers, libvfn, NVMe command libraries, and zoned-storage interfaces.
Linux ublk, Rust block servers, NBD plugins, and TCMU/SCSI backstore handlers.
io_uring libraries, Rust asynchronous file runtimes, C++ low-level file I/O, and Seastar disk scheduling.
FUSE libraries in C, Go, and Rust; Windows filesystem frameworks.
User-space NFS/iSCSI clients, Open CAS caching engines, and GPU-initiated storage.
Additional Java storage-runtime and GPU/direct-storage searches, which mostly produced general-purpose runtimes, applications, wrappers, or kernel-focused components outside the chosen scope.

Later searches increasingly repeated these implementation families or returned weaker fits. The selection therefore stops at 21 substantial repositories rather than padding the list. It spans C, C++, Rust, Go, and CUDA, from small device/protocol libraries through runtime subsystems and operating-system integration frameworks. The prevalence of C and Linux reflects the discovered interfaces and projects, not a requirement of the category.

Canonical repository URLs were checked by opening GitHub pages or reading GitHub API metadata. Linked implementation files and documentation were opened and read; repository descriptions and search snippets were not treated as sufficient implementation evidence. None of the retained repositories was marked archived at the time of checking. This is a metadata observation, not a maintenance guarantee; the specific libfuse, tokio-uring, and TCMU-runner caveats above are more informative. Each monorepo is counted once, and no ordinary fork is included as a duplicate implementation.

Important exclusions are deliberate. libguestfs/nbdkit explicitly labels its GitHub repository unmaintained and redirects development to GitLab, so that stale copy is not counted. libblkio's official documentation points to GitLab; no official substantive GitHub mirror was verified during this search. These are gaps imposed by the GitHub-only requirement, not judgments that the projects lack technical depth. Benchmark clients, complete storage products, generic networking frameworks, kernel-only drivers, and thin bindings were also excluded.

Criteria assessments are grounded engineering judgments about the cited mechanisms. No candidates were built, benchmarked, or subjected to a security/correctness audit. Links to moving branches and published documentation describe the material available on the research date; they are not immutable version pins. Hardware, kernel, protocol, and API-version restrictions should be checked against a prospective deployment, particularly for BaM and newer asynchronous interfaces.