fix(p7-devices): sendfile falls back to buffered read for non-regular sources (--copy-devices no longer hangs); test/doc hardening

RSYNC_COMPAT.md

@@ -250,7 +250,7 @@ why plain `--append` works on the normal atomic path, not only with `--inplace`.
 | `-D` | Same as --devices --specials | ✅ Implemented | Implies `--devices --specials`. `-D` was unassigned in FastSync (verified: no collision), so it is free to imply both device-node and special-file preservation. See the `--devices`/`--specials` rows and the Phase-4 devices notes below |
 | `--devices` | Preserve device files | ✅ Implemented | Recreates char/block device nodes on the destination via `mknod` instead of transferring content. Type + rdev are validated strictly (S_IFMT from the transmitted mode; major/minor range-checked, non-negative), and creation is **privilege-gated**: `mknod` needs `CAP_MKNOD`, so a non-root receiver (CI runs via setpriv as non-root) logs a warning and **skips the device entry safely** — the whole transfer never aborts just because the node could not be made. The node is created fd-relative below the receive root (`mknodat` on the confined secure parent), so it can never be placed outside the authorized root, never follows a symlink, and never replaces an existing directory. Only a char/block mode is honored. Crosses the wire (a new `STATUS_SPECIAL` frame carries the path + metadata mode + rdev; `PROTOCOL_VERSION` bumped **2.12.0 → 2.13.0**). Divergence: per-entry skip (not a hard error) when the receiver lacks `CAP_MKNOD`, documented in the Phase-4 devices notes |
 | `--specials` | Preserve special files | ⛔ Impossible/Divergence | **FIFO recreation works**: FIFOs are recreated on the destination via `mkfifo` (unprivileged, so this is a real, assertable behavior under CI). **Only socket recreation is impossible**: a socket entry can be created only by `bind(2)` on a live socket, not by any filesystem call, so a source socket is skipped with an explicit note. That one unsupported node kind is why the flag is classified Impossible/Divergence even though FIFO recreation itself works; its normal path is otherwise complete. FIFO creation is privileged-gated only in the sense of graceful skip on any permission failure. Node creation is confined below the receive root (`mkfifoat` on the secure fd-relative parent; no `..`, no symlink follow). Crosses the wire like `--devices` (the `STATUS_SPECIAL` frame; `PROTOCOL_VERSION` bumped **2.12.0 → 2.13.0**). See the Phase-4 devices notes |
-| `--copy-devices` | Copy device contents as file | ✅ Implemented | Copy a device's CONTENT into an ordinary regular file on the destination instead of recreating the node — non-privileged and safe. FastSync scans a device/FIFO as a regular file: its reported size (`st_size`, typically 0 for char devices and FIFOs) is copied, so a FIFO or a non-readable device becomes an empty (or size-bounded) regular file without ever blocking or reading unbounded pseudo-device streams. The run always succeeds and never crashes on such input. **Deliberate, safe divergence from rsync's dd-like unbounded device read.** See the Phase-4 devices notes |
+| `--copy-devices` | Copy device contents as file | ✅ Implemented | Copy a device's CONTENT into an ordinary regular file on the destination instead of recreating the node — non-privileged and safe. FastSync scans a device/FIFO as a regular file: its reported size (`st_size`, typically 0 for char devices and FIFOs) is copied, so a FIFO or a non-readable device becomes an empty (or size-bounded) regular file. The default data path is size-bounded and never blocks (it sends exactly `st_size` bytes, never an unbounded pseudo-device stream); with `--sendfile`, a non-regular source (FIFO/device) is detected from its `stat` mode and falls back to that same buffered read, so `--copy-devices --sendfile` cannot hang either. The run always succeeds and never crashes on such input. **Deliberate, safe divergence from rsync's dd-like unbounded device read.** See the Phase-4 devices notes |
 | `--write-devices` | Write to devices as files | ✅ Implemented | Write the received data directly into an **existing** device node on the destination instead of creating a regular file. Restricted and best-effort: the destination must already exist and be a char/block device (opened only under the confined receive root, with `O_NOFOLLOW` + `O_NONBLOCK`); a missing, symlinked, FIFO-with-no-reader (`ENXIO`), non-device destination, or any write failure is **skipped with a warning** rather than allowed, so a run can never clobber the system, never blocks on a special-file target, and never aborts on an unusable target. See the Phase-4 devices notes |
 | `-U`, `--atimes` | Preserve access times | ✅ Implemented | Captures the source access time (from the scanner's pre-read stat, so it is not clobbered by reading the file for transfer) and transmits it over the wire; the receiver restores it together with the mtime via `futimens`/`utimensat`. Implies metadata transmission (the times travel inside the `-M` metadata payload), but does not enable ownership application (that stays opt-in via the identity flags). Wire: new `atime` fields on the metadata frame + a `preserve_atimes` config boolean; `PROTOCOL_VERSION` bumped **2.11.0 → 2.12.0** |
 | `-N`, `--crtimes` | Preserve create times | ⚠️ Partial | Captures the source birth time via `statx(STATX_BTIME)` on Linux and transmits it (recorded as a wire field), but there is **no portable way to set a birth time** (`utimensat` can only set atime/mtime), so the receiver explicitly does NOT apply it: it logs a debug note and continues — never failing the transfer and never pretending it worked. On platforms without `statx` it parses as a documented no-op (flag accepted; nothing is captured). Implies metadata transmission. Wire: new `crtime` fields + a `preserve_crtimes` config boolean; `PROTOCOL_VERSION` bumped **2.11.0 → 2.12.0** (see the Phase-4 metadata-time notes) |

src/client/client_send.c

@@ -1259,6 +1259,19 @@ static int send_single_file(Client* client, File* file, Config* config, bool use
   return 0;
 }
 
+/* Sendfile calls a blocking open() on the source (file_send_sendfile_with_skip
+ * -> file_open_for_read), which never returns for a FIFO/device with no writer.
+ * Only a regular file may take the zero-copy sendfile path; a non-regular source
+ * (FIFO/device copied by --copy-devices) must use the buffered, size-bounded
+ * read path instead.  `stat` follows symlinks, so a dereferenced symlink to a
+ * regular file keeps the sendfile fast path. */
+static bool source_is_regular_file(const File* file) {
+  if (!file || !file->path)
+    return false;
+  struct stat st;
+  return stat(file->path, &st) == 0 && S_ISREG(st.st_mode);
+}
+
 static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
                                    ArrayList* remove_sources) {
   if (config->use_chunk_serialization) {
@@ -1337,8 +1350,9 @@ static int send_chunk_with_removal(Client* client, Chunk* chunk, Config* config,
       continue;
     }
     bool stream = f->data->data == NULL && f->data->size > 0;
-    bool use_sendfile =
-        (config->use_sendfile && !config->use_compression) || (stream && !config->use_compression);
+    bool use_sendfile = ((config->use_sendfile && !config->use_compression) ||
+                         (stream && !config->use_compression)) &&
+                        source_is_regular_file(f);
     SourceFile* source = remove_sources ? source_file_create(f) : NULL;
     int rc = send_single_file(client, f, config, config->use_incremental, use_sendfile);
     if (rc == 1) {

tests/integration/test_features.py

@@ -16,7 +16,7 @@ from common import (
     run_client, CountingProxy,
     generate_test_files, verify_transfer, clean_dir, make_result,
     get_dest_received_dir, CLIENT_CMD, SERVER_CMD, ServerManager,
-    _find_free_port, _wait_for_port,
+    _find_free_port, _wait_for_port, _wait_proc,
 )
 
 SOURCE_DIR = os.path.join(TEST_DATA_DIR, "feature_source")
@@ -25,6 +25,31 @@ DEVICE_SOURCE = os.path.join(TEST_DATA_DIR, "device_source")
 DEVICE_DEST = os.path.join(TEST_DATA_DIR, "device_dest")
 
 
+def _start_captured_server(prefix=None, extra_args=None):
+    """Start a plain-TCP server with captured stdout/stderr for one test.
+
+    Returns (proc, port).  The caller owns `proc` and must terminate it via
+    `_wait_proc` so a server that ignores SIGTERM is killed instead of leaving
+    a zombie or raising TimeoutExpired.  The shared session server discards its
+    output, so tests that lock in a receiver-side warning need their own.  The
+    server's SIGTERM handler exits via `_exit`, which does not flush stdio, so
+    `stdbuf -oL` keeps stdout line-buffered and the warning observable."""
+    port = _find_free_port()
+    cmd = ["stdbuf", "-oL"] + (prefix or []) + SERVER_CMD + ["-p", str(port), "--allow-unauthenticated"]
+    if extra_args:
+        cmd += extra_args
+    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
+    _wait_for_port(port)
+    return proc, port
+
+
+def _stop_captured_server(proc):
+    """Terminate a captured server and return its (stdout, stderr) text."""
+    proc.terminate()
+    _wait_proc(proc)
+    return proc.communicate()
+
+
 class TestDeviceSpecial:
     """Phase 4: --devices / --specials / -D / --copy-devices / --write-devices.
 
@@ -66,19 +91,22 @@ class TestDeviceSpecial:
         received = get_dest_received_dir(DEVICE_DEST, DEVICE_SOURCE)
         assert stat.S_ISFIFO(os.stat(os.path.join(received, "pipe.fifo")).st_mode)
 
-    def test_specials_socket_source_skipped_safely(self, shared_server):
+    @pytest.mark.ci
+    def test_specials_socket_source_skipped_safely(self):
         """A socket cannot be recreated by any standard filesystem call, so
         --specials must skip it with a note and still complete the run (the
         adjacent regular file transfers normally; no socket node appears)."""
         self._setup()
         sock_path = os.path.join(DEVICE_SOURCE, "source.sock")
         s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+        server, port = _start_captured_server()
         try:
             s.bind(sock_path)
             result, _ = run_client(DEVICE_SOURCE, DEVICE_DEST,
-                                   flags=["--specials"], port=shared_server.port)
+                                   flags=["--specials"], port=port)
         finally:
             s.close()
+            out, err = _stop_captured_server(server)
         assert result.returncode == 0, f"Exit {result.returncode}: {result.stderr[:200]}"
         received = get_dest_received_dir(DEVICE_DEST, DEVICE_SOURCE)
         with open(os.path.join(received, "plain.txt")) as f:
@@ -86,15 +114,23 @@ class TestDeviceSpecial:
         assert not os.path.lexists(os.path.join(received, "source.sock")), (
             "socket source must be skipped, not materialized"
         )
+        assert "socket not recreated" in (out + err), (
+            f"receiver did not log the documented socket skip: out={out!r} err={err!r}"
+        )
 
-    def test_copy_devices_fifo_becomes_regular_file(self, shared_server):
+    @pytest.mark.ci
+    @pytest.mark.parametrize("flags", [["--copy-devices"], ["--copy-devices", "--sendfile"]])
+    def test_copy_devices_fifo_becomes_regular_file(self, shared_server, flags):
         """--copy-devices treats a special source as an ordinary regular-file
         copy: a FIFO (st_size 0) becomes a zero-length REGULAR file on the
-        destination (never a FIFO, never a hang), and the run succeeds."""
+        destination (never a FIFO, never a hang), and the run succeeds.  The
+        --sendfile variant previously blocked forever in the sendfile open();
+        the non-regular source now falls back to the buffered read path, so it
+        must complete within the bounded-time assertion below."""
         self._setup()
         os.mkfifo(os.path.join(DEVICE_SOURCE, "device_copy.fifo"))
-        result, _ = run_client(DEVICE_SOURCE, DEVICE_DEST,
-                               flags=["--copy-devices"], port=shared_server.port)
+        result, dur = run_client(DEVICE_SOURCE, DEVICE_DEST,
+                                 flags=flags, port=shared_server.port)
         assert result.returncode == 0, f"Exit {result.returncode}: {result.stderr[:200]}"
         received = get_dest_received_dir(DEVICE_DEST, DEVICE_SOURCE)
         copied = os.path.join(received, "device_copy.fifo")
@@ -104,6 +140,7 @@ class TestDeviceSpecial:
             f"copy-devices must produce a regular file, got mode {oct(st.st_mode)}"
         )
         assert st.st_size == 0, f"expected a size-bounded 0-byte copy, got {st.st_size}"
+        assert dur < 60, f"{' '.join(flags)} hung on a FIFO source"
 
     def test_write_devices_non_crash(self, shared_server):
         """--write-devices writes into an existing device only; when the
@@ -117,6 +154,7 @@ class TestDeviceSpecial:
                                flags=["--write-devices"], port=shared_server.port)
         assert result.returncode == 0, f"Exit {result.returncode}: {result.stderr[:200]}"
 
+    @pytest.mark.ci
     def test_write_devices_regular_file_target_skipped(self, shared_server):
         """--write-devices only ever writes into an existing char/block node: a
         pre-existing REGULAR file at the destination path is left byte-identical
@@ -148,18 +186,13 @@ class TestDeviceSpecial:
         # The unprivileged receiver must be able to create the destination tree.
         os.makedirs(DEVICE_DEST, exist_ok=True)
         os.chmod(DEVICE_DEST, 0o777)
-        port = _find_free_port()
-        server = subprocess.Popen(
-            ["setpriv", "--reuid=65534", "--regid=65534", "--clear-groups"] +
-            SERVER_CMD + ["-p", str(port), "--allow-unauthenticated"],
-            stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+        server, port = _start_captured_server(
+            prefix=["setpriv", "--reuid=65534", "--regid=65534", "--clear-groups"])
         try:
-            _wait_for_port(port)
             result, _ = run_client(DEVICE_SOURCE, DEVICE_DEST,
                                    flags=["--devices"], port=port)
         finally:
-            server.terminate()
-            server.wait(timeout=5)
+            out, err = _stop_captured_server(server)
         assert result.returncode == 0, f"Exit {result.returncode}: {result.stderr[:300]}"
         received = get_dest_received_dir(DEVICE_DEST, DEVICE_SOURCE)
         with open(os.path.join(received, "plain.txt")) as f:
@@ -167,6 +200,9 @@ class TestDeviceSpecial:
         assert not os.path.lexists(os.path.join(received, "chardev")), (
             "a receiver without CAP_MKNOD must skip the device node, not create it"
         )
+        assert "cannot create device node" in (out + err), (
+            f"receiver did not log the documented CAP_MKNOD skip: out={out!r} err={err!r}"
+        )
 
     @pytest.mark.skipif(os.geteuid() != 0, reason="requires root to create device nodes")
     def test_devices_recreates_real_char_device(self, shared_server):