7 Commits
Author SHA1 Message Date
TapTap a196522010 style: clang-format test_client_cli.c after merge resolution
CI / lint (push) Successful in 23s
CI / sanitizers (undefined) (push) Successful in 42s
CI / sanitizers (address) (push) Successful in 43s
CI / fuzz-build (push) Successful in 17s
CI / coverage (push) Successful in 34s
CI / valgrind (push) Successful in 36s
CI / build-and-test (push) Successful in 1m45s
2026-09-06 14:29:28 +02:00
TapTap 342dd152ef Merge feat/p2-files-from-filter: files-from/from0/filter/-F/-C
Client-side file-list selection and filter-rule layer (client-only; no wire
change). rsync-consistent inner-first per-dir filter precedence; listed-but-
missing files-from entries hard-error; root '/' scan regression fixed;
Summary recounted (63 implemented / 75 not).
c-review REQUEST CHANGES -> blockers fixed; PR #265.
2026-09-06 14:25:51 +02:00
TapTap a6c982cd86 Merge feat/p2-delay-updates: implement --delay-updates
Receiver stages writes and publishes only after the full transfer succeeds
(single and -m, before the outcomes frame). delete walker skips staging;
backup-dir name reserved; flock serializes concurrent delayed sessions;
protocol bump to 2.6.0. c-review REQUEST CHANGES -> blockers fixed; PR #264.
2026-09-06 14:19:38 +02:00
TapTap 4295fefaa3 fix: --delay-updates delete/backup collisions, publish-failure test, staging lock
Review fixes for --delay-updates:

- --delete no longer deletes the staged files: the delete walker gains a
  skip_root_child parameter and receive_manifest passes DELAY_UPDATES_STAGING_DIR
  when delay_updates is active, so deletion removes genuine extras while the
  staging dir (a direct child of the receive root) is left for publication in
  both single and -m modes.
- --backup-dir is rejected when it collides with the reserved internal staging
  name .fastsync-stage (trailing slash normalized), in client validation and in
  the received-config wire validation, preventing old backups from being
  silently installed as new files.
- Staging dir is now held under an exclusive advisory flock for the whole
  transfer (context lifetime): two simultaneous delayed transfers to one
  destination root no longer share/destroy each other's staged data - the
  second fails cleanly.  Cleanup only touches the staging dir when this context
  owns the lock, so a lock-contention failure cannot wipe a live session.
- Post-publish staging cleanup now returns/logs instead of discarding failures
  (warning when the staging dir cannot be fully removed).
- Reworked the publish-failure integration test to exercise real mid-publish
  semantics (top-level file published, nested rename fails, no rollback,
  sources retained under --remove-source-files) and added integration tests for
  --delete + --delay-updates ordering and reserved --backup-dir rejection.
- RSYNC_COMPAT note documents delete ordering, the reserved-name hazard, and
  the concurrency guard.
2026-09-06 14:19:18 +02:00
TapTap 7f2180ef90 fix: filter precedence, files-from errors, NUL/CRLF and filter-rule rejections
Address an independent c-review of the files-from/filter feature:

- .rsync-filter precedence now matches rsync: evaluate the innermost
  (current) directory's rules first, then ancestors, then the command-line
  base (--filter/-C), so a deeper file's '+' can re-include what a shallower
  '-' excluded (regression tests in both scan modes; single-thread and -m).
- --files-from: a listed entry missing on disk and an empty list are now hard
  errors surfaced pre-transfer in send_files, send_files_multithreaded,
  dry-run and --list-only; '.' (whole tree) and empty listed dirs stay valid.
- scanner_path_relative now handles a transfer root of / (previously the
  scanner aborted on children of /).
- Reject unsupported rsync filter syntax explicitly (no silent no-ops):
  +/- modifiers other than '/' (! C s r p x) and rules beginning with ':'/'.'
  /'!' (merge/dir-merge/list-clear shorthands). Docs updated.
- -0/--from0 NUL mode preserves entry bytes (no CR/LF trimming); only newline
  mode trims. Absolute-entry error message no longer includes the newline.
- --no-from0/--no-cvs-exclude registered as negatable booleans.
- RSYNC_COMPAT rows updated for the precedence, rejection list, NUL-mode
  detail and the documented O(entries x files) scalability bound of the
  allow-set (Summary unchanged: 62/3/5/1/76 = 147).
2026-09-06 14:16:12 +02:00
TapTap f4c15a7b78 feat: add --files-from/-0, --filter, -C and -F filter layer
CI / lint (pull_request) Successful in 23s
CI / sanitizers (address) (pull_request) Successful in 40s
CI / sanitizers (undefined) (pull_request) Successful in 39s
CI / fuzz-build (pull_request) Successful in 17s
CI / coverage (pull_request) Successful in 34s
CI / build-and-test (pull_request) Successful in 1m28s
CI / valgrind (pull_request) Successful in 36s
Implement the RSYNC_COMPAT Phase-2 filter/parser feature group:
- --files-from=FILE (repeatable) plus -0/--from0 NUL delimiters: parse the
  source file list relative to the source root into a shared read-only
  allow-set; the scanner transfers listed files and the whole subtree of
  listed directories and prunes everything else in single- and
  multithreaded mode. Absolute/'..' entries and missing files are hard
  CLI errors.
- --filter=RULE: rsync-style +/- rules (anchored '/', dir-only trailing '/',
  word include/exclude forms) evaluated first-match-wins with a default of
  include, as an independent layer from legacy --exclude/--include.
  Unsupported directives (merge/hide/... ) are rejected explicitly. -f stays
  sendfile.
- -C/--cvs-exclude: well-known rsync CVS default exclude set.
- -F: per-directory .rsync-filter files read during traversal and applied to
  the owning directory's subtree (single + parallel), never transferred.
- Delete manifest still derives from what was actually sent.

Client-only config fields; no wire/protocol change. Adds unit coverage
(CLI parse, allow-set and filter scanning single+parallel) and integration
tests (TestFilesFrom, TestFilters). RSYNC_COMPAT matrix rows updated:
5 rows move to Implemented (Summary 62/3/5/1/76 = 147).
2026-09-06 13:43:34 +02:00
TapTap a2a82dd856 feat: implement --delay-updates receiver staging and publication
CI / lint (pull_request) Failing after 22s
CI / build-and-test (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
Stage every successfully written file under a private 0700 .fastsync-stage
directory inside the receive root and atomically publish all staged files
only after the whole protocol stream (manifest/delete handling included)
has completed, immediately before the success/outcome frame.  On any
abort/error before publication nothing is installed and staging is removed;
a publish failure aborts the transfer with best-effort cleanup of the
remainder (already-published files are not rolled back).  Crash leftovers
are wiped when the next delayed transfer starts.

Wire: new delay_updates config flag (selection-options block), protocol
version bumped to 2.6.0, client/server validation rejects --inplace.
CLI/usage/validation updated.  Works in single-threaded and -m modes
(exactly one write_thread stages files; the staged-file registry is
mutex-protected; publication runs once after both threads join).
--existing/--ignore-existing/--update decide against the final destination
at stage time; --backup is deferred to publication.  remove_source_files
outcomes are only sent after publication so skipped/unpublished sources are
never deleted.  Default (no flag) behavior is unchanged.

Tests: config wire round-trip, CLI parse, --inplace rejection, new
test_delay_updates unit suite (27 suites total), and integration
TestDelayUpdates covering single/-m parity, incremental reruns, remove
source files, receiver-skip ordering, and a deterministic publish-failure
abort path.
2026-09-06 13:20:03 +02:00
29 changed files with 3159 additions and 94 deletions

No files matched your search

+8 -8
View File
@@ -6,11 +6,11 @@ This document maps rsync's full feature set to FastSync's current implementation
| Status | Count | Description |
|--------|-------|-------------|
| ✅ Implemented | 57 | Feature works end-to-end |
| ✅ Implemented | 63 | Feature works end-to-end |
| 🔀 Alt Arg | 3 | Functionality exists but under different flag/semantics |
| ⚠️ Partial | 5 | Flag parsed/stored but behavior incomplete |
| 🔄 Compatibility No-op | 1 | Flag is accepted for CLI compatibility but has no effect |
| ❌ Not Implemented | 81 | Flag not recognized or no behavior |
| ❌ Not Implemented | 75 | Flag not recognized or no behavior |
| **Total** | **147** | |
---
@@ -30,7 +30,7 @@ This document maps rsync's full feature set to FastSync's current implementation
| `--no-motd` | Suppress daemon MOTD | ❌ Not Implemented | |
| `--exclude=PATTERN` | Exclude files matching pattern | ✅ Implemented | Glob matching in scanner |
| `--include=PATTERN` | Include files matching pattern | ✅ Implemented | Glob matching in scanner |
| `-C`, `--cvs-exclude` | Auto-ignore CVS files | ❌ Not Implemented | Removed because it had no effect |
| `-C`, `--cvs-exclude` | Auto-ignore CVS files | ✅ Implemented | Applies the well-known rsync default exclude set as exclude rules during scanning (RCS SCCS CVS CVS.adm RCSLOG cvslog.* tags TAGS .make.state .nse_depinfo *~ #* .#* ,* _$* *$ *.old *.bak *.BAK *.orig *.rej .del-* *.a *.olb *.o *.obj *.so *.exe *.Z *.elc *.ln core .svn/ .git/ .hg/ .bzr/); `.git/`-style repo dirs are pruned without descending |
## 2. Modifying Output
@@ -53,9 +53,9 @@ This document maps rsync's full feature set to FastSync's current implementation
|------|-------------------|-----------------|-------|
| `--exclude-from=FILE` | Read exclude patterns from file | ✅ Implemented | Reads patterns from file |
| `--include-from=FILE` | Read include patterns from file | ✅ Implemented | Reads patterns from file |
| `--filter=RULE` | Add file-filtering rule | ❌ Not Implemented | Removed because it had no effect |
| `--files-from=FILE` | Read source file list from file | ❌ Not Implemented | Removed because it had no effect |
| `-0`, `--from0` | Delimit *-from files with NULs | ❌ Not Implemented | |
| `--filter=RULE` | Add file-filtering rule | ✅ Implemented | Long option only: rsync's short `-f` conflicts with FastSync sendfile (see FastSync-specific list), so `-f` is not reassigned. Supported subset: `+`/`-` include/exclude, implicit-exclude patterns, `include`/`exclude` word forms, a leading `/` anchor (to the transfer root, or to a `.rsync-filter` file's directory), and a trailing `/` for dir-only rules; first match wins with a default of include inside the filter layer. Filters are an independent layer from `--exclude`/`--include` (an entry must pass both). Rejected with a clear error (no silent no-ops): `merge`/`dir-merge`/`hide`/`show`/`protect`/`risk`/`clear` words, rules that begin with `:`/`.`/`!` (merge/dir-merge/list-clear shorthands), and include/exclude modifiers other than `/` (`! C s r p x`) |
| `--files-from=FILE` | Read source file list from file | ✅ Implemented | Entries are paths relative to the source root (leading `./` stripped, `..`/absolute entries rejected at parse time, blank lines ignored; NUL-delimited with `-0`). A listed regular file is transferred; a listed directory transfers its whole subtree (FastSync recursion is always on, unlike rsync's non-recursive default). Non-listed paths and their subtrees are pruned by the scanner. A listed entry that does not exist under the source (and an empty list) is a hard error reported before any transfer; listing `.` (whole tree) and empty listed directories are fine. Scalability note: `file_list_affects` is O(list size) per scanned entry, so a very large `--files-from` list against a huge tree is quadratic; lists are typically small enough that this is acceptable, but it is the documented bound. The delete manifest still derives from what was actually sent, so `--delete` stays consistent with the subset |
| `-0`, `--from0` | Delimit *-from files with NULs | ✅ Implemented | `--files-from` entries become NUL-delimited; the flag may appear before or after `--files-from` on the command line. NUL mode preserves entry bytes exactly (trailing CR/LF are part of the name; only newline mode trims them) |
| `--max-size=SIZE` | Skip files larger than SIZE | ✅ Implemented | `max_size` in scanner |
| `--min-size=SIZE` | Skip files smaller than SIZE | ✅ Implemented | `min_size` in scanner |
| `-I`, `--ignore-times` | Don't skip files matching size+time | ❌ Not Implemented | |
@@ -65,7 +65,7 @@ This document maps rsync's full feature set to FastSync's current implementation
| `--ignore-existing` | Skip updating existing files | ❌ Not Implemented | |
| `--remove-source-files` | Sender removes regular files after confirmed transfer | ✅ Implemented | |
| `-x`, `--one-file-system` | Do not cross filesystem boundaries | ✅ Implemented | Sender scanner captures the root device and skips descending into mount-point crossings (`st_dev` differs); cross-filesystem mount-point subdirectories are dropped entirely, matching rsync |
| `-F` | Add the default `.rsync-filter` rules | ❌ Not Implemented | |
| `-F` | Add the default `.rsync-filter` rules | ✅ Implemented | Reads one filter rule per line from each directory's `.rsync-filter` file during traversal and applies it to that directory's subtree; the current directory's rules are evaluated before its ancestors', so deeper files override shallower ones and per-directory files override the command-line `--filter`/`-C` base by default (matching rsync's first-match-wins precedence); `.rsync-filter` files are never transferred. The rsync `-FF` behavior (also `.cvsignore`) is out of scope; unsupported rule types inside the file abort with a clear error |
## 4. Directory Options
@@ -96,7 +96,7 @@ This document maps rsync's full feature set to FastSync's current implementation
| `-b`, `--backup` | Make backups of overwritten files | ✅ Implemented | Backup before overwrite |
| `--backup-dir=DIR` | Backup directory hierarchy | ✅ Implemented | `backup_dir` config field |
| `--suffix=SUFFIX` | Backup suffix (default ~) | ✅ Implemented | `suffix` config field |
| `--delay-updates` | Put updated files in place at end | ❌ Not Implemented | |
| `--delay-updates` | Put updated files in place at end | ✅ Implemented | Successfully received files are staged under a private 0700 `.fastsync-stage` dir inside the receive root and atomically renamed into their final destinations only after the whole transfer (manifest/delete handling included) succeeds, just before the success/outcome frame is sent. The delete walker deliberately skips the staging dir at the receive root, so `--delete` removes genuine extras but never the staged files (deletion runs before publication; rsync's delete-after ordering is not implemented). `--existing`/`--ignore-existing`/`--update` decide against the final destination path at stage time; `--backup` moves the old file aside at publication. Incompatible with `--inplace` and with `--backup-dir=.fastsync-stage` (the internal staging name is reserved; both are rejected). The staging dir name is fixed, so two simultaneous delayed transfers to the same destination root are serialized with an exclusive advisory lock held for the whole transfer: the second session fails cleanly instead of corrupting the first. Aborting or failing before publication installs nothing and removes the staging tree; a crash between stage and publish leaves staged leftovers that the next delayed run wipes at start (process death releases the lock). A stage→publish failure aborts the transfer (best-effort cleanup of the not-yet-published staged files; already-published files are not rolled back). Works in single-threaded and `-m` modes |
| `-T`, `--temp-dir=DIR` | Create temporary files in DIR | ✅ Implemented | `--temp-dir` only; `-T` stays FastSync's `--timeout` alias. Scratch dir is resolved under the receive root; temp copies use a unique name there and are atomically renamed into place. If the scratch dir and destination are on different filesystems the atomic rename fails with EXDEV and the file save fails, which aborts the whole transfer (FastSync has no per-file skip/resume on a save error; rsync's non-atomic copy fallback is deliberately not used). `--inplace` and `--partial-dir` writes bypass the scratch dir |
## 7. Deletion
+67
View File
@@ -4,6 +4,8 @@
#include "compression.h"
#include "config.h"
#include "delta.h"
#include "file_list.h"
#include "filter.h"
#include "log.h"
#include "protocol.h"
#include "transport_tcp.h"
@@ -313,6 +315,31 @@ static int config_add_pattern(char*** patterns, int* count, const char* value,
return 0;
}
/* Validate and append one --filter=RULE string. Returns 0 on success, -1 on error. */
static int config_add_filter(Config* config, const char* rule) {
char err[160];
FilterRule* parsed = filter_rule_parse(rule, err, sizeof(err));
if (!parsed) {
log_message(LOG_LEVEL_ERROR, "invalid --filter rule '%s': %s", rule, err);
return -1;
}
filter_rule_free(parsed);
if (!config->filters) {
config->filters = array_list_create(free);
if (!config->filters) {
log_message(LOG_LEVEL_ERROR, "memory allocation failed for --filter");
return -1;
}
}
char* dup = str_dup(rule);
if (!dup || !array_list_add(config->filters, dup)) {
free(dup);
log_message(LOG_LEVEL_ERROR, "memory allocation failed for --filter");
return -1;
}
return 0;
}
static int parse_skip_compress(Config* config, const char* value) {
char* list = str_dup(value);
if (!list)
@@ -396,6 +423,7 @@ static const OptionEntry OPTION_TABLE[] = {
{"--log-file-format", NULL, OPT_STRING, offsetof(Config, log_file_format)},
{"--existing", NULL, OPT_FLAG, offsetof(Config, existing)},
{"--ignore-existing", NULL, OPT_FLAG, offsetof(Config, ignore_existing)},
{"--delay-updates", NULL, OPT_FLAG, offsetof(Config, delay_updates)},
{"--chmod", NULL, OPT_STRING, offsetof(Config, chmod_spec)},
{"--dirs", "--old-dirs", OPT_UNSUPPORTED, 0},
{"--old-d", NULL, OPT_UNSUPPORTED, 0},
@@ -422,6 +450,9 @@ static const OptionEntry OPTION_TABLE[] = {
{"--max-size", NULL, OPT_ULL, offsetof(Config, max_size)},
{"--min-size", NULL, OPT_ULL, offsetof(Config, min_size)},
{"--one-file-system", "-x", OPT_FLAG, offsetof(Config, one_file_system)},
{"--from0", "-0", OPT_FLAG, offsetof(Config, from0)},
{"--cvs-exclude", "-C", OPT_FLAG, offsetof(Config, cvs_exclude)},
{"-F", NULL, OPT_FLAG, offsetof(Config, per_dir_filter)},
};
/* Only boolean options with no required argument are safe to negate. */
@@ -443,6 +474,8 @@ static const NegatableOption NEGATABLE_OPTIONS[] = {
{"sparse", "S", offsetof(Config, preserve_sparse)},
{"inplace", NULL, offsetof(Config, inplace)},
{"checksum", NULL, offsetof(Config, checksum)},
{"from0", NULL, offsetof(Config, from0)},
{"cvs-exclude", NULL, offsetof(Config, cvs_exclude)},
/* These options are also implied by --archive or handled outside the table. */
{"compress", "c", offsetof(Config, use_compression)},
@@ -861,6 +894,26 @@ int parse_args(Config* config, int argc, char* argv[], int* positional_args,
if (read_patterns_from_file(argv[++i], &config->include_patterns, &config->include_count) !=
0)
return -1;
} else if (strncmp(argv[i], "--filter=", 9) == 0) {
if (config_add_filter(config, argv[i] + 9) != 0)
return -1;
} else if (opt_is(argv[i], "--filter", NULL)) {
if (i + 1 >= argc) {
log_message(LOG_LEVEL_ERROR, "missing argument for %s", argv[i]);
return -1;
}
if (config_add_filter(config, argv[++i]) != 0)
return -1;
} else if (strncmp(argv[i], "--files-from=", 13) == 0) {
if (set_string_option(&config->files_from, argv[i] + 13, "--files-from") != 0)
return -1;
} else if (opt_is(argv[i], "--files-from", NULL)) {
if (i + 1 >= argc) {
log_message(LOG_LEVEL_ERROR, "missing argument for %s", argv[i]);
return -1;
}
if (set_string_option(&config->files_from, argv[++i], "--files-from") != 0)
return -1;
} else if (opt_is(argv[i], "-v", "--verbose")) {
verbose = true;
set_log_level(LOG_LEVEL_DEBUG);
@@ -935,6 +988,20 @@ int parse_args(Config* config, int argc, char* argv[], int* positional_args,
if (config->compress_choice)
config->use_compression = strcmp(config->compress_choice, "zstd") == 0;
/* --files-from is loaded after every argument is seen so that -0/--from0 may
* appear anywhere on the command line. A missing or unreadable file, and
* invalid (absolute / traversal) entries, are hard CLI errors. */
if (config->files_from) {
char err[256];
FileListSet* set = file_list_load(config->files_from, config->from0, err, sizeof(err));
if (!set) {
log_message(LOG_LEVEL_ERROR, "--files-from: %s", err);
return -1;
}
file_list_destroy((FileListSet*)config->files_from_set);
config->files_from_set = set;
}
/* Incremental and delta transfers need metadata unless the user disabled it. */
if ((config->use_incremental || config->use_delta) && !config->use_metadata &&
!config->metadata_explicitly_disabled) {
+154 -23
View File
@@ -7,6 +7,8 @@
#include "data.h"
#include "delta.h"
#include "file.h"
#include "file_list.h"
#include "filter.h"
#include "metadata.h"
#include "log.h"
#include "multiprocessing.h"
@@ -40,16 +42,112 @@ static const char* display_bytes(unsigned long long bytes, bool human_readable,
return buffer;
}
static ScannerOptions scanner_options_from_config(const Config* config, int num_threads) {
ScannerOptions options = {config->use_metadata, config->chunk_size,
config->exclude_patterns, config->exclude_count,
config->include_patterns, config->include_count,
config->max_size, config->min_size,
config->max_depth, num_threads,
config->follow_symlinks, config->copy_links,
config->safe_links, config->copy_unsafe_links,
config->checksum, config->one_file_system};
return options;
/* Compiled scanner inputs that are shared read-only across scanner instances
* and, in -m mode, across worker threads. `base_filters` owns the compiled
* command-line + -C rules; the FileListSet allow-set lives in the Config. */
typedef struct {
ScannerOptions options;
FilterRuleList* base_filters; /* owned; may be NULL */
} PreparedScanner;
/* Build the scanner options for one scan. Returns false and logs on failure. */
static bool prepare_scanner(const Config* config, int num_threads, PreparedScanner* out) {
if (!out)
return false;
out->base_filters = NULL;
memset(&out->options, 0, sizeof(out->options));
int rule_count = config->filters ? config->filters->size : 0;
const char** texts = NULL;
if (rule_count > 0) {
texts = malloc((size_t)rule_count * sizeof(char*));
if (!texts) {
log_message(LOG_LEVEL_ERROR, "memory allocation failed for filter rules");
return false;
}
for (int i = 0; i < rule_count; i++)
texts[i] = (const char*)config->filters->items[i];
}
if (rule_count > 0 || config->cvs_exclude) {
char err[160];
out->base_filters = filter_base_build(texts, rule_count, config->cvs_exclude, err, sizeof(err));
free(texts);
if (!out->base_filters) {
log_message(LOG_LEVEL_ERROR, "invalid filter rule: %s", err);
return false;
}
} else {
free(texts);
}
ScannerOptions* options = &out->options;
options->use_metadata = config->use_metadata;
options->chunk_size = config->chunk_size;
options->exclude_patterns = config->exclude_patterns;
options->exclude_count = config->exclude_count;
options->include_patterns = config->include_patterns;
options->include_count = config->include_count;
options->max_size = config->max_size;
options->min_size = config->min_size;
options->max_depth = config->max_depth;
options->num_threads = num_threads;
options->follow_symlinks = config->follow_symlinks;
options->copy_links = config->copy_links;
options->safe_links = config->safe_links;
options->copy_unsafe_links = config->copy_unsafe_links;
options->checksum = config->checksum;
options->one_file_system = config->one_file_system;
options->file_list = (const FileListSet*)config->files_from_set;
options->base_filters = out->base_filters;
options->per_dir_filters = config->per_dir_filter;
return true;
}
static void prepared_scanner_destroy(PreparedScanner* prepared) {
if (!prepared)
return;
filter_rule_list_free(prepared->base_filters);
prepared->base_filters = NULL;
}
/* --files-from semantics: every listed entry must resolve under the source
* root, otherwise rsync reports a hard error instead of silently transferring
* nothing. An empty list is also an error. An entry of "." (the whole tree)
* and listed-but-empty directories are valid. Runs before any transfer so the
* failure is surfaced uniformly in the single-threaded, -m, dry-run and
* --list-only paths. */
static bool files_from_list_valid(const Config* config) {
const FileListSet* set = (const FileListSet*)config->files_from_set;
if (!set)
return true;
if (!config->send_directory) {
log_message(LOG_LEVEL_ERROR, "--files-from requires a source directory");
return false;
}
if (set->count == 0) {
log_message(LOG_LEVEL_ERROR, "--files-from file '%s' contains no entries; nothing to transfer",
config->files_from ? config->files_from : "");
return false;
}
for (int i = 0; i < set->count; i++) {
const char* entry = set->entries[i];
if (entry[0] == '\0')
continue; /* "." == list the whole tree */
char* full = path_cat(config->send_directory, entry);
if (!full) {
log_message(LOG_LEVEL_ERROR, "memory allocation failed while validating --files-from");
return false;
}
struct stat st;
if (lstat(full, &st) != 0) {
log_message(LOG_LEVEL_ERROR, "--files-from entry '%s' not found in source '%s'", entry,
config->send_directory);
free(full);
return false;
}
free(full);
}
return true;
}
/* Select the configured transport for both transfer execution paths. */
@@ -249,11 +347,17 @@ static void pipeline_cancel(PipelineContextSender* context) {
/* Print dry-run manifest showing files that would be transferred. Returns 0 on success. */
static int send_dry_run_manifest(const Config* config) {
ScannerOptions options = scanner_options_from_config(config, 0);
DirectoryScanner* scanner =
directory_scanner_create_with_options(config->send_directory, &options);
if (!scanner)
if (!files_from_list_valid(config))
return -1;
PreparedScanner prepared;
if (!prepare_scanner(config, 0, &prepared))
return -1;
DirectoryScanner* scanner =
directory_scanner_create_with_options(config->send_directory, &prepared.options);
if (!scanner) {
prepared_scanner_destroy(&prepared);
return -1;
}
Chunk* chunk;
int file_count = 0;
unsigned long long total_bytes = 0;
@@ -267,6 +371,7 @@ static int send_dry_run_manifest(const Config* config) {
if (!escaped_path) {
chunk_destroy(chunk);
directory_scanner_destroy(scanner);
prepared_scanner_destroy(&prepared);
return -1;
}
if (config->human_readable)
@@ -283,6 +388,7 @@ static int send_dry_run_manifest(const Config* config) {
chunk_destroy(chunk);
}
directory_scanner_destroy(scanner);
prepared_scanner_destroy(&prepared);
if (!config->quiet) {
if (config->human_readable)
printf("Total: %d files, %s\n", file_count,
@@ -319,12 +425,18 @@ static int compare_list_entries(const void* left, const void* right) {
* Directory lines are not printed because the scanner only yields regular
* transfer candidates. Returns 0 on success, 1 on error. */
static int send_list_only(const Config* config) {
ScannerOptions options = scanner_options_from_config(config, 0);
options.use_metadata = true; /* capture mode + mtime for the listing */
DirectoryScanner* scanner =
directory_scanner_create_with_options(config->send_directory, &options);
if (!scanner)
if (!files_from_list_valid(config))
return 1;
PreparedScanner prepared;
if (!prepare_scanner(config, 0, &prepared))
return 1;
prepared.options.use_metadata = true; /* capture mode + mtime for the listing */
DirectoryScanner* scanner =
directory_scanner_create_with_options(config->send_directory, &prepared.options);
if (!scanner) {
prepared_scanner_destroy(&prepared);
return 1;
}
ListEntry* entries = NULL;
size_t count = 0;
size_t capacity = 0;
@@ -378,6 +490,7 @@ static int send_list_only(const Config* config) {
}
bool failed = oom || directory_scanner_failed(scanner);
directory_scanner_destroy(scanner);
prepared_scanner_destroy(&prepared);
if (failed) {
list_entries_destroy(entries, count);
if (oom)
@@ -770,14 +883,20 @@ static int send_chunks_multithreaded(void* pipeline_context) {
static int scan_directory_multithreaded(void* pipeline_context) {
PipelineContextSender* context = (PipelineContextSender*)pipeline_context;
protocol_session_bind(&context->allocation_session);
ScannerOptions options = scanner_options_from_config(context->config, 4);
PreparedScanner prepared;
if (!prepare_scanner(context->config, 4, &prepared)) {
pipeline_cancel(context);
protocol_session_unbind();
return thrd_error;
}
ParallelScanner* scanner = parallel_scanner_create_with_options(
context->config->send_directory, &options, &context->allocation_session);
context->config->send_directory, &prepared.options, &context->allocation_session);
Chunk* current_chunk;
if (scanner == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to create parallel scanner");
pipeline_cancel(context);
prepared_scanner_destroy(&prepared);
protocol_session_unbind();
return thrd_error;
}
@@ -790,6 +909,7 @@ static int scan_directory_multithreaded(void* pipeline_context) {
pipeline_cancel(context);
chunk_destroy(current_chunk);
parallel_scanner_destroy(scanner);
prepared_scanner_destroy(&prepared);
protocol_session_unbind();
return thrd_error;
}
@@ -801,12 +921,14 @@ static int scan_directory_multithreaded(void* pipeline_context) {
chunk_destroy(current_chunk);
pipeline_cancel(context);
parallel_scanner_destroy(scanner);
prepared_scanner_destroy(&prepared);
protocol_session_unbind();
return thrd_error;
}
}
if (parallel_scanner_failed(scanner)) {
parallel_scanner_destroy(scanner);
prepared_scanner_destroy(&prepared);
mtx_lock(&context->mutex_scanner);
context->scanner_done = true;
cnd_broadcast(&context->condition_not_empty_scanner);
@@ -822,6 +944,7 @@ static int scan_directory_multithreaded(void* pipeline_context) {
mtx_unlock(&context->mutex_scanner);
parallel_scanner_destroy(scanner);
prepared_scanner_destroy(&prepared);
protocol_session_unbind();
return thrd_success;
}
@@ -923,6 +1046,8 @@ int send_files(Config* config) {
return send_list_only(config);
if (config->dry_run)
return send_dry_run_manifest(config);
if (!files_from_list_valid(config))
return 1;
Client* client = connect_transfer_client(config);
if (!client) {
@@ -939,10 +1064,13 @@ int send_files(Config* config) {
DirectoryScanner* scanner = NULL;
ArrayList* manifest = NULL;
ArrayList* remove_sources = NULL;
PreparedScanner prepared;
memset(&prepared, 0, sizeof(prepared));
if (!config_send(client->file_descriptor, config))
goto send_fail;
ScannerOptions scanner_options = scanner_options_from_config(config, 0);
scanner = directory_scanner_create_with_options(config->send_directory, &scanner_options);
if (!prepare_scanner(config, 0, &prepared))
goto send_fail;
scanner = directory_scanner_create_with_options(config->send_directory, &prepared.options);
manifest = create_transfer_manifest(config);
if (config->remove_source_files)
remove_sources = array_list_create(source_file_destroy);
@@ -1043,6 +1171,7 @@ send_fail:
array_list_delete(remove_sources);
if (scanner)
directory_scanner_destroy(scanner);
prepared_scanner_destroy(&prepared);
disconnect_transfer_client(client);
protocol_session_unbind();
return ret;
@@ -1056,6 +1185,8 @@ int send_files_multithreaded(Config** config_ptr) {
return send_list_only(config);
if (config->dry_run)
return send_dry_run_manifest(config);
if (!files_from_list_valid(config))
return 1;
long pages = sysconf(_SC_AVPHYS_PAGES);
long page_size = sysconf(_SC_PAGE_SIZE);
+11
View File
@@ -1,4 +1,5 @@
#include "client_validation.h"
#include "delay_updates.h"
#include "log.h"
#include "usage.h"
#include <stdio.h>
@@ -60,5 +61,15 @@ bool validate_config(const Config* config) {
return false;
}
}
if (config->delay_updates && config->inplace) {
log_message(LOG_LEVEL_ERROR, "--delay-updates does not work with --inplace");
return false;
}
if (config->delay_updates && delay_updates_staging_name_conflict(config->backup_dir)) {
log_message(LOG_LEVEL_ERROR,
"--backup-dir is reserved when --delay-updates is active (used for the internal "
"staging directory)");
return false;
}
return true;
}
+296 -24
View File
@@ -17,8 +17,62 @@
typedef struct {
char* path;
int depth;
FilterNode* context; /* inherited per-directory filter context */
} DirEntry;
/* A chain node: `own` holds the .rsync-filter rules of one directory, `parent`
* the context that directory inherited (nearest ancestor with a filter file).
* The chain for a directory's contents runs from that directory's own node up
* to the root; the command-line base rules are evaluated after the whole
* chain. */
struct FilterNode {
FilterNode* parent;
FilterRuleList* own;
};
static void filter_node_destroy(void* item) {
if (item) {
FilterNode* node = (FilterNode*)item;
if (node->own)
filter_rule_list_free(node->own);
free(node);
}
}
static FilterNode* filter_node_alloc(FilterNode* parent, FilterRuleList* own) {
FilterNode* node = malloc(sizeof(FilterNode));
if (!node)
return NULL;
node->parent = parent;
node->own = own;
return node;
}
/* Evaluate a rule chain for an entry inside the directory whose content
* context is `node`. rsync precedence, highest first: the innermost (current)
* directory's .rsync-filter rules, then each ancestor's, then the root's, and
* finally the command-line base rules (--filter/-C). A deeper per-directory
* file therefore overrides a shallower one, and per-directory files override
* the base rules by default. Returns FILTER_ACTION_NONE when nothing matched. */
static FilterAction chain_rules_apply(const FilterRuleList* base, const FilterNode* node,
const char* rel, const char* leaf, bool is_dir) {
if (node) {
FilterAction own_action = filter_rules_apply(node->own, rel, leaf, is_dir);
if (own_action != FILTER_ACTION_NONE)
return own_action;
return chain_rules_apply(base, node->parent, rel, leaf, is_dir);
}
return base ? filter_rules_apply(base, rel, leaf, is_dir) : FILTER_ACTION_NONE;
}
static bool entry_allowed(const FilterRuleList* base, const FilterNode* node, const char* rel,
const char* leaf, bool is_dir, bool per_dir_filters) {
/* -F: per-directory .rsync-filter files are never transferred. */
if (per_dir_filters && !is_dir && strcmp(leaf, ".rsync-filter") == 0)
return false;
return chain_rules_apply(base, node, rel, leaf, is_dir) != FILTER_ACTION_EXCLUDE;
}
static void dir_entry_destroy(void* item) {
if (item) {
DirEntry* de = (DirEntry*)item;
@@ -27,7 +81,7 @@ static void dir_entry_destroy(void* item) {
}
}
static DirEntry* dir_entry_create(const char* path, int depth) {
static DirEntry* dir_entry_create(const char* path, int depth, FilterNode* context) {
DirEntry* de = malloc(sizeof(DirEntry));
if (!de)
return NULL;
@@ -37,6 +91,7 @@ static DirEntry* dir_entry_create(const char* path, int depth) {
return NULL;
}
de->depth = depth;
de->context = context;
return de;
}
@@ -66,6 +121,79 @@ bool scanner_same_filesystem(bool one_file_system, dev_t root_device, dev_t entr
return !one_file_system || entry_device == root_device;
}
/* Relative path of an on-disk path below `root`. The transfer root may be
* given with a trailing slash; the returned rel path never has one and is ""
* for the root itself. A root of "/" is handled (its children start at "/").
* Exposed so tests can exercise the mapping directly. */
char* scanner_path_relative(const char* root, const char* fs_path) {
size_t root_len = strlen(root);
while (root_len > 1 && root[root_len - 1] == '/')
root_len--;
if (strncmp(root, fs_path, root_len) != 0)
return NULL;
if (root_len == 1 && root[0] == '/') {
if (fs_path[1] == '\0')
return str_dup("");
return str_dup(fs_path + 1);
}
if (fs_path[root_len] == '\0')
return str_dup("");
if (fs_path[root_len] != '/')
return NULL;
return str_dup(fs_path + root_len + 1);
}
/* Relative path of a child entry below the current directory. */
static char* child_rel_path(const char* parent_rel, const char* name) {
if (!parent_rel || parent_rel[0] == '\0')
return str_dup(name);
return path_cat(parent_rel, name);
}
/* Apply the --files-from allow-set and the filter layer to one entry. */
static bool entry_passes_selection(const FileListSet* file_list, const FilterRuleList* base,
const FilterNode* node, const char* rel, const char* leaf,
bool is_dir, bool per_dir_filters) {
if (file_list && !file_list_affects(file_list, rel))
return false;
if (base || per_dir_filters)
return entry_allowed(base, node, rel, leaf, is_dir, per_dir_filters);
return true;
}
/* Merge the open directory's own .rsync-filter rules into the inherited
* context, returning the context used for this directory's entries. On a parse
* error the scanner is marked failed. Returns 0 on success, -1 on failure. */
static int open_directory_filter_context(DirectoryScanner* scanner, const FilterNode* inherited) {
if (!scanner->per_dir_filters) {
scanner->current_node = (FilterNode*)inherited;
return 0;
}
char err[256];
bool exists = false;
FilterRuleList* own =
filter_file_read(scanner->current_path, scanner->current_rel ? scanner->current_rel : "",
&exists, err, sizeof(err));
if (!own) {
log_message(LOG_LEVEL_ERROR, "invalid .rsync-filter in %s: %s", scanner->current_path, err);
scanner->failed = true;
return -1;
}
if (exists && own->count > 0) {
FilterNode* node = filter_node_alloc((FilterNode*)inherited, own);
if (!node || !array_list_add(scanner->filter_nodes, node)) {
filter_node_destroy(node);
scanner->failed = true;
return -1;
}
scanner->current_node = node;
} else {
filter_rule_list_free(own);
scanner->current_node = (FilterNode*)inherited;
}
return 0;
}
/* Inspect symlinks, resolve the entry type, and apply file filters once for both scanners. */
static int scanner_inspect_entry(const ScannerOptions* options, const char* source_root,
const char* containing_dir, const char* name,
@@ -165,25 +293,54 @@ DirectoryScanner* directory_scanner_create_with_options(const char* root_directo
scanner->checksum = options->checksum;
scanner->one_file_system = options->one_file_system;
scanner->failed = false;
scanner->root_path = str_dup(root_directory);
if (!scanner->root_path) {
queue_destroy(scanner->directories);
free(scanner);
return NULL;
}
scanner->current_rel = NULL;
scanner->at_seed_dir = true;
scanner->seed_node = NULL;
scanner->current_node = NULL;
scanner->file_list = options->file_list;
scanner->base_filters = options->base_filters;
scanner->per_dir_filters = options->per_dir_filters;
scanner->filter_nodes = NULL;
if (scanner->base_filters || scanner->per_dir_filters) {
scanner->filter_nodes = array_list_create(filter_node_destroy);
if (!scanner->filter_nodes) {
free(scanner->root_path);
queue_destroy(scanner->directories);
free(scanner);
return NULL;
}
}
if (scanner->one_file_system) {
struct stat root_stats;
if (stat(root_directory, &root_stats) != 0) {
log_perror("Could not stat source directory");
free(scanner->root_path);
queue_destroy(scanner->directories);
array_list_delete(scanner->filter_nodes);
free(scanner);
return NULL;
}
scanner->root_dev = root_stats.st_dev;
}
DirEntry* root = dir_entry_create(root_directory, 0);
DirEntry* root = dir_entry_create(root_directory, 0, NULL);
if (!root) {
free(scanner->root_path);
queue_destroy(scanner->directories);
array_list_delete(scanner->filter_nodes);
free(scanner);
return NULL;
}
if (!queue_enqueue(scanner->directories, root)) {
dir_entry_destroy(root);
free(scanner->root_path);
queue_destroy(scanner->directories);
array_list_delete(scanner->filter_nodes);
free(scanner);
return NULL;
}
@@ -197,14 +354,25 @@ DirectoryScanner* directory_scanner_create(const char* root_directory, bool use_
unsigned long long min_size, int max_depth,
bool follow_symlinks, bool copy_links, bool safe_links,
bool copy_unsafe_links, bool checksum) {
ScannerOptions options = {use_metadata, chunk_size,
exclude_patterns, exclude_count,
include_patterns, include_count,
max_size, min_size,
max_depth, 0,
follow_symlinks, copy_links,
safe_links, copy_unsafe_links,
checksum, false};
ScannerOptions options = {use_metadata,
chunk_size,
exclude_patterns,
exclude_count,
include_patterns,
include_count,
max_size,
min_size,
max_depth,
0,
follow_symlinks,
copy_links,
safe_links,
copy_unsafe_links,
checksum,
false,
NULL,
NULL,
false};
return directory_scanner_create_with_options(root_directory, &options);
}
@@ -216,6 +384,9 @@ void directory_scanner_destroy(DirectoryScanner* scanner) {
scanner->current_dir = NULL;
}
free(scanner->current_path);
free(scanner->current_rel);
free(scanner->root_path);
array_list_delete(scanner->filter_nodes);
queue_destroy(scanner->directories);
free(scanner);
}
@@ -246,7 +417,21 @@ static int open_next_directory(DirectoryScanner* scanner) {
DirEntry* de = (DirEntry*)queue_dequeue(scanner->directories);
scanner->current_path = de->path;
scanner->current_depth = de->depth;
/* The seed directory inherits the scanner's configured context (the root
* .rsync-filter context in parallel mode); other dirs inherit the context of
* the directory that enqueued them. */
const FilterNode* inherited = scanner->at_seed_dir ? scanner->seed_node : de->context;
scanner->at_seed_dir = false;
free(de);
free(scanner->current_rel);
scanner->current_rel = scanner_path_relative(scanner->root_path, scanner->current_path);
if (!scanner->current_rel) {
log_message(LOG_LEVEL_ERROR, "Could not compute relative path under %s", scanner->root_path);
scanner->failed = true;
return -1;
}
scanner->current_dir = opendir(scanner->current_path);
if (scanner->current_dir == NULL) {
log_perror("Could not open directory");
@@ -255,6 +440,11 @@ static int open_next_directory(DirectoryScanner* scanner) {
scanner->failed = true;
return -1;
}
if (open_directory_filter_context(scanner, inherited) != 0) {
closedir(scanner->current_dir);
scanner->current_dir = NULL;
return -1;
}
return 1;
}
@@ -294,7 +484,9 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
scanner->max_depth, 0,
scanner->follow_symlinks, scanner->copy_links,
scanner->safe_links, scanner->copy_unsafe_links,
scanner->checksum, scanner->one_file_system};
scanner->checksum, scanner->one_file_system,
scanner->file_list, scanner->base_filters,
scanner->per_dir_filters};
ScannerEntry inspected;
int inspection = scanner_inspect_entry(&options, scanner->current_path, scanner->current_path,
entry->d_name, &inspected);
@@ -307,14 +499,32 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
char* cur_path = inspected.path;
struct stat stats = inspected.stats;
if (inspected.is_directory) {
/* --files-from allow-set and the filter layer apply to files and to
* directories (an excluded directory is not descended into). */
bool is_dir = inspected.is_directory;
char* rel = child_rel_path(scanner->current_rel, entry->d_name);
if (!rel) {
free(cur_path);
scanner->failed = true;
break;
}
bool passes_selection =
entry_passes_selection(scanner->file_list, scanner->base_filters, scanner->current_node,
rel, entry->d_name, is_dir, scanner->per_dir_filters);
free(rel);
if (!passes_selection) {
free(cur_path);
continue;
}
if (is_dir) {
if (!scanner_same_filesystem(scanner->one_file_system, scanner->root_dev, stats.st_dev)) {
free(cur_path);
continue;
}
int next_depth = scanner->current_depth + 1;
if (scanner->max_depth <= 0 || next_depth < scanner->max_depth) {
DirEntry* de = dir_entry_create(cur_path, next_depth);
DirEntry* de = dir_entry_create(cur_path, next_depth, scanner->current_node);
if (!de || !queue_enqueue(scanner->directories, de)) {
dir_entry_destroy(de);
scanner->failed = true;
@@ -376,6 +586,7 @@ typedef struct {
ParallelScanner* ps;
char** dirs;
int dir_count;
char* root_dir; /* the transfer root, for relative-path computation */
ScannerOptions options;
ProtocolSession* allocation_session;
} ParallelWorkerArg;
@@ -398,6 +609,13 @@ static int parallel_worker_thread(void* arg) {
free(wa->dirs[j]);
break;
}
/* Root .rsync-filter rules (parsed by the parallel scanner) apply to the
* contents of every assigned subdirectory. Relative paths (used by the
* allow-set and per-directory rules) are computed against the transfer
* root, not the subdirectory the worker is seeded with. */
free(ds->root_path);
ds->root_path = str_dup(wa->root_dir);
ds->seed_node = wa->ps->root_filter_node;
Chunk* chunk;
while ((chunk = directory_scanner_next(ds)) != NULL) {
if (!queue_enqueue_multithreaded_cancel(wa->ps->result_queue, chunk, &wa->ps->result_mutex,
@@ -419,6 +637,7 @@ static int parallel_worker_thread(void* arg) {
free(wa->dirs[i]);
}
ParallelScanner* ps = wa->ps;
free(wa->root_dir);
free(wa->dirs);
free(wa);
mtx_lock(&ps->result_mutex);
@@ -549,9 +768,10 @@ static Chunk* batch_files(ArrayList* files, unsigned long long chunk_size, Queue
}
/* Scan one root-directory entry into either the subdirs or files list. */
static void scan_root_entry(const ScannerOptions* options, const char* root_directory,
const struct dirent* entry, ArrayList* root_files, ArrayList* subdirs,
dev_t root_dev, ParallelScanner* ps) {
static void scan_root_entry(const ScannerOptions* options, const FilterNode* root_node,
const char* root_directory, const struct dirent* entry,
ArrayList* root_files, ArrayList* subdirs, dev_t root_dev,
ParallelScanner* ps) {
ScannerEntry inspected;
int inspection =
scanner_inspect_entry(options, root_directory, root_directory, entry->d_name, &inspected);
@@ -563,7 +783,21 @@ static void scan_root_entry(const ScannerOptions* options, const char* root_dire
return;
char* cur_path = inspected.path;
struct stat st = inspected.stats;
if (inspected.is_directory) {
bool is_dir = inspected.is_directory;
char* rel = str_dup(entry->d_name);
if (!rel) {
free(cur_path);
ps->failed = true;
return;
}
bool passes = entry_passes_selection(options->file_list, options->base_filters, root_node, rel,
entry->d_name, is_dir, options->per_dir_filters);
free(rel);
if (!passes) {
free(cur_path);
return;
}
if (is_dir) {
if (!scanner_same_filesystem(options->one_file_system, root_dev, st.st_dev)) {
free(cur_path);
return;
@@ -597,8 +831,8 @@ static void scan_root_entry(const ScannerOptions* options, const char* root_dire
/* Scan the root directory itself, collecting root files and subdirectories.
* Returns false if the root directory could not be opened. */
static bool scan_root_directory(ParallelScanner* ps, const char* root_directory,
const ScannerOptions* options, dev_t root_dev,
ArrayList* root_files, ArrayList* subdirs) {
const ScannerOptions* options, const FilterNode* root_node,
dev_t root_dev, ArrayList* root_files, ArrayList* subdirs) {
DIR* dir = opendir(root_directory);
if (!dir) {
log_perror("Could not open root directory for parallel scan");
@@ -608,7 +842,7 @@ static bool scan_root_directory(ParallelScanner* ps, const char* root_directory,
while ((entry = readdir(dir)) != NULL) {
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
continue;
scan_root_entry(options, root_directory, entry, root_files, subdirs, root_dev, ps);
scan_root_entry(options, root_node, root_directory, entry, root_files, subdirs, root_dev, ps);
}
closedir(dir);
return true;
@@ -616,7 +850,8 @@ static bool scan_root_directory(ParallelScanner* ps, const char* root_directory,
/* Spawn worker threads, one per group of subdirectories. */
static void spawn_parallel_workers(ParallelScanner* ps, ArrayList* subdirs,
const ScannerOptions* options, unsigned long long cs) {
const ScannerOptions* options, const char* root_directory,
unsigned long long cs) {
if (subdirs->size <= 0)
return;
int n = options->num_threads > 0 ? options->num_threads : 4;
@@ -647,7 +882,10 @@ static void spawn_parallel_workers(ParallelScanner* ps, ArrayList* subdirs,
}
wa->ps = ps;
wa->dirs = calloc(count, sizeof(char*));
if (!wa->dirs) {
wa->root_dir = str_dup(root_directory);
if (!wa->dirs || !wa->root_dir) {
free(wa->root_dir);
free(wa->dirs);
free(wa);
parallel_scanner_creation_failed(ps);
break;
@@ -661,6 +899,7 @@ static void spawn_parallel_workers(ParallelScanner* ps, ArrayList* subdirs,
if (!dup_ok) {
for (int j = 0; j < count; j++)
free(wa->dirs[j]);
free(wa->root_dir);
free(wa->dirs);
free(wa);
parallel_scanner_creation_failed(ps);
@@ -674,6 +913,7 @@ static void spawn_parallel_workers(ParallelScanner* ps, ArrayList* subdirs,
if (thrd_create(&ps->threads[t], parallel_worker_thread, wa) != thrd_success) {
for (int j = 0; j < count; j++)
free(wa->dirs[j]);
free(wa->root_dir);
free(wa->dirs);
free(wa);
parallel_scanner_creation_failed(ps);
@@ -720,7 +960,37 @@ ParallelScanner* parallel_scanner_create_with_options(const char* root_directory
root_dev = root_stats.st_dev;
}
if (!scan_root_directory(ps, root_directory, options, root_dev, root_files, subdirs)) {
/* Build the root directory's .rsync-filter context once; workers seed their
* scanners with it so per-dir rules behave identically to the sequential
* scanner. */
FilterNode* root_node = NULL;
if (options->per_dir_filters) {
char err[256];
bool exists = false;
FilterRuleList* own = filter_file_read(root_directory, "", &exists, err, sizeof(err));
if (!own) {
log_message(LOG_LEVEL_ERROR, "invalid .rsync-filter in %s: %s", root_directory, err);
array_list_delete(root_files);
array_list_delete(subdirs);
parallel_scanner_destroy(ps);
return NULL;
}
if (exists && own->count > 0) {
root_node = filter_node_alloc(NULL, own);
if (!root_node) {
filter_rule_list_free(own);
array_list_delete(root_files);
array_list_delete(subdirs);
parallel_scanner_destroy(ps);
return NULL;
}
} else {
filter_rule_list_free(own);
}
}
ps->root_filter_node = root_node;
if (!scan_root_directory(ps, root_directory, options, root_node, root_dev, root_files, subdirs)) {
array_list_delete(root_files);
array_list_delete(subdirs);
parallel_scanner_destroy(ps);
@@ -731,7 +1001,7 @@ ParallelScanner* parallel_scanner_create_with_options(const char* root_directory
ps->initial_chunk = batch_files(root_files, cs, ps->result_queue, &ps->failed);
array_list_delete(root_files);
spawn_parallel_workers(ps, subdirs, options, cs);
spawn_parallel_workers(ps, subdirs, options, root_directory, cs);
array_list_delete(subdirs);
return ps;
}
@@ -774,6 +1044,8 @@ void parallel_scanner_destroy(ParallelScanner* ps) {
for (int i = 0; i < ps->num_threads; i++)
thrd_join(ps->threads[i], NULL);
free(ps->threads);
if (ps->root_filter_node)
filter_node_destroy(ps->root_filter_node);
if (ps->initial_chunk)
chunk_destroy(ps->initial_chunk);
queue_destroy(ps->result_queue);
+28
View File
@@ -2,6 +2,8 @@
#define SCANNER_H
#include "chunk.h"
#include "file_list.h"
#include "filter.h"
#include "protocol.h"
#include "queue.h"
#include <dirent.h>
@@ -27,8 +29,18 @@ typedef struct {
bool copy_unsafe_links;
bool checksum;
bool one_file_system;
/* Phase 2 (files-from / filter layer). All pointers are shared read-only
* across scanner instances and worker threads; ownership stays with the
* caller (client_send). */
const FileListSet* file_list; /* --files-from allow-set, or NULL */
const FilterRuleList* base_filters; /* command-line + -C rules, or NULL */
bool per_dir_filters; /* -F: read .rsync-filter per directory */
} ScannerOptions;
/* Internal per-scanner filter state. FilterNode chains represent the ordered
* per-directory .rsync-filter rules that apply below a directory. */
typedef struct FilterNode FilterNode;
typedef struct {
Queue* directories;
DIR* current_dir;
@@ -51,6 +63,16 @@ typedef struct {
bool one_file_system;
dev_t root_dev;
bool failed;
/* Phase 2 (files-from / filter layer). */
char* root_path; /* transfer root (fs path) for rel computation */
char* current_rel; /* rel path of the open directory ("" == root) */
bool at_seed_dir; /* next open is the seed directory */
FilterNode* seed_node; /* inherited context of the seed dir, or NULL */
FilterNode* current_node; /* filter context of the open directory */
ArrayList* filter_nodes; /* owned FilterNode arena (may be NULL) */
const FileListSet* file_list;
const FilterRuleList* base_filters;
bool per_dir_filters;
} DirectoryScanner;
typedef struct {
@@ -68,6 +90,7 @@ typedef struct {
int completed;
Chunk* initial_chunk;
ProtocolSession* allocation_session;
FilterNode* root_filter_node; /* root .rsync-filter context (owned by ps) */
} ParallelScanner;
DirectoryScanner* directory_scanner_create(const char* root_directory, bool use_metadata,
@@ -88,6 +111,11 @@ void directory_scanner_destroy(DirectoryScanner* scanner);
* the transfer root. Exposed so tests can exercise the rule directly. */
bool scanner_same_filesystem(bool one_file_system, dev_t root_device, dev_t entry_device);
/* Relative path of an on-disk path below `root` ("" == the root itself, NULL
* when `fs_path` is not under `root`). Handles trailing slashes and a root of
* "/". Exposed so tests can exercise the mapping directly. */
char* scanner_path_relative(const char* root, const char* fs_path);
ParallelScanner* parallel_scanner_create_with_options(const char* root_directory,
const ScannerOptions* options,
ProtocolSession* allocation_session);
+8
View File
@@ -25,6 +25,7 @@ void print_usage(void) {
printf(" -8, --8-bit-output Leave high-bit characters unescaped in output\n");
printf(" --delete Delete files on receiver not in source\n");
printf(" --ignore-existing Skip files that already exist on receiver\n");
printf(" --delay-updates Put updated files into place only at the end of transfer\n");
printf(
" --dirs, --old-dirs, --old-d Transfer directories without recursing (not implemented)\n");
printf(" --del Alias for --delete-during (not implemented)\n");
@@ -32,6 +33,13 @@ void print_usage(void) {
printf(" --include <pattern> Only include files matching pattern\n");
printf(" --exclude-from <file> Read exclude patterns from file\n");
printf(" --include-from <file> Read include patterns from file\n");
printf(" --files-from <file> Read the source file list from FILE (paths relative to the "
"source root)\n");
printf(" -0, --from0 Entries in --files-from are NUL-delimited\n");
printf(" --filter=RULE rsync-style filter rule (+/- include/exclude; repeatable; the\n");
printf(" rsync -f short form conflicts with FastSync sendfile -f)\n");
printf(" -C, --cvs-exclude Auto-ignore common CVS/SCM files (.git/, .svn/, *.o, *~, ...)\n");
printf(" -F Apply per-directory .rsync-filter files during the scan\n");
printf(" --max-size <n> Skip files larger than n bytes\n");
printf(" --min-size <n> Skip files smaller than n bytes\n");
printf(" --max-alloc <SIZE> Maximum single allocation (default: 1G)\n");
+15
View File
@@ -1,6 +1,8 @@
#include "receiver.h"
#include "chunk.h"
#include "config.h"
#include "delay_updates.h"
#include "file_receive.h"
#include "log.h"
#include "metadata.h"
@@ -217,6 +219,17 @@ static bool receiver_save_file(File* file, void* context_pointer) {
static bool receiver_send_success_frame(int fd, void* context_pointer) {
ReceiverSaveContext* context = context_pointer;
/* --delay-updates: the whole protocol stream (including manifest/delete
handling, which ran inside receiver_process) has succeeded and every
staged file was fully written. Publish them atomically now, before the
success/outcome frame tells a --remove-source-files sender it may delete
its sources. */
if (context->config->delay_updates && context->config->delay_context) {
if (!delay_updates_publish(context->config->delay_context, context->config)) {
send_status(fd, STATUS_ERROR);
return false;
}
}
return receiver_send_final_success(fd, context->config, &context->outcomes);
}
@@ -224,6 +237,8 @@ int receiver_receive_files(Config* config, int file_descriptor) {
ReceiverSaveContext context = {.config = config, .outcomes = {0}};
ReceiverSink sink = {receiver_save_file, &context, true, true, receiver_send_success_frame};
int ret = receiver_process(config, file_descriptor, &sink);
if (ret != 0 && config->delay_updates && config->delay_context)
delay_updates_cleanup(config->delay_context);
receiver_outcomes_destroy(&context.outcomes);
return ret;
}
+30 -1
View File
@@ -1,4 +1,5 @@
#include "config.h"
#include "delay_updates.h"
#include "file.h"
#include "log.h"
#include "multiprocessing.h"
@@ -164,6 +165,20 @@ void handler(int file_descriptor) {
return;
}
config->use_delete = config->use_delete && allow_delete;
/* A --delay-updates transfer stages under a private 0700 directory inside
the receive root. Create it up front (wiping leftovers of any previously
interrupted delayed transfer) so a fully-skipped run also starts clean. */
if (config->delay_updates) {
config->delay_context = delay_updates_context_create(config->receive_root_directory);
if (!config->delay_context || !delay_updates_prepare(config->delay_context)) {
log_message(LOG_LEVEL_ERROR, "Failed to initialize --delay-updates staging area");
delay_updates_cleanup(config->delay_context);
config_delete(config);
close(file_descriptor);
protocol_session_unbind();
return;
}
}
if (config->use_multithreading) {
Queue* q = queue_create(100, file_destroy);
if (q == NULL) {
@@ -214,14 +229,28 @@ void handler(int file_descriptor) {
thrd_join(receiver, &receiver_result);
thrd_join(writer, &writer_result);
bool transfer_ok = receiver_result == thrd_success && writer_result == thrd_success;
if (transfer_ok) {
/* --delay-updates: receive_thread has finished the whole protocol stream
(including manifest/delete handling) and write_thread has drained its
queue, so every staged file is complete. Publish atomically before the
success/outcome frame so a --remove-source-files sender only learns of
files that were actually installed. */
if (config->delay_updates && config->delay_context &&
!delay_updates_publish(config->delay_context, config)) {
transfer_ok = false;
}
}
if (transfer_ok) {
if (!receiver_send_final_success(file_descriptor, config, &context->outcomes))
transfer_ok = false;
} else {
send_status(file_descriptor, STATUS_ERROR);
}
if (!transfer_ok)
if (!transfer_ok) {
log_message(LOG_LEVEL_ERROR, "Transfer failed");
if (config->delay_updates && config->delay_context)
delay_updates_cleanup(config->delay_context);
}
pipeline_context_receiver_destroy(context);
} else {
if (receiver_receive_files(config, file_descriptor) != 0)
+24 -7
View File
@@ -1,6 +1,8 @@
#include "config.h"
#include "chmod.h"
#include "delay_updates.h"
#include "delta.h"
#include "file_list.h"
#include "log.h"
#include "protocol.h"
#include "utils.h"
@@ -84,6 +86,7 @@ static void config_set_defaults(Config* config) {
config->ignore_existing = false;
config->update = false;
config->inplace = false;
config->delay_updates = false;
config->use_fsync = false;
config->append = false;
config->append_verify = false;
@@ -92,7 +95,10 @@ static void config_set_defaults(Config* config) {
config->max_delete = 0;
config->filters = NULL;
config->files_from = NULL;
config->files_from_set = NULL;
config->from0 = false;
config->cvs_exclude = false;
config->per_dir_filter = false;
config->prune_empty_dirs = false;
config->one_file_system = false;
config->relative = false;
@@ -119,6 +125,7 @@ static void config_set_defaults(Config* config) {
config->skip_compress_suffixes = NULL;
config->skip_compress_count = 0;
config->skip_compress_set = false;
config->delay_context = NULL;
}
static bool valid_wire_bool(int value) {
@@ -152,6 +159,8 @@ static bool validate_received_config(const Config* config) {
valid_wire_bool(config->use_fsync) && valid_wire_bool(config->append_verify) &&
valid_wire_bool(config->delete_excluded) && valid_wire_bool(config->delete_after) &&
valid_wire_bool(config->relative) && valid_wire_bool(config->prune_empty_dirs) &&
valid_wire_bool(config->delay_updates) && !(config->delay_updates && config->inplace) &&
!(config->delay_updates && delay_updates_staging_name_conflict(config->backup_dir)) &&
valid_wire_bool(config->partial) && valid_wire_bool(config->delete_before) &&
valid_wire_bool(config->checksum) && valid_wire_bool(config->eight_bit_output) &&
!(config->skip_compress_set && config->use_chunk_serialization) &&
@@ -227,6 +236,7 @@ void config_delete(Config* config) {
free(config->out_format);
free(config->log_file_format);
free(config->files_from);
file_list_destroy((FileListSet*)config->files_from_set);
free(config->rsh_command);
free(config->rsync_path);
free(config->temp_dir);
@@ -248,6 +258,12 @@ void config_delete(Config* config) {
if (config->filters) {
array_list_delete(config->filters);
}
/* A --delay-updates staging tree is transient receiver state: remove any
leftovers on every exit path (success already emptied it). */
if (config->delay_context)
delay_updates_cleanup(config->delay_context);
delay_updates_context_destroy(config->delay_context);
config->delay_context = NULL;
free(config);
}
@@ -287,10 +303,11 @@ static bool send_file_options(int fd, const Config* c) {
static bool send_selection_options(int fd, const Config* c) {
return send_int(fd, c->ignore_existing) && send_int(fd, c->existing) && send_int(fd, c->update) &&
send_int(fd, c->inplace) && send_int(fd, c->append) && send_int(fd, c->use_fsync) &&
send_int(fd, c->append_verify) && send_int(fd, c->delete_excluded) &&
send_int(fd, c->delete_after) && send_n_data(fd, &c->max_delete, sizeof(c->max_delete)) &&
send_int(fd, c->relative) && send_int(fd, c->prune_empty_dirs);
send_int(fd, c->inplace) && send_int(fd, c->delay_updates) && send_int(fd, c->append) &&
send_int(fd, c->use_fsync) && send_int(fd, c->append_verify) &&
send_int(fd, c->delete_excluded) && send_int(fd, c->delete_after) &&
send_n_data(fd, &c->max_delete, sizeof(c->max_delete)) && send_int(fd, c->relative) &&
send_int(fd, c->prune_empty_dirs);
}
static bool send_skip_compress_options(int fd, const Config* c) {
@@ -382,9 +399,9 @@ static bool receive_file_options(int fd, Config* c) {
}
static bool receive_selection_options(int fd, Config* c) {
bool* flags[] = {&c->ignore_existing, &c->existing, &c->update,
&c->inplace, &c->append, &c->use_fsync,
&c->append_verify, &c->delete_excluded, &c->delete_after};
bool* flags[] = {&c->ignore_existing, &c->existing, &c->update, &c->inplace,
&c->delay_updates, &c->append, &c->use_fsync, &c->append_verify,
&c->delete_excluded, &c->delete_after};
for (size_t i = 0; i < sizeof(flags) / sizeof(flags[0]); i++) {
if (!receive_wire_bool(fd, flags[i]))
return false;
+18 -5
View File
@@ -8,6 +8,10 @@
typedef enum { TRANSPORT_TCP, TRANSPORT_SSH } TransportType;
/* Receiver-side staging state for --delay-updates. Forward-declared here so
Config can carry it; the concrete type lives in delay_updates.h. */
typedef struct DelayUpdatesContext DelayUpdatesContext;
typedef struct Config {
char* version;
char* send_directory;
@@ -91,6 +95,7 @@ typedef struct Config {
bool ignore_existing;
bool update;
bool inplace;
bool delay_updates;
bool use_fsync;
bool append;
bool append_verify;
@@ -100,10 +105,14 @@ typedef struct Config {
bool delete_after;
int max_delete;
// Issue #129: Advanced file selection
ArrayList* filters;
char* files_from;
bool cvs_exclude;
// Issue #129: Advanced file selection. These fields are CLIENT-ONLY: they are
// never serialized to the wire (the receiver must not learn them).
ArrayList* filters; /* --filter=RULE rule strings, in order */
char* files_from; /* --files-from path (may be NULL) */
void* files_from_set; /* parsed FileListSet* allow-set, or NULL */
bool from0; /* -0/--from0: NUL-delimited *-from files */
bool cvs_exclude; /* -C/--cvs-exclude: standard CVS ignore set */
bool per_dir_filter; /* -F: apply per-directory .rsync-filter files */
bool prune_empty_dirs;
bool one_file_system; /* -x/--one-file-system: do not cross filesystem boundaries */
bool relative;
@@ -147,9 +156,13 @@ typedef struct Config {
char** skip_compress_suffixes;
int skip_compress_count;
bool skip_compress_set;
// Receiver-side runtime staging registry for --delay-updates. Never sent
// over the wire and never set on the sender side.
DelayUpdatesContext* delay_context;
} Config;
#define PROTOCOL_VERSION "2.5.0"
#define PROTOCOL_VERSION "2.6.0"
#define DEFAULT_CHUNK_SIZE (10 * 1024 * 1024)
Config* config_create(void);
+338
View File
@@ -0,0 +1,338 @@
#include "delay_updates.h"
#include "config.h"
#include "file.h"
#include "log.h"
#include "utils.h"
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
#include <libgen.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/file.h>
#include <sys/stat.h>
#include <unistd.h>
DelayUpdatesContext* delay_updates_context_create(const char* root_directory) {
if (!root_directory)
return NULL;
DelayUpdatesContext* context = calloc(1, sizeof(DelayUpdatesContext));
if (!context)
return NULL;
context->root_directory = str_dup(root_directory);
if (!context->root_directory) {
free(context);
return NULL;
}
context->staging_root = path_cat(root_directory, DELAY_UPDATES_STAGING_DIR);
if (!context->staging_root) {
free(context->root_directory);
free(context);
return NULL;
}
context->entries = NULL;
context->count = 0;
context->capacity = 0;
context->prepared = false;
context->lock_fd = -1;
if (mtx_init(&context->mutex, mtx_plain) != thrd_success) {
free(context->staging_root);
free(context->root_directory);
free(context);
return NULL;
}
return context;
}
void delay_updates_context_destroy(DelayUpdatesContext* context) {
if (!context)
return;
mtx_destroy(&context->mutex);
if (context->lock_fd >= 0)
close(context->lock_fd);
context->lock_fd = -1;
free(context->staging_root);
free(context->root_directory);
for (size_t i = 0; i < context->count; i++) {
free(context->entries[i].staged_path);
free(context->entries[i].final_path);
free(context->entries[i].file_path);
}
free(context->entries);
free(context);
}
bool delay_updates_staging_name_conflict(const char* dir) {
if (!dir || !*dir)
return false;
size_t length = strlen(dir);
while (length > 0 && dir[length - 1] == '/')
length--;
size_t reserved_length = strlen(DELAY_UPDATES_STAGING_DIR);
if (length != reserved_length)
return false;
return strncmp(dir, DELAY_UPDATES_STAGING_DIR, length) == 0;
}
/* Recursively delete every entry inside an open directory (never following
symlinks). The directory itself is left in place. Mirrors the fd-relative
walk used by the delete code so a symlink planted inside the staging tree
can never redirect removal outside of it. */
static bool delay_wipe_dir_fd(int dirfd) {
int scanfd = dup(dirfd);
if (scanfd < 0)
return false;
DIR* dir = fdopendir(scanfd);
if (!dir) {
close(scanfd);
return false;
}
bool operation_ok = true;
const struct dirent* entry;
while ((entry = readdir(dir)) != NULL) {
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
continue;
struct stat st;
if (fstatat(dirfd, entry->d_name, &st, AT_SYMLINK_NOFOLLOW) != 0) {
if (errno != ENOENT)
operation_ok = false;
continue;
}
if (S_ISDIR(st.st_mode)) {
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_removed = false;
if (childfd >= 0) {
child_removed = delay_wipe_dir_fd(childfd);
close(childfd);
} else if (errno != ENOENT) {
operation_ok = false;
}
if (child_removed && unlinkat(dirfd, entry->d_name, AT_REMOVEDIR) != 0 && errno != ENOENT)
operation_ok = false;
} else {
if (unlinkat(dirfd, entry->d_name, 0) != 0 && errno != ENOENT)
operation_ok = false;
}
}
closedir(dir);
return operation_ok;
}
bool delay_updates_prepare(DelayUpdatesContext* context) {
if (!context)
return false;
if (context->prepared)
return true;
int fd = file_open_private_dir(context->staging_root);
if (fd < 0) {
int saved_errno = errno;
char* escaped = output_escape(context->staging_root, false);
log_message(LOG_LEVEL_ERROR, "could not create --delay-updates staging directory '%s': %s",
escaped ? escaped : "<allocation failed>", strerror(saved_errno));
free(escaped);
return false;
}
/* Hold an exclusive advisory lock on the staging directory for the whole
transfer. The staging directory name is fixed, so two simultaneous
delayed transfers to the same destination root would otherwise share it
and destroy each other's staged files. The lock makes the second session
fail cleanly instead of corrupting the first. The lock is released when
the context (and its file descriptor) is destroyed. */
if (flock(fd, LOCK_EX | LOCK_NB) != 0) {
int saved_errno = errno;
close(fd);
if (saved_errno == EWOULDBLOCK || saved_errno == EAGAIN) {
char* escaped = output_escape(context->staging_root, false);
log_message(LOG_LEVEL_ERROR,
"another --delay-updates transfer to '%s' is already in progress; refusing to "
"share the staging directory",
escaped ? escaped : "<allocation failed>");
free(escaped);
} else {
log_message(LOG_LEVEL_ERROR, "could not lock --delay-updates staging directory '%s': %s",
context->staging_root, strerror(saved_errno));
}
return false;
}
context->lock_fd = fd;
/* Only now, with exclusive ownership, wipe leftovers from an interrupted
earlier transfer; this can never race with a live session. */
bool ok = delay_wipe_dir_fd(fd);
if (!ok) {
log_message(LOG_LEVEL_ERROR, "could not clear stale --delay-updates staging files under '%s'",
context->staging_root);
close(context->lock_fd);
context->lock_fd = -1;
return false;
}
context->prepared = true;
return true;
}
bool delay_updates_record(DelayUpdatesContext* context, const char* staged_path,
const char* final_path, const char* file_path) {
if (!context || !staged_path || !final_path || !file_path)
return false;
char* staged_copy = str_dup(staged_path);
char* final_copy = str_dup(final_path);
char* file_copy = str_dup(file_path);
if (!staged_copy || !final_copy || !file_copy) {
free(staged_copy);
free(final_copy);
free(file_copy);
return false;
}
mtx_lock(&context->mutex);
bool ok = true;
if (context->count == context->capacity) {
size_t new_capacity = context->capacity == 0 ? 64 : context->capacity * 2;
if (new_capacity < context->capacity) {
ok = false;
} else {
StagedFileEntry* grown = realloc(context->entries, new_capacity * sizeof(StagedFileEntry));
if (!grown) {
ok = false;
} else {
context->entries = grown;
context->capacity = new_capacity;
}
}
}
if (ok) {
context->entries[context->count].staged_path = staged_copy;
context->entries[context->count].final_path = final_copy;
context->entries[context->count].file_path = file_copy;
context->count++;
}
mtx_unlock(&context->mutex);
if (!ok) {
free(staged_copy);
free(final_copy);
free(file_copy);
}
return ok;
}
/* Move an existing final destination file aside before the staged replacement
is installed. Deferred from stage time so the final destination is not
modified until publication. Mirrors the immediate-mode backup logic. */
static bool delay_publish_backup(const DelayUpdatesContext* context, const Config* config,
const StagedFileEntry* entry) {
bool backup_enabled = config && config->backup && !config->ignore_existing;
if (!backup_enabled)
return true;
const char* backup_suffix = (config && config->suffix) ? config->suffix : "~";
struct stat backup_stat;
if (!file_stat_secure(entry->final_path, &backup_stat))
return true; /* nothing to back up */
char* backup_path = NULL;
if (config->backup_dir) {
char* confined_backup = path_cat(context->root_directory, config->backup_dir);
if (!confined_backup)
return false;
backup_path = path_cat(confined_backup, entry->file_path);
free(confined_backup);
} else {
size_t path_len = strlen(entry->final_path);
size_t suffix_len = strlen(backup_suffix);
if (path_len > SIZE_MAX - suffix_len - 1)
return false;
backup_path = malloc(path_len + suffix_len + 1);
if (backup_path) {
memcpy(backup_path, entry->final_path, path_len);
memcpy(backup_path + path_len, backup_suffix, suffix_len + 1);
}
}
if (!backup_path)
return false;
char* parent_copy = str_dup(backup_path);
if (!parent_copy || !file_ensure_directory_secure(dirname(parent_copy))) {
free(parent_copy);
free(backup_path);
return false;
}
free(parent_copy);
bool ok = file_rename_secure(entry->final_path, backup_path);
free(backup_path);
return ok;
}
static bool delay_publish_entry(DelayUpdatesContext* context, const Config* config,
const StagedFileEntry* entry) {
if (!delay_publish_backup(context, config, entry))
return false;
if (!file_rename_secure(entry->staged_path, entry->final_path)) {
if (errno == EXDEV) {
char* escaped = output_escape(entry->final_path, false);
log_message(LOG_LEVEL_ERROR,
"staging directory is on a different filesystem than the destination; cannot "
"atomically install file (EXDEV): %s",
escaped ? escaped : "<allocation failed>");
free(escaped);
} else {
char* escaped = output_escape(entry->final_path, false);
log_message(LOG_LEVEL_ERROR, "could not install staged file '%s': %s",
escaped ? escaped : "<allocation failed>", strerror(errno));
free(escaped);
}
return false;
}
return true;
}
/* Remove the staging tree (contents plus the directory itself). Returns true
when nothing is left behind (including the case where it never existed). */
static bool delay_updates_remove_staging_tree(DelayUpdatesContext* context) {
int fd = open(context->staging_root, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (fd < 0)
return errno == ENOENT;
bool ok = delay_wipe_dir_fd(fd);
if (close(fd) != 0)
ok = false;
if (ok && rmdir(context->staging_root) != 0 && errno != ENOENT)
ok = false;
return ok;
}
bool delay_updates_publish(DelayUpdatesContext* context, const Config* config) {
if (!context)
return false;
mtx_lock(&context->mutex);
bool ok = true;
for (size_t i = 0; i < context->count; i++) {
if (!delay_publish_entry(context, config, &context->entries[i])) {
ok = false;
break;
}
}
mtx_unlock(&context->mutex);
/* Renaming files out of the staging tree leaves the mirrored directories
behind, and a mid-publish failure leaves the remaining staged files.
Remove whatever is left so a later run starts from a clean staging area
and no staged content can linger after a failed publish. If that cleanup
fails, tell the operator: a stale staging directory would otherwise
silently accumulate and make the next transfer's prepare-wipe fail. */
if (!delay_updates_remove_staging_tree(context)) {
log_message(LOG_LEVEL_WARNING,
"could not fully remove --delay-updates staging directory '%s' after publish; a "
"later --delay-updates transfer to this destination will try to clear it",
context->staging_root);
}
return ok;
}
void delay_updates_cleanup(DelayUpdatesContext* context) {
if (!context)
return;
/* Only a context that gained exclusive ownership may touch the shared
staging directory. If prepare never succeeded (e.g. lock contention with
another live session) the directory belongs to that other session and must
be left alone. */
if (!context->prepared)
return;
delay_updates_remove_staging_tree(context);
}
+68
View File
@@ -0,0 +1,68 @@
#ifndef DELAY_UPDATES_H
#define DELAY_UPDATES_H
#include <stdbool.h>
#include <stddef.h>
#include <threads.h>
/* Forward-declared in config.h; full type needed by file_save_to_disk. */
typedef struct Config Config;
/* One staged file awaiting publication. */
typedef struct {
char* staged_path; /* full path inside the staging tree */
char* final_path; /* full final destination path */
char* file_path; /* the file path as received on the wire */
} StagedFileEntry;
/* Receiver-side --delay-updates staging registry. All successfully written
files land under a private staging directory inside the receive root and are
atomically renamed into their final destination only at the very end of the
transfer. A single PipelineContextReceiver has exactly one writer thread,
but the registry is still mutex-protected so the same object can be safely
shared with the publish/cleanup phase that runs after the threads join. */
typedef struct DelayUpdatesContext {
char* root_directory; /* receive root the staging dir lives under */
char* staging_root; /* root_directory/<staging dir name> */
mtx_t mutex;
StagedFileEntry* entries;
size_t count;
size_t capacity;
bool prepared; /* staging dir created, wiped, and exclusively locked */
int lock_fd; /* advisory exclusive flock held on the staging dir, or -1 */
} DelayUpdatesContext;
/* Name of the private staging subdirectory created under the receive root. */
#define DELAY_UPDATES_STAGING_DIR ".fastsync-stage"
/* True when `dir` (ignoring a trailing "/") is the reserved staging directory
name. Used to reject a --backup-dir that would collide with the internal
staging area. */
bool delay_updates_staging_name_conflict(const char* dir);
/* Create an empty staging context rooted below root_directory. Does not touch
the filesystem yet. */
DelayUpdatesContext* delay_updates_context_create(const char* root_directory);
void delay_updates_context_destroy(DelayUpdatesContext* context);
/* Create the private 0700 staging directory (on first call) and wipe any
leftovers from a previously interrupted delayed transfer. Idempotent. */
bool delay_updates_prepare(DelayUpdatesContext* context);
/* Record a fully-written staged file for later publication. Copies all three
paths. Returns false on allocation failure. */
bool delay_updates_record(DelayUpdatesContext* context, const char* staged_path,
const char* final_path, const char* file_path);
/* Atomically rename every staged file into its final destination. Deferred
--backup handling runs immediately before each rename. On any failure the
remaining staged files are removed (best effort); already-published files
are not rolled back. Afterwards the staging tree is removed so a successful
or failed publish leaves no staging leftovers. */
bool delay_updates_publish(DelayUpdatesContext* context, const Config* config);
/* Best-effort removal of every staged file and the staging directory itself.
Safe to call when nothing was staged or after a successful publish. */
void delay_updates_cleanup(DelayUpdatesContext* context);
#endif
+11 -12
View File
@@ -338,23 +338,22 @@ bool file_rename_secure(const char* old_path, const char* new_path) {
return ok;
}
/* Open the configured --temp-dir scratch directory, creating it (and any
missing path components) on demand. scratch_path is expected to already be
confined below the authorized root by the caller; file_open_secure_parent
re-checks that confinement and rejects `..` components, so a scratch
directory can never be created or opened outside the destination root.
Returns an O_DIRECTORY|O_NOFOLLOW fd, or -1 on error. */
static int file_open_scratch_dir(const char* scratch_path) {
if (!scratch_path)
/* Open a private staging/scratch directory, creating it (and any missing path
components) on demand. dir_path is expected to already be confined below
the authorized root by the caller; file_open_secure_parent re-checks that
confinement and rejects `..` components, so a scratch directory can never be
created or opened outside the destination root. The directory itself is
created 0700 so other users cannot race on names inside it. Returns an
O_DIRECTORY|O_NOFOLLOW fd, or -1 on error. */
int file_open_private_dir(const char* dir_path) {
if (!dir_path)
return -1;
char* leaf = NULL;
int parent_fd = file_open_secure_parent(scratch_path, &leaf, true);
int parent_fd = file_open_secure_parent(dir_path, &leaf, true);
if (parent_fd < 0)
return -1;
int fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (fd < 0 && errno == ENOENT) {
/* A scratch directory holds transient working copies only; keep it
private (0700) so other users cannot race on temp names inside it. */
if (mkdirat(parent_fd, leaf, 0700) == 0 || errno == EEXIST)
fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
}
@@ -429,7 +428,7 @@ static bool file_to_disk_secure_impl(const char* path, const void* data,
file is created in the destination directory, exactly as historically. */
int scratch_dirfd = -1;
if (temp_dir) {
scratch_dirfd = file_open_scratch_dir(temp_dir);
scratch_dirfd = file_open_private_dir(temp_dir);
if (scratch_dirfd < 0) {
int saved_errno = errno;
log_message(LOG_LEVEL_ERROR, "could not open --temp-dir scratch directory '%s': %s",
+4
View File
@@ -31,6 +31,10 @@ bool file_destination_is_newer_secure(const char* path, const FileMetadata* meta
int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs);
bool file_ensure_directory_secure(const char* path);
bool file_rename_secure(const char* old_path, const char* new_path);
/* Open a private 0700 directory (creating it on demand) that must live below
the authorized root. Used for the --temp-dir scratch directory and the
--delay-updates staging directory. */
int file_open_private_dir(const char* dir_path);
/* The file_to_disk_secure* variants write a temporary copy in the destination
directory and atomically rename it over `path`. temp_dir is an absolute,
+191
View File
@@ -0,0 +1,191 @@
#include "file_list.h"
#include "log.h"
#include "utils.h"
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
typedef struct {
char** items;
int count;
int capacity;
} StringList;
static void string_list_destroy(StringList* list) {
if (!list)
return;
for (int i = 0; i < list->count; i++)
free(list->items[i]);
free(list->items);
}
static bool string_list_add(StringList* list, const char* text) {
if (list->count == list->capacity) {
int new_cap = list->capacity > 0 ? list->capacity * 2 : 16;
char** grown = realloc(list->items, (size_t)new_cap * sizeof(char*));
if (!grown)
return false;
list->items = grown;
list->capacity = new_cap;
}
list->items[list->count] = str_dup(text);
if (!list->items[list->count])
return false;
list->count++;
return true;
}
/* Validate and normalize one entry. Returns:
* 1 -> added to `out`
* 0 -> blank entry, skip
* -1 -> invalid (message set in `err`)
* `strip_line_endings` trims a trailing CR/LF (line mode only); NUL mode keeps
* the entry bytes verbatim so names ending in CR/LF survive. */
static int normalize_entry(const char* raw, size_t len, bool strip_line_endings, StringList* out,
char* err, size_t err_size) {
if (strip_line_endings) {
while (len > 0 && (raw[len - 1] == '\n' || raw[len - 1] == '\r'))
len--;
}
if (len == 0)
return 0;
if (raw[0] == '/') {
snprintf(err, err_size, "absolute path entries are not allowed: '%.*s'", (int)len, raw);
return -1;
}
/* Reject NUL bytes inside a token defensively (NUL-delimited mode splits on
* them, so this only guards against embedded garbage). */
char* dup = malloc(len + 1);
if (!dup) {
snprintf(err, err_size, "memory allocation failed");
return -1;
}
memcpy(dup, raw, len);
dup[len] = '\0';
/* Rebuild the path token-by-token: skip '.' and empty segments, reject '..'. */
size_t out_len = 0;
for (const char* part = dup;;) {
const char* slash = strchr(part, '/');
size_t part_len = slash ? (size_t)(slash - part) : strlen(part);
if (part_len == 1 && part[0] == '.') {
/* skip "." segment */
} else if (part_len == 2 && part[0] == '.' && part[1] == '.') {
snprintf(err, err_size, "path traversal entry is not allowed: '%s'", dup);
free(dup);
return -1;
} else if (part_len > 0) {
if (out_len > 0)
dup[out_len++] = '/';
memmove(dup + out_len, part, part_len);
out_len += part_len;
}
if (!slash)
break;
part = slash + 1;
}
dup[out_len] = '\0';
int result;
if (out_len == 0) {
/* "." / "./" lists the source root: the whole tree is transferred. */
result = string_list_add(out, "") ? 1 : -1;
if (result < 0)
snprintf(err, err_size, "memory allocation failed");
} else {
result = string_list_add(out, dup) ? 1 : -1;
if (result < 0)
snprintf(err, err_size, "memory allocation failed");
}
free(dup);
return result;
}
static FileListSet* string_list_to_set(StringList* raw, char* err, size_t err_size) {
FileListSet* set = malloc(sizeof(FileListSet));
if (!set) {
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
set->count = raw->count;
set->entries = raw->items;
raw->items = NULL;
raw->count = 0;
return set;
}
FileListSet* file_list_load(const char* path, bool null_separated, char* err, size_t err_size) {
if (err && err_size > 0)
err[0] = '\0';
if (!path || !*path) {
snprintf(err, err_size, "no file given");
return NULL;
}
FILE* fp = fopen(path, "r");
if (!fp) {
char* escaped = output_escape(path, false);
snprintf(err, err_size, "could not open '%s': %s", escaped ? escaped : path, strerror(errno));
free(escaped);
return NULL;
}
StringList raw = {0};
char* line = NULL;
size_t line_cap = 0;
ssize_t n;
bool ok = true;
char delim = null_separated ? '\0' : '\n';
while (ok && (n = getdelim(&line, &line_cap, delim, fp)) != -1) {
int r = normalize_entry(line, (size_t)n, !null_separated, &raw, err, err_size);
if (r < 0) {
ok = false;
break;
}
}
free(line);
fclose(fp);
if (!ok) {
string_list_destroy(&raw);
return NULL;
}
FileListSet* set = string_list_to_set(&raw, err, err_size);
if (!set)
string_list_destroy(&raw);
return set;
}
void file_list_destroy(FileListSet* set) {
if (!set)
return;
for (int i = 0; i < set->count; i++)
free(set->entries[i]);
free(set->entries);
free(set);
}
static bool path_has_prefix(const char* path, const char* prefix) {
size_t plen = strlen(prefix);
if (strncmp(path, prefix, plen) != 0)
return false;
return path[plen] == '/' || path[plen] == '\0';
}
bool file_list_affects(const FileListSet* set, const char* rel) {
if (!set)
return true;
if (!rel)
return false;
for (int i = 0; i < set->count; i++) {
const char* entry = set->entries[i];
if (entry[0] == '\0')
return true; /* whole tree listed */
if (strcmp(rel, entry) == 0)
return true; /* the entry itself is listed */
if (path_has_prefix(rel, entry))
return true; /* rel lives under a listed directory */
if (path_has_prefix(entry, rel))
return true; /* rel is an ancestor directory of a listed entry */
}
return false;
}
+35
View File
@@ -0,0 +1,35 @@
#ifndef FILE_LIST_H
#define FILE_LIST_H
#include <stdbool.h>
#include <stddef.h>
/* --files-from allow-set. The file lists source paths RELATIVE to the source
* root. A listed regular file is transferred; a listed directory transfers its
* whole subtree (FastSync's recursion is always on). Blank lines are ignored.
*
* Entries are normalized: leading "./" and duplicate "/" are removed, an entry
* of "." means the whole tree, absolute entries and ".." traversal are
* rejected at parse time. The set is immutable and shared read-only across
* scanner worker threads.
*/
typedef struct {
char** entries; /* normalized rel paths; "" means the whole tree */
int count;
} FileListSet;
/* Load and validate a --files-from file. When `null_separated` (-0/--from0)
* entries are delimited by NUL instead of newlines. Returns NULL with a message
* in `err` on open/validation failure. An empty file yields an empty set
* (nothing is transferred). */
FileListSet* file_list_load(const char* path, bool null_separated, char* err, size_t err_size);
void file_list_destroy(FileListSet* set);
/* True when `rel` (path relative to the source root, "" == root) is a listed
* entry, lives under a listed directory, or is an ancestor directory of a
* listed entry. Used to prune scanning: directories are descended only when
* this returns true, files are transferred only when it returns true. */
bool file_list_affects(const FileListSet* set, const char* rel);
#endif
+86 -2
View File
@@ -12,6 +12,7 @@
#include "compression.h"
#include "config.h"
#include "data.h"
#include "delay_updates.h"
#include "delta.h"
#include "file.h"
#include "log.h"
@@ -26,6 +27,72 @@ bool file_save_to_disk(const char* root_directory, const File* file, const Confi
return file_save_to_disk_full(root_directory, file, config) != FILE_SAVE_ERROR;
}
/* --delay-updates receiver path: write the file into a private staging tree
below the receive root instead of its final destination, and remember it so
it can be atomically renamed into place only once the whole transfer has
succeeded. Existence/update policies (--existing/--ignore-existing/--update)
are decided against the FINAL destination path at stage time so the run
decides exactly what an immediate (non-delayed) run would decide; the staged
file is then never re-checked at publication. Backups are deferred to
publication so the final destination is untouched until the transfer ends. */
static FileSaveResult file_stage_delayed_update(const char* root_directory,
const char* destination_path, const File* file,
Config* config) {
if (!config)
return FILE_SAVE_ERROR;
bool sparse = config->preserve_sparse;
bool preserve_executability = config->use_executability;
if (config->existing && !file_path_exists_secure(destination_path))
return FILE_SAVE_SKIPPED;
if (config->ignore_existing && file_path_exists_secure(destination_path))
return FILE_SAVE_SKIPPED;
if (config->update && file_destination_is_newer_secure(destination_path, file->metadata))
return FILE_SAVE_SKIPPED;
FileMetadata adjusted_metadata;
const FileMetadata* metadata = file->metadata;
if (metadata && config->chmod_spec && *config->chmod_spec) {
adjusted_metadata = *metadata;
if (!chmod_apply(adjusted_metadata.mode, config->chmod_spec, &adjusted_metadata.mode))
return FILE_SAVE_ERROR;
metadata = &adjusted_metadata;
}
if (!config->delay_context) {
config->delay_context = delay_updates_context_create(root_directory);
if (!config->delay_context)
return FILE_SAVE_ERROR;
}
DelayUpdatesContext* context = config->delay_context;
if (!delay_updates_prepare(context))
return FILE_SAVE_ERROR;
char* staged_path = path_cat(context->staging_root, file->path);
if (!staged_path)
return FILE_SAVE_ERROR;
/* The staged location is brand new (stale leftovers from a prior crash were
wiped by prepare), so the plain atomic temp+rename engine installs the
complete file there. --temp-dir scratch is deliberately not layered on
top of the delay-updates staging tree. */
bool ok =
file_to_disk_secure_with_fsync(staged_path, file->data->data, file->data->size, false, sparse,
metadata, preserve_executability, config->use_fsync, NULL);
if (!ok) {
free(staged_path);
return FILE_SAVE_ERROR;
}
if (!delay_updates_record(context, staged_path, destination_path, file->path)) {
unlink(staged_path);
free(staged_path);
return FILE_SAVE_ERROR;
}
free(staged_path);
return FILE_SAVE_WRITTEN;
}
FileSaveResult file_save_to_disk_full(const char* root_directory, const File* file,
const Config* config) {
/* Backups are incompatible with ignore-existing: moving the entry first
@@ -79,6 +146,18 @@ FileSaveResult file_save_to_disk_full(const char* root_directory, const File* fi
return FILE_SAVE_ERROR;
}
/* --delay-updates diverts the whole write into the staging tree; the rest of
this function is the immediate-install path. */
if (config && config->delay_updates) {
FileSaveResult result =
file_stage_delayed_update(root_directory, destination_path, file, (Config*)config);
free(confined_backup);
free(confined_partial);
free(destination_path);
free(disk_path);
return result;
}
/* --existing checks the final destination, not a temporary partial path. */
if (config && config->existing && !file_path_exists_secure(destination_path)) {
free(confined_backup);
@@ -718,8 +797,13 @@ int receive_manifest(int fd, const Config* config, int* next_status) {
return *status_out == STATUS_FINISHED ? 0 : -1;
}
fprintf(stderr, "Deleting files not in manifest...\n");
bool deletion_ok =
delete_extras_limited(config->receive_root_directory, manifest, MAX_SERVER_DELETE_COUNT);
/* With --delay-updates the staged (not yet published) files live directly
under the receive root in the staging directory; the delete walker must
not treat them as extras or it would remove every staged file before it
can be published. */
const char* skip_staging = config->delay_updates ? DELAY_UPDATES_STAGING_DIR : NULL;
bool deletion_ok = delete_extras_limited(config->receive_root_directory, manifest,
MAX_SERVER_DELETE_COUNT, skip_staging);
array_list_delete(manifest);
if (!deletion_ok)
send_status(fd, STATUS_ERROR);
+427
View File
@@ -0,0 +1,427 @@
#include "filter.h"
#include "log.h"
#include "utils.h"
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/* ---- Single rule parsing ---- */
static bool rule_text_is_unsupported_word(const char* p, size_t len) {
static const char* const words[] = {"merge", "dir-merge", "hide", "show",
"protect", "risk", "clear"};
for (size_t i = 0; i < sizeof(words) / sizeof(words[0]); i++) {
size_t wl = strlen(words[i]);
if (len == wl && strncmp(p, words[i], wl) == 0)
return true;
}
return false;
}
/* rsync include/exclude rule modifiers we do NOT implement. A rule whose +/- is
* immediately followed by one of these is rejected instead of being silently
* parsed as a literal pattern. */
static bool is_unsupported_rule_modifier(char c) {
return c == '!' || c == 'C' || c == 's' || c == 'r' || c == 'p' || c == 'x';
}
FilterRule* filter_rule_parse(const char* line, char* err, size_t err_size) {
if (err && err_size > 0)
err[0] = '\0';
if (!line)
return NULL;
char* text = str_dup(line);
if (!text) {
if (err)
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
size_t len = strlen(text);
while (len > 0 && (text[len - 1] == '\n' || text[len - 1] == '\r'))
text[--len] = '\0';
const char* p = text;
while (*p == ' ' || *p == '\t')
p++;
if (*p == '\0') {
snprintf(err, err_size, "empty filter rule");
free(text);
return NULL;
}
FilterAction action = FILTER_ACTION_EXCLUDE;
if (*p == '+' || *p == '-') {
action = *p == '+' ? FILTER_ACTION_INCLUDE : FILTER_ACTION_EXCLUDE;
p++;
/* rsync attaches rule modifiers directly to the +/- (e.g. "-s foo"). Only
* the '/' anchor modifier is supported; anything else is a clear error
* rather than a silently-ignored literal. */
if (*p != ' ' && *p != '\t' && *p != '\0' && is_unsupported_rule_modifier(*p)) {
snprintf(err, err_size,
"filter rule modifier '%c' is not supported (only the '/' anchor after +/- "
"is implemented; put a space between +/- and the pattern)",
*p);
free(text);
return NULL;
}
while (*p == ' ' || *p == '\t')
p++;
} else {
/* ':' (dir-merge) and '.' (merge) are rsync filter-rule shorthands. At the
* start of a rule they mean "merge this file", so reject them instead of
* silently turning them into inert exclude patterns. */
if (*p == ':' || *p == '.' || *p == '!') {
snprintf(err, err_size,
"filter rule starting with '%c' is not supported (merge/dir-merge/list-clear "
"shorthands are not implemented; use +/- include/exclude rules)",
*p);
free(text);
return NULL;
}
const char* sp = p;
while (*sp != '\0' && *sp != ' ' && *sp != '\t')
sp++;
size_t word_len = (size_t)(sp - p);
if (rule_text_is_unsupported_word(p, word_len)) {
snprintf(err, err_size,
"'%.*s' filter directives are not supported (only +/- include/exclude rules "
"with an optional '/' anchor and trailing '/' dir marker)",
(int)word_len, p);
free(text);
return NULL;
}
if (word_len == strlen("include") && strncmp(p, "include", word_len) == 0) {
action = FILTER_ACTION_INCLUDE;
p = sp;
} else if (word_len == strlen("exclude") && strncmp(p, "exclude", word_len) == 0) {
action = FILTER_ACTION_EXCLUDE;
p = sp;
}
while (*p == ' ' || *p == '\t')
p++;
}
if (*p == '\0') {
snprintf(err, err_size, "filter rule has no pattern");
free(text);
return NULL;
}
/* A pattern beginning with '/' is anchored (either as "-/foo" or "- /foo"). */
bool anchored = false;
if (*p == '/') {
anchored = true;
p++;
while (*p == ' ' || *p == '\t')
p++;
}
if (*p == '\0') {
snprintf(err, err_size, "filter rule has no pattern after '/' anchor");
free(text);
return NULL;
}
/* Pattern runs to the end of the rule; a single trailing '/' marks dir-only. */
size_t pat_len = strlen(p);
bool dir_only = false;
if (pat_len > 1 && p[pat_len - 1] == '/') {
dir_only = true;
pat_len--;
} else if (pat_len == 1 && p[0] == '/') {
/* "//" anchored with nothing after: meaningless. */
snprintf(err, err_size, "filter rule has no pattern");
free(text);
return NULL;
}
FilterRule* rule = calloc(1, sizeof(FilterRule));
if (!rule) {
snprintf(err, err_size, "memory allocation failed");
free(text);
return NULL;
}
rule->pattern = malloc(pat_len + 1);
if (!rule->pattern) {
free(rule);
snprintf(err, err_size, "memory allocation failed");
free(text);
return NULL;
}
memcpy(rule->pattern, p, pat_len);
rule->pattern[pat_len] = '\0';
rule->action = action;
rule->anchored = anchored;
rule->dir_only = dir_only;
rule->owner = NULL;
free(text);
return rule;
}
void filter_rule_free(FilterRule* rule) {
if (!rule)
return;
free(rule->pattern);
free(rule->owner);
free(rule);
}
/* ---- Ordered rule lists ---- */
FilterRuleList* filter_rule_list_create(void) {
return calloc(1, sizeof(FilterRuleList));
}
bool filter_rule_list_add(FilterRuleList* list, FilterRule* rule) {
if (!list || !rule)
return false;
if (list->count == list->capacity) {
int new_cap = list->capacity > 0 ? list->capacity * 2 : 8;
FilterRule** grown = realloc(list->items, (size_t)new_cap * sizeof(FilterRule*));
if (!grown)
return false;
list->items = grown;
list->capacity = new_cap;
}
list->items[list->count++] = rule;
return true;
}
bool filter_rule_list_parse_append(FilterRuleList* list, const char* line, char* err,
size_t err_size) {
FilterRule* rule = filter_rule_parse(line, err, err_size);
if (!rule)
return false;
if (!filter_rule_list_add(list, rule)) {
filter_rule_free(rule);
snprintf(err, err_size, "memory allocation failed");
return false;
}
return true;
}
void filter_rule_list_free(FilterRuleList* list) {
if (!list)
return;
for (int i = 0; i < list->count; i++)
filter_rule_free(list->items[i]);
free(list->items);
free(list);
}
static bool set_rule_owner(FilterRule* rule, const char* owner) {
char* dup = str_dup(owner ? owner : "");
if (!dup)
return false;
free(rule->owner);
rule->owner = dup;
return true;
}
/* ---- CVS default excludes (-C) ---- */
typedef struct {
const char* pattern;
bool dir_only;
} CvsDefaultRule;
static const CvsDefaultRule CVS_DEFAULTS[] = {
{"RCS", false}, {"SCCS", false}, {"CVS", false}, {"CVS.adm", false},
{"RCSLOG", false}, {"cvslog.*", false}, {"tags", false}, {"TAGS", false},
{".make.state", false}, {".nse_depinfo", false}, {"*~", false}, {"#*", false},
{".#*", false}, {",*", false}, {"_$*", false}, {"*$", false},
{"*.old", false}, {"*.bak", false}, {"*.BAK", false}, {"*.orig", false},
{"*.rej", false}, {".del-*", false}, {"*.a", false}, {"*.olb", false},
{"*.o", false}, {"*.obj", false}, {"*.so", false}, {"*.exe", false},
{"*.Z", false}, {"*.elc", false}, {"*.ln", false}, {"core", false},
{".svn/", true}, {".git/", true}, {".hg/", true}, {".bzr/", true},
};
static bool cvs_rule_list_append(FilterRuleList* list) {
for (size_t i = 0; i < sizeof(CVS_DEFAULTS) / sizeof(CVS_DEFAULTS[0]); i++) {
FilterRule* rule = calloc(1, sizeof(FilterRule));
if (!rule)
return false;
rule->action = FILTER_ACTION_EXCLUDE;
rule->dir_only = CVS_DEFAULTS[i].dir_only;
size_t plen = strlen(CVS_DEFAULTS[i].pattern);
if (rule->dir_only && plen > 0 && CVS_DEFAULTS[i].pattern[plen - 1] == '/')
plen--; /* keep the cleaned pattern, matching filter_rule_parse */
rule->pattern = malloc(plen + 1);
if (!rule->pattern) {
free(rule);
return false;
}
memcpy(rule->pattern, CVS_DEFAULTS[i].pattern, plen);
rule->pattern[plen] = '\0';
if (!set_rule_owner(rule, "")) {
filter_rule_free(rule);
return false;
}
if (!filter_rule_list_add(list, rule)) {
filter_rule_free(rule);
return false;
}
}
return true;
}
FilterRuleList* filter_base_build(const char* const* rule_texts, int rule_count, bool cvs_exclude,
char* err, size_t err_size) {
if (err && err_size > 0)
err[0] = '\0';
FilterRuleList* list = filter_rule_list_create();
if (!list) {
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
for (int i = 0; i < rule_count; i++) {
if (!rule_texts || !rule_texts[i])
continue;
FilterRule* rule = filter_rule_parse(rule_texts[i], err, err_size);
if (!rule) {
filter_rule_list_free(list);
return NULL;
}
if (!set_rule_owner(rule, "")) {
filter_rule_free(rule);
filter_rule_list_free(list);
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
if (!filter_rule_list_add(list, rule)) {
filter_rule_free(rule);
filter_rule_list_free(list);
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
}
if (cvs_exclude && !cvs_rule_list_append(list)) {
filter_rule_list_free(list);
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
return list;
}
/* ---- Per-directory .rsync-filter files ---- */
FilterRuleList* filter_file_read(const char* dir_path, const char* owner_rel, bool* exists,
char* err, size_t err_size) {
if (err && err_size > 0)
err[0] = '\0';
if (exists)
*exists = false;
char* filter_path = path_cat(dir_path, ".rsync-filter");
if (!filter_path) {
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
FILE* fp = fopen(filter_path, "r");
free(filter_path);
if (!fp) {
if (errno == ENOENT || errno == ENOTDIR)
return filter_rule_list_create();
log_message(LOG_LEVEL_WARNING, "Could not read .rsync-filter in %s: %s", dir_path,
strerror(errno));
return filter_rule_list_create();
}
if (exists)
*exists = true;
FilterRuleList* list = filter_rule_list_create();
if (!list) {
fclose(fp);
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
char* line = NULL;
size_t line_cap = 0;
ssize_t n;
bool ok = true;
while ((n = getline(&line, &line_cap, fp)) != -1) {
const char* p = line;
while (*p == ' ' || *p == '\t')
p++;
if (*p == '\0' || *p == '\n' || *p == '\r' || *p == '#')
continue;
FilterRule* rule = filter_rule_parse(p, err, err_size);
if (!rule) {
ok = false;
break;
}
if (!set_rule_owner(rule, owner_rel)) {
filter_rule_free(rule);
snprintf(err, err_size, "memory allocation failed");
ok = false;
break;
}
if (!filter_rule_list_add(list, rule)) {
filter_rule_free(rule);
snprintf(err, err_size, "memory allocation failed");
ok = false;
break;
}
}
free(line);
fclose(fp);
if (!ok) {
filter_rule_list_free(list);
return NULL;
}
return list;
}
/* ---- Rule matching ---- */
/* Match a pattern that contains '/' (non-anchored) against the end of the
* relative path, starting at any path-component boundary. */
static bool glob_suffix_match(const char* pattern, const char* str) {
if (glob_match(pattern, str))
return true;
for (const char* slash = strchr(str, '/'); slash; slash = strchr(slash + 1, '/')) {
if (glob_match(pattern, slash + 1))
return true;
}
return false;
}
static FilterAction rule_matches(const FilterRule* rule, const char* rel_path, const char* leaf,
bool is_dir) {
if (!rule || !rule->pattern)
return FILTER_ACTION_NONE;
if (rule->dir_only && !is_dir)
return FILTER_ACTION_NONE;
/* A rule applies only to entries below its owner directory. */
const char* rel2 = rel_path;
if (rule->owner && rule->owner[0] != '\0') {
size_t owner_len = strlen(rule->owner);
if (strncmp(rule->owner, rel_path, owner_len) != 0)
return FILTER_ACTION_NONE;
if (rel_path[owner_len] != '/')
return FILTER_ACTION_NONE;
rel2 = rel_path + owner_len + 1;
}
if (rel2[0] == '\0')
return FILTER_ACTION_NONE;
bool matched;
if (rule->anchored) {
matched = glob_match(rule->pattern, rel2);
} else if (strchr(rule->pattern, '/') != NULL) {
matched = glob_suffix_match(rule->pattern, rel2);
} else {
matched = glob_match(rule->pattern, leaf);
}
return matched ? rule->action : FILTER_ACTION_NONE;
}
FilterAction filter_rules_apply(const FilterRuleList* list, const char* rel_path, const char* leaf,
bool is_dir) {
if (!list)
return FILTER_ACTION_NONE;
for (int i = 0; i < list->count; i++) {
FilterAction action = rule_matches(list->items[i], rel_path, leaf, is_dir);
if (action != FILTER_ACTION_NONE)
return action;
}
return FILTER_ACTION_NONE;
}
+83
View File
@@ -0,0 +1,83 @@
#ifndef FILTER_H
#define FILTER_H
#include <stdbool.h>
#include <stddef.h>
/* rsync-style filter rule engine (client-side file selection).
*
* Supported rule syntax (documented subset):
* [+|-] [anchored '/' prefix] pattern [trailing '/' for dir-only]
*
* "+ PATTERN" include rule (first match wins)
* "- PATTERN" exclude rule
* "PATTERN" implicit exclude rule (rsync default)
* "include PATTERN" / "exclude PATTERN" word forms
* leading '/' after the +/- anchors the pattern to its owner directory
* (the transfer root for command-line/-C rules, the directory that
* contains a .rsync-filter file for per-directory rules)
* a trailing '/' makes the rule match directories only
*
* Rejected explicitly (no silent no-ops): the rsync merge/dir-merge/list-clear
* shorthands written as a rule that starts with ':' or '.' or '!', the
* merge/dir-merge/hide/show/protect/risk/clear words, and every include/exclude
* rule modifier other than '/' (! C s r p x). The pattern must be separated
* from +/- by a space (or a single '/' anchor), exactly like rsync's
* "-s foo"/"-p ..." modifier syntax is refused.
*/
typedef enum {
FILTER_ACTION_NONE = 0, /* no rule matched */
FILTER_ACTION_EXCLUDE = -1,
FILTER_ACTION_INCLUDE = 1
} FilterAction;
typedef struct {
FilterAction action;
bool anchored; /* pattern anchored to the rule's owner directory */
bool dir_only; /* pattern had a trailing '/': matches directories only */
char* owner; /* owning directory rel path ("" == transfer root) */
char* pattern; /* cleaned glob pattern (no leading '/', no trailing '/') */
} FilterRule;
typedef struct {
FilterRule** items; /* owned array of rule pointers */
int count;
int capacity;
} FilterRuleList;
/* Parse a single filter-rule line (no trailing newline required). Returns an
* owned rule, or NULL on unsupported/invalid syntax with a message in `err`. */
FilterRule* filter_rule_parse(const char* line, char* err, size_t err_size);
void filter_rule_free(FilterRule* rule);
FilterRuleList* filter_rule_list_create(void);
/* Append a fully-parsed rule (takes ownership). Returns false on OOM. */
bool filter_rule_list_add(FilterRuleList* list, FilterRule* rule);
/* Parse `line` and append it. Returns false and fills `err` on bad syntax. */
bool filter_rule_list_parse_append(FilterRuleList* list, const char* line, char* err,
size_t err_size);
void filter_rule_list_free(FilterRuleList* list);
/* Build the command-line filter set: `rule_texts` (--filter=RULE in the order
* given, 0..rule_count) followed by the -C CVS default excludes when
* cvs_exclude is true. All rules are owned by "" (the transfer root).
* Returns NULL on unsupported rule text (message in `err`). */
FilterRuleList* filter_base_build(const char* const* rule_texts, int rule_count, bool cvs_exclude,
char* err, size_t err_size);
/* Read "<dir_path>/.rsync-filter" and return its rules, each owned by
* `owner_rel`. A missing file yields an empty list with *exists=false; an
* unreadable file is treated as missing. Returns NULL only on parse or
* allocation failure (message in `err`). */
FilterRuleList* filter_file_read(const char* dir_path, const char* owner_rel, bool* exists,
char* err, size_t err_size);
/* Evaluate an entry against one ordered rule list. Returns FILTER_ACTION_NONE
* when no rule matched, otherwise the first matching rule's action.
* `rel_path` is the entry's path relative to the transfer root ("" == root),
* `leaf` its final name, `is_dir` whether it is a directory. */
FilterAction filter_rules_apply(const FilterRuleList* list, const char* rel_path, const char* leaf,
bool is_dir);
#endif
+15 -5
View File
@@ -205,7 +205,8 @@ static bool is_dir_in_manifest(const char* rel_path, ArrayList* manifest) {
}
static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifest,
size_t max_delete, size_t* deleted_count) {
size_t max_delete, size_t* deleted_count,
const char* skip_root_child) {
int scanfd = dup(dirfd);
if (scanfd < 0)
return false;
@@ -219,6 +220,13 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifes
while ((entry = readdir(dir)) != NULL) {
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
continue;
/* A --delay-updates run keeps its staging directory as a direct child of
the receive root. Its contents are not manifest entries yet (they are
published after deletion), so descending into it would delete every
staged file as an "extra". Skip only the top-level staging name; nested
directories with the same name are ordinary destination content. */
if (rel_path[0] == '\0' && skip_root_child && strcmp(entry->d_name, skip_root_child) == 0)
continue;
char* child_rel = path_cat((char*)rel_path, entry->d_name);
if (!child_rel) {
operation_ok = false;
@@ -240,7 +248,8 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifes
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_removed = false;
if (childfd >= 0) {
child_removed = delete_extras_fd(childfd, child_rel, manifest, max_delete, deleted_count);
child_removed = delete_extras_fd(childfd, child_rel, manifest, max_delete, deleted_count,
skip_root_child);
if (!child_removed)
operation_ok = false;
close(childfd);
@@ -291,7 +300,8 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifes
return operation_ok;
}
bool delete_extras_limited(const char* dest_root, ArrayList* manifest, size_t max_delete) {
bool delete_extras_limited(const char* dest_root, ArrayList* manifest, size_t max_delete,
const char* skip_root_child) {
if (!manifest)
return false;
int rootfd;
@@ -308,14 +318,14 @@ bool delete_extras_limited(const char* dest_root, ArrayList* manifest, size_t ma
if (rootfd < 0)
return false;
size_t deleted_count = 0;
bool ok = delete_extras_fd(rootfd, "", manifest, max_delete, &deleted_count);
bool ok = delete_extras_fd(rootfd, "", manifest, max_delete, &deleted_count, skip_root_child);
if (close(rootfd) != 0)
ok = false;
return ok;
}
bool delete_extras(const char* dest_root, ArrayList* manifest) {
return delete_extras_limited(dest_root, manifest, SIZE_MAX);
return delete_extras_limited(dest_root, manifest, SIZE_MAX, NULL);
}
bool has_path_traversal(const char* path) {
+6 -1
View File
@@ -10,7 +10,12 @@ char* output_escape(const char* string, bool eight_bit_output);
char* path_cat(const char* path1, const char* path2);
bool glob_match(const char* pattern, const char* str);
bool delete_extras(const char* dest_root, ArrayList* manifest);
bool delete_extras_limited(const char* dest_root, ArrayList* manifest, size_t max_delete);
/* Remove files/dirs under dest_root that are not listed in manifest. When
skip_root_child is non-NULL, a direct child of dest_root with that exact
name is left untouched (used to protect the --delay-updates staging
directory, which holds files that are still to be published). */
bool delete_extras_limited(const char* dest_root, ArrayList* manifest, size_t max_delete,
const char* skip_root_child);
bool utils_set_authorized_root(int fd, const char* canonical_path);
/* The fd-only compatibility form is fail-closed for path-based operations;
* callers should use utils_set_authorized_root with the canonical identity. */
+235 -1
View File
@@ -12,7 +12,7 @@ from common import (
PROJECT_ROOT, BUILD_DIR, TEST_DATA_DIR,
run_client,
generate_test_files, verify_transfer, clean_dir, make_result,
get_dest_received_dir, CLIENT_CMD,
get_dest_received_dir, CLIENT_CMD, ServerManager,
)
SOURCE_DIR = os.path.join(TEST_DATA_DIR, "feature_source")
@@ -1405,3 +1405,237 @@ class TestLogFileFormat:
expected = {f"{os.path.join(source, rel)} {len(data)}" for rel, data in files.items()}
for line in expected:
assert line in content, f"log file (-m) missing {line!r}"
class TestDelayUpdates:
"""--delay-updates stages every updated file under a private 0700 staging
directory inside the receive root and atomically publishes all of them only
after the whole transfer succeeds."""
STAGING = ".fastsync-stage"
def _make_source(self, name):
source = os.path.join(TEST_DATA_DIR, name)
clean_dir(source)
entries = {
"top.txt": b"top level\n",
"sub/deep.txt": b"deeply nested file\n",
"sub/another.txt": b"another nested file\n" * 20,
"binary.bin": bytes(range(256)) * 4,
}
for rel, content in entries.items():
full = os.path.join(source, rel)
os.makedirs(os.path.dirname(full), exist_ok=True)
with open(full, "wb") as fh:
fh.write(content)
return source
@pytest.mark.parametrize("mt", [False, True])
def test_delay_updates_matches_plain_transfer(self, shared_server, mt):
source = self._make_source("delay_match_src")
plain_dest = os.path.join(TEST_DATA_DIR, "delay_match_plain_dst")
delay_dest = os.path.join(TEST_DATA_DIR, "delay_match_delay_dst")
clean_dir(plain_dest)
clean_dir(delay_dest)
result, _ = run_client(source, plain_dest, port=shared_server.port)
assert result.returncode == 0, f"plain sync failed: {result.stderr[:200]}"
flags = ["--delay-updates"] + (["-m"] if mt else [])
result, _ = run_client(source, delay_dest, flags=flags, port=shared_server.port)
assert result.returncode == 0, f"delay-updates sync failed: {result.stderr[:200]}"
plain_received = get_dest_received_dir(plain_dest, source)
delay_received = get_dest_received_dir(delay_dest, source)
mismatches, missing = verify_transfer(source, delay_received)
assert not missing, f"Missing: {missing}"
assert not mismatches, f"Mismatch: {mismatches}"
for root, _dirs, files in os.walk(delay_received):
for name in files:
rel = os.path.relpath(os.path.join(root, name), delay_received)
assert filecmp.cmp(os.path.join(plain_received, rel),
os.path.join(delay_received, rel), shallow=False), rel
assert not os.path.isdir(os.path.join(delay_dest, self.STAGING)), \
"staging directory left behind after a successful delayed transfer"
@pytest.mark.parametrize("mt", [False, True])
def test_delay_updates_incremental_rerun_no_leftovers(self, shared_server, mt):
source = self._make_source("delay_rerun_src")
dest = os.path.join(TEST_DATA_DIR, "delay_rerun_dst")
clean_dir(dest)
flags = ["--delay-updates", "-M", "--incremental"] + (["-m"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode == 0, f"first delayed sync failed: {result.stderr[:200]}"
received = get_dest_received_dir(dest, source)
mismatches, missing = verify_transfer(source, received)
assert not missing and not mismatches
assert not os.path.isdir(os.path.join(dest, self.STAGING))
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode == 0, f"second delayed sync failed: {result.stderr[:200]}"
assert not os.path.isdir(os.path.join(dest, self.STAGING)), \
"fully-skipped delayed run left a staging directory"
@pytest.mark.parametrize("mt", [False, True])
def test_remove_source_files_with_delay_updates(self, shared_server, mt):
source = self._make_source("delay_rsf_src")
dest = os.path.join(TEST_DATA_DIR, "delay_rsf_dst")
clean_dir(dest)
flags = ["--remove-source-files", "--delay-updates"] + (["-m"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode == 0, f"delayed remove-source sync failed: {result.stderr[:200]}"
# Sources are removed only after the receiver published every file.
for root, _dirs, files in os.walk(source):
assert files == [], f"source files survived delayed remove-source-files: {files}"
received = get_dest_received_dir(dest, source)
assert os.path.isfile(os.path.join(received, "top.txt"))
assert os.path.isfile(os.path.join(received, "sub", "deep.txt"))
assert not os.path.isdir(os.path.join(dest, self.STAGING))
@pytest.mark.parametrize("mt", [False, True])
def test_delete_with_delay_updates(self, mt):
"""--delete runs before publication, so the delete walker must not treat
the staging directory as a set of extras: a changed file must still be
published after genuine extras are removed. Uses its own server started
with --allow-delete (the shared session server refuses deletion)."""
source = os.path.join(TEST_DATA_DIR, "delay_delete_src")
dest = os.path.join(TEST_DATA_DIR, "delay_delete_dst")
clean_dir(source)
clean_dir(dest)
with open(os.path.join(source, "f.txt"), "wb") as fh:
fh.write(b"AAAA")
with open(os.path.join(source, "extra.txt"), "wb") as fh:
fh.write(b"seed extra")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, port=server.port)
assert result.returncode == 0, f"seed sync failed: {result.stderr[:200]}"
received = get_dest_received_dir(dest, source)
assert _read_file(os.path.join(received, "extra.txt")) == b"seed extra"
# Second source state: f.txt changed, extra.txt removed from source.
with open(os.path.join(source, "f.txt"), "wb") as fh:
fh.write(b"BBBB")
os.remove(os.path.join(source, "extra.txt"))
flags = ["--delete", "--delay-updates"] + (["-m"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=server.port)
assert result.returncode == 0, \
f"delete+delay-updates sync failed: {result.stderr[:200]}"
assert _read_file(os.path.join(received, "f.txt")) == b"BBBB", \
"changed file was not published after deletion"
assert not os.path.exists(os.path.join(received, "extra.txt")), \
"genuine extra file was not deleted"
assert not os.path.isdir(os.path.join(dest, self.STAGING))
def test_delay_updates_rejects_reserved_backup_dir(self):
"""--backup-dir equal to the internal staging name must be rejected so
an old backup can never be silently installed as the "new" file."""
source = self._make_source("delay_reserved_bak_src")
for variant, suffix in (("bare", ""), ("slash", "/")):
dest = os.path.join(TEST_DATA_DIR, f"delay_reserved_bak_{variant}_dst")
clean_dir(dest)
flags = ["--delay-updates", "--backup", "--backup-dir",
".fastsync-stage" + suffix]
result, _ = run_client(source, dest, flags=flags, port=None)
assert result.returncode != 0, \
f"reserved --backup-dir '{suffix}' was accepted"
assert not os.path.isdir(os.path.join(dest, self.STAGING)), \
"staging directory created by a rejected run"
@pytest.mark.parametrize("remove_source_files", [False, True])
@pytest.mark.parametrize("mt", [False, True])
def test_mid_publish_failure_keeps_published_no_rollback(self, shared_server, mt,
remove_source_files):
"""A stage->publish rename failing part way through publication must
fail the whole transfer, keep the already-published top-level file (no
rollback), leave the not-yet-published nested file absent, and clean up
the staging area. A regular file is planted where the final "sub"
directory must be created, so the nested rename fails (mkdir over a
file is impossible even for root) while the top-level file, which is
always staged first, publishes. With --remove-source-files the sender
must keep every source because no success/outcome frame is ever sent."""
source = os.path.join(TEST_DATA_DIR, "delay_mid_src")
dest = os.path.join(TEST_DATA_DIR, "delay_mid_dst")
clean_dir(source)
clean_dir(dest)
top_path = os.path.join(source, "top.txt")
deep_path = os.path.join(source, "sub", "deep.txt")
with open(top_path, "wb") as fh:
fh.write(b"top payload\n")
os.makedirs(os.path.dirname(deep_path))
with open(deep_path, "wb") as fh:
fh.write(b"deep payload\n")
received = get_dest_received_dir(dest, source)
os.makedirs(received)
with open(os.path.join(received, "sub"), "wb") as fh:
fh.write(b"blocks the nested destination directory")
flags = ["--delay-updates"] + (["-m"] if mt else [])
if remove_source_files:
flags += ["--remove-source-files"]
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode != 0, "blocked nested publish did not fail"
# The top-level file was published before the nested rename failed and
# is intentionally NOT rolled back.
assert _read_file(os.path.join(received, "top.txt")) == b"top payload\n"
# The nested file was never published.
assert not os.path.lexists(os.path.join(received, "sub", "deep.txt")), \
"nested file appeared despite a failed publish"
assert not os.path.isdir(os.path.join(dest, self.STAGING)), \
"staging leftovers after a failed mid-publish"
# Sources survive: no success frame was sent, so a remove-source-files
# sender must not delete anything.
assert os.path.isfile(top_path)
assert os.path.isfile(deep_path)
@pytest.mark.parametrize("mt", [False, True])
def test_remove_source_files_keeps_receiver_skipped_source(self, shared_server, mt):
"""With --delay-updates + --ignore-existing a receiver-skipped source
must survive (its outcome is sent only after publication) while a
freshly delivered file is published and its source removed."""
source = os.path.join(TEST_DATA_DIR, "delay_rsf_skip_src")
dest = os.path.join(TEST_DATA_DIR, "delay_rsf_skip_dst")
clean_dir(source)
clean_dir(dest)
with open(os.path.join(source, "keep.txt"), "wb") as fh:
fh.write(b"existing on dest")
result, _ = run_client(source, dest, port=shared_server.port)
assert result.returncode == 0, f"seed sync failed: {result.stderr[:200]}"
with open(os.path.join(source, "keep.txt"), "wb") as fh:
fh.write(b"changed on source")
with open(os.path.join(source, "deliver.txt"), "wb") as fh:
fh.write(b"new file")
flags = ["--remove-source-files", "--ignore-existing", "--delay-updates"] + (["-m"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode == 0, f"delayed skip sync failed: {result.stderr[:200]}"
# keep.txt already existed at the destination: receiver skip -> source stays.
assert os.path.isfile(os.path.join(source, "keep.txt")), \
"receiver-skipped source was removed despite --ignore-existing"
# deliver.txt was new: staged, published, and its source removed.
assert not os.path.isfile(os.path.join(source, "deliver.txt")), \
"published source was not removed"
received = get_dest_received_dir(dest, source)
assert not os.path.isdir(os.path.join(dest, self.STAGING))
def test_delay_updates_rejects_inplace(self):
source = self._make_source("delay_inplace_src")
dest = os.path.join(TEST_DATA_DIR, "delay_inplace_dst")
clean_dir(dest)
result, _ = run_client(source, dest, flags=["--delay-updates", "--inplace"])
assert result.returncode != 0, "--inplace with --delay-updates was accepted"
assert not os.path.isdir(os.path.join(dest, self.STAGING))
class TestFilesFrom:
"""--files-from transfers exactly the listed files; a listed directory
transfers its whole subtree. The manifest (and thus --delete) derives from
what was actually sent."""
class TestFilters:
"""--filter/-C/-F rule layer: excludes prune, ordering is first-match-wins,
the default with no matching rule is include, and legacy --exclude remains
an independent layer."""
+2
View File
@@ -5,6 +5,7 @@
#include "test_compression.h"
#include "test_config.h"
#include "test_data.h"
#include "test_delay_updates.h"
#include "test_delta.h"
#include "test_file.h"
#include "test_file_sendfile.h"
@@ -51,6 +52,7 @@ int main() {
RUN_TEST(test_metadata);
RUN_TEST(test_glob);
RUN_TEST(test_file);
RUN_TEST(test_delay_updates);
RUN_TEST(test_file_sendfile);
RUN_TEST(test_multiprocessing);
RUN_TEST(test_log);
+229 -3
View File
@@ -2,6 +2,7 @@
#include "client_validation.h"
#include "chmod.h"
#include "config.h"
#include "file_list.h"
#include "log.h"
#include "test_utils.h"
#include "utils.h"
@@ -607,9 +608,6 @@ static void test_parse_args_rejects_unimplemented_options() {
"--delete-excluded",
"--delete-after",
"--max-delete",
"--filter",
"--files-from",
"--cvs-exclude",
"--prune-empty-dirs",
"-R",
"--relative",
@@ -1265,6 +1263,228 @@ static void test_parse_args_log_file_format() {
config_delete(cfg);
}
/* --delay-updates is a plain boolean receiver option. */
static void test_parse_args_delay_updates() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--delay-updates", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->delay_updates);
EXPECT_EQ_INT(positional_count, 2);
config_delete(cfg);
}
/* rsync rejects --delay-updates with --inplace; FastSync must too. */
static void test_validate_config_delay_updates_rejects_inplace() {
Config* cfg = valid_client_config();
cfg->delay_updates = true;
cfg->inplace = true;
EXPECT_FALSE(validate_config(cfg));
config_delete(cfg);
}
/* --backup-dir may not collide with the internal --delay-updates staging
directory (with or without a trailing slash), or old backups would silently
be installed as the "new" file. */
static void test_validate_config_delay_updates_rejects_reserved_backup_dir() {
static const char* const reserved[] = {".fastsync-stage", ".fastsync-stage/"};
for (size_t i = 0; i < sizeof(reserved) / sizeof(reserved[0]); i++) {
Config* cfg = valid_client_config();
cfg->delay_updates = true;
cfg->backup_dir = str_dup(reserved[i]);
EXPECT_FALSE(validate_config(cfg));
config_delete(cfg);
}
/* A non-colliding backup dir is fine alongside --delay-updates. */
Config* ok = valid_client_config();
ok->delay_updates = true;
ok->backup_dir = str_dup("backups");
EXPECT_TRUE(validate_config(ok));
config_delete(ok);
}
static void test_parse_args_filter_rules() {
Config* cfg = config_create();
int positional_args[2];
int positional_count = 0;
char* argv[] = {"fastsync", "--filter", "- *.tmp", "--filter=+ /keep.txt", "/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 6, argv, positional_args, &positional_count), 0);
EXPECT_NOT_NULL(cfg->filters);
EXPECT_EQ_INT(cfg->filters->size, 2);
EXPECT_EQ_STR((char*)cfg->filters->items[0], "- *.tmp");
EXPECT_EQ_STR((char*)cfg->filters->items[1], "+ /keep.txt");
config_delete(cfg);
/* An unsupported rsync rule type is rejected with a clear error. */
cfg = config_create();
positional_count = 0;
char* bad_argv[] = {"fastsync", "--filter=merge /tmp/excludes", "/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 4, bad_argv, positional_args, &positional_count), -1);
config_delete(cfg);
/* A trailing --filter with no rule is a missing-argument error. */
cfg = config_create();
positional_count = 0;
char* missing_argv[] = {"fastsync", "/src", "/dst", "--filter"};
EXPECT_EQ_INT(parse_args(cfg, 4, missing_argv, positional_args, &positional_count), -1);
config_delete(cfg);
/* rsync shorthands/modifiers we do not support are rejected instead of being
* silently parsed as literal patterns. */
static const char* const unsupported[] = {
": .rsync-filter", ". /tmp/rules", "-s foo", "-p bar", "-C", "-! *.o", "!",
};
for (size_t i = 0; i < sizeof(unsupported) / sizeof(unsupported[0]); i++) {
cfg = config_create();
positional_count = 0;
char* rule_argv[] = {"fastsync", "--filter", (char*)unsupported[i], "/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 5, rule_argv, positional_args, &positional_count), -1);
config_delete(cfg);
}
/* Supported spellings still parse: space- or slash-separated, attached
* wildcards, and anchored rules. */
cfg = config_create();
positional_count = 0;
char* ok_argv[] = {"fastsync", "--filter=-*.o", "--filter=- /foo",
"--filter=+ /bar/", "/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 6, ok_argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->filters->size, 3);
config_delete(cfg);
}
static void test_parse_args_from0_cvs_filter_file_flags() {
static const struct {
const char* arg;
bool from0;
bool cvs;
bool per_dir;
} cases[] = {
{"--from0", true, false, false},
{"-0", true, false, false},
{"--cvs-exclude", false, true, false},
{"-C", false, true, false},
{"-F", false, false, true},
};
for (size_t i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", (char*)cases[i].arg, "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->from0, cases[i].from0);
EXPECT_EQ_INT(cfg->cvs_exclude, cases[i].cvs);
EXPECT_EQ_INT(cfg->per_dir_filter, cases[i].per_dir);
config_delete(cfg);
}
/* The plain booleans are negatable (--no-* simply clears the flag). */
static const char* const on[][2] = {{"--from0", "--no-from0"}, {"-C", "--no-cvs-exclude"}};
for (size_t i = 0; i < sizeof(on) / sizeof(on[0]); i++) {
Config* cfg = config_create();
int positional_args[2];
int positional_count = 0;
char* argv[] = {"fastsync", (char*)on[i][0], (char*)on[i][1], "/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
EXPECT_FALSE(cfg->from0);
EXPECT_FALSE(cfg->cvs_exclude);
config_delete(cfg);
}
}
static void write_file_bytes(const char* path, const char* bytes, size_t len) {
FILE* fp = fopen(path, "wb");
EXPECT_NOT_NULL(fp);
EXPECT_EQ_INT((int)fwrite(bytes, 1, len, fp), (int)len);
fclose(fp);
}
static void test_parse_args_files_from() {
const char* list_path = "cli_files_from_list.txt";
write_file_bytes(list_path, "a.txt\nsub/b.bin\n\n./c.txt\n", 25);
Config* cfg = config_create();
int positional_args[2];
int positional_count = 0;
char* argv[] = {"fastsync", "--files-from", (char*)list_path, "/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
EXPECT_EQ_STR(cfg->files_from, list_path);
EXPECT_NOT_NULL(cfg->files_from_set);
FileListSet* set = (FileListSet*)cfg->files_from_set;
EXPECT_TRUE(file_list_affects(set, "a.txt"));
EXPECT_TRUE(file_list_affects(set, "sub/b.bin"));
EXPECT_TRUE(file_list_affects(set, "sub/b.bin/x"));
EXPECT_TRUE(file_list_affects(set, "sub"));
EXPECT_TRUE(file_list_affects(set, "c.txt"));
EXPECT_FALSE(file_list_affects(set, "other.txt"));
config_delete(cfg);
remove(list_path);
/* -0 switches the separator to NUL regardless of argument order, and NUL
* mode preserves entry bytes exactly (a trailing CR/LF is part of the name). */
write_file_bytes(list_path, "x.txt\0y/z.bin\0", 14);
cfg = config_create();
positional_count = 0;
char* nul_argv[] = {"fastsync",
"--files-from="
"cli_files_from_list.txt",
"-0", "/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 5, nul_argv, positional_args, &positional_count), 0);
set = (FileListSet*)cfg->files_from_set;
EXPECT_NOT_NULL(set);
EXPECT_TRUE(file_list_affects(set, "x.txt"));
EXPECT_TRUE(file_list_affects(set, "y/z.bin"));
EXPECT_TRUE(file_list_affects(set, "y"));
EXPECT_FALSE(file_list_affects(set, "z.txt"));
config_delete(cfg);
remove(list_path);
write_file_bytes(list_path, "crlf\n\0tail\0", 11);
cfg = config_create();
positional_count = 0;
char* nul_nl_argv[] = {"fastsync",
"--files-from="
"cli_files_from_list.txt",
"-0", "/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 5, nul_nl_argv, positional_args, &positional_count), 0);
set = (FileListSet*)cfg->files_from_set;
EXPECT_NOT_NULL(set);
EXPECT_TRUE(file_list_affects(set, "crlf\n"));
EXPECT_TRUE(file_list_affects(set, "tail"));
config_delete(cfg);
remove(list_path);
/* A missing list file is a hard parse-time error. */
cfg = config_create();
positional_count = 0;
char* missing_argv[] = {"fastsync", "--files-from", "does_not_exist_ff.txt", "/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 5, missing_argv, positional_args, &positional_count), -1);
config_delete(cfg);
/* Absolute and traversal entries are rejected. */
write_file_bytes(list_path, "/abs/path\n", 10);
cfg = config_create();
positional_count = 0;
char* abs_argv[] = {"fastsync",
"--files-from="
"cli_files_from_list.txt",
"/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 4, abs_argv, positional_args, &positional_count), -1);
config_delete(cfg);
write_file_bytes(list_path, "../escape\n", 10);
cfg = config_create();
positional_count = 0;
char* trav_argv[] = {"fastsync",
"--files-from="
"cli_files_from_list.txt",
"/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 4, trav_argv, positional_args, &positional_count), -1);
config_delete(cfg);
remove(list_path);
}
void test_client_cli() {
test_validate_config_required_paths();
test_validate_config_incompatible_options();
@@ -1344,4 +1564,10 @@ void test_client_cli() {
test_parse_args_checksum_choice_aliases();
test_parse_args_checksum_choice_requires_value();
test_parse_args_temp_dir();
test_parse_args_delay_updates();
test_validate_config_delay_updates_rejects_inplace();
test_validate_config_delay_updates_rejects_reserved_backup_dir();
test_parse_args_files_from();
test_parse_args_filter_rules();
test_parse_args_from0_cvs_filter_file_flags();
}
+28
View File
@@ -136,6 +136,7 @@ static void test_config_send_receive() {
send_cfg->modify_window = 4;
send_cfg->existing = true;
send_cfg->ignore_existing = true;
send_cfg->delay_updates = true;
send_cfg->skip_compress_set = true;
send_cfg->skip_compress_count = 1;
send_cfg->skip_compress_suffixes = calloc(1, sizeof(char*));
@@ -191,6 +192,8 @@ static void test_config_send_receive() {
ok = false;
if (!recv_cfg->ignore_existing)
ok = false;
if (!recv_cfg->delay_updates)
ok = false;
if (!recv_cfg->skip_compress_set || recv_cfg->skip_compress_count != 1 ||
strcmp(recv_cfg->skip_compress_suffixes[0], ".zip") != 0)
ok = false;
@@ -405,6 +408,30 @@ static void test_config_temp_dir_roundtrip() {
config_delete(c);
}
static void test_config_delay_updates_reserved_backup_rejected() {
if (is_running_under_valgrind())
return;
Config* c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->delay_updates = true;
c->backup_dir = str_dup(".fastsync-stage");
/* The receiver-side wire validation must reject a --backup-dir that collides
with the internal delay-updates staging directory. */
EXPECT_FALSE(roundtrip_config_ok(c));
config_delete(c);
c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->delay_updates = true;
c->backup_dir = str_dup("backups");
EXPECT_TRUE(roundtrip_config_ok(c));
config_delete(c);
}
static void test_config_is_remote_dest() {
/* Valid SSH-style destinations */
EXPECT_TRUE(config_is_remote_dest("user@host:/path"));
@@ -440,6 +467,7 @@ void test_config() {
test_config_receive_truncated();
test_config_string_null_vs_empty_roundtrip();
test_config_temp_dir_roundtrip();
test_config_delay_updates_reserved_backup_rejected();
}
test_config_is_remote_dest();
}
+296
View File
@@ -0,0 +1,296 @@
#include "test_delay_updates.h"
#include "config.h"
#include "delay_updates.h"
#include "file.h"
#include "file_receive.h"
#include "test_utils.h"
#include "utils.h"
#include <dirent.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>
/* Recursively remove a test tree (never follows symlinks). */
static void remove_tree(const char* path) {
struct stat st;
if (lstat(path, &st) != 0)
return;
if (S_ISDIR(st.st_mode)) {
DIR* dir = opendir(path);
if (!dir)
return;
const struct dirent* entry;
while ((entry = readdir(dir)) != NULL) {
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
continue;
char* child = path_cat(path, entry->d_name);
if (child) {
remove_tree(child);
free(child);
}
}
closedir(dir);
rmdir(path);
} else {
unlink(path);
}
}
/* Build a File that carries `content`. */
static File* make_file(const char* path, const char* content) {
File* f = file_create(path);
if (!f)
return NULL;
f->data->data = malloc(strlen(content));
if (!f->data->data) {
file_destroy(f);
return NULL;
}
memcpy(f->data->data, content, strlen(content));
f->data->size = strlen(content);
return f;
}
static char* read_all(const char* path) {
FILE* fp = fopen(path, "rb");
if (!fp)
return NULL;
char buf[256] = {0};
size_t n = fread(buf, 1, sizeof(buf) - 1, fp);
fclose(fp);
char* out = malloc(n + 1);
if (!out)
return NULL;
memcpy(out, buf, n);
out[n] = '\0';
return out;
}
static void test_delay_updates_no_final_before_publish() {
const char* root = "test_delay_tmp";
remove_tree(root);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->delay_updates = true;
File* f = make_file("sub/file.txt", "staged payload");
EXPECT_NOT_NULL(f);
// cppcheck-suppress knownConditionTrueFalse
if (!cfg || !f)
goto out;
EXPECT_EQ_INT(file_save_to_disk_full(root, f, cfg), FILE_SAVE_WRITTEN);
EXPECT_NOT_NULL(cfg->delay_context);
const char* final_path = "test_delay_tmp/sub/file.txt";
/* Before publication the final destination must not contain the file. */
EXPECT_FALSE(file_path_exists_secure(final_path));
/* The complete staged copy must live inside the staging tree. */
char* staged = path_cat("test_delay_tmp/.fastsync-stage", "/sub/file.txt");
EXPECT_NOT_NULL(staged);
// cppcheck-suppress knownConditionTrueFalse
if (staged) {
char* content = read_all(staged);
EXPECT_NOT_NULL(content);
// cppcheck-suppress knownConditionTrueFalse
if (content) {
EXPECT_EQ_STR(content, "staged payload");
free(content);
}
free(staged);
}
out:
file_destroy(f);
config_delete(cfg);
remove_tree(root);
}
static void test_delay_updates_publish_installs_files() {
const char* root = "test_delay_pub_tmp";
remove_tree(root);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->delay_updates = true;
File* f = make_file("sub/file.txt", "published payload");
EXPECT_NOT_NULL(f);
// cppcheck-suppress knownConditionTrueFalse
if (!cfg || !f)
goto out;
EXPECT_EQ_INT(file_save_to_disk_full(root, f, cfg), FILE_SAVE_WRITTEN);
const char* final_path = "test_delay_pub_tmp/sub/file.txt";
EXPECT_FALSE(file_path_exists_secure(final_path));
EXPECT_TRUE(delay_updates_publish(cfg->delay_context, cfg));
/* After a successful publish the file is installed and staging is gone. */
char* content = read_all(final_path);
EXPECT_NOT_NULL(content);
// cppcheck-suppress knownConditionTrueFalse
if (content) {
EXPECT_EQ_STR(content, "published payload");
free(content);
}
EXPECT_FALSE(file_path_exists_secure("test_delay_pub_tmp/.fastsync-stage"));
out:
file_destroy(f);
config_delete(cfg);
remove_tree(root);
}
/* The staged tree is cleaned on the error/abort path and final files that were
never published do not appear at the destination. */
static void test_delay_updates_cleanup_removes_staged() {
const char* root = "test_delay_clean_tmp";
remove_tree(root);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->delay_updates = true;
File* f = make_file("sub/file.txt", "never installed");
EXPECT_NOT_NULL(f);
// cppcheck-suppress knownConditionTrueFalse
if (!cfg || !f)
goto out;
EXPECT_EQ_INT(file_save_to_disk_full(root, f, cfg), FILE_SAVE_WRITTEN);
EXPECT_TRUE(file_path_exists_secure("test_delay_clean_tmp/.fastsync-stage/sub/file.txt"));
delay_updates_cleanup(cfg->delay_context);
EXPECT_FALSE(file_path_exists_secure("test_delay_clean_tmp/.fastsync-stage"));
EXPECT_FALSE(file_path_exists_secure("test_delay_clean_tmp/sub/file.txt"));
out:
file_destroy(f);
config_delete(cfg);
remove_tree(root);
}
/* With --backup the previous version is only moved aside at publication. */
static void test_delay_updates_backup_deferred_to_publish() {
const char* root = "test_delay_bak_tmp";
remove_tree(root);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->delay_updates = true;
cfg->backup = true;
EXPECT_TRUE(file_write_to_disk("test_delay_bak_tmp/file.txt", "AAAA", 4, false, false));
File* f = make_file("file.txt", "BBBB");
EXPECT_NOT_NULL(f);
// cppcheck-suppress knownConditionTrueFalse
if (!cfg || !f)
goto out;
EXPECT_EQ_INT(file_save_to_disk_full(root, f, cfg), FILE_SAVE_WRITTEN);
/* Stage time must not touch the final file or create the backup yet. */
char* before = read_all("test_delay_bak_tmp/file.txt");
EXPECT_NOT_NULL(before);
// cppcheck-suppress knownConditionTrueFalse
if (before) {
EXPECT_EQ_STR(before, "AAAA");
free(before);
}
EXPECT_FALSE(file_path_exists_secure("test_delay_bak_tmp/file.txt~"));
EXPECT_TRUE(delay_updates_publish(cfg->delay_context, cfg));
char* after = read_all("test_delay_bak_tmp/file.txt");
char* backup = read_all("test_delay_bak_tmp/file.txt~");
EXPECT_NOT_NULL(after);
EXPECT_NOT_NULL(backup);
// cppcheck-suppress knownConditionTrueFalse
if (after) {
EXPECT_EQ_STR(after, "BBBB");
free(after);
}
// cppcheck-suppress knownConditionTrueFalse
if (backup) {
EXPECT_EQ_STR(backup, "AAAA");
free(backup);
}
out:
file_destroy(f);
config_delete(cfg);
remove_tree(root);
}
/* Skip/update policy checks run against the final path at stage time, matching
what an immediate run would decide. */
static void test_delay_updates_skip_semantics() {
const char* root = "test_delay_skip_tmp";
remove_tree(root);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->delay_updates = true;
/* --existing: final destination missing -> skipped, nothing staged. */
File* missing = make_file("missing.txt", "new");
EXPECT_NOT_NULL(missing);
// cppcheck-suppress knownConditionTrueFalse
if (!cfg || !missing)
goto out;
cfg->existing = true;
EXPECT_EQ_INT(file_save_to_disk_full(root, missing, cfg), FILE_SAVE_SKIPPED);
cfg->existing = false;
/* --ignore-existing: final destination present -> skipped. */
EXPECT_TRUE(file_write_to_disk("test_delay_skip_tmp/existing.txt", "old", 3, false, false));
File* present = make_file("existing.txt", "new");
EXPECT_NOT_NULL(present);
// cppcheck-suppress knownConditionTrueFalse
if (!present)
goto out;
cfg->ignore_existing = true;
EXPECT_EQ_INT(file_save_to_disk_full(root, present, cfg), FILE_SAVE_SKIPPED);
cfg->ignore_existing = false;
/* Without a skip flag the file is staged and later published. */
File* fresh = make_file("fresh.txt", "content");
EXPECT_NOT_NULL(fresh);
// cppcheck-suppress knownConditionTrueFalse
if (!fresh)
goto out;
EXPECT_EQ_INT(file_save_to_disk_full(root, fresh, cfg), FILE_SAVE_WRITTEN);
EXPECT_TRUE(delay_updates_publish(cfg->delay_context, cfg));
char* content = read_all("test_delay_skip_tmp/fresh.txt");
EXPECT_NOT_NULL(content);
// cppcheck-suppress knownConditionTrueFalse
if (content) {
EXPECT_EQ_STR(content, "content");
free(content);
}
out:
file_destroy(missing);
file_destroy(present);
file_destroy(fresh);
config_delete(cfg);
remove_tree(root);
}
/* The reserved staging name must be recognizable for validation, including
with a trailing slash. */
static void test_delay_updates_reserved_name_helper() {
EXPECT_TRUE(delay_updates_staging_name_conflict(".fastsync-stage"));
EXPECT_TRUE(delay_updates_staging_name_conflict(".fastsync-stage/"));
EXPECT_TRUE(delay_updates_staging_name_conflict(".fastsync-stage///"));
EXPECT_FALSE(delay_updates_staging_name_conflict(NULL));
EXPECT_FALSE(delay_updates_staging_name_conflict(""));
EXPECT_FALSE(delay_updates_staging_name_conflict("backups"));
EXPECT_FALSE(delay_updates_staging_name_conflict(".fastsync-stage.bak"));
}
void test_delay_updates() {
test_delay_updates_reserved_name_helper();
test_delay_updates_no_final_before_publish();
test_delay_updates_publish_installs_files();
test_delay_updates_cleanup_removes_staged();
test_delay_updates_backup_deferred_to_publish();
test_delay_updates_skip_semantics();
}
+6
View File
@@ -0,0 +1,6 @@
#ifndef TEST_DELAY_UPDATES_H
#define TEST_DELAY_UPDATES_H
void test_delay_updates(void);
#endif
+440 -2
View File
@@ -1,7 +1,10 @@
#include "test_utils.h"
#include "scanner.h"
#include "file.h"
#include "file_list.h"
#include "filter.h"
#include "utils.h"
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>
@@ -394,8 +397,8 @@ static void test_parallel_scanner_root_chunks_without_workers() {
create_test_file(file1, "a");
create_test_file(file2, "b");
ScannerOptions options = {false, 1, NULL, 0, NULL, 0, 0, 0,
0, 0, false, false, false, false, false, false};
ScannerOptions options = {false, 1, NULL, 0, NULL, 0, 0, 0, 0, 0,
false, false, false, false, false, false, NULL, NULL, false};
ParallelScanner* scanner = parallel_scanner_create_with_options(dir, &options, NULL);
EXPECT_NOT_NULL(scanner);
@@ -653,6 +656,428 @@ static void test_scanner_one_file_system_cross_device() {
EXPECT_EQ_INT(par_on_total, 1);
}
/* Collect emitted file paths (relative to `root`) from a sequential scan.
* Returns 0 on success with *out and *count set (caller frees *out). */
static int collect_files(const char* root, const ScannerOptions* options, char*** out,
int* out_count) {
DirectoryScanner* scanner = directory_scanner_create_with_options(root, options);
if (!scanner)
return -1;
size_t root_len = strlen(root);
while (root_len > 0 && root[root_len - 1] == '/')
root_len--;
int cap = 16;
int count = 0;
char** paths = malloc((size_t)cap * sizeof(char*));
if (!paths) {
directory_scanner_destroy(scanner);
return -1;
}
Chunk* chunk;
while ((chunk = directory_scanner_next(scanner)) != NULL) {
for (int i = 0; i < chunk->element_count; i++) {
const char* rel = chunk->items[i]->path + root_len;
if (*rel == '/')
rel++;
if (count == cap) {
cap *= 2;
char** grown = realloc(paths, (size_t)cap * sizeof(char*));
if (!grown) {
for (int k = 0; k < count; k++)
free(paths[k]);
free(paths);
chunk_destroy(chunk);
directory_scanner_destroy(scanner);
return -1;
}
paths = grown;
}
paths[count++] = str_dup(rel);
}
chunk_destroy(chunk);
}
bool failed = directory_scanner_failed(scanner);
directory_scanner_destroy(scanner);
if (failed) {
for (int k = 0; k < count; k++)
free(paths[k]);
free(paths);
return -1;
}
*out = paths;
*out_count = count;
return 0;
}
static int collect_files_parallel(const char* root, const ScannerOptions* options, char*** out,
int* out_count) {
ParallelScanner* scanner = parallel_scanner_create_with_options(root, options, NULL);
if (!scanner)
return -1;
size_t root_len = strlen(root);
while (root_len > 0 && root[root_len - 1] == '/')
root_len--;
int cap = 16;
int count = 0;
char** paths = malloc((size_t)cap * sizeof(char*));
if (!paths) {
parallel_scanner_destroy(scanner);
return -1;
}
Chunk* chunk;
while ((chunk = parallel_scanner_next(scanner)) != NULL) {
for (int i = 0; i < chunk->element_count; i++) {
const char* rel = chunk->items[i]->path + root_len;
if (*rel == '/')
rel++;
if (count == cap) {
cap *= 2;
char** grown = realloc(paths, (size_t)cap * sizeof(char*));
if (!grown) {
for (int k = 0; k < count; k++)
free(paths[k]);
free(paths);
chunk_destroy(chunk);
parallel_scanner_destroy(scanner);
return -1;
}
paths = grown;
}
paths[count++] = str_dup(rel);
}
chunk_destroy(chunk);
}
bool failed = parallel_scanner_failed(scanner);
parallel_scanner_destroy(scanner);
if (failed) {
for (int k = 0; k < count; k++)
free(paths[k]);
free(paths);
return -1;
}
*out = paths;
*out_count = count;
return 0;
}
static bool has_path(char** paths, int count, const char* rel) {
for (int i = 0; i < count; i++)
if (strcmp(paths[i], rel) == 0)
return true;
return false;
}
static void free_paths(char** paths, int count) {
for (int i = 0; i < count; i++)
free(paths[i]);
free(paths);
}
static const char* FILE_LIST_PATH = "test_scan_files_from.txt";
/* --files-from: only the listed files (and the subtree of a listed directory)
* are emitted; unrelated files and directories are pruned. */
static void test_files_from_subset(bool parallel) {
const char* root = "test_scan_ff";
const char* sub = "test_scan_ff/sub";
const char* other = "test_scan_ff/other";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir(sub, 0755), 0);
EXPECT_EQ_INT(mkdir(other, 0755), 0);
create_test_file("test_scan_ff/root.txt", "root");
create_test_file("test_scan_ff/sub/keep.txt", "keep");
create_test_file("test_scan_ff/sub/skip.bin", "skip");
create_test_file("test_scan_ff/other/unrelated.txt", "unrelated");
/* List a root file and a file under sub: sub is descended but its other file
* is not listed, and the whole `other` directory is pruned. */
create_test_file(FILE_LIST_PATH, "root.txt\nsub/keep.txt\n");
char err[160];
FileListSet* set = file_list_load(FILE_LIST_PATH, false, err, sizeof(err));
EXPECT_NOT_NULL(set);
ScannerOptions options = {0};
options.file_list = set;
if (parallel)
options.num_threads = 2;
char** paths = NULL;
int count = 0;
int rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
: collect_files(root, &options, &paths, &count);
EXPECT_EQ_INT(rc, 0);
EXPECT_EQ_INT(count, 2);
EXPECT_TRUE(has_path(paths, count, "root.txt"));
EXPECT_TRUE(has_path(paths, count, "sub/keep.txt"));
EXPECT_FALSE(has_path(paths, count, "sub/skip.bin"));
EXPECT_FALSE(has_path(paths, count, "other/unrelated.txt"));
free_paths(paths, count);
file_list_destroy(set);
remove(FILE_LIST_PATH);
/* Listing a directory transfers its whole subtree. */
create_test_file(FILE_LIST_PATH, "sub\n");
set = file_list_load(FILE_LIST_PATH, false, err, sizeof(err));
EXPECT_NOT_NULL(set);
options.file_list = set;
rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
: collect_files(root, &options, &paths, &count);
EXPECT_EQ_INT(rc, 0);
EXPECT_EQ_INT(count, 2);
EXPECT_TRUE(has_path(paths, count, "sub/keep.txt"));
EXPECT_TRUE(has_path(paths, count, "sub/skip.bin"));
EXPECT_FALSE(has_path(paths, count, "root.txt"));
EXPECT_FALSE(has_path(paths, count, "other/unrelated.txt"));
free_paths(paths, count);
file_list_destroy(set);
remove(FILE_LIST_PATH);
unlink("test_scan_ff/root.txt");
unlink("test_scan_ff/sub/keep.txt");
unlink("test_scan_ff/sub/skip.bin");
unlink("test_scan_ff/other/unrelated.txt");
rmdir(other);
rmdir(sub);
rmdir(root);
}
/* Filter layer: '-' excludes, first-match-wins ordering with '+', anchored
* rules, and dir-only rules all prune during the scan. */
static void test_filter_rules(bool parallel) {
const char* root = "test_scan_filter";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
create_test_file("test_scan_filter/a.txt", "a");
create_test_file("test_scan_filter/b.tmp", "b");
create_test_file("test_scan_filter/c.txt", "c");
/* - *.tmp excludes only the tmp file; other files remain (default include). */
const char* exclude_only[] = {"- *.tmp"};
char err[160];
FilterRuleList* base = filter_base_build(exclude_only, 1, false, err, sizeof(err));
EXPECT_NOT_NULL(base);
ScannerOptions options = {0};
options.base_filters = base;
if (parallel)
options.num_threads = 2;
char** paths = NULL;
int count = 0;
int rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
: collect_files(root, &options, &paths, &count);
EXPECT_EQ_INT(rc, 0);
EXPECT_EQ_INT(count, 2);
EXPECT_TRUE(has_path(paths, count, "a.txt"));
EXPECT_TRUE(has_path(paths, count, "c.txt"));
EXPECT_FALSE(has_path(paths, count, "b.tmp"));
free_paths(paths, count);
filter_rule_list_free(base);
/* Anchored include then exclude-all: only root-level keep* survives. */
const char* anchored[] = {"+ /a.txt", "- *"};
base = filter_base_build(anchored, 2, false, err, sizeof(err));
EXPECT_NOT_NULL(base);
options.base_filters = base;
rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
: collect_files(root, &options, &paths, &count);
EXPECT_EQ_INT(rc, 0);
EXPECT_EQ_INT(count, 1);
EXPECT_TRUE(has_path(paths, count, "a.txt"));
free_paths(paths, count);
filter_rule_list_free(base);
unlink("test_scan_filter/a.txt");
unlink("test_scan_filter/b.tmp");
unlink("test_scan_filter/c.txt");
rmdir(root);
}
/* Anchored dir-only rules prune a whole subtree. */
static void test_filter_dir_only_and_anchored(bool parallel) {
const char* root = "test_scan_filter_dir";
const char* sub = "test_scan_filter_dir/sub";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir(sub, 0755), 0);
create_test_file("test_scan_filter_dir/sub/inner.txt", "x");
create_test_file("test_scan_filter_dir/keep.txt", "keep");
const char* rules[] = {"- /sub/"};
char err[160];
FilterRuleList* base = filter_base_build(rules, 1, false, err, sizeof(err));
EXPECT_NOT_NULL(base);
ScannerOptions options = {0};
options.base_filters = base;
if (parallel)
options.num_threads = 2;
char** paths = NULL;
int count = 0;
int rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
: collect_files(root, &options, &paths, &count);
EXPECT_EQ_INT(rc, 0);
EXPECT_EQ_INT(count, 1);
EXPECT_TRUE(has_path(paths, count, "keep.txt"));
EXPECT_FALSE(has_path(paths, count, "sub/inner.txt"));
free_paths(paths, count);
filter_rule_list_free(base);
unlink("test_scan_filter_dir/sub/inner.txt");
unlink("test_scan_filter_dir/keep.txt");
rmdir(sub);
rmdir(root);
}
/* -C default CVS excludes prune .git/ directories and *.o files. */
static void test_cvs_defaults(bool parallel) {
const char* root = "test_scan_cvs";
const char* git = "test_scan_cvs/.git";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir(git, 0755), 0);
create_test_file("test_scan_cvs/.git/config", "cfg");
create_test_file("test_scan_cvs/object.o", "o");
create_test_file("test_scan_cvs/keep.txt", "keep");
char err[160];
FilterRuleList* base = filter_base_build(NULL, 0, true, err, sizeof(err));
EXPECT_NOT_NULL(base);
ScannerOptions options = {0};
options.base_filters = base;
if (parallel)
options.num_threads = 2;
char** paths = NULL;
int count = 0;
int rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
: collect_files(root, &options, &paths, &count);
EXPECT_EQ_INT(rc, 0);
EXPECT_EQ_INT(count, 1);
EXPECT_TRUE(has_path(paths, count, "keep.txt"));
EXPECT_FALSE(has_path(paths, count, ".git/config"));
EXPECT_FALSE(has_path(paths, count, "object.o"));
free_paths(paths, count);
filter_rule_list_free(base);
unlink("test_scan_cvs/.git/config");
unlink("test_scan_cvs/object.o");
unlink("test_scan_cvs/keep.txt");
rmdir(git);
rmdir(root);
}
/* -F: a .rsync-filter placed in a directory governs its subtree and the file
* itself is never transferred. */
static void test_per_dir_filter(bool parallel) {
const char* root = "test_scan_perdir";
const char* sub = "test_scan_perdir/sub";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir(sub, 0755), 0);
create_test_file("test_scan_perdir/drop.tmp", "tmp");
create_test_file("test_scan_perdir/keep.txt", "keep");
create_test_file("test_scan_perdir/sub/nested.tmp", "tmp");
create_test_file("test_scan_perdir/.rsync-filter", "- *.tmp\n");
ScannerOptions options = {0};
options.per_dir_filters = true;
if (parallel)
options.num_threads = 2;
char** paths = NULL;
int count = 0;
int rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
: collect_files(root, &options, &paths, &count);
EXPECT_EQ_INT(rc, 0);
EXPECT_EQ_INT(count, 1);
EXPECT_TRUE(has_path(paths, count, "keep.txt"));
EXPECT_FALSE(has_path(paths, count, "drop.tmp"));
EXPECT_FALSE(has_path(paths, count, "sub/nested.tmp"));
EXPECT_FALSE(has_path(paths, count, ".rsync-filter"));
free_paths(paths, count);
unlink("test_scan_perdir/drop.tmp");
unlink("test_scan_perdir/keep.txt");
unlink("test_scan_perdir/sub/nested.tmp");
unlink("test_scan_perdir/.rsync-filter");
rmdir(sub);
rmdir(root);
}
/* scanner_path_relative maps an on-disk path to its transfer-relative path,
* including the "/" transfer-root edge case (regression: children of "/" used
* to abort the scan because the suffix was mis-read). */
static void test_scanner_path_relative() {
char* rel = NULL;
rel = scanner_path_relative("/", "/");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "");
free(rel);
rel = scanner_path_relative("/", "/etc");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "etc");
free(rel);
rel = scanner_path_relative("/", "/etc/passwd");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "etc/passwd");
free(rel);
/* Normal roots: with and without a trailing slash on the root. */
rel = scanner_path_relative("/tmp/foo", "/tmp/foo");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "");
free(rel);
rel = scanner_path_relative("/tmp/foo", "/tmp/foo/bar");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "bar");
free(rel);
rel = scanner_path_relative("/tmp/foo/", "/tmp/foo/bar/baz.txt");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "bar/baz.txt");
free(rel);
/* A path outside the root maps to NULL. */
EXPECT_NULL(scanner_path_relative("/tmp/foo", "/tmp"));
EXPECT_NULL(scanner_path_relative("/tmp/foo", "/tmp/foobar"));
}
/* rsync precedence: a deeper .rsync-filter overrides a shallower one, so an
* inner "+ *.tmp" re-includes what the outer "- *.tmp" excluded. */
static void test_per_dir_filter_override(bool parallel) {
const char* root = "test_scan_perdir_ovr";
const char* sub = "test_scan_perdir_ovr/sub";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir(sub, 0755), 0);
create_test_file("test_scan_perdir_ovr/.rsync-filter", "- *.tmp\n");
create_test_file("test_scan_perdir_ovr/sub/.rsync-filter", "+ *.tmp\n");
create_test_file("test_scan_perdir_ovr/top.tmp", "x");
create_test_file("test_scan_perdir_ovr/keep.txt", "keep");
create_test_file("test_scan_perdir_ovr/sub/inside.tmp", "x");
ScannerOptions options = {0};
options.per_dir_filters = true;
if (parallel)
options.num_threads = 2;
char** paths = NULL;
int count = 0;
int rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
: collect_files(root, &options, &paths, &count);
EXPECT_EQ_INT(rc, 0);
/* top.tmp is still excluded by the root file; inside.tmp is re-included by
* the subdir file; .rsync-filter files are never transferred. */
EXPECT_EQ_INT(count, 2);
EXPECT_TRUE(has_path(paths, count, "keep.txt"));
EXPECT_TRUE(has_path(paths, count, "sub/inside.tmp"));
EXPECT_FALSE(has_path(paths, count, "top.tmp"));
EXPECT_FALSE(has_path(paths, count, ".rsync-filter"));
EXPECT_FALSE(has_path(paths, count, "sub/.rsync-filter"));
free_paths(paths, count);
unlink("test_scan_perdir_ovr/top.tmp");
unlink("test_scan_perdir_ovr/keep.txt");
unlink("test_scan_perdir_ovr/sub/inside.tmp");
unlink("test_scan_perdir_ovr/.rsync-filter");
unlink("test_scan_perdir_ovr/sub/.rsync-filter");
rmdir(sub);
rmdir(root);
}
void test_scanner() {
test_scanner_single_file();
test_scanner_multiple_files();
@@ -672,4 +1097,17 @@ void test_scanner() {
test_scanner_one_file_system_same_device();
test_parallel_scanner_one_file_system_same_device();
test_scanner_one_file_system_cross_device();
test_files_from_subset(false);
test_files_from_subset(true);
test_filter_rules(false);
test_filter_rules(true);
test_filter_dir_only_and_anchored(false);
test_filter_dir_only_and_anchored(true);
test_cvs_defaults(false);
test_cvs_defaults(true);
test_per_dir_filter(false);
test_per_dir_filter(true);
test_scanner_path_relative();
test_per_dir_filter_override(false);
test_per_dir_filter_override(true);
}