Back to project page

NAME

continual-sync.conf - The configuration file for continual-sync

SYNOPSIS

The continual-sync.conf file is a configuration file for continual-sync(1). It defines one or more directory trees to be synchronised, where to synchronise them to, and the associated parameters.

FILE FORMAT

The file consists of sections and parameters. A section begins with the name of the section in square brackets and continues until the next section begins. Sections contain parameters of the form:

name = value

The file is line-based - that is, each newline-terminated line represents either a comment, a section name or a parameter.

Section and parameter lines are case sensitive.

Section names must consist only of alphanumeric characters, "-", ".", and "_".

Only the first equals sign in a parameter is significant. Whitespace before or after the first equals sign is discarded. Leading, trailing and internal whitespace in parameter names is irrelevant. Leading and trailing whitespace in a parameter value is discarded. Internal whitespace within a parameter value is retained verbatim.

Any hash ("#") character causes that hash, and the rest of that line, to be ignored.

A maximum of 999 sections can be defined, across all configuration files read by continual-sync.

SECTIONS

Each section defines a directory tree to be synchronised. It must have a unique name and give a source and a destination. The bare minimum for a valid section definition is therefore:

[section_name]
source = /path/to/source/directory/
destination = remotehost:/remote/path/to/sync/to/

Additional parameters may be specified to fine-tune the synchronisation process.

The exception to the above is the special defaults section, which defines default values which are used by all other sections. The defaults section must not specify either a source or a destination.

PARAMETERS

These parameters can only appear within a section, and apply only to that section. Unless otherwise specified, every parameter is optional; any which are not defined will use the setting from the defaults section if there is one.

source
The path to the directory to be used as the source for synchronisation. It must be a normal pathname (not prefixed with a hostname) and must be a directory.

This parameter is mandatory for every section except defaults, where it is forbidden.

It is usually best to ensure that the trailing slash (/) is present to ensure correct operation with rsync(1).

destination
The path, as given to rsync(1), to the directory to be used as the synchronisation destination. It must be a directory, but can be in any form accepted by rsync, such as remotehost:/path/.

This parameter is mandatory for every section except defaults, where it is forbidden.

It is usually best to ensure that the trailing slash (/) is present to ensure correct operation with rsync(1).

exclude
A glob(7) pattern to exclude. Any file or directory, in any subdirectory of the source directory, which matches this pattern will be ignored.

This parameter can be specified multiple times per section.

The default is to exclude *.tmp and *~, unless overridden by the defaults section.

source validation command
A command line (run with "sh -c") which must exit with 0 (success) before any watch can start and before any synchronisation will be performed.

This can be used to check that the source directory is mounted. For example:

source validation command = mountpoint -q %s

The default is to use no source validation command, unless overridden by the defaults section. To explicitly state that no source validation command is to be run, use a value of none.

destination validation command
A command line (run with "sh -c") which must exit with 0 (success) before any synchronisation will be performed.

This can be used to check that the destination host is up and the directory is mounted. For example:

destination validation command = ssh %h mountpoint -q %d

The default is to use no destination validation command, unless overridden by the defaults section. To explicitly state that no destination validation command is to be run, use a value of none.

full sync interval
The interval, in seconds, between full synchronisations. A full synchronisation is done by running rsync(1) over the entire source directory, regardless of what changes have been seen.

If no full sync marker file has been defined, then a full sync is run every time the watch starts.

The default full sync interval is 86400 seconds (1 day) unless overridden by the defaults section.

If the full sync interval is set to 0, then a full sync will never be run. This is not recommended unless you will be running a full sync yourself through some other mechanism, as the partial sync process is not perfect and so some changes may be missed.

full sync retry
The number of seconds to wait after an unsuccessful full sync before trying another one. An unsuccessful sync is one for which a validation command failed or where rsync returned nonzero.

The default full sync retry wait time is 3600 seconds (1 hour) unless overridden by the defaults section.

partial sync interval
The interval, in seconds, between partial synchronisations. A partial synchronisation only runs rsync(1) over the files and directories on which changes have been seen.

The default partial sync interval is 30 seconds unless overridden by the defaults section.

If the partial sync interval is set to 0, then the source directory will not be watched for changes and no partial synchronisations will be run.

partial sync retry
The number of seconds to wait after an unsuccessful partial sync before trying another one. An unsuccessful sync is one for which a validation command failed or where rsync returned nonzero.

The default partial sync retry wait time is 300 seconds (5 minutes) unless overridden by the defaults section.

recursion depth
The maximum number of subdirectories deep that a watch will descend. If this is set too high, then excessive system resources will be consumed.

The default recursion depth is 20 unless overridden by the defaults section.

Subdirectories deeper than this will not be watched for changes, but they will be picked up by rsync when a full sync is run.

full sync marker file
The path to a file which will have its last modification time updated every time a successful full sync is run.

This can be used to ensure that a full sync is not run more often than the full sync interval, even if the program is restarted.

The default is to use no full sync marker file, unless overridden by the defaults section. To explicitly state that no full sync marker file is to be used, use a value of none.

partial sync marker file
The path to a file which will have its last modification time updated every time a successful partial sync is run.

This can be used to expose the information to other programs.

The default is to use no partial sync marker file, unless overridden by the defaults section. To explicitly state that no partial sync marker file is to be used, use a value of none.

change queue
The path to a directory where watchdir(1) can record details of changes to the source directory. This directory must be empty, and not used by anything else, so must be unique for each section. The contents are deleted when the transfer list is generated before each partial sync.

The default is to create a temporary directory which is automatically removed when the program exits. If a value has been specified in the defaults section, you can override it on a per-section basis with a value of none to go back to the temporary directory behaviour.

transfer list
The path to a file which can be used to collate the changes from the change queue to produce a list of changes to pass to rsync(1). The file is deleted and recreated, so must be unique for each section and must exist in a directory writable by the program.

The default is to create a temporary file which is automatically removed when the program exits. If a value has been specified in the defaults section, you can override it on a per-section basis with a value of none to go back to the temporary file behaviour.

temporary directory
The temporary directory under which a working directory is created by each sync process, for the default change queue directory and transfer list, as well as other working files such as the excludes list.
  The default is /tmp, or whatever is specified in the defaults section.
sync lock
The path to a file which is locked with lockf(3) prior to running a full or a partial sync.

This can be used to ensure that only one sync at a time runs to a given destination; for example:

[dir1]
source = /home/dir1/
destination = host1:/home/dir1/
sync lock = /var/lock/sync-%h.lock

 
[dir2]
source = /home/dir2/
destination = host1:/home/dir2/
sync lock = /var/lock/sync-%h.lock

 
[dir3]
source = /home/dir3/
destination = host2:/home/dir3/
sync lock = /var/lock/sync-%h.lock

In the above example, dir1 and dir2 share the same sync lock of /var/lock/sync-host1.lock, and so only one of them at a time would ever be running a sync; dir3 uses a different lock and so could be running a sync at the same time as either dir1 or dir2.

The default is not to use a lock, unless overridden by the defaults section.

To explicitly state that no lock is to be used, specify a value of none.

full rsync options
The options to pass to rsync(1) when running a full sync, other than the source, destination, and exclusions.

The default is "--delete -axH".

partial rsync options
The options to pass to rsync(1) when running a partial sync, other than the source, destination, exclusions, and --files-from partial transfer list.

The default is "--delete -dlptgoDH".

ignore vanished files
If this is set to "yes" or "on", then an rsync(1) exit status of 24, meaning "partial transfer due to vanished source files", is treated as if it were a successful transfer.

Use this when files disappear from the source quite often, such as a Subversion repository that is in active use.

log file
The file to record all activity to. It is locked while writing, so can be shared by multiple sections, and closed while not being written to, so can be safely rotated with logrotate(8).

The default is to write no logs, unless overridden by the defaults section.

To explicitly state that no log file is to be written, use a value of none.

status file
A file which contains the current status of this sync process. It is rewritten every time something changes, and contains the following fields in the format parameter : value, one per line:
section
The section name.
current action
The current action being performed on this section, if any: one of -, VALIDATE-SOURCE, VALIDATE-DESTINATION, SYNC-FULL-AWAITING-LOCK, SYNC-FULL, SYNC-PARTIAL-AWAITING-LOCK, or SYNC-PARTIAL.
sync process
The process ID of this section's sync process.
watcher process
The process ID of the directory change watcher sub-process, or "-" if there isn't one. There only wouldn't be one if the partial sync interval is 0 or if the source validation command failed.
last full sync status
Whether the last full sync succeeded ("OK") or failed ("FAILED"), or "-" if none was run yet.
last partial sync status
Whether the last partial sync succeeded ("OK") or failed ("FAILED"), or "-" if none was run yet.
last full sync
The time of the last successful full sync.
last partial sync
The time of the last successful partial sync.
next full sync
The time when the next full sync will be attempted.
next partial sync
The time when the next partial sync will be attempted.
failed full sync
The time of the last failed full sync.
failed partial sync
The time of the last failed partial sync.
full sync failures
How many full syncs have failed in a row.
partial sync failures
How many partial syncs have failed in a row.
working directory
The temporary working directory used by this section's sync process for files like the rsync error log.
Times are in YYYY-MM-DD HH:MM:SS format, in the local time zone, or "-" to mean "never".

The status file is not read by the sync process - use the marker file options if you want to retain memory of the last full and partial sync times between invocations of continual-sync(1).

The default is to write no status file, unless overridden by the defaults section.

To explicitly state that no status file is to be written, use a value of none.

SPECIAL PARAMETERS

The following special parameters can appear anywhere in a configuration file:

include
Include the given configuration file(s) into this one. Filenames can be relative to the current configuration file, and glob patterns are expanded. For example, the following will include all configuration files found in /etc/continual-sync.conf.d/:

include = /etc/continual-sync.conf.d/*

Files ending in ~, .rpmsave, .rpmorig, and .rpmnew are automatically skipped.

VARIABLE SUBSTITUTIONS

The following character sequences may be used in the values of certain parameters:

%n
the current section name
%s
the source directory
%h
the destination host (defined as the part of destination before the first colon (":"), or "localhost" if there is no colon)
%d
the destination directory (defined as the part of destination after the last colon (":"), or the whole destination if there is no colon)

These may be used in any of the following parameter values:

source validation command
destination validation command
full sync marker file
partial sync marker file
change queue
transfer list
temporary directory
sync lock
full rsync options
partial rsync options
log file
status file

REPORTING BUGS

Report bugs in continual-sync by using the contact form linked from the package home page: <http://www.ivarch.com/programs/continual-sync.shtml>

SEE ALSO

continual-sync(1), watchdir(1), rsync(1), glob(7)

LICENSE

This is free software, distributed under the ARTISTIC 2.0 license.

Back to project page