ivarch.com: Continual Sync (Watchdir): Online Manual

NAME

watchdir - watch for changes in a directory tree

SYNOPSIS

watchdir [OPTION]... DIRECTORY OUTPUTDIR
watchdir [OPTION]... -o OUTPUTDIR DIRECTORY...
watchdir [-h|-V]

DESCRIPTION

watchdir monitors the given DIRECTORY and all its subdirectories for changes. At regular intervals, the names of the files and directories which have changed are written to a uniquely named file in the OUTPUTDIR directory:

If a file is changed or created, its filename will be listed.
If a file is removed, this is treated as a change to the directory it was in, so the directory will be listed.
If a file is renamed, this is treated as a removal and creation, so both the file's new path and the directory containing its old name will be listed.

Changes to directories are treated in the same way:

If a directory is created, the new directory will be listed.
If a directory is removed, its parent directory will be listed.
If a directory is renamed, its old parent directory will be listed, and its new path will be listed.

When there is only one top-level DIRECTORY, and --absolute is not switched on, all paths are given relative to that directory. Directory names always end in /, which means that just “/” on its own in a change file means that the top-level directory itself changed (i.e. a file or directory was removed from it).

The change files in OUTPUTDIR are given names of the form “YYYYMMDD-hhmmss.pid”, where YYYY is the 4-digit year, MM is the 2-digit month, DD is the 2-digit day of the month, hh, mm, ss are 2-digit hours, minutes, and seconds in the 24-hour clock, and pid is the process ID of watchdir.

Note that OUTPUTDIR must not be a subdirectory of DIRECTORY.

If DIRECTORY itself is removed, watchdir exits immediately.

OPTIONS

-i, --dump-interval SEC

Write change files every SEC seconds. The default is 30 seconds. No change files are written when there are no changes to report.

-0, --null

Use a null byte instead of a newline as the delimiter between filenames written to the change files. This prevents ambiguity when dealing with files that have newlines in their names.

-a, --absolute

Write absolute paths to the change files, instead of paths relative to the top-level directory. This is automatically switched on if more than one top-level directory is specified.

-F, --only-files

Do not write directory names to the change files. This means that directory changes are not listed, nor are file deletions. Use this when you are only interested in new or changed files, such as when looking for newly altered files to scan for malware.

-D, --only-directories

Do not write filenames to the change files. This means that only file deletions, and creation, removal, and renaming of directories, will cause changes to be listed. Use this when you are only interested in file removals or directory structure changes.

-n, --no-file-tracking

Do not track filenames, file sizes, or modification times. This means that files which are opened for appending may be treated as having changed even when no change is made, but removes the need to call lstat(2) on every file and hold their details in memory.

When this option is active, directory rescans (--full-scan-interval) will not look for added or removed files, only subdirectories.

Use this when there are so many files that memory usage becomes an issue, or if calling lstat(2) degrades performance, or when you are more interested in files being possibly changed than in minimising the size of the change list.

For example, this can be useful when using the output of watchdir(1) to trigger malware scans, or when monitoring directories containing a very large number of files.

-s, --suffix SUFFIX

Append SUFFIX to the name of each change file, such as “.list”.

-o, --output-dir OUTPUTDIR

Write change files to the OUTPUTDIR directory. Use this option when there is more than one directory to watch. If any of the watched directories are removed, watchdir will exit.

-f, --full-scan-interval SEC

Rescan DIRECTORY every SEC seconds, to pick up any changes that might have been missed. The default is 7200 seconds (2 hours).

-e, --exclude PATTERN

A glob(7) pattern to exclude. Any file or directory, in any subdirectory of DIRECTORY, which matches this pattern will be ignored.

This option can be specified multiple times.

The default is to exclude “*.tmp” and “*~”.

-r, --recursion-depth NUM

Descend no more than NUM directories deep into DIRECTORY. The default is 20. It is not advisable to set this too high as it may cause excessive consumption of system resources.

-q, --queue-run-interval SEC

Process the inotify(7) change queue every SEC seconds. The default is 2 seconds. This will rarely need to be changed.

-m, --queue-run-max SEC

Spend no longer than SEC seconds processing the change queue before checking inotify(7) again, to avoid overflows caused by many changes happening at once. The default is 5 seconds. This will rarely need to be changed.

-x, --exchange-dir DIR

Exchange information with other watchdir instances on other hosts using subdirectories of DIR. See the Clustering subsection under NOTES for more details.

-t, --exchange-interval SEC

Exchange information with other watchdir instances every SEC seconds. The default is 1. See the Clustering subsection under NOTES for more details.

-I, --identity NAME

Use NAME as the identity of this host under the exchange directory. The default is to use the current hostname (“uname -n”). See the Clustering subsection under NOTES for more details.

-M, --metrics FILE

Write metrics to FILE every few seconds, replacing the file each time. See the Metrics subsection under NOTES for more details.

-h, --help

Print a usage message on standard output and exit successfully.

-V, --version

Print version information on standard output and exit successfully.

NOTES

The change files can be used directly as input to rsync(1)'s --files-from option, like this:

rsync --delete -dlptgoDH --files-from=OUTPUTDIR/20140731-112501.1234 DIRECTORY remote:/dir/copy/

A similar command line is used by continual-sync(8) when doing a partial sync.

Changes to file permissions are not listed - only changes which alter the contents of a file or its last-modification time.

When watching a directory with a large number of subdirectories, it may take watchdir a long time to start watching for changes in all of them. It will also take longer for watchdir to terminate, as it closes down all of its open watches.

System tuning

If you watch a lot of directories, you will probably need to increase the kernel parameter fs.inotify.max_user_watches, for example:

sysctl -w fs.inotify.max_user_watches=1048576

Use the “--metrics” option and monitor the metrics file's directories_watching value to see whether the system-wide limit is in danger of being reached.

You may also need to increase fs.inotify.max_user_instances if many watchdirs are running on the same system.

Clustering

When watchdir is monitoring a directory on a shared filesystem, it won't know about subdirectories created by other hosts sharing the same filesystem, and so won't be able to start watching them until its next full scan.

For example, you could have a cluster of web or FTP servers with a shared filesystem, and use watchdir on each one to make lists of newly changed files. These lists could be read and de-duplicated by a separate admin server with the same shared filesystem, so that the admin server could scan the changed files for malware.

With the “--exchange-dir” option, each watchdir can tell the others about any directories they see being created, renamed, or deleted. This reduces the risk that changes will be missed.

Within the exchange directory, each watchdir creates a subdirectory named after its identity (the “--identity” option, or if that option is not given, its hostname). When it sees any directory changes, it writes a file describing them to every other subdirectory of the exchange directory except its own. When it sees any files appear in its own subdirectory of the exchange directory, it deletes them and uses them to update its directory watch list.

The files are written first to a subdirectory named “.tmp” under the exchange directory, and then, every exchange interval, they are hard-linked into every other subdirectory that starts with a letter or number (except for the one matching its own identity), and the original is removed.

When using the “--exchange-dir” option to allow clustering like this, ensure you have a separate cleanup process which regularly deletes files from it that are older than a few minutes, in case a watcher exits without removing its directory.

Metrics

The “--metrics” option writes counters to a file, replacing the file every few seconds. Each counter is on its own line, prefixed by a label and a space. The counters are:

directories_watching: The number of active inotify(7) directory watches.
change_files_output: The number of change files written to the output directory.
changed_paths_output: The number of paths written to change files.
exchange_files_output: The number of exchange information files written to the exchange directory for collection by other watchdir instances. Files are counted before the hard-link operation, so one exchange file is counted as one, regardless of how many exchange subdirectories it is linked into.
exchange_paths_output: The number of paths written to exchange information files.
exchange_files_input: The number of exchange information files read from other watchdir instances.
exchange_paths_input: The number of paths read from exchange information files, regardless of whether they were successfully matched to a watched directory.
watch_index_length: The total length of all of the tables for each top-level directory which track watched directories. This should be the same as directories_watching.
change_queue_length: The total length of all change queues for each top-level directory. Change queues track files and directories for which change events have been received, but not processed, such as directories that are waiting to be scanned. This will typically rise during initial scanning, or when a lot of changes happen, and then fall.

REPORTING BUGS

This is part of the continual-sync package.

Report bugs in continual-sync to the project's issue tracker or use the contact form linked from the project's home page.

COPYRIGHT

License GPLv3+: GNU GPL version 3 or later.

This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

Please see the package's ACKNOWLEDGEMENTS file for a complete list of contributors.

← Back to the project page