Archives/osync
2
0
Fork 0
mirror of https://github.com/deajan/osync synced 2024-11-03 15:40:14 +00:00
Code Issues Releases Wiki Activity
6f91d4a442
osync/osync v1.1.lyx
deajan 21994ce096 Updated documentation for release
2016-07-27 10:47:55 +02:00

3266 lines
61 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

#LyX 2.2 created this file. For more info see http://www.lyx.org/
\lyxformat 508
\begin_document
\begin_header
\save_transient_properties true
\origin unavailable
\textclass article
\use_default_options true
\maintain_unincluded_children false
\language english
\language_package default
\inputencoding utf8x
\fontencoding global
\font_roman "default" "default"
\font_sans "default" "default"
\font_typewriter "default" "default"
\font_math "auto" "auto"
\font_default_family default
\use_non_tex_fonts true
\font_sc false
\font_osf false
\font_sf_scale 100 100
\font_tt_scale 100 100
\graphics default
\default_output_format default
\output_sync 0
\bibtex_command default
\index_command default
\paperfontsize default
\spacing single
\use_hyperref true
\pdf_title "Osync Configuration guide"
\pdf_author "Orsiris "
\pdf_bookmarks true
\pdf_bookmarksnumbered false
\pdf_bookmarksopen false
\pdf_bookmarksopenlevel 1
\pdf_breaklinks false
\pdf_pdfborder true
\pdf_colorlinks false
\pdf_backref section
\pdf_pdfusetitle true
\papersize a4paper
\use_geometry true
\use_package amsmath 1
\use_package amssymb 1
\use_package cancel 0
\use_package esint 1
\use_package mathdots 1
\use_package mathtools 0
\use_package mhchem 1
\use_package stackrel 0
\use_package stmaryrd 0
\use_package undertilde 0
\cite_engine basic
\cite_engine_type default
\biblio_style plain
\use_bibtopic false
\use_indices false
\paperorientation portrait
\suppress_date false
\justification true
\use_refstyle 1
\boxbgcolor #d0d0d0
\index Index
\shortcut idx
\color #008000
\end_index
\leftmargin 2cm
\topmargin 2cm
\rightmargin 2cm
\bottommargin 2cm
\headheight 1cm
\headsep 1cm
\footskip 1cm
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\paragraph_indentation 2em
\quotes_language swedish
\papercolumns 1
\papersides 1
\paperpagestyle default
\listings_params "backgroundcolor={\color{white}},basicstyle={\ttfamily},breaklines=true,frame=single"
\bullet 0 0 6 -1
\tracking_changes false
\output_changes false
\html_math_output 0
\html_css_as_file 0
\html_be_strict false
\end_header
\begin_body
\begin_layout Title
osync v1.1 Documentation
\end_layout
\begin_layout Author
(C) 2013-2016 by Orsiris de Jong
\end_layout
\begin_layout Date
27 July 2016
\end_layout
\begin_layout Standard
\begin_inset CommandInset href
LatexCommand href
name "http://www.netpower.fr/osync"
target "http://www.netpower.fr/osync"
\end_inset
\end_layout
\begin_layout Standard
\begin_inset CommandInset line
LatexCommand rule
offset "0.5ex"
width "100col%"
height "1pt"
\end_inset
\end_layout
\begin_layout Standard
\begin_inset CommandInset toc
LatexCommand tableofcontents
\end_inset
\end_layout
\begin_layout Section
Introduction
\end_layout
\begin_layout Subsection
Quickstart guide
\end_layout
\begin_layout Standard
osync is a command line two way synchronization tool for Linux / BSD / MacOSX
and Windows, with an emphasis on reliability and automation on low bandwidth
links.
\end_layout
\begin_layout Standard
A quickstart guide can be found in the README.md file.
\end_layout
\begin_layout Subsection
Basic synchronization problems
\end_layout
\begin_layout Standard
Synchronization is usually found in two flavors, bloc level sync and file
level sync.
While whole bloc level synchronization is generally a good way of doing,
it's also very greedy in network ressources and is not easy to setup.
That's where file level sync comes in handy, where only some directories
& files need to be synced.
\end_layout
\begin_layout Standard
Imagine you're syncing two remote offices of a same company.
If you're syncing a user's home directory or it's roaming profile as a
night task, the next day, the user will find it's roaming profile up to
date at the remote office.
\end_layout
\begin_layout Standard
But what would happen if two users work on the same file in a public folder,
at the same time, on both offices ? Some sync software would stop sync
and ask what to do.
Others might simply deleted the oldest version of the file, even if it
was modified on both sides.
\end_layout
\begin_layout Standard
Also, what would happen if a user uploads a lot of data ? If the link between
both offices cannot handle enough data transfer in a given time, any other
sync task won't be run, and the sync would continue during the day, when
bandwidth is necessary elsewhere.
\end_layout
\begin_layout Standard
What would happen if a power fault occurs while synchronization is going
on ?
\end_layout
\begin_layout Subsection
What osync can do
\end_layout
\begin_layout Subsubsection
Making synchronization reliable
\end_layout
\begin_layout Standard
osync is designed to synchronize two folders on both local and / or remote
systems.
\end_layout
\begin_layout Standard
It is time controlled, which means you can decide how much time it should
spend on a sync task before stopping it and launching the next one.
\end_layout
\begin_layout Standard
It's designed to resume failed or stopped sync tasks, or totally restart
the sync task if resume fails.
\end_layout
\begin_layout Standard
It can keep multiple versions of a file in case of a conflict.
\end_layout
\begin_layout Standard
It handles soft deletion.
If a user deletes a file on replica A, it will move that file on replica
B to the
\begin_inset Quotes sld
\end_inset
.osync_workdir/deleted
\begin_inset Quotes srd
\end_inset
folder.
\end_layout
\begin_layout Standard
It will automatically clean old files (soft deleted and conflict backups)
after a defined amount of days.
\end_layout
\begin_layout Standard
It will perform various checks before launching a synchronization like free
disk space.
\end_layout
\begin_layout Subsubsection
Making a sysadmin's life easier
\end_layout
\begin_layout Standard
osync is also desgined to ease synchronization setups.
\end_layout
\begin_layout Standard
It will trigger an email alert including the whole sync process execution
log if a warning / error is found.
\end_layout
\begin_layout Standard
Pre-processing and post-processing commands can be launched on local and
/ or remote systems, which may be useful to launch snapshot software, flush
or standby virtual machines, etc).
\end_layout
\begin_layout Standard
Multiple concurrent instances of osync can be run as long as they don't
sync the same replicas at the same time.
\end_layout
\begin_layout Standard
A batch processing script is included (osync-batch.sh) to launch sequential
sync tasks.
Failed sync tasks are rerun if every other task has completed and there's
still some time left in a given timespan.
\end_layout
\begin_layout Standard
osync can use rsync or ssh tunnel compression to gain bandwidth.
Bandwidth can also be limited for slow link sharing.
\end_layout
\begin_layout Standard
It can be run in quicksync mode for the impatient (nothing to configure
except the replica paths), or with a full blown config file.
\end_layout
\begin_layout Standard
You may run osync manually, schedule it with cron, or have it monitor a
directory as a daemon and launch sync tasks on file modifications.
\end_layout
\begin_layout Standard
osync has been succesfully tested on RHEL / CentOS 5, CentOS 6, Centos 7,
Debian 6, Debian 7, Linux Mint 14, 17, FreeBSD 8.3, 10.3, pfSense 2.x, Mac
OS X, and Windows using msys & cygwin environment.
\end_layout
\begin_layout Subsubsection
What osync cannot do
\end_layout
\begin_layout Standard
osync is a simple bash script that relies on other tools like rsync.
\end_layout
\begin_layout Standard
Hence, it has some advantages and disavantages:
\end_layout
\begin_layout Standard
Advantages:
\end_layout
\begin_layout Standard
- It's easily customisable
\end_layout
\begin_layout Standard
- It's fast
\end_layout
\begin_layout Standard
Disavantages:
\end_layout
\begin_layout Standard
- There's no way to detect file moves.
If you move a directory on replica A, osync will soft delete the directory
on replica B and copy the new directory from replica A.
\end_layout
\begin_layout Standard
- There's no multi-master replication in osync V1.
Hence, if you want to sync replicas A, B, and C using osync, you'll have
to use one of the following schemas:
\end_layout
\begin_layout Paragraph
Replicate 3-way with A as master
\end_layout
\begin_layout Standard
Run the following tasks sequentially where A is the initiatior:
\end_layout
\begin_layout Standard
Replicate A & B
\end_layout
\begin_layout Standard
Replicate A & C
\end_layout
\begin_layout Standard
Replicate A & B
\end_layout
\begin_layout Paragraph
Replicate 3-way with A & B as masters
\end_layout
\begin_layout Standard
Run the following tasks from each system.
Be sure they won't run concurrently (osync will detect that another replication
is still running, and abort the current one):
\end_layout
\begin_layout Standard
Replicate A & B where A is the initiator
\end_layout
\begin_layout Standard
Replicate B & C where B is the initiator
\end_layout
\begin_layout Standard
Replicate C & A where C is the initiator
\end_layout
\begin_layout Subsection
Why use osync
\end_layout
\begin_layout Standard
There are a lot of file sync tools out there, some probably better than
osync depending on the use case.
\end_layout
\begin_layout Standard
osync has been basically written to be low bandwidth friendly, with resume
options for unstable internet links (but it will also do great on a fiber
link :)).
\end_layout
\begin_layout Standard
It has also been written to be a setup and forget tool, without any user
interaction like manual conflict resolution.
\end_layout
\begin_layout Standard
osync also is one of the few tools that support ACL synchronization.
\end_layout
\begin_layout Standard
At least, osync consumes very little RAM and CPU ressources and is suitable
from lower en hardware up to serveres.
\end_layout
\begin_layout Standard
Hence, it has some unique features that make it a good tool depending on
the use case.
\end_layout
\begin_layout Subsection
How osync tries to resolve sync issues
\end_layout
\begin_layout Standard
Let's get back to the example above, where osync is used to sync two remote
offices with users' home directories.
\end_layout
\begin_layout Standard
Now imagine a user uploaded 100GB of data, and the WAN link between local
and remote systems can only handle 6GB/hour of data transfer.
\end_layout
\begin_layout Standard
Now if osync is scheduled every night at 10:00 pm, and it's configured to
run for maximum 10 hours, it would stop at 6am, after having transferred
60GB.
\end_layout
\begin_layout Standard
Then, on the next day, it would transfer the remaining 40GB from 10:00 pm
to about 3:30am.
\end_layout
\begin_layout Standard
Also, if you run sequential instances of osync (with osync-batch), one per
user directory for example, you can decide how much time osync should spend
per user.
So if a user uploads too much data, and osync cannot finish the synchronization
task for that user directory in a given timespan, it will stop that sync
task and run next user synchronization task so every user sync task gets
run, regardless of the amount of data.
If there's time left, osync-batch reprograms the user sync task that has
been stopped.
\end_layout
\begin_layout Subsection
Naming in this document
\end_layout
\begin_layout Standard
osync's goal is to synchronize two directories, that can be hosted on the
same computer or two different ones.
\end_layout
\begin_layout Standard
The computer that runs osync must have at least one of these two directories
mounted, and will be called the
\emph on
local system.
\end_layout
\begin_layout Standard
The first directory to sync on the local system is called the
\emph on
initiator replica
\emph default
.
\end_layout
\begin_layout Standard
The other directory, called the
\emph on
target replica
\emph default
can be hosted on the
\emph on
local system
\emph default
, or on another computer which will be called the
\emph on
remote system
\emph default
.
In that case, the
\emph on
local system
\emph default
will connect to the
\emph on
remote system
\emph default
through an ssh tunnel and synchronize both
\emph on
initiator
\emph default
and
\emph on
target replicas
\emph default
.
\end_layout
\begin_layout Standard
Any osync configuration file that gets executed is called an
\emph on
osync task
\emph default
.
\end_layout
\begin_layout Standard
In the following examples, a special user called
\emph on
syncuser
\emph default
is used for osync.
\end_layout
\begin_layout Subsection
How osync solves sync conflicts
\end_layout
\begin_layout Standard
Conflict resolution is done automatically.
When a file is modified on both replicas, osync compares the timestamps
on both replicas and keeps the most recent file.
\end_layout
\begin_layout Standard
In the highly uncommon case where both files have the exact same timestamp,
a configuration value called CONFLICT_PREVALANCE choses the which replica's
file will be kept.
By default, Initiator is chosen, unless specified otherwise in the config
file.
\end_layout
\begin_layout Standard
The file that isn't kept is copied to
\begin_inset Quotes sld
\end_inset
.osync_wordir/backups
\begin_inset Quotes srd
\end_inset
directory of the replica by default, with its full path relative to the
replica, unless specified otherwise by the CONFLICT_BACKUP configuration
value.
\end_layout
\begin_layout Standard
After an amount of days set by CONFLICT_BACKUP_DAYS, which is 30 by default,
the backed up file is deleted.
\end_layout
\begin_layout Standard
If CONFLICT_BACKUP_MULTIPLE is set to YES (disabled by default), multiple
versions of the backed up files are kept, with a timestamp suffix like
\begin_inset Quotes sld
\end_inset
2016.12.31-12.00.01
\begin_inset Quotes srd
\end_inset
.
\end_layout
\begin_layout Section
Prerequisites
\end_layout
\begin_layout Subsection
General packages
\end_layout
\begin_layout Standard
osync is programmed in bash and will not run with ksh / tsh or other shells.
Usually bash comes with most distributions.
\end_layout
\begin_layout Standard
On FreeBSD, you might need to install bash with
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
pkg install bash
\end_layout
\end_inset
\end_layout
\begin_layout Standard
The following packages are needed on both local and remote systems:
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
rsync coreutils
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Also, the local system will send emails on errors.
\end_layout
\begin_layout Standard
Make sure you have a mail package like mailx, mutt, postfix installed on
Unix like systems.
\end_layout
\begin_layout Standard
On Winows, make sure you have mailsend (https://github.com/muquit/mailsend)
/ sendmail (http://caspian.dotconf.net/menu/Software/SendEmail/) installed
in executable path.
\end_layout
\begin_layout Standard
Additionnaly, if you intend to run osync in daemon mode, you'll need the
following package.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
inotify-tools
\end_layout
\end_inset
\end_layout
\begin_layout Standard
On MinGW, you will have to install msys-rsync and msys-coreutils-ext on
top of a base install.
\end_layout
\begin_layout Subsection
File synchronization
\end_layout
\begin_layout Standard
File sync tasks don't need any special configurations.
You only have to worry about your sync user having the filesystem permissions
to read / write on both replicas.
\end_layout
\begin_layout Standard
A good way is to make your user member of the files' group that has full
permissions.
\end_layout
\begin_layout Standard
Another way to achieve this is using ACLs if your filesystem supports them.
You can add the following permissions for user "syncuser" on directory
"/home/web".
Setting a default rule will add rights on new files.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
# setfacl -dRm u:syncuser:r-x /home/web
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Be aware that ACLs are tricky and default unix permissions serve as mask
for ACLs.
\end_layout
\begin_layout Standard
Make always sure you can read /write to both replicas with your sync user:
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
# su syncuser
\end_layout
\begin_layout Plain Layout
$ cat /initiator/replica/test.file
\end_layout
\begin_layout Plain Layout
$ touch /initiator/replica/othertest.file
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Repeat that step for the target replica.
\end_layout
\begin_layout Subsection
Time setup
\end_layout
\begin_layout Standard
WARNING: osync's conflict resolution relies on timestamps, so it is very
important for all systems osync runs / syncs to, to have a reliable and
common timesource.
\end_layout
\begin_layout Standard
Please consider setting up NTPD first before you plan to run osync.
\end_layout
\begin_layout Subsection
\begin_inset CommandInset label
LatexCommand label
name "subsec:Performing-superuser-backups"
\end_inset
Performing superuser sync
\end_layout
\begin_layout Standard
osync may be run as superuser, which should always be avoided by granting
the read / write permissions to a dedicated sync user to both replicas.
\end_layout
\begin_layout Standard
There are still some cases where osync needs to be run as superuser, especially
when syncing system files.
\end_layout
\begin_layout Standard
In those cases, osync can be run as dedicated sync user and ask for sudo
permissions for specific commands.
\end_layout
\begin_layout Standard
In order to be able to use the sudo command without having to enter a password,
you’ll need to modify the local and / or remote system to allow the following
commands to be run as superuser: rsync, du, find, mkdir, rm, echo, mv,
tee and cat.
\end_layout
\begin_layout Standard
Use visudo to edit the sudoers file (or carefully edit /etc/sudoers yourself)
and add the following
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
syncuser ALL= NOPASSWD:/usr/bin/rsync
\end_layout
\begin_layout Plain Layout
syncuser ALL= NOPASSWD:/usr/bin/du
\end_layout
\begin_layout Plain Layout
syncuser ALL= NOPASSWD:/bin/find
\end_layout
\begin_layout Plain Layout
syncuser ALL= NOPASSWD:/bin/mkdir
\end_layout
\begin_layout Plain Layout
syncuser ALL= NOPASSWD:/bin/rm
\end_layout
\begin_layout Plain Layout
syncuser ALL= NOPASSWD:/bin/mv
\end_layout
\begin_layout Plain Layout
syncuser ALL= NOPASSWD:/bin/echo
\end_layout
\begin_layout Plain Layout
syncuser ALL= NOPASSWD:/bin/cat
\end_layout
\begin_layout Plain Layout
syncuser ALL= NOPASSWD:/usr/bin/tee
\end_layout
\end_inset
\end_layout
\begin_layout Standard
You might check the right paths to your commands (example to get path for
rsync executable):
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
# type rsync
\end_layout
\end_inset
\end_layout
\begin_layout Standard
You'll also need to disable requiretty in /etc/sudoers by adding the following
line:
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
Defaults:syncuser !requiretty
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Once your standard sync user is granted to run what osync needs, you can
enable sudo in osync's config file:
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
SUDO_EXEC=yes
\end_layout
\end_inset
\end_layout
\begin_layout Standard
You should be aware that there is a risk with having rsync command run as
superuser.
A user who can run rsync command as superuser can upload any file he wants
to the system, including a tweaked /etc/sudoers or /etc/passwd file.
Please read chapter
\begin_inset CommandInset ref
LatexCommand ref
reference "subsec:Enhancing-remote-backup"
\end_inset
to secure your installation.
\end_layout
\begin_layout Subsection
Remote sync
\end_layout
\begin_layout Standard
osync can perform local or remote synchronization tasks.
For local sync, pelease refer to chapters
\begin_inset CommandInset ref
LatexCommand ref
reference "sec:Running-Osync-in"
\end_inset
,
\begin_inset CommandInset ref
LatexCommand ref
reference "sec:Running-Osync-with-config-file"
\end_inset
and
\begin_inset CommandInset ref
LatexCommand ref
reference "sec:Running-Osync-as-daemon"
\end_inset
.
\end_layout
\begin_layout Standard
Remote synchronization is done through an SSH tunnel.
To be able to establish such a tunnel without having to enter a password,
you’ll have to generate a pair of private and public RSA keys.
\end_layout
\begin_layout Standard
The private part is kept by the computer that initiates the connection,
the local system.
The public part is kept on the remote system.
\end_layout
\begin_layout Standard
The following steps will be required to generate a ssh key:
\end_layout
\begin_layout Standard
Create a dedicated sync user and log in as that user on the local system
to perform the following actions.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
$ ssh-keygen -t rsa
\end_layout
\end_inset
\end_layout
\begin_layout Standard
This should create two files named ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub
\end_layout
\begin_layout Standard
You should also create a dedicated sync user on the remote system.
\end_layout
\begin_layout Standard
Copy the public part of the RSA pair to the remote system with scp (replace
22 with your ssh port number if needed).
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
$ scp -p 22 ~/.ssh/id_rsa syncuser@remotesystem.tld:/home/syncuser/.ssh/authorized_
keys
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Make sure the file is only readable and owned by the syncuser on the remote
system.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
$ chmod 600 /home/syncuser/.ssh/authorized_keys
\end_layout
\begin_layout Plain Layout
$ chown syncuser:root /home/syncuser/.ssh/authorized_keys
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Now you should be able to login as "syncuser" on the remote system without
any password.
You can try to remotely login by entering the following on the local system:
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
$ ssh -p 22 syncuser@remotesystem.tld
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Be aware that only the user that generated the ssh key can remotely log
in.
\end_layout
\begin_layout Standard
You may optionnaly enhance remote login security by applying chapter
\begin_inset CommandInset ref
LatexCommand ref
reference "subsec:Enhancing-remote-backup"
\end_inset
methods.
\end_layout
\begin_layout Subsection
Mail transport agent
\end_layout
\begin_layout Standard
You should make sure your system can send emails so osync can warn you if
something bad happens.
osync will use mutt or mail command.
Please make sure you can send a test mail with at least one of the following
commands run by your sync user:
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
$ echo "your test message" | mutt -x -s "This is a test message" your@mail.tld
\end_layout
\begin_layout Plain Layout
$ echo "your test message" | mail -s "This is a test message" your@mail.tld
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Check your antispam if you don't get your message.
If you still don't get your message, check your distributions documentation
about the mail command.
\end_layout
\begin_layout Standard
If you run on windows environment, please make sure you can launch mailsend.exe
/ sendemail.exe by adding it to the %PATH% variable (found
\begin_inset CommandInset href
LatexCommand href
name "here"
target "http://github.com/muquit/mailsend"
\end_inset
and
\begin_inset CommandInset href
LatexCommand href
name "here"
target "http://caspian.dotconf.net/menu/Software/SendEmail/"
\end_inset
).
\end_layout
\begin_layout Subsection
\begin_inset CommandInset label
LatexCommand label
name "subsec:Enhancing-remote-backup"
\end_inset
Enhancing remote system security
\end_layout
\begin_layout Standard
You may want to secure a password-less ssh access by removing non necessary
services offered by SSH.
Edit the file ~/.ssh/authorized_keys created earlier on the remote system
and add the following line in the beginning of the file:
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Also, we may want to prevent any host except of our initiator replica system
to passwordless connect.
Add the following line:
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
from=*.my.initiator.replica.server.domain.tld
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Your authorized_keys file should look like this:
\end_layout
\begin_layout Standard
\begin_inset listings
lstparams "breaklines=true"
inline false
status open
\begin_layout Plain Layout
from="*.mydomain.tld",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-
pty ssh-rsa yourkey== syncuser@host.tld
\end_layout
\end_inset
\end_layout
\begin_layout Subsection
More security enhancing
\end_layout
\begin_layout Standard
We may also restrict the ssh session to only a couple of commands we'll
need.
osync comes with a script called
\emph on
ssh_filter.sh
\emph default
that will only allow execution of commands osync needs.
Once again edit your authorized_keys file and add the following.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
command="/usr/local/bin/ssh_filter.sh"
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Your file should then look like this:
\end_layout
\begin_layout Standard
\begin_inset listings
lstparams "breaklines=true,showstringspaces=false"
inline false
status open
\begin_layout Plain Layout
from="*.mydomain.tld",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-
pty,command="/usr/local/bin/ssh_filter.sh" ssh-rsa yourkey== syncuser@remotesyste
m.tld
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Copy then the script ssh_filter.sh to /usr/local/bin on the remote system.
Don't forget to make it executable and make it owned by root
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
# chmod 755 /usr/local/bin/ssh_filter.sh
\end_layout
\begin_layout Plain Layout
# chown root:root /usr/local/bin/ssh_filter.sh
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Now, only the commands
\begin_inset Quotes eld
\end_inset
find, du, rsync, echo, mv, mkdir and sudo
\begin_inset Quotes erd
\end_inset
may be executed via the ssh tunnel.
You may enable / disable the usage of sudo command by editing the following
value in the ssh_filter.sh script:
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
SUDO_EXEC=yes
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Also, adding remote pre- and postexecution commands in your configuration
files will not work if you use the ssh filter.
You'll have to add your optional commands in ssh_filter.sh.
Example if you want to perform remote snapshots you'll have to allow one
of the following:
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
CMD1=zfs
\end_layout
\begin_layout Plain Layout
CMD2=xfs
\end_layout
\begin_layout Plain Layout
CMD3=lvm
\end_layout
\end_inset
\end_layout
\begin_layout Subsection
\begin_inset CommandInset label
LatexCommand label
name "subsec:More-security-(or"
\end_inset
Security for the paranoid
\end_layout
\begin_layout Standard
Executing rsync as superuser is a security risk.
A way to prevent rsync usage allowing only a symlink to be executed.
Thus, a attacker script using rsync would not work.
This kind of security is called
\begin_inset Quotes eld
\end_inset
security by obscurity
\begin_inset Quotes erd
\end_inset
and should generally not be the only security process, but makes any attack
harder.
First, let's create a symlink to rsync called let's say o_rsync, on both
local and remote systems.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
# ln -s $(type rsync) $(dirname $(type rsync))/o_rsync
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Now edit ssh_filter.sh and change the following value:
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
RSYNC_EXECUTABLE=o_rsync
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Also, edit RSYNC_EXECUTABLE value on any of your sync configuration files
and you're done.
\end_layout
\begin_layout Section
Getting and running osync
\end_layout
\begin_layout Subsection
Downloading osync
\end_layout
\begin_layout Standard
osync can be downloaded on the author's site (stable version) or on github
(stable or latest dev snapshot).
\end_layout
\begin_layout Standard
Getting osync via author's site
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
wget http://netpower.fr/projects/osync/osync.v1.1.tar.gz
\end_layout
\begin_layout Plain Layout
tar xvf osync.v1.1.tar.gz
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Getting osync via github (remove the -b
\begin_inset Quotes sld
\end_inset
v1.00a
\begin_inset Quotes srd
\end_inset
if you want latest dev snapshot)
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
git clone -b "v1.1" https://github.com/deajan/osync
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Once you downloaded osync, enter into the newly created folder and run the
install script
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
sh ./install.sh
\end_layout
\end_inset
\end_layout
\begin_layout Standard
This will copy osync to /usr/local/bin and create /etc/osync with a test
sync.conf file.
\end_layout
\begin_layout Standard
It will also copy daemon required files to /etc/init.d or /usr/lib/systemd/system
and /etc/systemd/user depending on your distribution.
\end_layout
\begin_layout Subsection
\begin_inset CommandInset label
LatexCommand label
name "sec:Running-Osync-in"
\end_inset
Running osync in quicksync mode
\end_layout
\begin_layout Standard
You just osync to sync two local dirs like this:
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
$ ./osync.sh --initiator=/path/to/dir1 --target=/path/to/dir2
\end_layout
\end_inset
\end_layout
\begin_layout Standard
You also may want to sync a remote directory.
\end_layout
\begin_layout Standard
You may specify an alternate SSH port directly in the URI.
When ommited, SSH port 22 is used.
\end_layout
\begin_layout Standard
Also, if not set, the default RSA key will be read from ~/.ssh/id_rsa
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
$ ./osync.sh --initiator=/path/to/dir1 --target=ssh://remoteuser@remotehost.com//pa
th/to/dir2
\end_layout
\begin_layout Plain Layout
$ ./osync.sh --initiator=/path/to/dir2 --target=ssh://remoteuser@remotehost.com:22/
/path/to/dir2 --rsakey=/home/user/.ssh/other_key
\end_layout
\end_inset
\end_layout
\begin_layout Subsection
\begin_inset CommandInset label
LatexCommand label
name "sec:Running-Osync-with-config-file"
\end_inset
Running osync with a full blown configuration file
\end_layout
\begin_layout Standard
Running osync with a configuration will do just the same as in quicksync
mode, except that you have much more control of what's going on.
\end_layout
\begin_layout Standard
A sample configuration file is called sync.conf and is included with osync.
You may edit this file to fit your needs.
Basically configuration files should go to /etc/osync.
\end_layout
\begin_layout Standard
Every option of the configuration file is explained in the appendix.
\end_layout
\begin_layout Standard
Once you've setup a file according to your needs, you may go for a test
run.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
$ ./osync.sh /etc/osync/my_sync.conf --dry --verbose
\end_layout
\end_inset
\end_layout
\begin_layout Standard
osync should enumerate which changes will be done on both sides.
\end_layout
\begin_layout Standard
If everything worked out right, you might process the actual sync process.
\end_layout
\begin_layout Standard
A full configuration file specifies a maximum execution delay.
Initial sync tasks can take a huge amount of time depending on bandwidth
between replicas, in that case you might add parameter –no-maxtime to your
first sync run so execution time won't be enforced.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
$ ./osync.sh /etc/osync/my_sync.conf --no-maxtime
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Creating a regular sync scenario is quite simple as long as you don't schedule
twice the same sync task in a shorter time span than your HARD_MAX_EXEC_TIME_TO
TAL value.
Just create a crontab entry and add parameter –silent so your local mailbox
won't get filled up.
Example, having a sync scheduled every hour in /etc/crontab
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
00 * * * * syncuser /usr/local/bin/osync.sh /etc/osync/your_sync.conf --silent
\end_layout
\end_inset
You may find the sync log under /var/log/osync-your_sync.log or under the
current directory if /var/log is not writable.
\end_layout
\begin_layout Subsection
\begin_inset CommandInset label
LatexCommand label
name "sec:Running-Osync-as-daemon"
\end_inset
Running osync as deamon
\end_layout
\begin_layout Subsubsection
Manually
\end_layout
\begin_layout Standard
osync may also run in file monitor mode.
In this mode, osync checks the initiator replica, and runs a synchronization
as soon as there is file activity on initiator replica.
With this mode, you do not need a schedule anymore.
Be aware that only initiator replica is monitored, and target replica sync
updates only occur when initiator replica modifications happen.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
$ ./osync.sh /etc/osync/my_sync.conf --on-changes
\end_layout
\end_inset
\end_layout
\begin_layout Subsubsection
As a system service
\end_layout
\begin_layout Standard
If you plan to run osync on a regular basis in file monitor mode, you might
consider installing it as a system service.
\end_layout
\begin_layout Standard
From the directory you downloaded osync, run the install.sh script and enable
the service.
\end_layout
\begin_layout Standard
Remark: When exiting osync daemon, the process will continue to run for
up to a minute to unlock replicas and termine sub processes.
\end_layout
\begin_layout Paragraph
init.d service files
\end_layout
\begin_layout Standard
For init.d systems, syntax is:
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
# service osync-srv start
\end_layout
\begin_layout Plain Layout
# chkconfig osync-srv on
\end_layout
\end_inset
\end_layout
\begin_layout Standard
osync then scans for *.conf files in /etc/osync and will run an instance
per configuration file.
\end_layout
\begin_layout Standard
Service control just works like with standard system services.
\end_layout
\begin_layout Paragraph
systemd service files
\end_layout
\begin_layout Standard
For systemd systems, syntax is:
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
# systemctl start osync-srv@config_file
\end_layout
\begin_layout Plain Layout
# systemctl enable osync-srv@config_file
\end_layout
\end_inset
\end_layout
\begin_layout Standard
With systemd, every config file found in /etc/osync can be controlled as
a separate service.
\end_layout
\begin_layout Subsection
Running osync batch
\end_layout
\begin_layout Standard
If you have multiple configuration files in /etc/osync that you would like
to run sequentially, and re-run failed sync tasks, osync comes with a tool
called osync-batch.
\end_layout
\begin_layout Standard
It will execute all osync conf files in alphanumerical order, in a given
timespan.
\end_layout
\begin_layout Standard
osync-batch takes the following non mandatory parameters:
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–silent Will launch osync tasks silently
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–dry Will launch osnyc tasks as simulations only
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–verbose Will launch osync tasks with detailed output, including changed
and deleted files lists on both sides
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–no-maxtime Will launch osync taks without any maximum execution time
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–path= By default, osync-batch.sh searches for config files in /etc/osync.
This parameter overrides the default value.
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–max-reruns= By default osync-batch.sh tries to rerun failed tasks 3 times.
This parameter overrides the default value.
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–max-exec-time= By default, osync-batch.sh won't launch next task if 36000
seconds have passed.
This parameter overrides the default value.
\end_layout
\begin_layout Standard
You may program a cron task for osync-batch.sh like
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
00 * * * * syncuser /usr/local/bin/osync-batch.sh --silent
\end_layout
\end_inset
\end_layout
\begin_layout Section
Configuration appendix
\end_layout
\begin_layout Subsection
Quicksync only command line parameters
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–initiator=
\begin_inset Quotes srd
\end_inset
\begin_inset Quotes srd
\end_inset
Initiator replica path.
Will contain state and backup directory (is mandatory)
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–target=
\begin_inset Quotes srd
\end_inset
\begin_inset Quotes srd
\end_inset
Local or remote target replica path.
Can be a ssh uri like ssh://user@host.com:22//path/to/target/replica (is
mandatory)
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–rsakey Alternative path to rsa private key for ssh connection to target
replica (if not ~/.ssh/id_rsa)
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–instance-id Optional sync task name to identify this synchronization task
when using multiple targets
\end_layout
\begin_layout Subsection
Universal command line parameters
\end_layout
\begin_layout Standard
When run without any parameter, osync will show usage.
\end_layout
\begin_layout Standard
Both quicksync and config file modes can take the following optional parameters:
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–dry Will make osync run a simulation only
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–silent Will run osync silently, to be used in a cron schedule
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–verbose Will run osync with detailed output, including changed and deleted
files lists on both sides
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–stats Will add rsync transfer statistics to verbose output
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–partial osync will leave partial transfered files in order to be resumed
on later runs
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–no-maxtime Will disable MAX_EXEC_TIME checks, so a task can take as long
as it needs.
This is useful for performing initial big sync operations
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–force-unlock Will override any existing active or dead locks on initiator
and target replica
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–on-changes Will launch a sync task after a short wait period if there is
some file activity on initiator replica.
You should try daemon mode instead
\end_layout
\begin_layout Labeling
\labelwidthstring 00.00.0000
–help Will print osync version and usage
\end_layout
\begin_layout Subsection
Full list of configuration file parameters
\end_layout
\begin_layout Standard
Set this to whatever you want to identify your sync task.
This value also determines the log filename and appears in the warning
/ error mails.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
INSTANCE_ID=name_of_your_sync
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Initiator directory to sync (initiator replica), must be on the system you're
running osync on.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
INITIATOR_SYNC_DIR="/some/path"
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Target directory to sync (target replica), can be on the same system you're
running osync on or another remote system, reachable via an SSH tunnel.
\end_layout
\begin_layout Standard
Target directory can be a SSH uri like
\begin_inset Quotes sld
\end_inset
ssh://user@host.com:1234//some/other/path
\begin_inset Quotes srd
\end_inset
where 1234 is an optional port, and the first slash is a separator, meaning
that the full path is /some/other/path.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
TARGET_SYNC_DIR="/some/other/path"
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Location of the private RSA key.
If left empty, the default path
\begin_inset Quotes sld
\end_inset
~/.ssh/id_rsa
\begin_inset Quotes srd
\end_inset
will be used.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
SSH_RSA_PRIVATE_KEY=~/.ssh/id_rsa
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Tells osync to create initiator or target directories if they don't exist.
Default is no.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
CREATE_DIRS=yes|no
\end_layout
\end_inset
\end_layout
\begin_layout Standard
By default, leaving this empty sets the log file to /var/log/osync_INSTANCE_ID.lo
g.
You might change this to specify a personalized log file.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
LOGFILE=""
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Generate an alert if initiator or target replicas have less space than the
following given value in kilobytes.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
MINIMUM_SPACE=10240
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Bandwidth limit in kilobytes / second.
Leave this to zero to disable limitation.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
BANDWIDTH=0
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Synchronization tasks may be executed as root if you enable the following
parameter.
See prerequisites in chapter
\begin_inset CommandInset ref
LatexCommand ref
reference "subsec:Performing-superuser-backups"
\end_inset
.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
SUDO_EXEC=yes|no
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Paranoia option.
Don't change this unless you read chapter
\begin_inset CommandInset ref
LatexCommand ref
reference "subsec:More-security-(or"
\end_inset
and understand what you are doing.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
RSYNC_EXECUTABLE=rsync
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Remote Rsync Executable path.
Don't change this unless your remote rsync binary isn't in the execution
path.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
REMOTE_RSYNC_PATH=""
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Rsync include / exclude order.
If set to include, includes will be processed before excludes, and vice-versa.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
RSYNC_PATTERN_FIRST=include|exclude
\end_layout
\end_inset
\end_layout
\begin_layout Standard
List of files / directories to include / exclude from both replicas (see
rsync patterns for more explanations, wildcards won't work).
\end_layout
\begin_layout Standard
Poaths are relative to both replicas.
List is separated by PATH_SEPARATOR_CHAR defined below.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
RSYNC_INCLUDE_PATTERN=""
\end_layout
\begin_layout Plain Layout
RSYNC_EXCLUDE_PATTERN="tmp;archives;somepath"
\end_layout
\end_inset
\end_layout
\begin_layout Standard
File that contains the list of files /directories to include / exclude from
both replicas (see rsync pattern files for more explanations).
Leave this empty if you don't want to use an exclusion file.
This file has to be in the same directory as the config file.
Paths are relative to sync dirs.
One element per line.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
RSYNC_INCLUDE_FROM=""
\end_layout
\begin_layout Plain Layout
RSYNC_EXCLUDE_FROM="exclude.list"
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Path separator char for RSYNC_EXCLUDE_PATTERN, you might change this in
the unholy case that your filenames contains semicolons.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
PATH_SEPARATOR_CHAR=";"
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Enable / disable ssh compression.
Leave this enabled unless your connection to remote system is high speed
(LAN)
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
SSH_COMPRESSION=yes|no
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Tell ssh to not check the remote computer ssh fingerprint.
DANGER WILL ROBINSON ! This should generally lead to security issues.
Only enable this if you know exactly what you are doing.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
SSH_IGNORE_KNOWN_HOSTS=yes|no
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Ping remote host before launching synchronization.
Be sure the host is responding to ping.
Failing to ping will skip current task.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
REMOTE_HOST_PING=yes|no
\end_layout
\end_inset
\end_layout
\begin_layout Itemize
Check for internet access by pinging one or more hosts before launching
remote sync task.
Leave this empty do disable the check.
Failing to ping will skip current task.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
REMOTE_3RD_PARTY_HOST="www.kernel.org"
\end_layout
\end_inset
\end_layout
\begin_layout Itemize
Misc settings
\end_layout
\begin_layout Standard
Preserve ACLs.
Please check that your filesystem supports ACLs and is mounted with it's
support or rsync will get you loads of errors.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
PRESERVE_ACL=yes|no
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Preserve Xattr.
The same applies as for ACLs
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
PRESERVE_XATTR=yes|no
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Transforms symlinks into referent files/dirs when syncing replicas.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
COPY_SYMLINKS=yes|no
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Treat symlinked dirs as dirs.
CAUTION: This also follows symlinks outside of the replica root.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
KEEP_DIRLINKS=no
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Preserve hard links.
Make sure source and target FS can manage hard links or you will lose them.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
PRESERVE_HARDLINKS=yes|no
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Do a full checksum on files instead of comparing file sizes and modification
times.
Enabling this will make sync tasks longer.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
CHECKSUM=yes|no
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Use rsync compression for file transfers.
Leave this disabled unless your're not using SSH compression.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
RSYNC_COMPRESS=yes|no
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Maximum execution time (in seconds) for sync process.
Soft value generates a warning only.
Hard value generates a warning and stops sync task.
\end_layout
\begin_layout Standard
You may set this to 0 to disable time checks.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
SOFT_MAX_EXEC_TIME_FILE_TASK=7200
\end_layout
\begin_layout Plain Layout
HARD_MAX_EXEC_TIME_FILE_TASK=10600
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Minimum time (in seconds) in file monitor /daemon mode between modification
detection and sync task in order to let copy operations finish.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
MIN_WAIT=60
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Maximum time (in seconds) in file monitor / daemon mode.
After this amount of time, a sync operation is forced.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
MAX_WAIT=300
\end_layout
\end_inset
\end_layout
\begin_layout Itemize
Conflict and deletion option
\end_layout
\begin_layout Standard
Enabling this option will keep a backup of a file on the target replica
if it gets updated from the source replica.
Backups will be made to .osync_workdir/backups
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
CONFLICT_BACKUP=yes|no
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Keep multiple backup versions of the same file.
Warning, This can be very space consuming.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
CONFLICT_BACKUP_MULTIPLE=yes|no
\end_layout
\end_inset
\end_layout
\begin_layout Standard
osync will clean backup files after a given number of days.
Setting this to 0 will disable cleaning and keep backups forever.
Warning: This can be very space consuming.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
CONFLICT_BACKUP_DAYS=30
\end_layout
\end_inset
\end_layout
\begin_layout Standard
If the same file exists on both replicas, newer version will be synced.
However, if both files have the same timestamp but differ, CONFLICT_PREVALANCE
sets winner replica.
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
CONFLICT_PREVALANCE=initiator|target
\end_layout
\end_inset
\end_layout
\begin_layout Standard
On deletition propagation to the target replica, a backup of the deleted
files can be kept.
Deletions will be kept in .osync_workdir/deleted
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
SOFT_DELETE=yes|no
\end_layout
\end_inset
\end_layout
\begin_layout Standard
osync will clean deleted files after a given number of days.
Setting this to 0 will disable cleaning and keep deleted files forever.
Warning: This can be very space consuming.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
SOFT_DELETE_DAYS=30
\end_layout
\end_inset
\end_layout
\begin_layout Itemize
Resuming options
\end_layout
\begin_layout Standard
Try to resume an aborted sync task
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
RESUME_SYNC=yes|no
\end_layout
\end_inset
Number maximum resume tries before initating a fresh sync.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
RESUME_TRY=2
\end_layout
\end_inset
\end_layout
\begin_layout Standard
When a pidlock exists on target replica that does not correspond to initiator's
instance-id, force pidlock removal.
Be carefull with this option if you have multiple initiators.
\end_layout
\begin_layout Standard
\begin_inset listings
lstparams "breaklines=true"
inline false
status open
\begin_layout Plain Layout
FORCE_STRANGER_LOCK_RESUME=no
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Keep partial uploads that can be resumed on next run.
This can be very useful if big files must get updated though slow links.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
PARTIAL=no
\end_layout
\end_inset
\end_layout
\begin_layout Itemize
Alert Options
\end_layout
\begin_layout Standard
List of alert mails separated by spaces
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
DESTINATION_MAILS="your@alert.tld"
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Windows (MSYS / Cygwin environment) only mail options (used with mailsend.exe
from muquit or sendemail.exe from Brandon Zehm)
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
SENDER_MAIL="alert@your.system.tld"
\end_layout
\begin_layout Plain Layout
SMTP_SERVER=smtp.your.isp.tld
\end_layout
\begin_layout Plain Layout
SMTP_PORT=25
\end_layout
\begin_layout Plain Layout
SMTP_ENCRYPTION=tls|ssl|none
\end_layout
\begin_layout Plain Layout
SMTP_USER=optional_smtp_user
\end_layout
\begin_layout Plain Layout
SMTP_PASSWORD=optional_smtp_password
\end_layout
\end_inset
\end_layout
\begin_layout Itemize
Execution hooks
\end_layout
\begin_layout Standard
Commands can will be run before and / or after sync process (remote execution
will only happen if REMOTE_SYNC is set).
Multiple commands can be semicolon separated.
\end_layout
\begin_layout Standard
Command(s) to run locally before sync process starts.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
LOCAL_RUN_BEFORE_CMD=""
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Command(s) to run locally if sync process finishes.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
LOCAL_RUN_AFTER_CMD=""
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Command(s) to run on remote system before sync process starts.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
REMOTE_RUN_BEFORE_CMD=""
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Command(s) to run on remote system if sync process finishes.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
REMOTE_RUN_AFTER_CMD=""
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Max execution time of commands before they get force killed.
Leave 0 if you don't wan't this to happen.
Time is specified in seconds.
MAX_EXEC_TIME_PER_CMD_BEFORE=0
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
MAX_EXEC_TIME_PER_CMD_AFTER=0
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Stops osync execution if one of the above commands fail
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
STOP_ON_CMD_ERROR=yes|no
\end_layout
\end_inset
\end_layout
\begin_layout Standard
Run local and remote commands after a sync task even if it failed.
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
RUN_AFTER_CMD_ON_ERROR=yes|no
\end_layout
\end_inset
\end_layout
\begin_layout Section
Troubleshooting
\end_layout
\begin_layout Standard
osync has been tested successfully on multiple systems for a wide variety
of sync plans.
Please check the following steps before requesting help.
\end_layout
\begin_layout Subsection
Local-local sync
\end_layout
\begin_layout Standard
osync logs every of it's actions to /var/log/osync-version.instance_id.log
(or current directory if /var/log is not writable).
\end_layout
\begin_layout Standard
Please check the log file if something went wrong.
\end_layout
\begin_layout Standard
You might try running osync as root to check if your problem is filesystem
permission related.
\end_layout
\begin_layout Standard
You might add –verbose option to see what actually happens.
\end_layout
\begin_layout Standard
Also, running osync with the following command will give the exact commands
that actually happen:
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
DEBUG=yes /usr/local/bin/osync.sh /etc/osync/my_sync.conf --verbose
\end_layout
\end_inset
\end_layout
\begin_layout Subsection
Local-remote sync
\end_layout
\begin_layout Standard
Remote synchronization is a bit more tricky.
\end_layout
\begin_layout Standard
You might check that you can log in remotely with the command
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
$ ssh -p 22 remotesyncuser@remotehost.tld
\end_layout
\end_inset
Also, you might check that you can use rsync command remotely
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
$ ssh -p 22 remotesyncuser@remotehost.tld rsync --help
\end_layout
\end_inset
You can temporarily disable ssh security by removing lines you added in
chapter
\begin_inset CommandInset ref
LatexCommand ref
reference "subsec:Enhancing-remote-backup"
\end_inset
.
Additionnaly, you can check ssh_filter log in ~/.ssh/ssh_filter.log on the
remote system.
You might try running osync with SUDO_EXEC to check if your problem is
user permission related.
\end_layout
\begin_layout Subsection
File monitor mode
\end_layout
\begin_layout Standard
In file monitor mode, osync will still log it's execution to /var/log/osync.insta
nce_id.log (or current directory if /var/log is not writable).
\end_layout
\begin_layout Standard
Also, standard systemd log method is used if available.
You may check an execution with
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
systemctl status osync-srv@configfile
\end_layout
\end_inset
\end_layout
\begin_layout Standard
You mai also see osync logs in journalctl with
\end_layout
\begin_layout Standard
\begin_inset listings
inline false
status open
\begin_layout Plain Layout
journalctl -xn
\end_layout
\end_inset
\end_layout
\begin_layout Section
Final words
\end_layout
\begin_layout Standard
The idea of osync came in a discussion around a beer one evening.
It began as a project for a friend, whose company I was working for as
a consultant.
\end_layout
\begin_layout Standard
Today, osync is still used by this company, and a lot of others around the
globe.
\end_layout
\begin_layout Standard
I do provide technical help and support in my spare time, and will appreciate
every contribution i get on Github :)
\end_layout
\begin_layout Standard
\begin_inset CommandInset line
LatexCommand rule
offset "0.5ex"
width "100col%"
height "1pt"
\end_inset
\end_layout
\end_body
\end_document
Reference in New Issue View Git Blame Copy Permalink