2
0
mirror of https://github.com/deajan/osync synced 2024-11-11 07:10:40 +00:00

Added systemd file, installer stats and ssh known host ignore option

This commit is contained in:
deajan 2016-03-29 22:55:57 +02:00
parent 8e74fd2bff
commit fb53664877
12 changed files with 144 additions and 49 deletions

View File

@ -14,7 +14,12 @@ KNOWN ISSUES
RECENT CHANGES
--------------
! XX Mar 2016: osync v1.1 released
! XX Apr 2016: osync v1.1 released
! Updated documentation (+README.md file)
- Redirected ERROR and WARN messages to stderr to systemd catches them into it's journal
!- Added a systemd unit file (needs testing in file monitor mode yet)
- Added an option to ignore ssh known hosts (use with caution, can lead to security risks), also updated upgrade script accordingly
- Added optional installation statistics
- Fixed a nasty bug with log writing and tree_list function
- Improved mail fallback
- Improved more logging

View File

@ -1,8 +1,8 @@
osync
=====
A two way filesync script with fault tolerance, resuming, deletion backup and conflict backups running on linux and virtually any system supporting bash.
File synchronization is bidirectional, based on rsync, and can be run manually, by cron, or triggered whenever a file changes on master.
A two way filesync script with fault tolerance, resume, soft deletion, conflictual file backups running on bash (linux, BSD and virtually any system supporting bash).
File synchronization is bidirectional, based on rsync, can be run manually, as scheduled task, or triggered on file changes.
## About
@ -22,12 +22,14 @@ Osync uses a master / slave sync schema. It can sync local to local or local to
Osync uses pidlocks to prevent multiple concurrent sync processes on/to the same master / slave replica. Be sure a sync process is finished before launching next one, or use osync-batch.
You may launch concurrent sync processes on the same system but only for different master replicas.
Currently, it has been tested on CentOS 5.x, 6.x, 7.x, Debian 6.0.7, Linux Mint 14, 15 and 16, Ubuntu 12.04 and 12.10, FreeBSD 8.3 and 10.1.
Some users report osync to work on MacOS X too. Microsoft Windows is supported via MSYS environment.
Currently, it has been tested on CentOS 5.x, 6.x, 7.x, Debian 6.0.7, Linux Mint 14-17, Ubuntu 12.04 and 12.10, FreeBSD 8.3 and 10.1.
Microsoft Windows is supported via MSYS or Cygwin.
Some users report osync to work on MacOS X too.
## Installation
Please note that development of version 1.1 has begun. Stable release is v1.01.
Stable release is v1.1 and will be the last of v1 series.
Osync has been designed to not delete any data, but rather make backups of conflictual files or soft deletes.
Nevertheless, you should always have a neat backup of your data before trying a new sync tool.
@ -36,7 +38,7 @@ You can download the latest stable release of Osync at www.netpower.fr/osync or
You may also get the last development version at https://github.com/deajan/osync with the following command
$ git clone -b "v1.01" https://github.com/deajan/osync
$ git clone -b "v1.1" https://github.com/deajan/osync
$ sh install.sh
Osync will install itself to /usr/local/bin and an example configuration file will be installed to /etc/osync
@ -49,6 +51,17 @@ If bash is not your default shell, you may invoke it using
On *BSD, be sure to have bash installed.
On MSYS, On top of your basic install, you need msys-rsync and msys-coreutils-ext packages.
## Upgrade from v1.0x
Since osync v1.1 the config file format has changed in semantics and adds new config options.
Also, master is now called initiator and slave is now called target.
You can upgrade all v1.0x config files by running the upgrade script
$ ./upgrade-v1.0x-v1.1x.sh /etc/osync/your-config-file.conf
The script will backup your config file, update it's content and try to connect to master and remote replicas to update the state dir.
## Usage
Osync can work with in three flavors: Quick sync mode, configuration file mode, and daemon mode.
@ -114,6 +127,11 @@ You may run the install.sh script which should work in most cases or copy the fi
$ service osync-srv start
$ chkconfig osync-srv on
Systemd specific
$ systemctl start osync-srv@configfile
$ systemctl enable osync-srv@configfile
Troubleshooting
---------------
You may find osync's logs in /var/log/osync-*.log (or current directory if /var/log is not writable).

View File

@ -5,13 +5,13 @@
#TODO: explain why osync daemon process can still exist after quit for 30s
PROGRAM="osync" # Rsync based two way sync engine with fault tolerance
AUTHOR="(L) 2013-2016 by Orsiris de Jong"
AUTHOR="(C) 2013-2016 by Orsiris de Jong"
CONTACT="http://www.netpower.fr/osync - ozy@netpower.fr"
PROGRAM_VERSION=1.1-dev
PROGRAM_BUILD=2016031002
IS_STABLE=no
## FUNC_BUILD=2016032201
## FUNC_BUILD=2016032901
## BEGIN Generic functions for osync & obackup written in 2013-2016 by Orsiris de Jong - http://www.netpower.fr - ozy@netpower.fr
## type -p does not work on platforms other than linux (bash). If if does not work, always assume output is not a zero exitcode
@ -95,11 +95,15 @@ function Dummy {
}
function _Logger {
local svalue="${1}" # What to log to screen
local svalue="${1}" # What to log to stdout
local lvalue="${2:-$svalue}" # What to log to logfile, defaults to screen value
local evalue="${3}" # What to log to stderr
echo -e "$lvalue" >> "$LOG_FILE"
if [ $_SILENT -eq 0 ]; then
# <OSYNC SPECIFIC> Special case in daemon mode where systemctl doesn't need double timestamps
if [ "$sync_on_changes" == "1" ]; then
cat <<< "$evalue" 1>&2 # Log to stderr in daemon mode
elif [ $_SILENT -eq 0 ]; then
echo -e "$svalue"
fi
}
@ -117,15 +121,15 @@ function Logger {
# </OSYNC SPECIFIC>
if [ "$level" == "CRITICAL" ]; then
_Logger "$prefix\e[41m$value\e[0m" "$prefix$level:$value"
_Logger "$prefix\e[41m$value\e[0m" "$prefix$level:$value" "$level:$value"
ERROR_ALERT=1
return
elif [ "$level" == "ERROR" ]; then
_Logger "$prefix\e[91m$value\e[0m" "$prefix$level:$value"
_Logger "$prefix\e[91m$value\e[0m" "$prefix$level:$value" "$level:$value"
ERROR_ALERT=1
return
elif [ "$level" == "WARN" ]; then
_Logger "$prefix\e[93m$value\e[0m" "$prefix$level:$value"
_Logger "$prefix\e[93m$value\e[0m" "$prefix$level:$value" "$level:$value"
WARN_ALERT=1
return
elif [ "$level" == "NOTICE" ]; then
@ -276,7 +280,7 @@ function SendAlert {
# pfSense specific
if [ -f /usr/local/bin/mail.php ]; then
cmd="echo \"$MAIL_ALERT_MSG\" | /usr/local/bin/mail.php subject=\"$subject\""
cmd="echo \"$MAIL_ALERT_MSG\" | /usr/local/bin/mail.php -s=\"$subject\""
Logger "Mail cmd: $cmd" "DEBUG"
eval $cmd
if [ $? != 0 ]; then
@ -875,6 +879,11 @@ function PreInit {
SSH_COMP=
fi
## Ignore SSH known host verification
if [ "$SSH_IGNORE_KNOWN_HOSTS" == "yes" ]; then
SSH_OPTS="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"
fi
## Support for older config files without RSYNC_EXECUTABLE option
if [ "$RSYNC_EXECUTABLE" == "" ]; then
RSYNC_EXECUTABLE=rsync
@ -976,7 +985,7 @@ function PostInit {
__CheckArguments 0 $# ${FUNCNAME[0]} "$@" #__WITH_PARANOIA_DEBUG
# Define remote commands
SSH_CMD="$(type -p ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY $REMOTE_USER@$REMOTE_HOST -p $REMOTE_PORT"
SSH_CMD="$(type -p ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY $SSH_IGNORE_KNOWN_HOSTS $REMOTE_USER@$REMOTE_HOST -p $REMOTE_PORT"
SCP_CMD="$(type -p scp) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY -P $REMOTE_PORT"
RSYNC_SSH_CMD="$(type -p ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY -p $REMOTE_PORT"
}

View File

@ -49,5 +49,5 @@ Unexpand
Merge
CleanDebug
rm -f tmp_$PROGRAM.sh
rm -f tmp_install.sh
rm -f ../tmp_install.sh
CopyCommons

View File

@ -5,7 +5,7 @@
#TODO: explain why osync daemon process can still exist after quit for 30s
PROGRAM="osync" # Rsync based two way sync engine with fault tolerance
AUTHOR="(L) 2013-2016 by Orsiris de Jong"
AUTHOR="(C) 2013-2016 by Orsiris de Jong"
CONTACT="http://www.netpower.fr/osync - ozy@netpower.fr"
PROGRAM_VERSION=1.1-dev
PROGRAM_BUILD=2016031002

View File

@ -1,4 +1,4 @@
## FUNC_BUILD=2016032201
## FUNC_BUILD=2016032901
## BEGIN Generic functions for osync & obackup written in 2013-2016 by Orsiris de Jong - http://www.netpower.fr - ozy@netpower.fr
## type -p does not work on platforms other than linux (bash). If if does not work, always assume output is not a zero exitcode
@ -82,11 +82,15 @@ function Dummy {
}
function _Logger {
local svalue="${1}" # What to log to screen
local svalue="${1}" # What to log to stdout
local lvalue="${2:-$svalue}" # What to log to logfile, defaults to screen value
local evalue="${3}" # What to log to stderr
echo -e "$lvalue" >> "$LOG_FILE"
if [ $_SILENT -eq 0 ]; then
# <OSYNC SPECIFIC> Special case in daemon mode where systemctl doesn't need double timestamps
if [ "$sync_on_changes" == "1" ]; then
cat <<< "$evalue" 1>&2 # Log to stderr in daemon mode
elif [ $_SILENT -eq 0 ]; then
echo -e "$svalue"
fi
}
@ -104,15 +108,15 @@ function Logger {
# </OSYNC SPECIFIC>
if [ "$level" == "CRITICAL" ]; then
_Logger "$prefix\e[41m$value\e[0m" "$prefix$level:$value"
_Logger "$prefix\e[41m$value\e[0m" "$prefix$level:$value" "$level:$value"
ERROR_ALERT=1
return
elif [ "$level" == "ERROR" ]; then
_Logger "$prefix\e[91m$value\e[0m" "$prefix$level:$value"
_Logger "$prefix\e[91m$value\e[0m" "$prefix$level:$value" "$level:$value"
ERROR_ALERT=1
return
elif [ "$level" == "WARN" ]; then
_Logger "$prefix\e[93m$value\e[0m" "$prefix$level:$value"
_Logger "$prefix\e[93m$value\e[0m" "$prefix$level:$value" "$level:$value"
WARN_ALERT=1
return
elif [ "$level" == "NOTICE" ]; then
@ -263,7 +267,7 @@ function SendAlert {
# pfSense specific
if [ -f /usr/local/bin/mail.php ]; then
cmd="echo \"$MAIL_ALERT_MSG\" | /usr/local/bin/mail.php subject=\"$subject\""
cmd="echo \"$MAIL_ALERT_MSG\" | /usr/local/bin/mail.php -s=\"$subject\""
Logger "Mail cmd: $cmd" "DEBUG"
eval $cmd
if [ $? != 0 ]; then
@ -862,6 +866,11 @@ function PreInit {
SSH_COMP=
fi
## Ignore SSH known host verification
if [ "$SSH_IGNORE_KNOWN_HOSTS" == "yes" ]; then
SSH_OPTS="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"
fi
## Support for older config files without RSYNC_EXECUTABLE option
if [ "$RSYNC_EXECUTABLE" == "" ]; then
RSYNC_EXECUTABLE=rsync
@ -963,7 +972,7 @@ function PostInit {
__CheckArguments 0 $# ${FUNCNAME[0]} "$@" #__WITH_PARANOIA_DEBUG
# Define remote commands
SSH_CMD="$(type -p ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY $REMOTE_USER@$REMOTE_HOST -p $REMOTE_PORT"
SSH_CMD="$(type -p ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY $SSH_IGNORE_KNOWN_HOSTS $REMOTE_USER@$REMOTE_HOST -p $REMOTE_PORT"
SCP_CMD="$(type -p scp) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY -P $REMOTE_PORT"
RSYNC_SSH_CMD="$(type -p ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY -p $REMOTE_PORT"
}

View File

@ -4,7 +4,7 @@ PROGRAM=osync
PROGRAM_VERSION=1.1-dev
PROGRAM_BINARY=$PROGRAM".sh"
PROGRAM_BATCH=$PROGRAM"-batch.sh"
SCRIPT_BUILD=2016032502
SCRIPT_BUILD=2016032903
## osync / obackup daemon install script
## Tested on RHEL / CentOS 6 & 7, Fedora 23, Debian 7 & 8, Mint 17 and FreeBSD 8 & 10
@ -12,7 +12,10 @@ SCRIPT_BUILD=2016032502
CONF_DIR=/etc/$PROGRAM
BIN_DIR=/usr/local/bin
SERVICE_DIR=/etc/init.d
SERVICE_DIR_INIT=/etc/init.d
SERVICE_FILE_INIT="osync-srv"
SERVICE_DIR_SYSTEMD=/usr/lib/systemd/system
SERVICE_FILE_SYSTEMD="osync-srv@.service"
USER=root
@ -38,6 +41,17 @@ if [ "$(whoami)" != "$USER" ]; then
exit 1
fi
if [ -f /sbin/init ]; then
if file /sbin/init | grep systemd > /dev/null; then
init=systemd
else
init=init
fi
else
echo "Can't detect init system."
exit 1
fi
if [ ! -d "$CONF_DIR" ]; then
mkdir "$CONF_DIR"
if [ $? == 0 ]; then
@ -95,13 +109,26 @@ if [ -f "./ssh_filter.sh" ]; then
fi
fi
if [ -f "./osync-srv" ]; then
cp "./osync-srv" "$SERVICE_DIR"
if ([ -f "./$SERVICE_FILE_INIT" ] || [ -f "./$SERVICE_FILE_SYSTEMD" ] ); then
if [ "$init" == "systemd" ]; then
cp "./$SERVICE_FILE_SYSTEMD" "$SERVICE_DIR_SYSTEMD"
if [ $? != 0 ]; then
echo "Cannot copy osync-srv to [$SERVICE_DIR]."
echo "Cannot copy the systemd file to [$SERVICE_DIR_SYSTEMD]."
else
chmod 755 "$SERVICE_DIR/osync-srv"
echo "Created osync-srv service in [$SERVICE_DIR]."
echo "Created osync-srv service in [$SERVICE_DIR_SYSTEMD]."
echo "Activate with [systemctl start osync-srv@instance.conf] where instance.conf is the name of the config file in /etc/osync."
echo "Enable on boot with [systemctl enable osync-srv@instance.conf]."
fi
elif [ "$init" == "init" ]; then
cp "./$SERVICE_FILE_INIT" "$SERVICE_DIR_INIT"
if [ $? != 0 ]; then
echo "Cannot copy osync-srv to [$SERVICE_DIR_INIT]."
else
chmod 755 "$SERVICE_DIR_INIT/$SERVICE_FILE_INIT"
echo "Created osync-srv service in [$SERVICE_DIR_INIT]."
echo "Activate with [service $SERVICE_FILE_INIT start]."
echo "Enable on boot with [chkconfig $SERVICE_FILE_INIT on]."
fi
fi
fi

View File

@ -1,7 +1,7 @@
#!/usr/bin/env bash
SUBPROGRAM=osync
PROGRAM="$SUBPROGRAM-batch" # Batch program to run osync / obackup instances sequentially and rerun failed ones
AUTHOR="(L) 2013-2016 by Orsiris de Jong"
AUTHOR="(C) 2013-2016 by Orsiris de Jong"
CONTACT="http://www.netpower.fr - ozy@netpower.fr"
PROGRAM_BUILD=2016031301

12
osync-srv@.service Normal file
View File

@ -0,0 +1,12 @@
[Unit]
Description=A robust two way (bidirectional) file sync script based on rsync with fault tolerance
After=time-sync.target local-fs.target network-online.target
Requires=time-sync.target local-fs.target
Wants=network-online.target
[Service]
Type=simple
ExecStart=/usr/local/bin/osync.sh /etc/osync/%i --on-changes --silent
[Install]
WantedBy=multi-user.target

View File

@ -5,13 +5,13 @@
#TODO: explain why osync daemon process can still exist after quit for 30s
PROGRAM="osync" # Rsync based two way sync engine with fault tolerance
AUTHOR="(L) 2013-2016 by Orsiris de Jong"
AUTHOR="(C) 2013-2016 by Orsiris de Jong"
CONTACT="http://www.netpower.fr/osync - ozy@netpower.fr"
PROGRAM_VERSION=1.1-dev
PROGRAM_BUILD=2016031002
IS_STABLE=no
## FUNC_BUILD=2016032201
## FUNC_BUILD=2016032901
## BEGIN Generic functions for osync & obackup written in 2013-2016 by Orsiris de Jong - http://www.netpower.fr - ozy@netpower.fr
## type -p does not work on platforms other than linux (bash). If if does not work, always assume output is not a zero exitcode
@ -90,11 +90,15 @@ function Dummy {
}
function _Logger {
local svalue="${1}" # What to log to screen
local svalue="${1}" # What to log to stdout
local lvalue="${2:-$svalue}" # What to log to logfile, defaults to screen value
local evalue="${3}" # What to log to stderr
echo -e "$lvalue" >> "$LOG_FILE"
if [ $_SILENT -eq 0 ]; then
# <OSYNC SPECIFIC> Special case in daemon mode where systemctl doesn't need double timestamps
if [ "$sync_on_changes" == "1" ]; then
cat <<< "$evalue" 1>&2 # Log to stderr in daemon mode
elif [ $_SILENT -eq 0 ]; then
echo -e "$svalue"
fi
}
@ -112,15 +116,15 @@ function Logger {
# </OSYNC SPECIFIC>
if [ "$level" == "CRITICAL" ]; then
_Logger "$prefix\e[41m$value\e[0m" "$prefix$level:$value"
_Logger "$prefix\e[41m$value\e[0m" "$prefix$level:$value" "$level:$value"
ERROR_ALERT=1
return
elif [ "$level" == "ERROR" ]; then
_Logger "$prefix\e[91m$value\e[0m" "$prefix$level:$value"
_Logger "$prefix\e[91m$value\e[0m" "$prefix$level:$value" "$level:$value"
ERROR_ALERT=1
return
elif [ "$level" == "WARN" ]; then
_Logger "$prefix\e[93m$value\e[0m" "$prefix$level:$value"
_Logger "$prefix\e[93m$value\e[0m" "$prefix$level:$value" "$level:$value"
WARN_ALERT=1
return
elif [ "$level" == "NOTICE" ]; then
@ -265,7 +269,7 @@ function SendAlert {
# pfSense specific
if [ -f /usr/local/bin/mail.php ]; then
cmd="echo \"$MAIL_ALERT_MSG\" | /usr/local/bin/mail.php subject=\"$subject\""
cmd="echo \"$MAIL_ALERT_MSG\" | /usr/local/bin/mail.php -s=\"$subject\""
Logger "Mail cmd: $cmd" "DEBUG"
eval $cmd
if [ $? != 0 ]; then
@ -806,6 +810,11 @@ function PreInit {
SSH_COMP=
fi
## Ignore SSH known host verification
if [ "$SSH_IGNORE_KNOWN_HOSTS" == "yes" ]; then
SSH_OPTS="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"
fi
## Support for older config files without RSYNC_EXECUTABLE option
if [ "$RSYNC_EXECUTABLE" == "" ]; then
RSYNC_EXECUTABLE=rsync
@ -906,7 +915,7 @@ function PreInit {
function PostInit {
# Define remote commands
SSH_CMD="$(type -p ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY $REMOTE_USER@$REMOTE_HOST -p $REMOTE_PORT"
SSH_CMD="$(type -p ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY $SSH_IGNORE_KNOWN_HOSTS $REMOTE_USER@$REMOTE_HOST -p $REMOTE_PORT"
SCP_CMD="$(type -p scp) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY -P $REMOTE_PORT"
RSYNC_SSH_CMD="$(type -p ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY -p $REMOTE_PORT"
}

View File

@ -1,8 +1,8 @@
#!/usr/bin/env bash
###### Osync - Rsync based two way sync engine with fault tolerance
###### (L) 2013-2015 by Orsiris "Ozy" de Jong (www.netpower.fr)
###### Config file rev 2016012701
###### (C) 2013-2015 by Orsiris "Ozy" de Jong (www.netpower.fr)
###### Config file rev 2016032901
## ---------- GENERAL OPTIONS
@ -63,6 +63,9 @@ PATH_SEPARATOR_CHAR=";"
## ssh compression should be used unless your remote connection is good enough (LAN)
SSH_COMPRESSION=yes
## Ignore ssh known hosts. DANGER WILL ROBINSON DANGER ! This can lead to security issues. Only enable this if you know what you're doing.
SSH_IGNORE_KNOWN_HOSTS=no
## Check for connectivity to remote host before launching remote sync task. Be sure the hosts responds to ping. Failing to ping will stop sync.
REMOTE_HOST_PING=no

View File

@ -2,11 +2,11 @@
PROGRAM="osync instance upgrade script"
SUBPROGRAM="osync"
AUTHOR="(L) 2015 by Orsiris \"Ozy\" de Jong"
AUTHOR="(C) 2015 by Orsiris \"Ozy\" de Jong"
CONTACT="http://www.netpower.fr/osync - ozy@netpower.fr"
OLD_PROGRAM_VERSION="1.0x"
NEW_PROGRAM_VERSION="v1.1x"
PROGRAM_BUILD=2016021101
PROGRAM_BUILD=2016032901
function Init {
OSYNC_DIR=".osync_workdir"
@ -333,6 +333,9 @@ function RewriteConfigFiles {
sed -i '/^LOGFILE=*/a RSYNC_PATTERN_FIRST=include' "$config_file"
fi
if ! grep "^SSH_IGNORE_KNOWN_HOSTS=" "$config_file" > /dev/null; then
sed -i '/^SSH_COMPRESSION=*/a SSH_IGNORE_KNOWN_HOSTS=no' "$config_file"
if ! grep "^RSYNC_INCLUDE_PATTERN=" "$config_file" > /dev/null; then
sed -i '/^RSYNC_EXCLUDE_PATTERN=*/a RSYNC_INCLUDE_PATTERN=""' "$config_file"
fi