Archives/osync
2
0
Fork 0
mirror of https://github.com/deajan/osync synced 2024-11-19 03:25:37 +00:00
Code Issues Releases Wiki Activity

Initial uplaod of function merge

Browse Source
This commit is contained in:
deajan 2015-11-12 01:37:43 +01:00
parent 04e056d720
commit 956f16ea9c
10 changed files with 5457 additions and 686 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
CHANGELOG.md
View File

@ -12,7 +12,8 @@ KNOWN ISSUES
RECENT CHANGES RECENT CHANGES
-------------- --------------
! Check for deletion failed list file before trying to copy it back (normally empty file is created anyway) ! XX Dec 2015: osync v1.1 released
- Fully merged codebase with obackup
- Passed shellCheck.net - Passed shellCheck.net
- Simplified EscapeSpaces to simple bash substitution - Simplified EscapeSpaces to simple bash substitution
- Corrected a lot of minor warnings in order to make code more bullet proof - Corrected a lot of minor warnings in order to make code more bullet proof
@ -23,6 +24,7 @@ RECENT CHANGES
- Unlocking happens after TrapQuit has successfully killed any child processes - Unlocking happens after TrapQuit has successfully killed any child processes
- Replace child_pid by $? directly, add a better sub process killer in TrapQuit - Replace child_pid by $? directly, add a better sub process killer in TrapQuit
- Refactor [local master, local slave, remote slave] code to [local, remote][initiator, target]code - Refactor [local master, local slave, remote slave] code to [local, remote][initiator, target]code
- Renamed a lot of code in order to prepare v2 code (master becomes initiator, slave becomes target, sync_id becomes instance_id)
- Added some automatic checks in code, for _DEBUG mode (and _PARANOIA_DEBUG now) - Added some automatic checks in code, for _DEBUG mode (and _PARANOIA_DEBUG now)
- Improved Logging - Improved Logging
- Updated osync to be fully compliant with coding style - Updated osync to be fully compliant with coding style

60
CODING_STYLE.TXT
View File

@ -1,8 +1,9 @@
Coding style used for my bash projects (v2 Sep 2015) Coding style used for my bash projects (v2.1 Oct 2015)
++++ Header ++++++ Header
Always use the following header Always use the following header
----BEGIN HEADER ----BEGIN HEADER
#!/usr/bin/env bash #!/usr/bin/env bash
@ -17,31 +18,38 @@ PROGRAM_BUILD=YYYYMMDDVV
Using bind style versionning: Using bind style versionning:
YYYYMMDDVV (Year, Month, Day, Revision): Example: 2015012402 = 2nd revision of 24 Jan 2015 YYYYMMDDVV (Year, Month, Day, Revision): Example: 2015012402 = 2nd revision of 24 Jan 2015
#!/usr/bin/env bash instead of #!/bin/bash
++++ Indentation Change old scripts with
for i in $(grep -r '#!/bin/bash' * |cut -f1 -d':'); do sed -i 's&#!/bin/bash&#!/usr/bin/env bash&g' $i; done
type instead of type -p for bash test (other shells don't know -p)
++++++ Indentation
Using tabs Using tabs
Transform old shell scripts using unexpand command Transform old shell scripts using unexpand command
++++ Comments ++++++ Comments
Some command # comment Some command # comment
## Some comment on a new line ## Some comment on a new line
################################################# Some separation ################################################# Some separation
++++ Work comments ++++++ Work comments
Whenever there is some idea to postpone, use #TODO:dev-name: some remark Whenever there is some idea to postpone, use #TODO[-version]:[dev-name:] some remark
A marker must be left where on the line a dev is working (when the work isn't finished). Marker is #WIP:dev-name: some remark A marker must be left where on the line a dev is working (when the work isn't finished). Marker is #WIP:dev-name: some remark
dev-name is mandatory if more than one person is coding dev-name is mandatory if more than one person is coding
Example: #TODO-v2.1:deajan: need to do something
++++ Variables ++++++ Variables
All local variables are lowercase, separated by _ (ex: low_wait) All local variables are lowercase, separated by _ (ex: low_wait)
All global variables full upercase, separated by _ (ex: EXEC_TIME) All global variables full upercase, separated by _ (ex: EXEC_TIME)
All environment variables (verbose, silent, debug, etc) have prefix _ and are full upercase, separated by _ (ex: _PARANOIA_DEBUG) All environment variables (verbose, silent, debug, etc) have prefix _ and are full upercase, separated by _ (ex: _PARANOIA_DEBUG)
++++ Functions ++++++ Functions
Every word in a function begins with an uppercase (ex: SomeFunctionDoesThings) Every word in a function begins with an uppercase (ex: SomeFunctionDoesThings)
@ -64,11 +72,11 @@ function thirdthing {
return $? return $?
} }
++++ Sub functions ++++++ Sub functions
When a function is a subroutine of another function, it is called _SomethingAsSubFunction When a function is a subroutine of another function, it is called _SomethingAsSubFunction
++++ Function argument check ++++++ Function argument check
Bash does not provide any checks against missing function arguments. Also, missing quotes can lead to an inconsistent number of arguments. Bash does not provide any checks against missing function arguments. Also, missing quotes can lead to an inconsistent number of arguments.
Every function call will be checked by __CheckArguments which takes the number of arguments, $# (the real number of args given), the parent function name and the parent function's arguments. Every function call will be checked by __CheckArguments which takes the number of arguments, $# (the real number of args given), the parent function name and the parent function's arguments.
@ -84,7 +92,7 @@ function Something {
__CheckArguments will only trigger if script is called with DEBUG=yes __CheckArguments will only trigger if script is called with DEBUG=yes
Also, with PARANOIA_DEBUG=yes, __CheckArguments will recount all arguments given by "$*" and compare. This can mislead if arguments contain spaces. Also, with PARANOIA_DEBUG=yes, __CheckArguments will recount all arguments given by "$*" and compare. This can mislead if arguments contain spaces.
++++ If statements ++++++ If statements
If statements will be fully written (word "if" must be used). then is written on the same line. If statements will be fully written (word "if" must be used). then is written on the same line.
(Use sed ':a;N;$!ba;s/]\n\t*then/]; then/g' to convert files to this format... Replace "],new line, zero or more tabs, then" by "; then") (Use sed ':a;N;$!ba;s/]\n\t*then/]; then/g' to convert files to this format... Replace "],new line, zero or more tabs, then" by "; then")
@ -94,7 +102,7 @@ else
other stuff other stuff
fi fi
++++ Logging ++++++ Logging
A logging function is available with the following levels of logging: A logging function is available with the following levels of logging:
@ -104,26 +112,36 @@ A logging function is available with the following levels of logging:
- ERROR: Program produced an error but continues execution - ERROR: Program produced an error but continues execution
- CRITICAL: Program execution is halted - CRITICAL: Program execution is halted
++++ Eval ++++++ Eval
Most commands should be logged to a tmp file. Most commands should be logged to a tmp file.
The basic way of doing is: The basic way of doing is:
cmd='something > some_file 2>&1' cmd='"something '$somevar'" > some_file 2>&1'
eval $cmd & eval $cmd &
WaitForTaskCompletion $! 0 0 "some function name" WaitForTaskCompletion $! 0 0 $FUNCNAME
When executing rsync commands, errors should always be logged
rsync_cmd='something > some_file'
eval $cmd 2>> $LOG_FILE &
WaitForTaskCompletion $! 0 0 "some other funcion"
Remote commands should exist as: Remote commands should exist as:
cmd=$SSH_CMD' "some; commands \"'$VARIABLE'\" some; other; commands" > some_file 2>&1' cmd=$SSH_CMD' "some; commands \"'$VARIABLE'\" some; other; commands" > some_file 2>&1'
++++ File variables ++++++ File variables
All eval cmd should exit their content to a file called "$RUNDIR/osync.$FUNCNAME.$SCRIPT_PID" All eval cmd should exit their content to a file called "$RUNDIR/osync.$FUNCNAME.$SCRIPT_PID"
Dots are used instead of '_' so variables can be separated with a forbidden char in variables, so they get detected. Dots are used instead of '_' so variables can be separated with a forbidden char in variables, so they get detected.
++++++ Finding code errors
Use shellcheck.net now and then (ignore SC2086 in our case)
Use a low tech approach to find uneven number of quotes per line
tr -cd "'\n" < my_bash_file.sh | awk 'length%2==1 {print NR, $0}'
tr -cd "\"\n" < my_bash_file.sh | awk 'length%2==1 {print NR, $0}'
++++++ ofunctions
As obackup and osync share alot of common functions, ofunctions.sh will host all shared code.
Dev programs n_osync.sh and n_obackup.sh will source ofunctions.sh
Release programs will still include ofunctions.sh in order to enhance ease of use.

2
LICENCE.txt → LICENCE.TXT
View File

@ -1,4 +1,4 @@
Copyright (c) 2013, Orsiris "Ozy" de Jong. ozy@netpower.fr Copyright (c) 2013-2015, Orsiris "Ozy" de Jong. ozy@netpower.fr
All rights reserved. All rights reserved.
Redistribution and use in source and binary forms, with or without Redistribution and use in source and binary forms, with or without

2404
dev/debug_osync.sh Executable file
View File

File diff suppressed because it is too large Load Diff

49
dev/merge.sh Executable file
View File

@ -0,0 +1,49 @@
#!/usr/bin/env bash
## Merges ofunctions.sh and $PROGRAM
PROGRAM=osync
FUNC_PATH=/home/git/common
PARANOIA_DEBUG_LINE="#__WITH_PARANOIA_DEBUG"
PARANOIA_DEBUG_BEGIN="#__BEGIN_WITH_PARANOIA_DEBUG"
PARANOIA_DEBUG_END="#__END_WITH_PARANOIA_DEBUG"
function Unexpand {
unexpand n_$PROGRAM.sh > tmp_$PROGRAM.sh
}
function Merge {
sed "/source \"\/home\/git\/common\/ofunctions.sh\"/r /home/git/common/ofunctions.sh" tmp_$PROGRAM.sh | grep -v 'source "/home/git/common/ofunctions.sh"' > debug_$PROGRAM.sh
chmod +x debug_$PROGRAM.sh
}
function CleanDebug {
# sed explanation
#/pattern1/{ # if pattern1 is found
# p # print it
# :a # loop
# N # and accumulate lines
# /pattern2/!ba # until pattern2 is found
# s/.*\n// # delete the part before pattern2
#}
#p
sed -n '/'$PARANOIA_DEBUG_BEGIN'/{p; :a; N; /'$PARANOIA_DEBUG_END'/!ba; s/.*\n//}; p' debug_$PROGRAM.sh | grep -v "$PARANOIA_DEBUG_LINE" > ../$PROGRAM.sh
chmod +x ../$PROGRAM.sh
}
function CopyCommons {
sed "s/\[prgname\]/$PROGRAM/g" /home/git/common/common_install.sh > ../install.sh
sed "s/\[prgname\]/$PROGRAM/g" /home/git/common/common_batch.sh > ../$PROGRAM-batch.sh
chmod +x ../install.sh
chmod +x ../$PROGRAM-batch.sh
}
Unexpand
Merge
CleanDebug
rm -f tmp_$PROGRAM.sh
CopyCommons

1520
dev/n_osync.sh Executable file
View File

File diff suppressed because it is too large Load Diff

885
dev/ofunctions.sh Normal file
View File

@ -0,0 +1,885 @@
FUNC_BUILD=2015111102
## BEGIN Generic functions for osync & obackup written in 2013-2015 by Orsiris de Jong - http://www.netpower.fr - ozy@netpower.fr
## type -p does not work on platforms other than linux (bash). If if does not work, always assume output is not a zero exitcode
if ! type "$BASH" > /dev/null; then
echo "Please run this script only with bash shell. Tested on bash >= 3.2"
exit 127
fi
# Environment variables
_DRYRUN=0
_SILENT=0
# Initial error status, logging 'WARN', 'ERROR' or 'CRITICAL' will enable alerts flags
ERROR_ALERT=0
WARN_ALERT=0
## allow function call checks #__WITH_PARANOIA_DEBUG
if [ "$_PARANOIA_DEBUG" == "yes" ];then #__WITH_PARANOIA_DEBUG
_DEBUG=yes #__WITH_PARANOIA_DEBUG
fi #__WITH_PARANOIA_DEBUG
## allow debugging from command line with _DEBUG=yes
if [ ! "$_DEBUG" == "yes" ]; then
_DEBUG=no
SLEEP_TIME=.1
_VERBOSE=0
else
SLEEP_TIME=1
trap 'TrapError ${LINENO} $?' ERR
_VERBOSE=1
fi
SCRIPT_PID=$$
LOCAL_USER=$(whoami)
LOCAL_HOST=$(hostname)
## Default log file until config file is loaded
if [ -w /var/log ]; then
LOG_FILE="/var/log/$PROGRAM.log"
else
LOG_FILE="./$PROGRAM.log"
fi
## Default directory where to store temporary run files
if [ -w /tmp ]; then
RUN_DIR=/tmp
elif [ -w /var/tmp ]; then
RUN_DIR=/var/tmp
else
RUN_DIR=.
fi
## Log a state message every $KEEP_LOGGING seconds. Should not be equal to soft or hard execution time so your log will not be unnecessary big.
KEEP_LOGGING=1801
## Correct output of sort command (language agnostic sorting)
export LC_ALL=C
# Standard alert mail body
MAIL_ALERT_MSG="Execution of $PROGRAM instance $INSTANCE_ID on $(date) has warnings/errors."
# Default alert attachment filename
ALERT_LOG_FILE="$RUN_DIR/$PROGRAM.last.log"
# Set error exit code if a piped command fails
set -o pipefail
set -o errtrace
function Dummy {
__CheckArguments 0 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
sleep .1
}
function _Logger {
local svalue="${1}" # What to log to screen
local lvalue="${2:-$svalue}" # What to log to logfile, defaults to screen value
echo -e "$lvalue" >> "$LOG_FILE"
if [ $_SILENT -eq 0 ]; then
echo -e "$svalue"
fi
}
function Logger {
local value="${1}" # Sentence to log (in double quotes)
local level="${2}" # Log level: PARANOIA_DEBUG, DEBUG, NOTICE, WARN, ERROR, CRITIAL
# <OSYNC SPECIFIC> Special case in daemon mode we should timestamp instead of counting seconds
if [ "$sync_on_changes" == "1" ]; then
prefix="$(date) - "
else
prefix="TIME: $SECONDS - "
fi
# </OSYNC SPECIFIC>
if [ "$level" == "CRITICAL" ]; then
_Logger "$prefix\e[41m$value\e[0m" "$prefix$value"
ERROR_ALERT=1
return
elif [ "$level" == "ERROR" ]; then
_Logger "$prefix\e[91m$value\e[0m" "$prefix$value"
ERROR_ALERT=1
return
elif [ "$level" == "WARN" ]; then
_Logger "$prefix\e[93m$value\e[0m" "$prefix$value"
WARN_ALERT=1
return
elif [ "$level" == "NOTICE" ]; then
_Logger "$prefix$value"
return
elif [ "$level" == "DEBUG" ]; then
if [ "$_DEBUG" == "yes" ]; then
_Logger "$prefix$value"
return
fi
elif [ "$level" == "PARANOIA_DEBUG" ]; then #__WITH_PARANOIA_DEBUG
if [ "$_PARANOIA_DEBUG" == "yes" ]; then #__WITH_PARANOIA_DEBUG
_Logger "$prefix$value" #__WITH_PARANOIA_DEBUG
return #__WITH_PARANOIA_DEBUG
fi #__WITH_PARANOIA_DEBUG
else
_Logger "\e[41mLogger function called without proper loglevel.\e[0m"
_Logger "$prefix$value"
fi
}
# Portable child (and grandchild) kill function tester under Linux, BSD and MacOS X
function KillChilds {
local pid="${1}"
local self="${2:-false}"
if children="$(pgrep -P "$pid")"; then
for child in $children; do
KillChilds "$child" true
done
fi
# Try to kill nicely, if not, wait 30 seconds to let Trap actions happen before killing
if [ "$self" == true ]; then
kill -s SIGTERM "$pid" || (sleep 30 && kill -9 "$pid" &)
fi
}
function TrapError {
local job="$0"
local line="$1"
local code="${2:-1}"
if [ $_SILENT -eq 0 ]; then
echo -e " /!\ ERROR in ${job}: Near line ${line}, exit code ${code}"
fi
}
function Spinner {
if [ $_SILENT -eq 1 ]; then
return 0
fi
case $toggle
in
1)
echo -n " \ "
echo -ne "\r"
toggle="2"
;;
2)
echo -n " | "
echo -ne "\r"
toggle="3"
;;
3)
echo -n " / "
echo -ne "\r"
toggle="4"
;;
*)
echo -n " - "
echo -ne "\r"
toggle="1"
;;
esac
}
function SedStripQuotes {
echo $(echo $1 | sed "s/^\([\"']\)\(.*\)\1\$/\2/g")
}
function StripSingleQuotes {
local string="${1}"
string="${string/#\'/}" # Remove singlequote if it begins string
string="${string/%\'/}" # Remove singlequote if it ends string
echo "$string"
}
function StripDoubleQuotes {
local string="${1}"
string="${string/#\"/}"
string="${string/%\"/}"
echo "$string"
}
function StripQuotes {
local string="${1}"
echo "$(StripSingleQuotes $(StripDoubleQuotes $string))"
}
function EscapeSpaces {
local string="${1}" # String on which spaces will be escaped
echo "${string// /\ }"
}
function IsNumeric {
eval "local value=\"${1}\"" # Needed so variable variables can be processed
local re="^-?[0-9]+([.][0-9]+)?$"
if [[ $value =~ $re ]]; then
echo 1
else
echo 0
fi
}
function CleanUp {
__CheckArguments 0 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
if [ "$_DEBUG" != "yes" ]; then
rm -f "$RUN_DIR/$PROGRAM."*".$SCRIPT_PID"
fi
}
function SendAlert {
__CheckArguments 0 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
if [ "$_DEBUG" == "yes" ]; then
Logger "Debug mode, no warning email will be sent." "NOTICE"
return 0
fi
# <OSYNC SPECIFIC>
if [ "$_QUICK_SYNC" == "2" ]; then
Logger "Current task is a quicksync task. Will not send any alert." "NOTICE"
return 0
fi
# </OSYNC SPECIFIC>
eval "cat \"$LOG_FILE\" $COMPRESSION_PROGRAM > $ALERT_LOG_FILE"
MAIL_ALERT_MSG="$MAIL_ALERT_MSG"$'\n\n'$(tail -n 25 "$LOG_FILE")
if [ $ERROR_ALERT -eq 1 ]; then
subject="Error alert for $INSTANCE_ID"
elif [ $WARN_ALERT -eq 1 ]; then
subject="Warning alert for $INSTANCE_ID"
else
subject="Alert for $INSTANCE_ID"
fi
# Need better fallback if mail sending does not succeed
if type mutt > /dev/null 2>&1 ; then
echo "$MAIL_ALERT_MSG" | $(type -p mutt) -x -s "$subject" $DESTINATION_MAILS -a "$ALERT_LOG_FILE"
if [ $? != 0 ]; then
Logger "WARNING: Cannot send alert email via $(type -p mutt) !!!" "WARN"
else
Logger "Sent alert mail using mutt." "NOTICE"
return 0
fi
fi
if type mail > /dev/null 2>&1 ; then
echo "$MAIL_ALERT_MSG" | $(type -p mail) -a "$ALERT_LOG_FILE" -s "$subject" $DESTINATION_MAILS
if [ $? != 0 ]; then
Logger "WARNING: Cannot send alert email via $(type -p mail) with attachments !!!" "WARN"
echo "$MAIL_ALERT_MSG" | $(type -p mail) -s "$subject" $DESTINATION_MAILS
if [ $? != 0 ]; then
Logger "WARNING: Cannot send alert email via $(type -p mail) without attachments !!!" "WARN"
else
Logger "Sent alert mail using mail command without attachment." "NOTICE"
return 0
fi
else
Logger "Sent alert mail using mail command." "NOTICE"
return 0
fi
fi
if type sendmail > /dev/null 2>&1 ; then
echo -e "$subject\r\n$MAIL_ALERT_MSG" | $(type -p sendmail) $DESTINATION_MAILS
if [ $? != 0 ]; then
Logger "WARNING: Cannot send alert email via $(type -p sendmail) !!!" "WARN"
else
Logger "Sent alert mail using sendmail command without attachment." "NOTICE"
return 0
fi
fi
if type sendemail > /dev/null 2>&1 ; then
if [ "$SMTP_USER" != "" ] && [ "$SMTP_PASSWORD" != "" ]; then
SMTP_OPTIONS="-xu $SMTP_USER -xp $SMTP_PASSWORD"
else
SMTP_OPTIONS=""
fi
$(type -p sendemail) -f $SENDER_MAIL -t $DESTINATION_MAILS -u "$subject" -m "$MAIL_ALERT_MSG" -s $SMTP_SERVER $SMTP_OPTIONS > /dev/null 2>&1
if [ $? != 0 ]; then
Logger "WARNING: Cannot send alert email via $(type -p sendemail) !!!" "WARN"
else
Logger "Sent alert mail using sendemail command without attachment." "NOTICE"
return 0
fi
fi
# If function has not returned 0 yet, assume it's critical that no alert can be sent
Logger "/!\ CRITICAL: Cannot send alert" "ERROR" # Is not marked critical because execution must continue
# Delete tmp log file
if [ -f "$ALERT_LOG_FILE" ]; then
rm "$ALERT_LOG_FILE"
fi
}
function LoadConfigFile {
__CheckArguments 1 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
local config_file="${1}"
if [ ! -f "$config_file" ]; then
Logger "Cannot load configuration file [$config_file]. Cannot start." "CRITICAL"
exit 1
elif [[ "$1" != *".conf" ]]; then
Logger "Wrong configuration file supplied [$config_file]. Cannot start." "CRITICAL"
exit 1
else
grep '^[^ ]*=[^;&]*' "$config_file" > "$RUN_DIR/$PROGRAM.$FUNCNAME.$SCRIPT_PID" # WITHOUT COMMENTS
# Shellcheck source=./sync.conf
source "$RUN_DIR/$PROGRAM.$FUNCNAME.$SCRIPT_PID"
fi
}
function GetLocalOS {
__CheckArguments 0 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
local local_os_var=$(uname -spio 2>&1)
if [ $? != 0 ]; then
local local_os_var=$(uname -v 2>&1)
if [ $? != 0 ]; then
local local_os_var=($uname)
fi
fi
case $local_os_var in
*"Linux"*)
LOCAL_OS="Linux"
;;
*"BSD"*)
LOCAL_OS="BSD"
;;
*"MINGW32"*)
LOCAL_OS="msys"
;;
*"Darwin"*)
LOCAL_OS="MacOSX"
;;
*)
Logger "Running on >> $local_os_var << not supported. Please report to the author." "ERROR"
exit 1
;;
esac
Logger "Local OS: [$local_os_var]." "DEBUG"
}
function GetRemoteOS {
__CheckArguments 0 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
if [ "$REMOTE_OPERATION" == "yes" ]; then
CheckConnectivity3rdPartyHosts
CheckConnectivityRemoteHost
local cmd=$SSH_CMD' "uname -spio" > "'$RUN_DIR/$PROGRAM.$FUNCNAME.$SCRIPT_PID'" 2>&1'
Logger "cmd: $cmd" "DEBUG"
eval "$cmd" &
WaitForTaskCompletion $! 120 240 $FUNCNAME"-1"
retval=$?
if [ $retval != 0 ]; then
local cmd=$SSH_CMD' "uname -v" > "'$RUN_DIR/$PROGRAM.$FUNCNAME.$SCRIPT_PID'" 2>&1'
Logger "cmd: $cmd" "DEBUG"
eval "$cmd" &
WaitForTaskCompletion $! 120 240 $FUNCNAME"-2"
retval=$?
if [ $retval != 0 ]; then
local cmd=$SSH_CMD' "uname" > "'$RUN_DIR/$PROGRAM.$FUNCNAME.$SCRIPT_PID'" 2>&1'
Logger "cmd: $cmd" "DEBUG"
eval "$cmd" &
WaitForTaskCompletion $! 120 240 $FUNCNAME"-3"
retval=$?
if [ $retval != 0 ]; then
Logger "Cannot Get remote OS type." "ERROR"
fi
fi
fi
local remote_os_var=$(cat $RUN_DIR/$PROGRAM.$FUNCNAME.$SCRIPT_PID)
case $remote_os_var in
*"Linux"*)
REMOTE_OS="Linux"
;;
*"BSD"*)
REMOTE_OS="BSD"
;;
*"MINGW32"*)
REMOTE_OS="msys"
;;
*"Darwin"*)
REMOTE_OS="MacOSX"
;;
*"ssh"*|*"SSH"*)
Logger "Cannot connect to remote system." "CRITICAL"
exit 1
;;
*)
Logger "Running on remote OS failed. Please report to the author if the OS is not supported." "CRITICAL"
Logger "Remote OS said:\n$remote_os_var" "CRITICAL"
exit 1
esac
Logger "Remote OS: [$remote_os_var]." "DEBUG"
fi
}
function WaitForTaskCompletion {
local pid="${1}" # pid to wait for
local soft_max_time="${2}" # If program with pid $pid takes longer than $soft_max_time seconds, will log a warning, unless $soft_max_time equals 0.
local hard_max_time="${3}" # If program with pid $pid takes longer than $hard_max_time seconds, will stop execution, unless $hard_max_time equals 0.
local caller_name="${4}" # Who called this function
Logger "$FUNCNAME called by [$caller_name]." "PARANOIA_DEBUG" #__WITH_PARANOIA_DEBUG
__CheckArguments 4 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
local soft_alert=0 # Does a soft alert need to be triggered, if yes, send an alert once
local log_ttime=0 # local time instance for comparaison
local seconds_begin=$SECONDS # Seconds since the beginning of the script
local exec_time=0 # Seconds since the beginning of this function
while eval "$PROCESS_TEST_CMD" > /dev/null
do
Spinner
exec_time=$(($SECONDS - $seconds_begin))
if [ $((($exec_time + 1) % $KEEP_LOGGING)) -eq 0 ]; then
if [ $log_ttime -ne $exec_time ]; then
log_ttime=$exec_time
Logger "Current task still running." "NOTICE"
fi
fi
if [ $exec_time -gt $soft_max_time ]; then
if [ $soft_alert -eq 0 ] && [ $soft_max_time -ne 0 ]; then
Logger "Max soft execution time exceeded for task." "WARN"
soft_alert=1
SendAlert
fi
if [ $exec_time -gt $hard_max_time ] && [ $hard_max_time -ne 0 ]; then
Logger "Max hard execution time exceeded for task. Stopping task execution." "ERROR"
kill -s SIGTERM $pid
if [ $? == 0 ]; then
Logger "Task stopped succesfully" "NOTICE"
else
Logger "Sending SIGTERM to proces failed. Trying the hard way." "ERROR"
sleep 5 && kill -9 $pid
if [ $? != 0 ]; then
Logger "Could not stop task." "ERROR"
fi
fi
return 1
fi
fi
sleep $SLEEP_TIME
done
wait $pid
local retval=$?
Logger "$FUNCNAME ended for [$caller_name] with status $retval." "PARANOIA_DEBUG" #__WITH_PARANOIA_DEBUG
return $retval
}
function WaitForCompletion {
local pid="${1}" # pid to wait for
local soft_max_time="${2}" # If program with pid $pid takes longer than $soft_max_time seconds, will log a warning, unless $soft_max_time equals 0.
local hard_max_time="${3}" # If program with pid $pid takes longer than $hard_max_time seconds, will stop execution, unless $hard_max_time equals 0.
local caller_name="${4}" # Who called this function
Logger "$FUNCNAME called by [$caller_name]" "PARANOIA_DEBUG" #__WITH_PARANOIA_DEBUG
__CheckArguments 4 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
local soft_alert=0 # Does a soft alert need to be triggered, if yes, send an alert once
local log_ttime=0 # local time instance for comparaison
local seconds_begin=$SECONDS # Seconds since the beginning of the script
local exec_time=0 # Seconds since the beginning of this function
while eval "$PROCESS_TEST_CMD" > /dev/null
do
Spinner
if [ $((($SECONDS + 1) % $KEEP_LOGGING)) -eq 0 ]; then
if [ $log_time -ne $SECONDS ]; then
log_time=$SECONDS
Logger "Current task still running." "NOTICE"
fi
fi
if [ $SECONDS -gt $soft_max_time ]; then
if [ $soft_alert -eq 0 ] && [ $soft_max_time != 0 ]; then
Logger "Max soft execution time exceeded for script." "WARN"
soft_alert=1
SendAlert
fi
if [ $SECONDS -gt $hard_max_time ] && [ $hard_max_time != 0 ]; then
Logger "Max hard execution time exceeded for script. Stopping current task execution." "ERROR"
kill -s SIGTERM $pid
if [ $? == 0 ]; then
Logger "Task stopped succesfully" "NOTICE"
else
Logger "Sending SIGTERM to proces failed. Trying the hard way." "ERROR"
kill -9 $pid
if [ $? != 0 ]; then
Logger "Could not stop task." "ERROR"
fi
fi
return 1
fi
fi
sleep $SLEEP_TIME
done
wait $pid
retval=$?
Logger "$FUNCNAME ended for [$caller_name] with status $retval." "PARANOIA_DEBUG" #__WITH_PARANOIA_DEBUG
return $retval
}
function RunLocalCommand {
local command="${1}" # Command to run
local hard_max_time="${2}" # Max time to wait for command to compleet
__CheckArguments 2 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
if [ $_DRYRUN -ne 0 ]; then
Logger "Dryrun: Local command [$command] not run." "NOTICE"
return 1
fi
Logger "Running command [$command] on local host." "NOTICE"
eval "$command" > "$RUN_DIR/$PROGRAM.$FUNCNAME.$SCRIPT_PID" 2>&1 &
WaitForTaskCompletion $! 0 $hard_max_time $FUNCNAME
retval=$?
if [ $retval -eq 0 ]; then
Logger "Command succeded." "NOTICE"
else
Logger "Command failed." "ERROR"
fi
if [ $_VERBOSE -eq 1 ] || [ $retval -ne 0 ]; then
Logger "Command output:\n$(cat $RUN_DIR/$PROGRAM.$FUNCNAME.$SCRIPT_PID)" "NOTICE"
fi
if [ "$STOP_ON_CMD_ERROR" == "yes" ] && [ $retval -ne 0 ]; then
Logger "Stopping on command execution error." "CRITICAL"
exit 1
fi
}
## Runs remote command $1 and waits for completition in $2 seconds
function RunRemoteCommand {
local command="${1}" # Command to run
local hard_max_time="${2}" # Max time to wait for command to compleet
__CheckArguments 2 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
CheckConnectivity3rdPartyHosts
CheckConnectivityRemoteHost
if [ $_DRYRUN -ne 0 ]; then
Logger "Dryrun: Local command [$command] not run." "NOTICE"
return 1
fi
Logger "Running command [$command] on remote host." "NOTICE"
cmd=$SSH_CMD' "$command" > "'$RUN_DIR/$PROGRAM.$FUNCNAME.$SCRIPT_PID'" 2>&1'
Logger "cmd: $cmd" "DEBUG"
eval "$cmd" &
WaitForTaskCompletion $! 0 $hard_max_time $FUNCNAME
retval=$?
if [ $retval -eq 0 ]; then
Logger "Command succeded." "NOTICE"
else
Logger "Command failed." "ERROR"
fi
if [ -f "$RUN_DIR/$PROGRAM.$FUNCNAME.$SCRIPT_PID" ] && ([ $_VERBOSE -eq 1 ] || [ $retval -ne 0 ])
then
Logger "Command output:\n$(cat $RUN_DIR/$PROGRAM.$FUNCNAME.$SCRIPT_PID)" "NOTICE"
fi
if [ "$STOP_ON_CMD_ERROR" == "yes" ] && [ $retval -ne 0 ]; then
Logger "Stopping on command execution error." "CRITICAL"
exit 1
fi
}
function RunBeforeHook {
__CheckArguments 0 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
if [ "$LOCAL_RUN_BEFORE_CMD" != "" ]; then
RunLocalCommand "$LOCAL_RUN_BEFORE_CMD" $MAX_EXEC_TIME_PER_CMD_BEFORE
fi
if [ "$REMOTE_RUN_BEFORE_CMD" != "" ]; then
RunRemoteCommand "$REMOTE_RUN_BEFORE_CMD" $MAX_EXEC_TIME_PER_CMD_BEFORE
fi
}
function RunAfterHook {
__CheckArguments 0 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
if [ "$LOCAL_RUN_AFTER_CMD" != "" ]; then
RunLocalCommand "$LOCAL_RUN_AFTER_CMD" $MAX_EXEC_TIME_PER_CMD_AFTER
fi
if [ "$REMOTE_RUN_AFTER_CMD" != "" ]; then
RunRemoteCommand "$REMOTE_RUN_AFTER_CMD" $MAX_EXEC_TIME_PER_CMD_AFTER
fi
}
function CheckConnectivityRemoteHost {
__CheckArguments 0 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
if [ "$_PARANOIA_DEBUG" != "yes" ]; then # Do not loose time in paranoia debug
if [ "$REMOTE_HOST_PING" != "no" ] && [ "$REMOTE_OPERATION" != "no" ]; then
eval "$PING_CMD $REMOTE_HOST > /dev/null 2>&1" &
WaitForTaskCompletion $! 180 180 $FUNCNAME
if [ $? != 0 ]; then
Logger "Cannot ping $REMOTE_HOST" "CRITICAL"
return 1
fi
fi
fi
}
function CheckConnectivity3rdPartyHosts {
__CheckArguments 0 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
if [ "$_PARANOIA_DEBUG" != "yes" ]; then # Do not loose time in paranoia debug
if [ "$REMOTE_3RD_PARTY_HOSTS" != "" ]; then
remote_3rd_party_success=0
OLD_IFS=$IFS
IFS=$' \t\n'
for i in $REMOTE_3RD_PARTY_HOSTS
do
eval "$PING_CMD $i > /dev/null 2>&1" &
WaitForTaskCompletion $! 360 360 $FUNCNAME
if [ $? != 0 ]; then
Logger "Cannot ping 3rd party host $i" "WARN"
else
remote_3rd_party_success=1
fi
done
IFS=$OLD_IFS
if [ $remote_3rd_party_success -ne 1 ]; then
Logger "No remote 3rd party host responded to ping. No internet ?" "CRITICAL"
return 1
fi
fi
fi
}
#__BEGIN_WITH_PARANOIA_DEBUG
function __CheckArguments {
# Checks the number of arguments of a function and raises an error if some are missing
if [ "$_DEBUG" == "yes" ]; then
local number_of_arguments="${1}" # Number of arguments the tested function should have
local number_of_given_arguments="${2}" # Number of arguments that have been passed
local function_name="${3}" # Function name that called __CheckArguments
if [ "$_PARANOIA_DEBUG" == "yes" ]; then
Logger "Entering function [$function_name]." "DEBUG"
fi
# All arguments of the function to check are passed as array in ${4} (the function call waits for $@)
# If any of the arguments contains spaces, bash things there are two aguments
# In order to avoid this, we need to iterate over ${4} and count
local iterate=4
local fetch_arguments=1
local arg_list=""
while [ $fetch_arguments -eq 1 ]; do
cmd='argument=${'$iterate'}'
eval $cmd
if [ "$argument" = "" ]; then
fetch_arguments=0
else
arg_list="$arg_list [Argument $(($iterate-3)): $argument]"
iterate=$(($iterate+1))
fi
done
local counted_arguments=$((iterate-4))
if [ $counted_arguments -ne $number_of_arguments ]; then
Logger "Function $function_name may have inconsistent number of arguments. Expected: $number_of_arguments, count: $counted_arguments, see log file." "ERROR"
Logger "Arguments passed: $arg_list" "ERROR"
fi
fi
}
function old__CheckArguments {
# Checks the number of arguments and raises an error if some are missing
if [ "$_DEBUG" == "yes" ]; then
local number_of_arguments="${1}" # Number of arguments a function should have
local number_of_given_arguments="${2}" # Number of arguments that have been passed
local function_name="${3}" # Function name that called __CheckArguments
local arguments="${4}" # All other arguments
if [ "$_PARANOIA_DEBUG" == "yes" ]; then
Logger "Entering function [$function_name]." "DEBUG"
# Paranoia check... Can help finding empty arguments. __CheckArguments should be grepped out in production builds.
local count=-3 # Number of arguments minus the function calls for __CheckArguments
for i in $@; do
count=$((count + 1))
done
if [ $count -ne $1 ]; then
Logger "Function $function_name may have inconsistent number of arguments. Expected: $number_of_arguments, count: $count, see log file." "WARN"
echo "Argument list (including checks): $*" >> "$LOG_FILE"
fi
fi
if [ $number_of_arguments -ne $number_of_given_arguments ]; then
Logger "Inconsistnent number of arguments in $function_name. Should have $number_of_arguments arguments, has $number_of_given_arguments arguments, see log file." "CRITICAL"
# Cannot user Logger here because $@ is a list of arguments
echo "Argumnt list: $4" >> "$LOG_FILE"
fi
fi
}
#__END_WITH_PARANOIA_DEBUG
function PreInit {
__CheckArguments 0 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
## SSH compression
if [ "$SSH_COMPRESSION" != "no" ]; then
SSH_COMP=-C
else
SSH_COMP=
fi
## Support for older config files without RSYNC_EXECUTABLE option
if [ "$RSYNC_EXECUTABLE" == "" ]; then
RSYNC_EXECUTABLE=rsync
fi
## Sudo execution option
if [ "$SUDO_EXEC" == "yes" ]; then
if [ "$RSYNC_REMOTE_PATH" != "" ]; then
RSYNC_PATH="sudo $RSYNC_REMOTE_PATH/$RSYNC_EXECUTABLE"
else
RSYNC_PATH="sudo $RSYNC_EXECUTABLE"
fi
COMMAND_SUDO="sudo"
else
if [ "$RSYNC_REMOTE_PATH" != "" ]; then
RSYNC_PATH="$RSYNC_REMOTE_PATH/$RSYNC_EXECUTABLE"
else
RSYNC_PATH="$RSYNC_EXECUTABLE"
fi
COMMAND_SUDO=""
fi
## Set rsync default arguments
RSYNC_ARGS="-rlptgoD"
if [ "$PRESERVE_ACL" == "yes" ]; then
RSYNC_ARGS=$RSYNC_ARGS" -A"
fi
if [ "$PRESERVE_XATTR" == "yes" ]; then
RSYNC_ARGS=$RSYNC_ARGS" -X"
fi
if [ "$RSYNC_COMPRESS" == "yes" ]; then
RSYNC_ARGS=$RSYNC_ARGS" -z"
fi
if [ "$COPY_SYMLINKS" == "yes" ]; then
RSYNC_ARGS=$RSYNC_ARGS" -L"
fi
if [ "$KEEP_DIRLINKS" == "yes" ]; then
RSYNC_ARGS=$RSYNC_ARGS" -K"
fi
if [ "$PRESERVE_HARDLINKS" == "yes" ]; then
RSYNC_ARGS=$RSYNC_ARGS" -H"
fi
if [ "$CHECKSUM" == "yes" ]; then
RSYNC_ARGS=$RSYNC_ARGS" --checksum"
fi
if [ $_DRYRUN -eq 1 ]; then
RSYNC_ARGS=$RSYNC_ARGS" -n"
DRY_WARNING="/!\ DRY RUN"
fi
if [ "$BANDWIDTH" != "" ] && [ "$BANDWIDTH" != "0" ]; then
RSYNC_ARGS=$RSYNC_ARGS" --bwlimit=$BANDWIDTH"
fi
## Set compression executable and extension
COMPRESSION_LEVEL=3
if type xz > /dev/null 2>&1
then
COMPRESSION_PROGRAM="| xz -$COMPRESSION_LEVEL"
COMPRESSION_EXTENSION=.xz
elif type lzma > /dev/null 2>&1
then
COMPRESSION_PROGRAM="| lzma -$COMPRESSION_LEVEL"
COMPRESSION_EXTENSION=.lzma
elif type pigz > /dev/null 2>&1
then
COMPRESSION_PROGRAM="| pigz -$COMPRESSION_LEVEL"
COMPRESSION_EXTENSION=.gz
COMPRESSION_OPTIONS=--rsyncable
elif type gzip > /dev/null 2>&1
then
COMPRESSION_PROGRAM="| gzip -$COMPRESSION_LEVEL"
COMPRESSION_EXTENSION=.gz
COMPRESSION_OPTIONS=--rsyncable
else
COMPRESSION_PROGRAM=
COMPRESSION_EXTENSION=
fi
ALERT_LOG_FILE="$ALERT_LOG_FILE$COMPRESSION_EXTENSION"
}
function PostInit {
__CheckArguments 0 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
# Define remote commands
SSH_CMD="$(type -p ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY $REMOTE_USER@$REMOTE_HOST -p $REMOTE_PORT"
SCP_CMD="$(type -p scp) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY -P $REMOTE_PORT"
RSYNC_SSH_CMD="$(type -p ssh) $SSH_COMP -i $SSH_RSA_PRIVATE_KEY -p $REMOTE_PORT"
}
function InitLocalOSSettings {
__CheckArguments 0 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
## If running under Msys, some commands do not run the same way
## Using mingw version of find instead of windows one
## Getting running processes is quite different
## Ping command is not the same
if [ "$LOCAL_OS" == "msys" ]; then
FIND_CMD=$(dirname $BASH)/find
#TODO: The following command needs to be checked on msys. Does the $1 variable substitution work ?
# PROCESS_TEST_CMD assumes there is a variable $pid
PROCESS_TEST_CMD='ps -a | awk "{\$1=\$1}\$1" | awk "{print \$1}" | grep $pid'
PING_CMD="ping -n 2"
else
FIND_CMD=find
# PROCESS_TEST_CMD assumes there is a variable $pid
PROCESS_TEST_CMD='ps -p$pid'
PING_CMD="ping -c 2 -i .2"
fi
## Stat command has different syntax on Linux and FreeBSD/MacOSX
if [ "$LOCAL_OS" == "MacOSX" ] || [ "$LOCAL_OS" == "BSD" ]; then
STAT_CMD="stat -f \"%Sm\""
else
STAT_CMD="stat --format %y"
fi
}
function InitRemoteOSSettings {
__CheckArguments 0 $# $FUNCNAME "$@" #__WITH_PARANOIA_DEBUG
## MacOSX does not use the -E parameter like Linux or BSD does (-E is mapped to extended attrs instead of preserve executability)
if [ "$LOCAL_OS" != "MacOSX" ] && [ "$REMOTE_OS" != "MacOSX" ]; then
RSYNC_ARGS=$RSYNC_ARGS" -E"
fi
if [ "$REMOTE_OS" == "msys" ]; then
REMOTE_FIND_CMD=$(dirname $BASH)/find
else
REMOTE_FIND_CMD=find
fi
}
## END Generic functions

72
install.sh
View File

@ -1,12 +1,15 @@
#!/usr/bin/env bash #!/usr/bin/env bash
SCRIPT_BUILD=2015092801 PROGRAM=osync
PROGRAM_BINARY=$PROGRAM".sh"
PROGRAM_BATCH=$PROGRAM"-batch.sh"
SCRIPT_BUILD=2015102701
## Osync daemon install script ## osync / obackup daemon install script
## Tested on RHEL / CentOS 6 & 7 and Mint 17 ## Tested on RHEL / CentOS 6 & 7 and Mint 17
## Please adapt this to fit your distro needs ## Please adapt this to fit your distro needs
OSYNC_CONF_DIR=/etc/osync CONF_DIR=/etc/$PROGRAM
BIN_DIR=/usr/local/bin BIN_DIR=/usr/local/bin
SERVICE_DIR=/etc/init.d SERVICE_DIR=/etc/init.d
@ -15,28 +18,41 @@ if [ "$(whoami)" != "root" ]; then
exit 1 exit 1
fi fi
if [ ! -d "$OSYNC_CONF_DIR" ]; then if [ ! -d "$CONF_DIR" ]; then
mkdir "$OSYNC_CONF_DIR" mkdir "$CONF_DIR"
if [ $? == 0 ]; then if [ $? == 0 ]; then
echo "Created directory [$OSYNC_CONF_DIR]." echo "Created directory [$CONF_DIR]."
else
echo "Cannot create directory [$CONF_DIR]."
exit 1
fi fi
else else
echo "Config directory [$OSYNC_CONF_DIR] exists." echo "Config directory [$CONF_DIR] exists."
fi fi
cp ./sync.conf /etc/osync/sync.conf.example if [ -f ./sync.conf ]; then
cp ./exclude.list.example /etc/osync cp ./sync.conf /etc/$PROGRAM/sync.conf.example
cp ./osync.sh "$BIN_DIR"
if [ $? != 0 ]; then
echo "Cannot copy osync.sh to [$BIN_DIR]."
else
echo "Copied osync.sh to [$BIN_DIR]."
fi fi
cp ./osync-batch.sh /usr/local/bin
if [ -f ./host_backup.conf ]; then
cp ./host_backup.conf /etc/$PROGRAM/host_backup.conf.example
fi
if [ -f ./exlude.list.example ]; then
cp ./exclude.list.example /etc/$PROGRAM
fi
cp ./$PROGRAM_BINARY "$BIN_DIR"
if [ $? != 0 ]; then if [ $? != 0 ]; then
echo "Cannot copy osync-batch.sh to [$BIN_DIR]." echo "Cannot copy $PROGRAM_BINARY to [$BIN_DIR]."
else else
echo "Copied osync-batch.sh to [$BIN_DIR]." echo "Copied $PROGRAM_BINARY to [$BIN_DIR]."
fi
cp ./$PROGRAM_BATCH /usr/local/bin
if [ $? != 0 ]; then
echo "Cannot copy $PROGRAM_BATCH to [$BIN_DIR]."
else
echo "Copied $PROGRAM_BATCH to [$BIN_DIR]."
fi fi
cp ./ssh_filter.sh /usr/local/bin cp ./ssh_filter.sh /usr/local/bin
if [ $? != 0 ]; then if [ $? != 0 ]; then
@ -44,16 +60,18 @@ if [ $? != 0 ]; then
else else
echo "Copied ssh_filter.sh to [$BIN_DIR]." echo "Copied ssh_filter.sh to [$BIN_DIR]."
fi fi
cp ./osync-srv "$SERVICE_DIR"
if [ $? != 0 ]; then if [ -f ./osync-srv ]; then
echo "Cannot copy osync-srv to [$SERVICE_DIR]." cp ./osync-srv "$SERVICE_DIR"
else if [ $? != 0 ]; then
echo "Created osync-srv service in [$SERVICE_DIR]." echo "Cannot copy osync-srv to [$SERVICE_DIR]."
else
echo "Created osync-srv service in [$SERVICE_DIR]."
chmod 755 /etc/init.d/osync-srv
fi
fi fi
chmod 755 /usr/local/bin/osync.sh
chmod 755 /usr/local/bin/osync-batch.sh chmod 755 /usr/local/bin/$PROGRAM_BINARY
chmod 755 /usr/local/bin/$PROGRAM_BATCH
chmod 755 /usr/local/bin/ssh_filter.sh chmod 755 /usr/local/bin/ssh_filter.sh
chown root:root /usr/local/bin/ssh_filter.sh chown root:root /usr/local/bin/ssh_filter.sh
chmod 755 /etc/init.d/osync-srv

64
osync-batch.sh
View File

@ -1,15 +1,15 @@
#!/usr/bin/env bash #!/usr/bin/env bash
SUBPROGRAM=osync
PROGRAM="$SUBPROGRAM-batch" # Batch program to run osync / obackup instances sequentially and rerun failed ones
AUTHOR="(L) 2013-2015 by Orsiris \"Ozy\" de Jong"
CONTACT="http://www.netpower.fr - ozy@netpower.fr"
PROGRAM_BUILD=2015103001
PROGRAM="Osync-batch" # Batch program to run osync instances sequentially and rerun failed ones ## Runs an osync /obackup instance for every conf file found
AUTHOR="(L) 2013-2014 by Orsiris \"Ozy\" de Jong"
CONTACT="http://www.netpower.fr/osync - ozy@netpower.fr"
PROGRAM_BUILD=2015092901
## Runs an osync instance for every conf file found
## If an instance fails, run it again if time permits ## If an instance fails, run it again if time permits
## Configuration file path. The path where all the osync conf files are, usually /etc/osync ## Configuration file path. The path where all the osync / obackup conf files are, usually /etc/osync or /etc/obackup
CONF_FILE_PATH=/etc/osync CONF_FILE_PATH=/etc/$SUBPROGRAM
## If maximum execution time is not reached, failed instances will be rerun. Max exec time is in seconds. Example is set to 10 hours. ## If maximum execution time is not reached, failed instances will be rerun. Max exec time is in seconds. Example is set to 10 hours.
MAX_EXECUTION_TIME=36000 MAX_EXECUTION_TIME=36000
@ -19,9 +19,9 @@ MAX_RERUNS=3
## Log file path ## Log file path
if [ -w /var/log ]; then if [ -w /var/log ]; then
LOG_FILE=/var/log/osync-batch.log LOG_FILE=/var/log/$SUBPROGRAM-batch.log
else else
LOG_FILE=./osync-batch.log LOG_FILE=./$SUBPROGRAM-batch.log
fi fi
# No need to edit under this line ############################################################## # No need to edit under this line ##############################################################
@ -30,7 +30,7 @@ function _logger {
local value="${1}" # What to log local value="${1}" # What to log
echo -e "$value" >> "$LOG_FILE" echo -e "$value" >> "$LOG_FILE"
if [ $silent -eq 0 ]; then if [ $_SILENT -eq 0 ]; then
echo -e "$value" echo -e "$value"
fi fi
} }
@ -62,18 +62,18 @@ function Logger {
} }
function CheckEnvironment { function CheckEnvironment {
## Osync executable full path can be set here if it cannot be found on the system ## osync / obackup executable full path can be set here if it cannot be found on the system
if ! type -p osync.sh > /dev/null 2>&1 if ! type -p $SUBPROGRAM.sh > /dev/null 2>&1
then then
if [ -f /usr/local/bin/osync.sh ] if [ -f /usr/local/bin/$SUBPROGRAM.sh ]
then then
OSYNC_EXECUTABLE=/usr/local/bin/osync.sh SUBPROGRAM_EXECUTABLE=/usr/local/bin/$SUBPROGRAM.sh
else else
Logger "Could not find osync.sh" "CRITICAL" Logger "Could not find $SUBPROGRAM.sh" "CRITICAL"
exit 1 exit 1
fi fi
else else
OSYNC_EXECUTABLE=$(type -p osync.sh) SUBPROGRAM_EXECUTABLE=$(type -p $SUBPROGRAM.sh)
fi fi
## Check for CONF_FILE_PATH ## Check for CONF_FILE_PATH
@ -97,10 +97,10 @@ function Batch {
RERUNS=0 RERUNS=0
while ([ $MAX_EXECUTION_TIME -gt $SECONDS ] || [ $MAX_EXECUTION_TIME -eq 0 ]) && [ "$RUN" != "" ] && [ $MAX_RERUNS -gt $RERUNS ] while ([ $MAX_EXECUTION_TIME -gt $SECONDS ] || [ $MAX_EXECUTION_TIME -eq 0 ]) && [ "$RUN" != "" ] && [ $MAX_RERUNS -gt $RERUNS ]
do do
Logger "Osync instances will be run for: $RUN" "NOTICE" Logger "$SUBPROGRAM instances will be run for: $RUN" "NOTICE"
for i in $RUN for i in $RUN
do do
$OSYNC_EXECUTABLE "$i" $opts & $SUBPROGRAM_EXECUTABLE "$i" $opts &
wait $! wait $!
if [ $? != 0 ]; then if [ $? != 0 ]; then
Logger "Run instance $(basename $i) failed" "ERROR" Logger "Run instance $(basename $i) failed" "ERROR"
@ -124,37 +124,37 @@ function Usage {
echo $AUTHOR echo $AUTHOR
echo $CONTACT echo $CONTACT
echo "" echo ""
echo "Batch script to sequentially run osync instances and rerun failed ones." echo "Batch script to sequentially run osync or obackup instances and rerun failed ones."
echo "Usage: osync-batch.sh [OPTIONS]" echo "Usage: $SUBPROGRAM-batch.sh [OPTIONS]"
echo "" echo ""
echo "[OPTIONS]" echo "[OPTIONS]"
echo "--path=/path/to/conf Path to osync conf files, defaults to /etc/osync" echo "--path=/path/to/conf Path to osync / obackup conf files, defaults to /etc/osync or /etc/obackup"
echo "--max-reruns=X Number of runs max for failed instances, (defaults to 3)" echo "--max-reruns=X Number of runs max for failed instances, (defaults to 3)"
echo "--max-exec-time=X Retry failed instances only if max execution time not reached (defaults to 36000 seconds). Set to 0 to bypass execution time check." echo "--max-exec-time=X Retry failed instances only if max execution time not reached (defaults to 36000 seconds). Set to 0 to bypass execution time check."
echo "--no-maxtime Run osync without honoring conf file defined timeouts" echo "--no-maxtime Run osync / obackup without honoring conf file defined timeouts"
echo "--dry Will run osync without actually doing anything; just testing" echo "--dry Will run osync / obackup without actually doing anything; just testing"
echo "--silent Will run osync without any output to stdout, used for cron jobs" echo "--silent Will run osync / obackup without any output to stdout, used for cron jobs"
echo "--verbose Increases output" echo "--verbose Increases output"
exit 128 exit 128
} }
silent=0 _SILENT=0
dry=0 _DRY=0
verbose=0 _VERBOSE=0
opts="" opts=""
for i in "$@" for i in "$@"
do do
case $i in case $i in
--silent) --silent)
silent=1 _SILENT=1
opts=$opts" --silent" opts=$opts" --silent"
;; ;;
--dry) --dry)
dry=1 _DRY=1
opts=$opts" --dry" opts=$opts" --dry"
;; ;;
--verbose) --verbose)
verbose=1 _VERBOSE=1
opts=$opts" --verbose" opts=$opts" --verbose"
;; ;;
--no-maxtime) --no-maxtime)
@ -180,5 +180,5 @@ do
done done
CheckEnvironment CheckEnvironment
Logger "$(date) Osync batch run" "NOTICE" Logger "$(date) $SUBPROGRAM batch run" "NOTICE"
Batch Batch

1083
osync.sh
View File

File diff suppressed because it is too large Load Diff