Coding style used for my bash projects (v3.0 Dec 2016)
As bash is clearly an error prone script language, we'll use as much standard coding as possible, including some quick and dirty debug techniques described here.

++++++ Header

Always use the following header

----BEGIN HEADER
#!/usr/bin/env bash

PROGRAM="program-name" # Long description
AUTHOR="(C) 20XX-20YY by Orsiris de Jong"
CONTACT="http://www.example.com me@example.com"
PROGRAM_BUILD=YYYYMMDDVV

## Optional instructions
----END HEADER

Using bind style versionning:
YYYYMMDDVV (Year, Month, Day, Revision): Example: 2015012402 = 2nd revision of 24 Jan 2015

#!/usr/bin/env bash instead of #!/bin/bash

Change old scripts with
for i in $(grep -r '#!/bin/bash' * |cut -f1 -d':'); do sed -i 's&#!/bin/bash&#!/usr/bin/env bash&g' $i; done

type instead of type -p for bash test (other shells don't know -p)

++++++ Indentation

Using tabs
Transform old shell scripts using unexpand command

++++++ Comments

Some command # comment
## Some comment on a new line
################################################# Some separation

++++++ Work comments

Whenever there is some idea to postpone, use #TODO(priority):[dev-name:] some remark
Priority can be critical, high, medium, low, verylow. No release can happen if there are TODOs other than low or verylow.
Example: #TODO(high):deajan: need to do something

A "work in progress" marker must be left on the line a dev is working when it's work isn't finished). Marker is #WIP:dev-name: some remark
dev-name is mandatory if more than one person is coding
Example: #WIP:deajan: missing function something

++++++ Variables

All local variables names have each first letter of the words uppercase and all others lowercase, except for the first word where all letters are lowercase
Example: someLongVariable
All global variables are full upercase, separated by _
Example: EXEC_TIME
All environment variables (verbose, silent, debug, etc) have prefix _ and are full upercase, separated by _
Example: _PARANOIA_DEBUG

Exec time variables that can take boolean values should use true and false instead of 1 and 0.

++++++ Functions

All function names should begin with an uppercase letter for every word, the other letters should be lowercase
Example: SomeFunctionThatRocks

Bash does not provide any checks against missing function arguments. Also, missing quotes can lead to an inconsistent number of arguments.
Most functions should have a first line that calls the special function __CheckArguments, which checks the number of given arguments for a function in order
to find possible problems. Number of arguments are given as first argument to __CheckArguments. May be a number or a range, eg 0-2 if the function takes optional arguments.
__CheckArguments will only trigger when the script is launched with _PARANOIA_DEBUG=yes. Also, it will only exist in the debug version.
Use the following convention for function definition:

function SomeFunction {
	__CheckArguments 0 $# "$@"               #__WITH_PARANOIA_DEBUG
	...	
}

Use sed ':a;N;$!ba;s/\n{\n/ {\n/g' to convert functions that have opening brackets on a new line.

If the function has arguments, use local variable names that are more readable than $1...$n. Explain via comments what those variables contain if needed.
Declare arguments before launching __CheckArguments:

function AnotherFunction {
	local varName="${1}"
	local otherVarName="${2}" # This variable contains stuff
	__CheckArguments 2 $# "$@"		#__WITH_PARANOIA_DEBUG
	...
}

Functions should always have return status
function RandomFunction {
	...
	return $?
}

++++++ Sub functions

When a function is a subroutine of another function, it is called _SomethingAsSubFunction:
Example:

function _ApplyLocally
function _ApplyRemotely
function Apply

++++++ For and While statements

For and while statements will have the "do" part on the first line.
Example:

for i in "${var[@]}"; do
	...
done

while [ $i -eq 1 ]; do
	...
done

++++++ If statements

If statements will be fully written (word "if" must be used). then is written on the same line.
(Use sed ':a;N;$!ba;s/]\n\t*then/]; then/g' to convert files to this format... Replace "],new line, zero or more tabs, then" by "; then")
if [ something ]; then
	stuff
else
	other stuff
fi

++++++ Logging

A logging function is available that writes both to log file and stdout/stderr.
It has the following global variable modifiers:

_LOGGER_SILENT=true/false: disables any output to stdout/stderr
_LOGGER_VERBOSE=true/false: logs messages with log level VERBOSE
_LOGGER_ERR_ONLY=true/false: disables logging to log file and stdout/stderr except for CRITICAL, ERROR, WARN and ALWAYS log levels. 

The following log levels exist:

- PARANOIA_DEBUG: Only used by debugging functions themselves
- DEBUG: Only log this when _DEBUG flag is set in program. Any command forged for eval instruction should be logged by this.
- NOTICE: Standard messages
- ALWAYS: Standard messages, regardless of _LOGGER_ERR_ONLY
- WARN: Requires attention
- ERROR: Program produced an error but continues execution
- CRITICAL: Program execution is halted

Can be called with:
Logger "My message" "LOGLEVEL" $retval

$retval is an optional parameter that passes the exit code of the command that triggered the logging message
$retval, along with function stack, script pid and current pid can be found in the ERROR /WARN alert files ($RUN_DIR/$PROGRAM.Logger.error.$SCRIPT_PID)

++++++ Eval

Most commands should be logged to a tmp file.
The basic way of doing is:

cmd='"something '$somevar'" > some_file 2>&1'
eval $cmd &
WaitForTaskCompletion $! 0 0 1 0 true false true false
retval=$?
if [ $retval -ne 0 ]; then
	Logger "Some error message" "ERROR" $retval
fi

Remote commands should always invoke bash (using '"'"' to escape single quotes of 'bash -c "command"'). It is preferable to use ssh heredoc in order to use plain code.
If local and remote code is identical, wrap remote code in a function so only minor modifications are needed.
Remote code return code is transmitted via exit.

cmd=$SSH_CMD' '"'"'bash -c "some; commands \"'$VARIABLE'\" some; other; commands" > some_file'"'"' 2>&1'

Better formule

$SSH_CMD remoteVar="'$localVar'" 'bash -s' << 'ENDSSH' > 2>&1
function remoteSub {
	some code
	return 0
}

remoteSub
exit $?
ENDSSH
retval=$?
if [ $retval -ne 0 ]; then
	Logger "Some error message" "ERROR" $retval
fi

++++++ File variables

All eval cmd should exit their content to a file called "$RUNDIR/$PROGRAM.${FUNCNAME[0]}.$SCRIPT_PID"
Dots are used instead of '_' so variables can be separated with a forbidden char in variable names, so the separtors apply as wished.


++++++ String function calls

String returning functions should only be called this way in order to deal with spaces:
Quoting happens outside the function call.

echo "$(myStringFunction $myStringVar)"

++++++ ofunctions

As obackup and osync share alot of common functions, ofunctions.sh will host all shared code.
Dev programs n_osync.sh and n_obackup.sh will source ofunctions.sh
Release programs will still include ofunctions.sh in order to enhance ease of use.

Ofunctions are defined like:

#__FUNC:FunctionName
function FunctionName {

}
#__ENDFUNC

These functions are inserted into code that has placeholders like #__FUNC:FuncName

+++++++ includes

ofunctions parts can be directly included in shell scripts.
The parts that needs to be included must be contained within specific comments:

#### MyFunction SUBSET ####
function MyFunction {
	...
}
#### MyFunction SUBSET END ####

These can later be included in shell scripts with:


include #### MyFunction SUBSET ####

In order to have those includes parsed, we use bootstrap.sh to launch the original shell script.
Original shell script will not work because include is not a bash statement.
Include the following code into original shell script

include #### _OFUNCTIONS_BOOTSTRAP SUBSET ####
[ "$_OFUNCTIONS_BOOTSTRAP" != true ] && echo "Please use bootstrap.sh to load this dev version of $(basename $0)" && exit 1
+++++++ Exit codes

Normal exit code = 0
Run with errors exit code = 1
Run with warnings exit code = 2
Wrong shell exit code = 127
Usage function exit code = 128

+++++++ Detailled debugging

When launching the program with 'bash -x', add SLEEP_TIME=1 so wait functions won't spam output
Ex:

SLEEP_TIME=1 bash -x ./program.sh

++++++ Finding code errors

Before every release, shellcheck must be run
Also a grep -Eri "TODO|WIP" osync/* must be run in order to find potential release blockers

Use shellcheck.net now and then (ignore SC2086 in our case)

Use a low tech approach to find uneven number of quotes per line

tr -cd "'\n" < my_bash_file.sh | awk 'length%2==1 {print NR, $0}'
tr -cd "\"\n" < my_bash_file.sh | awk 'length%2==1 {print NR, $0}'