* ask for answer on same line

* added script to debootstrap Ubuntu Trusty
* added Dockerfile for Ubuntu Trusty kernel builder

.gitignore

@@ -1,4 +0,0 @@
-rkt/nginx/nginx.aci
-lxd/busybox-nonroot/busybox-nonroot.tar.xz
-
-docker/ubuntu-kernelbuilder/packages

docker/dmd/Dockerfile

@@ -1,18 +0,0 @@
-## dmd
-##
-## VERSION 2.072
-##
-## a minimal image installing the official .deb for dmd
-## the official image would instead compile and install too many dependencies
-##
-#
-
-## this image must be debootstrapped with build-wheezy.sh
-FROM gdm85/wheezy
-
-## install deps
-RUN export DEBIAN_FRONTEND=noninteractive && apt-get update -y && apt-get install libc6-dev gcc libcurl3 -y
-
-## install dmd+dub
-RUN export DEBIAN_FRONTEND=noninteractive && cd /tmp && wget --quiet http://downloads.dlang.org/releases/2.x/2.072.0/dmd_2.072.0-0_amd64.deb && \
-echo 'cd61bc9d5e511db5d35b3e1410c21f0d38941e038e06c6db2db18e73177efbae  dmd_2.072.0-0_amd64.deb' | sha256sum -c && dpkg -i dmd_2.072.0-0_amd64.deb && rm dmd_2.072.0-0_amd64.deb

docker/gitian-bitcoin-host/Dockerfile

@@ -3,26 +3,19 @@
 ## VERSION 0.1.0
 ##
 
-FROM gdm85/gitian-host-vms
+FROM gdm85/gitian-host
 
 MAINTAINER Giuseppe Mazzotta "gdm85@users.noreply.github.com"
 
-RUN aptitude install make -y
+RUN apt-get install -y patch
 
-COPY input-sources/ /home/debian/input-sources/
+ADD gitian.patch /home/debian/
-COPY bin/build-bitcoin.sh /home/debian/
-COPY bin/sign.sh /home/debian/
 
-## patch to not attempt removal of build directory (will be a volume)
+## patch to allow paralle creation of VMs
-COPY gitian-gbuild.patch /home/debian/
+RUN cd /home/debian/gitian-builder && patch -p1 < ../gitian.patch
-RUN cd gitian-builder && patch -p1 < ../gitian-gbuild.patch && rm ../gitian-gbuild.patch
 
-## patch to write asserts only (best for detached signatures)
+ADD input-sources/ /home/debian/input-sources/
-COPY gitian-gasserts.patch /home/debian/
+ADD build-bitcoin.sh /home/debian/
-RUN cd gitian-builder && cp bin/gsign bin/gasserts && patch -p1 < ../gitian-gasserts.patch && rm ../gitian-gasserts.patch
+ADD sign.sh /home/debian/
 
-WORKDIR /home/debian
+RUN chown -R debian.debian /home/debian/
-
-RUN mkdir gitian-builder/inputs
-
-RUN chown -R debian.debian input-sources build-bitcoin.sh sign.sh gitian-builder/inputs

docker/gitian-bitcoin-host/README.md

@@ -1,35 +1,24 @@
 Building bitcoin with a gitian-builder Docker container
 =======================================================
 
-This image allows automated Gitian builds of bitcoin core using a Docker container.
+This image allows automated gitian builds of bitcoin using a docker container.
+Before proceeding make sure you have created the necessary *gdm85/wheezy*, *gdm85/gitian-host* and *gdm85/gitian-host-vms* images, see [these instructions](../gitian-host/README.md).
 
-Prerequisites
--------------
-Before proceeding make sure you have created these necessary images:
-* gdm85/wheezy
-* gdm85/gitian-host
-* gdm85/gitian-host-vms
-
-Instructions to build them are available [here](../gitian-host/README.md).
-
-Image creation
----------------
 Afterwards you can create the *gdm85/gitian-bitcoin-host* image by running [scripts/create-gitian-bitcoin-host.sh](../scripts/create-gitian-bitcoin-host.sh).
 
-Bitcoin input sources
+NOTE: this image currently supports only building of bitcoin 0.9.1, but it can be easily adapted to build other versions.
----------------------
+You can submit the source lists for other versions as a patch or pull request, see directory [input-sources/](input-sources/) for currently available versions.
-Since version 0.10.0 it's no more needed to use [input-sources/](input-sources/).
 
-Do not forget to read also the [Preamble here](../gitian-host/README.md#preamble) to correctly use Gitian builder and provided scripts.
+Do not forget to read also the [Preamble here](../gitian-host/README.md#preamble] to correctly use Gitian builder and these provided scripts.
 
 Spawning a container
 --------------------
 
-A new container for Gitian bitcoin builds can be spawned with:
+You can spawn a new container for Gitian bitcoin builds with:
 
 - [scripts/spawn-gitian-bitcoin-host.sh](scripts/spawn-gitian-bitcoin-host.sh)
 
-This script will spawn a running Docker container and provide details about how to connect via SSH to the container, example:
+This script will create the running docker container and provide details about how to connect via SSH to the container, example:
 ```
 $ scripts/spawn-gitian-bitcoin-host.sh
 You can now SSH into container 3bc0d0611374ca4d4730fd5fb1067808b1bcfd072ec7cf029393a7fd99ec856e:
@@ -44,21 +33,24 @@ Preparing the gitian environment
 
 To initiate a gitian build of bitcoin you will run:
 
-- [./build-bitcoin.sh](bin/build-bitcoin.sh) 0.10.0rc1
+- [./build-bitcoin.sh](build-bitcoin.sh) 0.9.1
 
-Notice the parameter 0.10.0, that is the version we are going to build and must be available in [input-sources](input-sources/).
+Notice the parameter 0.9.1, that is the version we are going to build and must be available in [input-sources](input-sources/).
 
-[build-bitcoin.sh](bin/build-bitcoin.sh) is a script that will download & build all the dependencies and then bitcoin itself, for both i386 and amd64 Linux architectures.
+[build-bitcoin.sh](build-bitcoin.sh) is a script that will download & build all the dependencies and then bitcoin itself, for both i386 and amd64 Linux architectures.
 
 Signing
 -------
 
-In order to sign the build you can either import your private key in container's debian user gpg, or perform the signing externally.
+Once you have completed the build of bitcoin, you will be ready to perform the signing; before doing that you should verify that signatures are matching with those of [other developers](https://github.com/bitcoin/gitian.sigs) by peeking inside *~/gitian.sigs* of the running container.
+Only the out_manifest signatures do matter for this purpose.
 
-In this example we will cover the former case; run [~/sign.sh](bin/sign.sh) script and check that your generated assert file (in a subdirectory of ~/gitian.sigs) matches with those of [other developers](https://github.com/bitcoin/gitian.sigs).
+In order to sign you have to either put your private key in the container's *~/.gnupg* or perform the signing externally, at your option.
-**NOTE:** Only the out_manifest signatures do matter, not all the dependencies.
+
+If you have the private key in the container (also displayed by `gpg -K`), then you can use the [sign.sh](sign.sh) script that is already provided, otherwise
+run it (with failure) and then copy the *~/gitian.sigs~ directory to another machine to apply the GPG signature.
 
 Submitting your signature
 -------------------------
 
-If signatures do match, you can fork the [gitian sigs repo](https://github.com/bitcoin/gitian.sigs), add & commit your signatures and submit a pull request for inclusion.
+If everything went well, you can fork the [gitian sigs repo](https://github.com/bitcoin/gitian.sigs), commit your signatures and submit a pull request for inclusion.

docker/gitian-bitcoin-host/bin/build-bitcoin.sh

@@ -1,82 +0,0 @@
-#!/bin/bash
-## automatic Gitian build of bitcoin
-## @author gdm85
-## @version 0.3.1
-## see also https://github.com/gdm85/tenku/blob/master/docker/gitian-bitcoin-host/
-##
-#
-
-if [ $# -lt 2 ]; then
-	echo "Usage: build-bitcoin.sh commit linux [win] [osx] [...]" 1>&2
-	exit 1
-fi
-
-COMMIT="$1"
-shift
-## remaining parameters are OS targets to be build (e.g. win,osx,linux)
-
-CLONE="$HOME/bitcoin"
-
-function verlte() {
-    [  "$1" = "`echo -e "$1\n$2" | sort -V | head -n1`" ]
-}
-
-NPROC=$(nproc) && \
-cd gitian-builder && \
-mkdir -p inputs && \
-cd .. || exit $?
-
-if [ ! -d bitcoin ]; then
-	git clone https://github.com/bitcoin/bitcoin.git && \
-	cd bitcoin && \
-	git checkout $COMMIT && \
-	cd .. || exit $?
-fi
-
-## old logic using descriptors (only linux supported)
-if echo "$COMMIT" | grep ^v >/dev/null && ! verlte v0.10.0rc1 $COMMIT; then
-	## make sure only Linux is being built
-	if [[ ! $# -eq 1 && "$1" != "linux" ]]; then
-		echo "For versions before 0.10.0rc1, only Linux building is supported" 1>&2
-		exit 1
-	fi
-
-	VERSION=$(echo "$COMMIT" | awk '{ print substr($0, 2) }')
-
-	cd gitian-builder/inputs || exit $?
-	## get each dependency
-	## they are validated afterwards by gbuild
-	while read -r URL FNAME; do
-	        if [ -z "$URL" ]; then
-	                continue
-	        fi
-	        if [ ! -f $FNAME ]; then
-	                echo "echo 'Downloading $FNAME'"
-	                echo "wget -q --no-check-certificate '$URL' -O '$FNAME' || echo 'Failed to download $FNAME from $URL'"
-	        fi
-	done < ../../input-sources/${VERSION}-inputs.txt | parallel -j10 || exit $?
-
-	## verify that all sources are correct before continuing
-	md5sum -c < ../../input-sources/${VERSION}-inputs.md5 && \
-	DESCRIPTORS="$(<../input-sources/${VERSION}-descriptors.txt)" && \
-	cd .. || exit $?
-
-	for DESC in $DESCRIPTORS; do
-		./bin/gbuild -j$NPROC ../bitcoin/contrib/gitian-descriptors/${DESC}.yml && \
-		mv -v $(find build/out -type f -name '*gz' -o -name '*.zip') inputs/ || exit $?
-	done
-else
-	cd bitcoin/depends || exit $?
-	for DESC in $@; do
-		make download-${DESC} SOURCES_PATH="$HOME/gitian-builder/cache/common" || exit $?
-	done
-	cd ../.. || exit $?
-fi
-
-## proceed to build of each of the specified gitian descriptors
-cd gitian-builder || exit $?
-for DESC in $@; do
-	./bin/gbuild -j$NPROC --commit bitcoin=$COMMIT -u bitcoin=$CLONE "$CLONE/contrib/gitian-descriptors/gitian-${DESC}.yml" || exit $?
-done
-
-echo "Successfully built gitian-${DESC} at $COMMIT"

docker/gitian-bitcoin-host/bin/sign.sh

@@ -1,13 +0,0 @@
-#!/bin/bash
-
-if [ ! $# -eq 3 ]; then
-	echo "Usage: sign.sh version signer-id gitian-descriptor.yml" 1>&2
-	exit 1
-fi
-
-VERSION="$1"
-SIGNER="$2"
-DESC="$3"
-
-cd gitian-builder && \
-./bin/gsign --signer $SIGNER --release ${VERSION} --destination ../gitian.sigs/ "../bitcoin/contrib/gitian-descriptors/$DESC"

docker/gitian-bitcoin-host/build-bitcoin.sh

@@ -0,0 +1,38 @@
+#!/bin/bash
+
+if [[ ! $# -eq 1 ]]; then
+	echo "Please specify version" 1>&2
+	exit 1
+fi
+
+VERSION="$1"
+
+if [ ! -d bitcoin ]; then
+	git clone https://github.com/bitcoin/bitcoin.git || exit $?
+fi
+cd bitcoin && \
+git checkout v${VERSION} || exit $?
+
+cd ../gitian-builder && \
+mkdir -p inputs && cd inputs/ || exit $?
+
+## get each dependency
+## they are validated afterwards by gbuild
+while read -r URL FNAME; do
+	if [ -z "$URL" ]; then
+		continue
+	fi
+	wget --continue --no-check-certificate "$URL" -O "$FNAME" || exit $?
+done < ../../input-sources/${VERSION}.txt || exit $?
+
+## verify that all sources are correct before continuing
+md5sum -c < ../../input-sources/${VERSION}.txt.md5 || exit $?
+
+cd ..
+./bin/gbuild ../bitcoin/contrib/gitian-descriptors/boost-linux.yml || exit $?
+mv build/out/boost-*.zip inputs/
+./bin/gbuild ../bitcoin/contrib/gitian-descriptors/deps-linux.yml || exit $?
+mv build/out/bitcoin-deps-*.zip inputs/
+./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml || exit $?
+echo "Completed successfully."
+echo "The output files are in: gitian-builder/build/out/"

docker/gitian-bitcoin-host/gitian-gasserts.patch

@@ -1,8 +0,0 @@
---- a/bin/gasserts	2015-01-13 15:39:29.000000000 +0000
-+++ b/bin/gasserts	2015-01-14 13:52:49.287255738 +0000
-@@ -85,4 +85,3 @@
- File.open(assert_path, "w") do |io|
-   io.write result.to_yaml
- end
--system!("gpg --detach-sign -u \"#{signer}\" \"#{assert_path}\"")
-

docker/gitian-bitcoin-host/gitian-gbuild.patch

@@ -1,13 +0,0 @@
---- a/bin/gbuild	2015-01-14 10:25:27.275143892 +0100
-+++ b/bin/gbuild	2015-01-14 10:25:56.171144093 +0100
-@@ -182,8 +182,8 @@
- cache_dir = 'cache'
- enable_cache = build_desc["enable_cache"]
- 
--FileUtils.rm_rf(build_dir)
--FileUtils.mkdir(build_dir)
-+## build dir is cleaned up outside container running gitian-builder
-+FileUtils.mkdir_p(build_dir)
- FileUtils.mkdir_p(result_dir)
- 
- package_name = build_desc["name"] or raise "must supply name"

docker/gitian-bitcoin-host/gitian.patch

@@ -0,0 +1,21 @@
+diff --git a/bin/make-base-vm b/bin/make-base-vm
+index c1920f3..8a44f13 100755
+--- a/bin/make-base-vm
++++ b/bin/make-base-vm
+@@ -109,16 +109,3 @@ rm -rf $OUT
+ sudo vmbuilder kvm ubuntu --rootsize 10240 --arch=$ARCH --suite=$SUITE --addpkg=$addpkg --removepkg=$removepkg --ssh-key=var/id_dsa.pub --ssh-user-key=var/id_dsa.pub --mirror=$MIRROR --secu
+ mv $OUT/*.qcow2 $OUT.qcow2
+ rm -rf $OUT
+-
+-if [ $LXC = "1" ]; then
+-    #sudo debootstrap --include=$addpkg --arch=$ARCH $SUITE $OUT-root $MIRROR
+-    echo Extracting partition for lxc
+-    qemu-img convert $OUT.qcow2 $OUT.raw
+-    loop=`sudo kpartx -av $OUT.raw|sed -n '/loop.p1/{s/.*loop\(.\)p1.*/\1/;p}'`
+-    sudo cp --sparse=always /dev/mapper/loop${loop}p1 $OUT
+-    sudo chown $USER $OUT
+-    sudo kpartx -d /dev/loop$loop
+-    rm -f $OUT.raw
+-    # bootstrap-fixup is done in libexec/make-clean-vm
+-fi
+-

docker/gitian-bitcoin-host/input-sources/0.9.1-descriptors.txt

@@ -1,2 +0,0 @@
-boost-linux
-deps-linux

docker/gitian-bitcoin-host/input-sources/0.9.1.txt

@@ -2,6 +2,7 @@ http://miniupnp.free.fr/files/download.php?file=miniupnpc-1.8.tar.gz	miniupnpc-1
 https://www.openssl.org/source/openssl-1.0.1g.tar.gz	openssl-1.0.1g.tar.gz
 http://download.oracle.com/berkeley-db/db-4.8.30.NC.tar.gz	db-4.8.30.NC.tar.gz 
 http://zlib.net/zlib-1.2.8.tar.gz	zlib-1.2.8.tar.gz
+ftp://ftp.simplesystems.org/pub/png/src/history/libpng16/libpng-1.6.8.tar.gz	libpng-1.6.8.tar.gz
 https://fukuchi.org/works/qrencode/qrencode-3.4.3.tar.bz2	qrencode-3.4.3.tar.bz2
 https://downloads.sourceforge.net/project/boost/boost/1.55.0/boost_1_55_0.tar.bz2	boost_1_55_0.tar.bz2
 https://download.qt-project.org/official_releases/qt/5.2/5.2.0/single/qt-everywhere-opensource-src-5.2.0.tar.gz	qt-everywhere-opensource-src-5.2.0.tar.gz

docker/gitian-bitcoin-host/input-sources/0.9.1.txt.md5

@@ -1,5 +1,6 @@
 d6eef4b4cacb2183f2bf265a5a03a354  boost_1_55_0.tar.bz2
 a14a5486d6b4891d2434039a0ed4c5b7  db-4.8.30.NC.tar.gz
+29b7065906e2551508a0d7eacd19174e  libpng-1.6.8.tar.gz
 065bf20a20ebe605c675b7a5aaef340a  miniupnpc-1.8.tar.gz
 de62b43dfcd858e66a74bee1c834e959  openssl-1.0.1g.tar.gz
 a72001a9067a4c2c4e0e836d0f92ece4  protobuf-2.5.0.tar.bz2

docker/gitian-bitcoin-host/input-sources/0.9.2-descriptors.txt

@@ -1,3 +0,0 @@
-boost-linux
-deps-linux
-qt-linux

docker/gitian-bitcoin-host/input-sources/0.9.2-inputs.md5

@@ -1,11 +0,0 @@
-d6eef4b4cacb2183f2bf265a5a03a354  boost_1_55_0.tar.bz2
-efe08e2f3ca478486037b053acd512e9  cdrkit-1.1.11.tar.gz
-cea2d01b3206e92a8df7b079935c070b  clang-llvm-3.2-x86-linux-ubuntu-12.04.tar.gz
-a14a5486d6b4891d2434039a0ed4c5b7  db-4.8.30.NC.tar.gz
-5ef3ba321e6df72d6519b728b292073e  miniupnpc-1.9.tar.gz
-8d6d684a9430d5cc98a62a5d8fbda8cf  openssl-1.0.1h.tar.gz
-a72001a9067a4c2c4e0e836d0f92ece4  protobuf-2.5.0.tar.bz2
-89a90a3b8cbca60ac412b2e0e0c776e7  qrencode-3.4.3.tar.bz2
-8ac880cc07a130c39607b65efd5e1421  qt-everywhere-opensource-src-4.6.4.tar.gz
-228b6384dfd7272de00fd8b2c144fecd  qt-everywhere-opensource-src-5.2.0.tar.gz
-44d667c142d7cda120332623eab69f40  zlib-1.2.8.tar.gz

docker/gitian-bitcoin-host/input-sources/0.9.2-inputs.txt

@@ -1,11 +0,0 @@
-http://miniupnp.free.fr/files/download.php?file=miniupnpc-1.9.tar.gz    miniupnpc-1.9.tar.gz                                                                                                  
-https://www.openssl.org/source/openssl-1.0.1h.tar.gz    openssl-1.0.1h.tar.gz
-http://download.oracle.com/berkeley-db/db-4.8.30.NC.tar.gz      db-4.8.30.NC.tar.gz
-http://zlib.net/zlib-1.2.8.tar.gz       zlib-1.2.8.tar.gz
-https://fukuchi.org/works/qrencode/qrencode-3.4.3.tar.bz2       qrencode-3.4.3.tar.bz2
-https://downloads.sourceforge.net/project/boost/boost/1.55.0/boost_1_55_0.tar.bz2       boost_1_55_0.tar.bz2
-https://download.qt-project.org/official_releases/qt/5.2/5.2.0/single/qt-everywhere-opensource-src-5.2.0.tar.gz qt-everywhere-opensource-src-5.2.0.tar.gz
-https://download.qt-project.org/archive/qt/4.6/qt-everywhere-opensource-src-4.6.4.tar.gz        qt-everywhere-opensource-src-4.6.4.tar.gz
-https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.bz2    protobuf-2.5.0.tar.bz2
-http://cdrkit.org/releases/cdrkit-1.1.11.tar.gz cdrkit-1.1.11.tar.gz
-http://llvm.org/releases/3.2/clang+llvm-3.2-x86-linux-ubuntu-12.04.tar.gz     clang-llvm-3.2-x86-linux-ubuntu-12.04.tar.gz

docker/gitian-bitcoin-host/input-sources/0.9.2rc2-descriptors.txt

@@ -1,3 +0,0 @@
-boost-linux
-deps-linux
-qt-linux

docker/gitian-bitcoin-host/input-sources/0.9.2rc2-inputs.md5

@@ -1,11 +0,0 @@
-d6eef4b4cacb2183f2bf265a5a03a354  boost_1_55_0.tar.bz2
-efe08e2f3ca478486037b053acd512e9  cdrkit-1.1.11.tar.gz
-cea2d01b3206e92a8df7b079935c070b  clang-llvm-3.2-x86-linux-ubuntu-12.04.tar.gz
-a14a5486d6b4891d2434039a0ed4c5b7  db-4.8.30.NC.tar.gz
-5ef3ba321e6df72d6519b728b292073e  miniupnpc-1.9.tar.gz
-8d6d684a9430d5cc98a62a5d8fbda8cf  openssl-1.0.1h.tar.gz
-a72001a9067a4c2c4e0e836d0f92ece4  protobuf-2.5.0.tar.bz2
-89a90a3b8cbca60ac412b2e0e0c776e7  qrencode-3.4.3.tar.bz2
-8ac880cc07a130c39607b65efd5e1421  qt-everywhere-opensource-src-4.6.4.tar.gz
-228b6384dfd7272de00fd8b2c144fecd  qt-everywhere-opensource-src-5.2.0.tar.gz
-44d667c142d7cda120332623eab69f40  zlib-1.2.8.tar.gz

docker/gitian-bitcoin-host/input-sources/0.9.2rc2-inputs.txt

@@ -1,11 +0,0 @@
-http://miniupnp.free.fr/files/download.php?file=miniupnpc-1.9.tar.gz    miniupnpc-1.9.tar.gz                                                                                                  
-https://www.openssl.org/source/openssl-1.0.1h.tar.gz    openssl-1.0.1h.tar.gz
-http://download.oracle.com/berkeley-db/db-4.8.30.NC.tar.gz      db-4.8.30.NC.tar.gz
-http://zlib.net/zlib-1.2.8.tar.gz       zlib-1.2.8.tar.gz
-https://fukuchi.org/works/qrencode/qrencode-3.4.3.tar.bz2       qrencode-3.4.3.tar.bz2
-https://downloads.sourceforge.net/project/boost/boost/1.55.0/boost_1_55_0.tar.bz2       boost_1_55_0.tar.bz2
-https://download.qt-project.org/official_releases/qt/5.2/5.2.0/single/qt-everywhere-opensource-src-5.2.0.tar.gz qt-everywhere-opensource-src-5.2.0.tar.gz
-https://download.qt-project.org/archive/qt/4.6/qt-everywhere-opensource-src-4.6.4.tar.gz        qt-everywhere-opensource-src-4.6.4.tar.gz
-https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.bz2    protobuf-2.5.0.tar.bz2
-http://cdrkit.org/releases/cdrkit-1.1.11.tar.gz cdrkit-1.1.11.tar.gz
-http://llvm.org/releases/3.2/clang+llvm-3.2-x86-linux-ubuntu-12.04.tar.gz     clang-llvm-3.2-x86-linux-ubuntu-12.04.tar.gz

docker/gitian-bitcoin-host/input-sources/0.9.3rc2-descriptors.txt

@@ -1,3 +0,0 @@
-boost-linux
-deps-linux
-qt-linux

docker/gitian-bitcoin-host/input-sources/0.9.3rc2-inputs.md5

@@ -1,10 +0,0 @@
-d6eef4b4cacb2183f2bf265a5a03a354  boost_1_55_0.tar.bz2
-cea2d01b3206e92a8df7b079935c070b  clang-llvm-3.2-x86-linux-ubuntu-12.04.tar.gz
-a14a5486d6b4891d2434039a0ed4c5b7  db-4.8.30.NC.tar.gz
-56b4ac6a7a3e7d64f8cfeeac87b212bc  miniupnpc-1.9.20140701.tar.gz
-c8dc151a671b9b92ff3e4c118b174972  openssl-1.0.1i.tar.gz
-a72001a9067a4c2c4e0e836d0f92ece4  protobuf-2.5.0.tar.bz2
-89a90a3b8cbca60ac412b2e0e0c776e7  qrencode-3.4.3.tar.bz2
-8ac880cc07a130c39607b65efd5e1421  qt-everywhere-opensource-src-4.6.4.tar.gz
-228b6384dfd7272de00fd8b2c144fecd  qt-everywhere-opensource-src-5.2.0.tar.gz
-44d667c142d7cda120332623eab69f40  zlib-1.2.8.tar.gz

docker/gitian-bitcoin-host/input-sources/0.9.3rc2-inputs.txt

@@ -1,10 +0,0 @@
-http://miniupnp.free.fr/files/download.php?file=miniupnpc-1.9.20140701.tar.gz    miniupnpc-1.9.20140701.tar.gz
-https://www.openssl.org/source/openssl-1.0.1i.tar.gz    openssl-1.0.1i.tar.gz
-http://download.oracle.com/berkeley-db/db-4.8.30.NC.tar.gz      db-4.8.30.NC.tar.gz
-http://zlib.net/zlib-1.2.8.tar.gz       zlib-1.2.8.tar.gz
-https://fukuchi.org/works/qrencode/qrencode-3.4.3.tar.bz2       qrencode-3.4.3.tar.bz2
-https://downloads.sourceforge.net/project/boost/boost/1.55.0/boost_1_55_0.tar.bz2       boost_1_55_0.tar.bz2
-https://download.qt-project.org/official_releases/qt/5.2/5.2.0/single/qt-everywhere-opensource-src-5.2.0.tar.gz qt-everywhere-opensource-src-5.2.0.tar.gz
-https://download.qt-project.org/archive/qt/4.6/qt-everywhere-opensource-src-4.6.4.tar.gz        qt-everywhere-opensource-src-4.6.4.tar.gz
-https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.bz2    protobuf-2.5.0.tar.bz2
-http://llvm.org/releases/3.2/clang+llvm-3.2-x86-linux-ubuntu-12.04.tar.gz     clang-llvm-3.2-x86-linux-ubuntu-12.04.tar.gz

docker/gitian-bitcoin-host/sign.sh

@@ -0,0 +1,13 @@
+-#!/bin/bash
+set -e
+
+if [[ ! $# -eq 2 ]]; then
+	echo "Please specify version and signer id" 1>&2
+	exit 1
+fi
+
+VERSION="$1"
+SIGNER="$2"
+
+cd gitian-builder
+./bin/gsign --signer $SIGNER --release ${VERSION} --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml

docker/gitian-host/Dockerfile

@@ -12,19 +12,20 @@ FROM gdm85/wheezy
 
 MAINTAINER Giuseppe Mazzotta "gdm85@users.noreply.github.com"
 
+ENV     TERM xterm
+ENV     DEBIAN_FRONTEND noninteractive
+
 ## some core packages
-RUN apt-get update && apt-get install -y aptitude
+RUN apt-get update && apt-get install -y htop tmux less locales openssh-server
-RUN aptitude update && aptitude safe-upgrade -y
-RUN aptitude install htop tmux less locales openssh-server -y
 
 ## these 2 files can be customized
-COPY etc/timezone /etc/
+ADD etc/timezone /etc/timezone
-COPY etc/locale.gen /etc/
+ADD etc/locale.gen /etc/locale.gen
 RUN dpkg-reconfigure locales && dpkg-reconfigure tzdata
 
 ## continue installation of gitian-builder packages
 RUN echo "deb http://http.debian.net/debian wheezy main contrib non-free" > /etc/apt/sources.list.d/non-free.list
-RUN apt-get install -y iptables ca-certificates sudo bridge-utils git python-cheetah qemu-utils parted kpartx ruby apt-cacher-ng lxc parallel patch
+RUN apt-get install -y iptables ca-certificates sudo bridge-utils git python-cheetah qemu-utils parted kpartx ruby apt-cacher-ng lxc parallel
 
 ## disable password logins, reverse DNS lookups & privilege separation (due to chroot issues on latest docker)
 RUN sed -i -e 's/^#PasswordAuthentication yes/PasswordAuthentication no/' -e 's/^UsePrivilegeSeparation.*$/UsePrivilegeSeparation no/' /etc/ssh/sshd_config && echo 'UseDNS no' >> /etc/ssh/sshd_config
@@ -41,7 +42,7 @@ RUN useradd -m -s /bin/bash debian && adduser debian sudo && mkdir /home/debian/
 ## provide authorized keys to allow connections inside host
 ## NOTE: this is placed in debian user directory by setup.sh script
 ## reason is that there is a Docker glitch with file permissions when adding this
-COPY authorized_keys /root/
+ADD authorized_keys /root/
 
 WORKDIR /home/debian
 
@@ -50,17 +51,12 @@ RUN wget http://archive.ubuntu.com/ubuntu/pool/universe/v/vm-builder/vm-builder_
 RUN tar -zxvf vm-builder_0.12.4+bzr489.orig.tar.gz && cd vm-builder-0.12.4+bzr489 && python setup.py install
 
 ## these must always be loaded when you login as 'debian' user
-COPY bash_profile /home/debian/.bash_profile
+ADD bash_profile /home/debian/.bash_profile
 
 ## script used to build base VMs
-COPY bin/build-base-vms.sh /home/debian/
+ADD build-base-vms.sh /home/debian/
 
-RUN git clone https://github.com/devrandom/gitian-builder.git && cd gitian-builder && git checkout  ad3f9cc4c2c8c0899961a366f5b9fbd1483b0ee3
+RUN git clone https://github.com/devrandom/gitian-builder.git --depth=1
-
-COPY gitian.patch /home/debian/
-
-## patch to allow paralle creation of VMs
-RUN cd gitian-builder && patch -p1 < ../gitian.patch && rm ../gitian.patch
 RUN chown -R debian.debian .bash_profile . && chown -R apt-cacher-ng.apt-cacher-ng /var/cache/apt-cacher-ng/
 
 ## suggested: run this image with mounted volumes for pre-downloaded inputs and apt-cacher-ng cache
@@ -68,7 +64,7 @@ RUN chown -R debian.debian .bash_profile . && chown -R apt-cacher-ng.apt-cacher-
 ##NOTE: if you do use them, then not forget to set proper user rights for these mounted volumes
 
 ## all necessary setup instructions (SSH, LXC)
-COPY bin/setup.sh /usr/local/bin/setup.sh
+ADD setup.sh /usr/local/bin/setup.sh
 
 ## on run failure, check logs to see if setup failed. on success connect via SSH
 CMD /usr/local/bin/setup.sh && /etc/init.d/apt-cacher-ng start && /usr/sbin/sshd -D -e

docker/gitian-host/README.md

@@ -11,10 +11,11 @@ Some of the discussions leading to the creation of this set of Dockerfiles/scrip
 Preamble
 --------
 
-It is **necessary** to read the provided scripts and understand what they do before starting to use them.
+It is **necessary** that before you using these scripts you read them and understand what they do.
-The reason is that aiming at the goal of creating a Gitian build (deterministic) also means that your build is untampered, thus trust shall be correctly attributed during the process.
+Why? Because your goal is to create a gitian build (deterministic) that has not been tampered with, thus trust shall be correctly attributed during your process.
 
-For example, in this repository I provide the [Debian Archive keyring](../keyrings/debian-archive-keyring.gpg) that is used for the original debootstrap, however you **must** verify its authenticity and that it is exactly [as provided officially by Debian](https://packages.debian.org/wheezy/all/debian-archive-keyring/download)
+For example, in this repository I provide the [Debian Archive keyring](../keyrings/debian-archive-keyring.gpg) that is used for the original debootstrap,
+however you **must** verify its authenticity and that it is exactly [as provided officially by Debian](https://packages.debian.org/wheezy/all/debian-archive-keyring/download)
 in order to continue using a trusted chain of systems.
 
 See also:
@@ -28,19 +29,19 @@ See also:
 
 How to build the image
 ----------------------
-Images have not been pushed to my [Docker Registry](https://index.docker.io/) account, this is on purpose because even if generated images have my repository prefix ('gdm85/') you are supposed to create them from scratch with the provided (auditable) scripts.
+Images have not been pushed images to my [Docker Registry](https://index.docker.io/) account, this is on purpose because even if generated images have my repository prefix ('gdm85/') you are supposed
+to create them from scratch.
 
-**NOTE:** you must have *debootstrap* on your real host to run this script successfully, and also make sure you have a keyring with APT keys, see also https://wiki.debian.org/SecureApt
+**NOTE:** you must have debootstrap on your real host to run this script successfully, and also make sure you have a keyring with APT keys, see also https://wiki.debian.org/SecureApt
 
 First steps:
-- run [scripts/build-wheezy.sh](../scripts/build-wheezy.sh) to get a Debian Wheezy image debootstrapped from Debian repositories.
+- run **scripts/build-wheezy.sh** to get a Debian Wheezy image debootstrapped from Debian repositories.
-- run [scripts/create-gitian-host.sh](../scripts/create-gitian-host.sh), this will simply build the Dockerfile that installs the few necessary dependencies inside the prepared image
+- run **scripts/create-gitian-host.sh**, this will simply build the Dockerfile that installs the few necessary dependencies inside the prepared image, plus generate a second image with the i386 and amd64 VMs (see [build-base-vms.sh](build-base-vms.sh)).
-- generate a second image with the i386 and amd64 VMs byu using [build-base-vms.sh](../scripts/build-base-vms.sh)
 
-**NOTE:** when I say "run", what I really mean is "read the script, study it for your own learning purposes, then run it"
+**NOTE:** when I say "run", what I really mean is "read the script, study it for your own learning purposes, then run it" ;)
 
-After steps above you will have prepared a full Gitian builder environment for deterministic builds.
+After steps above you will have prepared a full gitian builder environment for deterministic builds.
-The image that contains the VMs is called *gdm85/gitian-host-vms*; in future you can spawn containers with this image for new Gitian builder environments.
+The image that contains the VMs is called *gdm85/gitian-host-vms*; in future you can spawn containers with this image for new gitian-builder environments.
 
 Example:
 ```
@@ -53,7 +54,7 @@ This will create a privileged running container that you can access with the SSH
 
 Derived images
 --------------
-A [bitcoin gitian host container](../gitian-bitcoin-host/README.md) is available.
+A [bitcoin gitian host container](../gitian-bitcoin/host/README.md) is available.
 
 Credits
 -------

docker/gitian-host/bash_profile

@@ -1,3 +1,10 @@
 export USE_LXC=1
 export GITIAN_HOST_IP=10.0.3.2
 export LXC_GUEST_IP=10.0.3.5
+
+## see bugs https://github.com/devrandom/gitian-builder/issues/56 and https://bugzilla.mindrot.org/show_bug.cgi?id=1285
+for ENV in $(env | grep ^LC | awk -F= '{ print $1 }'); do
+        unset $ENV
+done
+
+export LANG=en_US.UTF-8

docker/gitian-host/bin/build-base-vms.sh

@@ -1,65 +0,0 @@
-#!/bin/bash
-
-if [ $# -eq 0 ]; then
-	echo "Please specify: [i386|amd64]" 1>&2
-	exit 1
-fi
-
-if [ -z "$USE_LXC" ]; then
-	echo "Environment variables not correctly setup (source .bash_profile?)" 1>&2
-	exit 2
-fi
-
-export MIRROR_HOST=$GITIAN_HOST_IP
-SUITE=precise
-
-cd gitian-builder && \
-mkdir -p var || exit $?
-
-if [ ! -e var/id_dsa ]; then
-  ssh-keygen -t dsa -f var/id_dsa -N ""
-fi
-
-## build both VMs in parallel
-for TYPE in "$@"; do
-	echo -e "MIRROR_HOST=$GITIAN_HOST_IP bin/make-base-vm --lxc --arch $TYPE --suite $SUITE"
-done | parallel -j$# || exit $?
-
-function retry_remove() {
-	local RETRIES="$1"
-	local LOOP="$2"
-	while ! sudo losetup -d "/dev/loop${LOOP}" 2>/dev/null; do
-		let RETRIES-=1
-		if [ $RETRIES -eq 0 ]; then
-			echo "Failed removing /dev/loop${LOOP}" 1>&2
-			return 1
-		fi
-		sleep 1
-	done
-	sudo unlink /dev/mapper/loop${LOOP}p1 2>/dev/null
-}
-
-## this function corresponds to part removed from gbuild via custom patch
-function ext_partition() {
-	local loop
-	local OUT=$1
-	echo "Converting $OUT to raw format..." && \
-	qemu-img convert $OUT.qcow2 $OUT.raw && \
-	echo -n "Identifying partition..." && \
-	set -o pipefail && \
- 	loop=`sudo kpartx -av $OUT.raw | sed -n '/loop.p1/{s/.*loop\(.\)p1.*/\1/;p}'` && \
-	echo ": $loop" && \
-	echo "Copying partition to $OUT..." && \
-	sudo cp --sparse=always /dev/mapper/loop${loop}p1 $OUT && \
-	sudo chown $USER $OUT || return $?
-	## these are silenced because if former fails, second doesn't and viceversa
-	echo "Removing partition loop mount..." && \
-	retry_remove 5 "$loop" && \
-	echo "Removing raw image..." && \
-	rm -f "$OUT.raw" && \
-	echo "$OUT correctly extracted"
-}
-
-for TYPE in "$@"; do
-	ext_partition base-${SUITE}-${TYPE} || exit $?
-done

docker/gitian-host/build-base-vms.sh

@@ -0,0 +1,36 @@
+#!/bin/bash
+set -e
+
+
+source ~/.bash_profile
+
+cd gitian-builder
+mkdir -p var
+if [ ! -e var/id_dsa ]; then
+  ssh-keygen -t dsa -f var/id_dsa -N ""
+fi
+
+export MIRROR_HOST=$GITIAN_HOST_IP
+SUITE=precise
+
+## build both VMs in parallel
+echo -e "MIRROR_HOST=$GITIAN_HOST_IP bin/make-base-vm --lxc --arch i386 --suite $SUITE\nMIRROR_HOST=$GITIAN_HOST_IP bin/make-base-vm --lxc --arch amd64 --suite $SUITE" | parallel -j2 || exit $?
+
+function ext_partition() {
+	local OUT=$1
+	echo Extracting $OUT partition for lxc
+	qemu-img convert $OUT.qcow2 $OUT.raw
+ 	loop=`sudo kpartx -av $OUT.raw|sed -n '/loop.p1/{s/.*loop\(.\)p1.*/\1/;p}'`
+	sudo cp --sparse=always /dev/mapper/loop${loop}p1 $OUT
+	sudo chown $USER $OUT
+	## following 2 lines are a sloppy hack to an unknown problem with kpartx
+	sudo sync
+	sleep 5
+	## these are silenced because if former fails, second doesn't and viceversa
+	sudo kpartx -d /dev/loop$loop 2>/dev/null
+	sudo rm /dev/mapper/loop${loop}p1 2>/dev/null
+	rm -f $OUT.raw
+}
+
+ext_partition base-${SUITE}-i386 && \
+ext_partition base-${SUITE}-amd64 || exit $?

docker/gitian-host/etc/locale.gen

@@ -124,7 +124,7 @@
 # en_DK.UTF-8 UTF-8
 # en_GB ISO-8859-1
 # en_GB.ISO-8859-15 ISO-8859-15
-# en_GB.UTF-8 UTF-8
+en_GB.UTF-8 UTF-8
 # en_HK ISO-8859-1
 # en_HK.UTF-8 UTF-8
 # en_IE ISO-8859-1
@@ -322,7 +322,7 @@ en_US.UTF-8 UTF-8
 # nl_BE.UTF-8 UTF-8
 # nl_BE@euro ISO-8859-15
 # nl_NL ISO-8859-1
-# nl_NL.UTF-8 UTF-8
+nl_NL.UTF-8 UTF-8
 # nl_NL@euro ISO-8859-15
 # nn_NO ISO-8859-1
 # nn_NO.UTF-8 UTF-8

docker/gitian-host/gitian.patch

@@ -1,22 +0,0 @@
-diff --git a/bin/make-base-vm b/bin/make-base-vm
-index a7c182b..8dc4091 100755
---- a/bin/make-base-vm
-+++ b/bin/make-base-vm
-@@ -227,5 +227,17 @@ else
-   mv $OUT/*.qcow2 $OUT.qcow2
-   rm -rf $OUT
-   # bootstrap-fixup is done on first boot
-+
-+  if [ $LXC = "1" ]; then
-+      #sudo debootstrap --include=$addpkg --arch=$ARCH $SUITE $OUT-root $MIRROR
-+      echo Extracting partition for lxc
-+      qemu-img convert $OUT.qcow2 $OUT.raw
-+      loop=`sudo kpartx -av $OUT.raw|sed -n '/loop.p1/{s/.*loop\(.\)p1.*/\1/;p}'`
-+      sudo cp --sparse=always /dev/mapper/loop${loop}p1 $OUT
-+      sudo chown $USER $OUT
-+      sudo kpartx -d /dev/loop$loop
-+      rm -f $OUT.raw
-+      # bootstrap-fixup is done in libexec/make-clean-vm
-+  fi
- fi
- 

docker/scripts/bitcoin-gitian-build.sh

@@ -1,179 +0,0 @@
-#!/bin/bash
-## bitcoin-gitian-build.sh
-##
-## @author gdm85
-##
-## Automatically build latest version of Bitcoin Core using
-## Docker containers (nested LXC).
-##
-## User can specify target operative systems as arguments.
-## Several optional environment variables condition the build:
-## - OUTPUTDIR - where input/output volume directories will be read/created
-## - SIGNER - id of signer (no signature will be attempted, just directory structure created)
-## - COMMIT - commit/branch to use for build, by default is latest tag
-## - NOPURGE - set to non-empty to not dispose containers after build
-#
-
-SCRIPTS=$(dirname $(readlink -m $0)) || exit $?
-
-## place this file in script's directory in order to build for Mac OS X
-SDK=MacOSX10.7.sdk.tar.gz
-
-if [ $# -lt 1 ]; then
-	echo "Usage: gitian-build.sh linux [win] [osx] [...]" 1>&2
-	exit 1
-fi
-
-if docker info 2>/dev/null | grep ^Storage | grep aufs$ >/dev/null; then
-	echo "You are using AUFS as Docker storage drive, which is terribly broken and not supported by this script." 1>&2
-	exit 1
-fi
-
-## identify a CLI tool to run commands in parallel
-## coshell is preferred
-PARALLEL=""
-if type coshell 2>/dev/null >/dev/null; then
-	PARALLEL="coshell"
-else
-	if type parallel 2>/dev/null >/dev/null; then
-		PARALLEL="parallel -j$#"
-	else
-		echo "Please install coshell (https://github.com/gdm85/coshell) or GNU Parallel (https://www.gnu.org/software/parallel/)" 1>&2
-		exit 2
-	fi
-fi
-
-function read_commit() {
-	local SHA="$1"
-	local OUTPUT
-	set -o pipefail && \
-	OUTPUT=$(curl -s https://api.github.com/repos/bitcoin/bitcoin/commits/${SHA} | jq -r '.sha') && \
-	test ! -z "$OUTPUT" && \
-	test "$OUTPUT" != "null" && \
-	echo "$OUTPUT"
-}
-
-## run all necessary containers, detached
-## setup proper volumes for input/output collection
-function run_all() {
-	local OS
-
-	for OS in "$@"; do
-		mkdir -p "$LSOURCE/${OS}" && \
-		rm -rf "$LDEST/${OS}" && \
-		mkdir -p "$LDEST/${OS}" || return $?
-	done
-	mkdir -p "$LSIGS" && \
-	mkdir -p "$LSOURCE" && \
-	mkdir -p "$LRESULT" && \
-	chown -R 1000.1000 "$LDEST" "$LSOURCE" "$LSIGS" "$LRESULT" || return $?
-
-	for OS in "$@"; do
-		echo "docker run -d --privileged -v $LRESULT:$CRESULT -v $LSIGS:$CSIGS -v $LSOURCE/${OS}:${CSOURCE} -v $LDEST/${OS}:$CDEST gdm85/gitian-bitcoin-host" || return $?
-	done | $PARALLEL
-}
-
-function inject_mac_sdk() {
-	local CID="$1"
-
-	docker-inject "$CID" "$SCRIPTS/$SDK" /home/debian/gitian-builder/inputs/
-}
-
-function build_all() {
-	local ALL=($@)
-	local COUNT=$#
-	local LEN=$((COUNT/2))
-	local CREATED=(${ALL[@]:0:$LEN})
-	local OSES=(${ALL[@]:$LEN})
-	local CID
-	local OS
-
-	local I=0
-	for CID in "${CREATED[@]}"; do
-		OS=${OSES[$I]}
-
-		if [[ "$OS" == "osx" ]]; then
-			inject_mac_sdk "$CID" || return $?
-		fi
-		let I+=1
-	done
-
-	I=0
-	for CID in "${CREATED[@]}"; do
-		OS=${OSES[$I]}
-		local OS_LOG_FILE="$LLOGS/build-${OS}.log"
-		echo "Execution log for ${OS} ({$HCOMMIT}) --> $OS_LOG_FILE" 1>&2
-
-		echo -n "docker exec $CID su -c 'cd /home/debian && source .bash_profile && ./build-bitcoin.sh $COMMIT ${OS} && " && \
-		echo -n "cd gitian-builder && ./bin/gasserts --signer $SIGNER --release ${HCOMMIT} --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-${OS}.yml' debian " && \
-		echo    " >> $OS_LOG_FILE 2>&1"
-		let I+=1
-	done | $PARALLEL
-}
-
-## change the assert directory as desired
-if [ -z "$SIGNER" ]; then
-	SIGNER="$USER"
-fi
-
-## customize output volumes
-if [ -z "$OUTPUTDIR" ]; then
-	OUTPUTDIR="$SCRIPTS/output"
-fi
-
-set -o pipefail || exit $?
-
-## always get latest release/rc if no commit environment was specified
-if [ ! -z "$COMMIT" ]; then
-	HCOMMIT="$COMMIT"
-else
-	HCOMMIT="$(curl -s https://api.github.com/repos/bitcoin/bitcoin/tags | jq -r '.[0].name')" || exit $?
-fi
-
-## get commit short hash
-## NOTE: this overwrites environment provided by user
-COMMIT=$(read_commit "$HCOMMIT") || exit $?
-
-###
-### declarations for input/output data volumes
-###
-
-## always add human readable commit and commit to volume path variables
-REL_OD="$OUTPUTDIR/${HCOMMIT}-${COMMIT}"
-LRESULT="${REL_OD}/result-${HCOMMIT}-${COMMIT}"
-LSIGS="${REL_OD}/sigs"
-LDEST="${REL_OD}/built"
-LLOGS="${REL_OD}"
-## depends-cache does not sport human readable prefix, being the only input volume for containers
-LSOURCE="${OUTPUTDIR}/${COMMIT}/depends-cache"
-
-## path of above volumes inside the containers
-CRESULT="/home/debian/gitian-builder/result"
-CSIGS="/home/debian/gitian.sigs"
-CSOURCE="/home/debian/gitian-builder/cache"
-CDEST="/home/debian/gitian-builder/build"
-
-## ---------------- main -------------------- ##
-
-CREATED="$(run_all $@ | tr '\n' ' ')" && \
-echo "Building bitcoin (${HCOMMIT}) for $@" && \
-build_all ${CREATED[@]} $@
-RV=$?
-
-if [ -z "$NOPURGE" ]; then
-	## cleanup
-	#echo "Cleaning up created containers..."
-	for CID in $CREATED; do
-		docker stop $CID
-		docker rm $CID
-	done
-fi
-
-## return build exit code
-if [ $RV -eq 0 ]; then
-	echo -n "Completed successfully "
-else
-	echo -n "Failed "
-fi
-echo "with exit code = $RV"
-exit $RV

docker/scripts/build-macos121-wily-kernel.sh

@@ -1,29 +0,0 @@
-#!/bin/bash
-## build-macos121-wily-kernel.sh
-##
-## @author gdm85
-##
-## Build a kernel with fixed bluetooth support for Mac OS 12,1 and Ubuntu Wily
-## Based on instructions read from http://www.spinics.net/lists/linux-bluetooth/msg64123.html
-##
-#
-
-SCRIPTS=$(dirname $(readlink -m $0)) || exit $?
-
-set -e
-
-cd "$SCRIPTS"
-
-if ! docker inspect gdm85/wily >/dev/null 2>/dev/null; then
-	./build-ubuntu-image.sh wily
-fi
-
-cd ../ubuntu-pkgbuilder
-
-make wily
-
-cd ../ubuntu-kernelbuilder
-
-make wily linux-image-wily
-
-echo "Linux kernel .deb packages are now available in packages/"

docker/scripts/build-trusty.sh

@@ -1,12 +1,13 @@
 #!/bin/bash
 ## @author gdm85
 ##
-## build a base Ubuntu image, can be Trusty, Wily etc
+## build a base Ubuntu Trusty
 #
 
 BASENAME=$(dirname $(readlink -m $0))
 
 ## the distro we are going to use
+DISTNAME=trusty
 REPOSRC=http://archive.ubuntu.com/ubuntu/
 
 if [ ! $UID -eq 0 ]; then
@@ -14,16 +15,9 @@ if [ ! $UID -eq 0 ]; then
 	exit 1
 fi
 
-if [ ! $# -eq 1 ]; then
-	echo "Usage: build-ubuntu-image.sh release-name" 1>&2
-	exit 2
-fi
-
-DISTNAME="$1"
-
 ## check for prerequisites
 if ! type -P debootstrap >/dev/null; then
-	echo "You need to install debootstrap" 1>&2
+	echo "You need to install debootstrap" 1&2
 	exit 2
 fi
 
@@ -56,6 +50,7 @@ else
 fi
 
 echo "Will use $KEYRING"
+exit 0
 
 ## NOTE: a temporary directory under /tmp is not used because can't be mounted dev/exec
 mkdir $DISTNAME || exit $?

docker/scripts/build-wheezy.sh

@@ -50,6 +50,7 @@ else
 fi
 
 echo "Will use $KEYRING"
+exit 0
 
 ## NOTE: a temporary directory under /tmp is not used because can't be mounted dev/exec
 mkdir $DISTNAME || exit $?

docker/scripts/create-gitian-host.sh

@@ -2,24 +2,6 @@
 
 BASENAME=$(dirname $(readlink -m $0))
 
-function wait_for_ssh() {
-       local IP="$1"
-       local SECS="$2"
-       while [ $SECS -gt 0 ]; do
-               ssh -o ConnectTimeout=1 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no debian@$IP ls >/dev/null 2>/dev/null && return 0
-               sleep 1
-               let SECS-=1
-       done
-       return 1
-}
-
-function wait_remove() {
-	local CID="$1"
-	while [ ! docker rm $CID 2>/dev/null ]; do
-		sleep 2
-	done
-}
-
 cd $BASENAME/../gitian-host || exit $?
 
 if [ ! -f authorized_keys ]; then
@@ -37,16 +19,26 @@ if [ ! -f authorized_keys ]; then
 	fi
 fi
 
+function wait_for_ssh() {
+	local IP="$1"
+	local SECS="$2"
+	while [ $SECS -gt 0 ]; do
+		ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no debian@$IP ls >/dev/null 2>/dev/null && return 0
+		sleep 1
+		let SECS-=1
+	done
+	return 1
+}
+
 ##NOTE: can leave behind a running container of gitian-host
 docker build --tag=gdm85/gitian-host . && \
 CID=$(docker run -d --privileged gdm85/gitian-host) && \
-IP=$(docker exec $CID cat /etc/hosts | tail -n1 | awk '{ print $1 }') && \
+echo "Now building base VMs" && \
-wait_for_ssh "$IP" 10 && \
+IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' $CID) && \
-echo "$CID is now online ($IP), building base VMs on it" && \
+wait_for_ssh $IP 10 && \
-ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no debian@$IP bash -c 'cd /home/debian && source ./.bash_profile && ./build-base-vms.sh amd64' && \
+ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no debian@$IP ./build-base-vms.sh && \
 docker kill $CID && \
-docker wait $CID && \
 docker commit $CID gdm85/gitian-host-vms && \
-wait_remove $CID && \
+docker rm $CID && \
 echo "Gitian host images created successfully!" && \
 echo "You can now spawn containers with spawn-gitian-host.sh"

docker/scripts/create-trusty-kbuilder.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+BASENAME=$(dirname $(readlink -m $0))
+
+cd $BASENAME/../trusty-kbuilder || exit $?
+
+if [ ! -f authorized_keys ]; then
+	echo "No authorized_keys file found in $PWD"
+	if [ -f ~/.ssh/id_rsa.pub ]; then
+		echo -n "Do you want to use ~/.ssh/id_rsa.pub? (y/n) "
+		read -r ANSWER
+		if [[ "$ANSWER" == "y" ]]; then
+			cp -v ~/.ssh/id_rsa.pub authorized_keys || exit $?
+		else
+			exit 1
+		fi
+	else
+		exit 1
+	fi
+fi
+
+##NOTE: can leave behind a running container of gitian-host
+docker build --tag=gdm85/trusty-kbuilder . && \
+echo "Ubuntu Trusty kernel builder image created successfully!" && \
+echo "You can now spawn containers with:" && \
+echo "docker run -d gdm85/trusty-kbuilder"

docker/scripts/docker-hosts

@@ -0,0 +1,21 @@
+#!/bin/bash
+## docker-hosts
+##
+## @author gdm85
+## script to show docker containers and their hostnames/IPv4
+#
+
+function enumerate_container_ips() {
+	local CID
+
+	for CID in $(docker ps -q -a); do
+		local NAME=$(docker inspect --format '{{ .Name }}' $CID | awk '{ print substr($0, 2, length($0)-1) }')
+		local HPATH=$(docker inspect --format '{{ .HostnamePath }}' $CID)
+		local IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' $CID)
+		local HOSTNAME=$(<$HPATH)
+
+		echo -e "$NAME\t$HOSTNAME\t$IP"
+	done
+}
+
+enumerate_container_ips

docker/scripts/docker-iptables

@@ -0,0 +1,28 @@
+#!/bin/bash
+## docker-iptables
+##
+## @author gdm85
+## script to show iptables rules with docker names
+## can be used also to detect problems with dead containers and stale iptable rules
+## supports standard iptables-save syntax
+#
+
+function replace_iptables() {
+        local CID
+
+	local SEDCMD="-e s!172.17.42.1/32!dockerHost!g
+        for CID in $(docker ps -q -a); do
+                local NAME=$(docker inspect --format '{{ .Name }}' $CID | awk '{ print substr($0, 2, length($0)-1) }')
+                local IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' $CID)
+
+                if [ -z "$IP" ]; then
+                        continue
+                fi
+
+                SEDCMD="$SEDCMD -e s!$IP/32!${NAME}!g -e s!$IP!${NAME}!g"
+        done
+
+        sed $SEDCMD
+}
+
+iptables-save $@ | replace_iptables

docker/trusty-kbuilder/Dockerfile

@@ -0,0 +1,48 @@
+## trusty-kbuilder
+##
+## VERSION 0.1.0
+##
+## Trusty image to build kernel
+##
+#
+
+## NOTE: this image must be debootstrapped with build-trusty.sh
+FROM gdm85/trusty
+
+MAINTAINER Giuseppe Mazzotta "gdm85@users.noreply.github.com"
+
+ENV DEBIAN_FRONTEND noninteractive
+
+## replace sources
+ADD sources.list /etc/apt/sources.list
+
+RUN apt-get update && apt-get install -y apt-utils aptitude && aptitude update && aptitude safe-upgrade -y && \
+aptitude install -y nano tmux fakeroot build-essential crash kexec-tools makedumpfile kernel-wedge openssh-server
+
+## log rotation is managed outside, thus remove it
+## install necessary daemons
+RUN apt-get remove -y logrotate
+
+## set current user's id_rsa as only allowed key
+ADD authorized_keys /root/.ssh/authorized_keys
+RUN chmod -R go-rwx /root/.ssh
+
+## disable password logins & reverse DNS lookups
+RUN sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config && echo 'UseDNS no' >> /etc/ssh/sshd_config
+
+EXPOSE 22
+
+##NOTE: this will use current kernel version!
+RUN apt-get build-dep -y --no-install-recommends linux-image-$(uname -r)
+RUN apt-get install -y git-core libncurses5 libncurses5-dev libelf-dev asciidoc binutils-dev
+
+## user that will make the compilation
+RUN useradd -m -s /bin/bash rdeckard
+
+## add script for getting source
+ADD build-kernel.sh /home/rdeckard/
+ADD patches/ /home/rdeckard/patches/
+
+RUN chown -R rdeckard.rdeckard /home/rdeckard
+
+CMD /usr/sbin/sshd -D -e

docker/trusty-kbuilder/build-kernel.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+
+set -e
+
+apt-get source linux-image-$(uname -r)
+
+cd linux-3.13.0
+
+## will fail here if no patches are available
+## why are you recompiling kernel if no custom patches are there?
+for MYP in $(ls ../patches); do
+	patch -p1 < ../patches/$MYP
+done
+
+fakeroot debian/rules clean
+DEB_BUILD_OPTIONS=parallel=3 AUTOBUILD=1 NOEXTRAS=1 fakeroot debian/rules binary-generic

docker/trusty-kbuilder/sources.list

@@ -0,0 +1,11 @@
+deb http://archive.ubuntu.com/ubuntu/ trusty main restricted
+deb-src http://archive.ubuntu.com/ubuntu/ trusty main restricted
+deb http://archive.ubuntu.com/ubuntu/ trusty-updates main restricted
+deb-src http://archive.ubuntu.com/ubuntu/ trusty-updates main restricted
+
+deb http://security.ubuntu.com/ubuntu trusty-security main restricted
+deb-src http://security.ubuntu.com/ubuntu trusty-security main restricted
+deb http://security.ubuntu.com/ubuntu trusty-security universe
+deb-src http://security.ubuntu.com/ubuntu trusty-security universe
+deb http://security.ubuntu.com/ubuntu trusty-security multiverse
+deb-src http://security.ubuntu.com/ubuntu trusty-security multiverse

docker/ubuntu-kernelbuilder/Dockerfile.tpl

@@ -1,16 +0,0 @@
-FROM %IMAGE%-pkgbuilder
-
-ENV KERNEL_VERSION %KERNEL_VERSION%
-
-RUN	apt-get build-dep -y --no-install-recommends linux-image-$KERNEL_VERSION && \
-	apt-get install -y git-core libncurses5 libncurses5-dev libelf-dev asciidoc binutils-dev
-
-USER rdeckard
-
-## fetch source
-RUN mkdir build && cd build && apt-get source linux-image-$KERNEL_VERSION
-
-## add build script
-COPY build-kernel.sh /home/rdeckard/
-
-RUN chown rdeckard.rdeckard /home/rdeckard/build-kernel.sh

docker/ubuntu-kernelbuilder/Makefile

@@ -1,15 +0,0 @@
-ifeq ($(KERNEL_VERSION),)
-	KERNEL_VERSION := $(shell uname -r)
-endif
-
-linux-image-%:
-	mkdir -p packages
-	docker run -e KERNEL_VERSION=$(KERNEL_VERSION) -v $(CURDIR)/packages:/home/rdeckard/packages -v $(CURDIR)/patches:/home/rdeckard/patches:ro $*-kernelbuilder ./build-kernel.sh $(KERNEL_VERSION)
-
-## supports any Ubuntu version (trusty, wily etc)
-%:
-	TMPD=`mktemp -d` && cp build-kernel.sh "$$TMPD/" && \
-	sed -e 's~%IMAGE%~$*~g' -e 's~%KERNEL_VERSION%~$(KERNEL_VERSION)~' Dockerfile.tpl > "$$TMPD/Dockerfile" && cd "$$TMPD" && \
-	docker build --tag='$*-kernelbuilder' . && cd && rm -rf "$TMPD"
-
-.DEFAULT: trusty

docker/ubuntu-kernelbuilder/build-kernel.sh

@@ -1,17 +0,0 @@
-#!/bin/bash
-
-set -e
-
-## enter the source-unpacked location
-cd build/linux-*
-
-## will fail here if no patches are available
-for MYP in $(ls ../../patches); do
-	patch -p1 < ../../patches/$MYP
-done
-
-fakeroot debian/rules clean
-
-DEB_BUILD_OPTIONS=parallel=4 AUTOBUILD=1 NOEXTRAS=1 fakeroot debian/rules binary-generic
-
-mv ../*.deb $HOME/packages/

docker/ubuntu-kernelbuilder/patches/add-module-parms.patch

@@ -1,44 +0,0 @@
---- a/drivers/bluetooth/btusb.c	2015-11-24 09:53:01.230539986 +0000
-+++ b/drivers/bluetooth/btusb.c	2015-11-24 09:54:32.598314339 +0000
-@@ -38,6 +38,9 @@
- static bool disable_scofix;
- static bool force_scofix;
- 
-+static int probe_interface = 2;
-+static int usb_interface = 3;
-+
- static bool reset = true;
- 
- static struct usb_driver btusb_driver;
-@@ -2657,7 +2660,7 @@
- 	BT_DBG("intf %p id %p", intf, id);
- 
- 	/* interface numbers are hardcoded in the spec */
--	if (intf->cur_altsetting->desc.bInterfaceNumber != 2)
-+	if (intf->cur_altsetting->desc.bInterfaceNumber != probe_interface)
- 		return -ENODEV;
- 
- 	if (!id->driver_info) {
-@@ -2827,7 +2830,7 @@
- 		data->isoc = NULL;
- 	} else {
- 		/* Interface numbers are hardcoded in the specification */
--		data->isoc = usb_ifnum_to_if(data->udev, 3);
-+		data->isoc = usb_ifnum_to_if(data->udev, usb_interface);
- 	}
- 
- 	if (!reset)
-@@ -3057,6 +3060,12 @@
- module_param(force_scofix, bool, 0644);
- MODULE_PARM_DESC(force_scofix, "Force fixup of wrong SCO buffers size");
- 
-+module_param(probe_interface, int, 0644);
-+MODULE_PARM_DESC(probe_interface, "Interface to probe");
-+
-+module_param(usb_interface, int, 0644);
-+MODULE_PARM_DESC(probe_interface, "USB interface to use");
-+
- module_param(reset, bool, 0644);
- MODULE_PARM_DESC(reset, "Send HCI reset command on initialization");
- 
-

docker/ubuntu-pkgbuilder/Dockerfile.tpl

@@ -1,27 +0,0 @@
-## ubuntu-pkgbuilder
-##
-## VERSION 0.1.1
-##
-## Ubuntu image to build a .deb package
-##
-#
-
-FROM %IMAGE%
-
-MAINTAINER Giuseppe Mazzotta "gdm85@users.noreply.github.com"
-
-ENV DEBIAN_FRONTEND noninteractive
-
-## replace sources
-COPY sources.list /etc/apt/sources.list
-
-RUN	apt-get update && apt-get install -y apt-utils aptitude && aptitude update && aptitude safe-upgrade -y && \
-	aptitude install -y nano tmux fakeroot build-essential crash kexec-tools makedumpfile kernel-wedge
-
-## log rotation is not managed in this container, thus remove it
-RUN apt-get remove -y logrotate
-
-## user that will make the compilation
-RUN useradd -m -s /bin/bash rdeckard && mkdir /home/rdeckard/patches && chown rdeckard.rdeckard /home/rdeckard/patches
-
-WORKDIR /home/rdeckard

docker/ubuntu-pkgbuilder/Makefile

@@ -1,7 +0,0 @@
-.DEFAULT: trusty
-
-## supports any base Ubuntu image (gdm85/trusty, gdm85/wily etc)
-## they should be generated with make-ubuntu-image.sh
-%:
-	TMPD=`mktemp -d` && sed 's~%IMAGE%~gdm85/$@~g' Dockerfile.tpl > "$$TMPD/Dockerfile" && sed 's~%DIST%~$@~g' sources.list.tpl > "$$TMPD/sources.list" && cd "$$TMPD" && \
-	docker build --tag='$@-pkgbuilder' . && cd && rm -rf "$TMPD"

docker/ubuntu-pkgbuilder/sources.list.tpl

@@ -1,11 +0,0 @@
-deb http://archive.ubuntu.com/ubuntu/ %DIST% main restricted
-deb-src http://archive.ubuntu.com/ubuntu/ %DIST% main restricted
-deb http://archive.ubuntu.com/ubuntu/ %DIST%-updates main restricted
-deb-src http://archive.ubuntu.com/ubuntu/ %DIST%-updates main restricted
-
-deb http://security.ubuntu.com/ubuntu %DIST%-security main restricted
-deb-src http://security.ubuntu.com/ubuntu %DIST%-security main restricted
-deb http://security.ubuntu.com/ubuntu %DIST%-security universe
-deb-src http://security.ubuntu.com/ubuntu %DIST%-security universe
-deb http://security.ubuntu.com/ubuntu %DIST%-security multiverse
-deb-src http://security.ubuntu.com/ubuntu %DIST%-security multiverse

lxd/README.md

@@ -1,25 +0,0 @@
-# lxd-images
-
-Set of images for use with [LXD](https://linuxcontainers.org/lxd/).
-
-<a rel="license" href="http://creativecommons.org/licenses/by-sa/2.0/"><img alt="Creative Commons License" style="border-width:0" src="http://i.creativecommons.org/l/by-sa/2.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/2.0/">Creative Commons Attribution-ShareAlike 2.0 Generic License</a>.
-
-## busybox-nonroot
-
-Running the containers' processes as non-root with ``busybox-nonroot`` is useful when you design applications that do not need root privileges,
-as you will be able to detect problems that would otherwise be unnoticed when running with root privileges.
-
-Create the image tarball with:
-```
-make
-```
-
-This will import the standard ``busybox`` image if you don't already have it.
-
-Afterwards you can import ``busybox-nonroot`` image with:
-```
-make import
-```
-
-This image has a ``nobody`` user defined, so that you can run all processes inside the container as a non-privileged user.
-**NOTE:** this is fundamentally different from root uid/gid mapping (which you should still use, if possible), as explained in [this blog post](https://www.stgraber.org/2014/01/17/lxc-1-0-unprivileged-containers/).

lxd/busybox-nonroot/Makefile

@@ -1,19 +0,0 @@
-.DEFAULT := all
-.PHONY := all import have-busybox
-
-## current hash of busybox image
-BASEIMG := 6e9df9219c50317368ec1e2b37da8bb1f5edcbefae3bae4bd38f41b56a365d09 
-
-all: busybox-nonroot.tar.xz
-
-have-busybox:
-	lxc image info busybox >/dev/null || lxd-images import busybox --alias busybox
-
-busybox-nonroot.tar.xz: have-busybox
-	TMPD=`mktemp -d` && mkdir -p "$$TMPD/rootfs/etc" && \
-	cp overlay/etc/passwd overlay/etc/group "$$TMPD/rootfs/etc/" && cd "$$TMPD" && tar xf /var/lib/lxd/images/$(BASEIMG) && \
-        chmod go+w /tmp && chmod +x /tmp && \
-	tar pcfJ $(CURDIR)/$@ rootfs/ metadata.yaml && cd && rm -rf "$$TMPD"
-
-import:
-	lxc image info busybox-nonroot >/dev/null 2>/dev/null || lxc image import busybox-nonroot.tar.xz --alias busybox-nonroot

lxd/busybox-nonroot/metadata.yaml

@@ -1,11 +0,0 @@
-{
-    "architecture": "x86_64",
-    "creation_date": 1443204631,
-    "properties": {
-        "architecture": "x86_64",
-        "description": "Busybox with unprivileged user x86_64",
-        "name": "busybox-nonroot-x86_64",
-        "author": "gdm85",
-        "os": "Busybox"
-    }
-}

lxd/busybox-nonroot/overlay/etc/group

@@ -1 +0,0 @@
-nobody:x:1:

lxd/busybox-nonroot/overlay/etc/passwd

@@ -1 +0,0 @@
-nobody:x:1:1:nobody:/tmp:/bin/sh

rkt/.gitignore

@@ -1 +0,0 @@
-nginx/nginx.aci

rkt/Makefile

@@ -1,22 +0,0 @@
-## define here all recursively buildable targets
-SUBDIRS := nginx
-
-.PHONY := $(SUBDIRS) recurse clean get-tools
-.DEFAULT_GOAL := recurse
-
-RECURSE_TARGET ?= $(MAKECMDGOALS)
-
-recurse:
-	@for D in $(SUBDIRS) ; do \
-	   if [ -e $${D}/Makefile ]; then \
-		$(MAKE) -C "$${D}" $(RECURSE_TARGET) || exit $$? ; \
-	   fi ; \
-	done
-
-clean: ; @$(MAKE) RECURSE_TARGET=clean recurse
-
-## disable entering/exiting messages
-#.SILENT:
-
-get-tools:
-	go get github.com/gdm85/deb2aci

rkt/README.md

@@ -1,28 +0,0 @@
-rkt images
-==========
-
-This repository contains recipes for various [ACI](https://github.com/appc/spec/blob/master/spec/aci.md) images that can be used with [rkt](https://github.com/coreos/rkt).
-
-<a rel="license" href="http://creativecommons.org/licenses/by-sa/2.0/"><img alt="Creative Commons License" style="border-width:0" src="http://i.creativecommons.org/l/by-sa/2.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/2.0/">Creative Commons Attribution-ShareAlike 2.0 Generic License</a>.
-
-Building
-========
-
-You will need [deb2aci](https://github.com/gdm85/deb2aci) to build these ACI images; you can fetch it automatically into your ``GOPATH`` with:
-```
-make get-tools
-```
-
-To build all images, run:
-```
-make
-```
-
-To build a specific image, for example ``nginx``, run:
-```
-make -C nginx
-```
-
-**DO NOT** run ``make nginx``, it would not build anything.
-
-Each image subdirectory comes with a README.md for specific documentation and instructions.

rkt/nginx/Makefile

@@ -1,13 +0,0 @@
-.DEFAULT_GOAL := all
-.PHONY := all clean
-
-## all necessary deb packages
-DEBS := nginx
-
-all: nginx.aci
-
-nginx.aci:
-	deb2aci -pkg $(DEBS) -manifest build.manifest -image $@
-
-clean:
-	rm -f nginx.aci

rkt/nginx/build.manifest

@@ -1,38 +0,0 @@
-{
-    "acKind": "ImageManifest",
-    "acVersion": "0.6.1",
-    "name": "gdm85/rkt-images/nginx",
-    "app":{
-        "user": "65534",
-        "group": "65534",
-        "exec":[
-            "/usr/sbin/nginx"
-        ],
-        "mountPoints": [
-            {
-                "name": "etc",
-                "path": "/etc/nginx",
-                "readOnly": true
-            },
-            {
-                "name": "www",
-                "path": "/var/www",
-                "readOnly": true
-            }
-        ],
-	"ports": [
-            {
-                "name": "www",
-                "port": 80,
-                "protocol": "tcp",
-                "socketActivated": false
-            }
-        ]
-    },
-    "annotations":[
-        {
-            "name":"authors",
-            "value":"gdm85 <gdm85@users.noreply.github.com>"
-        }
-    ]
-}