Archives/lokinet
2
0
Fork 0
mirror of https://github.com/oxen-io/lokinet.git synced 2024-10-31 09:20:21 +00:00
Code Issues Projects Releases Wiki Activity
d583f8bd72
lokinet/.drone.jsonnet

294 lines
14 KiB
Plaintext
Raw Normal View History

lokinet-bootstrap native binary
2021-04-15 17:39:45 +00:00
local default_deps_base='libsystemd-dev python3-dev libuv1-dev libunbound-dev nettle-dev libssl-dev libevent-dev libsqlite3-dev libcurl4-openssl-dev';
Add several drone ci builds
2020-05-15 23:29:05 +00:00
local default_deps_nocxx='libsodium-dev ' + default_deps_base; // libsodium-dev needs to be >= 1.0.18
local default_deps='g++ ' + default_deps_nocxx; // g++ sometimes needs replacement
make drone ci use debian testing (#1413)
2020-10-21 21:31:42 +00:00
local default_windows_deps='mingw-w64 zip nsis';
redo ci pipeline: * make cached docker containers for ci * remove old docker files from root of repo * use cached docker containers for ci
2021-02-26 17:26:48 +00:00
local docker_base = 'registry.oxen.rocks/lokinet-ci-';
consolidate windows ci drone jizz
2020-06-17 13:42:11 +00:00
Add tags/submodules as separate step Drone's initial clone doesn't include tags or submodules; add a step to fetch both (rather than doing this in the build step).
2020-06-12 16:57:59 +00:00
local submodules = {
name: 'submodules',
image: 'drone/git',
Shallow clone submodules
2020-10-02 21:36:52 +00:00
commands: ['git fetch --tags', 'git submodule update --init --recursive --depth=1']
Add tags/submodules as separate step Drone's initial clone doesn't include tags or submodules; add a step to fetch both (rather than doing this in the build step).
2020-06-12 16:57:59 +00:00
};
Fix deb snapshot builds + misc other CI updates (#1462) * Remove unneeded libcurl-dev build dep * Make apt-get operations quieter * apt repo updates - store the apt repo gpg pubkey in the repo rather than having to fetch it every time - add the apt repo for all the deb builds - update apt repo URL to deb.loki.network instead of deb.imaginary.stream * Use http so that we don't need ca-certificates The repo packages are already signed so using https really isn't important. * Add hack to ignore jsonnet conflicts in deb builds * Add build machine name to build output * We don't install any shared libs currently
2020-11-10 10:25:37 +00:00
local apt_get_quiet = 'apt-get -o=Dpkg::Use-Pty=0 -q';
drone tweaks - add some comments - exclude commits to the debian/ubuntu branches from getting built - disable "Updating man-db" which chews up several seconds during package install
2020-05-23 03:32:00 +00:00
// Regular build on a debian-like system:
Disable lto for most linux builds Keeps it on for the first three (sid/release, sid/debug, and clang) and turns them off for the rest.
2020-05-20 23:35:25 +00:00
local debian_pipeline(name, image,
arch='amd64',
deps=default_deps,
build_type='Release',
lto=false,
werror=true,
cmake_extra='',
extra_cmds=[],
Limit builds to 6 jobs (4 for arm) The CI runners don't use fixed CPU cores anymore, so that they can better allocate jobs across idle cores, but this means ninja's default is running an insane number of parallel compilations that likely just makes things slower. Add 6-job limiting to match the core limits that used to be in place.
2021-04-27 20:34:17 +00:00
jobs=6,
Fix deb snapshot builds + misc other CI updates (#1462) * Remove unneeded libcurl-dev build dep * Make apt-get operations quieter * apt repo updates - store the apt repo gpg pubkey in the repo rather than having to fetch it every time - add the apt repo for all the deb builds - update apt repo URL to deb.loki.network instead of deb.imaginary.stream * Use http so that we don't need ca-certificates The repo packages are already signed so using https really isn't important. * Add hack to ignore jsonnet conflicts in deb builds * Add build machine name to build output * We don't install any shared libs currently
2020-11-10 10:25:37 +00:00
loki_repo=false,
Disable lto for most linux builds Keeps it on for the first three (sid/release, sid/debug, and clang) and turns them off for the rest.
2020-05-20 23:35:25 +00:00
allow_fail=false) = {
Add several drone ci builds
2020-05-15 23:29:05 +00:00
kind: 'pipeline',
type: 'docker',
name: name,
platform: { arch: arch },
drone tweaks - add some comments - exclude commits to the debian/ubuntu branches from getting built - disable "Updating man-db" which chews up several seconds during package install
2020-05-23 03:32:00 +00:00
trigger: { branch: { exclude: ['debian/*', 'ubuntu/*'] } },
Add several drone ci builds
2020-05-15 23:29:05 +00:00
steps: [
Add tags/submodules as separate step Drone's initial clone doesn't include tags or submodules; add a step to fetch both (rather than doing this in the build step).
2020-06-12 16:57:59 +00:00
submodules,
Add several drone ci builds
2020-05-15 23:29:05 +00:00
{
name: 'build',
image: image,
[if allow_fail then "failure"]: "ignore",
Add tar+upload to builds.lokinet.dev of static build Relies on an SSH_KEY secret being set in the repository. Also move the linked library check to a bash script because escaping a value in jsonnet (escape once) that gets passed through yaml (escape twice) and then bash (escape again) is too painful.
2020-05-23 03:36:30 +00:00
environment: { SSH_KEY: { from_secret: "SSH_KEY" } },
Add several drone ci builds
2020-05-15 23:29:05 +00:00
commands: [
Fix deb snapshot builds + misc other CI updates (#1462) * Remove unneeded libcurl-dev build dep * Make apt-get operations quieter * apt repo updates - store the apt repo gpg pubkey in the repo rather than having to fetch it every time - add the apt repo for all the deb builds - update apt repo URL to deb.loki.network instead of deb.imaginary.stream * Use http so that we don't need ca-certificates The repo packages are already signed so using https really isn't important. * Add hack to ignore jsonnet conflicts in deb builds * Add build machine name to build output * We don't install any shared libs currently
2020-11-10 10:25:37 +00:00
'echo "Building on ${DRONE_STAGE_MACHINE}"',
drone tweaks - add some comments - exclude commits to the debian/ubuntu branches from getting built - disable "Updating man-db" which chews up several seconds during package install
2020-05-23 03:32:00 +00:00
'echo "man-db man-db/auto-update boolean false" | debconf-set-selections',
Fix deb snapshot builds + misc other CI updates (#1462) * Remove unneeded libcurl-dev build dep * Make apt-get operations quieter * apt repo updates - store the apt repo gpg pubkey in the repo rather than having to fetch it every time - add the apt repo for all the deb builds - update apt repo URL to deb.loki.network instead of deb.imaginary.stream * Use http so that we don't need ca-certificates The repo packages are already signed so using https really isn't important. * Add hack to ignore jsonnet conflicts in deb builds * Add build machine name to build output * We don't install any shared libs currently
2020-11-10 10:25:37 +00:00
apt_get_quiet + ' update',
apt_get_quiet + ' install -y eatmydata',
] + (if loki_repo then [
'eatmydata ' + apt_get_quiet + ' install -y lsb-release',
'cp contrib/deb.loki.network.gpg /etc/apt/trusted.gpg.d',
'echo deb http://deb.loki.network $$(lsb_release -sc) main >/etc/apt/sources.list.d/loki.network.list',
'eatmydata ' + apt_get_quiet + ' update'
Use imaginary repo for sodium on bionic/arm64 Building both sodium + libzmq is kind of a pain, so just use the backport libsodium from the loki repo on deb.imaginary.stream.
2020-06-26 21:52:04 +00:00
] else []
) + [
Fix deb snapshot builds + misc other CI updates (#1462) * Remove unneeded libcurl-dev build dep * Make apt-get operations quieter * apt repo updates - store the apt repo gpg pubkey in the repo rather than having to fetch it every time - add the apt repo for all the deb builds - update apt repo URL to deb.loki.network instead of deb.imaginary.stream * Use http so that we don't need ca-certificates The repo packages are already signed so using https really isn't important. * Add hack to ignore jsonnet conflicts in deb builds * Add build machine name to build output * We don't install any shared libs currently
2020-11-10 10:25:37 +00:00
'eatmydata ' + apt_get_quiet + ' dist-upgrade -y',
'eatmydata ' + apt_get_quiet + ' install -y gdb cmake git ninja-build pkg-config ccache ' + deps,
Add several drone ci builds
2020-05-15 23:29:05 +00:00
'mkdir build',
'cd build',
explicitly turn off setcap on ci builds so they pass
2021-03-04 11:00:03 +00:00
'cmake .. -G Ninja -DWITH_SETCAP=OFF -DCMAKE_CXX_FLAGS=-fdiagnostics-color=always -DCMAKE_BUILD_TYPE='+build_type+' ' +
Add several drone ci builds
2020-05-15 23:29:05 +00:00
(if werror then '-DWARNINGS_AS_ERRORS=ON ' else '') +
Move resolvconf from deb branches to contrib/ (#1439) * Move resolvconf from deb branches to contrib/ This script gets invoked by the systemd service after startup to update resolvconf with the lokinet dns server. It was previously living in debian/lokinet-resolvconf in the debian/ubuntu branches, but really belongs in contrib/ instead. * Disable LTO on sid gcc/clang-11
2020-10-31 09:21:55 +00:00
'-DWITH_LTO=' + (if lto then 'ON ' else 'OFF ') +
add gdb to ci runners on debian to catch backtraces if they happen
2020-06-16 13:21:19 +00:00
cmake_extra,
Limit builds to 6 jobs (4 for arm) The CI runners don't use fixed CPU cores anymore, so that they can better allocate jobs across idle cores, but this means ninja's default is running an insane number of parallel compilations that likely just makes things slower. Add 6-job limiting to match the core limits that used to be in place.
2021-04-27 20:34:17 +00:00
'ninja -j' + jobs + ' -v',
catchAll -> testAll
2021-03-04 20:45:52 +00:00
'../contrib/ci/drone-gdb.sh ./test/testAll --use-colour yes',
Add several drone ci builds
2020-05-15 23:29:05 +00:00
] + extra_cmds,
}
Build (and upload) debs by merging into the deb branches Similar to the static files, the debs also end up under builds.lokinet.dev/deb.
2020-05-23 03:39:04 +00:00
],
};
disallow fail on ci pipeline for android builds
2021-03-02 20:21:39 +00:00
local apk_builder(name, image, extra_cmds=[], allow_fail=false) = {
Large collection of changes to make android work - Previous android java and jni code updated to work, but with much love still needed to make it work nicely, e.g. handling when the VPN is turned off. - DNS handling refactored to allow android to intercept and handle DNS requests as we can't set the system DNS to use a high port (and apparently Chrome ignores system DNS settings anyway) - add packet router structure to allow separate handling of specific intercepted traffic, e.g. UDP traffic to port 53 gets handled by our DNS handler rather than being naively forwarded as exit traffic. - For now, android lokinet is exit-only and hard-coded to use exit.loki as its exit. The exit will be configurable before release, but allowing to not use exit-only mode is more of a challenge. - some old gitignore remnants which were matching to things we don't want them to (and are no longer relevant) removed - some minor changes to CI configuration
2021-03-02 18:18:22 +00:00
kind: 'pipeline',
type: 'docker',
name: name,
platform: {arch: "amd64"},
trigger: { branch: { exclude: ['debian/*', 'ubuntu/*'] } },
steps: [
submodules,
{
name: 'build',
image: image,
[if allow_fail then "failure"]: "ignore",
environment: { SSH_KEY: { from_secret: "SSH_KEY" }, ANDROID: "android" },
commands: [
"cd android",
"rm -f local.properties",
"echo 'sdk.dir=/usr/lib/android-sdk' >> local.properties",
"echo 'ndk.dir=/usr/lib/android-ndk' >> local.properties",
Allow multiple gradles to run on the same box /cache is shared among multiple runners that can potentiall run at once, but gradle's cache is retarded and just completely blocks anything from using the cache for the span of the entire build. Work around it by sticking the unique builder identifier in the cache path.
2021-03-03 01:23:37 +00:00
"GRADLE_USER_HOME=/cache/gradle/${DRONE_STAGE_MACHINE} gradle --no-daemon assembleDebug",
Large collection of changes to make android work - Previous android java and jni code updated to work, but with much love still needed to make it work nicely, e.g. handling when the VPN is turned off. - DNS handling refactored to allow android to intercept and handle DNS requests as we can't set the system DNS to use a high port (and apparently Chrome ignores system DNS settings anyway) - add packet router structure to allow separate handling of specific intercepted traffic, e.g. UDP traffic to port 53 gets handled by our DNS handler rather than being naively forwarded as exit traffic. - For now, android lokinet is exit-only and hard-coded to use exit.loki as its exit. The exit will be configurable before release, but allowing to not use exit-only mode is more of a challenge. - some old gitignore remnants which were matching to things we don't want them to (and are no longer relevant) removed - some minor changes to CI configuration
2021-03-02 18:18:22 +00:00
] + extra_cmds
}
]
};
// windows cross compile on debian
consolidate windows ci drone jizz
2020-06-17 13:42:11 +00:00
local windows_cross_pipeline(name, image,
windows builds on alpine lunix
2020-06-12 20:55:24 +00:00
arch='amd64',
build_type='Release',
turn off lto, add patches
2020-06-17 14:37:41 +00:00
lto=false,
windows builds on alpine lunix
2020-06-12 20:55:24 +00:00
werror=false,
cmake_extra='',
pass in WINDOWS_BUILD_NAME to uploader to mark 64bit and 32bit builds distinctly
2020-06-18 11:30:07 +00:00
toolchain='32',
windows builds on alpine lunix
2020-06-12 20:55:24 +00:00
extra_cmds=[],
Limit builds to 6 jobs (4 for arm) The CI runners don't use fixed CPU cores anymore, so that they can better allocate jobs across idle cores, but this means ninja's default is running an insane number of parallel compilations that likely just makes things slower. Add 6-job limiting to match the core limits that used to be in place.
2021-04-27 20:34:17 +00:00
jobs=6,
windows builds on alpine lunix
2020-06-12 20:55:24 +00:00
allow_fail=false) = {
kind: 'pipeline',
type: 'docker',
name: name,
platform: { arch: arch },
trigger: { branch: { exclude: ['debian/*', 'ubuntu/*'] } },
steps: [
submodules,
{
name: 'build',
image: image,
[if allow_fail then "failure"]: "ignore",
pass in WINDOWS_BUILD_NAME to uploader to mark 64bit and 32bit builds distinctly
2020-06-18 11:30:07 +00:00
environment: { SSH_KEY: { from_secret: "SSH_KEY" }, WINDOWS_BUILD_NAME: toolchain+"bit" },
windows builds on alpine lunix
2020-06-12 20:55:24 +00:00
commands: [
Fix deb snapshot builds + misc other CI updates (#1462) * Remove unneeded libcurl-dev build dep * Make apt-get operations quieter * apt repo updates - store the apt repo gpg pubkey in the repo rather than having to fetch it every time - add the apt repo for all the deb builds - update apt repo URL to deb.loki.network instead of deb.imaginary.stream * Use http so that we don't need ca-certificates The repo packages are already signed so using https really isn't important. * Add hack to ignore jsonnet conflicts in deb builds * Add build machine name to build output * We don't install any shared libs currently
2020-11-10 10:25:37 +00:00
'echo "Building on ${DRONE_STAGE_MACHINE}"',
make drone ci use debian testing (#1413)
2020-10-21 21:31:42 +00:00
'echo "man-db man-db/auto-update boolean false" | debconf-set-selections',
Fix deb snapshot builds + misc other CI updates (#1462) * Remove unneeded libcurl-dev build dep * Make apt-get operations quieter * apt repo updates - store the apt repo gpg pubkey in the repo rather than having to fetch it every time - add the apt repo for all the deb builds - update apt repo URL to deb.loki.network instead of deb.imaginary.stream * Use http so that we don't need ca-certificates The repo packages are already signed so using https really isn't important. * Add hack to ignore jsonnet conflicts in deb builds * Add build machine name to build output * We don't install any shared libs currently
2020-11-10 10:25:37 +00:00
apt_get_quiet + ' update',
apt_get_quiet + ' install -y eatmydata',
Add automake+libtool for libuv static build
2021-01-19 00:45:04 +00:00
'eatmydata ' + apt_get_quiet + ' install -y build-essential cmake git ninja-build pkg-config ccache g++-mingw-w64-x86-64-posix nsis zip automake libtool',
make drone ci use debian testing (#1413)
2020-10-21 21:31:42 +00:00
'update-alternatives --set x86_64-w64-mingw32-gcc /usr/bin/x86_64-w64-mingw32-gcc-posix',
'update-alternatives --set x86_64-w64-mingw32-g++ /usr/bin/x86_64-w64-mingw32-g++-posix',
windows builds on alpine lunix
2020-06-12 20:55:24 +00:00
'mkdir build',
'cd build',
Remove CMAKE_CROSSCOMPILE We set CMAKE_CROSSCOMPILE in a few places, but that is just a typo for CMAKE_CROSSCOMPILING. However we don't need to set it at all: cmake does so automatically whenever CMAKE_SYSTEM_NAME is set (which is why our cross compiling actually works currently.)
2021-01-25 17:06:59 +00:00
'cmake .. -G Ninja -DCMAKE_EXE_LINKER_FLAGS=-fstack-protector -DCMAKE_CXX_FLAGS=-fdiagnostics-color=always -DCMAKE_TOOLCHAIN_FILE=../contrib/cross/mingw'+toolchain+'.cmake -DCMAKE_BUILD_TYPE='+build_type+' ' +
windows builds on alpine lunix
2020-06-12 20:55:24 +00:00
(if werror then '-DWARNINGS_AS_ERRORS=ON ' else '') +
(if lto then '' else '-DWITH_LTO=OFF ') +
make drone ci use debian testing (#1413)
2020-10-21 21:31:42 +00:00
"-DBUILD_STATIC_DEPS=ON -DDOWNLOAD_SODIUM=ON -DBUILD_PACKAGE=ON -DBUILD_SHARED_LIBS=OFF -DBUILD_TESTING=OFF -DNATIVE_BUILD=OFF -DSTATIC_LINK=ON" +
add docs to gdb-filter.py turn off thread logging in gdb make sure everything runs in gdb when running test add note in contrib/format.sh about a GNU-ism
2020-06-18 13:44:13 +00:00
cmake_extra,
Limit builds to 6 jobs (4 for arm) The CI runners don't use fixed CPU cores anymore, so that they can better allocate jobs across idle cores, but this means ninja's default is running an insane number of parallel compilations that likely just makes things slower. Add 6-job limiting to match the core limits that used to be in place.
2021-04-27 20:34:17 +00:00
'ninja -j' + jobs + ' -v package',
windows builds on alpine lunix
2020-06-12 20:55:24 +00:00
] + extra_cmds,
}
],
64-bit LTO cursed for ages
2020-06-12 22:48:49 +00:00
};
Build (and upload) debs by merging into the deb branches Similar to the static files, the debs also end up under builds.lokinet.dev/deb.
2020-05-23 03:39:04 +00:00
// Builds a snapshot .deb on a debian-like system by merging into the debian/* or ubuntu/* branch
Fix deb snapshot builds + misc other CI updates (#1462) * Remove unneeded libcurl-dev build dep * Make apt-get operations quieter * apt repo updates - store the apt repo gpg pubkey in the repo rather than having to fetch it every time - add the apt repo for all the deb builds - update apt repo URL to deb.loki.network instead of deb.imaginary.stream * Use http so that we don't need ca-certificates The repo packages are already signed so using https really isn't important. * Add hack to ignore jsonnet conflicts in deb builds * Add build machine name to build output * We don't install any shared libs currently
2020-11-10 10:25:37 +00:00
local deb_builder(image, distro, distro_branch, arch='amd64', loki_repo=true) = {
Build (and upload) debs by merging into the deb branches Similar to the static files, the debs also end up under builds.lokinet.dev/deb.
2020-05-23 03:39:04 +00:00
kind: 'pipeline',
type: 'docker',
name: 'DEB (' + distro + (if arch == 'amd64' then '' else '/' + arch) + ')',
platform: { arch: arch },
environment: { distro_branch: distro_branch, distro: distro },
steps: [
Add tags/submodules as separate step Drone's initial clone doesn't include tags or submodules; add a step to fetch both (rather than doing this in the build step).
2020-06-12 16:57:59 +00:00
submodules,
Build (and upload) debs by merging into the deb branches Similar to the static files, the debs also end up under builds.lokinet.dev/deb.
2020-05-23 03:39:04 +00:00
{
name: 'build',
image: image,
failure: 'ignore',
environment: { SSH_KEY: { from_secret: "SSH_KEY" } },
commands: [
Fix deb snapshot builds + misc other CI updates (#1462) * Remove unneeded libcurl-dev build dep * Make apt-get operations quieter * apt repo updates - store the apt repo gpg pubkey in the repo rather than having to fetch it every time - add the apt repo for all the deb builds - update apt repo URL to deb.loki.network instead of deb.imaginary.stream * Use http so that we don't need ca-certificates The repo packages are already signed so using https really isn't important. * Add hack to ignore jsonnet conflicts in deb builds * Add build machine name to build output * We don't install any shared libs currently
2020-11-10 10:25:37 +00:00
'echo "Building on ${DRONE_STAGE_MACHINE}"',
'echo "man-db man-db/auto-update boolean false" | debconf-set-selections'
] + (if loki_repo then [
'cp contrib/deb.loki.network.gpg /etc/apt/trusted.gpg.d',
'echo deb http://deb.loki.network $${distro} main >/etc/apt/sources.list.d/loki.network.list'
Build (and upload) debs by merging into the deb branches Similar to the static files, the debs also end up under builds.lokinet.dev/deb.
2020-05-23 03:39:04 +00:00
] else []) + [
Fix deb snapshot builds + misc other CI updates (#1462) * Remove unneeded libcurl-dev build dep * Make apt-get operations quieter * apt repo updates - store the apt repo gpg pubkey in the repo rather than having to fetch it every time - add the apt repo for all the deb builds - update apt repo URL to deb.loki.network instead of deb.imaginary.stream * Use http so that we don't need ca-certificates The repo packages are already signed so using https really isn't important. * Add hack to ignore jsonnet conflicts in deb builds * Add build machine name to build output * We don't install any shared libs currently
2020-11-10 10:25:37 +00:00
apt_get_quiet + ' update',
apt_get_quiet + ' install -y eatmydata',
'eatmydata ' + apt_get_quiet + ' install -y git devscripts equivs ccache git-buildpackage python3-dev',
Build (and upload) debs by merging into the deb branches Similar to the static files, the debs also end up under builds.lokinet.dev/deb.
2020-05-23 03:39:04 +00:00
|||
# Look for the debian branch in this repo first, try upstream if that fails.
if ! git checkout $${distro_branch}; then
Fix build server name & path (#1516) builds.lokinet.dev -> oxen.rocks /loki-project/ -> /oxen-io/
2021-01-19 12:39:53 +00:00
git remote add --fetch upstream https://github.com/oxen-io/loki-network.git &&
Build (and upload) debs by merging into the deb branches Similar to the static files, the debs also end up under builds.lokinet.dev/deb.
2020-05-23 03:39:04 +00:00
git checkout $${distro_branch}
fi
|||,
Fix deb snapshot builds + misc other CI updates (#1462) * Remove unneeded libcurl-dev build dep * Make apt-get operations quieter * apt repo updates - store the apt repo gpg pubkey in the repo rather than having to fetch it every time - add the apt repo for all the deb builds - update apt repo URL to deb.loki.network instead of deb.imaginary.stream * Use http so that we don't need ca-certificates The repo packages are already signed so using https really isn't important. * Add hack to ignore jsonnet conflicts in deb builds * Add build machine name to build output * We don't install any shared libs currently
2020-11-10 10:25:37 +00:00
# Tell the merge how to resolve conflicts in the source .drone.jsonnet (we don't
# care about it at all since *this* .drone.jsonnet is already loaded).
'git config merge.ours.driver true',
'echo .drone.jsonnet merge=ours >>.gitattributes',
Build (and upload) debs by merging into the deb branches Similar to the static files, the debs also end up under builds.lokinet.dev/deb.
2020-05-23 03:39:04 +00:00
'git merge ${DRONE_COMMIT}',
'export DEBEMAIL="${DRONE_COMMIT_AUTHOR_EMAIL}" DEBFULLNAME="${DRONE_COMMIT_AUTHOR_NAME}"',
'gbp dch -S -s "HEAD^" --spawn-editor=never -U low',
Fix deb snapshot builds + misc other CI updates (#1462) * Remove unneeded libcurl-dev build dep * Make apt-get operations quieter * apt repo updates - store the apt repo gpg pubkey in the repo rather than having to fetch it every time - add the apt repo for all the deb builds - update apt repo URL to deb.loki.network instead of deb.imaginary.stream * Use http so that we don't need ca-certificates The repo packages are already signed so using https really isn't important. * Add hack to ignore jsonnet conflicts in deb builds * Add build machine name to build output * We don't install any shared libs currently
2020-11-10 10:25:37 +00:00
'eatmydata mk-build-deps --install --remove --tool "' + apt_get_quiet + ' -o Debug::pkgProblemResolver=yes --no-install-recommends -y"',
Build (and upload) debs by merging into the deb branches Similar to the static files, the debs also end up under builds.lokinet.dev/deb.
2020-05-23 03:39:04 +00:00
'export DEB_BUILD_OPTIONS="parallel=$$(nproc)"',
Fix deb snapshot builds + misc other CI updates (#1462) * Remove unneeded libcurl-dev build dep * Make apt-get operations quieter * apt repo updates - store the apt repo gpg pubkey in the repo rather than having to fetch it every time - add the apt repo for all the deb builds - update apt repo URL to deb.loki.network instead of deb.imaginary.stream * Use http so that we don't need ca-certificates The repo packages are already signed so using https really isn't important. * Add hack to ignore jsonnet conflicts in deb builds * Add build machine name to build output * We don't install any shared libs currently
2020-11-10 10:25:37 +00:00
#'grep -q lib debian/lokinet-bin.install || echo "/usr/lib/lib*.so*" >>debian/lokinet-bin.install',
Build (and upload) debs by merging into the deb branches Similar to the static files, the debs also end up under builds.lokinet.dev/deb.
2020-05-23 03:39:04 +00:00
'debuild -e CCACHE_DIR -b',
'./contrib/ci/drone-debs-upload.sh ' + distro,
]
}
Add several drone ci builds
2020-05-15 23:29:05 +00:00
]
};
Large collection of changes to make android work - Previous android java and jni code updated to work, but with much love still needed to make it work nicely, e.g. handling when the VPN is turned off. - DNS handling refactored to allow android to intercept and handle DNS requests as we can't set the system DNS to use a high port (and apparently Chrome ignores system DNS settings anyway) - add packet router structure to allow separate handling of specific intercepted traffic, e.g. UDP traffic to port 53 gets handled by our DNS handler rather than being naively forwarded as exit traffic. - For now, android lokinet is exit-only and hard-coded to use exit.loki as its exit. The exit will be configurable before release, but allowing to not use exit-only mode is more of a challenge. - some old gitignore remnants which were matching to things we don't want them to (and are no longer relevant) removed - some minor changes to CI configuration
2021-03-02 18:18:22 +00:00
macos static build & upload
2020-05-23 14:46:45 +00:00
// Macos build
Limit builds to 6 jobs (4 for arm) The CI runners don't use fixed CPU cores anymore, so that they can better allocate jobs across idle cores, but this means ninja's default is running an insane number of parallel compilations that likely just makes things slower. Add 6-job limiting to match the core limits that used to be in place.
2021-04-27 20:34:17 +00:00
local mac_builder(name,
build_type='Release',
werror=true,
cmake_extra='',
extra_cmds=[],
jobs=6,
allow_fail=false) = {
macos static build & upload
2020-05-23 14:46:45 +00:00
kind: 'pipeline',
type: 'exec',
name: name,
platform: { os: 'darwin', arch: 'amd64' },
steps: [
Add tags/submodules as separate step Drone's initial clone doesn't include tags or submodules; add a step to fetch both (rather than doing this in the build step).
2020-06-12 16:57:59 +00:00
{ name: 'submodules', commands: ['git fetch --tags', 'git submodule update --init --recursive'] },
macos static build & upload
2020-05-23 14:46:45 +00:00
{
name: 'build',
environment: { SSH_KEY: { from_secret: "SSH_KEY" } },
commands: [
Fix deb snapshot builds + misc other CI updates (#1462) * Remove unneeded libcurl-dev build dep * Make apt-get operations quieter * apt repo updates - store the apt repo gpg pubkey in the repo rather than having to fetch it every time - add the apt repo for all the deb builds - update apt repo URL to deb.loki.network instead of deb.imaginary.stream * Use http so that we don't need ca-certificates The repo packages are already signed so using https really isn't important. * Add hack to ignore jsonnet conflicts in deb builds * Add build machine name to build output * We don't install any shared libs currently
2020-11-10 10:25:37 +00:00
'echo "Building on ${DRONE_STAGE_MACHINE}"',
macos static build & upload
2020-05-23 14:46:45 +00:00
// If you don't do this then the C compiler doesn't have an include path containing
// basic system headers. WTF apple:
'export SDKROOT="$(xcrun --sdk macosx --show-sdk-path)"',
add ulimit -n 1024 to mac ci stuff because lol apple
2021-01-12 14:46:44 +00:00
'ulimit -n 1024', // because macos sets ulimit to 256 for some reason yeah idk
macos static build & upload
2020-05-23 14:46:45 +00:00
'mkdir build',
'cd build',
'cmake .. -G Ninja -DCMAKE_CXX_FLAGS=-fcolor-diagnostics -DCMAKE_BUILD_TYPE='+build_type+' ' +
add docs to gdb-filter.py turn off thread logging in gdb make sure everything runs in gdb when running test add note in contrib/format.sh about a GNU-ism
2020-06-18 13:44:13 +00:00
(if werror then '-DWARNINGS_AS_ERRORS=ON ' else '') + cmake_extra,
Limit builds to 6 jobs (4 for arm) The CI runners don't use fixed CPU cores anymore, so that they can better allocate jobs across idle cores, but this means ninja's default is running an insane number of parallel compilations that likely just makes things slower. Add 6-job limiting to match the core limits that used to be in place.
2021-04-27 20:34:17 +00:00
'ninja -j' + jobs + ' -v',
catchAll -> testAll
2021-03-04 20:45:52 +00:00
'./test/testAll --use-colour yes',
macos static build & upload
2020-05-23 14:46:45 +00:00
] + extra_cmds,
}
]
};
Build (and upload) debs by merging into the deb branches Similar to the static files, the debs also end up under builds.lokinet.dev/deb.
2020-05-23 03:39:04 +00:00
Add several drone ci builds
2020-05-15 23:29:05 +00:00
[
{
name: 'lint check',
kind: 'pipeline',
type: 'docker',
steps: [{
redo ci pipeline: * make cached docker containers for ci * remove old docker files from root of repo * use cached docker containers for ci
2021-02-26 17:26:48 +00:00
name: 'build', image: 'registry.oxen.rocks/lokinet-ci-lint',
Add several drone ci builds
2020-05-15 23:29:05 +00:00
commands: [
Fix deb snapshot builds + misc other CI updates (#1462) * Remove unneeded libcurl-dev build dep * Make apt-get operations quieter * apt repo updates - store the apt repo gpg pubkey in the repo rather than having to fetch it every time - add the apt repo for all the deb builds - update apt repo URL to deb.loki.network instead of deb.imaginary.stream * Use http so that we don't need ca-certificates The repo packages are already signed so using https really isn't important. * Add hack to ignore jsonnet conflicts in deb builds * Add build machine name to build output * We don't install any shared libs currently
2020-11-10 10:25:37 +00:00
'echo "Building on ${DRONE_STAGE_MACHINE}"',
apt_get_quiet + ' update',
apt_get_quiet + ' install -y eatmydata',
Upgrade to clang-format 11 The formatting changes here are not too large and also nice: in particular clang-format-11 appears to do better handling of `{...}` constructor braces/arguments.
2021-03-05 17:30:54 +00:00
'eatmydata ' + apt_get_quiet + ' install -y git clang-format-11',
remove the dreaded makefile and update docs
2020-06-12 18:32:13 +00:00
'./contrib/ci/drone-format-verify.sh']
Add several drone ci builds
2020-05-15 23:29:05 +00:00
}]
},
drone tweaks - add some comments - exclude commits to the debian/ubuntu branches from getting built - disable "Updating man-db" which chews up several seconds during package install
2020-05-23 03:32:00 +00:00
// Various debian builds
Move resolvconf from deb branches to contrib/ (#1439) * Move resolvconf from deb branches to contrib/ This script gets invoked by the systemd service after startup to update resolvconf with the lokinet dns server. It was previously living in debian/lokinet-resolvconf in the debian/ubuntu branches, but really belongs in contrib/ instead. * Disable LTO on sid gcc/clang-11
2020-10-31 09:21:55 +00:00
debian_pipeline("Debian sid (amd64)", "debian:sid"),
Disable LTO on sid for now (#1472)
2020-11-13 14:00:04 +00:00
debian_pipeline("Debian sid/Debug (amd64)", "debian:sid", build_type='Debug'),
redo ci pipeline: * make cached docker containers for ci * remove old docker files from root of repo * use cached docker containers for ci
2021-02-26 17:26:48 +00:00
debian_pipeline("Debian sid/clang-11 (amd64)", docker_base+'debian-sid', deps='clang-11 '+default_deps_nocxx,
Move resolvconf from deb branches to contrib/ (#1439) * Move resolvconf from deb branches to contrib/ This script gets invoked by the systemd service after startup to update resolvconf with the lokinet dns server. It was previously living in debian/lokinet-resolvconf in the debian/ubuntu branches, but really belongs in contrib/ instead. * Disable LTO on sid gcc/clang-11
2020-10-31 09:21:55 +00:00
cmake_extra='-DCMAKE_C_COMPILER=clang-11 -DCMAKE_CXX_COMPILER=clang++-11 '),
Add several drone ci builds
2020-05-15 23:29:05 +00:00
debian_pipeline("Debian buster (i386)", "i386/debian:buster", cmake_extra='-DDOWNLOAD_SODIUM=ON'),
redo ci pipeline: * make cached docker containers for ci * remove old docker files from root of repo * use cached docker containers for ci
2021-02-26 17:26:48 +00:00
debian_pipeline("Ubuntu focal (amd64)", docker_base+'ubuntu-focal'),
Drone CI job shuffling (#1359) - Disable bionic arm64 job to reduce the load on the ARM box a bit. - Add bionic amd64 build so that we have a (normal) build on bionic. - Remove sid/g++-10 job because g++ is now the default on sid (so the debian/sid build is already doing the same thing).
2020-09-22 20:10:38 +00:00
debian_pipeline("Ubuntu bionic (amd64)", "ubuntu:bionic", deps='g++-8 ' + default_deps_nocxx,
Fix deb snapshot builds + misc other CI updates (#1462) * Remove unneeded libcurl-dev build dep * Make apt-get operations quieter * apt repo updates - store the apt repo gpg pubkey in the repo rather than having to fetch it every time - add the apt repo for all the deb builds - update apt repo URL to deb.loki.network instead of deb.imaginary.stream * Use http so that we don't need ca-certificates The repo packages are already signed so using https really isn't important. * Add hack to ignore jsonnet conflicts in deb builds * Add build machine name to build output * We don't install any shared libs currently
2020-11-10 10:25:37 +00:00
cmake_extra='-DCMAKE_C_COMPILER=gcc-8 -DCMAKE_CXX_COMPILER=g++-8', loki_repo=true),
drone tweaks - add some comments - exclude commits to the debian/ubuntu branches from getting built - disable "Updating man-db" which chews up several seconds during package install
2020-05-23 03:32:00 +00:00
// ARM builds (ARM64 and armhf)
Limit builds to 6 jobs (4 for arm) The CI runners don't use fixed CPU cores anymore, so that they can better allocate jobs across idle cores, but this means ninja's default is running an insane number of parallel compilations that likely just makes things slower. Add 6-job limiting to match the core limits that used to be in place.
2021-04-27 20:34:17 +00:00
debian_pipeline("Debian sid (ARM64)", "debian:sid", arch="arm64", jobs=4),
debian_pipeline("Debian buster (armhf)", "arm32v7/debian:buster", arch="arm64", cmake_extra='-DDOWNLOAD_SODIUM=ON', jobs=4),
Add full-static-deps armhf build (+upload) (#1505) * Add armhf static build + upload * openssl build fix for static armhf build Without this it tries building for arm64. * Disable LTO for armhf static build * Fix static lib check for arm
2021-01-04 21:17:29 +00:00
// Static armhf build (gets uploaded)
Add automake+libtool for libuv static build
2021-01-19 00:45:04 +00:00
debian_pipeline("Static (buster armhf)", "arm32v7/debian:buster", arch="arm64", deps='g++ python3-dev automake libtool',
Add full-static-deps armhf build (+upload) (#1505) * Add armhf static build + upload * openssl build fix for static armhf build Without this it tries building for arm64. * Disable LTO for armhf static build * Fix static lib check for arm
2021-01-04 21:17:29 +00:00
cmake_extra='-DBUILD_STATIC_DEPS=ON -DBUILD_SHARED_LIBS=OFF -DSTATIC_LINK=ON ' +
'-DCMAKE_CXX_FLAGS="-march=armv7-a+fp" -DCMAKE_C_FLAGS="-march=armv7-a+fp" -DNATIVE_BUILD=OFF ' +
'-DWITH_SYSTEMD=OFF',
extra_cmds=[
'../contrib/ci/drone-check-static-libs.sh',
'UPLOAD_OS=linux-armhf ../contrib/ci/drone-static-upload.sh'
Limit builds to 6 jobs (4 for arm) The CI runners don't use fixed CPU cores anymore, so that they can better allocate jobs across idle cores, but this means ninja's default is running an insane number of parallel compilations that likely just makes things slower. Add 6-job limiting to match the core limits that used to be in place.
2021-04-27 20:34:17 +00:00
],
jobs=4),
Large collection of changes to make android work - Previous android java and jni code updated to work, but with much love still needed to make it work nicely, e.g. handling when the VPN is turned off. - DNS handling refactored to allow android to intercept and handle DNS requests as we can't set the system DNS to use a high port (and apparently Chrome ignores system DNS settings anyway) - add packet router structure to allow separate handling of specific intercepted traffic, e.g. UDP traffic to port 53 gets handled by our DNS handler rather than being naively forwarded as exit traffic. - For now, android lokinet is exit-only and hard-coded to use exit.loki as its exit. The exit will be configurable before release, but allowing to not use exit-only mode is more of a challenge. - some old gitignore remnants which were matching to things we don't want them to (and are no longer relevant) removed - some minor changes to CI configuration
2021-03-02 18:18:22 +00:00
// android apk builder
apk_builder("android apk", "registry.oxen.rocks/lokinet-ci-android", extra_cmds=['UPLOAD_OS=anrdoid ../contrib/ci/drone-static-upload.sh']),
64-bit LTO cursed for ages
2020-06-12 22:48:49 +00:00
Install less stuff on windows CI mingw-w64 is a metapackage that pulls in 4 different compilers {i686,x86_64}×{-posix,-win32} but we only need x86_64-posix, so just install that package.
2020-11-11 00:10:09 +00:00
// Windows builds (x64)
redo ci pipeline: * make cached docker containers for ci * remove old docker files from root of repo * use cached docker containers for ci
2021-02-26 17:26:48 +00:00
windows_cross_pipeline("Windows (amd64)", docker_base+'debian-win32-cross',
pass in WINDOWS_BUILD_NAME to uploader to mark 64bit and 32bit builds distinctly
2020-06-18 11:30:07 +00:00
toolchain='64', extra_cmds=[
DRY out ci upload script
2020-06-22 14:30:38 +00:00
'../contrib/ci/drone-static-upload.sh'
consolidate windows ci drone jizz
2020-06-17 13:42:11 +00:00
]),
drone tweaks - add some comments - exclude commits to the debian/ubuntu branches from getting built - disable "Updating man-db" which chews up several seconds during package install
2020-05-23 03:32:00 +00:00
Add tar+upload to builds.lokinet.dev of static build Relies on an SSH_KEY secret being set in the repository. Also move the linked library check to a bash script because escaping a value in jsonnet (escape once) that gets passed through yaml (escape twice) and then bash (escape again) is too painful.
2020-05-23 03:36:30 +00:00
// Static build (on bionic) which gets uploaded to builds.lokinet.dev:
redo ci pipeline: * make cached docker containers for ci * remove old docker files from root of repo * use cached docker containers for ci
2021-02-26 17:26:48 +00:00
debian_pipeline("Static (bionic amd64)", docker_base+'ubuntu-bionic', deps='g++-8 python3-dev automake libtool', lto=true,
static build fixes in cmake stuff I hate cmake so much.
2020-06-16 22:17:41 +00:00
cmake_extra='-DBUILD_STATIC_DEPS=ON -DBUILD_SHARED_LIBS=OFF -DSTATIC_LINK=ON -DCMAKE_C_COMPILER=gcc-8 -DCMAKE_CXX_COMPILER=g++-8 ' +
Don't use -march=native for static linux build (#1390)
2020-10-03 19:23:13 +00:00
'-DCMAKE_CXX_FLAGS="-march=x86-64 -mtune=haswell" -DCMAKE_C_FLAGS="-march=x86-64 -mtune=haswell" -DNATIVE_BUILD=OFF ' +
Merge remote-tracking branch 'origin/stable' into dev
2020-11-06 18:48:28 +00:00
'-DWITH_SYSTEMD=OFF',
Add tar+upload to builds.lokinet.dev of static build Relies on an SSH_KEY secret being set in the repository. Also move the linked library check to a bash script because escaping a value in jsonnet (escape once) that gets passed through yaml (escape twice) and then bash (escape again) is too painful.
2020-05-23 03:36:30 +00:00
extra_cmds=[
'../contrib/ci/drone-check-static-libs.sh',
'../contrib/ci/drone-static-upload.sh'
]),
add pybind and router hive to ci
2020-06-08 19:15:54 +00:00
// integration tests
debian_pipeline("Router Hive", "ubuntu:focal", deps='python3-dev python3-pytest python3-pybind11 ' + default_deps,
disable running router hive just build it
2020-08-28 13:59:29 +00:00
cmake_extra='-DWITH_HIVE=ON'),
add pybind and router hive to ci
2020-06-08 19:15:54 +00:00
Build (and upload) debs by merging into the deb branches Similar to the static files, the debs also end up under builds.lokinet.dev/deb.
2020-05-23 03:39:04 +00:00
// Deb builds:
deb_builder("debian:sid", "sid", "debian/sid"),
Fix deb snapshot builds + misc other CI updates (#1462) * Remove unneeded libcurl-dev build dep * Make apt-get operations quieter * apt repo updates - store the apt repo gpg pubkey in the repo rather than having to fetch it every time - add the apt repo for all the deb builds - update apt repo URL to deb.loki.network instead of deb.imaginary.stream * Use http so that we don't need ca-certificates The repo packages are already signed so using https really isn't important. * Add hack to ignore jsonnet conflicts in deb builds * Add build machine name to build output * We don't install any shared libs currently
2020-11-10 10:25:37 +00:00
deb_builder("debian:buster", "buster", "debian/buster"),
Build (and upload) debs by merging into the deb branches Similar to the static files, the debs also end up under builds.lokinet.dev/deb.
2020-05-23 03:39:04 +00:00
deb_builder("ubuntu:focal", "focal", "ubuntu/focal"),
deb_builder("debian:sid", "sid", "debian/sid", arch='arm64'),
macos static build & upload
2020-05-23 14:46:45 +00:00
// Macos builds:
mac_builder('macOS (Release)'),
mac_builder('macOS (Debug)', build_type='Debug'),
static build flag for win/mac static builds in drone config
2020-06-17 00:36:18 +00:00
mac_builder('macOS (Static)', cmake_extra='-DBUILD_STATIC_DEPS=ON -DBUILD_SHARED_LIBS=OFF -DSTATIC_LINK=ON -DDOWNLOAD_SODIUM=FORCE -DDOWNLOAD_CURL=FORCE -DDOWNLOAD_UV=FORCE',
macos static build & upload
2020-05-23 14:46:45 +00:00
extra_cmds=[
'../contrib/ci/drone-check-static-libs.sh',
'../contrib/ci/drone-static-upload.sh'
]),
Add several drone ci builds
2020-05-15 23:29:05 +00:00
]
Reference in New Issue Copy Permalink