- oxen-logging updated to bump fmt version
- version bump oxen-logging to fix fmt version
- version bump oxen-mq to solve uniform distribution error
- misc errors introduced by above version bumps
- clang-format 14 -> 15
when setting libunbound's upstream dns, we need to not pass in the square braces of an ipv6 address.
we also net udp handles have ipv6 address for the local ip.
This allows bencode-dump.py autodetect hex input and decode it on the
fly, which is quite convenient when working with binary-containing
bencoded data strings.
in rpc client, contention on a null lock happened.
fix this by making the sending of pings always done in the logic
thread. this is done by wrapping the lambda we made with EventLoop::make_caller()
-- Moved all RPCServer initialization logic to rpcserver constructor
-- Fixed config logic, fxn binding to rpc address, fxn adding rpc cats
-- router hive failed CI/CD resulting from outdated reference to rpcBindAddr
-- ipc socket as default hidden from windows (for now)
refactored config endpoint
- added rpc call script (contrib/omq-rpc.py)
- added new fxns to .ini config stuff
- added delete .ini file functionality to config endpoint
- added edge case control for config endpoint
add commented out line in clang-form for header reorg later
* Updated RpcServer Initialization and Logic
-- Moved all RPCServer initialization logic to rpcserver constructor
-- Fixed config logic, fxn binding to rpc address, fxn adding rpc cats
-- router hive failed CI/CD resulting from outdated reference to rpcBindAddr
-- ipc socket as default hidden from windows (for now)
Previously oxen-logging was erroneously hard-coded to use the target
"lokinet" for system logs. Obviously this is wrong for anything else
which uses oxen-logging and the system log. This changes our call to
add_sink to pass "lokinet" as the target rather than the config
filename, and updates oxen-logging to use that argument correctly.
Default & Required makes no sense: if we have a default it makes no
sense to make it required. The previous behaviour when this was
specified was to force an (uncommented) value in the config with the
value, but this was only used in the test suite.
Required & Hidden makes no sense either: if it's required to be
specified we definitely don't want to hide it from the generated config
file.
These are now compile-time failures.
When running as a service node we can't do anything without a lokid rpc
URL, and we don't necessarily have a good default for it.
This makes it required so that we fail with an appropriate error message
(rather than connect timeouts) if it is not specified.
At some point between 0.9.9 and 0.9.10 we removed the printing of option
names when a value doesn't have a default, but this means the config is
littered with things like:
# This option sets the greater foo value.
with no actual option name printed out when there is no default.
This fixes it by always printing the option name in such a case, just
with an empty value, e.g.:
# This option sets the greater foo value.
#big-foo=
The bg has to get encoded in a multi-format TIFF to make it work.
Also increase the vertical size a bit so that it still looks okay in
case you are a crazy person with a bunch of toolbars and other junk
cluttering up the window.
certain files needed to include either fstream and our shim for std::filesystem.
this includes fstream into our shim and includes this shim in places
that require fstream. this is done because some toolchains (cough
cough broke af arch linux amalgums) can have weird subsets of the
requirements of C++17 that overlap, except when they dont, denoted by
unknowable undisclosed circumstances.
this issue was reported by a user in the wild, and this fixes it.
previously we had a checking style function that passes in an optional
defaulting to nullopt as a micro optimzation, this makes the code
unnessarily obtuse.
simplify this by splitting up into 2 functions,
one for getting the unique endpoints and one for checking if the
number of them is above the minimum.
add overload for ReadyToDoLookup() that checks against constant but
can do more in the future if desired to reduce the burden on future contributors.
Query->Cancel() will remove the Query, but that introduces a race
condition where unbound may still try to invoke the callback (with a
no-longer-valid pointer) if we do it before the ub_ctx_delete call.
Move to it afterwards so that we only cancel things that unbound didn't
Occasionally during shutdown windivert will crash because a thread tries
sending after we've called wd::shutdown, which isn't allowed. Add an
atomic bool to prevent this.
we were calling llarp::Context::HandleSignal from a non mainloop
thread when running as a win32 service. this caused issues with a non
clean destruction.
call our signal handler instead of llarp::Context::HandleSignal
Get rid of the --win32-daemon hack (which was removed from the service
itself earlier in this PR, by mistake) and replace it with detection of
the error code for "not running as a service" that windows gives us back
if we try to set up service controller dispatching but aren't a service.
If wintun fails it seems to take about 15s, so extend the startup
timeout so that it can fail gracefully (and let us clean up before
exiting).
Also refactors the timeouts to chrono constants.
Fixes windows shutdown crashes:
- windivert wasn't handling an ERROR_NO_DATA, which it gets when
finished handling everything after a shutdown.
- wintun ReadPacket still gets invoked after end_session is called, but
shouldn't be. This adds an atomic<bool> to early return.
- fixes up some settings we send for windows service manager notify
- win32_platform.cpp is dead
- win32_platform.hpp is useless
Style changes from clang-tidy warnings:
- remove `virtual` from some definitions that already have `override`
- remove virtual destructor from NetworkInterface because it already has
a virtual destructor via the base type (and clang-tiny warns about it)
the win32 and sd_notify components provided a disjointed set of
similar high level functionality so we consolidate these duplicate
code paths into one that has the same lifecycle regardless of platform
to reduce complexity of this feature.
this new component is responsible for reporting state changes to the
system layer and optionally propagating state change to lokinet
requested by the system layer (used by windows service).
We should send STOP_PENDING rather than STOPPED while we aren't yet
actually stopped; STOPPED is already sent when we actually finish
stopping.
Also fixes some silly argc/argv shenanigans, I think.
We're defining formats for std::chrono types, which feels wrong (because
fmt itself also has these), so just replace them with functions:
short_time_from_now(...) gives a short "in 14m12s" or "5.123s ago" time
span relative to now, given a time point. Precision gets reduced for
larger deviations from now (e.g. "4h12m ago").
ToString(Duration_t) gives a string such as "-3h22m02.123s" for a
duration.
The time_delta<T> was using the wrong duration type when formatting, so
was outputting millisecond precision in the systemd status string which
is pointless (and unintended).
Currently you can't use GUI_EXE without BUILD_GUI, but BUILD_GUI also
requires the yarn command (even though it will never use it when GUI_EXE
is set).
This commit fixes it:
- Make `GUI_EXE` a windows-only top-level project options, rather than
being guarded by `BUILD_GUI`.
- Make `BUILD_GUI` control *building* the GUI instead of bundling it.
- GUI_EXE and BUILD_GUI are now mutually exclusive.
The options we need in rsvg-convert are apparently too new for bullseye,
so split the build so that we do the gui separately (in the nodejs-lts
container) and then build lokinet in bookworm.
The iterator here to skip an obsolete bootstrap wasn't properly
reassigning the iterator, so "didn't work" (though why it was hanging
for me is entirely non-obvious).
Also refactored it to simplify/clarify it a bit.
- Move logging initialization to early in Configure rather than at the
end of FromConfig so that we can add debug logging inside
Configure/FromConfig/etc.
- add said debug logging to Configure/FromConfig/etc.
Without this, old config (with now-irrelevant settings) won't work in
newer lokinet, making lokinet fatal error on startup if one of the
no-longer-used options is still present.
when read/writing a .loki privkey file we dont rewind a llarp_buffer_t
after use. this is an argument in favor of just removing that type
from the code entirely.
fixes by using 2 distinct locally scoped llarp_buffer_t, one for read,
one for write.
Set -D_WIN32_WINNT for static deps; unbound, in particular, needs this
as the latest version appears to rely on something only provided in
non-ancient windows to build properly.
This required moving _winver into the toolchain file so that it is
available earlier in cmake code (StaticBuild is included long before
win32.cmake), but also this seems a more appropriate place for it.
- cd .. after the build, before running extra_cmds, because the scripts
we invoke expect to be in the root, not in the build dir (and it's
dirtier for the build function to not undo the `cd build` that it
runs).
- fix unclosed parenthesis in mac static lib checker
cpr: 1.9.2
cxxopts: 3.0.0
ghc-filesystem: 1.5.12
nlohmann-json: 3.11.2
pybind11: 2.10.0
sqlite_orm: 1.7.1
Plus other updates need to make these work:
- cpr needs a cprver.h configured with the version (cmake code copied
from oxen-core).
Currently (from a recent PR) we aren't pinging oxend if not active, but
that behaviour ended up being quite wrong because lokinet needs to ping
even when decommissioned or deregistered (when decommissioned we need
the ping to get commissioned again, and if not registered we need the
ping to get past the "lokinet isn't pinging" nag screen to prepare a
registration).
This considerably revises the pinging behaviour:
- We ping oxend *unless* there is a specific error with our connections
(i.e. we *should* be establishing peer connections but don't have any)
- If we do have such an error, we send a new oxend "error" ping to
report the error to oxend and get oxend to hold off on sending uptime
proofs.
Along the way this also changes how we handle the current node state:
instead of just tracking deregistered/decommissioned, we now track three
states:
- LooksRegistered -- which means the SN is known to the network (but not
necessarily active or fully staked)
- LooksFunded -- which means it is known *and* is fully funded, but not
necessarily active
- LooksDecommissioned -- which means it is known, funded, and not
currently active (which implies decommissioned).
The funded (or more precisely, unfunded) state is now tracked in
rc_lookup_handler in a "greenlist" -- i.e. new SNs that are so new (i.e.
"green") that they aren't even fully staked or active yet.
This aligns service node updating logic a bit closer to what happens in
storage server, and should make it a bit more resilient, hopefully
tracking down the (off-Github) reported issue where lokinet sometimes
doesn't see itself as active.
- Initiate a service node list update in the 30s timer lokinet ping
timer (in case we miss a block notify for some reason); although this
is expensive, the next point mitigates it:
- Retrieve the block hash with the SN state update, and feed it back
into the next get_service_nodes call (as "poll_block_hash") so that
oxend just sends back a mostly-empty response when the block hasn't
changed, allowing both oxend and lokinet to skip nearly all of the
work of a service node list update when the block hasn't changed since
the last poll. (This was already partially implemenated--we were
already looking for "unchanged"--but without a block hash to get from
and pass back to oxend we'd never actually get an "unchanged" result).
- Tighten up the service node list handling by moving the "unchanged"
handling into the get_service_nodes response handler: this way the
HandleNewServiceNodeList function is only handling the list but not
the logic as to whether there actually is a new list or not.
Lots and lots of places in the code had broken < operators because they
are returning something like:
foo < other.foo or bar < other.bar;
but this breaks both the strict weak ordering requirements that are
required for the "Compare" requirement for things like
std::map/set/priority_queue.
For example:
a = {.foo=1, .bar=3}
b = {.foo=3, .bar=1}
does not have an ordering over a and b (both `a < b` and `b < a` are
satisfied at the same time).
This needs to be instead something like:
foo < other.foo or (foo == other.foo and bar < other.bar)
but that's a bit clunkier, and it is easier to use std::tie for tuple's
built-in < comparison which does the right thing:
std::tie(foo, bar) < std::tie(other.foo, other.bar)
(Initially I noticed this in SockAddr/sockaddr_in6, but upon further
investigation this extends to the major of multi-field `operator<`'s.)
This fixes it by using std::tie (or something similar) everywhere we are
doing multi-field inequalities.
The non-mac icon was an old version with white foreground and a
completely transparent background, but this looks bad (or invisible)
depending on where you view it. This updates it based on the macos
icon, but with a round white circle background instead of the macos
"squircle" background.
This also replaces the .ico file for the installer with one that we
build during the win32 build rather than a pregenerated one.
Bumps the gui as well to a version with the new icons in place.
If we get back an IPv6 address as the first gateway then we won't have
the expected IPv4 gateway that the route poker needs to operate.
This iterates through them separately so that we treat the IPv4 and IPv6
sides of an address as separate interfaces which should allow the route
poker to find the one it wants (and just skip the IPv6 one).
DRY a chunk of repeated code for finding a free private range.
Also fix it so that it will consider 10.255.0.1/16 and 192.168.255.1/24
(previously it would only check up to octet 254).
If running as a service node, we ping core on a regular interval to
inform it we're running and in a good state. If we're an active
(not decommissioned or deregistered) service node and have too few
peers and thus we're not actually connected to lokinet, we should skip
that ping so core doesn't think we're ok.
Adds a fallback bootstrap file path parameter to CMake, specify
-DBOOTSTRAP_SYSTEM_PATH="/path/to/file" to use.
Adds a list of (currently 1) obsolete bootstrap RouterIDs to check
bootstrap RCs against. Will not use bootstrap RCs if they're on that
list.
Log an error periodically if we appear to be an active service node but
have fewer than a set number (5) known peers.
Bumps oxen-logging version for literal _format.
The `sign` target on macos was not working properly -- the signing
script would run before the build is finished. This was caused by
cmake/macos.cmake having an `if(BUILD_GUI)`, but BUILD_GUI isn't defined
as an option until cmake/gui.cmake, which hadn't been included yet where
macos.cmake was included.
This extracts just the `option(BUIL_GUI)` from gui.cmake into a separate
gui-option.cmake file that we can load earlier to fix it.
While here I also noticed the GUI_EXE setting was defined as an option,
but isn't actually a boolean value, as an option, but isn't actually a
boolean value, so fixed it by making it a `set(... CACHE FILEPATH ...)`.
If you stop/start the GUI but it doesn't exit on start, the second
--start (when lokinet is already running) waits for a state change that
doesn't come (because lokinet is already running). This add a check for
already-running so that we exit right away in such a case.
The tools to create a dmg on Apple are flakey, of course, and fail in
cryptic ways if the file already exists, so purge it in the
contrib/mac.sh script.
No more llarp_buffer_t here!
(I was tracking down a segfault which led me in here and it was easier
to rewrite this to use bt_dict_{consumer,producer} than to decipher all
the cursed llarp_buffer_t and bencode callback nest).
We have basically this same bit of code in tons of places; consolidate
it into llarp::util::slurp_file/llarp::util::dump_file.
Also renames all the extra junk that crept into llarp/util/fs.hpp out of
there into llarp/util/file.hpp instead.
The old one was way too big on mac relative to other icons. This scales
the background down, while keeping the black logo parts the same, and
changes the rounding of the corners to match native macos apps.
This also rewrites it from scratch to use a useful coordinate system
which allows drawing all the fundamentals in much more useful units.
- Add a function to extract a value from parsed options, to DRY out the
code a little bit.
- Add a exit_error function to format a message to stdout and then
return the code, to simplify the repeated print-and-return code used
when errors occur.
- Use fmt for output formatting
- Add an error if multiple modes are specified at once
(--up/--down/--status/--exit)
- Add error printing around unmap
- Accept empty string or `null` for token to mean "no token."
- Accept `null` for range to mean "default range."
- Don't use a default range (::0/0) in lokinet-vpn because this will
fail if IPv6 ranges aren't supported on the platform (e.g. on
Windows), and isn't necessary: if we omit it then the rpc code already
uses ::0/0 or 0.0.0.0/0 by default, as needed.
This is not likely to be usable to many people, and people who it *is*
useful for are knowledgeable enough to modify it themselves. Most users
get no use at all and it most likely just confuses them instead.
- ANDROID_NDK_ROOT must be set in env
- cmake should be setting `-DANDROID_API=23`
- specify the correct android API via a define when building openssl; it
has to be in CPPFLAGS (not CFLAGS) because otherwise openssl's
configure script doesn't notice and overrides our define with the
latest API version.
- openssl configure puts $(ANDROID_NDK_ROOT) in the makefile, so we have
to be sure that we put it in the environment for the build command,
too.
- Split up mac.sh into a configure + build scripts (like Windows).
- Don't attempt to build the 'package' target in CI: apparently you have
to have a logged in user at the GUI in order to build a .dmg because
being obtuse is the Apple way.
- Upload the raw Lokinet unsigned app in a .tar.xz, rather than dmg,
because of the above.
- make mac.sh respect JOBS (pun not intended (but still good))
- ReconfigureDNS wasn't returning the old servers; made it void instead
(the Apple code can just store a copy of the original upstream
servers instead).
- Reconfiguring DNS reset the unbound context but didn't replace it, so
a Down()/Up() would crash.
- Simplify Resolver() destructor to just call Down(), and make it final
just so that no one tries to inherit from us (so that calling a
virtual function from the destructor is safe).
- Rename CancelPendingQueries() to Down(); the former cancelled but also
shut down the object, so the name seemed a bit misleading.
- Rename SetInternalState in Resolver_Base to ResetResolver, so that we
aren't conflicting with ResetInternalState from Endpoint (which was a
problem because TunEndpoint inherited from both; it could be resolved
through the different argument type if we removed the default, but
that seems gross).
- Make Resolver use a bare unbound context pointer rather than a
shared_ptr; since Resolver (now) entirely manages it already we don't
need an extra management layer, and it saves a bunch of `.get()`s.
On Apple, the network extension is outside the tunnel routing, so we
cannot have libunbound talk directly to upstream (it would leak DNS when
exit mode is enabled). Instead unbound *always* talks to a localhost
port where we have a "dns trampoline" that takes UDP packets and shoves
them through the tunnel.
We were doing that already, but recent changes here were overwriting the
libunbound settings with.
This also moves the upstream DNS configuration part of `Up()` into its
own method.
We don't have a resolver on macos, so we were running through this loop
with fails == 0 == m_Impls.size() and throwing, crashing the process.
Early return to avoid the failure and fix macos crash.
Apple supports anything here that Clang supports and should have them
set the same as everywhere else.
Most importantly this gives apple the -Wno-deprecated-declarations flag
which has been driving me nuts on macos.
This also version-gates the -Wno-deprecated-declarations so that it
will turn on again when we bump the version beyond .10.
We were requiring `->Next` be true, which means we skipped the last (and
often only) entry of the linked lists and so never properly found the
gateway.
- We need to pass a flag to get Windows to include gateway info.
- Refactor it to use microsoft's recommended magic default 15000 buffer
size and repeat in a loop a few times until it works. Developers,
developers, developers, developers!
- a `static` is less verbose and otherwise identical to an empty
namespace for a single declaration like this.
- operator== on two optionals already does exactly what the `is_equal`
lambda here is doing.
- formatting
- windivert was being set up *before* DNS is set up, so the DNS port was
nullopt and thus we couldn't properly identify upstream DNS traffic.
- close() doesn't close a socket on Windows, so the socket-bind-close
approach to get a free UDP port wasn't actually closing, and thus
unbound upstream constrained to the given port were completely
failing.
- The unbound thread was accessing the same shared_ptr instance as the
outer code, which isn't thread-safe; changed it to copy a weak_ptr
into the lambda instead.
- Exclude upstream DNS traffic in the filter rather than capturing and
reinjecting it.
The inner lambda here wasn't keeping the `Query` (`this`) alive, so
`src` wasn't valid anymore. This changes it to copy the `src`
shared_ptr into the lambda instead of capturing `this`, and fixes it.
The current code isn't working and gives a 0 (which then fails unbound
initialization). This replaces it by doing a socket+bind to find a free
port then immediately closes (but passes the port we got into unbound).
- Replaces RAII handling of DLLs with global function pointers. (We
don't unload the dll this way, but that seems unnecessary anyway).
- Simplifies code by just needing to call an init function, but not
needing to pass around an object holding the function pointers.
- Adds a templated dll loader that takes the dll and a list of
name/pointer pairs to load the dll and set the pointers in one shot.
ip_header wasn't 20 bytes on windows compilations for some unholy
reason. This restructures it to avoid the template and just use two
different structs for le/be with a condition_t for the ifdef, which
resolves it (and *also* apparently avoids the need for the pack).
Also add a static_assert to check the size.
Also do the same for ipv6.
Cast via an ordinary function pointer rather than a function pointer
reference to avoid the warning.
Also make the pointer in `Func_t` explicit rather than implicit (deduced
into the `Func_t` type) to make it clearer what is going on here.
3.13...3.xx means "minimum is 3.13, but use any new cmake policies
introduced up to 3.xx".
There was, in particular, a policy w.r.t. external project timestamps
causing warnings under 3.24.
Lots of tools struggle with non-default DNS port, so keep a listener on
127.3.2.1:53 (by default).
This required various changes to the config handling to hold a vector
(instead of an optional) of defaults and values, and now allows passing
in an array of defaults instead of just a single default.
It didn't do equality, it did "does the remaining space start with the
argument" (and so the replacement in the previous commit was broken).
This renames it to avoid the confusion and restores to what it was doing
on dev.
errno is only set if read returns < 0 and won't be set to 0 if read
succeeds, so we were bailing here frequently on successful reads
(whenever errno happened to be non-0).
This class is cursed, but also broken under gcc-12. Apply some lipstick
to get it moving again (but we really need to refactor this because it
is a mess).
add jason's suggested changes for artifact upload
use lokinet-ci-nodejs-lts as base image so we can build the installer
update ci pipeline for windows to have building gui toggle-able
by default we will build the gui from this repo, but this allows it to
easily run using a custom gui asset if needed
* wintun vpn platform for windows
* bundle config snippets into nsis installer for exit node, keyfile persisting, reduced hops mode.
* use wintun for vpn platform
* isolate all windows platform specific code into their own compilation units and libraries
* split up internal libraries into more specific components
* rename liblokinet.a target to liblokinet-amalgum.a to elimiate ambiguity with liblokinet.so
* DNS platform for win32
* rename llarp/ev/ev_libuv.{c,h}pp to llarp/ev/libuv.{c,h}pp as the old name was idiotic
* split up net platform into win32 and posix specific compilation units
* rename lokinet_init.c to easter_eggs.cpp as that is what they are for and it does not need to be a c compilation target
* add cmake option STRIP_SYMBOLS for seperating out debug symbols for windows builds
* intercept dns traffic on all interfaces on windows using windivert and feed it into lokinet
* allow specifying a custom yarn binary for building the gui using -DYARN= cmake option
* unset DISPLAY when calling wine because i hate popups
* do not rebuild gui when building for windows
* by setting the magical undocumented env var USE_SYSTEM_7ZA to 'true' we can have the pile of npm bullshit code use our system's local 7z binary instead of the probably not backdoored binary from npm, yes for real. i hate nodejs so god damn much you have no fucking idea
* allow providing a custom gui from a zip file via -DGUI_ZIP_FILE cmake option
we want to be able to have multiple locally bound dns sockets in lokinet so
i restructured most of the dns subsystem in order to make this easier.
specifically, we have a new structure to dns subsystem:
* dns::QueryJob_Base
base type for holding a dns query and response with virtual methods
in charge of sending a reply to whoever requested.
* dns::PacketSource_Base
base type for reading and writing dns messages to and from wherever they came from
* dns::Resolver_Base
base type for filtering and handling of dns messages asynchronously.
* dns::Server
contextualized per endpoint dns object, responsible for all dns related isms.
this change hides all impelementation details of all of the dns components.
adds some more helper functions for parsing dns and dealing with OwnedBuffer.
overall dns becomes less of a pain with this new structure. probably.
Even if we aren't codesigning, things like the `package` target expect
to be able to depend on `notarize` (and thus implicitly sign ->
assemble) to require a built package.
Also add a `-UNSIGNED` into the built dmg filename.
Apple introduced a bug in macOS 11 that they can't be arsed to fix which
breaks PNG loading into icns files by dropping the blue channel of the
last pixel, leaving a streak of yellow pixels at the bottom of the
image.
This hacks around it by setting a fully transparent, non-white (actually
yellow) pixel in the bottom-right corner of the images.
This is such inexcusable trash.
- Add the version number into the .dmg filename
- Set the lokinet icon on the .dmg. This is done via a swift program
because all the Apple CLI tools to do this are deprecated.
The macOS PR that was merged accidentally dropped a cmake option that
result in the extension's provisioning profile not getting copied into
place (but was working locally because I was using a build dir where the
variable was still set). This restores the option to fix the
codesigning.
CMake apparently doesn't do anything with CMAKE_OSX_DEPLOYMENT_TARGET
for swift, which results in a 12+ minimum version. This fixes it
(albeit in a hacky way since the only apple-sanctioned way to properly
set this appears to be "use xcode").
Shame on Apple, as usual.
* make socket bind errors have a distinct message reported when caught using their own exception type
* omit printing banner in setup when we run from the lokinet executable (but not the liblokinet.so entry point)
Apple's servers have a gateway timeout a small but noticeable percentage
of the time, which was breaking the script. Detect such Apple flakiness
and keep trying.
Adds support for building Lokinet as a system extension, and fixes
various problems in the macos implementation found during development of
the system extension support.
outbound=:1234
outbound=0.0.0.0:1234
outbound=
outbound=0.0.0.0
now all default to use the inbound= IP. (If multiple inbound= IPs are
given, we raise an exception to abort startup).
Only applies to routers (since clients don't have inbound IPs), and
eliminates potential weird edge cases with local system IP and routing
shenanigans.
The general section comments contained all the descriptions for the
inbound/outbound settings, while inbound/outbound had no comment at all.
This moves the comments around to the individual settings, plus updates
some of the wording in the section.
We were failing when using `inbound=:1234`, rather than looking for a
default IP. This fixes it to still use the default IP, and change only
the default port.
Outbound behaviour should remain unchanged: i.e. `outbound=:2345` means
bind-to-wildcard-IP with a specific port.
Fixes:
- tighten reserved name detection to not match fooloki.loki, but instead
only match "foo.loki.loki" and "loki.loki" (and similar for reserved
name "snode.loki").
- IPv6 PTR parsing was completely broken.
- Added tests for the above two issues.
Cleanups:
- Eliminate llarp::dns::Name_t typedef for std::string
- Use optional return instead of bool + output param
- Use string_views; we were doing a *lot* of string substr's during
parsing, each of which allocates a new string.
- Use fmt instead of stringstream
- Simplify IPv4 PTR parsing
Using constructor inheritance DRYs the code, but unfortunately confuses
GCC as to where the proper "required from here" location is, which makes
debugging formatting errors very difficult. Avoid it (and update
oxen-logging to avoid it there as well).
Replaces custom logging system with spdlog-based oxen logging. This
commit mainly replaces the backend logging with the spdlog-based system,
but doesn't (yet) convert all the existing LogWarn, etc. to use the new
format-based logging.
New logging statements will look like:
llarp::log::warning(cat, "blah: {}", val);
where `cat` should be set up in each .cpp or cluster of .cpp files, as
described in the oxen-logging README.
As part of spdlog we get fmt, which gives us nice format strings, where
are applied generously in this commit.
Making types printable now requires two steps:
- add a ToString() method
- add this specialization:
template <>
constexpr inline bool llarp::IsToStringFormattable<llarp::Whatever> = true;
This will then allow the type to be printed as a "{}" value in a
fmt::format string. This is applied to all our printable types here,
and all of the `operator<<` are removed.
This commit also:
- replaces various uses of `operator<<` to ToString()
- replaces various uses of std::stringstream with either fmt::format or
plain std::string
- Rename some to_string and toString() methods to ToString() for
consistency (and to work with fmt)
- Replace `stringify(...)` and `make_exception` usage with fmt::format
(and remove stringify/make_exception from util/str.hpp).
Having it there (even defaulted, like this) means endpoint.hpp doesn't
have to include endpoint_state.hpp (which it otherwise would need,
because of the std::unique_ptr<EndpointState> default deleter
requirements).
We only check for definedness, not truth, in the code so make the cmake
definitions agree with that.
This also avoids warnings when building on macos (because swift only
allowed defined/undefined but not values)
We shouldn't be compiling these .cpp files at all on other platforms,
rather than compiling empty .cpp files (which later results in "... has
no symbols" warnings).
Currently I maintain a patch in the debs to do the same thing here, but
it fails to apply often enough; this change makes the behaviour
consistent with oxen-core/oxen-ss and will let me drop that patch and
just pass in the cmake option.
(Recommend ignore-whitespace for viewing the diff)
Fixes a bug on older cmake linking against oxenmq (older cmake hates the
direct oxenmq::oxenmq -> PkgConfig::OXENMQ alias), and also makes it
easier to handle things like nlohmann::json (which we can use from the
system *or* submodule).
Borrowed from oxen-ss/oxen-core.
bump submodules
Fix dynamic version generation
- GIT_FOUND OR Git_FOUND wasn't available because we hadn't done a
find_package(Git) yet.
- required version variables weren't being passed through to the cmake
script
clean up version cmake stuff
clean up generated cpp version stuff
make all the windows rc stuff get generated by cmake
bump release motto message
properly inject release motto into version
if a pending inbound session did not complete a handshake after an unclean close from a previous session the
remote udp endpoint would remain stuck mapped as authed and thus any further attempts from the remote would
be silently dropped as it entered a stuck state in the state machine. this was happening as a small part
of the state machine was hidden in the implementation details of iwp, but instead should be in the super type
as it is logic exclusively outside the details which every dialect would have regardless of its details.
this commit will unmap the udp endpoint every time it needs to in the link layer state machine, independat of
the implementation details of the diact.
only send close packet once, before we were sending a close after we got a close causing excess log spam.
include handshake phase when checking for connection timeouts.
when we change our rc make sure to put it into nodedb too when we are a service node to prevent weirdness in dht lookups.
this allows you to use exit nodes without forcing routes over the interface, useful for using lokinet with an exit and selectively routing over the lokinet interface using an external socks proxy or binding to device explicitly.
* make route poker configurable, defaults to enabled but allows disabling it on runtime if desired
* add config option [network]:auto-routing to enable/disable route poker
on win32/apple reading packets from the interface does not count as an io operation.
manually trigger pump on win32/apple to pretend that it is an io event.
add platform quark function MaybeWakeUpperLayers on vpn::Interface to manaully wake up the other components on platforms that need that (ones on which packet io is not done via io events).
on non linux platforms, use uv_prepare_t instead of uv_check_t as the former triggers before blocking for io, instead of after. this better matches linux's order of operations in libuv.
We are calling time_now() a huge amount, and it is a major consumer of
CPU cycles, but we don't need it: most of the time the current event
loop time is enough.
We have a few cases where we're making an extra shared_ptr which we copy
into a lambda, which then results in an extra unnecessary refcount
decrement in the parent; this changes them to give an rvalue reference
to the lambda to avoid the extra incr/decr instead.
The one in Session::Pump is particularly noticeable and shows up in
profiling.
- Replace m_FlushWakeup with a call to the router's god mode pump
method. m_FlushWakeup apparently isn't enough to get things out, and
we can end up with incoming packets that don't get properly handled
right away without it.
- The shared_ptr around the ihophandler queues isn't needed and is just
adding a layer of obfuscation; instead just exchange the list directly
into the lambda.
- Use std::exchange rather than swap
- A couple other small code cleanups.
The TriggerPump just below this is *already* going to trigger a flush,
so the extra flush call here can't do anything useful (and in
particular, it won't clear up the queue *immediately*, which is what
this code looks like it was aimed at doing).
- Make the main PumpLL also pump hidden services, rather than using
separate wakers in each TunEndpoint. It seems there is some
interactions that just one or the other is not enough.
- Eliminate TunEndpoint send queue -- it isn't needed as we can just
send directly.
If something needs to wake up the event loop it should be using an
async, as we are now with PumpLL(); but we had various code triggering a
wakeup, expecting that PumpLL gets called on every wakeup, which isn't
true anymore.
We trigger a pump immediately, but this is racey because we add to our
plaintext data in a worker thread; if the worker thread runs after the
pump then it ends up leaving plaintext to be handled, but there's no
wakeup until the next one.
This was the cause of seeing a random +1s and bunching added to ping
responses sometimes: it wasn't until the *next* ping goes through the
network that the plaintext queue gets processed, at which point it
flushes the old one and often the new one together.
The fix here gets rid of the map of sessions needing wakeups and instead
adds an atomic flag to all of them to let us figure out which ones
need to be flushed.
call_soon wakes up the mainloop and is often reentrant, we dont want to
ever way up the event loop but we also want to always defer the call so
we always use the workers
- split debian sid into base/sid/clang images
- similarly for debian stable
- add jsonnet to lint
- add `--pull` to docker build so that we always pull the latest images
(otherwise we were building on whatever local cache we have for
`debian:sid`, etc., which made the base image update layer much
larger).
- don't install Recommends by default
- add libzmq3-dev
- split android into android (base) and flutter
- hard-code registry.oxen.rocks into the dockerfile stuff because that
seems to be the only way to properly depend on other docker builds.
- update a few CI builds that should have been using our images but
weren't.
- Update a few CI images to bullseye instead of buster
Add openssh-client (for sftp to upload builds)
`which` is a debian tool that is being deprecated in favour of
posix-standard `command -v`, so which to that to avoid deprecation
warnings.
Change the exit codes of `contrib/format.sh verify` to be unique for
each formatting program.
* attempt path timeout bullshittery fix
* make sure ServiceInfo always has its address set up
* do not copy intros in constuctor, ammend logging and
add assert
* use std::source_location instead of godawful macros in logging
* remove unused/absolutely haram af json logstream
* fix bug in android logger where it doesn't respect eLogNone
We were linking/loading it in different ways, one with cmake option
`USE_JELLOC` and the other, older version `WITH_JEMALLOC`. This removes
the latter (which was default OFF) and keeps the former (which was added
and has been default ON since 0.9.4 or so).
Also removes the `ifdef`ed JEMALLOC code in lokinet.cpp because we don't
need it; just linking to jemalloc is enough to get the malloc/free
replacements.
We use them everywhere now (include the debs) so it makes sense to have
them bundled here rather than needing a download during package build
(in particular for debs that's considered a bad thing).
ngtcp2's top-level CMakeLists.txt is not friendly to being used as a
subdirectory (for instance, it always adds a `check` target when cunit
is installed), so stop using it in favour of skipping directly into the
ngtcp2/lib subdir.
This requires some hackery to set up a couple of the things the
top-level cmake does, but it isn't *too* painful.
It's failing to link with LTO on bionic because of an internal compiler
error for some random reason, so just disable building the tests there
for now (keeping LTO on is more useful since that is also what gets used
for a static build).
before when we get the list of router ids for gossip it was highly bias towards outbound sessions.
instead now we get a full list of link session router ids in random order, truncate them to be at most MaxGossipPeers number of keys, and then put them into an unordered set
Removes stuff we didn't end up needing/using:
- Lokinet.modulemap
- apple bits from lokinet.cpp (we don't use lokinet.cpp at all on macos
anymore).
- dnsproxy/extension C++ headers
- apple-specific network extension config in llarp::config::Config
When we enable/disable exit mode on this restarts the unbound DNS
responder with the DNS trampoline (or restores upstream, when disabling)
to properly route DNS requests through the tunnel (because libunbound's
direct requests don't get tunneled because unbound is inside the network
extension).
This runs a DNS listener on localhost:1053 that bounces requests to the
upstream DNS through the tunnel. The idea here is that, when we turn on
exit mode, we start libunbound bouncing the requests through the
trampoline (since if it makes direct requests they won't go through the
tunnel).
(The actual libunbound configuration is still to follow).
Thus when a user goes looking for it they'll find the (commented out)
default in the right place and can edit it.
(That right place is: ~/Library/Containers/com.loki-project.lokinet.network-extension/Data/lokinet.ini)
Don't squash this commit so that the swift version stays around in
history in case we need to resurrect it again some day (i.e. when Apple
decides to kill off Objective-C support).
- Add a C callback interface (context_wrapper.h) between lokinet and the
objective-C code so that:
- we can use objective-C (rather than objective-C++), which seems more
likely to be supported by Apple into the future;
- we minimize the amount of code that needs to be aware of the Apple
APIs.
- this replaces apple logger objective c++ implementation with a plain
c++ implementation that takes a very simple C callback (provided
from the obj-c code) to actually make the call to NSLog.
- Add various documentation to the code of what is going on.
- Send all DNS traffic to the primary IP on the tun interface. The
match prefixes simply don't work as advertised, and have weird shit
(like even if you get it working for some domains, "instagram.com"
still doesn't because of god-knows-what Apple internal politics).
- Drop the dns proxy code as we don't need it anymore.
- Don't use 9.9.9.9 for default DNS. (We might consider the unfiltered
9.9.9.10 as an alternative default, but if we do it should be a global
lokinet change rather than a Mac-specific change).
- Parse a lokinet.ini in the data directory, if it exists. (Since we
are sandboxed, it is an app-specific "home" directory so is probably
buried god knows where, but at least the GUI ought to be able to get
it to let users add things to it).
- This commit also adds a swift version of the PacketTunnelProvider
glue, which ought to work in theory, but the *tooling* for cmake is so
underdeveloped that I couldn't find any way to actually get the damn
thing working. So I'm committing it here anyway (and will revert it
away in the next commit) in case we someday want to switch to it.
-
- Added contrib/macos/README.txt with description of the cancer
happening here.
- Add provisioningprofiles that Apple wants to make things work properly
- Made the entitlements files match the provisioningprofiles
- Remove configured entitlements files; we *can't* change any of the
things here because they are closedly tied to the provisioningprofiles
-- which means if someone wants to build their own Lokinet, they have
to replace a bunch of crap and change application IDs throughout.
This is the hostile-to-open-source Apple way.
- Remove unused old lokinet binary, as we're no longer using it on macos
- Use a POST_BUILD rather than install to copy things around into the
right places
- Convert all the configure_file's to consistently use @ONLY
- Misc cleanups
PR #1725 reversed argument orders but UnboundResolver was still using
(from,to) ordering in its callbacks, which leaked through to make a
wrong order in our reply function (which simply forwards arguments).
This fixes that bug by making UnboundResolver callback argument order
consistent (i.e. using to, from) with the PacketHandler argument order.
The reason the dns fix on android didn't work is that the DnsInterceptor
had a reversed to/from argument order for its
`SendServerMessageBufferTo` overload, and so android/mac needed the
to/from to be reversed so that the second reverse cancelled out the
first one.
Upon review, the DnsInterceptor order (to, from) is more intuitive than
the base order (from, to), so this reapplies the dns fix and swaps
everything *except* DnsInterceptor to match the (to, from) argument
order.
This reverts commit dace0224ec.
This reportedly didn't fix things on Android, and most definitely breaks
macOS (with this we get a bunch of errors about expecting inbound when
we have outbound).
- Reduce buffer size to INET6_ADDRSTRLEN, and use a single buf rather
than two identical ones in each branch.
- Don't pre-reserve because doing so is usually going to over-allocate,
but also because it prevents SSO, especially for the IPv4 case which
should fit in SSO for all IPv4 addresses.
The default upstream DNS was being set to 1.1.1.1:0, which doesn't work.
This fixes it to also set the port so that default upstream resolution
(i.e. with an empty config) works again.
This function had a bug in stable (fixed in dev) when `last` returns
npos, but the function also appears to basically be duplicating what the
next split version can do, so this just removes it and uses the single
more generic split(strview, strview) method.
* make it so that we don't set up unbound resolver when we have no resolvers provided by config
* clean up dns codepath and make it use llarp::SockAddr instead of llarp::IpAddress
* modify contrib/window.sh now uses makefiles
* create contrib/android.sh for building shared libs for android apk
* update drone ci jizz for android
* update cmake/StaticBuild.cmake to use $(MAKE) so that builds can parallelize (super epic)
* remove android directory as it is no longer needed
* add option to persist address mappings between restarts using [network]:persist-addrmap-file
* make it work
* only persist address map for inbound convos
* turn persisting address map on by default
* dont load addrmap file if it has been modified last over a minute ago to prevent foot cannons fired from loading a really old version of it
* when a path build times out, shitlist every router in the path except the first hop, this way eventually we get the nodedb pruned to only the routers that are currently actually alive, any ones we nuke that we need later we can always do lookups for.
* introset path haodver tweaks
* improve warn/error messages to convey more information
* dont block on queue insertion
* reset convotag on decrypt/verify fail
* add multiple ready hooks on outbound context
* lookup introsets from close routers on dht
* continue to tick dead sessions so they expire their paths
* introset spacing
* reduce lns lookup diversity requirement for speed
* add a function to send reset convotag message
* only have 1 outbound context at a time
* include stricter router profiling checks in path::Builder hop slection algorithm
* make intro selection function nicer by returning a std::optional instead of a bool with an "out" variable
* do FEC for latency tests so if we fail one test it doesn't kill the entire path
* ignore FEC'd responses on latency tests
* track latency history and report the mean latency instead of just the last sample
* add remove_node_from_failing to remove a node by pubkey from the failing set
* if a router is deregistered we remove it from the failing set so we don't retest it
* remove a router from the failing set if we get a test success
currently creating an outbound session will cancel if we have any session
at all with the relay. instead, only cancel if we have an outbound session
to that relay. this is useful for reachability testing.
It's there for polling, which we aren't doing anymore; we just got the
hash from oxend's push notification, so if it pass it in then we will
always get an "unchanged" result because we're telling oxend that we
already have the data for that hash updated.
This just drops the hash completely because we don't need it anymore.
* immediately poke routes when we are told to use an exit so that packets get pushed which makes an exit path happen
* fix up cmake oddity in nsis section
* not gossip our rc
* not explore the network to prevent outbound session attempts
* not establish sessions to other service nodes
* close all open sessions we have to tell clients we don't want them
* catch exceptions flushing peerdb in disk thread
* don't connect out to non allowed routers
* simplify logic in RCLookupHandler::RemoteIsAllowed()
* add HaveReceivedWhitelist to I_RCLookupHandler base type
* add LooksDeregistered to Router type that tells us if we think we are deregistered
* don't allow building paths over us if we are deregistered
so it doesn't time out and get into a state that's totally screwed.
add virtual function service::Endpont::DefaultPathAlignmentTimeout() to get the timeout for path alignment
and use it for resetablishing outbound sessions
* increase publish introset timeout so that it does not time out on the network
* remove pedantic log warn
* make sure the path we are using for replying on inbound sessions is alive
* include convotag in log message so we know wtf is going on
* appease tom's autism, improve log message text
adds [network] section parameter called path-alignment-timeout that allows configring the timeout
for optional name lookup + introset lookup + aligned path build, used by tun endpoint dns, provided
as milliseconds.
Fixes a subtle memory leak that was a result of outbound messages which
were in the shared queue (not yet sorted into a per-path queue) when a
path was removed, resulting in a ghost path queue (and thus round-robin
order entry as well).
Adds much needed documentation to the outbound message handler class.
Wires up systemd support to configure DNS on startup and when
enabling/disabling exit mode.
On startup (and when turning off an exit) we tell systemd-resolved to
direct .loki and .snode lookups to lokinet (leaving other DNS traffic
alone).
On exit enabling, we reconfigure it to resolve "." (i.e. the root DNS
domain) so that all lookups come into it.
The CI runners don't use fixed CPU cores anymore, so that they can
better allocate jobs across idle cores, but this means ninja's default
is running an insane number of parallel compilations that likely just
makes things slower.
Add 6-job limiting to match the core limits that used to be in place.
* add srv records in RCs if we have any
* add mechanism to add SRV records for plainquic exposed ports
* resign and republish rc or introset on srv record changes
* llarp::service::NameIsValid was not checking that the tld was .loki, add this check.
* make link layer initial connection timeout 5s not the session activity timeout which happens to be 60 god damn seconds.
* add lokinet_add_bootstrap_rc function for adding an rc from memory
* prevent stack overflow on error closing connection in quic
* add in memory nodedb
* refactor how convotags are set as active
* add initial stubs for endpoint statistics
* refactor time stuff to be a bit cleaner
* update lnproxy script with more arguments
Replace stream_reset (which typically isn't called) with a stream_close
handler (which is already called whether or not it was a reset). Most
importantly, the server side needs to extend the max bidi streams
counter during stream_close (otherwise we run out when we hit the
limit and new connections just stall).
Stream forward on the client-side TCP connection gets set up within in
initial_client_data_handler (which also handles reading the initial
stream version byte).
Also fix a potential transmission delay because `again()` wasn't being
called when the expiry is already passed (i.e. meaning we should run
immediately).
Make stream closing with expiring connections work better. Fixes an
issue where the stream's uv_async could outlive the stream and/or
connection and segfault.
In the standalone plainquic code we triggered a retransmit when the
socket became writeable again, but that doesn't work here, so just
schedule it right away to let ngtcp2 worry about retrying.
For now we still steal buffers from uvw.
In the future I'd like to change that, but it's still uvw
work-in-progress to support custom data allocators, and so for now we
still steal data buffers from uvw.
ngtcp2 was rejecting them because we have the port when constructing,
but then it was 0 on the return packet (which ngtcp2 drops because it's
coming from an unknown/invalid path).
* wire up last of the quic stuff
* clean up udp packet generation code
* pass EndpointBase not quic tunnel for quic stuff
* add {n,h}uint16_t::FromString
* add nuint_t::FromString
* make AlignedBuffer::IsZero non constant time call for speed
Refactors how quic packets get handled: the actual tunnels now live in
tunnel.hpp's TunnelManager which holds and manages all the quic<->tcp
tunnelling. service::Endpoint now holds a TunnelManager rather than a
quic::Server. We only need one quic server, but we need a separate quic
client instance per outgoing quic tunnel, and TunnelManager handles all
that glue now.
Adds QUIC packet handling to get to the right tunnel code. This
required multiplexing incoming quic packets, as follows:
Adds a very small quic tunnel packet header of 4 bytes:
[1, SPORT, ECN] for client->server packets, where SPORT is our
source "port" (really: just a uint16_t unique quic instance
identifier)
or
[2, DPORT, ECN] for server->client packets where the DPORT is the SPORT
from above.
(This also reworks ECN bits to get properly carried over lokinet.)
We don't need a destination/source port for the server-side because
there is only ever one quic server (and we know we're going to it when
the first byte of the header is 1).
Removes the config option for quic exposing ports; a full lokinet will
simply accept anything incoming on quic and tunnel it to the requested
port on the the local endpoint IP (this handler will come in a following
commit).
Replace ConvoTags with full addresses: we need to carry the port, as
well, which the ConvoTag can't give us, so change those to more general
SockAddrs from which we can extract both the ConvoTag *and* the port.
Add a pending connection queue along with new quic-side handlers to call
when a stream becomes available (TunnelManager uses this to wire up
pending incoming conns with quic streams as streams open up).
Completely get rid of tunnel_server/tunnel_client.cpp code; it is now
moved to tunnel.hpp.
Add listen()/forget() methods in TunnelManager for setting up quic
listening sockets (for liblokinet usage).
Add open()/close() methods in TunnelManager for spinning up new quic
clients for outgoing quic connections.
They were failing to compile if output because implicit integer
promotion doesn't work for std::byte's (but rather needs an explicit
std::to_integer call).
- LogTrace() (and LogTraceTag, etc.) are now no-ops for release builds.
(hoping there are no side effects in trace logging!)
- renamed llarp::_Log to llarp::_log because _Log is a reserved keyword
- change logging code to implicitly convert 1-byte types (char, unsigned
char, uint8_t) to ints so that we print them as numeric values rather
than raw chars because, more often than not, printing a single char is
trying to log an 8-bit value.
- Move uvw target to external instead of llarp/CMakeLists.txt
- Add sqlite_orm headers via interface library rather than shoving it
into the global include search path.
- Find sqlite3 as part of the sqlite_orm target deps rather than linking
it separately. (Also don't link it into lokinet-util, which doesn't
appear to actually need it).
* add ip6-range option to network section to control which range we want to use for ipv6 traffic
* make ip6-range able to disable ipv6 exits ( i dont like this but eh )
- Make SockAddr endian arguments explicit
- Consolidate port-less contructors and port constructors into one with
a default port of 0.
- Add {h,n}uint128_t ctors for construction from IPv6 addrs
- De-lambda a lambda that is only called once, immediately after it is
defined.
- C++17 modernize key-value for loops
- Remove unused (and useless) method
a file that was accidentally committed after being moved was
suddenly being chosen as an include target over the correct
one due to an amusing coincidence of cmake configuration. that
file is now gone.
All #ifndef guards on headers have been removed, I think,
in favor of #pragma once
Headers are now included as `#include "filename"` if the included file
resides in the same directory as the file including it, or any
subdirectory therein. Otherwise they are included as
`#include <project/top/dir/relative/path/filename>`
The above does not include system/os headers.
ip packet on read error was returning an uninitialized ip packet with non zero size so it was thinking it was a valid packet.
value initialize it to set the packet size to zero by default.
this fixed it in an emulator at least...
The formatting changes here are not too large and also nice: in
particular clang-format-11 appears to do better handling of `{...}`
constructor braces/arguments.
call(), when invoked from the logic thread, wasn't triggering a wakeup
which stalled some traffic (such as client-to-snode packets).
Fix it by triggering a wakeup on `call()`, and expose it because this is
a useful thing to be able to do.
The `const sockaddr*` conversion was returning a pointer to the
sockaddr_in6, but with its family set to AF_INET, so the receiving code
would have treated interpreting the result as a sockaddr_in pointer --
but it isn't, and trying to get the IPv4 from the mis-reinterpreted
sockaddr_in6 fails; this then prevented uvw UDP binding (which uses the
sockaddr* operator) from working.
This changes it to always leave the family value at the proper value and
using the ipv6_is_mapped_ipv4 check to determine when we have (and
should return) a pointer to the sockaddr_in rather than the
sockaddr_in6.
Also fixes toString() to compare the full prefix instead of just bytes
10 and 11.
loop->call(...) is similar to the old logic->Call(...), but is smart
about the current thread: if called from within the event loop it simply
runs the argument directly, otherwise it queues it.
Similarly most of the other event loop calls are also now thread-aware:
for example, `call_later(...)` can queue the job directly when called if
in the event loop rather than having to double-queue through the even
loop (once to call, then inside the call to initiate the time).
- removes all the llarp_ev_* functions, replacing with methods/classes/functions in the llarp
namespace.
- banish ev/ev.h to the void
- Passes various things by const lvalue ref, especially shared_ptr's that don't need to be copied
(to avoid an atomic refcount increment/decrement).
- Add a llarp::UDPHandle abstract class for UDP handling
- Removes the UDP tick handler; code that needs tick can just do a separate handler on the event
loop outside the UDP socket.
- Adds an "OwnedBuffer" which owns its own memory but is implicitly convertible to a llarp_buffer_t.
This is mostly needed to take over ownership of buffers from uvw without copying them as,
currently, uvw does its own allocation (pending some open upstream issues/PRs).
- Logic:
- add `make_caller`/`call_forever`/`call_every` utility functions to abstract Call wrapping and
dependent timed tasks.
- Add inLogicThread() so that code can tell its inside the logic thread (typically for
debugging assertions).
- get rid of janky integer returns and dealing with cancellations on call_later: the other methods
added here and the event loop code remove the need for them.
- Event loop:
- redo everything with uvw instead of libuv
- rename EventLoopWakeup::Wakeup to EventLoopWakeup::Trigger to better reflect what it does.
- add EventLoopRepeater for repeated events, and replace the code that reschedules itself every
time it is called with a repeater.
- Split up `EventLoop::run()` into a non-virtual base method and abstract `run_loop()` methods;
the base method does a couple extra setup/teardown things that don't need to be in the derived class.
- udp_listen is replaced with ev->udp(...) which returns a new UDPHandle object rather that
needing gross C-style-but-not-actually-C-compatible structs.
- Remove unused register_poll_fd_(un)readable
- Use shared_ptr for EventLoopWakeup rather than returning a raw pointer; uvw lets us not have to
worry about having the event loop class maintain ownership of it.
- Add factory EventLoop::create() function to create a default (uvw-based) event loop (previously
this was one of the llarp_ev_blahblah unnamespaced functions).
- ev_libuv: this is mostly rewritten; all of the glue code/structs, in particular, are gone as
they are no longer needed with uvw.
- DNS:
- Rename DnsHandler to DnsInterceptor to better describe what it does (this is the code that
intercepts all DNS to the tun IP range for Android).
- endpoint:
- remove unused "isolated network" code
- remove distinct (but actually always the same) variables for router/endpoint logic objects
- llarp_buffer_t
- make constructors type-safe against being called with points to non-size-1 values
- tun packet reading:
- read all available packets off the device/file descriptor; previously we were reading one packet
at a time then returning to the event loop to poll again.
- ReadNextPacket() now returns a 0-size packet if the read would block (so that we can implement
the previous point).
- ReadNextPacket() now throws on I/O error
- Miscellaneous code cleanups/simplifications
-fpermissive allows nasty things that are (apparently) needed on
Windows, so only enable it there.
vla are evil, so don't just warn but error on it.
remove duplicate -Wno-unknown-pragmas
Turn off modernize-use-trailing-return because it's *really* annoying
(and clangd picks it up when using it for editing).
Turn off modernize-use-nodiscard as well because it's already annoying.
Add bugprone-* as well because it includes a lot of useful checks.
Current lokinet dev relies on something from oxenmq dev branch which is
in the not-yet-released next version; without this lokinet tries using
the system 1.2.3 install, which then fails the build.
Ignore any root directory starting with build so that you can have many
build directories (build, build2, build-win32, ...) that get ignored.
Add .cache which clangd creates
Ignore compile_commands.json (and generate it by default) which helps
clangd, but needs to be symlinked to a compile_commands.json in a build
dir so isn't suitable for inclusion in the repo.
when retransmissions happen, they will be executed in a deterministic order by their delivery id.
this reduces the jitter from delayed message delivery on link layer as with an unordered map the order of iteration is "random",
so when we iterate all messages for retransmission we do it lowest id first (the messages queued first).
* redo setcap on linux so that it's required to do install targets on linux
* add -DWITH_SETCAP=OFF option to bypass setcap requirement on linux
* remove lokinet-bootstrap.ps1 as we bundle the bootstrap in the win32 installer now
* remove references to lokinet-bootstrap.ps1 in cmake
/cache is shared among multiple runners that can potentiall run at
once, but gradle's cache is retarded and just completely blocks anything
from using the cache for the span of the entire build. Work around it
by sticking the unique builder identifier in the cache path.
- Previous android java and jni code updated to work, but with much love
still needed to make it work nicely, e.g. handling when the VPN is
turned off.
- DNS handling refactored to allow android to intercept and handle DNS
requests as we can't set the system DNS to use a high port
(and apparently Chrome ignores system DNS settings anyway)
- add packet router structure to allow separate handling of specific
intercepted traffic, e.g. UDP traffic to port 53 gets handled by our
DNS handler rather than being naively forwarded as exit traffic.
- For now, android lokinet is exit-only and hard-coded to use exit.loki
as its exit. The exit will be configurable before release, but
allowing to not use exit-only mode is more of a challenge.
- some old gitignore remnants which were matching to things we don't
want them to (and are no longer relevant) removed
- some minor changes to CI configuration
- Remove SIIT from method names & comments because we're doing IPv4
mapped addresses (::ffff:0:0/96) rather than actual SIIT
(::ffff:0:0:0/96).
- add constructor taking a string+numeric port (and then don't allow a
port in the string).
- simplify IP string parsing by using parse_int()
- replace addrIsV4 with call to ipv6_is_mapped_ipv4 (this also fixes a
bug where addrIsV4 was not checking for leading 0s and so could return
true for a public IPv6 that happened to have ffff in the wrong spot).
add nameservice abstraction
give profile a name
allow to read conf, tmp files, etc.
remove /lib/@{multiarch}/ld-*.so mr, already covered by abstractions/base
allow local additions
* bump zmq static dep
* lokimq -> oxenmq
* llarp_nodedb -> llarp::NodeDB
* remove all crufty api parts of NodeDB
* make NodeDB rc selection api not suck
* make path builder api not suck
* propagate all above changes so that unit tests work and it all compiles
We set CMAKE_CROSSCOMPILE in a few places, but that is just a typo for
CMAKE_CROSSCOMPILING. However we don't need to set it at all: cmake
does so automatically whenever CMAKE_SYSTEM_NAME is set (which is why
our cross compiling actually works currently.)
when a path expires or fails or something causes its lifecycle to end we have cleanup
logic that handles clean up for it. this removes a code path that was crashing that
is duplicated elsewhere and is thus probably safe to bin. yolo.
* partial tun code refactor
* take out the trash
* move vpn platform code into llarp/vpn/platform.cpp
* fix hive build
* fix win32
* fix memory leak on win32
* reduce cpu use
* make macos compile
* win32 patches:
* use wepoll for zmq
* use all cores on windows iocp read loop
* fix zmq patch for windows
* clean up cmake for win32
* add uninstall before reinstall option to win32 installer
* more ipv6 stuff
* make it compile
* fix up route poker
* remove an unneeded code block in macos wtf
* always use call to system
* fix route poker behavior on macos
* disable ipv6 on windows for now
* cpu perf improvement:
* colease calls to Router::PumpLL to 1 per event loop wakeup
* set up THEN add addresses
* emulate proactor event loop on win32
* remove excessively verbose error message
* fix issue #1499
* exclude uv_poll from win32 so that it can start up
* update logtag to include directory
* create minidump on windows if there was a crash
* make windows happy
* use dmp suffix on minidump files
* typo fix
* address feedback from jason
* use PROJECT_SOURCE_DIR instead of CMAKE_SOURCE_DIR
* quote $@ in apply-patches in case path has spaces in it
* address feedback from tom
* remove llarp/ev/pipe
* add comments for clairification
* make event loop queue size constant named
* Add armhf static build + upload
* openssl build fix for static armhf build
Without this it tries building for arm64.
* Disable LTO for armhf static build
* Fix static lib check for arm
* Update gui win32 package url (#1466)
* Dev to stable (#1471)
* Install less stuff on windows CI
mingw-w64 is a metapackage that pulls in 4 different compilers
{i686,x86_64}×{-posix,-win32} but we only need x86_64-posix, so just
install that package.
* Update gui win32 package url (#1466) (#1468)
Co-authored-by: Jeff <jeff@i2p.rocks>
* stop lokinet before uninstalling service (#1469)
* stop lokinet before uninstalling service
* kill gui in uninstaller
* add helper script for building windows installers because i forgot how to most of the time.
* Update win32 gui download with smaller svg icon
Co-authored-by: Jeff <jeff@i2p.rocks>
Co-authored-by: Jason Rhinelander <jason@imaginary.ca>
* macos fluff
* make uninstaller work, the uninstall script needs to be called with /usr/bin/sudo to work because god know why it needs that even though it's already root.
* add logrotation
* make the uninstall script kill all the things
* revert
* use correct regular expression
* remove logrotate configs rename logrotate config file name to be more descriptive
* Bump version to 0.8.2
* Let cmake set PROJECT_NAME
`project(x ...)` already sets PROJECT_NAME (and CMAKE_PROKECT_NAME if
this is the top-level build) to `x`; the logic here seemed sort of
backwards.
* Get rid of uninstaller entitlement
We don't seem to need it anymore with Qt 5.15.1.
* Bump windows lokinet-gui build
this fixes the gui bug where it hangs on adding an exit becuase loopback gets routed over the exit.
before it was causing the lmq connection to hang becuase of that route change.
* Install less stuff on windows CI
mingw-w64 is a metapackage that pulls in 4 different compilers
{i686,x86_64}×{-posix,-win32} but we only need x86_64-posix, so just
install that package.
* Update gui win32 package url (#1466) (#1468)
Co-authored-by: Jeff <jeff@i2p.rocks>
* stop lokinet before uninstalling service (#1469)
* stop lokinet before uninstalling service
* kill gui in uninstaller
* add helper script for building windows installers because i forgot how to most of the time.
* Update win32 gui download with smaller svg icon
Co-authored-by: Jeff <jeff@i2p.rocks>
* stop lokinet before uninstalling service
* kill gui in uninstaller
* add helper script for building windows installers because i forgot how to most of the time.
mingw-w64 is a metapackage that pulls in 4 different compilers
{i686,x86_64}×{-posix,-win32} but we only need x86_64-posix, so just
install that package.
* Remove unneeded libcurl-dev build dep
* Make apt-get operations quieter
* apt repo updates
- store the apt repo gpg pubkey in the repo rather than having to fetch
it every time
- add the apt repo for all the deb builds
- update apt repo URL to deb.loki.network instead of
deb.imaginary.stream
* Use http so that we don't need ca-certificates
The repo packages are already signed so using https really isn't
important.
* Add hack to ignore jsonnet conflicts in deb builds
* Add build machine name to build output
* We don't install any shared libs currently
* fix deadlock in #1453
* remove use of shared_lock in profiling
* clean up Save and Load to acquire lock
* bencode and bdecode are now non locking
* remove all places using std::shared_lock
* bail if write fails
* deny message delivery from non registered nodes when whitelist is enabled and applicable
* use DoCallback and return true so that the event gets fired right
* add exit.localhost.loki cname record
* return cname in localhost.loki A/AAAA lookup reply
* correctly handle case where we have no exit
* typofix
* simplify logic
* Move resolvconf from deb branches to contrib/
This script gets invoked by the systemd service after startup to update
resolvconf with the lokinet dns server.
It was previously living in debian/lokinet-resolvconf in the
debian/ubuntu branches, but really belongs in contrib/ instead.
* Disable LTO on sid gcc/clang-11
So apparently dnsConfig in Router was never set.
Router::networkConfig was set, but only used in the function
which set it and otherwise only passed to another function once.
dnsConfig was only used in one place.
both member variables have been removed in favor of just
referencing Router::m_Config.
Co-authored-by: Thomas Winget <tewinget@gmail.com>
* llarp::Config constructor can take an empty fs::path now and it will default to fs::current_path
* llarp::ensureConfig now throws on fs error
* updates to daemon/main.cpp to reflect changes to llarp::ensureConfig's new throwability
* dowse parts of the code with holy water for the 0.8.1 tag
* Update how we build libuv
- Update submoduled libuv to latest stable (1.40.0)
- Don't look for a system libuv if we're under BUILD_STATIC_DEPS
- Add a libuv interface library rather than using globals
- Make the windows build fall back to the submodule if not explicitly
given a LIBUV_ROOT
* Replace ${LIBS} global with `base_libs` interface
This simplifies linking and include handling a bit.
* Remove unneeded header
* Add missing csignal header
(This was previously being pulled in incredibly indirectly via some
stuff that eventually includes some other stuff that eventually included
uv.h)
* Use GNUInstallDirs to get lib dir instead of hard-coding lib
Fixes#1429
* enable client relay profiling by default
* macos dns fixes
* improve peer profiling algorithm to track timeouts vs failures
* remove debug ioctl call in tuntap code
* use ub_wait instead of ub_process as that was what was there before
* const correctness
* DRY out checking for SIIT
* typofix
* correct name
* more aggressive path building.
* do more than one in parallel path builds at a time
* correct last commit's logic
* rename numPaths -> numDesiredPaths to clarify intent
* revert string change as it will break a lot
* don't prematurly short circuit on snode builds
Co-authored-by: Thomas Winget <tewinget@gmail.com>
* Should fix some windows service issues
* fix return condition inversion
* Add some Trace level logging
also make the logger actually respect the log level you set.
* event loop should not queue things to itself...
at present, logic thread queue continues until it is empty, so
queueing things onto itself is just wasteful.
* call_later(foreach thing) is better than foreach thing (call later)
also if you already queued those things but they have not happened yet,
there is no sense to queue them to happen again.
* do not queue read on write finish, only on read finish
* failure to start DNS server should be proper startup failure.
without the DNS server working lokinet is...kinda pointless, right?
* format
* don't queue stuff to logic thread if in logic thread
the thing that clears the queue...clears it. So you're just delaying and adding overhead.
* windows unbound thread sleep instead of just busy-waiting
also clang-format decided I can't have a blank line for some reason...
* fix unbound async worker on windows
* add conf.d directory for config overrides
* refactor llarp::Config
* add explicit constructor with datadir as parameter
* have all config files be passed as std::optional
* make Config::LoadDefault private and use std::optional in Config::Load to remove ambiguity
* update rest of codebase to reflect above changes
* fix pybind
* rename bootstrap config skipBootstrap to seednode as it's more descriptive
* make seednode configurable
* make pybind layer compile
* make pybind layer run
If not using an exit, messing with the routing table is not good.
As such, the ability to keep track of routes we *might* want is good,
but the ability to set/unset those routes is necessary, to correspond
to enabling/disabling exit functionality.
This is relatively painless for lokinet as it already had workarounds
during 0.8 dev work for the things macos hated in 10.13.
Dependencies, however, were not being built with the proper macos target
junk, so this fixes that.
* add some dns txt records for stuff we want to expose
* fix txt records
* txt records for snode info
* dont send cname as it mangles the response
* check for 3 parts not 3 characters
* Config file API/comment improvements
API improvements:
=================
Make the config API use position-independent tag parameters (Required,
Default{123}, MultiValue) rather than a sequence of bools with
overloads. For example, instead of:
conf.defineOption<int>("a", "b", false, true, 123, [] { ... });
you now write:
conf.defineOption<int>("a", "b", MultiValue, Default{123}, [] { ... });
The tags are:
- Required
- MultiValue
- Default{value}
plus new abilities (see below):
- Hidden
- RelayOnly
- ClientOnly
- Comment{"line1", "line2", "line3"}
Made option definition more powerful:
=====================================
- `Hidden` allows you to define an option that won't show up in the
generated config file if it isn't set.
- `RelayOnly`/`ClientOnly` sets up an option that is only accepted and
only shows up for relay or client configs. (If neither is specified
the option shows up in both modes).
- `Comment{...}` lets the option comments be specified as part of the
defineOption.
Comment improvements
====================
- Rewrote comments for various options to expand on details.
- Inlined all the comments with the option definitions.
- Several options that were missing comments got comments added.
- Made various options for deprecated and or internal options hidden by
default so that they don't show up in a default config file.
- show the section comment (but not option comments) *after* the
[section] tag instead of before it as it makes more sense that way
(particularly for the [bind] section which has a new long comment to
describe how it works).
Disable profiling by default
============================
We had this weird state where we use and store profiling by default but
never *load* it when starting up. This commit makes us just not use
profiling at all unless explicitly enabled.
Other misc changes:
===================
- change default worker threads to 0 (= num cpus) instead of 1, and fix
it to allow 0.
- Actually apply worker-threads option
- fixed default data-dir value erroneously having quotes around it
- reordered ifname/ifaddr/mapaddr (was previously mapaddr/ifaddr/ifname)
as mapaddr is a sort of specialization of ifaddr and so makes more
sense to come after it (particularly because it now references ifaddr
in its help message).
- removed peer-stats option (since we always require it for relays and
never use it for clients)
- removed router profiles filename option (this doesn't need to be
configurable)
- removed defunct `service-node-seed` option
- Change default logging output file to "" (which means stdout), and
also made "-" work for stdout.
* Router hive compilation fixes
* Comments for SNApp SRV settings in ini file
* Add extra blank line after section comments
* Better deprecated option handling
Allow {client,relay}-only options in {relay,client} configs to be
specified as implicitly deprecated options: they warn, and don't set
anything.
Add an explicit `Deprecated` tag and move deprecated option handling
into definition.cpp.
* Move backwards compat options into section definitions
Keep the "addBackwardsCompatibleConfigOptions" only for options in
sections that no longer exist.
* Fix INI parsing issues & C++17-ify
- don't allow inline comments because it seems they aren't allowed in
ini formats in general, and is going to cause problems if there is a
comment character in a value (e.g. an exit auth string). Additionally
it was breaking on a line such as:
# some comment; see?
because it was treating only `; see?` as the comment and then producing
an error message about the rest of the line being invalid.
- make section parsing stricter: the `[` and `]` have to be at the
beginning at end of the line now (after stripping whitespace).
- Move whitespace stripping to the top since everything in here does it.
- chop off string_view suffix/prefix rather than maintaining position
values
- fix potential infinite loop/segfault when given a line such as `]foo[`
* Make config parsing failure fatal
Load() LogError's and returns false on failure, so we weren't aborting
on config file errors.
* Formatting: allow `{}` for empty functions/structs
Instead of using two lines when empty:
{
}
* Make default dns bind 127.0.0.1 on non-Linux
* Don't show empty section; fix tests
We can conceivably have sections that only make sense for clients or
relays, and so want to completely omit that section if we have no
options for the type of config being generated.
Also fixes missing empty lines between tests.
Co-authored-by: Thomas Winget <tewinget@gmail.com>
- bumps deployment target to 10.15 because earlier versions don't
support C++17.
- remove double-include of installer.cmake
- use new static dep lokinet build system
- replace lokinetctl with lokinet-vpn
If you specify upstream= then you get no upstream, if you give one then
you use that instead of the default, but you can still list multiple.
unbound doesn't support an upstream port, so bail if the user gives a
non-port-53 response.
0.7 supported both public-address and public-ip, but only the former was
being allowed in 0.8.
This makes `public-ip=...` work again, and makes it the canonical option
(because it makes a bit more sense than address when we also have the
public-port parameter), while making `public-address=` accepted with a
deprecation warning.
Also adds default descriptions for port and ip for the generated ini.
* serialize auth codes in rpc
* format
* format
* config and route fixups
* bump lokimq
* fix compile
* discard unknown members in service keys
* format
* prepare for keytool script
* dont serialize ephemeral members in service key file
* regnerate ephemeral members in service identity on load
* add keygen script
* use nacl for generating keys
* format
* initial work for RC SRVs.
Needs tests for new RC format.
Needs SRVs added to new RC, and associated tests.
* convert rc sign/verify test to catch2, add test for new rc
Also fixes a mistake in new rc serialization
* bump loki-mq submodule
need support for viewing bt deserialize consumer buffer so we
know how much it has consumed.
* fix some behavior errors
need to advance llarp_buffer_t after consuming data from it
only rewind and modify size of llarp_buffer_t in owning context.
* Add test for router contact (de-)serialization
Adds a test that makes a list of RouterContact with mixed
versions and ensures it serializes and deserializes correctly.
* * refactor route poking out of llarp::Router and into llarp::RoutePoker
* only poke routes when we have an exit enabled as a client
* add route_poker header so it compiles
* ignore tun interfaces on windows for mitigating foot cannons
* add flag for git add -p when using format verifier as git hook
* use explicit path for route command on windows
* fix typo
* fix typo
* remove hunk for win32 route exclusion based off being a tun interface
* add metric to win32 route command
* * refactor win32 route poking to use a common function for iterating over routes
* put interface in route poking for default route
* mnake it compile
* use correct route command on windows
* use fs::path for service::Identity::EnsureKeys
* Add ignored [dns]no-resolvconf option for deb backwards compat
The debs hack a `#no-resolvconf=1` into the config file in 0.7 to allow
a user to disable resolvconf setup during startup by uncommenting it.
That doesn't work anymore since 0.8 errors on invalid config options, so
add it as an ignored option so that 0.7 deb config files can still be
compatible.
* Fix `[dns]upstream` comment not being produced in generated conf file
* fix test
Co-authored-by: Jeff <jeff@i2p.rocks>
The debs don't want it (and so this will save needing to patch it out),
nor do you need it if running via a systemd service file that sets the
capabilities.
- Disable bionic arm64 job to reduce the load on the ARM box a bit.
- Add bionic amd64 build so that we have a (normal) build on bionic.
- Remove sid/g++-10 job because g++ is now the default on sid (so the
debian/sid build is already doing the same thing).
* it lives?
* clean up
* add readme and add x86_64 to abi filters
* disable route poking on android
* make it compile on android
* it compiles!!111
* typofix
* re-enable ccache for android
* Update requirements and other clean up
git is now required due to submodules
libcurl/libcap: not sure if the minimum version is correct, The number I used is based on the bionic package version
add recursive option to all cloning
other tweaks to make concise and precise
* Remove libcurl
* Add nproc back
* add libcurl back for windows
* Remove libcurl from WIndows
as lokinet-bootstrap.exe is just curl.exe now and will be removed soon
* - allow running lokinet as root on windows
($PROGRAMDATA/.lokinet/lokinet.ini)
- oops forgot to eat my own command line args
* cursed windows
* now that we eat our own startup options the entry in cxxopts is fully
ded
* aha our command line options get smashed up
* revert plus DRY fix
* initial relay side lns
* fix typo
* add reserved names and refactor test for dns
* lns name decryption
* all wired up (allegedly)
* refact to use service::EncryptedName for LNS responses to include nonce with ciphertext
* fully rwemove tag_lookup_job
* replace lns cache with DecayingHashTable
* check for lns name validity against the following rules:
* not localhost.loki, loki.loki, or snode.loki
* if it contains no dash then max 32 characters long, not including the .loki tld (and also assuming a leading subdomain has been stripped)
* These are from general DNS requirements, and also enforced in
registrations:
* Must be all [A-Za-z0-9-]. (A-Z will be lower-cased by the RPC call).
* cannot start or end with a -
* max 63 characters long if it does contain a dash
* cannot contain -- in the third and fourth characters unless it starts with xn--
* handle timeout in name lookup job by calling the right handler with std::nullopt
* fix up macos route poker logic
* fix typo
* use string_view
* add forgotten header
* full paths
* add debugging
* catch exception on adding route
* workarround for macos
* typofix
* typofix
* fix for macos
* fix command for macos
* because we autopoke remove explicit route poking in rpc
* probably final fix of macos route poking
* split routes instead of deleting them
* dynamic route poking
* move log statement for introset lookup and dont consider bad sessions as able to send
* send convotag reset frame when we have no session
* add exit map to rpc
* use split_any
* route poking:
* remove popen() call, replace with reading /proc/net/route for getting default route
* dynamically poke and unpoke routes on runtime
* swap intros and fix rpc endpoint for version to return what the ui expects
* use std::string::find_first_not_of instead of using a lambda
* update loki-mq submodule for tuple support
* srv record reply implementation
still need to encode srv records into intro sets / router contacts
as well as decode from them and match against queried service.proto
* inverted condition fix in config code
* SRV record struct (de-)serialization for intro sets
* parsing and using srv records from config (for/in introsets)
* adopt str utils from core and use for srv parsing
* changes to repeat requests
no longer drop repeat requests on the floor, but do not make
an *actual* request for them if one is in progress.
do not call reply hook for each reply for a request, as
each userland request is actually made into several lokinet
requests and this would result in duplicate replies.
* fetch SRVs from introsets for .loki
* make format
* dns and srv fixes, srv appears to be working
* run unbound stuff in another thread because LOL windows
* because unbound runs in another thread callbacks for libunbound need to be wrapped in a deferred call so they are done in the logic thread
* bump sqlite3 dep because it's gone, repin hash.
* split up ipv6 netmask
* revert iwp ack interval change
* c++17-isms in ip range map
* lambda-ize nodedb
* mutable lambdas in transit hops
* perfect forwarding of args in abstract router
* mutable lambdas in lokid rpc client
* notes in readme about nproc
* use exceptions when fetching identity key instead of std::optional, will throw on fail
* fix up config options for endpoint auth and add better docs
* add llarp::serive::AuthType enum for controlling what kind of endpoint auth to use
These variables need to persist past the end of the lambda and weren't;
release builds didn't seem to mind access the freed memory, but Debug
builds triggered failures.
* rework net code for ip ranges to be cleaner
* clean up endpoint auth code
* refactor config to validate network configs before setting up endpoints
* remove buildone from path/pathbuilder.cpp so we don't spam connection attempts
llarp::dns::Message does not fully support DNS packets,
so converting the upstream response to one will not work
for all query types. Also it is kinda silly to begin with,
as the upstream reply is already a network packet, ready to go.
CMAKE_<LANG>_COMPILER_LAUNCHER was added around cmake 3.4 and, usefully,
can be enabled/disabled via cmake invocation flags (unlike the older
RULE_LAUNCH_COMPILE property).
- fix ldap disabling flag
- disable a ton of protocols (include ldap, properly)
- remove INSTALL_COMMAND (the default seems to work fine)
- disable --quiet so we can see the configuration summary
Relies on an SSH_KEY secret being set in the repository.
Also move the linked library check to a bash script because escaping a
value in jsonnet (escape once) that gets passed through yaml (escape
twice) and then bash (escape again) is too painful.
- add some comments
- exclude commits to the debian/ubuntu branches from getting built
- disable "Updating man-db" which chews up several seconds during
package install
It was a workaround for pre-C++17 std::string which didn't support
passing a string_view to various functions/operators. There's only one
place left that needs an explicit conversion, and that's where it is
used as a map key; so just be explicit there and remove llarp::str()
everywhere else.
- The `grep` had a typo of `-e` instead of `-E` which made it not work
- Made the command actually fail the pipeline if unwanted linked libs
were found.
- Print in green if everything is okay.
- Add librt.so to the whitelist
This replaces all use of std::optional's `opt.value()` with `*opt`
because macOS is great and the ghost of Steve Jobs says that actually
supporting std::optional's value() method is not for chumps before macOS
10.14. So don't use it because Apple is great.
Pretty much all of our use of it actually is done better with operator*
anyway (since operator* doesn't do a check that the optional has a
value).
Also replaced *most* of the `has_value()` calls with direct bool
context, except for one in the config section which looked really
confusing at a glance without a has_value().
macOS doing a shared library build is not working without untangling
some of the interdependencies. This commit does that, at least enough
to get macOS to compile.
This isn't the cleanest as currently implemented (we have some net/
things in `liblokinet-platform` and some in `liblokinet`, and likewise
ev/vpnio.cpp is in `liblokinet` while the rest of `ev/*` is in
`liblokinet-platform`).
Identifier names with `__` in them are reserved for the implementation.
One leading underscore followed by a lower-case letter is fine though
(leading underscore followed by *upper*-case is, however, also
reserved).
The fixed-size array + not needing to do an atomic operation in the
function should allow better compiler optimization.
(And with C++17 the type and size are inferred).
- Move IPRange into its own net/ip_range.hpp
- Move the static net::IPPacket::TruncateV6, etc. functions to free
net::TruncateV6, etc. functions (now from net/ip.hpp instead of
net/ip_packet.hpp).
- Make net::TruncateV6 and net::ExpandV4 constexpr.
- Add IPRange::FromIPv4 factory function (to replace the iprange_ipv4
free function)
Rename net/ip.{cpp,hpp} to net/ip_packet.{cpp,hpp}.
(Doing this in two commits because I want to repurpose ip.hpp/ip.cpp,
and want git to figure out the history properly).
This simplifies the build a bit by moving the avx stubs into a single .c
file: we can avoid compiling all of the avx2 code and just add a single
stub file instead when the compiler doesn't have -mavx2/-mfma.
This also simplifies cmake to just always test the flags; there are some
cases (like using NATIVE_BUILD) where USE_AVX2 doesn't necessarily
apply, and it's cheap to just check them.
Refactors many things in cmake to improve and simplify:
- don't use variable indirection for target names; target names are
*already* a variable of sorts. (e.g. ${UTIL_LIB} is now just
lokinet-util). cmake/basic_definitions.cmake is now gone.
- fix LTO enabling to use the standard cmake (3.9+) LTO mechanism rather
than shoving a bunch of flag hacks through link_libraries and
add_compile_options. This also now enables LTO when building a shared
library (because previously the -flto hacks were only turned on in the
static code for some reason).
- build liblokinet as *either* shared library or static library, but not
both. Building both makes things more complicated because they had
different names (lokinet-shared or lokinet-static) and seems pointless:
you generally want one or the other. Now there is just the liblokinet
target, which will be shared or static depending on the value of
BUILD_SHARED_LIBS.
- Simplify lokinet-cryptography AVX2 code: just build *one* library, and
add in the additional AVX2 files when possible, rather than building two
and needing to merge them.
- Compress STATIC_LINK and STATIC_LINK_RUNTIME into just STATIC_LINK.
It makes no sense to use one of these (_RUNTIME) on Windows and the
other on non-Windows when they appear to try to do the same thing.
- remove a bunch of annotations from `endif(FOO)` -> `endif()`.
- move all the tuntap compilation code (including OS-specific source
file selection) into vendor/CMakeLists.txt and build tuntap as an
intermediate OBJECT library rather than keeping a global variable in 5
different files.
- move release motto define to root cmake; it made no sense being
duplicated in both unix.cmake and win32.cmake
- fix add_log_tag to not stomp on any existing source compile flags with
its definition. Also use proper compile definition property instead of
cramming it into compile flags.
- make optimization/linker flags less hacky. There's no reason for us
to force particular optimization flags because the cmake build type
already does that (e.g. -DCMAKE_BUILD_TYPE=Release does -O3). Not doing
that also silences a bunch of cmake warnings because it thinks "-O0 -g3"
etc. are link libraries (which is reasonable: that's what the code was
telling cmake they are).
- sets the default build type to RelWithDebInfo which gives us `-O2 -g`
if you don't specify a build type.
- Move PIC up (so that the things loaded in unix.cmake, notably libuv,
have it set).
- Add a custom `curl` interface library that carries the correct link
target and include paths for curl (system or bundled).
- class template argument deduction lets us write `std::unique_lock
foo{mutex}` instead of `std::unique_lock<mutex_type> foo{mutex}` which
makes the `unique_lock` and `shared_lock` functions unnecessary.
- Replace GNU-specific warn_unused_result attribute with C++17-standard
[[nodiscard]]
- Remove pre-C++17 workaround code for fold expressions, void_t
llarp/config/config.cpp:
respect [network]:type option
llarp/handlers/exit.cpp:
when [network]:type is null dont init tun interface
llarp/service/context.cpp:
respect [network]:type option
change endpoint name back to "default"
llarp/tooling/router_hive.cpp:
dont use LogicCall for obtaining RCs from underlying relays, it crashes the mainloop and it's probably safe to readonly access RCs.
pybind/common.hpp:
remove typecasters as we use C++17 now
pybind/llarp/config.cpp:
remove SnappConfig
wire up NetworkConfig
pybind/llarp/handlers/pyhandler.hpp:
remove SnappConfig from constructor
pybind/llarp/handlers/pyhandler.cpp:
update constructor implementation to match header
test/hive/hive.py:
remove broke endpoint related code
wire up null endpoint option using NetworkConfig
use index at 0 for relays and clients instead of 1
dont add a python endpoint to all clients
* rev-parse --short without any argument defaults to 8 chars
* truncation is absolutely moot here
* rev-list --abbrev-commit also uses 8 chars by default
because of length mismatch (COMMIT - 9 chars, TAGGEDCOMMIT - 8 chars) it always was mismatching them even when it should've been detecting that it's at release tag.
cut out explicit lengths and leave everything to use defaults, because they work (they both rely on the same core.abbrev setting so no length mismatch should happen).
Right after project() which contains the version, so that version +
motto updates are basically in one place (and so that non-Makefile cmake
invocations get it).
Codesigns and notarizes the package. Requires a bunch of stuff to
actually get this working. (see contrib/macos-signing.txt).
I'm currently also building with
-DLOKINET_GUI_REPO=https://github.com/jagerman/loki-network-control-panel.git -DLOKINET_GUI_CHECKOUT=cmake
because it needs the camke-ification PR for the control panel to
properly sign everything there.
CMake will set version variables itself if you give the version in the
project(), which is cleaner. Also removes some (nearly) duplicate
definitions and settings added in basic_definitions.cmake for unknown
reasons.
Removes some redundant settings (name, description, version) from the
cpack settings which already default to the values from the project()
call.
Right after project() which contains the version, so that version +
motto updates are basically in one place (and so that non-Makefile cmake
invocations get it).
Codesigns and notarizes the package. Requires a bunch of stuff to
actually get this working. (see contrib/macos-signing.txt).
I'm currently also building with
-DLOKINET_GUI_REPO=https://github.com/jagerman/loki-network-control-panel.git -DLOKINET_GUI_CHECKOUT=cmake
because it needs the camke-ification PR for the control panel to
properly sign everything there.
CMake will set version variables itself if you give the version in the
project(), which is cleaner. Also removes some (nearly) duplicate
definitions and settings added in basic_definitions.cmake for unknown
reasons.
Removes some redundant settings (name, description, version) from the
cpack settings which already default to the values from the project()
call.
* update travis ci clang-format to clang-format-9 because we use that now
add python3-dev package becuase something inside ci thinks it has python but really does not
* try using python3.6
* try working around hot garbage that is travis-ci
* add deadsnakes repo for python3.8
* prevent nullptr deference when running in unit tests
* move python3.8 to main dependancy matrix and add python3.8 to homebrew deps
* add deadsnake apt repo
* add deadsnakes and python3.8 back to previous matrix
* dev package for python
* toggle hive build in ci
* dont add pybind11 if not bulding hive
* revert setting pyenv shim for travis ci
* make native builds on by default except for windows ci
* only apply native build being off for windows release target becuase that broke macos
This is an initial pass at doing explicit value checks when handling
config parsing, as opposed to using a visiting pattern. The latter
made it difficult to check for conditions such as missing required
values, multiple values, etc.
It was also generally less readable (think declarative) which further
made it difficult to get a grasp for what our actual configuration file
requirements were.
This commit reflects changes to clang-format rules. Unfortunately,
these rule changes create a massive change to the codebase, which
causes an apparent rewrite of git history.
Git blame's --ignore-rev flag can be used to ignore this commit when
attempting to `git blame` some code.
This template-ifies Router::NotifyRouterEvent() up so that it accepts
the arguments to instantiate the specified event type, forwarding them
to std::make_unique. This would allow (in the future) the function to
no-op the call and avoid memory allocation. It also slightly reduces
the amount of code required to fire an event.
This commit also simplifies some of the RouterEvent code to reduce
redundancy.
DHT PubIntroSentEvent
some helper functions added to RouterHive (C++ class) as well as RouterHive(Python class)
hive.py main() continues to be a testbed for new event types
some more internal classes in pybind
allows enough time for every relay (at least in a hive of 50) to connect
to the bootstrap node so all will get all gossips, but not too long so
tests can run relatively quickly.
hive.py is currently largely for testing the pybind stuff, so changes to it will likely
be frequent and arbitrary for now.
Added pybind for llarp::path::PathHopConfig, but not every member -- just rc and upstream routerID
Hive now uses std::queue with mutex instead of our lockless queue.
Removed some functions from Hive that will not be necessary as things are being handled from python.
a bunch of routers logging to stdout at the same time is a complete
charlie foxtrot. until we take the time to make logger not a singleton
(and probably make each router able to log to its own file rather than stdout)
just make it not log.
Note: this is very temporary, as the logs will be annoying for testing the
pybind stuff and shouldn't be necessary for debugging it
Router now has a hive pointer if LOKINET_HIVE is set.
llarp::Context has a method InjectHive to give Router the pointer.
Router has a method NotifyRouterEvent which does:
- when LOKINET_HIVE is set, passes the event to RouterHive
- else when LOKINET_DEBUG is set, prints the event at a low log level
- else NOP
macro_ensure_out_of_source_build("${PROJECT_NAME} requires an out-of-source build. Create a build directory and run 'cmake ${CMAKE_SOURCE_DIR} [options]'.")
message(FATAL_ERROR"Could not find libsodium >= 1.0.18; either install it on your system or use -DDOWNLOAD_SODIUM=ON to download and build an internal copy")
endif()
message(STATUS"Sodium >= 1.0.18 not found, but DOWNLOAD_SODIUM specified, so downloading it")
message(WARNING"This configuration is optimised for older hardware and/or constrained node operation, may result in poor performance on desktop systems")
message(WARNING"For deployment on such systems, all external code (currently, libuv) must also be compiled for the target!")
message(STATUS"Submodule '${relative_path}' is up-to-date")
else()
message(FATAL_ERROR"Submodule '${relative_path}' is not up-to-date. Please update with\ngit submodule update --init --recursive\nor run cmake with -DSUBMODULE_CHECK=OFF")
endif()
endfunction()
message(STATUS"Checking submodules")
check_submodule(external/nlohmann)
check_submodule(external/googletest)
check_submodule(external/cxxopts)
check_submodule(external/ghc-filesystem)
check_submodule(external/optional-lite)
check_submodule(external/date)
pkg_check_modules(SDlibsystemdIMPORTED_TARGET)
#DefaultWITH_SYSTEMDtotrueifwefoundit
option(WITH_SYSTEMD"enable systemd integration for sd_notify"${SD_FOUND})
<uses-permissionandroid:name="android.permission.INTERNET"/><!-- normal perm, per https://developer.android.com/guide/topics/permissions/normal-permissions.html -->
<!-- The project.properties file is created and updated by the 'android'
tool, as well as ADT.
This contains project specific properties such as project target, and library
dependencies. Lower level build properties are stored in ant.properties
(or in .classpath for Eclipse projects).
This file is an integral part of the build system for your
application and should be checked into Version Control Systems. -->
<loadpropertiessrcFile="project.properties"/>
<!-- quick check on sdk.dir -->
<fail
message="sdk.dir is missing. Insert sdk.dir=... into './local.properties'. Make sure to generate local.properties using 'android update project' or to inject it through the ANDROID_HOME environment variable."
unless="sdk.dir"
/>
<fail
message="ndk.dir is missing. Insert ndk.dir=... into './local.properties'."
unless="ndk.dir"
/>
<!--
Import per project custom build rules if present at the root of the project.
This is the place to put custom intermediary targets such as:
-pre-build
-pre-compile
-post-compile (This is typically used for code obfuscation.
message(STATUS"Enabling notarization with account ${MACOS_NOTARIZE_ASC}/${MACOS_NOTARIZE_USER}")
else()
message(WARNING"You have not set one or more of MACOS_NOTARIZE_USER, MACOS_NOTARIZE_PASS, MACOS_NOTARIZE_ASC: notarization will fail; see contrib/macos/README.txt")
endif()
else()
message(WARNING"Codesigning disabled; the resulting build will not run on most macOS systems")
message(WARNING"Missing a ${prof} provisioning profile: Apple will most likely log an uninformative error message to the system log and then kill harmless kittens if you try to run the result")
elseif(NOTEXISTS"${${prof}}")
message(FATAL_ERROR"Provisioning profile ${${prof}} does not exist; fix your -D${prof} path")
Jason Rhinelander
dr7ana
Jeff Becker
GNU/LongYap
majestrate
Kamay Xutax
drouhana
Thomas Winget
Audric Ackermann
jeff
Pebu
Jeff
Benjamin Henrion
Sean
Nuc1eoN
David Arnold
lyyn
Anton Nesterov
jahway603
Yevhen Stasiv
Dimitris Apostolou
DEADBLACKCLOVER
Rick V
Ryan Tharp
cathugger
Rick V
Stephen Shelton
Linux User
