Refactors how quic packets get handled: the actual tunnels now live in
tunnel.hpp's TunnelManager which holds and manages all the quic<->tcp
tunnelling. service::Endpoint now holds a TunnelManager rather than a
quic::Server. We only need one quic server, but we need a separate quic
client instance per outgoing quic tunnel, and TunnelManager handles all
that glue now.
Adds QUIC packet handling to get to the right tunnel code. This
required multiplexing incoming quic packets, as follows:
Adds a very small quic tunnel packet header of 4 bytes:
[1, SPORT, ECN] for client->server packets, where SPORT is our
source "port" (really: just a uint16_t unique quic instance
identifier)
or
[2, DPORT, ECN] for server->client packets where the DPORT is the SPORT
from above.
(This also reworks ECN bits to get properly carried over lokinet.)
We don't need a destination/source port for the server-side because
there is only ever one quic server (and we know we're going to it when
the first byte of the header is 1).
Removes the config option for quic exposing ports; a full lokinet will
simply accept anything incoming on quic and tunnel it to the requested
port on the the local endpoint IP (this handler will come in a following
commit).
Replace ConvoTags with full addresses: we need to carry the port, as
well, which the ConvoTag can't give us, so change those to more general
SockAddrs from which we can extract both the ConvoTag *and* the port.
Add a pending connection queue along with new quic-side handlers to call
when a stream becomes available (TunnelManager uses this to wire up
pending incoming conns with quic streams as streams open up).
Completely get rid of tunnel_server/tunnel_client.cpp code; it is now
moved to tunnel.hpp.
Add listen()/forget() methods in TunnelManager for setting up quic
listening sockets (for liblokinet usage).
Add open()/close() methods in TunnelManager for spinning up new quic
clients for outgoing quic connections.
- Move uvw target to external instead of llarp/CMakeLists.txt
- Add sqlite_orm headers via interface library rather than shoving it
into the global include search path.
- Find sqlite3 as part of the sqlite_orm target deps rather than linking
it separately. (Also don't link it into lokinet-util, which doesn't
appear to actually need it).
loop->call(...) is similar to the old logic->Call(...), but is smart
about the current thread: if called from within the event loop it simply
runs the argument directly, otherwise it queues it.
Similarly most of the other event loop calls are also now thread-aware:
for example, `call_later(...)` can queue the job directly when called if
in the event loop rather than having to double-queue through the even
loop (once to call, then inside the call to initiate the time).
- removes all the llarp_ev_* functions, replacing with methods/classes/functions in the llarp
namespace.
- banish ev/ev.h to the void
- Passes various things by const lvalue ref, especially shared_ptr's that don't need to be copied
(to avoid an atomic refcount increment/decrement).
- Add a llarp::UDPHandle abstract class for UDP handling
- Removes the UDP tick handler; code that needs tick can just do a separate handler on the event
loop outside the UDP socket.
- Adds an "OwnedBuffer" which owns its own memory but is implicitly convertible to a llarp_buffer_t.
This is mostly needed to take over ownership of buffers from uvw without copying them as,
currently, uvw does its own allocation (pending some open upstream issues/PRs).
- Logic:
- add `make_caller`/`call_forever`/`call_every` utility functions to abstract Call wrapping and
dependent timed tasks.
- Add inLogicThread() so that code can tell its inside the logic thread (typically for
debugging assertions).
- get rid of janky integer returns and dealing with cancellations on call_later: the other methods
added here and the event loop code remove the need for them.
- Event loop:
- redo everything with uvw instead of libuv
- rename EventLoopWakeup::Wakeup to EventLoopWakeup::Trigger to better reflect what it does.
- add EventLoopRepeater for repeated events, and replace the code that reschedules itself every
time it is called with a repeater.
- Split up `EventLoop::run()` into a non-virtual base method and abstract `run_loop()` methods;
the base method does a couple extra setup/teardown things that don't need to be in the derived class.
- udp_listen is replaced with ev->udp(...) which returns a new UDPHandle object rather that
needing gross C-style-but-not-actually-C-compatible structs.
- Remove unused register_poll_fd_(un)readable
- Use shared_ptr for EventLoopWakeup rather than returning a raw pointer; uvw lets us not have to
worry about having the event loop class maintain ownership of it.
- Add factory EventLoop::create() function to create a default (uvw-based) event loop (previously
this was one of the llarp_ev_blahblah unnamespaced functions).
- ev_libuv: this is mostly rewritten; all of the glue code/structs, in particular, are gone as
they are no longer needed with uvw.
- DNS:
- Rename DnsHandler to DnsInterceptor to better describe what it does (this is the code that
intercepts all DNS to the tun IP range for Android).
- endpoint:
- remove unused "isolated network" code
- remove distinct (but actually always the same) variables for router/endpoint logic objects
- llarp_buffer_t
- make constructors type-safe against being called with points to non-size-1 values
- tun packet reading:
- read all available packets off the device/file descriptor; previously we were reading one packet
at a time then returning to the event loop to poll again.
- ReadNextPacket() now returns a 0-size packet if the read would block (so that we can implement
the previous point).
- ReadNextPacket() now throws on I/O error
- Miscellaneous code cleanups/simplifications
- Previous android java and jni code updated to work, but with much love
still needed to make it work nicely, e.g. handling when the VPN is
turned off.
- DNS handling refactored to allow android to intercept and handle DNS
requests as we can't set the system DNS to use a high port
(and apparently Chrome ignores system DNS settings anyway)
- add packet router structure to allow separate handling of specific
intercepted traffic, e.g. UDP traffic to port 53 gets handled by our
DNS handler rather than being naively forwarded as exit traffic.
- For now, android lokinet is exit-only and hard-coded to use exit.loki
as its exit. The exit will be configurable before release, but
allowing to not use exit-only mode is more of a challenge.
- some old gitignore remnants which were matching to things we don't
want them to (and are no longer relevant) removed
- some minor changes to CI configuration
* bump zmq static dep
* lokimq -> oxenmq
* llarp_nodedb -> llarp::NodeDB
* remove all crufty api parts of NodeDB
* make NodeDB rc selection api not suck
* make path builder api not suck
* propagate all above changes so that unit tests work and it all compiles
* partial tun code refactor
* take out the trash
* move vpn platform code into llarp/vpn/platform.cpp
* fix hive build
* fix win32
* fix memory leak on win32
* reduce cpu use
* make macos compile
* win32 patches:
* use wepoll for zmq
* use all cores on windows iocp read loop
* fix zmq patch for windows
* clean up cmake for win32
* add uninstall before reinstall option to win32 installer
* more ipv6 stuff
* make it compile
* fix up route poker
* remove an unneeded code block in macos wtf
* always use call to system
* fix route poker behavior on macos
* disable ipv6 on windows for now
* cpu perf improvement:
* colease calls to Router::PumpLL to 1 per event loop wakeup
* set up THEN add addresses
* emulate proactor event loop on win32
* remove excessively verbose error message
* fix issue #1499
* exclude uv_poll from win32 so that it can start up
* update logtag to include directory
* create minidump on windows if there was a crash
* make windows happy
* use dmp suffix on minidump files
* typo fix
* address feedback from jason
* use PROJECT_SOURCE_DIR instead of CMAKE_SOURCE_DIR
* quote $@ in apply-patches in case path has spaces in it
* address feedback from tom
* remove llarp/ev/pipe
* add comments for clairification
* make event loop queue size constant named
* Update how we build libuv
- Update submoduled libuv to latest stable (1.40.0)
- Don't look for a system libuv if we're under BUILD_STATIC_DEPS
- Add a libuv interface library rather than using globals
- Make the windows build fall back to the submodule if not explicitly
given a LIBUV_ROOT
* Replace ${LIBS} global with `base_libs` interface
This simplifies linking and include handling a bit.
* Remove unneeded header
* Add missing csignal header
(This was previously being pulled in incredibly indirectly via some
stuff that eventually includes some other stuff that eventually included
uv.h)
* Use GNUInstallDirs to get lib dir instead of hard-coding lib
Fixes#1429
* * refactor route poking out of llarp::Router and into llarp::RoutePoker
* only poke routes when we have an exit enabled as a client
* add route_poker header so it compiles
* initial relay side lns
* fix typo
* add reserved names and refactor test for dns
* lns name decryption
* all wired up (allegedly)
* refact to use service::EncryptedName for LNS responses to include nonce with ciphertext
* fully rwemove tag_lookup_job
* replace lns cache with DecayingHashTable
* check for lns name validity against the following rules:
* not localhost.loki, loki.loki, or snode.loki
* if it contains no dash then max 32 characters long, not including the .loki tld (and also assuming a leading subdomain has been stripped)
* These are from general DNS requirements, and also enforced in
registrations:
* Must be all [A-Za-z0-9-]. (A-Z will be lower-cased by the RPC call).
* cannot start or end with a -
* max 63 characters long if it does contain a dash
* cannot contain -- in the third and fourth characters unless it starts with xn--
* handle timeout in name lookup job by calling the right handler with std::nullopt
* update loki-mq submodule for tuple support
* srv record reply implementation
still need to encode srv records into intro sets / router contacts
as well as decode from them and match against queried service.proto
* inverted condition fix in config code
* SRV record struct (de-)serialization for intro sets
* parsing and using srv records from config (for/in introsets)
* adopt str utils from core and use for srv parsing
* changes to repeat requests
no longer drop repeat requests on the floor, but do not make
an *actual* request for them if one is in progress.
do not call reply hook for each reply for a request, as
each userland request is actually made into several lokinet
requests and this would result in duplicate replies.
* fetch SRVs from introsets for .loki
* make format
* dns and srv fixes, srv appears to be working
macOS doing a shared library build is not working without untangling
some of the interdependencies. This commit does that, at least enough
to get macOS to compile.
This isn't the cleanest as currently implemented (we have some net/
things in `liblokinet-platform` and some in `liblokinet`, and likewise
ev/vpnio.cpp is in `liblokinet` while the rest of `ev/*` is in
`liblokinet-platform`).
- Move IPRange into its own net/ip_range.hpp
- Move the static net::IPPacket::TruncateV6, etc. functions to free
net::TruncateV6, etc. functions (now from net/ip.hpp instead of
net/ip_packet.hpp).
- Make net::TruncateV6 and net::ExpandV4 constexpr.
- Add IPRange::FromIPv4 factory function (to replace the iprange_ipv4
free function)
Rename net/ip.{cpp,hpp} to net/ip_packet.{cpp,hpp}.
(Doing this in two commits because I want to repurpose ip.hpp/ip.cpp,
and want git to figure out the history properly).
Refactors many things in cmake to improve and simplify:
- don't use variable indirection for target names; target names are
*already* a variable of sorts. (e.g. ${UTIL_LIB} is now just
lokinet-util). cmake/basic_definitions.cmake is now gone.
- fix LTO enabling to use the standard cmake (3.9+) LTO mechanism rather
than shoving a bunch of flag hacks through link_libraries and
add_compile_options. This also now enables LTO when building a shared
library (because previously the -flto hacks were only turned on in the
static code for some reason).
- build liblokinet as *either* shared library or static library, but not
both. Building both makes things more complicated because they had
different names (lokinet-shared or lokinet-static) and seems pointless:
you generally want one or the other. Now there is just the liblokinet
target, which will be shared or static depending on the value of
BUILD_SHARED_LIBS.
- Simplify lokinet-cryptography AVX2 code: just build *one* library, and
add in the additional AVX2 files when possible, rather than building two
and needing to merge them.
- Compress STATIC_LINK and STATIC_LINK_RUNTIME into just STATIC_LINK.
It makes no sense to use one of these (_RUNTIME) on Windows and the
other on non-Windows when they appear to try to do the same thing.
- remove a bunch of annotations from `endif(FOO)` -> `endif()`.
- move all the tuntap compilation code (including OS-specific source
file selection) into vendor/CMakeLists.txt and build tuntap as an
intermediate OBJECT library rather than keeping a global variable in 5
different files.
- move release motto define to root cmake; it made no sense being
duplicated in both unix.cmake and win32.cmake
- fix add_log_tag to not stomp on any existing source compile flags with
its definition. Also use proper compile definition property instead of
cramming it into compile flags.
- make optimization/linker flags less hacky. There's no reason for us
to force particular optimization flags because the cmake build type
already does that (e.g. -DCMAKE_BUILD_TYPE=Release does -O3). Not doing
that also silences a bunch of cmake warnings because it thinks "-O0 -g3"
etc. are link libraries (which is reasonable: that's what the code was
telling cmake they are).
- sets the default build type to RelWithDebInfo which gives us `-O2 -g`
if you don't specify a build type.
- Move PIC up (so that the things loaded in unix.cmake, notably libuv,
have it set).
- Add a custom `curl` interface library that carries the correct link
target and include paths for curl (system or bundled).
Howard Hinnart's date.h is the library that was accepted as C++20
date/calendar support, so this is essentially a backport of C++20 date
time support.
(It does support timezone support, but requires more of the library and
that seems like overkill for what we need; this just prints UTC
timestamps instead, which need only a header-only include).
These aren't needed: CMake already knows how to follow #includes and
rebuild when headers change as long as the headers are included
*somewhere*. The extra .cpp files here just require building a bunch of
.cpp files with just header content that we just end up throw away
during linking (since the same things will also be compiled in whatever
other compilation units include the same headers).
- util::Mutex is now a std::shared_timed_mutex, which is capable of
exclusive and shared locks.
- util::Lock is still present as a std::lock_guard<util::Mutex>.
- the locking annotations are preserved, but updated to the latest
supported by clang rather than using abseil's older/deprecated ones.
- ACQUIRE_LOCK macro is gone since we don't pass mutexes by pointer into
locks anymore (WTF abseil).
- ReleasableLock is gone. Instead there are now some llarp::util helper
methods to obtain unique and/or shared locks:
- `auto lock = util::unique_lock(mutex);` gets an RAII-but-also
unlockable object (std::unique_lock<T>, with T inferred from
`mutex`).
- `auto lock = util::shared_lock(mutex);` gets an RAII shared (i.e.
"reader") lock of the mutex.
- `auto lock = util::unique_locks(mutex1, mutex2, mutex3);` can be
used to atomically lock multiple mutexes at once (returning a
tuple of the locks).
This are templated on the mutex which makes them a bit more flexible
than using a concrete type: they can be used for any type of lockable
mutex, not only util::Mutex. (Some of the code here uses them for
getting locks around a std::mutex). Until C++17, using the RAII types
is painfully verbose:
```C++
// pre-C++17 - needing to figure out the mutex type here is annoying:
std::unique_lock<util::Mutex> lock(mutex);
// pre-C++17 and even more verbose (but at least the type isn't needed):
std::unique_lock<decltype(mutex)> lock(mutex);
// our compromise:
auto lock = util::unique_lock(mutex);
// C++17:
std::unique_lock lock(mutex);
```
All of these functions will also warn (under gcc or clang) if you
discard the return value. You can also do fancy things like
`auto l = util::unique_lock(mutex, std::adopt_lock)` (which lets a
lock take over an already-locked mutex).
- metrics code is gone, which also removes a big pile of code that was
only used by metrics:
- llarp::util::Scheduler
- llarp:🧵:TimerQueue
- llarp::util::Stopwatch
Step 1 of removing abseil from lokinet.
For the most part this is a drop-in replacement, but there are also a
few changes here to the JSONRPC layer that were needed to work around
current gcc 10 dev snapshot:
- JSONRPC returns a json now instead of an optional<json>. It doesn't
make any sense to have a json rpc call that just closes the connection
with returning anything. Invoked functions can return a null (default
constructed) result now if they don't have anything to return (such a
null value won't be added as "result").
Our current abseil won't build with gcc 10 (its `optional`
implementation appears broken), and spews warnings under slightly older
compilers; updating to the latest stable 2019 branch fixes both issues.
This rewrites the version info using lokid's approach of compiling it
into a .cpp file that gets generated as part of the build (*not* during
the configure stage).
Among other things, this means that changing the version no longer
invalidates ccache or cmake dependencies, and because it depends on
`.git/index` git commits will cause the version to be regenerated,
making the commit tag more reliable (currently if you rebuild without
running cmake your git commit tag doesn't update).
