* Updated RpcServer Initialization and Logic
-- Moved all RPCServer initialization logic to rpcserver constructor
-- Fixed config logic, fxn binding to rpc address, fxn adding rpc cats
-- router hive failed CI/CD resulting from outdated reference to rpcBindAddr
-- ipc socket as default hidden from windows (for now)
* wintun vpn platform for windows
* bundle config snippets into nsis installer for exit node, keyfile persisting, reduced hops mode.
* use wintun for vpn platform
* isolate all windows platform specific code into their own compilation units and libraries
* split up internal libraries into more specific components
* rename liblokinet.a target to liblokinet-amalgum.a to elimiate ambiguity with liblokinet.so
* DNS platform for win32
* rename llarp/ev/ev_libuv.{c,h}pp to llarp/ev/libuv.{c,h}pp as the old name was idiotic
* split up net platform into win32 and posix specific compilation units
* rename lokinet_init.c to easter_eggs.cpp as that is what they are for and it does not need to be a c compilation target
* add cmake option STRIP_SYMBOLS for seperating out debug symbols for windows builds
* intercept dns traffic on all interfaces on windows using windivert and feed it into lokinet
we want to be able to have multiple locally bound dns sockets in lokinet so
i restructured most of the dns subsystem in order to make this easier.
specifically, we have a new structure to dns subsystem:
* dns::QueryJob_Base
base type for holding a dns query and response with virtual methods
in charge of sending a reply to whoever requested.
* dns::PacketSource_Base
base type for reading and writing dns messages to and from wherever they came from
* dns::Resolver_Base
base type for filtering and handling of dns messages asynchronously.
* dns::Server
contextualized per endpoint dns object, responsible for all dns related isms.
this change hides all impelementation details of all of the dns components.
adds some more helper functions for parsing dns and dealing with OwnedBuffer.
overall dns becomes less of a pain with this new structure. probably.
Replaces custom logging system with spdlog-based oxen logging. This
commit mainly replaces the backend logging with the spdlog-based system,
but doesn't (yet) convert all the existing LogWarn, etc. to use the new
format-based logging.
New logging statements will look like:
llarp::log::warning(cat, "blah: {}", val);
where `cat` should be set up in each .cpp or cluster of .cpp files, as
described in the oxen-logging README.
As part of spdlog we get fmt, which gives us nice format strings, where
are applied generously in this commit.
Making types printable now requires two steps:
- add a ToString() method
- add this specialization:
template <>
constexpr inline bool llarp::IsToStringFormattable<llarp::Whatever> = true;
This will then allow the type to be printed as a "{}" value in a
fmt::format string. This is applied to all our printable types here,
and all of the `operator<<` are removed.
This commit also:
- replaces various uses of `operator<<` to ToString()
- replaces various uses of std::stringstream with either fmt::format or
plain std::string
- Rename some to_string and toString() methods to ToString() for
consistency (and to work with fmt)
- Replace `stringify(...)` and `make_exception` usage with fmt::format
(and remove stringify/make_exception from util/str.hpp).
this allows you to use exit nodes without forcing routes over the interface, useful for using lokinet with an exit and selectively routing over the lokinet interface using an external socks proxy or binding to device explicitly.
* make route poker configurable, defaults to enabled but allows disabling it on runtime if desired
* add config option [network]:auto-routing to enable/disable route poker
on win32/apple reading packets from the interface does not count as an io operation.
manually trigger pump on win32/apple to pretend that it is an io event.
add platform quark function MaybeWakeUpperLayers on vpn::Interface to manaully wake up the other components on platforms that need that (ones on which packet io is not done via io events).
on non linux platforms, use uv_prepare_t instead of uv_check_t as the former triggers before blocking for io, instead of after. this better matches linux's order of operations in libuv.
* make it so that we don't set up unbound resolver when we have no resolvers provided by config
* clean up dns codepath and make it use llarp::SockAddr instead of llarp::IpAddress
* add option to persist address mappings between restarts using [network]:persist-addrmap-file
* make it work
* only persist address map for inbound convos
* turn persisting address map on by default
* dont load addrmap file if it has been modified last over a minute ago to prevent foot cannons fired from loading a really old version of it
adds [network] section parameter called path-alignment-timeout that allows configring the timeout
for optional name lookup + introset lookup + aligned path build, used by tun endpoint dns, provided
as milliseconds.
* add lokinet_add_bootstrap_rc function for adding an rc from memory
* prevent stack overflow on error closing connection in quic
* add in memory nodedb
* refactor how convotags are set as active
* add initial stubs for endpoint statistics
* refactor time stuff to be a bit cleaner
* update lnproxy script with more arguments
Refactors how quic packets get handled: the actual tunnels now live in
tunnel.hpp's TunnelManager which holds and manages all the quic<->tcp
tunnelling. service::Endpoint now holds a TunnelManager rather than a
quic::Server. We only need one quic server, but we need a separate quic
client instance per outgoing quic tunnel, and TunnelManager handles all
that glue now.
Adds QUIC packet handling to get to the right tunnel code. This
required multiplexing incoming quic packets, as follows:
Adds a very small quic tunnel packet header of 4 bytes:
[1, SPORT, ECN] for client->server packets, where SPORT is our
source "port" (really: just a uint16_t unique quic instance
identifier)
or
[2, DPORT, ECN] for server->client packets where the DPORT is the SPORT
from above.
(This also reworks ECN bits to get properly carried over lokinet.)
We don't need a destination/source port for the server-side because
there is only ever one quic server (and we know we're going to it when
the first byte of the header is 1).
Removes the config option for quic exposing ports; a full lokinet will
simply accept anything incoming on quic and tunnel it to the requested
port on the the local endpoint IP (this handler will come in a following
commit).
Replace ConvoTags with full addresses: we need to carry the port, as
well, which the ConvoTag can't give us, so change those to more general
SockAddrs from which we can extract both the ConvoTag *and* the port.
Add a pending connection queue along with new quic-side handlers to call
when a stream becomes available (TunnelManager uses this to wire up
pending incoming conns with quic streams as streams open up).
Completely get rid of tunnel_server/tunnel_client.cpp code; it is now
moved to tunnel.hpp.
Add listen()/forget() methods in TunnelManager for setting up quic
listening sockets (for liblokinet usage).
Add open()/close() methods in TunnelManager for spinning up new quic
clients for outgoing quic connections.
* add ip6-range option to network section to control which range we want to use for ipv6 traffic
* make ip6-range able to disable ipv6 exits ( i dont like this but eh )
All #ifndef guards on headers have been removed, I think,
in favor of #pragma once
Headers are now included as `#include "filename"` if the included file
resides in the same directory as the file including it, or any
subdirectory therein. Otherwise they are included as
`#include <project/top/dir/relative/path/filename>`
The above does not include system/os headers.
- Previous android java and jni code updated to work, but with much love
still needed to make it work nicely, e.g. handling when the VPN is
turned off.
- DNS handling refactored to allow android to intercept and handle DNS
requests as we can't set the system DNS to use a high port
(and apparently Chrome ignores system DNS settings anyway)
- add packet router structure to allow separate handling of specific
intercepted traffic, e.g. UDP traffic to port 53 gets handled by our
DNS handler rather than being naively forwarded as exit traffic.
- For now, android lokinet is exit-only and hard-coded to use exit.loki
as its exit. The exit will be configurable before release, but
allowing to not use exit-only mode is more of a challenge.
- some old gitignore remnants which were matching to things we don't
want them to (and are no longer relevant) removed
- some minor changes to CI configuration
* bump zmq static dep
* lokimq -> oxenmq
* llarp_nodedb -> llarp::NodeDB
* remove all crufty api parts of NodeDB
* make NodeDB rc selection api not suck
* make path builder api not suck
* propagate all above changes so that unit tests work and it all compiles
* llarp::Config constructor can take an empty fs::path now and it will default to fs::current_path
* llarp::ensureConfig now throws on fs error
* updates to daemon/main.cpp to reflect changes to llarp::ensureConfig's new throwability
* dowse parts of the code with holy water for the 0.8.1 tag
* add conf.d directory for config overrides
* refactor llarp::Config
* add explicit constructor with datadir as parameter
* have all config files be passed as std::optional
* make Config::LoadDefault private and use std::optional in Config::Load to remove ambiguity
* update rest of codebase to reflect above changes
* fix pybind
* rename bootstrap config skipBootstrap to seednode as it's more descriptive
* make seednode configurable
* make pybind layer compile
* make pybind layer run
* Config file API/comment improvements
API improvements:
=================
Make the config API use position-independent tag parameters (Required,
Default{123}, MultiValue) rather than a sequence of bools with
overloads. For example, instead of:
conf.defineOption<int>("a", "b", false, true, 123, [] { ... });
you now write:
conf.defineOption<int>("a", "b", MultiValue, Default{123}, [] { ... });
The tags are:
- Required
- MultiValue
- Default{value}
plus new abilities (see below):
- Hidden
- RelayOnly
- ClientOnly
- Comment{"line1", "line2", "line3"}
Made option definition more powerful:
=====================================
- `Hidden` allows you to define an option that won't show up in the
generated config file if it isn't set.
- `RelayOnly`/`ClientOnly` sets up an option that is only accepted and
only shows up for relay or client configs. (If neither is specified
the option shows up in both modes).
- `Comment{...}` lets the option comments be specified as part of the
defineOption.
Comment improvements
====================
- Rewrote comments for various options to expand on details.
- Inlined all the comments with the option definitions.
- Several options that were missing comments got comments added.
- Made various options for deprecated and or internal options hidden by
default so that they don't show up in a default config file.
- show the section comment (but not option comments) *after* the
[section] tag instead of before it as it makes more sense that way
(particularly for the [bind] section which has a new long comment to
describe how it works).
Disable profiling by default
============================
We had this weird state where we use and store profiling by default but
never *load* it when starting up. This commit makes us just not use
profiling at all unless explicitly enabled.
Other misc changes:
===================
- change default worker threads to 0 (= num cpus) instead of 1, and fix
it to allow 0.
- Actually apply worker-threads option
- fixed default data-dir value erroneously having quotes around it
- reordered ifname/ifaddr/mapaddr (was previously mapaddr/ifaddr/ifname)
as mapaddr is a sort of specialization of ifaddr and so makes more
sense to come after it (particularly because it now references ifaddr
in its help message).
- removed peer-stats option (since we always require it for relays and
never use it for clients)
- removed router profiles filename option (this doesn't need to be
configurable)
- removed defunct `service-node-seed` option
- Change default logging output file to "" (which means stdout), and
also made "-" work for stdout.
* Router hive compilation fixes
* Comments for SNApp SRV settings in ini file
* Add extra blank line after section comments
* Better deprecated option handling
Allow {client,relay}-only options in {relay,client} configs to be
specified as implicitly deprecated options: they warn, and don't set
anything.
Add an explicit `Deprecated` tag and move deprecated option handling
into definition.cpp.
* Move backwards compat options into section definitions
Keep the "addBackwardsCompatibleConfigOptions" only for options in
sections that no longer exist.
* Fix INI parsing issues & C++17-ify
- don't allow inline comments because it seems they aren't allowed in
ini formats in general, and is going to cause problems if there is a
comment character in a value (e.g. an exit auth string). Additionally
it was breaking on a line such as:
# some comment; see?
because it was treating only `; see?` as the comment and then producing
an error message about the rest of the line being invalid.
- make section parsing stricter: the `[` and `]` have to be at the
beginning at end of the line now (after stripping whitespace).
- Move whitespace stripping to the top since everything in here does it.
- chop off string_view suffix/prefix rather than maintaining position
values
- fix potential infinite loop/segfault when given a line such as `]foo[`
* Make config parsing failure fatal
Load() LogError's and returns false on failure, so we weren't aborting
on config file errors.
* Formatting: allow `{}` for empty functions/structs
Instead of using two lines when empty:
{
}
* Make default dns bind 127.0.0.1 on non-Linux
* Don't show empty section; fix tests
We can conceivably have sections that only make sense for clients or
relays, and so want to completely omit that section if we have no
options for the type of config being generated.
Also fixes missing empty lines between tests.
Co-authored-by: Thomas Winget <tewinget@gmail.com>
* serialize auth codes in rpc
* format
* format
* config and route fixups
* bump lokimq
* fix compile
* discard unknown members in service keys
* format
* update loki-mq submodule for tuple support
* srv record reply implementation
still need to encode srv records into intro sets / router contacts
as well as decode from them and match against queried service.proto
* inverted condition fix in config code
* SRV record struct (de-)serialization for intro sets
* parsing and using srv records from config (for/in introsets)
* adopt str utils from core and use for srv parsing
* changes to repeat requests
no longer drop repeat requests on the floor, but do not make
an *actual* request for them if one is in progress.
do not call reply hook for each reply for a request, as
each userland request is actually made into several lokinet
requests and this would result in duplicate replies.
* fetch SRVs from introsets for .loki
* make format
* dns and srv fixes, srv appears to be working
