From 9df6cd74c7cb5ba8f6ab82cee679f5f6ad1ba029 Mon Sep 17 00:00:00 2001 From: Jason Rhinelander Date: Tue, 26 Jan 2021 18:26:24 -0400 Subject: [PATCH] Add draft liblokinet TCP-over-QUIC design doc --- docs/tcp-over-quic.md | 216 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 216 insertions(+) create mode 100644 docs/tcp-over-quic.md diff --git a/docs/tcp-over-quic.md b/docs/tcp-over-quic.md new file mode 100644 index 000000000..7412759ce --- /dev/null +++ b/docs/tcp-over-quic.md @@ -0,0 +1,216 @@ +# "liblokinet" TCP-over-QUIC + +In order for lokinet to work in an embedded version (which I will call "liblokinet" in this +document), which lokinet cannot create TUN device (either because the host OS doesn't support them, +or because lokinet needs to run without permissions to manage them) lokinet needs a solution for +sending TCP data from the device to a remote lokinet client (i.e. a snapp, a snode, or another +liblokinet client). Since the vast majority of network connectivity relies on TCP stream +connections, not supporting them would be a severe limitation of a lokinet library that would make +it nearly useless. + +Traditional "full" lokinet does not need to solve this problem: it creates virtual IPs on the TUN +interface that map to every looked-up `.loki` address and then the host system's in-kernel TCP layer +handles the intricacies of TCP including acknowledgement, retry, and so on. While there are +user-space TCP implementations available, they are generally incomplete, unmaintained, or both, +which would mean substantial work and ongoing maintenance for us to adopt or reimplement such a +user-space TCP layer, for which we would most likely be the only user and contributor. + +Instead this proposal is for lokinet to support a tunneled TCP stream mode where TCP traffic is +carried over lokinet via a subset of the +[QUIC](https://datatracker.ietf.org/doc/draft-ietf-quic-transport/) protocol. Unlike TCP, QUIC has +several well-maintained user-space implementations which allow us to use, rather than create, a +well-maintained QUIC implementation. + +## Overview + +The high-level strategy of how we handle such a stream connection is to have TCP connections +established only within the local device. A liblokinet application would invoke a lokinet call to +establish such a connection to proxy to a remote host by lokinet name and TCP port. This would +first establish a lokinet connection to the remote host, then open a QUIC connection over it and +start listening for TCP connections on a local port. When a new TCP connection is established on +this port lokinet will establish a new QUIC stream over the existing connection, specifying the +destination port while initializing the stream. (The client is free to establish as many TCP +connections as it wants: each one becomes a separate QUIC stream). + +The situation is similar for the receiving lokinet client: it would listen for incoming QUIC +connections on the local lokinet IP and, when establishing a QUIC stream, would establish a local +TCP connection to the requested port on the lokinet IP. Any incoming stream data is then forwarded +into this TCP connection, and any responses are sent back via the QUIC stream. + +## Example + +For example, suppose `snap7.loki` is a lokinet snapp with a web server listening on port 80 and a +liblokinet client `omg42.loki` wants to connect to it to retrieve a cat photo. With a full +lokinet client, the DNS request for `omg56789.loki` triggers creation of a virtual IP on the TUN +device, returns the IP to the system, and any TCP packets sent to this IP are forwarded to the +primary lokinet IP of `azfoj123.loki`, where an HTTP server is ready and waiting to provide cat +photos. + +With a liblokinet client, this process will looks a little different: the client will first make a +call to the liblokinet library (rather than a DNS request) specifying the lokinet host name and TCP +port it wants to connect to (note that this is pseudo-code; the actual implementation calls will +have to deal with various details such as connection delays and timeouts that are omitted here): + + result = lokinet_stream_connect(lokinet_addr, port) + if result->connection_established: + http_get("http://" + result->local_address + ":" + result->local_port + "/cat.jpg") + +Here `http_get` would need no knowledge of lokinet at all: it will simply connect via TCP to an +address such as `127.0.0.1:4716` for the HTTP request. It will send the request, and receive it, +over this localhost TCP socket. + +Internally, lokinet will have established a QUIC connection to the remote host, and started +listening for TCP connections on the localhost port. When `http_get` establishes a TCP connection +on this local port it will create a QUIC stream on the established QUIC connection and forward all +stream data received from the TCP connection into the QUIC stream, and any data that comes back over +the QUIC stream will similarly be copied into the localhost TCP connection. + +Effectively the data path of data send from the app on omg42.loki to the HTTP snapp on omg42.loki +looks like this: + + ┌omg42.loki────────────┐ ┌snap7.loki───────────┐ + │ Main app thread │ │ HTTP │ + │ TCP localhost:4567 ─>│─┐ │ TCP 172.16.0.1:80 <─│─┐ + ├──────────────────────┤ │ ╞═════════════════════╡ │ + │ liblokinet (in app) │ │ │ lokinet (on host) │ │ + │ TCP localhost:4567 <─│─┘ ┌>│─> QUIC UDP │ │ + │ QUIC UDP ─>│───... Lokinet routers ...─┘ │ TCP 172.16.0.1:80 ─>│─┘ + └──────────────────────┘ └─────────────────────┘ + +(These connections are all bi-direction, so any TCP stream data replied from omg42.loki follows the +same path in reverse.) + +## Implementation details/notes + +Implementation library: `ngtcp2` is a robust, maintained library that fits our needs well. + +### Not a general QUIC server/client + +The QUIC tunnel described here is *only* for Lokinet TCP streams; it is not intended to be +interoperable with general QUIC clients, which allows us some leeway to not support some aspects of +QUIC that are of no advantage over a lokinet conversation. + +### No encryption + +Since lokinet traffic is itself encrypted and private, the built-in TLS layers of QUIC are something +that we don't need or want. Thus the QUIC implementation used will simply use no-op encryption to +pass data and avoid/ignore certificates. (`ngtcp2`, in particular, allows pluggable authentication +to allow this). + +### No address verification + +QUIC recommends address verification (among other things, to avoid amplification attacks). Lokinet +connections already provide this and so we can safely not use it. + +### Stream establishing + +Establishing a QUIC stream requires sending additional information during connection: namely the +target connection port. Thus establishing a new stream will require some additional data to be +passed, likely as the initial stream data. (Specification of how this data is to be encoded is not +yet specified). + +### Incoming TCP-over-QUIC connections + +Handling of incoming connections to a liblokinet client will require a similar process, but in +reverse: + +- the client starts listening on a localhost TCP port +- the client makes a call to liblokinet to inform it of this available listening port +- incoming QUIC tunneled streams attempting to connect to that registered port are accepted and + establish a new TCP connection as long as the stream stays open; data is forwarded between the two + connections. +- Should the client require end-point verification liblokinet will provide a function that can look + up the remote lokinet address based on the source port of the TCP connection. (This is different + from but analogous to a snapp doing a reverse DNS lookup on the source address to determine the + remote address). + +Note: to be externally reachable by other lokinet clients, a liblokinet client would have to publish +an introset; this introset would also include an additional flag indicating that TCP connections +must be tunneled through a TCP-over-QUIC connection. + +Note 2: we additionally may want to signal during connection that new TCP connections back to us +should be done over a QUIC tunnel, which requires also adding a flag when establishing the +lokinet conversation. + +### Non-tunneled incoming TCP connections + +Without a controllable TCP stack we have no ability to accept these, however since the introset (and +conversation initiation) indicates that TCP should be tunneled, we should just drop these packets. + +## Lokinet implementation notes + +### Outbound connections -- liblokinet + +The application makes a liblokinet library call such as + + lokinet_stream_result res; + lokinet_outbound_stream(&res, "some-snapp.loki", 2345); + +This initiates an outbound connection to the given lokinet remote, asking to connect to port 2345 on +the remote. Plainquic begins listening on a random localhost port, and returns this via an entry in +`res`. New connections establishes to this localhost port initiate new streams on the quic +connection which are tunneled to the remote end. + +### Inbound connections -- liblokinet + +The application needs to start listening on one or more TCP ports (e.g. on localhost, but doesn't +have to be) and then registers a callback with lokinet about the availability of this port for +incoming plainquic connections by setting up a callback: + +```C + int accept_inbound(const char *lokinet_addr, uint16_t port, sockaddr *addr, void *context) { + // lokinet_addr is the remote lokinet client trying to establish a stream + // port is the port they are trying to reach + // If the client is allowed then set the local TCP socket address that the tunnel should + // connect to in `addr` (which is big enough to allow either sockaddr_in or sockaddr_in6) + sockaddr_in* a = (sockaddr_in*)addr; + a->sin_family = AF_INET; + a->sin_addr = INADDR_LOOPBACK; + a->sin_port = htons(5678); // NB: Doesn't have to be the passed-in `port` + return 0; + // If this callback doesn't handle the requested port (will try other callbacks): + return -1; + // If this callback does handle it and the connection should be refused: + return -2; + // (Return values other than 0/-1/-2 are reserved and should not be used). + } + lokinet_inbound_stream(&accept_inbound, NULL /*context*/); +``` +or, for the very simple case where connections should be available on some localhost port: +```C + // All incoming tunneled connections for port 5678 should go to localhost:5678 + lokinet_inbound_stream_simple(5678); +``` + +When a new plainquic connection arrives, if such a callback has been registered it will be called to +determine whether the connection should be accepted and, if it is, where streams opened on that +connection should be sent. (For the simple version, all inbound connections on port 5678 would be +accepted and would be forwarded to localhost:5678; inbound connections for other ports would be +refused). + +Each new plainquic stream initiated by the remote connection then establishes a new TCP connection +to the IP/port set by the callback. + +### Outbound connections - full lokinet + +When attempting to connect to a client who has indicated in its introset that it requires plainquic +connections then plainquic will bind to and listen on the virtual (tun) TCP/IP port and establish a +plainquic connection to the remote liblokinet on the given port. (Future connections to this port +will establish new streams on the existing connection). + +Setting up the initial listener involves intercepting the initial TCP connection attempt (i.e. the +SYN packet), starting to listen on it while simultaneously initiating the plainquic connection over +lokinet. It may work sufficiently well (investigation required) to simply drop this initial SYN +packet and let the initiator retry in a few moments to attack to the new listener which now goes +into the plainquic listener which establishes a new stream. + +Thereafter the local application simply talks to this local listener and all stream data gets +tunneled over lokinet to the remote liblokinet. + +### Inbound connections - full lokinet + +This is fairly simple: when incoming quic-tunneled packets arrive we start up a plainquic server (if +not already running), deliver the packets into it, and it tunnels incoming stream data into TCP +connections to the primary lokinet IP (using the IP mapped to the lokinet endpoint as the source +address). +