mirror of https://github.com/oxen-io/lokinet
Add draft liblokinet TCP-over-QUIC design doc
parent
8ee80bc13d
commit
9df6cd74c7
@ -0,0 +1,216 @@
|
||||
# "liblokinet" TCP-over-QUIC
|
||||
|
||||
In order for lokinet to work in an embedded version (which I will call "liblokinet" in this
|
||||
document), which lokinet cannot create TUN device (either because the host OS doesn't support them,
|
||||
or because lokinet needs to run without permissions to manage them) lokinet needs a solution for
|
||||
sending TCP data from the device to a remote lokinet client (i.e. a snapp, a snode, or another
|
||||
liblokinet client). Since the vast majority of network connectivity relies on TCP stream
|
||||
connections, not supporting them would be a severe limitation of a lokinet library that would make
|
||||
it nearly useless.
|
||||
|
||||
Traditional "full" lokinet does not need to solve this problem: it creates virtual IPs on the TUN
|
||||
interface that map to every looked-up `.loki` address and then the host system's in-kernel TCP layer
|
||||
handles the intricacies of TCP including acknowledgement, retry, and so on. While there are
|
||||
user-space TCP implementations available, they are generally incomplete, unmaintained, or both,
|
||||
which would mean substantial work and ongoing maintenance for us to adopt or reimplement such a
|
||||
user-space TCP layer, for which we would most likely be the only user and contributor.
|
||||
|
||||
Instead this proposal is for lokinet to support a tunneled TCP stream mode where TCP traffic is
|
||||
carried over lokinet via a subset of the
|
||||
[QUIC](https://datatracker.ietf.org/doc/draft-ietf-quic-transport/) protocol. Unlike TCP, QUIC has
|
||||
several well-maintained user-space implementations which allow us to use, rather than create, a
|
||||
well-maintained QUIC implementation.
|
||||
|
||||
## Overview
|
||||
|
||||
The high-level strategy of how we handle such a stream connection is to have TCP connections
|
||||
established only within the local device. A liblokinet application would invoke a lokinet call to
|
||||
establish such a connection to proxy to a remote host by lokinet name and TCP port. This would
|
||||
first establish a lokinet connection to the remote host, then open a QUIC connection over it and
|
||||
start listening for TCP connections on a local port. When a new TCP connection is established on
|
||||
this port lokinet will establish a new QUIC stream over the existing connection, specifying the
|
||||
destination port while initializing the stream. (The client is free to establish as many TCP
|
||||
connections as it wants: each one becomes a separate QUIC stream).
|
||||
|
||||
The situation is similar for the receiving lokinet client: it would listen for incoming QUIC
|
||||
connections on the local lokinet IP and, when establishing a QUIC stream, would establish a local
|
||||
TCP connection to the requested port on the lokinet IP. Any incoming stream data is then forwarded
|
||||
into this TCP connection, and any responses are sent back via the QUIC stream.
|
||||
|
||||
## Example
|
||||
|
||||
For example, suppose `snap7.loki` is a lokinet snapp with a web server listening on port 80 and a
|
||||
liblokinet client `omg42.loki` wants to connect to it to retrieve a cat photo. With a full
|
||||
lokinet client, the DNS request for `omg56789.loki` triggers creation of a virtual IP on the TUN
|
||||
device, returns the IP to the system, and any TCP packets sent to this IP are forwarded to the
|
||||
primary lokinet IP of `azfoj123.loki`, where an HTTP server is ready and waiting to provide cat
|
||||
photos.
|
||||
|
||||
With a liblokinet client, this process will looks a little different: the client will first make a
|
||||
call to the liblokinet library (rather than a DNS request) specifying the lokinet host name and TCP
|
||||
port it wants to connect to (note that this is pseudo-code; the actual implementation calls will
|
||||
have to deal with various details such as connection delays and timeouts that are omitted here):
|
||||
|
||||
result = lokinet_stream_connect(lokinet_addr, port)
|
||||
if result->connection_established:
|
||||
http_get("http://" + result->local_address + ":" + result->local_port + "/cat.jpg")
|
||||
|
||||
Here `http_get` would need no knowledge of lokinet at all: it will simply connect via TCP to an
|
||||
address such as `127.0.0.1:4716` for the HTTP request. It will send the request, and receive it,
|
||||
over this localhost TCP socket.
|
||||
|
||||
Internally, lokinet will have established a QUIC connection to the remote host, and started
|
||||
listening for TCP connections on the localhost port. When `http_get` establishes a TCP connection
|
||||
on this local port it will create a QUIC stream on the established QUIC connection and forward all
|
||||
stream data received from the TCP connection into the QUIC stream, and any data that comes back over
|
||||
the QUIC stream will similarly be copied into the localhost TCP connection.
|
||||
|
||||
Effectively the data path of data send from the app on omg42.loki to the HTTP snapp on omg42.loki
|
||||
looks like this:
|
||||
|
||||
┌omg42.loki────────────┐ ┌snap7.loki───────────┐
|
||||
│ Main app thread │ │ HTTP │
|
||||
│ TCP localhost:4567 ─>│─┐ │ TCP 172.16.0.1:80 <─│─┐
|
||||
├──────────────────────┤ │ ╞═════════════════════╡ │
|
||||
│ liblokinet (in app) │ │ │ lokinet (on host) │ │
|
||||
│ TCP localhost:4567 <─│─┘ ┌>│─> QUIC UDP │ │
|
||||
│ QUIC UDP ─>│───... Lokinet routers ...─┘ │ TCP 172.16.0.1:80 ─>│─┘
|
||||
└──────────────────────┘ └─────────────────────┘
|
||||
|
||||
(These connections are all bi-direction, so any TCP stream data replied from omg42.loki follows the
|
||||
same path in reverse.)
|
||||
|
||||
## Implementation details/notes
|
||||
|
||||
Implementation library: `ngtcp2` is a robust, maintained library that fits our needs well.
|
||||
|
||||
### Not a general QUIC server/client
|
||||
|
||||
The QUIC tunnel described here is *only* for Lokinet TCP streams; it is not intended to be
|
||||
interoperable with general QUIC clients, which allows us some leeway to not support some aspects of
|
||||
QUIC that are of no advantage over a lokinet conversation.
|
||||
|
||||
### No encryption
|
||||
|
||||
Since lokinet traffic is itself encrypted and private, the built-in TLS layers of QUIC are something
|
||||
that we don't need or want. Thus the QUIC implementation used will simply use no-op encryption to
|
||||
pass data and avoid/ignore certificates. (`ngtcp2`, in particular, allows pluggable authentication
|
||||
to allow this).
|
||||
|
||||
### No address verification
|
||||
|
||||
QUIC recommends address verification (among other things, to avoid amplification attacks). Lokinet
|
||||
connections already provide this and so we can safely not use it.
|
||||
|
||||
### Stream establishing
|
||||
|
||||
Establishing a QUIC stream requires sending additional information during connection: namely the
|
||||
target connection port. Thus establishing a new stream will require some additional data to be
|
||||
passed, likely as the initial stream data. (Specification of how this data is to be encoded is not
|
||||
yet specified).
|
||||
|
||||
### Incoming TCP-over-QUIC connections
|
||||
|
||||
Handling of incoming connections to a liblokinet client will require a similar process, but in
|
||||
reverse:
|
||||
|
||||
- the client starts listening on a localhost TCP port
|
||||
- the client makes a call to liblokinet to inform it of this available listening port
|
||||
- incoming QUIC tunneled streams attempting to connect to that registered port are accepted and
|
||||
establish a new TCP connection as long as the stream stays open; data is forwarded between the two
|
||||
connections.
|
||||
- Should the client require end-point verification liblokinet will provide a function that can look
|
||||
up the remote lokinet address based on the source port of the TCP connection. (This is different
|
||||
from but analogous to a snapp doing a reverse DNS lookup on the source address to determine the
|
||||
remote address).
|
||||
|
||||
Note: to be externally reachable by other lokinet clients, a liblokinet client would have to publish
|
||||
an introset; this introset would also include an additional flag indicating that TCP connections
|
||||
must be tunneled through a TCP-over-QUIC connection.
|
||||
|
||||
Note 2: we additionally may want to signal during connection that new TCP connections back to us
|
||||
should be done over a QUIC tunnel, which requires also adding a flag when establishing the
|
||||
lokinet conversation.
|
||||
|
||||
### Non-tunneled incoming TCP connections
|
||||
|
||||
Without a controllable TCP stack we have no ability to accept these, however since the introset (and
|
||||
conversation initiation) indicates that TCP should be tunneled, we should just drop these packets.
|
||||
|
||||
## Lokinet implementation notes
|
||||
|
||||
### Outbound connections -- liblokinet
|
||||
|
||||
The application makes a liblokinet library call such as
|
||||
|
||||
lokinet_stream_result res;
|
||||
lokinet_outbound_stream(&res, "some-snapp.loki", 2345);
|
||||
|
||||
This initiates an outbound connection to the given lokinet remote, asking to connect to port 2345 on
|
||||
the remote. Plainquic begins listening on a random localhost port, and returns this via an entry in
|
||||
`res`. New connections establishes to this localhost port initiate new streams on the quic
|
||||
connection which are tunneled to the remote end.
|
||||
|
||||
### Inbound connections -- liblokinet
|
||||
|
||||
The application needs to start listening on one or more TCP ports (e.g. on localhost, but doesn't
|
||||
have to be) and then registers a callback with lokinet about the availability of this port for
|
||||
incoming plainquic connections by setting up a callback:
|
||||
|
||||
```C
|
||||
int accept_inbound(const char *lokinet_addr, uint16_t port, sockaddr *addr, void *context) {
|
||||
// lokinet_addr is the remote lokinet client trying to establish a stream
|
||||
// port is the port they are trying to reach
|
||||
// If the client is allowed then set the local TCP socket address that the tunnel should
|
||||
// connect to in `addr` (which is big enough to allow either sockaddr_in or sockaddr_in6)
|
||||
sockaddr_in* a = (sockaddr_in*)addr;
|
||||
a->sin_family = AF_INET;
|
||||
a->sin_addr = INADDR_LOOPBACK;
|
||||
a->sin_port = htons(5678); // NB: Doesn't have to be the passed-in `port`
|
||||
return 0;
|
||||
// If this callback doesn't handle the requested port (will try other callbacks):
|
||||
return -1;
|
||||
// If this callback does handle it and the connection should be refused:
|
||||
return -2;
|
||||
// (Return values other than 0/-1/-2 are reserved and should not be used).
|
||||
}
|
||||
lokinet_inbound_stream(&accept_inbound, NULL /*context*/);
|
||||
```
|
||||
or, for the very simple case where connections should be available on some localhost port:
|
||||
```C
|
||||
// All incoming tunneled connections for port 5678 should go to localhost:5678
|
||||
lokinet_inbound_stream_simple(5678);
|
||||
```
|
||||
|
||||
When a new plainquic connection arrives, if such a callback has been registered it will be called to
|
||||
determine whether the connection should be accepted and, if it is, where streams opened on that
|
||||
connection should be sent. (For the simple version, all inbound connections on port 5678 would be
|
||||
accepted and would be forwarded to localhost:5678; inbound connections for other ports would be
|
||||
refused).
|
||||
|
||||
Each new plainquic stream initiated by the remote connection then establishes a new TCP connection
|
||||
to the IP/port set by the callback.
|
||||
|
||||
### Outbound connections - full lokinet
|
||||
|
||||
When attempting to connect to a client who has indicated in its introset that it requires plainquic
|
||||
connections then plainquic will bind to and listen on the virtual (tun) TCP/IP port and establish a
|
||||
plainquic connection to the remote liblokinet on the given port. (Future connections to this port
|
||||
will establish new streams on the existing connection).
|
||||
|
||||
Setting up the initial listener involves intercepting the initial TCP connection attempt (i.e. the
|
||||
SYN packet), starting to listen on it while simultaneously initiating the plainquic connection over
|
||||
lokinet. It may work sufficiently well (investigation required) to simply drop this initial SYN
|
||||
packet and let the initiator retry in a few moments to attack to the new listener which now goes
|
||||
into the plainquic listener which establishes a new stream.
|
||||
|
||||
Thereafter the local application simply talks to this local listener and all stream data gets
|
||||
tunneled over lokinet to the remote liblokinet.
|
||||
|
||||
### Inbound connections - full lokinet
|
||||
|
||||
This is fairly simple: when incoming quic-tunneled packets arrive we start up a plainquic server (if
|
||||
not already running), deliver the packets into it, and it tunnels incoming stream data into TCP
|
||||
connections to the primary lokinet IP (using the IP mapped to the lokinet endpoint as the source
|
||||
address).
|
||||
|
Loading…
Reference in New Issue