@ -40,11 +40,11 @@ We've already seen the +commitment_signed+ and +revoke_and_ack+ in <<payment_cha
Alice and Bob start with a payment channel that has 70,000 satoshi balance on each side.
As we saw in <<payment_channel>> this means that Alice and Bob have negotiated and each hold commitment transactions. These commitment transactions are asymmetric, delayed and revocable, and look like the example in <<alice_bob_commitment_txs_1>> below:
As we saw in <<payment_channels>> this means that Alice and Bob have negotiated and each hold commitment transactions. These commitment transactions are asymmetric, delayed and revocable, and look like the example in <<alice_bob_commitment_txs_1>> below:
[[alice_bob_commitment_txs_1]]
.Alice and Bob's initial commitment transactions:
image:images/alice_bob_commitment_txs_1.png[]
.Alice and Bob's initial commitment transactions
image:images/alice_bob_commitment_txs_1.png["Alice and Bob's initial commitment transactions"]
==== Adding an HTLC
@ -55,10 +55,11 @@ To add the HTLC, Alice starts the flow we saw in <<HTLC_commitment_message_flow>
[[update_add_htlc]]
==== The +update_add_htlc+ message
Alice sends the `update_add_HTLC` Lightning message to Bob. This message is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#adding-an-htlc-update_add_htlc[BOLT #2 - Peer Protocol - update_add_HTLC], and is shown in <<update_add_HTLC_message_fields>> below:
Alice sends the `update_add_HTLC` Lightning message to Bob. This message is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#adding-an-htlc-update_add_htlc[BOLT #2 - Peer Protocol - update_add_HTLC], and is shown below:
[[update_add_HTLC_message_fields]]
.The update_add_HTLC message
====
----
[channel_id:channel_id]
[u64:id]
@ -67,6 +68,7 @@ Alice sends the `update_add_HTLC` Lightning message to Bob. This message is defi
[u32:cltv_expiry]
[1366*byte:onion_routing_packet]
----
====
+channel_id+:: This is the channel that Alice has with Bob that she wants to add the HTLC. Remember that Alice and Bob may have multiple channels with each other.
@ -91,34 +93,34 @@ The received information is enough for Bob to create a new commitment transactio
We've already seen the basic structure of an HTLC in <<routing>>. The complete script of an offered HTLC is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/03-transactions.md#offered-htlc-outputs[BOLT #3 - Transactions - Offered HTLC output] and is shown in <<offered_htlc_output_script>> below:
<1> The first clause of the OP_IF conditional is redeemable by Alice with a revocation key. If this commitment is later revoked, Alice will have a revocation key to claim this output in a penalty transaction, taking the whole channel balance.
<2> The second clause is redeemable by the pre-image (payment secret or in our example Dina's secret) if it is revealed. This allows Bob to claim this output if he has the secret from Dina, meaning he has successfully delivered the payment to Dina.
<3> The third and final clause is a refund of the HTLC to Alice, if the HTLC expires without reaching Dina. It is timelocked with the expiration +cltv_expiry+. This ensures that Alice's balance is not "stuck" in an HTLC that can't be routed to Dina.
====
There are three ways to claim this output. Try to read the script and see if you can figure it out (remember, it is a stack-based language so things appear "backwards"). In the order they appear in the script, these are:
revocation:: The first clause of the OP_IF conditional is redeemable by Alice with a revocation key. If this commitment is later revoked, Alice will have a revocation key to claim this output in a penalty transaction, taking the whole channel balance.
redemption:: The second clause is redeemable by the pre-image (payment secret or in our example Dina's secret) if it is revealed. This allows Bob to claim this output if he has the secret from Dina, meaning he has successfully delivered the payment to Dina.
refund:: The third and final clause is a refund of the HTLC to Alice, if the HTLC expires without reaching Dina. It is timelocked with the expiration +cltv_expiry+. This ensures that Alice's balance is not "stuck" in an HTLC that can't be routed to Dina.
There are three ways to claim this output. Try to read the script and see if you can figure it out (remember, it is a stack-based language so things appear "backwards").
==== New commitment with HTLC output
@ -136,12 +138,14 @@ Alice sends +commitment_signed+ to Bob, with the signature for the new commitmen
[[commitment_signed_message]]
.The commitment_signed message
====
----
[channel_id:channel_id]
[signature:signature]
[u16:num_htlcs]
[num_htlcs*signature:htlc_signature]
----
====
The fields +num_htlcs+ and +htlc_signature+ now make more sense:
@ -163,11 +167,13 @@ Now that Bob has a new signed commitment, he needs to acknowledge it and revoke
[[revoke_and_ack_message_2]]
.The +revoke_and_ack+ message
====
----
[channel_id:channel_id]
[32*byte:per_commitment_secret]
[point:next_per_commitment_point]
----
====
Bob sends the +per_commitment_secret+ that allows Alice to construct a revocation key to build a penalty transaction spending Bob's old commitment. Once Bob has sent this, he cannot ever publish "Commitment #2" without risking a penalty transaction and losing all his money. So, the old commitment is effectively revoked.
@ -35,6 +35,7 @@ There may be many paths that make it possible for Alice to reach Dina. We will e
As a reminder, the path selected by Alice is shown in <<alice_dina_path>>, below:
[[alice_dina_path]]
.Path: Alice to Bob to Chan to Dina
image::images/alice_dina_path.png["Alice to Bob to Chan to Dina"]
Let's see how Alice can use this path without revealing information to intermediaries Bob and Chan.
@ -110,7 +111,7 @@ From <<routing>> we know that Alice will send a 50,000 satoshi payment to Dina v
.Payment path with HTLCs from Alice to Dina
image::images/alice-dina-htlc.png[Payment path with HTLCs from Alice to Dina]
As we see in <<gossip>>, Alice is able to construct this path to Dina because Lightning nodes announce their channels to the entire Lightning Network using the _Lightning Gossip Protocol_. After the initial channel announcement, Bob and Chan each sent out an additional "channel update" message with their routing fee and timelock expectations for payment routing.
As we will see in <<gossip>>, Alice is able to construct this path to Dina because Lightning nodes announce their channels to the entire Lightning Network using the _Lightning Gossip Protocol_. After the initial channel announcement, Bob and Chan each sent out an additional "channel update" message with their routing fee and timelock expectations for payment routing.
From the announcements and updates, Alice knows the following information about the channels between Bob, Chan and Dina:
@ -132,7 +133,12 @@ Alice already knows her own channel to Bob, and therefore doesn't need this info
==== Alice constructs the payloads
There are two possible formats that Alice can use for the information communicated to each hop: A legacy fixed-length format called the _hop data_ and a more flexible Type-Length-Value (TLV) based format called the _hop payload_. The TLV message format is explained in more detail in <<tlv>>. It offers flexibility by allowing fields to be added to the protocol at will. Both formats are specified in https://github.com/lightningnetwork/lightning-rfc/blob/master/04-onion-routing.md#packet-structure[BOLT #4 - Onion Routing - Packet Structure]
There are two possible formats that Alice can use for the information communicated to each hop: A legacy fixed-length format called the _hop data_ and a more flexible Type-Length-Value (TLV) based format called the _hop payload_. The TLV message format is explained in more detail in <<tlv>>. It offers flexibility by allowing fields to be added to the protocol at will.
[NOTE]
====
Both formats are specified in https://github.com/lightningnetwork/lightning-rfc/blob/master/04-onion-routing.md#packet-structure[BOLT #4 - Onion Routing - Packet Structure.]
====
Alice will start building the hop data from the end of the path backwards: Dina, Chan, Bob.
@ -222,7 +228,7 @@ image::images/onion_hop_payloads.png[Hop payloads for all the hops]
Alice must now generate several keys that will be used to encrypt the various layers in the onion.
Remember the goals of onion routing, that Alice can achieve with these keys:
With these keys, Alice can achieve a high degree of privacy and integrity:
* Alice can encrypt each layer of the onion so that only the intended recipient can read it.
* Every intermediary can check that the message is not modified.
@ -232,7 +238,7 @@ Remember the goals of onion routing, that Alice can achieve with these keys:
[WARNING]
====
Like a chopped onion, the following technical details may bring tears to your eyes. Feel free to skip to the next section if you get confused. Come back to this and read BOLT #4 if you want to learn more.
Like a chopped onion, the following technical details may bring tears to your eyes. Feel free to skip to the next section if you get confused. Come back to this and read https://github.com/lightningnetwork/lightning-rfc/blob/master/04-onion-routing.md#packet-construction[BOLT #4 - Onion Routing - Packet Construction], if you want to learn more.
====
@ -278,56 +284,58 @@ One important detail that seems almost magical is the ability for Alice to creat
// To editor: Maybe put this in an appendix instead of a sidebar?
[[ecdh]]
[[ecdh_explained]]
.Elliptic Curve Diffie-Hellman (ECDH) explained
****
Assume Alice's private key is +a+ and Bob's private key is +b+. Using the Elliptic Curve, they multiply each private key by the generator point +G+ to produce their public keys +A+ and +B+ respectively:
Assume Alice's private key is +a+ and Bob's private key is +b+. Using the Elliptic Curve, Alice and Bob each multiply their private key by the generator point +G+ to produce their public keys +A+ and +B+ respectively:
A = aG
B = bG
Now Alice and Bob can create a shared secret +ss+, a value that they can both calculate independently without exchanging any information, such that
Now Alice and Bob can use _Elliptic Curve Diffie-Hellman Key Exchange_ to create a shared secret +ss+, a value that they can both calculate independently without exchanging any information
The shared secret +ss+ is calculated by each by multiplying their own private key with the *other's* public key, such that:
ss = aB = bA
But why would these two multiplications result in the same value +ss+?
Follow along, as we demonstrate the math that proves this is possible:
ss
= aB
calculated by Alice who knows both +a+ and +B+
calculated by Alice who knows both +a+ (her private key) and +B+ (Bob's public key)
= a(bG)
because we know
because we know that B = bG, we substitute
= (ab)G
because of associativity
because of associativity, we can move the parentheses
= (ba)G
because the curve is an abelian group
because xy = yx (the curve is an abelian group)
= b(aG)
because of associativity
because of associativity, we can move the parentheses
= bA
can be calculated by Bob who knows +b+ and +A+
because A = aG, we can substitute
The result bA can be calculated independently by Bob who knows +b+ (his private key) and +A+ (Alice's public key).
We have therefore shown that
ss = aB = bA
Alice can multiple her private key with Bob's public key to calculate +ss+
ss = aB (Alice can calculate this)
ss = bA (Bob can calculate this)
Bob can multiply his private key with Alice's public key to calculate +ss+
Thus, they will both get the same result which they can use as a shared key to symmetrically encrypt secrets between the two of them without communicating the shared secret.
Thus, they can each independently calculate +ss+ which they can use as a shared key to symmetrically encrypt secrets between the two of them without communicating the shared secret.
****
@ -612,7 +620,7 @@ onion_routing_packet:: The final onion packet Alice constructed with all the hop
==== Bob checks the onion
As we saw in <<channel_operations>>, Bob will add the HTLC to the commitment transactions and update the state of the channel with Alice.
As we saw in <<channel_operation>>, Bob will add the HTLC to the commitment transactions and update the state of the channel with Alice.
Bob will unwrap the onion he received from Alice as follows:
@ -823,3 +831,5 @@ One proposed solution to this problem is called "stuckless payments", and it dep
=== Conclusion
The Lightning Network's onion routing protocol is adapted from the Sphinx Protocol to better serve the needs of a payment network. As such, it offers a huge improvement in privacy and counter-surveillance compared to the public and transparent Bitcoin blockchain.
In <<path_finding>> we will see how the combination of source routing and onion routing is used by Alice to find a good path and route the payment to Dina. To find a path, Alice first needs to learn about the network topology, which is the topic of the next section <<gossip>>.
@ -89,27 +89,26 @@ existing Internet protocols to maintain and distribute a set of bootstrapping pe
* Internet Relay Chat (IRC)
* Hyper-Text Transfer Protocol (HTTP)
Similar to the Bitcoin protocol, the primary initial peer discovery mechanism used in the Lightning Network happens via DNS. As initial peer discovery is a critical and universal task for the network, the process has been _standardized_ in https://github.com/lightningnetwork/lightning-rfc/blob/master/10-dns-bootstrap.md[BOLT #10 - DNS Bootstrap]
Similar to the Bitcoin protocol, the primary initial peer discovery mechanism used in the Lightning Network happens via DNS. As initial peer discovery is a critical and universal task for the network, the process has been _standardized_ in https://github.com/lightningnetwork/lightning-rfc/blob/master/10-dns-bootstrap.md[BOLT #10 - DNS Bootstrap].
==== DNS Bootstrapping
The BOLT 10 document describes a standardized way of implementing peer
The https://github.com/lightningnetwork/lightning-rfc/blob/master/10-dns-bootstrap.md[BOLT #10] document describes a standardized way of implementing peer
discovery using the Domain Name System (DNS). Lightning's flavor of DNS based bootstrapping uses up to 3 distinct record types:
* +SRV+ records for discovering a set of _node public keys_.
* +A+ records for mapping a node's public key to its current +IPv4+ address.
* +AAA+ records for mapping a node's public key to its current +IPv6+ address (if one exists).
* +AAA+ records for mapping a node's public key to its current +IPv6+ address.
Those somewhat familiar with the DNS protocol may already be familiar with the +A+ and +AAA+ record types, but not the +SRV+ type. The +SRV+ record type is used by protocols built on top of DNS, to determine the
_location_ for a specified service. In our context, the service in question is a given Lightning node, and the location its IP address. We need to use this additional record type as unlike nodes within the Bitcoin protocol, we need both a public key _and_ an IP address in order to connect to a node. As we see in <<wire_protocol>> the transport encryption protocol used in LN requires knowledge of the public key of a node before connecting, so as to implement identity hiding for nodes in the network.
Those somewhat familiar with the DNS protocol may already be familiar with the +A+ (name to IPv4 address) and +AAA+ (name to IPv6 address) record types, but not the +SRV+ type. The +SRV+ record type is used by protocols built on top of DNS, to determine the _location_ for a specified service. In our context, the service in question is a given Lightning node, and the location its IP address. We need to use this additional record type as unlike nodes within the Bitcoin protocol, we need both a public key _and_ an IP address in order to connect to a node. As we see in <<wire_protocol>> the transport encryption protocol used in LN requires knowledge of the public key of a node before connecting, so as to implement identity hiding for nodes in the network.
===== A new peer's bootstrapping workflow
Before diving into the specifics of BOLT 10, we'll first outline the high level flow of a new node that wishes to use BOLT 10 to join the network.
Before diving into the specifics of https://github.com/lightningnetwork/lightning-rfc/blob/master/10-dns-bootstrap.md[BOLT #10], we'll first outline the high level flow of a new node that wishes to use BOLT #10 to join the network.
First, a node needs to identify a single, or set of DNS servers that understand BOLT 10 so they can be used for p2p bootstrapping.
First, a node needs to identify a single, or set of DNS servers that understand BOLT #10 so they can be used for p2p bootstrapping.
While BOLT 10 uses lseed.bitcoinstats.com as the seed server, there exists no "official" set of DNS seeds for this purpose, but each of the major implementations maintain their own DNS seed, and cross query each other's seeds for redundancy purposes. In <<dns_seeds>> you'll see a list non-exhaustive list of some popular DNS seed servers.
While BOLT #10 uses lseed.bitcoinstats.com as the seed server, there exists no "official" set of DNS seeds for this purpose, but each of the major implementations maintain their own DNS seed, and cross query each other's seeds for redundancy purposes. In <<dns_seeds>> you'll see a list non-exhaustive list of some popular DNS seed servers.
[[dns_seeds]]
.Table of known lightning dns seed servers
@ -124,9 +123,9 @@ While BOLT 10 uses lseed.bitcoinstats.com as the seed server, there exists no "o
DNS seeds exist for both Bitcoin's mainnet and testnet. For the sake
of our example, we'll assume the existence of a valid BOLT 10 DNS seed at +nodes.lightning.directory+.
of our example, we'll assume the existence of a valid BOLT #10 DNS seed at +nodes.lightning.directory+.
Next, our new node will issue an +SRV+ query to obtain a set of _candidate bootstrap peers_. The response to our query will be a series of _bech32_ encoded public keys. As DNS is a text based protocol, we can't send raw binary data, so an encoding scheme is required. BOLT 10 specifies a bech32 encoding due to its use in the wider Bitcoin ecosystem. The number of encoded public keys returned depends on the server returning the query, as well as all the resolvers that stand between the client and the authoritative server.
Next, our new node will issue an +SRV+ query to obtain a set of _candidate bootstrap peers_. The response to our query will be a series of _bech32_ encoded public keys. As DNS is a text based protocol, we can't send raw binary data, so an encoding scheme is required. BOLT #10 specifies a bech32 encoding due to its use in the wider Bitcoin ecosystem. The number of encoded public keys returned depends on the server returning the query, as well as all the resolvers that stand between the client and the authoritative server.
Using the widely available +dig+ command-line tool, we can query the _testnet_ version of the DNS seed mentioned above with the following command:
We use the +@+ argument to force resolution via Google's nameserver (with IP address 8.8.8.8) as they do not filter large SRV query responses. At the end of the command, we specify that we only want +SRV+ records to be returned. A sample response looks something like:
test.nodes.lightning.directory. 59 IN SRV 10 10 15735 ln1qtgsl3efj8verd4z27k44xu0a59kncvsarxatahm334exgnuvwhnz8dkhx8.test.nodes.lightning.directory.
<SNIP>
[...]
;; Query time: 89 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Dec 31 16:41:07 PST 2020
----
<1> TCP port number where LN node can be reached.
<2> Node public key (ID) encoded as a virtual domain name.
====
We've truncated the response for brevity and show only two of the returned responses. Starting from the right-most column, we have a candidate "virtual" domain name for a target node, then to the left we have the _TCP port_ that this node can be reached using. The first response uses the standard TCP port for LN: +9735+. The second response uses a custom port which is permitted by the protocol.
We've truncated the response for brevity and show only two of the returned responses. The responses contain a "virtual" domain name for a target node, then to the left we have the _TCP port_ where this node can be reached. The first response uses the standard TCP port for LN: +9735+. The second response uses a custom port which is permitted by the protocol.
Next, we'll attempt to obtain the other piece of information we need to connect to a node: its IP address. Before we can query for this however, we'll first _decode_ the returned sub-domain which is the bech32 encoding of the public key:
Next, we'll attempt to obtain the other piece of information we need to connect to a node: its IP address. Before we can query for this however, we'll first _decode_ the bech32 encoding of the public key from the virtual domain name:
@ -178,6 +180,8 @@ Decoding this bech32 string we obtain the following valid
----
Now that we have the raw public key, we'll now ask the DNS server to _resolve_ the virtual host given so we can obtain the IP information (+A+ record) for the node:
====
----
$ dig ln1qfkxfad87fxx7lcwr4hvsalj8vhkwta539nuy4zlyf7hqcmrjh40xx5frs7.test.nodes.lightning.directory A
<1> The DNS server returns an IP Address X.X.X.X. We've replaced it with X's in the text here so as to avoid presenting a real IP address.
====
In the above command, we've queried the server so we can obtain an +IPv4+ (+A+ record) address for our target node (replaced by X.X.X.X in the example above). Now that we have both the raw public key, IP address, and TCP port, we can connect to the node transport protocol at:
Querying for the current +A+ record for a given node can also be used to look up the _latest_ set of addresses for a given node. Such queries can be used to more quickly (compared to waiting on gossip as we'll cover later) sync the latest addressing information for a node.
Querying the current DNS +A+ record for a given node can also be used to look up the _latest_ set of addresses. Such queries can be used to more quickly sync the latest addressing information for a node, compared to waiting for address updates on the gossip network (see <<node_announcement>>).
At this point in our journey, our new Lightning node has found its first
peer and established its first connection! Now we can begin the second phase of new peer bootstrapping: channel graph synchronization and validation.
@ -213,16 +219,15 @@ First, we'll explore more of the intricacies of BOLT 10 itself to take a deeper
==== SRV Query Options
The BOLT 10 standard is highly extensible due to its usage of nested
The https://github.com/lightningnetwork/lightning-rfc/blob/master/10-dns-bootstrap.md[BOLT #10] standard is highly extensible due to its usage of nested
sub-domains as a communication layer for additional query options. The
bootstrapping protocol allows clients to further specify the _type_ of nodes they're attempting to query for vs the default of receiving a random subset of nodes in the query responses.
The query option sub-domain scheme uses a series of key-value pairs where the key itself is a _single letter_ and the remaining set of text is the value itself. The following query types exist in the current version of the BOLT 10 standards document:
The query option sub-domain scheme uses a series of key-value pairs where the key itself is a _single letter_ and the remaining set of text is the value itself. The following query types exist in the current version of the https://github.com/lightningnetwork/lightning-rfc/blob/master/10-dns-bootstrap.md[BOLT #10] standards document:
* +r+: The "realm" byte which is used to determine which chain or realm queries should be returned for. As is, the only value for this key is +0+ which denotes "Bitcoin".
* +a+: Allows clients to filter out returned nodes based on the _types_ of addresses they advertise. As an example, this can be used to only obtain nodes that advertise a valid IPv6 address.
* The value that follows this type is based on a bitfled that _indexes_ into the set of specified address _type_ which are defined in BOLT 7. The default value for this field is +6+, which represents both IPv4 and IPv6 (bits 1 and 2 are set)
* +a+: Allows clients to filter out returned nodes based on the _types_ of addresses they advertise. As an example, this can be used to only obtain nodes that advertise a valid IPv6 address. The value that follows this type is based on a bitfled that _indexes_ into the set of specified address _type_ which are defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/07-routing-gossip.md[BOLT #7]. The default value for this field is +6+, which which represents both IPv4 and IPv6 (bits 1 and 2 are set)
* +l+: A valid node public key serialized in compressed format. This allows a client to query for a specified node rather than receiving a set of random nodes.
@ -253,16 +258,20 @@ Now that our new node is able to use the DNS bootstrapping protocol to connect t
==== A directed graph
A graph in computer science is a special data structure composed of vertices (typically referred to as nodes) and edges (also known as links). Two nodes may be connected by one or more edges. The channel graph is also _directed_ given that a payment is able to flow in either direction over a given edge (a channel). As we're concerned with _routing payments_, in our model a node with no edges (no payment channels) isn't considered to be a part of the graph as it isn't useful.
A graph in computer science is a special data structure composed of vertices (typically referred to as nodes) and edges (also known as links). Two nodes may be connected by one or more edges. The channel graph is also _directed_ given that a payment is able to flow in either direction over a given edge (a channel). An example of a _directed graph_ is shown below:
In the context of the Lightning Network, our vertices are the Lightning nodes themselves, with our edges being the payment channels connecting these nodes.
In the context of the Lightning Network, our vertices are the Lightning nodes themselves, with our edges being the payment channels connecting these nodes. As we're concerned with _routing payments_, in our model a node with no edges (no payment channels) isn't considered to be a part of the graph as it isn't useful.
As channels themselves are UTXOs (funded 2-of-2 multisig addresses), we can view the channel graph as a special subset of the Bitcoin UTXO set, on top of which we can add some additional information (the nodes, etc) to arrive at the final overlay structure which is the channel graph. This anchoring of fundamental components of the channel graph in the
base Bitcoin blockchain means that it's impossible to _fake_ a valid channel graph, which has useful properties when it comes to spam prevention as we'll see later.
=== Gossip protocol messages
The channel graph information is propagated across the Lightning P2P Network as three messages, which are described in BOLT 7:
The channel graph information is propagated across the Lightning P2P Network as three messages, which are described in https://github.com/lightningnetwork/lightning-rfc/blob/master/07-routing-gossip.md[BOLT #7]:
* +node_announcement+: The vertex in our graph which communicates the public key of a node, as well as how to reach the node over the internet and some additional metadata describing the set of _features_ the node supports.
@ -275,6 +284,7 @@ themselves _authenticated_ allowing a 3rd party to ensure that the owner of a ch
With the high level structure of the channel graph laid out, we'll now dive down into the precise structure of each of the three messages used to gossip the channel graph. We'll also explain how one can also _verify_ each message and component of the channel graph.
[[node_announcement]]
==== The node_announcement message
First, we have the +node_announcement+ message, which serves two primary
@ -340,12 +350,13 @@ Due to the existence of unadvertised channels, the _true_ size of the channel gr
As mentioned earlier, the channel graph is authenticated due to its usage of public key cryptography, as well as the Bitcoin blockchain as a spam prevention system. In order to have a node accept a new +channel_announcement+, the advertisement must _prove_ that the channel actually exists in the Bitcoin blockchain. This proof system adds an upfront cost to adding a new entry to the channel graph (the on-chain fees one must pay to create the UTXO of the channel). As a result, we mitigate spam and ensure that a dishonest node on the network can't fill up the memory of an honest node at no cost with bogus channels.
Given that we need to construct a proof of the existence of a channel, a
natural question that arises is: how do we "point to" or reference a given channel for the verifier? Given that a payment channel is anchored in a UTXO, an initial thought might be to first attempt to just advertise the full outpoint (+txid:index+) of the channel. Given the outpoint is globally unique and confirmed in the chain, this sounds like a good idea, however it has a drawback: the verifier must maintain a full copy of the UTXO set in order to verify channels. This works fine for Bitcoin full-nodes, but clients which rely on lightweight verification don't typically maintain a full UTXO set. As we want
to ensure we can support mobile nodes in the Lightning Network, we're forced to find another solution.
natural question that arises is: how do we "point to" or reference a given channel for the verifier? Given that a payment channel is anchored in a Unspent Transaction Output (see <<utxo>>), an initial thought might be to first attempt to advertise the full outpoint (+txid:index+) of the channel. Given the outpoint is globally unique and confirmed in the chain, this sounds like a good idea, however it has a drawback: the verifier must maintain a full copy of the UTXO set in order to verify channels. This works fine for Bitcoin full-nodes, but clients which rely on lightweight verification don't typically maintain a full UTXO set. As we want to ensure we can support mobile nodes in the Lightning Network, we're forced to find another solution.
What if rather than referencing a channel by its UTXO, we reference it based on its "location" in the chain? In order to do this, we'll need a scheme that allows us to "index" into a given block, then a transaction within that block, and finally a specific output created by that transaction. Such an identifier is described in BOLT 7 and is referred to as a _short channel ID_, or +scid+.
What if rather than referencing a channel by its UTXO, we reference it based on its "location" in the chain? In order to do this, we'll need a scheme that allows us to reference a given block, then a transaction within that block, and finally a specific output created by that transaction. Such an identifier is described in https://github.com/lightningnetwork/lightning-rfc/blob/master/07-routing-gossip.md[BOLT #7] and is referred to as a _short channel ID_, or +scid+.
The +scid+ is used both in +channel_announcement+ (and +channel_update+) as well as within the onion encrypted routing packet included within HTLCs as we learned <<onion_routing>>.
[[short_channel_id]]
[[scid]]
===== The Short Channel ID
Based on the information above, we have three pieces of information we need to encode in order to uniquely reference a given channel. As we want a compact representation, we'll attempt to encode the information into a _single_ integer. Our integer format of choice is an unsigned 64-bit integer, comprised of 8 bytes.
Payment delivery on the Lightning Network depends on finding a path from the sender to the recipient, a process called _path finding_.
Payment delivery on the Lightning Network depends on finding a path from the sender to the recipient, a process called _path finding_. Since the routing is done by the sender, the sender must find a suitable path to reach the destination. This path is then encoded in an onion, as we saw in <<onion_routing>>.
In this chapter we will examine the problem of path finding, understand how uncertainty about channel balances complicates this problem and look at how a typical path finding implementation attempts to solve it.
=== Path finding in the Lightning protocol suite
@ -23,9 +25,10 @@ That's because path finding isn't an activity that requires any form of coordina
The term path finding may be somewhat misleading, because it implies a search for _a single path_ connecting two nodes. In the beginning, when the Lightning Network was small and not well interconnected, the problem was indeed about finding a way to join payment channels to reach the recipient.
But, as the Lightning Network has grown explosively, the path finding problem's nature has shifted. In mid-2021, as we finish this book, the Lightning Network consists of 20,000 nodes connected by at least 55,000 public channels with an aggregate capacity of almost 2,000 BTC. A node has on average 8.8 channels, while the top 10 most connected nodes have between 400 and 2000 channels _each_. A visualization of just a small subset of the Lightning Network channel graph (as of July 2021) is shown in <<lngraph>>:
But, as the Lightning Network has grown explosively, the path finding problem's nature has shifted. In mid-2021, as we finish this book, the Lightning Network consists of 20,000 nodes connected by at least 55,000 public channels with an aggregate capacity of almost 2,000 BTC. A node has on average 8.8 channels, while the top 10 most connected nodes have between 400 and 2000 channels _each_. A visualization of just a small subset of the Lightning Network channel graph is shown in <<lngraph>>:
[[lngraph]]
.A visualization of part of the Lightning Network as of July 2021
image::images/LNGraphJuly2021.png[]
[NOTE]
@ -98,7 +101,7 @@ Our channel liquidity uncertainty range is the range between the minimum and max
Finding a path through a graph is a problem modern computers can solve rather efficiently.
Developers mainly choose breadth-first search if the edges are all of equal weight.
In cases where the edges are not of equal weight, an algorithm based on Dijkstra Algorithm is used, such as A* ("a-star") or
In cases where the edges are not of equal weight, an algorithm based on Dijkstra Algorithm is used, such as https://en.wikipedia.org/wiki/A*_search_algorithm[A* ("A-star")].
In our case the weights of the edges can represent the routing fees.
Only edges with a capacity larger than the amount to be sent will be included in the search.
In this basic form, path finding in the Lightning network is very simple and straight forward.
@ -128,9 +131,7 @@ Most Lightning node and wallet implementations improve on this approach, by orde
Path finding and payment delivery involves several steps, which we list below. Different implementations may use different algorithms and strategies, but the basic steps are likely to be very similar:
* Create a _channel graph_ from announcements and updates, containing the capacity of each channel.
* Filter the graph ignoring any channels with insufficient capacity for the amount we want to send.
* Create a _channel graph_ from announcements and updates, containing the capacity of each channel and filter the graph ignoring any channels with insufficient capacity for the amount we want to send.
* Find paths connecting the sender to the recipient.
@ -150,7 +151,7 @@ These three activities can be repeated in a _payment round_ if we use the failur
In the next sections we will look at each of these steps in more detail, as well as more advanced payment strategies.
==== Channel graph construction
=== Channel graph construction
In <<gossip>> we covered the three main messages that nodes "gossip": +node_announcement+, +channel_announcement+, and +channel_update+. These three messages allow any node to gradually construct a "map" of the Lightning Network in the form of a _channel graph_. Each of these messages provides a critical piece of information for the channel graph:
@ -160,11 +161,11 @@ channel_announcement:: This contains the capacity and channel ID of a public (an
channel_update:: This contains a node's fee and timelock (CLTV) expectations for routing an outgoing (from that node's perspective) payment over a specified channel.
In terms of a mathematical graph, the node_announcement is the information needed to create the nodes or _vertices_ of the graph. The channel_announcement allows us to create the _edges_ of the graph representing the payment channels. Since each direction of the payment channel has its own balance, we create a directed graph. The channel_update allows us to incorporate fees and timelocks to set the _cost_ or _weight_ of the graph edges.
In terms of a mathematical graph, the +node_announcement+ is the information needed to create the nodes or _vertices_ of the graph. The +channel_announcement+ allows us to create the _edges_ of the graph representing the payment channels. Since each direction of the payment channel has its own balance, we create a directed graph. The +channel_update+ allows us to incorporate fees and timelocks to set the _cost_ or _weight_ of the graph edges.
Depending on the algorithm we will use for path finding, we may establish a number of different cost functions for the edges of the graph.
For now, let's ignore the cost function and simply establish a channel graph showing nodes and channels, using the node_announcement and channel_announcement messages.
For now, let's ignore the cost function and simply establish a channel graph showing nodes and channels, using the +node_announcement+ and +channel_announcement+ messages.
In this chapter we will see how Selena attempts to find a path to pay Rashid 1,000,000 (1m) satoshis. To start, Selena is constructing a channel graph using the information from the Lightning Network gossip to discover nodes and channels. Selena will then explore her channel graph to find a path to send a payment to Rashid.
@ -279,7 +280,7 @@ The fee and timelock information are very important not just as path selection m
It is also important to note that a channel must have liquidity that is sufficient not only for the payment amount but also for the cumulative fees of all the subsequent hops. Even though Selena's channel to Xavier (S-->X) has enough liquidity for a 1m satoshi payment, it *does not* have enough liquidity once we consider fees. We need to know fees because only paths that have sufficient liquidity for *both payment and all fees* will be considered.
==== Finding candidate paths
=== Finding candidate paths
Finding a suitable path through a directed graph like this is a well-studied computer science problem (known broadly as the "Shortest Path problem"), which can be solved by a variety of algorithms depending on the desired optimization and resource constraints.
@ -302,7 +303,7 @@ But, as we saw previously, the channel +S->X+ does not have enough liquidity for
With two possible paths, Selena is ready to attempt delivery!
==== Payment delivery (Trial-and-error loop)
=== Payment delivery (Trial-and-error loop)
Selena's node starts the trial-and-error loop, by constructing the HTLCs, building the onion and attempting delivery of the payment. For each attempt, there are three possible outcomes:
@ -314,7 +315,7 @@ If the payment fails, then it can be re-tried via a different path by updating t
We'll look at what happens if the payment is "stuck" in <<stuck_payments>>. The important detail is that a stuck payment is the worst outcome because we cannot retry with another HTLC as both (the stuck one and the retry one) might go through eventually and cause a double payment.
===== First attempt (path #1)
==== First attempt (path #1)
Selena attempts the first path (S->A->B->R). She constructs the onion and sends it, but receives a failure code from Bob's node. Bob reports back a +temporary channel failure+ with a +channel_update+ identifying the channel B->R as the one that can't deliver. This attempt is shown in <<path_1_fail>>:
From this failure code, Selena will deduce that Bob doesn't have enough liquidity to deliver the payment to Rashid on that channel. Importantly, this failure narrows the uncertainty of the liquidity of that channel! Previously, Selena's node assumed that the liquidity on Bob's side of the channel was somewhere in the range (0, 4m). Now, she can assume that the liquidity is in the range (0, 999999). Similarly, Selena can now assume that the liquidity of that channel on Rashid's side is in the range (1m, 4m), instead of (0, 4m). Selena has learned a lot from this failure.
===== Second attempt (path #4)
==== Second attempt (path #4)
Now Selena attempts the fourth candidate path (S->A->B->X->Y->R). This is a longer path and will incur more fees, but it's now the best option for delivery of the payment.
@ -137,6 +137,7 @@ More information about signature hash types can be found in https://github.com/b
Transactions are data structures that encode the transfer of value between participants in the bitcoin system.
[[utxo]]
==== Inputs and outputs
The fundamental building block of a bitcoin transaction is a transaction output. Transaction outputs are indivisible chunks of bitcoin currency, recorded on the blockchain, and recognized as valid by the entire network. A transaction spends "inputs" and creates "outputs". Transaction inputs are simply references to outputs of previously recorded transactions. This way, each transaction spends the outputs of previous transactions and creates new outputs.
Andreas M. Antonopoulos
3 years ago
