It's important to note that we separate the concept of _routing_ from the concept of _path finding_. These two concepts are often confused and the term "routing" is often used to describe both concepts. Let's remove the ambiguity, before we proceed any further.
Path Finding, which is covered in <<path_finding>> is the process of finding and choosing a contiguous path made of payment channels which connects the sender A to the recipient B. The sender of a payment does the path finding, by examining the _channel graph_ which they have assembled from channel announcements gossiped by other nodes.
Routing refers to the series of interactions across the network that allow a payment to _flow_ from A to B, across the path previously selected by path finding. Routing is the _active_ process of sending a payment on a path, which involves the cooperation of all the intermediary nodes along that path.
An important rule of thumb is that it's possible for a _path_ to exist between Alice and Bob, yet there may not be an active _route_ on which to send the payment.
One example is the scenario where all the nodes connecting Alice and Bob are currently off-line.
In theory, one can examine the _channel graph_ and connect a series of payment channels from Alice to Bob, hence a _path_ exists. However, as the intermediary nodes are offline, the payment cannot be sent and so no _route_ exists.
In this section we will examine routing from the perspective of Dina, a gamer who receives donations from her fans while she livestreams her game sessions.
The innovation of routed payment channels allows Dina to receive tips without maintaining a separate channel with every one of her fans who want to tip her.
Another important detail is that due to the use of onion routing, intermediary nodes are only explicitly aware of the one node preceding them and the one node following them in the route.
They will not necessarily know who is the sender and recipient of the payment.
This process of connecting a series of payment channels with end-to-end security, and the incentive structure for nodes to _forward_ payments, is one of the key innovations of the Lightning Network.
In this chapter, we'll dive into the mechanism of routing in the Lightning Network, detailing the precise manner in-which payments flow through the network.
First, we will cover the concept of a conditional chained end-to-end secure payment, most commonly referred to as a Hash Time Locked Transaction (HTLC).
Having learned how payments can be transmitted through the network, we will then discuss the concept of source-based routing and contrast it to the privacy preserving onion routing used in the network today.
Finally, we will explore the exact mechanism of payment forwarding.
We will discuss how the _structure_ (edges, fees, time-locks, etc) of the route is determined by the sender, and is then transmitted to each individual node along the route.
Bob, in turn, has an open channel with the software developer Chan who helps him with the point of sale system he uses in his coffee shop.
Chan is also the owner of a large software company which develops the game that Dina plays, and they already have an open channel which Dina uses to pay for the game's license and in-game items.
If we draw out this series of payment channels, it's possible to manually trace a _path_ from Alice to Dina that uses Bob and Chan as intermediary routing nodes.
Alice can then craft a _route_ from this outlined path, and use it to send a tip of a few thousand satoshis to Dina, with the payment being _forwarded_ by Bob and Chan.
Essentially, Alice will pay Bob, who will pay Chan, who will pay Dina.
And no direct channel from Alice to Dina is required.
To understand how the Lightning Network protects the payment while being routed, we can compare to an example of routing physical payments with golden coins in the real world.
While this contract is nice in the abstract, in the real world, Alice runs the risk that Bob might breach the contract and hope to not get caught by law enforcement.
Even if Bob gets caught by law enforcement, Alice faces the risk that he might be bankrupt and be unable to return her 10 gold coins.
Assuming these issues are magically solved, it's still unclear how to leverage such a contract to achieve our desired outcome: the coins ultimately being delivered to Dina.
_I (Alice) will reimburse you (Bob) with 10 gold coins if you can prove to me (for example via a receipt) that you already have delivered 10 gold coins to Chan_
He has to pay Chan but ultimately gets nothing out of the exchange, and he runs the risk that Alice might not reimburse him.
Bob could offer Chan a similar contract to pay Dina, but similarly Chan has no reason to accept it either.
Even putting aside the risk, Bob and Chan must _already_ have 10 gold coins to send, otherwise they wouldn't be able to participate in the contract.
Thus Bob and Chan face both risk and opportunity cost for agreeing to this contract, and they would need to be compensated in order for them to accept it.
_I (Alice) will reimburse you (Bob) with 12 gold coins if you can prove to me (for example via a receipt) that you already have delivered 11 golden coins to Chan_
At the start of the exchange, Alice could "lock up" these 12 golden coins in escrow that will only be paid to Bob once he proves that he's paid 11 golden coins to Chan.
In practice, this secret would be a large random number that is large enough to prevent others from guessing it (typically _very, very_ large number, encoded using 256 bits!).
In order to "commit" to this secret, she computes the SHA256 hash which when encoded in hex, can be displayed as: `+f23c83babfb0e5f001c5030cf2a06626f8a940af939c1c35bd4526e90f9759f5+`.
To facilitate Alice's payment, Dina will create the secret and the payment hash and send the payment hash to Alice. Alice doesn't know the secret but she can rewrite her contract to use the hash of the secret as a proof of payment:
This new contract now protects Alice from Bob not forwarding to Chan, protects Bob from not being reimbursed by Alice, and ensures that there will be proof that Dina was ultimately paid via the hash of Dina's secret.
After Bob and Alice agree to the contract, and Bob receives the message from the escrow that Alice has deposited the 12 gold coins, Bob can now negotiate a similar contract with Chan.
He checks that +"Dinas secret" hashes to +f23c83...+. Chan now has proof of payment and so instructs the escrow service to release the 10 golden coins to Dina.
We could amend the contract so that if it is not fulfilled by a certain deadline, then the contract expires and the escrow service returns the money to the person who made the original deposit.
Even if he successfully pays Chan, if he receives the proof of payment later than 24 hours he will not be reimbursed. To remove that risk, Bob must give Chan and even shorter deadline.
With such a chain of contracts we can ensure that, after 24 hours, the payment will successfully deliver from Alice to Bob to Chan to Dina, or it will fail and everyone will be refunded.
While this seems like a minor detail we will see in later this chapter that this requirement is actually one of the more difficult issues for Lightning Network nodes.
It becomes progressively more difficult as the size of the payment increases.
Thus users forwarding payments face an opportunity cost for locking the money, which is ultimately reimbursed through routing fees, as we saw in the example above.
In the following two sections we will discuss how the Bitcoin scripting language can be used to set up conditional chained end-to-end secure payment contracts _without_ third party escrows, similar to the gold coin contracts described above.
These are called Hash Time Locked Contracts (HTLCs).
For HTLCs, there are no trusted third parties who act as an escrow; the Bitcoin Network itself becomes the "escrow" service.
Then, we will discuss how users are able to use an HTLC to securely "route" a payment through the Lightning Network.
Currently (in 2020), the Lightning Network uses a routing protocol called source-based onion routing, although it is possible to route payments with other routing protocols.
Finally we will discuss the precise details of forwarding, settling, and canceling HTLCs in the network.
Our example in the prior section using "golden coins", was intended to lay same base intuition which we'll leverage in this section to explain how HTLCs work in practice.
A payment can be said to be conditional, if the completion of the payment relies on the completion of a certain event.
In the golden coins example, this "condition" was the reveal of a hash pre-image.
We could feasibly substitute this hash pre-image reveal for any other construct with "hardness" properties. Namely: it should be infeasible for a party that doesn't know the proper "solution" of the condition to satisfy it, the "description" of the condition shouldn't give away any information about the true "solution", and once a solution has been chosen and a description created from it, it shouldn't be possible to "alter" that solution and have it still be a valid condition for the description.
The payment should _only_ be able to be redeemed if a valid solution is revealed. Critical, all conditions need to be timed in order to allow the construct to return the funds back to the sender if a solution to this condition isn't revealed.
The combination of the condition, and a timeout on the condition gives the payment a trait we commonly refer to as atomicity: either the payment happens, or the receiver if refunded the funds.
Building upon our conditional payment, it may be possible to *chain* this payment, allowing it to involve the payer, the payee, and possibly several intermediaries.
Each intermediary, is able to present a _slightly_ modified version of the condition (without invalidating it all together), and so on in an iterated manner until the conditional payment reaches the payee.
Each chaining creates an "incoming" and "outgoing" conditional payment.
A node receives a conditional payment from a party (incoming condition), and then extends the conditional payment to the next party in the chain (outgoing condition).
The payment is extended in from payer to payee, but settled from payee to payer, as each of the intermediaries gain the solution to the outgoing condition, and use that (possibly augmenting it) to satisfy the incoming solution.
Typically the payer rewards the intermediaries by sending slightly more than the payment amount, in order to allow the intermediaries to send out less with their outgoing payment than what they received from the incoming payment.
The difference between these two payment values makes up the "forwarding fee" collected by the intermediary.
With our final addition, we'll achieve "end to end security".
By this we mean that: no intermediaries are able to "claim" the payment without first obtaining the solution from someone further down from them in the chain.
Additionally, we also require that the amount the payer intended to send is fully received by the payee.
Finally, we require that non of the intermediaries are able to "contaminate" the payment beyond giving incorrect directions to the party that directly follows them.
In other words, the intermediary shouldn't be able to materially affect the propagation of the payment several hops away from it.
At each step we'll add a new component, then examine it in light of our original definition to ensure all requirements and security properties are reached.
First, the "condition". For an HTLC, the condition is typically the reveal of a hash pre-image that matches a particular hash.
This hash is typically referred to as the "payment hash", with the pre-image being called the "payment pre-image".
If the name didn't give too much away, for an HTLC, we'll use a _cryptographically secure_ hash function as one part of our condition.
By using a cryptographic hash function, we ensure that it's infeasible for another party to "guess" the solution of our condition, it's easy for anyone to verify the solution, and there's only one "solution" to the condition.
In order to implement the "refund" functionality, we rely on the "absolute time lock" functionality of Bitcoin script.
With all that said, a basic Bitcoin script implementing a hash time-locked contract would look something like the following:
Each node in the hop needs some time to be able to settle the outgoing, then incoming payment on-chain in the worst case.
As a result, when constructing the final route, we need to give each node some buffer time, we call this before time, the "time lock delta".
Factoring in this time-lock delta, the time-lock of the outgoing HTLC will decrease as the route progresses, as the outgoing HTLC will expire before the incoming HTLC.
This set of decrementing time-locks is critical to the operation of the system, as it ensure out atomicity property for each hop, assuming they're able to get into the chain in time.
In the next section, we'll go into the exact mechanism of how Alice is able to deliver forwarding details to each hop in the route.
In addition, we'll dive further into proper time-lock construction, as incorrect time-lock set up can violate our atomicity property and lead to a loss of funds.
=== HTLC Packet Forwarding: Source Based Onion Routing
So far you have learnt that payment channels can be connected to a network which can be utilized to send payment from one participant to another one through a path of payment channels.
You have seen that with the use of HTLCs the intermediary nodes along the path are not able to steal any funds that they are supposed to forward and also how a node can set up and settle an HTLC.
With this bare foundation laid, the following questions may have come across you mind:
- Who chooses the path for a candidate route?
- How is a path selected as a candidate to attempt to route the HTLC for a payment?
- How much information do nodes know about the total path?
- How exactly does a payment flow through the network at each node?
In the network today, the sender is the one that selects the route and decides nearly all the details of the resulting route.
Instead, the answer to the second question has a very large solution space, meaning there are several algorithms and heuristics used in the network today.
Most commonly, a variation of Dijkstra's algorithm is used which takes into account additional Lightning Network details such as fees and time-locks.
Remember from earlier that a path turns into a route which is used to trigger a payment attempt.
As several conditions need to be satisfied for the HTLC to be completely extended, the sender may need to try several routes until one succeeds.
However, the user of the wallet typically will not be aware of these failed path finding attempts, just as when we load a web-page on the Internet, we don't learn of any TCP packet retransmissions.
In the early days of the network, a payment could only utilize a single channel in its final route.
With the rise of Multi-Path Payments, the sender is able to split the amount into smaller pieces, and use distinct strategies to route all the payment chunks.
This splitting behavior is similar to IP packet fragmentation on the IP layer: each node expresses its Maximum Payment Unit, with the sender using this as a guide to adequately split all payments.
In later chapters, we'll discuss further details of payment splitting and combination once we get to advanced path finding.
At a high level, each node in the route is only _explicitly_ told: how to validate the incoming HTLC packet (remember all details need to be correct for a payment to flow!), who the next hop in the route is, and how to modify the incoming HTLC packet into a valid outgoing HTLC packet to forward to the next node.
Combined with the fact that intermediate forwarding nodes aren't explicitly given the sender and receiver of a payment, nodes are given the _least_ amount of information they need to successfully forward a payment.
In addition to these privacy enhancing attributes, intermediate nodes aren't able to arbitrarily modify an HTLC packet, as all information is encrypted and cryptically authenticated with integrity checks carried out at each hop to ensure contents haven't been modified.
Readers familiar with onion routing may have realized that we'll be using some clever cryptographic technique application to achieve all thees traits.
We call this series of clever application of cryptographic techniques: sourced based onion routing!
Source based routing (the non-cryptographic portion of onion routing), is distinct from how packets are typically transmitted on the IP layer.
On the Internet today, packet switching is widely used to transmit data across the Internet.
Packet switching typically explicitly indicates the sender and receiver of a given packet.
Intermediate routing nodes then attempt to deliver the packet on a best effort basis, with great freedom with to exactly _how_ they select the next node in the route.
However, the lack of encryption, end-to-end integrity checks, and arbitrary choice of routes may this a poor system to use in a _payment network_.
Source routing instead has the sender select the route entirely (which all we'll learn later is important due to fees and timelocks).
The onion routing layers then gives the sender nearly completely control of the route, and allows the sender to only tell the intermediate nodes what they need to successfully forward a payment.
Onion routing is used in several popular protocols on the Internet, with the most notable of them being Tor.
In the Lightning Network, we use a specific onion routing _packet_ format called Sphinx, with some special modifications made in order to make it more suited to the unique constraints of the Lightning Network.
While the Lightning Network also uses an onion routing scheme it is actually very different to the onion routing scheme that is used in the TOR network.
Aside from the distinct cryptographic techniques they use, the biggest difference is that TOR is being used for arbitrary data to be exchanged between two participants where on the Lightning Network the main use case is to pay people and transfer data that encodes monetary value.
In the Lightning Network, we're only concerned with transmitting the details that are needed for a successful payment.
On the Lightning Network there is no analogy to the exit nodes of the Tor Network as there's no need to "exit" the network: all payments flow within the network.
Although, in an idea model only a precise amount of information is leaked by a route, in practice several "side channels' exist, that may allow an adversary to deduce more information about a route.
As an example, information about CTLV deltas, or the set of possible routes in the network may give away additional information about a given route.
Similar to Tor, onion routing in the Lightning Network isn't secure against a global passive adversary (one that can monitor all links and information flows in the network).
Today in the network, every node in the route sees the same payment hash, meaning that if two nodes are "compromised" more details of the route are leaked.
On the TOR network nodes can theoretically be connected via a full graph as every node could create an encrypted connection with every other node on top of the Internet Protocol almost instantaneously and at no cost.
On the Lightning Network payments can only flow along existing payment channels.
On the Lightning Network nodes might not be able to forward a payment package because they do not own enough funds on their side of the payment channel.
In order to kick off the transfer, Alice needs to send a special message to Bob to kick off the multi-hop transfer.
You'll learn about the specific structure of this message in later chapters, but for now we'll call it an "HTLC Add" message.
Aside from the amount, the payment hash, and the time-lock, this message also contains an opaque field use to store encrypted forwarding information.
Today in the network, this field is 1366 bytes, as that's the _fixed_ size length of the onion packet. #TODO(roasbeef): explain security properties earlier
However Bob who receives the onion cannot read all the information about the path as most of the onion is hidden from him through a sequence of encryptions.
For example after Bob received the onion from Alice he will be able to decrypt the first layer and he will only see the information that he is supposed to forward the onion to Chan by setting up an HTLC with Chan.
The HTLC with Chan should use the same Payment Hash as the receiving HTLC from Alice.
For example Bob will explicitly be told that Alice offered him an HTLC and sent him an onion and that he is supposed to offer an HTLC to Chan and forward a slightly modified onion.
While the Onion is decrypted layer by layer while it travels along the path from Alice via Bob and Chan to Dina it is created from the inside layer to the outside layers via several rounds of encryption.
Being created from the inside means that the construction starts with the Onion Package that Dina is supposed to receive in plain text.
Let us now look at the construction of the Onion that Alice has to follow and at the exact information that is being put inside each layer of the onion.
2. The header consisting of a public key that can be used by the recipient to produce the shared secret for decrypting the outer layer and to derive the public key that has to be put in the header of the modified onion for the next recipient.
While the TLV format offers more flexibility in both cases the routing information that is encoded into the onion is the same for every but the last hop.
We can see that Alice has created some per hop data for Dina.
The short channel id is set to 0 signaling Dina that this payment is intended to be for her.
Note that this example is slightly simplified, in that Dina can also use attributes of the onion packet format itself to be able to know when she's the final hop.
Finally the CLTV delta which Dina should use to forward the payment is also set to block height 800 (the current height minus Dina's CLTV grace delta) as Dina is the final hop.
As we'll see in later sections, the more modern onion payloads used in Lightning today are much more flexible in that they allow a series of arbitrary key-values pairs.
These arbitrary key-value pairs can be used to extend the protocol in an end-to-end manner, as it many cases, only the sender and receiver need to know how to interpret the data.
On important feature to protect the privacy is to make sure that onions are always of equal length independent of their position along the payment path.
If this wasn't the case, and the onion packet shrank as it was being processed, then this would leak information about the true path length to nodes in the route as the packet would be smaller the further down the route we went.
This ephemeral key is then used to generate a series of additional keys, which are themselves used for encryption, authentication, and also as input to a CSPRNG to deterministically generate the set of random filler bytes.
In the spirit of onion encryption, Alice will begin encrypting the payload from the last hop, adding a new layer of encryption with each new hop.
During processing, each node will authenticate the contents of the payload, then process the packet (decrypting it and shifting around some bytes) to prepare it for processing by the next node in the route.
As we want each node to use a new shared secret to authenticate and encrypt its portion of the packet, the Sphinx onion packet format uses a _re-randomization_ scheme to allow Alice to generate a single ephemeral Diffie-Hellman key for the entire route.
Rather than occupying space in the routing payload for N public keys, with this little trick, we're able to only include a single public key, which is used for ECDH at each step, and randomized in a deterministic manner for the next hop.
You can see that Alice put the encrypted payload inside the full Onion Package which contains the public keys from the secret key that she used to derive the shared secret.
The full onion packet also has a version byte in the beginning (for future extensibility) and an HMAC for the entire Onion.
Finally she uses the CLTV delta added to the current height that was announced for this channel on the gossip protocol and that Chan should use for the HTLC when he forwards the Onion.
As we'll see later, this series of decrementing time-locks must carefully be set in order to avoid time-based race conditions in the created contracts.
Luckily the ephemeral keys that Alice used for the ECDH with Dina can be derived from the ephemeral key that she used for Chan.
Thus after Chan decrypts his layer he can use the shared secret and his ephemeral public key to derive the ephemeral public key that Dina is supposed to use and store it in the header of the Onion that he forwards to Dina.
The exact process to generate the ephemeral keys for every hop will be explained at the very end of the chapter.
Similarly it is important to recognize that Alice removed data from the end of Davids onion payload to create space for the per hop data in Chan's onion.
Thus when Chan has received his onion and removed his routing hints and per hop data the onion would be too short and he somehow needs to be able to append the 65 Bytes of filled junk data in a way that the HMACs will still be valid.
This process of filler generation as well as the process of deriving the ephemeral keys is described in the end of this chapter.
What is important to know is that every hop can derive the Ephemeral Public key that is necessary for the next hop and that the onions save space by always storing only one ephemeral key instead of all the keys for all the hops.
Bob on the other hand will only forward the onion if the difference between the amount to forward and the HTLC that Alice sets up while transferring the Onion to him is large enough to cover for the fees that he would like to earn.
We have not discussed the exact cryptographic algorithms and schemes that are being used to compute the ciphertext from the plain text.
Also we have not discussed how the HMACs are being computed at every step and how everything fits together while the Onions are always being truncated and modified on the outer layer.
Since we use the network itself for transport of these onion packets, Alice is able to construct the entire onion without needing to communicate directly with each node in the route.
She only needs a public key from each participant which is the public `node_id` of the Lightning node and known to Alice.
In the network today, Alice learns about the public key via the gossip network, which is described in Chapter N.
In the second part you have seen how the necessary information for setting up HTLCs along a path of hops are being transferred via onion packets from the source to the sender.
However, in the above scenarios, we only discussed flows where everything goes as expected (the optimistic path).
In this section, we'll now turn out attention into the various scenarios where the payment flow across the route breaks down.
First, it's important to know that once a node sends a fully valid onion packet out to the first hop, they cannot directly influence the course of the route.
When sending out an HTLC and its corresponding onion packet, you as the sender must be prepared to wait the worst-case CTLV timelock period before funds are returned back to the sender (if the route fails).
This explicit awareness of the worst-case delay when sending a payment may be difficult to explain properly from a user experience perspective for end user wallets.
You want to quickly pay a person but the payment path that your node has chosen has CLTV deltas that quickly add up to several 100 blocks which is a couple of days on the Bitcoin network.
This means now that if nodes on the path misbehave - on purpose or maybe just because they have a downtime which your node didn't know about - you will have to wait even though you don't see a preimage.
You must not send out another onion along a different path which uses the same payment hash because there is a risk that both payments will eventually settle.
While our own experience is that most payments find a path and settle in far less than 10 seconds, the Lightning Network protocol cannot and does not give any service level agreement that within this time payments will settle or fail.
There are some ideas that might solve this issue to some degree by allowing the payer to abort a payment. You can find more about this idea under the terms `cancelable payments` or `stuckless payments`. However the proposals that exist only reverse the problem as now the sender can misbehave and the recipient loses control. Another solution is to use many paths in a multipath payment and include some redundancy and ignore the problem that a path takes longer to complete.
This is why the onion protocol has the ability to send back errors in a fail-fast manner that allows nodes to remove the HTLC *off-chain*, without the need to close the channels.
* A node does not have enough liquidity to set up the next HTLC
* The next payment channel does not exist anymore as it might have been closed while the onion was being routed to the node which was supposed to forward the onion along its channel.
* While the channel might still be open - as the funding transaction was never spent - it might happen that the other peer is offline. This of course prevents the node to forward the onion.
* The key exchanges of the sender might have been wrong so the decryption of the onion failed or the HMACs do not match. (also in case someone tried to tamper with the onion)
The process makes sure that the sender of the onion and recipient of the reply can be sure that the error really originated from the node that the error message says it's from.
If however the HTLC between Alice and Bob are set up and the HTLC between Bob and Chan are set up but Chan encounters problems with forwarding the onion it is perfectly Chan has more options than Alice.
Removing an HTLC is the reverse of adding one in the first place from the PoV of the commitment transaction.
In case of errors, peers signal that they wish to remove the HTLC by sending an `update_fail_htlc` or `update_fail_malformed_htlc` message.
These messages contain the id of an HTLC that should be removed in the next version of the commitment transaction.
In the same handshake-like process that was used to exchange `commitment_signed` and `revoke_and_ack` messages, the new state and thus pair of commitment signatures has to be negotiated and agreed upon.
This also means while the balance of a channel that was involved in a failed routing process will not have changed at the end it will have negotiated two new commitment transactions.
Despite having the same balance it must not got back to the previous commitment transaction which did not include the HTLC as this commitment transaction was revoked.
Of course HTLCs also need to be removed and the balance needs to be updated if the chain of HTLCs was successfully set up to the destination and the preimage is being released.
Recalling the full-duplex protocol with `commitment_signed` and `revoke_and_ack` messages you might wonder how to make this exchange `preimage` for the new state atomic.
Accepting an HTLC removes funds from a peer that the peer cannot utilize unless the HTLC is removed due to success or failure.
Similarly forwarding an HTLC binds some of the funds from your nodes payment channel until the HTLC has been removed again.
As we've explained at the very beginning of the chapter, engaging into the forwarding process of HTLCs does neither yield a direct risk to lose any funds nor does it gain the chance to gain any funds.
Owners of routing nodes might want to monitor the routing behavior and opportunities and compare them to the on-chain costs and the opportunity costs in order to compute their own routing fees that they wish to charge in order to accept and forward HTLCs.
Operators have to consider if they want to risk overpaying fees or losing funds in the forced channel closure cases because the commitment transactions have been added to the fees.