Now that we've followed Alice as she set up a Lightning Wallet and purchased a coffee from Bob, we'll look under the hood and unpack the different components of the Lightning Network involved in that process.
This chapter will give a high-level overview and will not delve into all the technical details.
The goal is rather to help you to become aware of the most important concepts and building blocks of the Lightning Network.
If you have experience in computer science, cryptography, Bitcoin, and protocol development, then this chapter should be enough for you to be able to fill out the connecting details by yourself.
If you are less experienced, this chapter shall give you a good enough overview so you will have an easier time understanding the formal protocol specifications, known as BOLTs (Basis of Lightning Technology).
**The Lightning Network (LN) is a peer-to-peer network of _payment channels_ implemented as smart contracts on the _Bitcoin blockchain_ as well as a communication protocol that defines how participants set up and execute these smart contracts**
We will see that a payment channel is simply a 2-out-of-2 multisignature address on Bitcoin, for which you hold one key and your channel partner holds the other key.
You and your channel partner negotiate a sequence of transactions that spend from this multisignature address. Instead of transmitting and recording these transactions on the Bitcoin blockchain, you both hold on to them, unspent.
The latest transaction in that sequence encodes the balance of the channel and defines how that balance is divided between you and your channel partner.
Thus, adding a new transaction to this sequence is equivalent to moving some part of the channel balance from one channel partner to the other, without the Bitcoin network being aware of it. As you negotiate each new transaction, changing the allocation of funds in the channel, you also revoke the previous transaction, so that neither party can regress to a previous state.
Each transaction in the sequence makes use of Bitcoin's scripting language, and thus the negotiation of funds between you and your channel partner is managed by a Bitcoin smart contract.
If you have an unpublished transaction from a 2-of-2 multisignature address, that pays you part of the balance, then a signature from the other party ensures that you can independently publish this transaction anytime by adding your own signature.
The ability to hold a partially-signed transaction, offline and unpublished, with the option to publish and own that balance at any time, is the basis of the Lightning Network.
Once several participants have channels from one party to another, a payment can also be "forwarded" from payment channel to payment channel, by setting up a route across the network connecting several payment channels together.
For example, Alice can send money to Charlie if Alice has a channel with Bob and Bob has a channel with Charlie.
By the design of the Lightning Network, it is possible to extend the smart contracts which operate the channel so that Bob has no way of stealing the funds that are being forwarded through his channels.
In the same way that the smart contract protects the channel partners, so they don't need to trust each other, the entire network protects the participants so that they can forward payments without trusting any of the other participants.
Since the channels are constructed from multisignature addresses and the balance update transactions are pre-signed Bitcoin transactions, all the trust that is needed to operate the Lightning Network comes from the trust in the decentralized Bitcoin network!
The aforementioned innovations are certainly the major breakthrough that allowed the creation of the Lightning Network.
It is a comprehensive communication protocol that allows peers to exchange Lightning messages to achieve the transfer of bitcoin.
The communication protocol defines how Lightning messages are encrypted and exchanged.
The Lightning Network also uses a "gossip" protocol to distribute public information about the channels (network topology) to all the participants.
Alice, for example, needs the network topology information to be aware of the channel between Bob and Charlie, so that she can construct a route to Charlie.
Last but not least, it is important to understand that the Lightning Network is nothing more than an application on top of Bitcoin, using Bitcoin transactions and Bitcoin Script. There is no "Lightning coin" or "Lightning blockchain".
Beyond all the technical primitives, the Lightning Network protocol is a creative way to get more benefits out of Bitcoin by allowing an arbitrary amount of instant payments with instant settlements without the necessity to trust anyone else but the Bitcoin network.
First, the time it takes for the internet to transfer the few bytes of data that the protocol requires to move funds from one end of the channel to the other.
Second, the capacity of the channel, meaning the amount of bitcoin that is committed to the channel when it is opened.
Payment channels have a few very interesting and useful properties:
* Because the time to update a channel is primarily bound by the communication speed of the internet, making a payment on a payment channel is almost instant.
* If the channel is open, making a payment does not require the confirmation of Bitcoin blocks. In fact - as long as you and your channel partner follow the protocol - it does not require any interaction with the Bitcoin network or anyone else other than your channel partner.
* The cryptographic protocol is constructed such that there is little to no trust needed between you and your channel partner. If your partner becomes unresponsive or tries to cheat you, you can ask the Bitcoin system to act as a "court" resolving the smart contract you and your partner have previously agreed upon.
* Payments made in a payment channel are only known to you and your partner. In that sense you gain privacy compared to Bitcoin, where every transaction is public. Only the final balance, which is the aggregate of all payments in that channel, will become visible on the Bitcoin blockchain.
Bitcoin was about 5 years old when talented developers first figured out how payment channels could be constructed and by now there are at least 3 different methods known.
This chapter will focus on the channel construction method first described in the Lightning Network whitepaper by Joseph Poon and Thaddeus Dryja in 2015. These are known as _Poon-Dryja_ channels, and are the channel construction method currently used in the Lightning Network.
The other two proposed methods are _Duplex Micropayment_ channels, introduced by Christian Decker around the same time as the "Poon-Dryja" channels and _eltoo_ channels, introduced in 2018 by Christian Decker, Rusty Russel and our co-author Olaoluwa Osuntokun.
Eltoo channels have some interesting properties and simplify the implementation of payment channels. However, eltoo channels require a change in the Bitcoin scripting language and therefore cannot be implemented on the Bitcoin mainnet at this time.
Payment channels are built on top of 2-of-2 multisignature addresses, timelocks and Segregated Witness transaction outputs. We will not revise these relatively advanced concepts of the Bitcoin system. Instead, in this section we will provide a high-level overview of multisignature scripts and how they allow us to construct payment channels.
If you have already studied Bitcoin and are familiar with multisignature addresses, feel free to skip this section.
A deep dive into the topic discussed here is presented in chapter 7 of Mastering Bitcoin which can be found at: https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch07.asciidoc.
Also, in case you are not familiar with P2PKH addresses and the basic format and scripting language of Bitcoin we encourage you to study chapter 6 of Mastering Bitcoin which can be found at: https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch06.asciidoc.
There is also a video on Rene's YouTube channel which dissects the bits and bytes of a transaction spending from a P2PKH output at: https://youtu.be/1n4g3eYX1UI
The Bitcoin scripting language provides a multisignature building block (primitive), that can be used to build escrow services and complex ownership configurations between several stakeholders. An arrangement that requires multiple signatures to spend Bitcoin is called a _multisignature scheme_, further specified as an _K-of-N_ scheme, where:
* N is the total number of signers identified in the multisignature scheme, and
* K is the _quorum_ or _threshold_ - the minimum number of signatures to authorize spending.
where K is the total number of listed public keys (Public Key 1 through Public Key N) and K is the threshold of required signatures to spend the output.
The Lightning Network uses a 2-of-2 multisignature scheme to build a payment channel. For example, a payment channel between Alice and Bob would be build on a 2-of-2 multisignature like this:
The fundamental building block of a payment channel, is a 2-of-2 multisignature address. The two channel partners fund the payment channel by sending bitcoin to the multisignature address. This transaction is called the _funding transaction_, and is recorded on the Bitcoin blockchain.
Even though the funding transaction is public, it is not obvious that it is a Lightning payment channel until it is closed. Furthermore, channel payments are still not visible to anyone other than the channel partners, nor is the distribution of the channel balance between them.
The amount deposited in the multisignature address is the called the _channel capacity_ and sets the maximum amount that can be sent across the payment channel. However, since funds can be sent back and forth, the channel capacity is not the upper limit on how much value can flow across the channel. That's because if the channel capacity is exhausted with payments in one direction, it can be used to send payments in the opposite direction again.
The funds sent to the multisignature address in the funding transaction are sometimes referred to as "locked in a Lightning channel". However in practice, funds in a Lightning channel are not "locked" but rather "unleashed". Lightning channel funds are more liquid than funds on the Bitcoin blockchain as they can be spent faster, cheaper and more privately. Opening a Lightning channel unleashes your Bitcoin!
If you think carefully about 2-of-2 multisignature addresses, you will realize that putting your funds into such an address seems to carry some risk. What if your channel partner refuses to sign a transaction to "release" the funds? Are they stuck forever? Let's look at that scenario and how the Lightning Network protocol avoids it, next.
Alice and Bob want to create a payment channel. They each create a private/public key pair and then exchange public keys. Now, they can construct a multisignature 2-of-2 with the two public keys, forming the foundation for their payment channel.
Next, Alice constructs a Bitcoin transaction sending a few mBTC to the multisignature address created from Alice's and Mallory's public keys. If Alice doesn't take any additional steps and simply broadcasts this transaction, she has to trust that Bob will provide his signature to spend from the multisignature address. Bob on the other hand has the chance to blackmail Alice by withholding his signature and denying Alice access to her funds.
In order to prevent this, Alice will need to create an additional transaction which spends from the multisignature address, refunding her mBTC. Alice then has Bob sign the refund transaction _before_ broadcasting her funding transaction to the Bitcoin network. This way, Alice can get a refund even if Bob disappears or fails to cooperate.
The "refund" transaction that protects Alice is the first of a class of transactions called _commitment transactions_, which we will examine in more detail next.
A _commitment transaction_ is a transaction that pays each channel partner their channel balance and ensures that the channel partners do not have to trust each other. By signing a commitment transaction, each channel partner "commits" to the current balance and gives the other channel partner the ability to get their funds back whenever they want.
By holding a signed commitment transaction, each channel partner can get their funds even without the cooperation of the other channel partner. This protects them against the other channel partner's disappearance, refusal to cooperate or attempt to cheat by violating the payment channel protocol.
The commitment transaction that Alice prepared in the previous example, was a "refund" of her initial payment to the multisignature address. More generally however, a commitment transaction splits the funds of the payment channel, paying the two channel partners according to the distribution (balance) they each hold. At first, Alice holds all the balance, so it is a simple refund. But as funds flow from Alice to Bob, they will each sign and exchange new commitment transactions that represent the new balance distribution, with some part of the funds paid to Alice and some paid to Bob.
. Alice creates a new private / public key pair and informs Bob that she wishes to open a channel via the `open_channel` message (a message in the Lightning Network protocol).
. Bob also creates a new private / public key pair and agrees to accept a channel from Alice, sending his public key to Alice via the `accept_channel` message.
. Alice now creates a funding transaction from her wallet that sends 10 mBTC to the multisignature address with a locking script +2 <PubKey Alice> <PubKey Bob> 2 CHECKMULTISIG+.
. Alice does not yet broadcast this funding transaction but sends Bob the transaction ID in a `funding_created` message.
. Both Alice and Bob create their version of a commitment transaction. This transaction will spend from the funding transaction and send all the bitcoin back to an address controlled by Alice.
. Alice and Bob don't need to exchange these commitment transactions, since they each know how they are constructed and can build both independently. They only need to exchange signatures.
. Alice provides a signature for Bob's commitment transaction. This signature was already included in the `funding_created` message.
By following this protocol, Alice does not give up ownership of her 10 mBTC even though the funds are sent to a 2-of-2 multisignature address for which Alice controls only one key.
Her only costs are the fees for the two on-chain transactions.
As long as she follows the protocol this is her only risk when opening a channel.
After this initial exchange, commitment transactions are created each time the channel balance changes. In other words, each time a payment is sent between Alice and Bob, new commitment transactions are created and signatures are exchanged. Each new commitment transaction encodes the latest balance between Alice and Bob.
If Alice wants to send 3 mBTC to Bob, both would create a new version of their commitment transactions which would now pay 7mBTC to Alice and 3 mBTC to Bob. By encoding a new balance for Alice and Bob, the new commitment transactions are the means by which a payment is "sent" across the channel.
Now that we understand commitment transactions, let's look at some of the more subtle details. You may notice that this protocol leaves a way for either Alice or Bob to cheat.
==== Cheating with prior state
How many commitment transactions does Alice hold after she pays 3 mBTC to Bob? She holds two: the original one paying her 10 mBTC and the more recent one, paying her 7 mBTC and Bob 3 mBTC.
In the channel protocol we have seen so far, nothing stops Alice from publishing a previous commitment transaction. A cheating Alice could publish the commitment transaction which grants her 10 mBTC.
Since that commitment transaction was signed by Bob he can't prevent Alice from transmitting it.
Some mechanism is needed to prevent Alice from publishing an old commitment transaction. Let us now find out how this can be achieved and how it enables the Lightning Network to operate without requiring any trust between Alice and Bob.
Because Bitcoin is censorship resistant, no one can prevent someone from publishing an old commitment transaction. To prevent this form of cheating, we construct commitment transactions so that if an old one is transmitted, the cheater can be punished. By making the penalty large enough, we create a strong incentive against cheating and this makes the system secure.
The way the penalty works is by giving the cheated party an opportunity to claim the balance of the cheater. So if someone attempts to cheat by broadcasting an old commitment transaction, in which they are paid a higher balance than they are due, the other party can punish them by taking *both* their own balance and the balance of the cheater. The cheater loses everything.
Let us go through the channel construction scenario again, adding a penalty mechanism to protect against cheating:
* Alice creates a channel with Bob and put 10 mBTC into it.
* Alice send 3 mBTC to Bob.
* Alice tries to cheat Bob out of his earned 3 mBTC by publishing an old commitment transaction claiming the full 10 mBTC for herself.
* Bob detects the fraud and punishes Alice by taking the full 10 mBTC for himself.
* Bob ends up with 10 mBTC, gaining 7 mBTC for catching Alice cheat.
* Alice ends up with 0 mBTC.
* Trying to cheat Bob out of 3 mBTC, she loses the 7 mBTC she owned.
With a strong penalty mechanism, Alice is not tempted to cheat by publishing an old commitment transaction as she risks losing her entire balance.
A key characteristic of bitcoin is that once a transaction is valid, it remains valid and does not expire. The only way to cancel a transaction is by double-spending its inputs with another transaction before it was mined. That's why we used timelocks [...] to ensure that more recent commitments could be spent before older commitments were valid.
Now that we understand _why_ a penalty mechanism is needed and how it will prevent cheating, let's see _how_ it works in detail.
Usually, the commitment transaction has at least two outputs, paying each channel partner. We change this to add a _timelock delay_ and a _revocation secret_, to one of the payments. The timelock prevents the owner of the output from spending it immediately once the commitment transaction is included in a block. The revocation secret, allows either party to immediately spend that payment, bypassing the timelock.
So, in our example, Bob holds a commitment transaction which pays Alice _immediately_, but his own payment is delayed and revocable. Alice also holds a commitment transaction, but hers is the opposite: it pays Bob immediately but her own payment is delayed and revocable.
The two channel partners hold half of the revocation secret, so that neither one knows the whole secret. If they share their half, then the other channel partner has the full secret and can use it to exercise the revocation condition. When signing a new commitment transaction, they revoke the previous commitment by exchanging the signature for the revocation secret.
In simple terms, Alice signs Bob's new commitment transaction only if Bob offers his half of the revocation secret for the previous commitment. Bob only signs Alice's new commitment transaction if she gives him her half of the revocation secrect from the previous commitment.
With each new commitment, they exchange the necessary "punishment" secret that allows them to effectively _revoke_ the prior commitment transaction by making it unprofitable to transmit. Essentially, they destroy the ability to use the old commitments as they sign the new ones.
The timelock is set to a number of blocks, up to 2016 (approximately two weeks). If either channel partner publishes a commitment transaction without cooperating with the other partner, they will have to wait for that number of blocks (e.g. 2 weeks) to claim their balance. The other channel partner can claim their own balance at any time. Furthermore, if the commitment they published was previously revoked, the channel partner can *also* immediately claim the cheating party's balance, bypassing the timelock and punishing the cheater.
The timelock is adjustable and can be negotiated between channel partners. Usually, it is longer for larger capacity channels, and shorter for smaller channels, to align the incentives with the value of the funds.
For every new update of the channel balance, new commitment transactions and new revocation secrets have to be created and saved. As long as a channel remains open, all revocation secrets _ever created_ for the channel need to be kept as they might be needed in the future. Fortunately, the secrets are rather small and it is only the channel partners who need to keep them, not the entire network.
Nevertheless, managing and storing the revocation secrets is one of the more elaborate parts of Lightning nodes that require node operators to maintain backups.
Technologies such as Watchtower services or changing the channel construction protocol to the "eltoo" protocol might be future strategies to mitigate these issues and reduce the need for revocation secrets, penalty transactions and channel backups.
Alice can close the channel at any time if Bob does not respond, claiming her fair share of the balance.
After publishing the *last* commitment transaction on-chain Alice has to wait for the time lock to expire before she can spend her funds from the commitment transaction. As we will see later, there is an easier way to close a channel without waiting, as long as Alice and Bob are both online and cooperate to close the channel with the correct balance allocation. But the commitment transactions stored by each channel partner act as a failsafe, ensuring they do not lose funds if there is a problem with their channel partner.
Channels partners can agree to announce their channel to the whole Lightning Network, making it a _public channel_. To announce the channel, they use the Lightning Network's gossip protocol to tell other nodes about the existence, capacity and fees of the channel.
Announcing channels publicly allows other nodes to use them for payment routing, thereby also generating routing fees for the channel partners.
By contrast, the channel partners may decide to keep the existence of the channel private and not announce it, making it a _private channel_.
[NOTE]
====
While a private channel will not be known to others while it is in use, it's existence and capacity will be revealed when the channel closes, because those details will be visible on-chain in the final settlement transaction.
====
Private channels are still used to route payments but only by the nodes which are aware of their existence, or given "routing hints" about a path that includes a private channel.
When a channel and its capacity is publicly announced using the gossip protocol, the announcement can also include information about the channel (metadata), such as it's routing fees and timelock duration.
When new nodes join the Lightning Network they collect the channel announcements propagated via the gossip protocol from their peers, building an internal "map" of the Lightning Network. This map can then be used to find paths for payments, connecting channels together end-to-end.
The best way to close a channel is... to not close it! Opening and closing channels requires an on-chain transaction, which will incur transaction fees. So it's best to keep channels open as long as possible. You can keep using your channel to make and forward payments, as long as you have sufficient capacity on your end of the channel. But even if you send all the balance to the other end of the channel, you can then use the channel to receive payments from your channel partner. This concept of using a channel in one direction and then using it in the opposite direction is called "re-balancing" and we will examine it in more detail in another chapter. By re-balancing a channel, it can be kept open almost indefinitely and used for thousands of payments.
* You want to reduce the balance held on your Lightning channels for security reasons and want to send funds to "cold storage"
* Your channel partner becomes unresponsive for a long time and you cannot use the channel anymore.
* The channel is not being used often because your channel partner is not a well-connected node, so you want to use the funds for another channel with a better-connected node.
* Your channel partner has breached the protocol either due to a software bug or on purpose forcing you to close the channel to protect your funds.
Which way you close a channel will depend on the circumstances. For example, if your channel partner is offline you will not be able to follow "the good way" because a mutual close cannot be done without a cooperating partner.
Usually, you Lightning Network software will automatically select the best closing mechanism available under the circumstances.
Whether you (the user) issue a command to close a channel or the Lightning Network software automatically closes a channel due to a detected anomaly (e.g. cheating partner), it is usually the Lightning Network software that chooses the appropriate way to close the channel.
===== The good way - Mutual close
The good way to close a channel is the mutual close and this way is always preferred, if possible. When you decide that you want to close a channel, your Lightning Network node will inform your channel partner about your intention. Now both your node and the channel parter's node work together to close the channel. No new routing attempts will be accepted from either channel partner and any ongoing routing attempts will be settled or removed after they time-out. Finalizing the routing attempts takes time, so a mutual close can also take some time to complete.
Once there are no pending routing attempts, the nodes cooperate to prepare a _closing transaction_. This transaction is similar to the commitment transaction, in that it distributes the channel balance to each channel partner according to their last commitment. Unlike the last commitment transaction, the outputs are not encumbered with a time lock.
The on-chain transaction fees for the closing transaction are paid by the channel partner who opened the channel and not by the one who initiated the closing procedure. Using the on-chain fee estimator, the channel partners agree on the appropriate fee and both sign the closing transaction.
Once the closing transaction is broadcast and confirmed by the Bitcoin network, the channel is effectively closed and each channel partner has received their share of the channel balance. A mutual close is usually the fastest way to close a channel.
Mutual close is not always possible. Sometimes, for example if your channel partner cannot be reached, you have to unilaterally close the channel in order to "free" the funds. Closing the channel unilaterally is called a _force close_. To do a force close, Alice can simply publish the last commitment transaction. After all, that's what commitment transactions are for - they offer a guarantee that Alice doesn't need to trust Bob to retrieve the balance of the channel that belongs to her.
Once Alice broadcasts the last commitment transaction she has to the Bitcoin network and it is confirmed, it will create two spendable outputs, one for Alice and one for Bob. As we discussed previously, because this commitment transaction is held by Alice, it gives a slight "advantage" to Bob. Alice's output will have a timelock delay and Bob's will be spendable immediately. This is to protect Bob, so that Alice can't broadcast an old commitment transaction and steal some of the balance. The timelock delay gives Bob the opportunity to "dispute" the transaction using the revocation secret and punish Alice for cheating.
1. When the commitment transaction was negotiated, the channel partners didn't know how much the on-chain fees would be at the future time the transaction would be broadcast. Since the fees cannot be changed without changing the outputs of the commitment transaction (needs both signatures) and since the force close happens when a channel partner is not available to sign, the protocol developers decided to be very generous with the fee rate included in the commitment transactions. It can be up to 5 times higher than the fee estimators suggest at the time the commitment transaction is negotiated.
2. The commitment transaction includes additional outputs for any pending routing attempts (HTLCs), which makes the commitment transactio bigger than a mutual close transaction. Bigger transactions incur more fees.
3. Any pending routing attempts will have to be resolved on-chain causing additional on-chain transactions. These additional transactions don't have to overestimate the fees but they nonetheless incur fees.
Hash Time-Locked Contracts (HTLCs) will be covered in detail in <<htlcs>>. For now, assume that these are payments that are routed across the Ligntning Network, rather than payments made directly between the two channel partners. These HTLCs are carried as additional outputs in the commitment transactions, thereby increasing the transaction size and on-chain fees.
====
In general, you should not do a "force close" unless absolutely necessary. Your funds will be locked for a longer time and the person who opened the channel will have to pay higher fees. Furthermore, you might have to pay on-chain fees to abort or settle routing attempts even if you haven't opened the channel.
If the channel partner is known to you, you might consider contacting that individual or company and inquire why their Lightning Node is down and request that they restart it so that you can achieve a mutual close of the channel.
You should consider a force close, only as the last resort.
===== The ugly way - Protocol breach
If your channel partner broadcasts an old (revoked) commitment transaction - whether deliberately or not - your node should be able to detect this protocol breach. Your node must be online and watching new blocks and transactions on the Bitcoin blockchain to detect this. Because your channel partner's payment will be encumbered by a timelock, your node has some time to act, before your channel partner can spend the payment and profit from the protocol breach.
During the timelock, your node can broadcast a _punishment transaction_ that uses the revocation secret to take the balance of your channel partner, as well as your own balance.
In this scenario the channel closure will be rather fast. You will have to pay on-chain fees to publish the punishment transaction, but your node can set these fees according to the fee estimation and not overpay. Furthermore, you will be taking the funds from your channel partner's balance too, so in a way the fees are unlikely to be a factor.
If your node is not online and the timelock expires, your channel partner will be able to take whatever balance was allocated to them in that old commitment, perhaps as much as the entire channel balance. If there is any balance allocated to you, you will have to pay on-chain fees to collect that balance.
Also all pending routing attempts will have to be resolved just as we saw in the force close scenario.
The "ugly way" of channel closure can be executed faster than both the "good" and "bad" ways, because you can immediately publish the punishment transaction and collect all the funds.
Game theory predicts that cheating is not a successful strategy because it is easy to detect a cheater and the cheater will lose all funds committed in the channel.
Hence, we do not recommend cheating. We do, however, recommend that anyone catching a cheater punish them by taking their funds.
You do so by running software that monitors the public Bitcoin blockchain for on-line transactions that correspond to any commitment transactions for any of your channels. This software is either:
* A properly maintained Lightning node, running 24x7.
* A single-purpose _watchtower_ node that you run to watch your channels.
* A third-party watchtower node that you pay to watch your channels.
We will look at watchtowers in more detail in <<watchtowers>>.
As long as you run your Lightning node once before the timeout period is reached, it will catch all cheats. It is not advisable to take this kind of risk; it is just as easy to keep a well maintained node running continuously.
Most payments on the Lightning Network start with an invoice, generated by the recipient of the payment. In our previous example, Bob creates an invoice to "request" a payment from Alice.
[NOTE]
====
There is a way to send an "unsolicited" payment without an invoice, using a work-around in the protocol called _keysend_. We will examine this in <<keysend>>.
====
An invoice is a simple payment instruction containing information such as a unique payment identifier, called a payment hash, a recipient, an amount, and an optional text description.
The most important part of the invoice is the payment hash, that allows the payment to travel across multiple channel in an _atomic_ way. Atomic, in computer science, means any action or state change that is either completed successfully or not at all - there is no possibility of an intermediate state or partial action. In the Lightning Network that means that the payment either travels the whole path or fails completely. It cannot be partially completed such that an intermediate node on the path can receive the payment and keep it.
There is no such thing as a "partial payment" or "partly successful payment".
Invoices are not communicated over the Lightning Network. Instead, they are communicated "out of band", using any other communication mechanism. This is similar to how Bitcoin addresses are communicated to senders outside the Bitcoin network, as a QR code, over email, or a text message. For example, Bob can present a Lightning invoice to Alice as a QR code, or send it via email, or any other message channel.
Invoices are usually encoded either as a long bech32-encoded string or as a QR code, to be scanned by a smartphone Lightning wallet. The invoice contains the amount of bitcoin that is requested and a signature of the recipient. The sender uses the signature to extract the public key (also known as the node ID) of the recipient so that the sender knows where to send the payment.
Did you notice how this contrasts with Bitcoin and how different terms are used? In Bitcoin, the recipient passes an address to the sender. In Lightning, the recipient creates an invoice and sends an invoice to the sender. In Bitcoin, the sender sends funds to an address. In Lightning, the sender pays an invoice and the payment gets routed to the recipient. Bitcoin is based on the concept of an "address", and Lightning is a payment network and based on the concept of an "invoice". In Bitcoin we create a "transaction" whereas in Lightning we send a "payment".
==== Payment Hash and Preimage
The most important part of the invoice is the _payment hash_. When constructing the invoice, Bob will make a payment hash as follows:
1. Bob chooses a random number +r+. This random number is called the _preimage_ or _payment secret_.
2. Bob uses +SHA256+ to calculate the hash +H+ of +r+ called the _payment hash_
latexmath:[H = SHA256(r)].
[NOTE]
====
The term _preimage_ comes from mathematics. In any function _y = f(x)_, the set of inputs that produce a certain value _y_ are called the preimage of _y_. In this case, the function is the SHA256 hash algorithm and any value _r_ that produces the hash _H_ is called a preimage.
====
Since the value +r+ is secret, only Bob knows it's value. No one can guess the value +r+, but once Bob reveals +r+, anyone who has the hash +H+ can check that +r+ is the correct secret, by calculating +SHA256(r)+ and seeing that it matches +H+.
The payment process of the Lightning Network is only secure if +r+ is chosen completely randomly and is not predictable. This security relies on the fact that hash functions cannot be inverted or feasibly brute-forced and therefore no one can find +r+ from +H+.
Invoices can optionally include other useful meta data such as a short text description. If a user has several invoices to pay, the user can read the description and be reminded what the invoice is about.
The invoice can also include some _routing hints_, which are private channels that have not been publicly announced, but can be used by the sender to construct a route to the recipient. Routing hints can also be used to suggest public channels, for example channels known by the recipient to have enough inbound capacity to route the payment.
In case the sender's Lightning node is unable to send the payment over the Lightning Network, invoices can optionally include an on-chain Bitcoin address as a fallback.
[NOTE]
====
While it is always possible to "fall back" to an on-chain Bitcoin transaction, it is actually better to open a new channel to the recipient instead. If you have to incur on-chain fees to make a payment, you might as well incur those fees to open a channel and make the payment over Lightning. After the payment is made, you are left with an open channel that has liquidity on the recipient's end and can be used to route payments back to your Lightning node in the future. One on-chain transaction gives you a payment and a channel for future use.
====
Lightning invoices contain an expiry date. Since the recipient must keep the preimage +r+ for every invoice issued, it is useful to have invoices expire so that these preimages do not need to be kept forever. Once an invoice expires or is paid, the recipient can discard the preimage.
We have seen how the recipient creates an invoice that contains a payment hash. This payment hash will be used to move the payment across a series of payment channels, from sender to recipient, even if they do not have a direct payment channel between them.
In the next few sections we will dive into the ideas and methods that are being used to deliver a payment over the Lightning Network and use all the concepts we have presented so far.
First, let's look at the Lightning network's communication protocol.
As we mentioned previously, when a payment channel is constructed, the channel partners have the option of making it public, announcing its existence and details to the whole Lightning network.
Channel announcements are communicated over a peer-to-peer _gossip protocol_. A peer-to-peer protocol is a communications protocol where each node connects to a random selection of other nodes in the network, usually over TCP/IP. Each of the nodes that are directly connected (over TCP/IP) to your node are called your _peers_. Your node in turn is one of their peers. Keep in mind, when we say that your node is connected to other peers, we don't mean that you have payment channels, but only via the gossip protocol.
After opening a channel, a node may choose to send out an announcement of the channel via the `channel_announcement` message to its peers.
Every peer validates the information from the `channel_announcement` message and verifies that the funding transaction is confirmed on the Bitcoin blockchain.
After verification the node will forward the gossip message to its own peers, and they will forward to their peers and so on, spreading the announcement across the entire network.
In order to avoid excessive communication the channel announcement is only forwarded by each node if it has not already forwarded that announcement previously.
The gossip protocol is also used to announce information about known nodes, with the `node_announcement` message.
For this message to be forwarded a node has to have at least one public channel announced on the gossip protocol, again to avoid excessive communication traffic.
Since nodes might occasionally change the meta data of their channels, this information is shared in a `channel_update` message.
These messages will only be forwarded approximately four times a day (per channel) to prevent excessive communication.
The gossip protocol also has a variety of queries and messages to initially synchronize a node with the view of the network or to update the node's view after being offline for a while.
A major challenge for the participants of the Lightning Network is that the topology information that is being shared by the gossip protocol is only partial.
1. To protect the privacy of the users and not shout out every financial transaction and payment that is being conducted. Channel balance updates would reveal that a payment has moved across the channel. This information could be correlated to reveal all payment sources and destinations.
2. To scale the amount of payments that can be conducted with the Lightning Network. Remember that the Lightning Network was created in the first place because notifying every participant about every payment does not scale well. Thus, the Lightning Network cannot be designed in a way that balance updates of channels are shared among participants.
3. The Lightning Network is a dynamic system. It changes constantly and frequently. Nodes are being added, other nodes are being turned off, balances change, etc. Even if everything is always communicated, the information will be valid only for a short amount of time. As a matter of fact, information is often outdated by the time it is received.
We will examine the details of the gossip protocol in a later chapter.
For now, it is only important to know that the gossip protocol exists and that it is used to share topology information of the Lightning Network.
This topology information is crucial for delivering payments through the network of payment channels.
Payments on the Lightning Network are forwarded along a _path_ made of channels linking one participant to another, from the payment source to the payment destination. The process of finding a path from source to destination is called _path finding_. The process of using that path to make the payment is called _routing_.
[NOTE]
====
A frequent criticism of the Lightning network is that "routing" is not solved, or even is an "unsolvable" problem. In fact, routing is trivial. Path finding, on the other hand, is a difficult problem. The two terms are often confused and need to be clearly defined to identify which problem we are attempting to solve.
====
As we will see next, the Lightning Network currently uses a _source-based onion-routed_ protocol for path finding and routing payments. Source-based means that the sender of the payment has to find a path through the network to the intended destination. Onion-routed means that the elements of the path are layered (like an onion), with each layer encrypted so that it can only be seen by one node at a time. We will discuss onion routing in the next section.
=== Source-based Path Finding
If we knew the exact channel balances of every channel we could easily compute a payment path using any of the standard path finding algorithms taught in any computer science program. This could even be solved in a way that optimizes the fees paid to nodes for forwarding the payment.
However, the balance information of all channels is not and cannot be known to all participants of the network. We need more innovative path finding strategies.
With only partial information about the network topology, path finding is a real challenge and active research is still being conducted into this part of the Lightning Network. The fact that the path finding problem is not "fully solved" in the Lightning Network is a major point of criticism towards the technology.
[NOTE]
====
One common criticism of path-finding in the Lightning network is that it is unsolvable because it is equivalent to the NP-complete _Traveling Salesperson Problem_, a fundamental problem in computational complexity theory. In fact, path finding in Lightning is not equivalent to TSP and falls into a different class of problems. We successfully solve these types of problems (path finding in graphs with incomplete information) every time we ask Google to give us driving directions with traffic avoidance. We also successfully solve this problem every time we route a payment on the Lightning network.
====
Path finding and routing can be implemented in a number of different ways and multiple path-finding and routing algorithms can co-exist on the Lightning network, just as multiple routing algorithms exist on the internet. Source-based routing is one of many possible solutions and is successful at the current scale of the Lightning network.
The path finding strategy currently implemented by Lightning nodes is to "probe" paths until one is found that has enough liquidity to forward the payment. This is an iterative process of trial and error, until success is achieved or no path is found. The algorithm currently does not necessarily result in the path with the lowest fees. While this is not optimal and certainly can be improved, even this simplistic strategy works quite well.
This "probing" is done by the Lightning node or wallet and is not directly seen by the user.
The user might only realize that probing is taking place if the payment does not complete instantly.
On the Internet we use the internet protocol and an IP forwarding algorithm to forward internet packages from the sender to the destination. While these protocols have the nice property of allowing internet hosts to collaboratively find a path for information flow through the internet, we cannot reuse and adopt this protocol for forwarding payments on the Lightning Network. Unlike the internet, Lightning payments have to be _atomic_ and channel balances have to remain _private_. These constraints require novel strategies.
Of course, path finding is trivial if we want to pay our direct channel partner and we have enough balance on our side of the channel to do so. In all other cases, our node uses information from the gossip protocol to do path finding. This includes currently known public payment channels, known nodes, known topology (how known nodes are connected), known channel capacities, and known fee policies set by the node owners.
If the sending node of a payment has selected a path that is supposed to be used to make the payment, the Lightning Network uses an onion routing scheme similar to the famous TOR-network.
Note that Lightning Onion routing via "SPHINX mixformat" is only similar to the TOR network routing in concept, but as a protocol as well as in implementation it is definitely different from the TOR network.
The node than places the remaining, but one layer smaller, onion into the box and sends the box to the next node on the routing path.
This continues until the receiving node is the payee and the routing stops and the payee can read the information encrypted in the innermost layer of the onion.
1. The most important property is that a routing node can only see on which channel it received an onion and on which channel to set up an HTLCs and thus to which peer to forward the onion.
This means that no routing node can know who initiated the payment and to whom the payment is destined.
4. Onions can have up to 20 hops or onion layers if you prefer. This allows for sufficiently long paths.
5. The encryption of the onion for every hop uses different ephemeral encryption keys with every single onion. Should a key (in particular the private key of the public node key) leak at some point in time an attacker who collected onions cannot decrypt other onions.
In simpler terms, keys are never reused in order to achieve more security.
6. Errors can be sent back from the erring node in an encrypted way to the original sender. This is particularly useful as we have seen that Lightning nodes who initiate the onions select a path without knowing whether every node has enough liquidity along their channels to forward the payment.
We will discuss the details of the Onion Format later. But we note already that the `Payment Hash` which is required to set up the HTLCs for the payment is not transported within the onions.
The `Payment Hash` is rather included in the Lightning Message alongside the onion.
In stark contrast to the Internet Protocol Forwarding Algorithm on the Lightning Network it is only the sender of a payment that decides the path it takes to the recipient.
It contains information about the payment for the recipient, routing hints for the intermediary nodes.
Routing fees which those nodes will earn if (and only if) the payment is successfully delivered and settles with the release of the `Payment Hash` are also included to the onion.
All Lightning Nodes must follow the payment forwarding algorithm.
Upon receiving a Lightning Message called `update_add_htlc` which contains the `Payment Hash` and the onion package it needs to follow through X steps:
1. It needs to decrypt the outer layer of the onion and use the HMAC to make sure that the onion was not corrupted.
2. It will check that it can fulfill the routing hints in the way that the fees are respecting its fee announcements and that it has enough liquidity on the outgoing channel or is even owner of the specified channel.
3. It needs to work with its peer on the incoming channel to update the channel state to include the offered HTLC.
4. It will have to add some padding data to the end of the onion to keep it as a constant size even though it removed some data from the beginning.
5. It will then follow the routing hints from its layer of the onion to forward the modified onion package on its outgoing payment channel by also sending and `update_add_htlc` message which includes the same `Payment Hash` and the onion.
If for example in the first step the node detects that the onion was corrupted it will not proceed to the second and subsequent steps to move forward the channel state on the receiving channel to include the HTLC.
Similarly if in the second step the node discovers that it won't be able to fulfill the routing request or that not enough fees have been included it will also abort the payment forwarding algorithm and rather send back an error.
Even though the node sending the error does not know who initially sent out the payment the error message will be encrypted with the key that was used to encrypt the forwarding nodes layer.
While the likelihood for a payment to fail is pretty high if it does not settle quickly a node SHOULD never initiate another payment attempt along a different path before the onion returned with an error as a node might just have delayed the forwarding of the payment.
Nodes which act maliciously by delaying the forwarding of payments or errors are actually hard to detect due to the privacy properties that are gained with the onion routing scheme.
Most importantly communication is needed to open and close payment channels, to send and receive onions, to set up and settle or fail HTLCs and for exchanging gossip information.
This makes development a little bit tricky as one cannot easily monitor one's own traffic on a tool like wireshark for debugging. footnote:[Luckily tools exist to make developer's lives easier: https://github.com/nayutaco/lightning-dissector]
As long as a person follows the protocol and has their node secured, there is no major risk of losing funds when participating in the Lightning Network.
However, there is the cost of paying on-chain fees when opening a channel.
Any cost should come with a corresponding benefit.
In our case the reward for Alice for bearing the cost of opening a channel is that Alice can send and receive payments of bitcoin on the Lightning Network at any time and that she can earn fees in bitcoin by forwarding payments for other people.
Alice decides that the reward is enough for her to take on the cost of the on-chain fee for creating a channel to Bob.
In contrast, Alice will not open a channel to someone unknown in a foreign country who just sent her an uninvitedly email asking her to open a channel to him.
While the Lightning Network is built on top of Bitcoin and inherits many of its features and properties, there are important differences that users of both networks need to be aware of.
==== Addresses vs Invoices, Transactions vs Payments
Bitcoin and Lightning have different terminology.
In Bitcoin, a typical operation is that a user receives a Bitcoin address (e.g. he reads it on a webpage, or he received it in an instant message from a friend).
For instance, a user making a payment of 1 BTC can use a single output with value 1 BTC, two outputs with value 0.25 BTC and 0.75 BTC, or four outputs with value 0.25 BTC each.
The entire UTXO needs to be spent, so if a user wishes to spend 0.8 BTC, but only has a 1 BTC UTXO, then they need to send 0.8 BTC to the receiver, and 0.2 BTC back to themselves as change.
This is because the channel partners simply update the channel balance and only create a new UTXO when the channel is eventually closed using the Bitcoin network.
These fees are paid to the miner who mines that particular block.
The amount of the fee is based on the _size_ of the transaction in _bytes_ that the transaction is using in a block, as well as how quickly the user wants that transaction mined.
As miners will typically mine the most profitable transactions first, a user who wants their transaction mined immediately will pay a _higher_ fee-per-byte, while a user who is not in a hurry will pay a _lower_ fee-per-byte.
On the Lightning Network, users pay fees to other users to route payments through their channels.
In order to route a payment, a routing user will have to move the funds in two or more channels they own, as well as transmit the data for the sender's payment.
Typically, the routing user will charge the sender based on the _value_ of the payment, as well as setting their own fees they established to route the payment.
Higher value payments will thus cost more to route, and a market for capacity will exist where different users will charge different fees for routing through their channels.
On the Bitcoin network, miners are profit-seeking and will typically include as many transactions in a block as possible, while staying within the block size limit (actually, a modified form called the block weight limit).
If there are more transactions in the queue (called the mempool) than can fit in a block, they will begin by mining the transactions that pay the highest fees per byte (highest fee per weight).
Thus, if there are many transactions in the queue, users will have to pay a higher fee to be included in the next block, or they will have to wait until there are fewer transactions in the queue.
The scarce resource on the Bitcoin network is the space in the blocks. Bitcoin users compete for block space. And the Bitcoin fee market is based on available block space. The scarce resource in the Lightning network is the capacities on the channels and channel connectivity. Lightning users compete for capacity and connectivity. The Lightning fee market is driven by capacity and connectivity.
On the Lightning Network, users are paying fees to the users routing their payments. Routing a payment in economic terms is nothing else than providing and assigning capacity to the payer.
Different routers will charge different fees for routing through their channels.
Following general economic principles, the more capacity and connectivity is provided the more competition there will be and as a result the lower the fees will be.
The growth of the Lightning Network might lead to lower Lightning fees, while the growth in the Bitcoin Network likely leads to higher Bitcoin fees.
While the addresses involved are pseudonymous and are not typically tied to identity, they will still be collected and validated by every other user on the network.
As payments are onion-routed, the users who route the payment will only be aware of the amount of the payment, but neither the source nor the destination.
This is because the security model of Lightning only requires users to control the balances in their channels.
In summary, Bitcoin transactions are broadcasted publicly and stored forever. Lightning payments are executed between a few selected peers and information about is privately stored and only until the channel is closed.
Creating en-masse analysis tools equivalent to the ones on the Bitcoin network will be impossible on Lightning.
When the channel is closed, a transaction will be made on the Bitcoin network and, only once that transaction is confirmed will the channel be considered closed.
For instance, if a user owns one channel with 0.4 BTC on their side, and another channel with 0.2 BTC on their side, then the maximum they can send with one payment is 0.4 BTC.
At the time of writing, Atomic Multi-Path Payments (AMPs) are in development which, in the above example, would allow the user to combine both their 0.4 BTC and 0.2 BTC channels to be able to send a maximum of 0.6 BTC with one payment.
On the Lightning Network, the balance in a channel at a particular time is known only to the two channel partners, and is only made visible to the rest of the network when the channel is closed.
When the channel is closed, the final balance of the channel is submitted to the Bitcoin blockchain, and each partner receives their share of the bitcoin in that channel.
For instance, if the opening balance was 1 BTC paid by Alice, and Alice made a payment of 0.3 BTC to Bob, then the final balance of the channel is 0.7 BTC for Alice and 0.3 BTC for Bob.
If Alice tries to cheat by submitting the opening state of the channel to the Bitcoin blockchain, with 1 BTC for Alice and 0 BTC for Bob, then Bob can retaliate by submitting the true final state of the channel, as well as create a penalty transaction that gives him all bitcoin in the channel.
For the Lightning network, the Bitcoin blockchain acts as a court system.
Like an arbitrer Bitcoin records the initial and final balances of each channel, and approves penalties if one of the parties tries to cheat.
The recipient could be logically speaking "online" (e.g. run a node) or could be "offline" (e.g. does not run a node, does not use any wallet currently).
The transaction is registered in the blockchain, and at any later point in time the recipient can have a look at it and see that has has received funds in the past.
Since Lightning sits on top of Bitcoin and relies on Bitcoin as its underlying base layer, it is clear that the security model of Lightning reduces to the security of the base blockchain.
