diff --git a/README.md b/README.md index 8d2e10b..17c9e60 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,9 @@ The book is suitable for technical readers with an understanding of the fundamen ## Status -The current status of the book is "RELEASE PREP". See below for status of specific chapters and read the contribution guide to learn how and where to contribute. +The current status of the book is "IN PRODUCTION": ONLY COPY-EDIT CONTRIBUTIONS ARE ACCEPTED. + +See below for status of specific chapters and read the contribution guide to learn how and where to contribute. ### Legend @@ -45,10 +47,13 @@ The current status of the book is "RELEASE PREP". See below for status of specif | [LN Security and Privacy](13_security_privacy_ln.asciidoc) | ################ | :heavy_check_mark: | | APPENDICES | APPENDICES | APPENDICES | | [A1 - Bitcoin Fundamentals Review](appendix-bitcoin-fundamentals-review.asciidoc) | ########### | :heavy_check_mark: | -| [An - License Notices](appendix_license_notices.asciidoc) | # | :heavy_check_mark: | +| [A2 - Payment Requests (Invoices)](appendix_payment_requests.asciidoc) | #### | :heavy_check_mark: | +| [A3 - Wire Protocol](appendix_wire_protocol.asciidoc) | ################# | :heavy_check_mark: | +| [A4 - Encrypted Message Transport (Brontide)](appendix_encrypted_transport.asciidoc) | ########## | :heavy_check_mark: | +| [A5 - License Notices](appendix_license_notices.asciidoc) | # | :heavy_check_mark: | -Total Word Count: 110718 +Total Word Count: 125288 Target Word Count: 100,000-120,000 diff --git a/contrib/brontide.asciidoc b/appendix_encrypted_transport.asciidoc similarity index 96% rename from contrib/brontide.asciidoc rename to appendix_encrypted_transport.asciidoc index de0bb79..d96b850 100644 --- a/contrib/brontide.asciidoc +++ b/appendix_encrypted_transport.asciidoc @@ -1,6 +1,11 @@ -= Brontide: Lightning's Encrypted Message Transport +[appendix] +[[brontide]] +[[encrypted_message_transport]] +== Lightning's Encrypted Message Transport (Brontide) -== Intro +In this chapter we will review the Lightning Network's _Encrypted Message Transport_, also known as the _Brontide Protocol_, which allows peers to establish end-to-end encrypted communication, authentication and integrity checking. + +=== Introduction Unlike the vanilla Bitcoin P2P network, every node in the Lightning Network is identified by a unique public key which serves as it identity. By default, this @@ -18,7 +23,7 @@ custom encrypted message transport Lightning uses, commonly referred to as "Brontide" (more on that later) can be dropped into any context that requires encrypted communication between two parties. -== The Channel Graph as Decentralized Public Key Infrastructure +=== The Channel Graph as Decentralized Public Key Infrastructure As we learned in the chapter on multi-hop forwarding, very node has a long-term identity that is used as the identifier for a vertex during path finding and @@ -48,7 +53,7 @@ Lightning network is able to significantly simply its encrypted transport protocol as it doesn't need to deal with all the complexities that come along with TLS, the Transport Layer Security protocol. -== Why Not TLS? +=== Why Not TLS? Readers familiar with the TLS system may be wondering at this point: why wasn't TLS used in spite of the drawbacks of the existing PKI system? It is indeed a @@ -80,7 +85,7 @@ to continual scrutiny of the TLS protocol over decades. In a way, the existence of Noise allows the community to effective "start over", with a more compact, simplified protocol that retains all the added benefits of TLS. -== The Noise Protocol Framework +=== The Noise Protocol Framework The Noise Protocol Framework is a modern, extensible, and flexible message encryption protocol designed by the creators of the Signal protocol. The Signal @@ -114,7 +119,7 @@ Lightning Network, the flavor of Noise use will be referred to from here on as "Brontide". A brontide is a low billowing noise, similar to what one would hear during a thunderstorm when very far away. -=== Noise Protocol Handshakes +==== Noise Protocol Handshakes The Noise protocol is extremely flexible in that it advertises several handshakes, each with different security and privacy properties for a would be @@ -129,7 +134,7 @@ handshake. Instead, a clever series of Elliptic-Curve Diffie-Hellman (ECDH) and Message Authentication Code (MAC) checks are used to authenticate the responder. -=== Handshake Notation & Protocol Flow +==== Handshake Notation & Protocol Flow Each handshakes typically consist of several steps. At each step some (possibly) encrypted material is sent to the opposite party, an ECDH (or @@ -148,12 +153,12 @@ refer to a new ephemeral key. ECDH operations between two keys are notated as the concatenation of two keys. As an example, `ee` represents an ECDH operation between two ephemeral keys. -== Brontide: Lightning's P2P Encryption +=== Brontide: Lightning's P2P Encryption -=== High-Level Overview +==== High-Level Overview Using the notation laid out earlier, we can succinctly describe the `Noise_XK` -as follows: +as follows: ``` Noise_XK(s, rs): <- rs @@ -196,7 +201,7 @@ variant, the ASCII string is hashed into a digest, which is used to initialize the starting handshake state. In the context of Brontide, the ASCII string describing the protocol is: `Noise_XK_secp256k1_ChaChaPoly_SHA256`. -=== Brontide: A Handshake in Three Acts +==== Brontide: A Handshake in Three Acts The handshake portion of Brontide can be see prated into three distinct "acts". The entire handshake takes 1.5 round trips between the initiator and responder. @@ -229,7 +234,7 @@ functions that will operate on the handshake and messaging state. When describing the handshake protocol, we'll use these variables in a manner similar to pseudo-code in order to reduce the verbosity of the explanation of each step in the protocol. We'll define the _functional_ primitives of the -handshake as: +handshake as: * `ECDH(k, rk)`: performs an Elliptic-Curve Diffie-Hellman operation using `k`, which is a valid `secp256k1` private key, and `rk`, which is a valid public key @@ -263,7 +268,7 @@ handshake as: * `a || b` denotes the concatenation of two byte strings `a` and `b` -==== Handshake Session State Initialization +===== Handshake Session State Initialization Before starting the handshake process, both sides need to initialize the starting state that they'll use to advance the handshake process. To start, @@ -295,13 +300,13 @@ does indeed know the public key of the responder. Bitcoin's compressed format: * `h = SHA-256(h || ls.pub.serializeCompressed())` -==== Handshake Acts +===== Handshake Acts After the initial handshake initialization, we can begin the actual execution of the handshake process. The Brontide handshake is compromised of a series of three messages sent between the initiator and responder, hence referred to as "acts". As each act is a single message sent between the parties, a handshake -is completed in a total of 1.5 round trips (0.5 for each act). +is completed in a total of 1.5 round trips (0.5 for each act). The first act completes the initial portion of the incremental Triple Diffie Hellman key exchange (using a new ephemeral key generated by the initiator), @@ -313,7 +318,7 @@ act, the initiator transmits their long-term static public key to the responder, and executes the final DH operation to mix that into the final resulting shared secret. -===== Act One +====== Act One ``` -> e, es @@ -374,7 +379,7 @@ and 16 bytes for the `poly1305` tag. * The received ciphertext is mixed into the handshake digest. This step serves to ensure the payload wasn't modified by a MITM. -===== Act Two +====== Act Two ``` <- e, ee @@ -432,7 +437,7 @@ for the `poly1305` tag. * The received ciphertext is mixed into the handshake digest. This step serves to ensure the payload wasn't modified by a MITM. -===== Act Three +====== Act Three ``` -> s, se @@ -502,7 +507,7 @@ construction, and 16 bytes for a final authenticating tag. 10. `rn = 0, sn = 0` * The sending and receiving nonces are initialized to 0. -==== Transport Message Encryption +===== Transport Message Encryption At the conclusion of Act Three, both sides have derived the encryption keys, which will be used to encrypt and decrypt messages for the remainder of the @@ -545,7 +550,7 @@ The structure of packets on the wire resembles the following: The prefixed message length is encoded as a 2-byte big-endian integer, for a total maximum packet length of `2 + 16 + 65535 + 16` = `65569` bytes. -===== Encrypting and Sending Messages +====== Encrypting and Sending Messages In order to encrypt and send a Lightning message (`m`) to the network stream, given a sending key (`sk`) and a nonce (`sn`), the following steps are @@ -566,7 +571,7 @@ completed: * The nonce `sn` MUST be incremented after this step. 5. Send `lc || c` over the network buffer. -===== Receiving and Decrypting Messages +====== Receiving and Decrypting Messages In order to decrypt the _next_ message in the network stream, the following steps are completed: @@ -583,7 +588,7 @@ steps are completed: plaintext packet `p`. * The nonce `rn` MUST be incremented after this step. -==== Lightning Message Key Rotation +===== Lightning Message Key Rotation Changing keys regularly and forgetting previous keys is useful to prevent the decryption of old messages, in the case of later key leakage (i.e. backwards diff --git a/contrib/payment-requests.asciidoc b/appendix_payment_requests.asciidoc similarity index 93% rename from contrib/payment-requests.asciidoc rename to appendix_payment_requests.asciidoc index d3aa5d4..0810a12 100644 --- a/contrib/payment-requests.asciidoc +++ b/appendix_payment_requests.asciidoc @@ -1,8 +1,13 @@ -= Lightning Payment Requests +[appendix] +[[invoices]] +[[lightning_payment_requests]] +== Lightning Payment Requests -== Intro +In this chapter we will look at _Lightning Payment Requests_ or as they are more commonly known _Lightning Invoices_. -As we've learned in prior chapters, minimally two prides of data are required +=== Introduction + +As we've learned in prior chapters, minimally two pieces of data are required to complete a Lightning payment: a payment hash, and a destination. As `SHA-256` is used in the Lightning Network to implement HTLCs, this information requires 32-bytes in order to communicate. Destinations on the other hand are @@ -15,10 +20,10 @@ to complete a payment from receiver to sender. In practice, more than just the payment hash and destination are communicated in a payment request in order to make the encoding more fully feature. -== Lightning Payment Requests vs Bitcoin Addresses +=== Lightning Payment Requests vs Bitcoin Addresses A commonly asked question when people first encounter a Lightning Payment -request is: why can't a normal static address format be used instead? +request is: why can't a normal static address format be used instead? In order to answer this question, one must first internalize how Lightning differs from base layer Bitcoin as a payment method. Compared to a Bitcoin @@ -27,7 +32,7 @@ address which may be used to make a potentially unbounded number of payments payment request should only ever be used *once*. This is due to the fact that sending a payment to a Bitcoin address essentially uses a public key cryptosystem to "encode" the payment in a manner that only the true "owner" of -that Bitcoin address can redeem it. +that Bitcoin address can redeem it. In contrast, in order to complete a Lightning payment, the recipient must reveal a "secret" to the entire payment route including the sender. This can be @@ -40,7 +45,7 @@ been reveled, all nodes in the path will keep it around _forever_, then rather than forward the HTLC in order to collect a routing fee if the payment is completed, they can simply _settle_ the payment at that instance and gain the entire payment amount in return. As a result, it's unsafe to ever use a payment -request more than once. +request more than once. As we'll see later in the book, there exist new variants of the original Lightning Payment request that allow the sender to -reuse them as many times as @@ -51,7 +56,7 @@ a mechanism that allows a sender to typically request a new payment request from the receiver, then an interactive protocol can be used in order to allow a degree of payment request re-use. -== BOLT 11: Lightning Payment Request Serialization & Interpretation +=== BOLT 11: Lightning Payment Request Serialization & Interpretation In this section, we'll describe the mechanism used to encode the set of information required to complete a payment on the Lightning Network. As @@ -60,16 +65,16 @@ information required to complete a payment. However in practice, more information such as time-lock information, payment request expiration, and possibly an on-chain fallback address are also communicated. -=== Payment Request Encoding in Practice +==== Payment Request Encoding in Practice First, let's examine what a real payment request looks like in practice. The following is a valid payment request that could have been used to complete a -payment on the mainnet Lightning Network at time it was created: +payment on the mainnet Lightning Network at time it was created: ``` lnbc2500u1pvjluezpp5qqqsyqcyq5rqwzqfqqqsyqcyq5rqwzqfqqqsyqcyq5rqwzqfqypqdq5xysxxatsyp3k7enxv4jsxqzpuaztrnwngzn3kdzw5hydlzf03qdgm2hdq27cqv3agm2awhz5se903vruatfhq77w3ls4evs3ch9zw97j25emudupq63nyw24cg27h2rspfj9srp ``` -=== The Human Readable Prefix +==== The Human Readable Prefix Looking at the string, we can tease out a portion that we can parse with our eyes, while the rest of it just looks like a random set of strings. The part @@ -120,7 +125,7 @@ A full list of the currently defined multipliers is a follows: -=== Bech32 & the Data Segment +==== Bech32 & the Data Segment If the "unreadable" portion of looks familiar, then that's because it uses the very same encoding scheme as segwit compatible Bitcoin addresses use today, @@ -128,7 +133,7 @@ namely `bech32`. Describing the `bech32` encoding scheme is outside the scope of this chapter. In brief, it's a sophisticated way to encode short strings that has very good error correction as well as detection properties. -The data portion can be separated into 3 sections: +The data portion can be separated into 3 sections: * The timestamp. * Zero or more tagged key-value pairs. @@ -143,7 +148,7 @@ Similar to the TLV format we learned about in Chapter XXX, the BOLT 11 invoice format uses a series of extensible key-value pairs to encode information needed to satisfy a payment. As key-value pairs are used, it's easy for add new values in the future if a new payment type or additional -requirement/functionality is introduced. +requirement/functionality is introduced. Finally a signature is included ed that covers the entire invoice signed by the destination of the payment. This signature allows the sender to verify that the @@ -155,7 +160,7 @@ allows public key extraction. When verifying the signature, the verifies extracts the public key, then verifies that against the public key included in the invoice. -==== Tagged Invoice Fields +===== Tagged Invoice Fields The tagged invoice fields are encoded in the main "body" of the invoice. These fields represent different key=value pairs that express either additional @@ -163,7 +168,7 @@ information that may help complete the payment, or information which is _required_ to complete the payment. As a slight variant of `bech32` is utilized, each of these fields are actually in the "base 5" domain. -A given tag field is comprised of 3 components: +A given tag field is comprised of 3 components: * The `type` of the field (5 bits). * The `length` of the data of the field (10 bits) diff --git a/contrib/wire_protocol.asciidoc b/appendix_wire_protocol.asciidoc similarity index 96% rename from contrib/wire_protocol.asciidoc rename to appendix_wire_protocol.asciidoc index 461a000..362ad5c 100644 --- a/contrib/wire_protocol.asciidoc +++ b/appendix_wire_protocol.asciidoc @@ -1,6 +1,8 @@ -= Wire Protocol: Framing & Extensibility +[appendix] +[[wire_protocol]] +== Wire Protocol: Framing & Extensibility -== Intro +=== Introduction In this chapter, we'll dive into the wire protocol of the Lightning network, and also cover all the various extensibility levers that have been built into @@ -10,7 +12,7 @@ to being able to write a custom wire protocol parser, a reader of this chapter will gain a deep understanding with respect of the various upgrade mechanisms that have been built into the protocol. -== Wire Framing +=== Wire Framing First, we being by describing the high level structure of the wire _framing_ within the protocol. When we say framing, we mean the way that the bytes are @@ -30,7 +32,7 @@ framing, we assume the encryption layer has already been stripped away (when decoding), or that we haven't yet encrypted the set of bytes before we send them on the wire (encoding). -=== High-Level Wire Framing +==== High-Level Wire Framing With that said, we're ready to being describe the high-level schema used to encode messages on the wire: @@ -58,7 +60,7 @@ nodes are able to provide information in the wire messages that older nodes feature combined with a very flexible wire message extensibility format also allows the protocol to achieve _forwards_ compatibility as well. -=== Type Encoding +==== Type Encoding With this high level background provided, we'll now start at the most primitive layer: parsing primitive types. In addition to encoding integers, the Lightning @@ -76,39 +78,39 @@ high-level routine used to encode/decode the type. .High-level message types [options="header"] |================================================================================ -| High Level Type | Framing | Comment -| `node_alias` | A 32-byte fixed-length byte slice. | When decoding, reject if contents are not a valid UTF-8 string. -| `channel_id` | A 32-byte fixed-length byte slice that maps an outpoint to a 32 byte value. | Given an outpoint, one can convert it to a `channel_id` by taking the txid of the outpoint and XOR'ing it with the index (interpreted as the lower 2 bytes). -| `short_chan_id` | An unsigned 64-bit integer (`uint64`) | Composed of the block height (24 bits), transaction index (24 bits), and output index (16 bits) packed into 8 bytes. -| `milli_satoshi` | An unsigned 64-bit integer (`uint64`) | Represents 1000th of a satoshi. -| `satoshi` | An unsigned 64-bit integer (`uint64`) | The based unit of bitcoin. -| `satoshi` | An unsigned 64-bit integer (`uint64`) | The based unit of bitcoin. -| `pubkey` | An secp256k1 public key encoded in _compressed_ format, occupying 33 bytes. | Occupies a fixed 33-byte length on the wire. +| High Level Type | Framing | Comment +| `node_alias` | A 32-byte fixed-length byte slice. | When decoding, reject if contents are not a valid UTF-8 string. +| `channel_id` | A 32-byte fixed-length byte slice that maps an outpoint to a 32 byte value. | Given an outpoint, one can convert it to a `channel_id` by taking the txid of the outpoint and XOR'ing it with the index (interpreted as the lower 2 bytes). +| `short_chan_id` | An unsigned 64-bit integer (`uint64`) | Composed of the block height (24 bits), transaction index (24 bits), and output index (16 bits) packed into 8 bytes. +| `milli_satoshi` | An unsigned 64-bit integer (`uint64`) | Represents 1000th of a satoshi. +| `satoshi` | An unsigned 64-bit integer (`uint64`) | The based unit of bitcoin. +| `satoshi` | An unsigned 64-bit integer (`uint64`) | The based unit of bitcoin. +| `pubkey` | An secp256k1 public key encoded in _compressed_ format, occupying 33 bytes. | Occupies a fixed 33-byte length on the wire. | `sig` | An ECDSA signature of the secp256k1 Elliptic Curve. | Encoded as a _fixed_ 64-byte byte slice, packed as `R \|\| S` | `uint8` | An 8-bit integer. | | `uint16` | A 16-bit integer. | | `uint64` | A 64-bit integer. | -| `[]byte` | A variable length byte slice. | Prefixed with a 16-bit integer denoting the length of the bytes. -| `color_rgb` | RGB color encoding. | Encoded as a series if 8-bit integers. -| `net_addr` | The encoding of a network address. | Encoded with a 1 byte prefix that denotes the type of address, followed by the address body. +| `[]byte` | A variable length byte slice. | Prefixed with a 16-bit integer denoting the length of the bytes. +| `color_rgb` | RGB color encoding. | Encoded as a series if 8-bit integers. +| `net_addr` | The encoding of a network address. | Encoded with a 1 byte prefix that denotes the type of address, followed by the address body. |================================================================================ In the next section, we'll describe the structure of each of the wire messages including the prefix type of the message along with the contents of its message body. -=== Type Length Value (TLV) Message Extensions +==== Type Length Value (TLV) Message Extensions Earlier in this chapter we mentioned that messages can be up to 65 KB in size, and if while parsing a messages, extra bytes are left over, then those bytes are to be _ignored_. At an initial glance, this requirement may appear to be somewhat arbitrary, however upon close inspection it's actually the case that -this requirement allows for de-coupled de-synchronized evolution of the Lighting +this requirement allows for de-coupled de-synchronized evolution of the Lighting Protocol itself. We'll opine further upon this notion towards the end of the chapter. First, we'll turn our attention to exactly what those "extra bytes" at the end of a message can be used for. -==== The Protcol Buffer Message Format +===== The Protcol Buffer Message Format The Protocol Buffer (protobuf) message serialization format started out as an internal format used at Google, and has blossomed into one of the most popular @@ -130,7 +132,7 @@ there're types/fields that it doesn't understand, then it simply _ignores_ them. This allows old clients and new clients to _co-exist_, as all clients can parse _some_ portion of the newer message format. -==== Forwards & Backwards Compatibility +===== Forwards & Backwards Compatibility Protobufs are extremely popular amongst developers as they have built in support for both _forwards_ and _backwards_ compatibility. Most developers are @@ -148,7 +150,7 @@ clients that don't update can still use Bitcoin, and if they encounters any transactions they don't understand, then they simply ignore them as their funds aren't using those new features. -==== Lighting's Protobuf Inspired Message Extension Format: `TLV` +===== Lighting's Protobuf Inspired Message Extension Format: `TLV` In order to be able to upgrade messages in both a forwards and backwards compatible manner, in addition to feature bits (more on that later), the LN @@ -167,7 +169,7 @@ In comparison lnd's implementation of the TLV message format weighs in at only With the necessary background presented, we're now ready to describe the TLV format in detail. A TLV message extension is said to be a _stream_ of individual TLV records. A single TLV record has three components: the type of -the record, the length of the record, and finally the opaque value of the +the record, the length of the record, and finally the opaque value of the record: * `type`: An integer representing the name of the record being encoded. @@ -192,7 +194,7 @@ size of the integer to be encoded. * If the value is _less than_ `0xfd` (`253`): * Then the discriminant isn't really used, and the encoding is simply the - integer itself. + integer itself. * This value allows us to encode very small integers with no additional overhead @@ -230,7 +232,7 @@ acceptable within teh context of Lighting, was many messages contain a signature of the message digest. If it's possible for a message to be encoded in two different ways, then it would be possible to break the authentication of a signature inadvertently by re-encoding a message using a slightly different -set of bytes on the wire. +set of bytes on the wire. In order to ensure that all encoded messages are canonical, the following constraints are defined when encoding: @@ -240,7 +242,7 @@ constraints are defined when encoding: * All records must _minimally encode_ the `type` and `length` fields. In orther woards, the smallest BigSIze representation for an integer MUST be - used at all times. + used at all times. * Each `type` may only appear _once_ within a given TLV stream. @@ -251,7 +253,7 @@ chapter cone we talked about how the Lighting Protocol is upgraded in practice and in theory. -=== Wire Messages +==== Wire Messages In this section, well outline the precise structure of each of the wire messages within the protocol. We'll do so in two parts: first we'll enumerate @@ -264,18 +266,18 @@ First, we'll lead with an enumeration of all the currently defined types: .Message Types [options="header"] |============================================================================== -| Type Integer | Message Name | Category -| 16 | `init` | Connection Establishment -| 17 | `error` | Error Communication -| 18 | `ping` | Connection Liveness +| Type Integer | Message Name | Category +| 16 | `init` | Connection Establishment +| 17 | `error` | Error Communication +| 18 | `ping` | Connection Liveness | 19 | `pong` | Connection Liveness | 32 | `open_channel` | Channel Funding | 33 | `accept_channel` | Channel Funding | 34 | `funding_created` | Channel Funding | 35 | `funding_signed` | Channel Funding | 36 | `funding_locked` | Channel Funding + Channel Operation -| 38 | `shutdown` | Channel Closing -| 39 | `closing_signed` | Channel Closing +| 38 | `shutdown` | Channel Closing +| 39 | `closing_signed` | Channel Closing | 128 | `update_add_htlc` | Channel Operation | 130 | `update_fulfill_hltc` | Channel Operation | 131 | `update_fail_htlc` | Channel Operation @@ -283,7 +285,7 @@ First, we'll lead with an enumeration of all the currently defined types: | 133 | `revoke_and_ack` | Channel Operation | 134 | `update_fee` | Channel Operation | 135 | `update_fail_malformed_htlc` | Channel Operation -| 136 | `channel_reestablish` | Channel Operation +| 136 | `channel_reestablish` | Channel Operation | 256 | `channel_announcement` | Channel Announcement | 257 | `node_announcement` | Channel Announcement | 258 | `channel_update` | Channel Announcement @@ -297,7 +299,7 @@ First, we'll lead with an enumeration of all the currently defined types: In the above table, the `Category` field allows us to quickly categonize a message based on its functionality within the protocol itself. At a high level, -we place a message into one of 8 (non exhaustive) buckets including: +we place a message into one of 8 (non exhaustive) buckets including: * *Connection Establishment*: Sent when a peer to peer connection is first established. Also used in order to negotiate the set of _feature_ supported @@ -307,7 +309,7 @@ we place a message into one of 8 (non exhaustive) buckets including: protocol level errors to each other. * *Connection Liveness*: Used by peers to check that a given transport - connection is still live. + connection is still live. * *Channel Funding*: Used by peers to create a new payment channel. This process is also known as the channel funding process. @@ -330,7 +332,7 @@ this roadmap laid out, we'll now visit each message category in order to define the precise structure and semantics of all defined messages within the LN protocol. -==== Connection Establishment Messages +===== Connection Establishment Messages Messages in this category are the very first message sent between peers once they establish a transport connection. At the time of writing of this chapter, @@ -339,12 +341,12 @@ The `init` message is sent by _both_ sides of the connection once it has been first established. No other messages are to be sent before the `init` message has been sent by both parties. -The structure of the `init` message is defined as follows: +The structure of the `init` message is defined as follows: `init` message: * type: `16` - * fields: + * fields: * `uint16`: `global_features_len` * `global_features_len*byte`: `global_features` * `uint16`: `features_len` @@ -368,12 +370,12 @@ they're used later in the chapter. An `init` message is then examined by a peer in order to determine if the connection is well defined based on the set of optional and required feature -bits advertised by both sides. +bits advertised by both sides. An optional feature means that a peer knows about a feature, but they don't consider it critical to the operation of a new connection. An example of one would be something like the ability to understand the semantics of a newly -added field to an existing message. +added field to an existing message. On the other hand, required feature indicate that if the other peer doesn't know about the feature, then the connection isn't well defined. An example of @@ -381,7 +383,7 @@ such a feature would be a theoretical new channel type within the protocol: if your peer doesn't know of this feature, they you don't want to keep the connection as they're unable to open your new preferred channel type. -==== Error Communication Messages +===== Error Communication Messages Messages in this category are used to send connection level errors between two peers. As we'll see later, another type of error exists in the protocol: an @@ -392,7 +394,7 @@ broadcast the latest signed commitment) The sole message in this category is the `error` message: * type: `17` - * fields: + * fields: * `channel_id`: `chan_id` * `uint16`: `data_len` * `data_len*byte`: `data` @@ -408,14 +410,14 @@ have a channel with may indicate that the channel cannot continue without manual intervention, so the only option at that point is to force close the channel by broadcasting the latest commitment state of the channel. -==== Connection Liveness +===== Connection Liveness Messages in this section are used to probe to determine if a connection is still live or not. As the LN protocol somewhat abstracts over the underlying transport being used to transmit the messages, a set of protocol level `ping` and `pong` messages are defined. -First, the `ping` message: +First, the `ping` message: * type: `18` * fields: @@ -430,7 +432,7 @@ Next it's companion, the `pong` message: * `uint16`: `pong_body_len` * `ping_body_len*bytes`: `pong_body` -A `ping` message can be sent by either party at any time. +A `ping` message can be sent by either party at any time. The `ping` message includes a `num_pong_bytes` field that is used to instruct the receiving node with respect to how large the payload it sends in its `pong` @@ -449,7 +451,7 @@ default the LN uses an _encrypted_ transport, so a passive network monitor cannot read the plaintext bytes, thus only has timing and packet sizes to go off of. -==== Channel Funding +===== Channel Funding As we go on, we enter into the territory of the core messages that govern the functionality and semantics of the Lightning Protocol. In this section, we'll @@ -494,7 +496,7 @@ commitment transaction. At the time of writing of this chapter, a single TLV record is defined within the set of optional TLV records that may be appended to the end of a defined -message: +message: * type: 0 * data: `upfront_shutdown_script` @@ -543,7 +545,7 @@ channel. In response, the initiator will send the `funding_created` message: * type: `34` - * fields: + * fields: * `32*byte`: `temp_chan_id` * `32*byte`: `funding_txid` * `uint16`: `funding_output_index` @@ -588,7 +590,7 @@ Once the funding transaction obtains a `minimum_depth` number of confirmations, then the `funding_locked` message is to be sent by both sides. Only after this message has been received, and sent can the channel being to be used. -==== Channel Closing +===== Channel Closing * type: `38` * fields: @@ -701,7 +703,7 @@ transactions: * `uint32`: `feerate_per_kw` This message can only be sent by the initiator of the channel they're the ones -that will pay for the commitment fee of the channel as along as it's open. +that will pay for the commitment fee of the channel as along as it's open. The `update_fail_malformed_htlc` is sent to remove a corrupted HTLC: @@ -720,7 +722,7 @@ then it won't be able to decrypt the packet. As a result it also can't properly forward the HTLC, therefore it'll send this message to signify that the HTLC has been corrupted somewhere along the route back to the sender. -==== Channel Announcement +===== Channel Announcement Messages in this category are used to announce components of the Channel Graph authenticated data structure to the wider network. The Channel Graph has a @@ -793,7 +795,7 @@ an active channel edge within the Channel graph: * `uint64`: `htlc_minimum_msat` * `uint32`: `fee_base_msat` * `uint32`: `fee_proportional_millionths` - * `uint16`: `htlc_maximum_msat` + * `uint16`: `htlc_maximum_msat` In addition to being able to enable/disable a channel this message allows a node to update it's routing fees as well as other fields that shape the type of @@ -815,7 +817,7 @@ advertise their channel to the network, then they'll each send the `announce_signatures` message which allows both sides to emplace the 4 signatures required to generate a `announce_signatures` message. -==== Channel Graph Syncing +===== Channel Graph Syncing The `query_short_chan_ids` allows a peer to obtain the channel information related to a series of short channel IDs: @@ -857,7 +859,7 @@ of a channel in the chain, a node on the network can use a block height as a sort of _cursor_ to seek through the chain in order to discover a set of newly opened channels. In Chapter XXX, we'll go through the protocol peers use to sync the channel graph in more detail. - + The `reply_channel_range` message is the response to `query_channel_range` and includes the set of short channel IDs for known channels within that range: @@ -875,7 +877,7 @@ As a response to `query_channel_range`, this message sends back the set of channels that were opened within that range. This process can be repeated with the requester advancing their cursor further down the chain in order to continue syncing the Channel Graph. - + The `gossip_timestamp_range` message allows a peer to start receiving new incoming gossip messages on the network: * type: `265: @@ -890,7 +892,7 @@ also set the `first_timestamp` and `timestamp_range` fields if they wish to receive a backlog of updates they may have missed while they were down. -== Feature Bits & Protocol Extensibility +=== Feature Bits & Protocol Extensibility As the Lighting Network is a decentralized system, no one entity can enforce a protocol change or modification upon all the users of the system. This @@ -912,7 +914,7 @@ extensibility mechanisms within the network which can be used to upgrade the network partially or fully in a decoupled, desynchronized, decentralized manner. -=== Feature Bits as an Upgrade Discoverability Mechanism +==== Feature Bits as an Upgrade Discoverability Mechanism An astute reader may have noticed the various locations that "feature bits" are included within the Lightning Protocol. A "feature bit" is a bitfield that can @@ -971,13 +973,13 @@ not. The feature bits within the `init` message all peers to understand kif they can maintain a connection, and also which features are negotiated for the lifetime of a given connection. -=== Utilizing TLV Records for Forwards+Backwards Compatibility +==== Utilizing TLV Records for Forwards+Backwards Compatibility As we learned earlier in the chapter, Type Length Value, or TLV records can be used to extend messages in a forwards and backwards compatible manner. Overtime, these records have been used to _extend_ existing messages without breaking the protocol by utilizing the "undefined" area within a message beyond -that set of known bytes. +that set of known bytes. As an example, the original Lighting Protocol didn't have a concept of the _largest_ HTLC that could traverse through a channel as dictated by a routing @@ -986,7 +988,7 @@ message to phase in such a concept over time. Peers that held a `channel_update` that set such a field but didn't even know the upgrade existed where unaffected by the change, but may see their HTLCs rejected if they are beyond the said limit. Newer peers on the other hand are able to parse, verify -and utilize the new field at will. +and utilize the new field at will. Those familiar with the concept of soft-forks in Bitcoin may now see some similarities between the two mechanism. Unlike Bitcoin consensus-level @@ -996,7 +998,7 @@ network need to understand new upgrade in order to start utilizing it without any permission. Commonly these tow peers may be the receiver and sender of a payment, or it may the initiator and responder of a new payment channel. -=== A Taxonomy of Upgrade Mechanisms +==== A Taxonomy of Upgrade Mechanisms Rather than there being a single widely utilized upgrade mechanism within the network (such as soft forks for base layer Bitcoin), there exist a wide @@ -1004,7 +1006,7 @@ gradient of possible upgrade mechanisms within the Lighting Network. In this section, we'll enumerate the various upgrade mechanism within the network, and provide a real-world example of their usage in the past. -==== Internal Network Upgrades +===== Internal Network Upgrades We'll start with the upgrade type that requires the most extra protocol-level coordination: internal network upgrades. An internal network upgrade is @@ -1035,7 +1037,7 @@ are required to forward the payment. However, if a new upgrade type instead changed the _HTLC_ format, then the entire path would need to be upgraded, otherwise the payment wouldn't be able to be fulfilled. -==== End to End Upgrades +===== End to End Upgrades To contrast the internal network upgrade, in this section we'll describe the _end to end_ network upgrade. This upgrade type differs from the internal @@ -1065,7 +1067,7 @@ end encrypted, this payment type was safe, since none of the intermediate nodes are able to fully unwrap the onion to uncover the payment pre-image that corresponded to that payment hash. -==== Channel Construction Level Updates +===== Channel Construction Level Updates The final broad category of updates within the network are those that happen at the channel construction level, but which don't modify the structure of the