Appendices and README update

3 years ago · e21944239c
parent 78da6f5b50
commit e21944239c
4 changed files with 122 additions and 105 deletions
--- a/README.md
+++ b/README.md
@ -13,7 +13,9 @@ The book is suitable for technical readers with an understanding of the fundamen

 ## Status

-The current status of the book is "RELEASE PREP". See below for status of specific chapters and read the contribution guide to learn how and where to contribute.
+The current status of the book is "IN PRODUCTION": ONLY COPY-EDIT CONTRIBUTIONS ARE ACCEPTED.
+
+See below for status of specific chapters and read the contribution guide to learn how and where to contribute.

 ### Legend

@ -45,10 +47,13 @@ The current status of the book is "RELEASE PREP". See below for status of specif
 | [LN Security and Privacy](13_security_privacy_ln.asciidoc) | ################ | :heavy_check_mark: |
 | APPENDICES | APPENDICES | APPENDICES |
 | [A1 - Bitcoin Fundamentals Review](appendix-bitcoin-fundamentals-review.asciidoc) | ########### | :heavy_check_mark: |
-| [An - License Notices](appendix_license_notices.asciidoc) | # | :heavy_check_mark: |
+| [A2 - Payment Requests (Invoices)](appendix_payment_requests.asciidoc) | #### | :heavy_check_mark: |
+| [A3 - Wire Protocol](appendix_wire_protocol.asciidoc) | ################# | :heavy_check_mark: |
+| [A4 - Encrypted Message Transport (Brontide)](appendix_encrypted_transport.asciidoc) | ########## | :heavy_check_mark: |
+| [A5 - License Notices](appendix_license_notices.asciidoc) | # | :heavy_check_mark: |


-Total Word Count: 110718
+Total Word Count: 125288

 Target Word Count: 100,000-120,000

--- a/appendix_encrypted_transport.asciidoc
+++ b/appendix_encrypted_transport.asciidoc
@ -1,6 +1,11 @@
-= Brontide: Lightning's Encrypted Message Transport
+[appendix]
+[[brontide]]
+[[encrypted_message_transport]]
+== Lightning's Encrypted Message Transport (Brontide)

-== Intro
+In this chapter we will review the Lightning Network's _Encrypted Message Transport_, also known as the _Brontide Protocol_, which allows peers to establish end-to-end encrypted communication, authentication and integrity checking.
+
+=== Introduction

 Unlike the vanilla Bitcoin P2P network, every node in the Lightning Network is
 identified by a unique public key which serves as it identity. By default, this
@ -18,7 +23,7 @@ custom encrypted message transport Lightning uses, commonly referred to as
 "Brontide" (more on that later) can be dropped into any context that requires
 encrypted communication between two parties.

-== The Channel Graph as Decentralized Public Key Infrastructure
+=== The Channel Graph as Decentralized Public Key Infrastructure

 As we learned in the chapter on multi-hop forwarding, very node has a long-term
 identity that is used as the identifier for a vertex during path finding and
@ -48,7 +53,7 @@ Lightning network is able to significantly simply its encrypted transport
 protocol as it doesn't need to deal with all the complexities that come along
 with TLS, the Transport Layer Security protocol.

-== Why Not TLS?
+=== Why Not TLS?

 Readers familiar with the TLS system may be wondering at this point: why wasn't
 TLS used in spite of the drawbacks of the existing PKI system? It is indeed a
@ -80,7 +85,7 @@ to continual scrutiny of the TLS protocol over decades. In a way, the existence
 of Noise allows the community to effective "start over", with a more compact,
 simplified protocol that retains all the added benefits of TLS.

-== The Noise Protocol Framework
+=== The Noise Protocol Framework

 The Noise Protocol Framework is a modern, extensible, and flexible message
 encryption protocol designed by the creators of the Signal protocol. The Signal
@ -114,7 +119,7 @@ Lightning Network, the flavor of Noise use will be referred to from here on as
 "Brontide". A brontide is a low billowing noise, similar to what one would hear
 during a thunderstorm when very far away.

-=== Noise Protocol Handshakes
+==== Noise Protocol Handshakes

 The Noise protocol is extremely flexible in that it advertises several
 handshakes, each with different security and privacy properties for a would be
@ -129,7 +134,7 @@ handshake. Instead, a clever series of Elliptic-Curve Diffie-Hellman (ECDH) and
 Message Authentication Code (MAC) checks are used to authenticate the
 responder.

-=== Handshake Notation & Protocol Flow
+==== Handshake Notation & Protocol Flow

 Each handshakes typically consist of several steps. At each step some
 (possibly) encrypted material is sent to the opposite party, an ECDH (or
@ -148,9 +153,9 @@ refer to a new ephemeral key. ECDH operations between two keys are notated as
 the concatenation of two keys. As an example, `ee` represents an ECDH operation
 between two ephemeral keys.

-== Brontide: Lightning's P2P Encryption
+=== Brontide: Lightning's P2P Encryption

-=== High-Level Overview
+==== High-Level Overview

 Using the notation laid out earlier, we can succinctly describe the `Noise_XK`
 as follows:
@ -196,7 +201,7 @@ variant, the ASCII string is hashed into a digest, which is used to initialize
 the starting handshake state. In the context of Brontide, the ASCII string
 describing the protocol is: `Noise_XK_secp256k1_ChaChaPoly_SHA256`.

-=== Brontide: A Handshake in Three Acts
+==== Brontide: A Handshake in Three Acts

 The handshake portion of Brontide can be see prated into three distinct "acts".
 The entire handshake takes 1.5 round trips between the initiator and responder.
@ -263,7 +268,7 @@ handshake as:

  * `a || b` denotes the concatenation of two byte strings `a` and `b`

-==== Handshake Session State Initialization
+===== Handshake Session State Initialization

 Before starting the handshake process, both sides need to initialize the
 starting state that they'll use to advance the handshake process. To start,
@ -295,7 +300,7 @@ does indeed know the public key of the responder.
   Bitcoin's compressed format:
   * `h = SHA-256(h || ls.pub.serializeCompressed())`

-==== Handshake Acts
+===== Handshake Acts

 After the initial handshake initialization, we can begin the actual execution
 of the handshake process. The Brontide handshake is compromised of a series of
@ -313,7 +318,7 @@ act, the initiator transmits their long-term static public key to the
 responder, and executes the final DH operation to mix that into the final
 resulting shared secret.

-===== Act One
+====== Act One

 ```
    -> e, es
@ -374,7 +379,7 @@ and 16 bytes for the `poly1305` tag.
     * The received ciphertext is mixed into the handshake digest. This step serves
       to ensure the payload wasn't modified by a MITM.

-===== Act Two
+====== Act Two

 ```
   <- e, ee
@ -432,7 +437,7 @@ for the `poly1305` tag.
     * The received ciphertext is mixed into the handshake digest. This step serves
       to ensure the payload wasn't modified by a MITM.

-===== Act Three
+====== Act Three

 ```
   -> s, se
@ -502,7 +507,7 @@ construction, and 16 bytes for a final authenticating tag.
 10. `rn = 0, sn = 0`
     * The sending and receiving nonces are initialized to 0.

-==== Transport Message Encryption
+===== Transport Message Encryption

 At the conclusion of Act Three, both sides have derived the encryption keys, which
 will be used to encrypt and decrypt messages for the remainder of the
@ -545,7 +550,7 @@ The structure of packets on the wire resembles the following:
 The prefixed message length is encoded as a 2-byte big-endian integer, for a
 total maximum packet length of `2 + 16 + 65535 + 16` = `65569` bytes.

-===== Encrypting and Sending Messages
+====== Encrypting and Sending Messages

 In order to encrypt and send a Lightning message (`m`) to the network stream,
 given a sending key (`sk`) and a nonce (`sn`), the following steps are
@ -566,7 +571,7 @@ completed:
    * The nonce `sn` MUST be incremented after this step.
 5. Send `lc || c` over the network buffer.

-===== Receiving and Decrypting Messages
+====== Receiving and Decrypting Messages

 In order to decrypt the _next_ message in the network stream, the following
 steps are completed:
@ -583,7 +588,7 @@ steps are completed:
    plaintext packet `p`.
    * The nonce `rn` MUST be incremented after this step.

-==== Lightning Message Key Rotation
+===== Lightning Message Key Rotation

 Changing keys regularly and forgetting previous keys is useful to prevent the
 decryption of old messages, in the case of later key leakage (i.e.  backwards
--- a/appendix_payment_requests.asciidoc
+++ b/appendix_payment_requests.asciidoc
@ -1,8 +1,13 @@
-= Lightning Payment Requests
+[appendix]
+[[invoices]]
+[[lightning_payment_requests]]
+== Lightning Payment Requests

-== Intro
+In this chapter we will look at _Lightning Payment Requests_ or as they are more commonly known _Lightning Invoices_.

-As we've learned in prior chapters, minimally two prides of data are required
+=== Introduction
+
+As we've learned in prior chapters, minimally two pieces of data are required
 to complete a Lightning payment: a payment hash, and a destination. As
 `SHA-256` is used in the Lightning Network to implement HTLCs, this information
 requires 32-bytes in order to communicate. Destinations on the other hand are
@ -15,7 +20,7 @@ to complete a payment from receiver to sender. In practice, more than just the
 payment hash and destination are communicated in a payment request in order to
 make the encoding more fully feature.

-== Lightning Payment Requests vs Bitcoin Addresses
+=== Lightning Payment Requests vs Bitcoin Addresses

 A commonly asked question when people first encounter a Lightning Payment
 request is: why can't a normal static address format be used instead?
@ -51,7 +56,7 @@ a mechanism that allows a sender to typically request a new payment request
 from the receiver, then an interactive protocol can be used in order to allow a
 degree of payment request re-use.

-== BOLT 11: Lightning Payment Request Serialization & Interpretation
+=== BOLT 11: Lightning Payment Request Serialization & Interpretation

 In this section, we'll describe the mechanism used to encode the set of
 information required to complete a payment on the Lightning Network. As
@ -60,7 +65,7 @@ information required to complete a payment. However in practice, more
 information such as time-lock information, payment request expiration, and
 possibly an on-chain fallback address are also communicated.

-=== Payment Request Encoding in Practice
+==== Payment Request Encoding in Practice

 First, let's examine what a real payment request looks like in practice. The
 following is a valid payment request that could have been used to complete a
@ -69,7 +74,7 @@ payment on the mainnet Lightning Network at time it was created:
 lnbc2500u1pvjluezpp5qqqsyqcyq5rqwzqfqqqsyqcyq5rqwzqfqqqsyqcyq5rqwzqfqypqdq5xysxxatsyp3k7enxv4jsxqzpuaztrnwngzn3kdzw5hydlzf03qdgm2hdq27cqv3agm2awhz5se903vruatfhq77w3ls4evs3ch9zw97j25emudupq63nyw24cg27h2rspfj9srp
 ```

-=== The Human Readable Prefix
+==== The Human Readable Prefix

 Looking at the string, we can tease out a portion that we can parse with our
 eyes, while the rest of it just looks like a random set of strings. The part
@ -120,7 +125,7 @@ A full list of the currently defined multipliers is a follows:



-=== Bech32 & the Data Segment
+==== Bech32 & the Data Segment

 If the "unreadable" portion of looks familiar, then that's because it uses the
 very same encoding scheme as segwit compatible Bitcoin addresses use today,
@ -155,7 +160,7 @@ allows public key extraction. When verifying the signature, the verifies
 extracts the public key, then verifies that against the public key included in
 the invoice.

-==== Tagged Invoice Fields
+===== Tagged Invoice Fields

 The tagged invoice fields are encoded in the main "body" of the invoice. These
 fields represent different key=value pairs that express either additional
--- a/appendix_wire_protocol.asciidoc
+++ b/appendix_wire_protocol.asciidoc
@ -1,6 +1,8 @@
-= Wire Protocol: Framing & Extensibility
+[appendix]
+[[wire_protocol]]
+== Wire Protocol: Framing & Extensibility

-== Intro
+=== Introduction

 In this chapter, we'll dive into the wire protocol of the Lightning network,
 and also cover all the various extensibility levers that have been built into
@ -10,7 +12,7 @@ to being able to write a custom wire protocol parser, a reader of this chapter
 will gain a deep understanding with respect of the various upgrade mechanisms
 that have been built into the protocol.

-== Wire Framing
+=== Wire Framing

 First, we being by describing the high level structure of the wire _framing_
 within the protocol. When we say framing, we mean the way that the bytes are
@ -30,7 +32,7 @@ framing, we assume the encryption layer has already been stripped away (when
 decoding), or that we haven't yet encrypted the set of bytes before we send
 them on the wire (encoding).

-=== High-Level Wire Framing
+==== High-Level Wire Framing

 With that said, we're ready to being describe the high-level schema used to
 encode messages on the wire:
@ -58,7 +60,7 @@ nodes are able to provide information in the wire messages that older nodes
 feature combined with a very flexible wire message extensibility format also
 allows the protocol to achieve _forwards_ compatibility as well.

-=== Type Encoding
+==== Type Encoding

 With this high level background provided, we'll now start at the most primitive
 layer: parsing primitive types. In addition to encoding integers, the Lightning
@ -97,7 +99,7 @@ In the next section, we'll describe the structure of each of the wire messages
 including the prefix type of the message along with the contents of its message
 body.

-=== Type Length Value (TLV) Message Extensions
+==== Type Length Value (TLV) Message Extensions

 Earlier in this chapter we mentioned that messages can be up to 65 KB in size,
 and if while parsing a messages, extra bytes are left over, then those bytes
@ -108,7 +110,7 @@ Protocol itself. We'll opine further upon this notion towards the end of the
 chapter. First, we'll turn our attention to exactly what those "extra bytes" at
 the end of a message can be used for.

-==== The Protcol Buffer Message Format
+===== The Protcol Buffer Message Format

 The Protocol Buffer (protobuf) message serialization format started out as an
 internal format used at Google, and has blossomed into one of the most popular
@ -130,7 +132,7 @@ there're types/fields that it doesn't understand, then it simply _ignores_
 them. This allows old clients and new clients to _co-exist_, as all clients can
 parse _some_ portion of the newer message format.

-==== Forwards & Backwards Compatibility
+===== Forwards & Backwards Compatibility

 Protobufs are extremely popular amongst developers as they have built in
 support for both _forwards_ and _backwards_ compatibility. Most developers are
@ -148,7 +150,7 @@ clients that don't update can still use Bitcoin, and if they encounters any
 transactions they don't understand, then they simply ignore them as their funds
 aren't using those new features.

-==== Lighting's Protobuf Inspired Message Extension Format: `TLV`
+===== Lighting's Protobuf Inspired Message Extension Format: `TLV`

 In order to be able to upgrade messages in both a forwards and backwards
 compatible manner, in addition to feature bits (more on that later), the LN
@ -251,7 +253,7 @@ chapter cone we talked about how the Lighting Protocol is upgraded in practice
 and in theory.


-=== Wire Messages
+==== Wire Messages

 In this section, well outline the precise structure of each of the wire
 messages within the protocol. We'll do so in two parts: first we'll enumerate
@ -330,7 +332,7 @@ this roadmap laid out, we'll now visit each message category in order to define
 the precise structure and semantics of all defined messages within the LN
 protocol.

-==== Connection Establishment Messages
+===== Connection Establishment Messages

 Messages in this category are the very first message sent between peers once
 they establish a transport connection. At the time of writing of this chapter,
@ -381,7 +383,7 @@ such a feature would be a theoretical new channel type within the protocol: if
 your peer doesn't know of this feature, they you don't want to keep the
 connection as they're unable to open your new preferred channel type.

-==== Error Communication Messages
+===== Error Communication Messages

 Messages in this category are used to send connection level errors between two
 peers. As we'll see later, another type of error exists in the protocol: an
@ -408,7 +410,7 @@ have a channel with may indicate that the channel cannot continue without
 manual intervention, so the only option at that point is to force close the
 channel by broadcasting the latest commitment state of the channel.

-==== Connection Liveness
+===== Connection Liveness

 Messages in this section are used to probe to determine if a connection is
 still live or not. As the LN protocol somewhat abstracts over the underlying
@ -449,7 +451,7 @@ default the LN uses an _encrypted_ transport, so a passive network monitor
 cannot read the plaintext bytes, thus only has timing and packet sizes to go
 off of.

-==== Channel Funding
+===== Channel Funding

 As we go on, we enter into the territory of the core messages that govern the
 functionality and semantics of the Lightning Protocol. In this section, we'll
@ -588,7 +590,7 @@ Once the funding transaction obtains a `minimum_depth` number of confirmations,
 then the `funding_locked` message is to be sent by both sides. Only after this
 message has been received, and sent can the channel being to be used.

-==== Channel Closing
+===== Channel Closing

 * type: `38`
 * fields:
@ -720,7 +722,7 @@ then it won't be able to decrypt the packet. As a result it also can't properly
 forward the HTLC, therefore it'll send this message to signify that the HTLC
 has been corrupted somewhere along the route back to the sender.

-==== Channel Announcement
+===== Channel Announcement

 Messages in this category are used to announce components of the Channel Graph
 authenticated data structure to the wider network. The Channel Graph has a
@ -815,7 +817,7 @@ advertise their channel to the network, then they'll each send the
 `announce_signatures` message which allows both sides to emplace the 4
 signatures required to generate a `announce_signatures` message.

-==== Channel Graph Syncing
+===== Channel Graph Syncing

 The `query_short_chan_ids` allows a peer to obtain the channel information
 related to a series of short channel IDs:
@ -890,7 +892,7 @@ also set the `first_timestamp` and `timestamp_range` fields if they wish to
 receive a backlog of updates they may have missed while they were down.


-== Feature Bits & Protocol Extensibility
+=== Feature Bits & Protocol Extensibility

 As the Lighting Network is a decentralized system, no one entity can enforce a
 protocol change or modification upon all the users of the system. This
@ -912,7 +914,7 @@ extensibility mechanisms within the network which can be used to upgrade the
 network partially or fully in a decoupled, desynchronized, decentralized
 manner.

-=== Feature Bits as an Upgrade Discoverability Mechanism
+==== Feature Bits as an Upgrade Discoverability Mechanism

 An astute reader may have noticed the various locations that "feature bits" are
 included within the Lightning Protocol. A "feature bit" is a bitfield that can
@ -971,7 +973,7 @@ not. The feature bits within the `init` message all peers to understand kif
 they can maintain a connection, and also which features are negotiated for the
 lifetime of a given connection.

-=== Utilizing TLV Records for Forwards+Backwards Compatibility
+==== Utilizing TLV Records for Forwards+Backwards Compatibility

 As we learned earlier in the chapter, Type Length Value, or TLV records can be
 used to extend messages in a forwards and backwards compatible manner.
@ -996,7 +998,7 @@ network need to understand new upgrade in order to start utilizing it without
 any permission. Commonly these tow peers may be the receiver and sender of a
 payment, or it may the initiator and responder of a new payment channel.

-=== A Taxonomy of Upgrade Mechanisms 
+==== A Taxonomy of Upgrade Mechanisms

 Rather than there being a single widely utilized upgrade mechanism within the
 network (such as soft forks for base layer Bitcoin), there exist a wide
@ -1004,7 +1006,7 @@ gradient of possible upgrade mechanisms within the Lighting Network. In this
 section, we'll enumerate the various upgrade mechanism within the network, and
 provide a real-world example of their usage in the past.

-==== Internal Network Upgrades
+===== Internal Network Upgrades

 We'll start with the upgrade type that requires the most extra protocol-level
 coordination: internal network upgrades. An internal network upgrade is
@ -1035,7 +1037,7 @@ are required to forward the payment. However, if a new upgrade type instead
 changed the _HTLC_ format, then the entire path would need to be upgraded,
 otherwise the payment wouldn't be able to be fulfilled.

-==== End to End Upgrades
+===== End to End Upgrades

 To contrast the internal network upgrade, in this section we'll describe the
 _end to end_ network upgrade. This upgrade type differs from the internal
@ -1065,7 +1067,7 @@ end encrypted, this payment type was safe, since none of the intermediate nodes
 are able to fully unwrap the onion to uncover the payment pre-image that
 corresponded to that payment hash.

-==== Channel Construction Level Updates
+===== Channel Construction Level Updates

 The final broad category of updates within the network are those that happen at
 the channel construction level, but which don't modify the structure of the