From 846e589abd9501f26f037f9c5b3e1479bac1b25f Mon Sep 17 00:00:00 2001 From: Olaoluwa Osuntokun Date: Sun, 11 Apr 2021 18:53:27 -0700 Subject: [PATCH] wire_protocol+upgrades: add section on protocol upgrades --- wire_protocol.asciidoc | 196 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 188 insertions(+), 8 deletions(-) diff --git a/wire_protocol.asciidoc b/wire_protocol.asciidoc index a60f29c..d810652 100644 --- a/wire_protocol.asciidoc +++ b/wire_protocol.asciidoc @@ -886,13 +886,193 @@ also set the `first_timestamp` and `timestamp_range` fields if they wish to receive a backlog of updates they may have missed while they were down. -### Type Length Value Fields - -// TODO(roasbeef): move up after the framing discussion - ## Feature Bits & Protocol Extensibility -diff types of upgrade: - * e2e - * internal - * link level +As the Lighting Network is a decentralized system, no one entity can enforce a +protocol change or modification upon all the users of the system. This +characteristic is also seen in other decentralized networks such as Bitcoin. +However, unlike Bitcoin overwhelming consensus *is not* require to change a +subset of the Lightning Network. Lighting is able to evolve at will without a +strong requirement of coordination, as unlike Bitcoin, there is no *global& +consensus required in the Lightning Network. Due to this fact and the several +upgrade mechanisms embedded in the Lighting Network, at most, only the +participants that wish to use these new Lighting Network feature need to +upgrade, and then they are able to interact w/ each other. + +In this section, we'll explore the various ways that developers and users are +able to design, roll out, deploy new features to the Lightning Network. The +designers of the origin Lightning Network knew at the time of drafting the +initial specification, that there were many possible future directions the +network could evolves towards. As a results, they made sure to emplace several +extensibility mechanisms within the network which can be used to upgrade the +network partially or fully in a decoupled, desynchronized, decentralized +manner. + +### Feature Bits as an Upgrade Discoverability Mechanism + +An astute reader may have noticed the various locations that "feature bits" are +included within the Lightning Protocol. A "feature bit" is a bitfield that can +be used to advertise understanding or adherence to a possible network protocol +update. Feature bits are commonly assigned in *pairs*, meaning that each +potential new feature/upgrade always defines *two* bits within the bitfield. +One bit signals that the advertised feature is _optional_, meaning that the +node knows a about the feature, and can use it if compelled, but doesn't +consider it required for normal operation. The other bit signals that the +feature is instead *required*, meaning that the node will not continue +operation if a prospective peer doesn't understand that feature. + +Using these two bits optional and required, we can construct a simple +compatibility matrix that nodes/users can consult in order to determine if a +peer is compatible with a desired feature: + +|Bit Type|Remote Optional|Remote Required|Remote Unknown +|--------|--------|--------| +|Local Optional|✅|✅|✅| +|Local Required|✅|✅|❌| +|Local Unknown|✅|❌|❌| + +From this simplified compatibility matrix, we can see that as long as the other +party *knows* about our feature bit, then can interact with them using the +protocol. If the party doesn't even know about what bit we're referring to +*and* they require the feature, then we are incompatible with them. Within the +network, optional features are signalled using an _odd bit number_ while +required feature are signalled using an _even bit number_. As an example, if a +peer signals that they known of a feature that uses bit _15_, then we know that +this is an _optional_ feature, and we can interact with them or respond to +their messages even if we don't know about the feature. On the other hand, if +they instead signalled the feature using bit _16_, then we know this is a +required feature, and we can't interact with them unless our node also +understands that feature. + +The Lighting developers have come up with an easy to remember phrase that +encodes this matrix: "it's OK to be odd". This simple rule set allows for a +rich set of interactions within the protocol, as a simple bitmask operation +between two feature bit vectors allows peers to determine if certain +interactions are compatible with each other or not. In other words, feature +bits are used as an upgrade discoverability mechanism: they easily allow to +peers to understand if they are compatible or not based on the concepts of +optional, required, and unknown feature bits. + +Feature bits are found in the: `node_announcement`, `channel_announcement`, and +`init` messages within the protocol. As a result, these three messages can be +used to *signal* the knowledge and/or understanding of in-flight protocol +updates within the network. The feature bits found in the `node_announcement` +message can allow a peer to determine if their _connections_ are compatible or +not. The feature bits within the `channel_announcement` messages allows a peer +to determine if a given payment ype or HTLC can transit through a given peer or +not. The feature bits within the `init` message all peers to understand kif +they can maintain a connection, and also which features are negotiated for the +lifetime of a given connection. + +### Utilizing TLV Records for Forwards+Backwards Compatibility + +As we learned earlier in the chapter, Type Length Value, or TLV records can be +used to extend messages in a forwards and backwards compatible manner. +Overtime, these records have been used to _extend_ existing messages without +breaking the protocol by utilizing the "undefined" area within a message beyond +that set of known bytes. + +As an example, the original Lighting Protocol didn't have a concept of the +_largest_ HTLC that could traverse through a channel as dictated by a routing +policy. Later on, the `max_htlc` field was added to the `channel_update` +message to phase in such a concept over time. Peers that held a +`channel_update` that set such a field but didn't even know the upgrade existed +where unaffected by the change, but may see their HTLCs rejected if they are +beyond the said limit. Newer peers on the other hand are able to parse, verify +and utilize the new field at will. + +Those familiar with the concept of soft-forks in Bitcoin may now see some +similarities between the two mechanism. Unlike Bitcoin consensus-level +soft-forks, upgrades to the Lighting Network don't require overwhelming +consensus in order to adopt. Instead, at minimum, only two peers within the +network need to understand new upgrade in order to start utilizing it without +any permission. Commonly these tow peers may be the receiver and sender of a +payment, or it may the initiator and responder of a new payment channel. + +### A Taxonomy of Upgrade Mechanisms + +Rather than there being a single widely utilized upgrade mechanism within the +network (such as soft forks for base layer Bitcoin), there exist a wide +gradient of possible upgrade mechanisms within the Lighting Network. In this +section, we'll enumerate the various upgrade mechanism within the network, and +provide a real-world example of their usage in the past. + +#### Internal Network Upgrades + +We'll start with the upgrade type that requires the most extra protocol-level +coordination: internal network upgrades. An internal network upgrade is +characterized by one that requires *every single node* within a prospective +payment path to understand the new feature. Such an upgrade is similar to any +upgrade within the known internet that requires hardware level upgrades within +the core relay portion of the upgrade. In the context of LN however, we deal +with pure software, so such upgrades are easier to deploy, yet they still +require much more coordination than any other upgrade type utilize within the +network. + +One example of such an upgrade within the network was the move to using a TLV +encoding for the routing information encoded within the onion encrypted routing +packets utilized within the network. The prior format used a hand encoded +format to communicate information such as the next hop to send the payment to. +As this format was _fixed_ it meant that new protocol-level upgrades such as +extensions that allowed feature such as packet switching weren't possible +without. The move to encoding the information using the more flexible TLV +format meant that after the single upgrade, then any sort of feature that +modified the _type_ of information communicated at each hop could be rolled out +at will. + +It's worth mentioning that the TLV onion upgrade was a sort of "soft" internal +network upgrade, in that if a payment wasn't using any _new_ feature beyond +that new routing information encoding, then a payment could be transmitted +using a _mixed_ set of nodes, as no new information would be transmitted that +are required to forward the payment. However, if a new upgrade type instead +changed the _HTLC_ format, then the entire path would need to be upgraded, +otherwise the payment wouldn't be able to be fulfilled. + +#### End to End Upgrade + +To contrast the internal network upgrade, in this section we'll describe the +_end to end_ network upgrade. This upgrade type differs from the internal +network upgrade in that it only requires the "ends" of the payment, the sender +and receiver to upgrade in order to be utilized. This type of upgrade allows +for a wide array of unrestricted innovation within the network, as due to the +onion encrypted nature of payments within the network, those forwarding HTLCs +within the center of the network may not even know that new feature are being +utilized. + +One example of an end to end upgrade within the network was the roll out of +MPP, or multi-path payments. MPP is a protocol-level feature that enables a +single payment to be split into multiple parts or paths, to be assembled at the +receiver for settlement. The roll out our MPP was coupled with a new +`node_announcement` level feature bit that indicates that the receiver knows +how to handle partial payments. Assuming a sender and receiver know about each +other (possibly via a BOLT 11 invoice), then they're able to use the new +feature without any further negotiation. + +Anothert example of an end to end upgrade are the various types of +_spontaneous_ payments deployed within the network. One early type of +spontaneous payments called "keysend" worked by simply placing the pre-image of +a payment within the encrypted onion packet that is only decrypted by the +destination o of the payment. Upon receipt, the destination would decrypt the +pre-image, then use that to settle the payment. As the entire packet is end to +end encrypted, this payment type was safe, since none of the intermediate nodes +are able to fully unwrap the onion to uncover the payment pre-image that +corresponded to that payment hash. + +#### Channel Construction Level Updates + +The final broad category of updates within the network are those that happen at +the channel construction level, but which don't modify the structure of the +HTLC used widely within the network. When we say channel construction, we mean +_how_ the channel is funded or created. As an example, the eltoo channel type +can be rolled out within the network using a new `node_announcement` level +feature bit as well as a `channel_announcement` level feature bit. Only the two +peers on the sides of the channels needs to understand and advertise these new +features. This channel pair can then be used to forward any payment type +granted the channel supports it. + +The "anchor outputs" channel format which allows the commitment fee to be +bumped via CPFP, and second-level HTLCs aggregated amongst other transactions +was rolled out in such a manner. Using the implicit feature bit negotiation, if +both sides established a connection, and advertised knowledge of the new +channel type, then it would be used for any future channel funding attempts in +that channel.