lnbook/p2p.asciidoc

Chapter overview:

  * high level description of p2p interaction for the LN

Relevant questions to answer:

  * Encrypted P2P Transport:
      - What is the noise protocol? How does it differ from TLS? Who created it

      - and what are some of primary design principles?

      - What is an authenticated key exchange?

      - Why does Noise offer various handshakes? What are some of unique properties certain handshakes offer?

      - What is key rotation in the context of a complete handshake? Why is it important?

      - What is "brontide"? How is it used in LN today? How can it be upgraded in the future?
  * LN Message Format:

      - What kind of framing is used in the LN message format? What's the max message size and why is it in place?

      - What is a varint? Why is it used in the protocol?

      - What are the message types of some of the popular messages in the protocol?

      - How can new messages be added in the future?
  * Feature bits:

      - What are feature bits in the network, and how+where are they advertised?

      - How can feature bits be used to phase in new features to the protocol?

      - Today, what are some of the major feature bits used in the system?

      - What's the difference between and end-to-end network upgrade and an internal network upgrade? How's the analogous to the evolution of routers and protocols in the existing internet?

  * TLV Message Extensions:

      - What does TLV stand for?

      - How is this related to the existing protobuf message format?

      - Where are TLV fields used in the protocol today?

      - How can TLV fields be used to extend the protocol, existing messages, and the onion itself?
      
      - Sidenote that TLV can be used by upcoming Instant Messaging chat apps like `Whatsat`, `Sphinx Chat` or `Juggernaut`


### Feature Bits

#### What is are Feature Bits and why do we need them?
 
Feature bits, or feature flags, are a string of 1s and 0s that Lightning nodes use to communicate to each other which features and upgrades they have enabled.
But why would they need to do this?

As discussed in earlier chapters, the Lightning Network protocol does not require all nodes to be in agreement on a common ruleset and list of features, which is quite different to the Bitcoin protocol.
If someone wants to introduce a new feature into Bitcoin (such as Replace-by-Fee) then either all nodes will need to accept this feature or the feature needs to be backwards-compatible.
However, if a new feature is introduced into the Lightning Network, Lightning users don't need to wait for the rest of the network to upgrade.
They can start using the new feature immedietely. 
Even if that feature is still in development, they can implement it as long as they can find other users willing to implement the feature along with them
footnote:[The Lightning Network itself was famously put into use before it's official "launch". Despite LN developers warning users that the software was still in beta and had bugs, users around the world set up their own nodes and used the software recklessly].

One example of a feature would be multi-part payments, discussed in the chapter on Path-Finding.
Despite being a feature still in development at the time of writing, it was already deployed by some nodes.
Given the inherent freedom in the Lightning Network's design there will never be a global consensus on the Lightning Network as to which features are supported and which aren't.
Some nodes may support multi-part payments, some may not, and some may never decide to support them.
As such, nodes need of a way of signalling to each other which features they support and which they do not.
They do so using pairs of 1s and 0s called Feature Bits.

#### How do Feature Bits work?

Whenever nodes communicate with each other, whether through invoices or other methods, a part of that message is reserved to signal which features the node has enabled.
For example, the `channel_announcement` and `node_announcement` messages described in BOLT #7 have a predefined `features` field reserved for this information.
The `features` field will take the form of a string of paired bits that will look something like this:

[feature-bits-example]
----
00101000001000000000
----

Flags are numbered from the least-significant bit (i.e. from right from left), starting from 0.
Flags are also paired, so that bits 0 and 1 form a pair, 2 and 3 form a pair, and so on.
BOLT #9 contains the full list of which digits stand for which features
footnote:[https://github.com/lightningnetwork/lightning-rfc/blob/master/09-features.md].
An odd-numbered bit pair (01) communicates that the feature is backwards-compatible.
An even-numbered bit pair (10) communicates that the feature is mandatory.
The rule of thumb is: "it's ok to be odd".

Using BOLT #9 we can break down the above string as follows:

[[feature-bits-breakdown]]
.Breakdown of a feature bit string
[options="header"]
|===
| Bit Number | Feature                          | Feature Bits | Status                         
| 0/1        | `option_data_loss_protect`       | 00     | Not enabled                    
| 2/3        | `initial_routing_sync`           | 00     | Not enabled                    
| 4/5        | `option_upfront_shutdown_script` | 00     | Not enabled                    
| 6/7        | `gossip_queries`                 | 00     | Not enabled                    
| 8/9        | `var_onion_optin`                | 01     | Enabled (Backwards-Compatible) 
| 10/11      | `gossip_queries_ex`              | 00     | Not enabled                    
| 12/13      | `option_static_remotekey`        | 00     | Not enabled                    
| 14/15      | `payment_secret`                 | 01     | Enabled (Backwards-Compatible) 
| 16/17      | `basic_mpp`                      | 01     | Enabled (Backwards-Compatible) 
| 18/19      | `option_support_large_channel`   | 00     | Not enabled                    
|===


If Alice sees this string in a node announcement message from Bob's node, then she knows from bits 16 and 17 that Bob's node supports multi-part payments.
And because the pair is odd, Alice knows that the feature is backwards-compatible.
If Alice also has multi-part payments enabled, then Alice and Bob can make use of this feature.
If Alice does not have multi-part payments enabled, she can simply ignore this and get on with her life.
She could still open a channel with Bob without having the feature enabled herself.
However, if the feature bits were even then the feature would be mandatory.
If Alice did not also have this feature enabled, she would have to find another node to open a channel with.
Alice also knows from bits 18 and 19 that Bob does not have large channels enabled.
If she would like to open a large channel, then she would have to find someone else who has large channels enabled to open it with.

In this way, upgrades and new features can be rolled out on the Lightning Network in an asynchronous bottom-up way.
Users can enable the features they want when they are ready to do so.
Power users can use and test features without waiting for them to be formally released, and more conservative users are not pressured into using a new feature before they are comfortable to do so.
Users can simply signal to each other which features they support, and even if they do not agree on the feature set, they can still connect and transact as long they agree on all mandatory features.