lnbook/node_operations.asciidoc

[[operating_ln_node]]
== Operating a Lightning Network Node

After reading this far, you've probably set up a Lightning wallet. In this chapter we will take things one step further and set up a full Lightning node, learning how to operate and maintain it over time.

There are many reasons why you might want to set up your own Lightning node:

* To be a full participant of the Lightning Network, not just an end-user.
* To run an e-commerce store, or earn income with Lightning payments.
* To develop new services, applications or plugins on the Lightning Network
* To increase your privacy while using Lightning

Of course, there are costs to running a Lightning Network node. You need a computer (of course), a permanent Internet connection, lots of disk space, and lots of time!

But the skills you will learn from this experience are valuable and can be applied to a variety of other tasks too.

Let's get started!

=== Choosing your platform

There are many ways you can run a Lightning node, from a small mini-PC hosted in your home, to a dedicated server, to a hosted server in the cloud. The method you choose will depend on the resources you have and how much money you want to spend.


==== Why is reliability important for running a Lightning Node?

In Bitcoin, unless one is specifically running a mining node, hardware is not particularly important.
The Bitcoin Core node software can be run on any machine that meets its minimum requirements and does not need to be online to receive payments; only to send them.
If a Bitcoin node goes down for an extended period of time, the user can simply reboot the node and once it connects to the rest of the network, it will resync the blockchain.

In Lightning however, the user needs to be online both to send _and_ to receive payments. If the Lightning user is offline it cannot receive any payments from anyone and thus its open invoices cannot be fulfilled.
Furthermore, the open channels of an offline node cannot be used to route payments. Your channel partners will notice that you are offline and cannot contact you to route a payment. If you are offline too often, they may consider the Bitcoin locked up in their channels with you to be "wasted" capacity, and may close those channels. We also consider the case of a Protocol Breach i.e. your channel partner tries to cheat you by submitting an earlier commitment transaction. If you are offline and your channels aren't being monitored, then the theft could succeed and you will have no recourse once the timelock expires.
Hence, node reliability is extremely important for a Lightning node.

There is also the issue of hardware failure and loss of data. In Bitcoin, hardware failure can be a trivial problem if the user has a backup of their mnemonic phrase or private key. The Bitcoin wallet and the bitcoin inside the wallet be easily restored from the private key on any computer or hardware wallet, and the blockchain can be re-downloaded from any peer.

In Lightning however, the information about the user's channels, including the commitment transactions and revocation secrets, are not publicly known and are only stored on the individual user's hardware.
Thus, hardware failure in the Lightning Network can easily result in loss of funds.

==== What are the types of hardware Lightning Nodes?

* **General Purpose Computers**: a Lightning Network node can be run on a home computer or laptop running Windows, MacOS, or Linux. Typically this is run alongside a Bitcoin node.
* **Dedicated Hardware**: a Lightning Node can also be run on dedicated hardware like a Raspberry Pi, Rock64, or mini-PC. This setup would usually run a software stack including a Bitcoin node and other applications. This setup is popular as the hardware is dedicated to running and maintaining the Lightning node only and is usually set up with an installation "helper".
* **Specialized Hardware**: a Lightning Network node can also be run on purpose-built hardware specifically designed for it. This would include "out-of-the-box" Lightning node solutions that can be purchased as a kit or turn-key system.

==== Running in the "cloud"

Virtual Private Server (VPS) and "cloud computing" services such as Microsoft Azure, Google Cloud, or Amazon Web Services (AWS) are quite affordable and can be set up very quickly. A Lightning node can be hosted for between $20 and $40 per month on such a service.

However, as the saying goes, "'Cloud' is just other people's computers". Using these services means running your node on other people's computers, with all the privacy and security implications that has. A Lightning node running in the "cloud", will always be less secure, less private that one running on your own computer. Additionally, these cloud computing services are very centralized. The vast majority of Bitcoin and Lightning nodes running on such services are located in a handful of data centers in Virginia, Sunnyvale, Seattle, London and Frankfurt. When the networks or data centers of these providers have service problems, it affects thousands of nodes on so-called "decentralized" networks.

Running a Lightning node in the cloud is better than not running a node at all, and worse than running it on your own computer at home or in your office.

==== Running a node at home

If you have a reasonable capacity internet link at home, or in your office, you can certainly run a Lightning node there. Any "broadband" connection is sufficient for this purpose (running a lightweight node), and a fast connection will allow you to run a Bitcoin full node too.

While you can run a Lightning node (and even a Bitcoin node) on your laptop, it will become annoying quite fast. These programs consume your computer's resources and need to run 24/7, and you will find yourself competing against the background services for your computer's attention (meaning your browser and other desktop workloads will be slowed down).

Instead, most users will choose to run a node on a dedicated computer. Fortunately, you don't need a "server" class computer to do this. You can run a Lightning node on a mini-PC, such as a Raspberry Pi or an Atom-based fanless PC, computers which are commonly used as a media server or home automation hub. These are relatively inexpensive, costing between $50 and $150 USD at that time of this writing. To run on a mini-PC, you will need an external USB hard drive, which again is relatively inexpensive, costing approximately $50 USD. The advantage of a dedicated mini-PC as a platform for Lightning and Bitcoin nodes is that it can run continuously, silently and unobtrusively on your home WiFi network, tucked behind your router or TV. No one will even know that this little box is actually a global banking system!

==== What hardware is required to run a Lightning node?

At a minimum, the following will be required to run a Lightning Node:

* **CPU**: sufficient processing power will be required to run a Bitcoin node, which will continuously download and validate new blocks. The user also needs to consider the Initial Block Download (IBD) when setting up a new Bitcoin node which can take anywhere from several hours to several days. A 2-core or 4-core CPU is recommended.

* **RAM**: a system running both Bitcoin and Lightning nodes may work on 2GB RAM _barely_, but will perform much better with at least 4GB of RAM. The Initial Block Download will be especially challenging with less than 4GB of RAM. More than 8GB of RAM is unnecessary, because the CPU is the greater bottleneck for these types of services, due to cryptographic operations such as signature validation.

* **Storage Drive**: this can be a Hard Drive or an SSD, although an SSD will be significantly quicker for running a Bitcoin node. Most of the storage is used for the Bitcoin blockchain, which can take up more than 300GB (as of June 2020).

* **Internet Connection**: a reliable internet connection will be required to download new Bitcoin blocks, as well as to communicate with other Lightning Peers. Estimated data use ranges from 10GB to 100GB per month, depending on configuration.

* **Power Supply**: a reliable power supply is required as Lightning nodes need to be online at all times. A power failure will cause in-progress payments to fail. For heavy duty routing nodes, a backup or uninterruptible power supply (UPS) is useful in the case of power outages.

In addition, the user will also want to consider some kind of data backup solution. This could be a cloud-based automated backup to a server or web service the user controls. Or, it could be a local hardware backup, such as a second drive. Or for best results, a combination of local and remote backup.

==== Switching server configuration in the cloud

When renting a cloud server, it is often cost effective to change the configuration between two phases of operation: A faster CPU and faster storage will be needed during the Initial Block Download (e.g. the first day). After the blockchain has synced, the CPU and storage speed requirements are much less, so the performance can be downgraded to a more cost-effective level.

For example, on Amazon's cloud, we would use a 8-16GB RAM, 8-core CPU (e.g. t3-large or m3.large) and faster 400GB SSD (1000+ provisioned IOPS) for the Initial Block Download (IBD), reducing its time to just 6-8 hours. Once that is complete, we would switch the server instance to a 2GB RAM, 2-core CPU (e.g. t3.small) and storage to a general purpose 1TB HDD. This will cost about the same as if you ran it on the slower sevrer the entire time, but will get you up and running in less than a day instead of having to wait almost a week for the IBD.

===== Permanent data storage (drive)

If you use a mini-PC or rent a server, the storage can be the costliest part, costing as much as the computer and connectivity (data) added together.

Let's have a look at the different options available. First there are two main types of drives, Hard Disk Drives (HDDs) and Solid State Drives (SSDs). HDDs are a lot cheaper and SSDs are a lot faster, but both would do the job.

The fastest SSDs up to date use the NVMe interface, but depending on your hardware they may not be better than the cheaper version, SATA SSDs. As an example, the Raspberry Pi 4 cannot benefit from them because of the limited bandwidth of its USB port.

To choose the size, let's look at the Bitcoin blockchain. As of July 2020 its size is approximately 330GB including the transaction index. If you want to have some margin available for the future or to install other stuff on your node, purchase at least a 512GB drive or better yet a 1TB drive.

=== Using an installer/helper

Installing a Lightning node (or also a Bitcoin node), may be daunting if you are not familiar with a command-line environment. However, there are a number of projects that make "helpers", software that installs and configures the various components for you. You will still need to learn some command-line incantations to interact with your node, but most of the initial work is done for you.

==== RaspiBlitz

One of the most popular and complete such "helpers", is _RaspibBlitz_, a project built by Christian Rootzoll, which is intended to be installed on a Raspberry Pi 4. RaspiBlitz comes with a recommended hardware "kit" that you can build in a matter of hours, or at most a weekend. If you attent a Lightning "hackathon" in your city, you are likely to see many people working on their RaspiBlitz set up, swapping tips and helping each other. You can find the RaspiBlitz project here:

https://github.com/rootzoll/raspiblitz

image::[images/raspiblitz.jpg]

In addition to a Bitocin and Lightning node, RaspiBlitz can install a number of additional services, such as:

* TOR (Run as Hidden Service)
* ElectRS (Electrum Server in Rust)
* BTCPayServer (Cryptocurrency Payment Processor)
* BTC-RPC-Explorer (Bitcoin Blockchain Explorer)
* LNbits (Lightning wallet/accounts System)
* SpecterDesktop (Multisig Trezor, Ledger, COLDCARDwallet & Specter-DIY)
* LNDmanage (Advanced Channel Management CLI)
* Loop (Submarine Swaps Service)
* JoinMarket (CoinJoin Service)


==== MyNode

MyNode is another popular open source project including a lot of Bitcoin related software. Is is easy to install: you "flash" the installer onto an SD card, and boot your mini-PC from the SD card. You do not need any screen to use myNode as the administrative tools are accessible from a browser. You can manage it from a computer or even from your smartphone. Once installed, go to http://mynode.local/ and create a lightning wallet and node in two clicks.

You can find the MyNode project here:

https://mynodebtc.com/

In addition to a Bitcoin and Lightning node, MyNode can optionally install a variety of additional services, such as:

- Ride The Lightning (Lightning Node Management GUI)
- VPN Support for remote management or wallet (OpenVPN)
- lndmanage (CLI management tool)
- btc-rpc-explorer (A Bitcoin blockchain explorer)

==== BTCPay Server

While not initially designed as an installation "helper", the e-commerce and payment platform BTCPay Server has an incredibly easy installation system that uses docker containers and docker-compose to install a Bitcoin node, Lightning node and payment gateway, among many other services. It can be installed on a variety of hardware platforms, from a simple Raspberry Pi 4 (4GB recommended) to a mini-PC, old laptop, desktop or server.

BTCPay server not only installs a full node for you, but that is a side-effect of what is a fully-featured self-hosted self-custody e-commerce platform that can be integrated with many e-commerce platforms such as Wordpress Woocommerce and others. Originally developed as a feature-for-feature replacement of the Bitpay commercial service and API, it has evolved past that to be a complete platform for BTC and Lighting services related to e-commerce.

More information can be found at:

https://btcpayserver.org/

In addition to a Bitcoin and Lightning node, BTCPay server can also install a variety of services including:

- c-Lightning or LND Lightning node
- Litecoin support
- Monero suport
- Spark server (c-lightning web wallet)
- Charge server (c-lightning e-commerce API)
- Ride The Lightning (Lightning node management web GUI)
- Many BTC forks
- BTCTransmuter (event-action automation service supporting currency exchange)

The number of additional services and features is expanding rapidly, so the list above is only a small subset of what is available in the BTCPay Server platform.

==== Bitcoin node or lightweight Lightning

One critical choice for your node will be the choice of Bitcoin node and its configuration. Bitcon Core, the reference implementation is the most common choice, but not the only choice available. One alternative choice is +btcd+, which is a Go-language implementation of a Bitcoin node. Btcd supports some features that are useful for running an LND Lightning node and are not available in Bitcoin Core.

A second consideration is whether you will run an archival Bitcoin node with a full copy of the blockchain (some 350GB in mid-2020), or a _pruned_ blockchain that only keeps the most recent blocks. A pruned blockchain can save you some disk space, but will still need to download the full blockchain at least one (during the Initial Block Download), so it won't save you anything on network utilization. Using a pruned node to run a Lightning node is still an experimental capability and might not support all the functionality, however many people are running a node like that successfully.

Finally, you also have the option of not running a Bitcoin node at all, instead operating the LND Lightning node in "lightweight" mode, using the _neutrino_ protocol to retrieve blockchain information from public Bitcoin nodes operated by others. Running like this means that you are taking resources from the Bitcoin network without offering any back, but it is still better than not running your own Lightning node at all.

Keep in mind that operating a Bitcoin node allows you to support other services (other than a Lightning node). These other services may require an arcihval (not pruned) Bitcoin node and often can't run without a Bitcoin node. Consider what other services you may need to run now or in the future, to make an informed decision on the type of Bitcoin node you run.

The bottom line for this decision is: If you can afford a > 500GB disk, run a full archival Bitcoin node. You will be contributing resources to the Bitcoin system and helping others who  cannot afford to do so. If you can't afford such a big disk, run a pruned node. If you can't afford the disk or the bandwidth for even a pruned node, run a lightweight LND node over neutrino.

==== Operating system choice

The next decision is on which operating system you build your nodes. The vast majority of internet servers run on some variant of Linux. Linux is the platform of choice for the internet because it is an open source and powerful operating system. Linux, however has a steep learning curve and requires familiarity with a command-line environment. It is often intimidating for new users.

Ultimately, most of the services can be run on any modern POSIX operating system, which includes Mac OS, Windows and of course Linux. Your choice should be driven more by your familiarity and comfort with an operating system and you learning objectives. If you want to expand your knowledge and learn how to operate a Linux system, this is a great opportunity to do so with a specific project and a clear goal. If you just want to get a node up and running, go with what you know.

Nowadays, many services are also delivered in the form of containers, usually based on the docker system. These containers can be deployed on a variety of operating systems, abstracting the underlying OS. You may need to leanr some Linux CLI commands, however, as most of the containers run some variant of Linux inside.

=== Choose your Lightning node implementation

As with the choice of operating system, your choice of Lightning node implementation should depend more on your familiarity with the programming language and development tools used for each project. While there are some small differences in features between the various node implementations, those are relatively minor and most implementations  converge on the common standards defined by the BOLTs.

Familiarity with the programming language and build system, on the other hand, is a good basis for choosing a node. That's because installation, configuration, ongoing maintenance, and troubleshooting will all involve interacting with the various tools used by the build system, such as:

* make, autotools, and GNU utilities for c-lightning
* go utilities for LND
* Java/Maven for Eclair

The programming language doesn't just influence the choice of build system but also many other aspects of the program. Each programming language comes with a whole design philosophy and affects many other things, such as:

* Format and syntax of configuration files
* File locations (in the filesystem)
* Command-line arguments and their syntax
* Error message formatting
* Pre-requisite libraries
* Remote Procedure Call interfaces

When you choose your Lightning node, you are also choosing all of the above characteristics, so your familiarity with these tools and design philosophies will make it easier to run a node. Or harder, if you land in an unfamiliar domain.

On the other hand, if this is your first foray into the command-line and server/service environment, you will find yourself unfamiliar with any implementation and have the opportunity to learn something completely new. In that case you might want to decide based on a number of other factors, such as:

* Quality of support forums and chat rooms
* Quality of documentation
* Degree of integration with other tools you want to run

As a final consideration, you may want to examine the performance and reliability of different node implementations. This is especially important if you will be using this node in a production environment (for example to run a shop), and expect heavy traffic and high reliability requirements.

=== Installing a Bitcoin or Lightning node

You decided not to use an installation "helper" and instead to dive into the command-line of a Linxu operating system? That is a brave decision and we'll try to help you make it work. If you'd rather not try to do this manually, consider using an application that helps you install the node software or a container based solution, as described in <<helpers>>.

[WARNING]
====
This section will delve into the advanced topic of system administration from the command-line. You will need to do additional research and learn more skills not covered here. Linux administration is a complicated topic and there are many pitfalls. Proceed with caution!
====

In the next few sections we will briefly describe how to install and configure a Bitcoin and Lightning node on a Linux operating system. You will need to review the installation instructions for the specific Bitcoin and Lightning node applications you decided to use. You can usually find these in a file called +INSTALL+ or in the +docs+ sub-directory of each project. We will only describe some of the common steps that apply to all such services, and the instructions we offer will be necessarily incomplete.

==== Background services

For those accustomed to running applications on their desktop or smartphone, an application always has a graphical user interface even if it may sometimes run in the background. The Bitcoin and Lightning node applications, however, are very different. These applications do not have a graphical user interface built in. Instead, they run as _headless_ background services, meaning they are always operating in the background and do not interact with the user directly.

This can create some confusion for users who are not used to running background services. How do you know if such a service is currently running? How do you start and stop it? How do you interact with it? The answers to these questions depend on the operating system you are using, but for now we will assume you are using some Linux variant and answer them in that context.

==== Process isolation

Background services usually run under a specific user account in order to isolate them from the operating system and each other. For example, Bitcoin Core is configured to run as user +bitcoin+. You will need to use the command-line to create a user for each of the services you run.

In addition, if you have connected an external drive, you will need to tell the operating system to relocate the user's home directory to that drive. That's because a service like Bitcoin Core will create files under the user's home directory. If you are setting it up to download the full Bitcoin blockchain, these files will take up several hundred GB. Here, we assume you have connected the external drive and it is located on the +/external_drive/+ path of the operating system.

On most Linux systems you can creatre a new user with the +useradd+ command, like this:

----
$ sudo useradd -d /external_drive/bitcoin -s /dev/null bitcoin
----

The +m+ flag assigns the user's home directory. In this case, we put it on the external drive. The +s+ flag assigns the user's interactive shell. In this case we set it to +/dev/null+ to disable interactive shell use. The last argument is the new user's username +bitcoin+.

==== Node configuration


==== Node startup

For both Bitcoin and Lightning node services, "installation" also involves creating a so called _startup script_ to make sure that the node starts when the computer boots. Startup and shutdown of background services is handled by an operating system process, which in Linux is called _init_ or _systemd_. You can usually find a system startup script in the +contrib+ subdirectory of each project. For example, if you are on a modern Linux OS that uses +systemd+, you would find a script called +bitcoind.service+, that can start the Bitcoin Core node service.

As the root user, install the script by copying it into the +systemd+ service folder +/lib/systemd/system/+ and then reload +systemd+:

----
$ sudo systemctl daemon-reload
----

Next, enable the service:

----
$ sudo systemctl enable bitcoind
----

You can now start and stop the service:

----
$ sudo systemctl start bitcoind
$ sudo systemctl stop bitcoind
----





==== NEXT....

* Auto-unlocking
* IBD configuration
* Fine tuning performance
* RPC security and authentication
* Logging
* Process monitoring

=== Node and channel backups

* Difference between Bitcoin wallet (BIP39) backups and LN backups
* Need for backup of channel state

==== Static channel backups

* How to set up
* Copying off the system, to account for loss of hardware

=== Security of your machine


==== Hot wallet risk

==== Sweeping funds

===== On-chain sweep

===== Off-chain sweep

==== Watchtowers

=== Channel management

==== Private vs public channels

==== Manually choosing nodes for outbound channels

==== Autopilot

==== Getting inbound liquidity

==== On-chain fees for channel management

==== Submarine swaps

==== Splice-in/Splice-out


=== Routing fees

* Earning fees from routing
* Setting routing fees
  * High volume/low cost vs. High cost/low volume
  * Zero fee routing

=== Node monitoring

==== RTL

Maintaining a Lightning node using the command-line can be a tedious task sometimes, fortunately we can use Ride The Lightning, most commonly known as RTL.

RTL is web graphical user interface to help users to manage lightning node operations for the three main lightning implementations (LND, c-lightning and Eclair), RTL is an open source project developed by Suheb, Shahana Farooqi and many other contributors. You can find the RTL software here:

https://github.com/Ride-The-Lightning/RTL

==== lndash

==== External node monitors (1ml etc.)

=== Channel maintenance

==== Inactive channels and nodes
==== When to force-close
==== Re-balancing channels

=== Running multiple Lightning Network nodes

==== Private channels
==== Topology