How to setup and run Stellar Core node?

Stellar is a decentralized network connecting several nodes, which are computers that keep a shared distributed ledger. The nodes communicate with each other to validate and add transactions to the ledger. Stellar Core is an implementation of the Stellar Consensus Protocol. Nodes use it to stay in sync while working on validating transactions and applying them to the ledger.

This article explains technical and operational aspects of installing, configuring, monitoring and maintaining a Stellar Core node. The basic flow of the article goes like this:

• Choose the type of node to run
• Environment setup
• Install Stellar Core
• Configuration
• Publish History Archives
• Running
• Monitor and maintain node

Types of nodes

All Stellar nodes’ basic functionalities are almost the same: running Stellar Core, connecting to peers, handling transactions, storing the ledger’s state in SQL database, and keeping a copy of the ledger in flat XDR files called buckets. Horizon is the Stellar API that is supported by all the nodes. Besides the basic functionalities, two key configuration options determine a node’s behavior. A node can:

• participate in consensus for transitions validation
• publish an archive that other nodes can access to find the complete history of the network.

Based on these two key points, nodes are categorized into the following four types:

1. Watcher
2. Basic validator
3. Full validator
4. Archiver

All types of nodes support Horizon and submit transactions to the network; the difference lies in validating transactions and publishing history.

1. Watcher
   A Watcher keeps track of the ledger and submits transactions, but it is not configured to participate in transaction validation or publish an archived history. It means a watcher doesn’t do anything to support the network or increase decentralization. It is the lightest node. Watcher pairs well with Horizon and if the requirement is only for Horizon instance, a Watcher would be the right choice.
2. Basic validator
   A Basic validator has similar operational requirements and provides similar advantages as a Watcher. The difference is that the Basic validator uses a secret key and it is configured to participate in the consensus.
3. Full validator
   Like the Basic validator, the Full validator also performs all similar tasks except that it also publishes a history archive containing the ledger’s snapshots. Though it’s a bit expensive and complex to run, it supports the network’s resilience and decentralization. New nodes that join the network or those that fall temporarily out of sync can consult archives provided by Full validators to catch up on the network’s history.
   A Full validator can also support Horizon, but the organizations that run them don’t use them to query network data or submit transactions. Instead, the organizations prefer to run Watcher in addition to Full validator to handle Horizon.
4. Archiver
   Like a Full validator, an archiver publishes activities of the network but doesn’t participate in consensus. Its usefulness is relatively limited. If the need is to run a Stellar-facing service such as blockchain explorer, archives need to run. Use an archiver if you want to referee the network, which is quite unlikely.

Environment setup

There are various ways to install Stellar Core. After installation, it can be configured to participate in the network at different levels: Watcher, Basic validator or Full validator. No matter how the Stellar Core is installed, it is necessary to connect to a peer-to-peer network, store the ledger’s state in the database and connect to the Stellar API, Horizon.

• Compute requirements
  Stellar Core PostgreSQL works well on an m5.large in AWS, having dual-core 2.5GHz Intel Xenon and 8GB RAM. Storage-wise, 1 TB storage space is suggested to be sufficient. If Stellar Core is running in conjunction with Horizon, ensure that the system setup is equipped to handle Horizon’s compute requirements.
• Network access
  Stellar Core connects to a peer-to-peer network to keep the ledger in sync. It means the node needs to maintain specific TCP ports available for inbound and outbound communication.
  Inbound: Stellar Core node allows all IPs to connect to its PEER_PORT over TCP. A port can be specified while configuring Stellar Core, but the default (11625) is mainly used.
  Outbound: Stellar Core needs to connect to other nodes via PEER_PORT over TCP. Information about other nodes PEER_PORT can be found on network explorer such as Stellarbeat.

Install Stellar Core

Developers can install Stellar Core in three ways: use a Docker image, pre-built packages or build from source. Here, the first two methods are discussed; installation from source will require build instructions.

• Docker-based installation: SDF maintains a quickstart image that bundles Stellar Core with PostgreSQL database and Horizon. It is a quick way to set up a non-validating ephemeral configuration that works for most developers. SDF also handles Stellar Corre only standalone image that starts a three node-local Stellar core network, all on the same docker host. Moreover, SatoshiPay maintains Docker’s separate image for Horizon and Stellar Core. It comes in a few varieties, including one with AWS CLI installed and the other with Google Cloud SDK installed.
• Package-based installation: For Ubuntu 16.40 LTS users, there’s Stellar Core and Horizon’s latest release in Debian binary package format. These packages can be installed individually, offering the most incredible flexibility but needing manual configuration files and the PostgreSQL database. However, the option to install a stellar quickstart package is also available, which configures a Testnet Stellar Core and Horizon, both backed by a local PostgreSQL database. It can also be modified once installed.

Configuration

After installation, the next step is to complete a configuration file that states important things about the nodes like what database it writes to, whether it connects to a public network or testnet and which other nodes are in the quorum set. It is done by using TOML. By default stellar core loads that file from ./stellar-core.cfg.  A different file can also be loaded using the command:

$ stellar-core --conf betterfile.cfg

1. Database

Stellar Core has two copies of the ledger: one in SQL database and the other in XDR files on local disks called buckets. During consensus, the database is consulted and modified automatically when a transaction set is applied to the ledger. It is fast, fine-grained and random access.

2. Buckets

Stellar Core stores a copy of the ledger in flat XDR files called buckets. These files are stored in the directory specified in the configuration file as BUCKET_DIR_PATH, which defaults to buckets. The bucket files do hashing and transmission of ledger differences to historical archives. Fast local disks with sufficient space are necessary to use to store buckets. However, for most parts, the content of buckets and databases is managed by Stellar Core to be ignored. While running stellar for the first time, initialize database and buckets with the following command:

$ stellar-core new-db

This command is also used if the database gets corrupted or needs to restart from scratch.

3. Network Passphrase

NETWORK_PASSPHRASE is used to specify whether the nodes connect to a public network or testnet. Choices are:

NETWORK_PASSPHRASE=”Public Global Stellar Network; September 2015″
NETWORK_PASSPHRASE=”Test SDF Network ; September 2015″

4. Validating

Stellar Core isn’t set up to validate by default. It needs to be configured if you want a node to be a Basic validator or Full validator. It means preparing the node to participate in SCP (Stellar Consensus Protocol) and sign messages pledging that the network agrees to the particular transaction set. It is a three-step process:

Generate a keypair stellar-core gen-seed
Add NODE_SEED=”SD7DN…” to the configuration file, where a secret key will be presented from the keypair.
Add NODE_IS_VALIDATOR=true to the configuration file.
It is crucial to safeguard the node’s secret key. If an unintended user gets access to it, they can send messages to the network and appear to originate from the root node. Every node has its secret key. If more than one node is running, HOME_DOMAIN must be set as a standard to those nodes using the NODE_HOME_DOMAIN property. It will allow nodes to group correctly during quorum set generation.

5. Choose quorum set

The selection of the quorum set doesn’t depend on the type of node. It consists of validators (specified by organization) that the node checks with to determine that apply transaction set to a ledger. An ideal quorum set:

aligns with your organization’s priorities
maintains good quorum intersection
has enough redundancy to handle a few node failures
Stellar automatically generates a quorum set based on structured information provided in the configuration file. The choice of validators is yours, while the Stellar Core configures them into an optimal quorum set. To accomplish this task, stellar does the following:

Classifies validators run by the same organization into a sub quorum
Assigns weights to those sub quorums based on quality
Specifies the threshold for each of those sub quorums
While this automatic quorum set by stellar doesn’t relieve us from all responsibilities, choosing a trustworthy validator and keeping an eye on them for consistency and reliability is an important task.

5.1 Validator discovery

It’s important to note that when adding a validating node to the quorum set, the node’s trust is on the organization running the node, the SDF and not on some anonymous Stellar public key. The validator declares a home domain on-chain using set_options operation and publishes information in stellar to generate a self-verified link between a node and the organization.toml file. This link allows to look up a node by its Stellar public key and checks stellar.toml to find who runs it. It is easier to consult the list of nodes on Stellarbeat.io rather than doing it manually. This list lets you discover that most reliable organizations run more than one validator. The critical point here is that you need to depend on precisely one entity or at least four entities for automatic quorum set configuration to work efficiently.

5.2 Home domain array

Stellar Core relies on two arrays of tables to create your quorum set: [[HOME_DOMAINS]] and [[VALIDATORS]].
[[HOME_DOMAINS]] defines a superset of validators: a home domain and information in [[HOME_DOMAINS]] table is shared when you add nodes hosted by the same organization to your configuration, specifically the quality rating that automatically applies to every validator.

5.3 Validator array

For each node added to the quorum set, complete [[VALIDATORS]] table consists of NAME, QUALITY, HOME_DOMAIN and PUBLIC_KEY fields. If the HOME_DOMAIN field of node aligns with the organization’s [[HOME_DOMAINS]] array, the quality rating mentioned there would be applied to the node.

5.4 Validator quality

QUALITY is a required field for every node in the quorum set. The quality rating for nodes can be HIGH, MEDIUM or LOW. Most weight in automatic quorum set configuration is given to High-quality validators. It is necessary to endure that the node has low latency and good uptime; the organization running the node is trustworthy and reliable.
A high-quality validator belongs to a suite of nodes that provide redundancy and publishes archives.
A medium-quality validator is nested under high-quality validators. The combined weight of all medium-quality validators is equivalent to a single high-quality entity.
A low-quality validator is nested under medium quality validator. The combined weight of all low-quality validators is equivalent to a single medium quality entity. They should be proved reliable over time.

5.5 Automatic quorum set generation

Once the validators are added to the configuration, the Stellar Core generates a quorum set based on the following rules:

Validators having the same home domain are grouped and provided a threshold requiring a simple majority.
Diverse groups of validators are provided thresholds assuming byzantine failure.
Based on QUALITY, entities are grouped and are nested from HIGH to LOW.
Decision-making priority is given to top HIGH-quality entities.
A single HIGH-quality entity weighs equal to the combined weights of all MEDIUM quality entities.
A single MEDIUM quality weighs equal to the combined weights of all LOW-quality entities.

5.6 Quorum and Overlay network

It’s a good practice to share the information with your validator about other reliable validators. KNOWN_PEERS and PREFERRED_PEERS are configured with addresses of these dependencies. PREFERRED_PEER_KEYS configured with keys from quorum set help to maintain the priority of nodes that allow reaching consensus.

5.7 Updating and coordinating quorum set

The best way to keep the quorum set updated and coordinated is to connect to the validators channel on Stellar Keybase and sign up for the validators google group. Whenever there’s a need for changes in the validator or quorum set – like take down validator for maintenance or add new to the quorum set – it is essential to staging changes to preserve quorum intersection and maintain the good health of the network:

Do not remove too many nodes from the quorum set before nodes are taken down. When different validators remove different nodes, the remaining sets do not overlap, causing network splits.
Do not add too many nodes to the quorum set at the same time. In such a case, nodes could overpower the configuration.
Adding or removing nodes begins by modifying your own nodes’ quorum sets and then coordinating with others to reflect changes gradually.

6. History

History archives are generally off-site commodity storage services such as Google Cloud Storage, Amazon S3 or custom SFTP/HTTP servers. Regardless of the kind of running node, it should be configured to get history from one or more public archives. It is possible to configure any number of archives, and Stellar Core will automatically round-robin between them.
While choosing a high-quality quorum set, high-quality nodes should be included, which add each archive’s location to the HISTORY field in the validators array.

7. Automatic maintenance

Stellar Core database has some tables that act as publishing queues for external systems like Horizon. These generate metadata for modifications made to the distributed ledger. The tables can grow without bounds if not appropriately managed.
A built-in scheduler deletes data from old ledgers that are not used anymore to avoid this situation. To control automatic maintenance behavior: AUTOMATIC_MAINTENANCE_PERIOD, AUTOMATIC_MAINTENANCE_COUNT and KNOWN_CURSORS are used.
Stellar Core performs this automatic maintenance by default. Ensure that it is disabled until appropriate data ingestion in downstream systems is done. If there’s a need to regenerate metadata, the easiest way is to replay ledgers for the required range after clearing the database in newdb.