Setting Up a Flow Node

This guide is for running a Collection, Consensus, Verification and Execution node. If you are planning to run an Access node then refer to access node setup.

First you'll need to provision a machine or virtual machine to run your node software. Please see follow the node-provisioning guide for it.

Pull the Flow Images

The flow-go binaries are distributed as container images, and need to be pulled down to your host with your image management tool of choice.

Replace $ROLE with the node type you are planning to run. Valid options are:

• collection
• consensus
• execution
• verification
• access

_10

# Docker

_10

docker pull gcr.io/flow-container-registry/${ROLE}:alpha-v0.0.1

_10

_10

# Containerd

_10

ctr images pull gcr.io/flow-container-registry/${ROLE}:alpha-v0.0.1",

Prepare Your Node to Start

Your nodes will need to boot at startup, and restart if they crash.

If you are running systemd you can use the service files provided by flow-go. Find them in the Flow Go.

If you are using some other system besides Systemd, you need to ensure that the Flow container is started, the appropriate key directories are mounted into the container, and that the container will automatically restart following a crash.

The systemd files pull runtime settings from /etc/flow/runtime-config.env and any .env files under /etc/flow/conf.d. Examples of these files are also available in the github repo. You will need to modify the runtime config file later.

Systemd

info

If you are not using Systemd, you can skip this step

1. Ensure that you pulled the latest changes from flow-go repository via git

_10

## Clone the repo if you haven't already done so

_10

git clone https://github.com/onflow/flow-go

_10

_10

## Get latest changes

_10

cd flow-go

_10

git pull origin master

2. Copy your respective systemd unit file to: /etc/systemd/system
3. Create directory sudo mkdir /etc/flow
4. Copy the runtime-conf.env file to: /etc/flow/
5. Enable your service sudo systemctl enable flow-$ROLE.service (replace $ROLE with your node role - eg. collection)

Docker Configuration

If you are not using Systemd, sample commands for running each Docker container are below. Be sure to replace /path/to/data and /path/to/bootstrap with the appropriate paths you are using.

warning

Do not run your node using docker run command directly without a mechanism for the node to automatically restart following a crash.

info

The actual Docker image tag can be found here for appropriate spork.

System Configuration

Flow nodes create connections to other nodes on the network, which are represented as file descriptors by the OS. Depending on the default limits for your machine, you may need to increase the soft limit available to the node software.

Make sure the soft limit is at least 8192.

You can configure the ulimit for the node's docker container. See the Docker documentation for more details.

Admin Server

Each node can be configured with an admin server, which allows you to control some of the node's configuration, as well as view some of its internal state. You can find a few of the commands in the Admin Server README. Two commands to highlight are:

• list-commands: which returns a list of all of the available commands for your node
• set-log-level: which allows you to change the log level of your node at runtime

You can enable the admin server by passing the --admin-addr flag with an interface and port.

⚠️ IMPORANT: The admin server can modify your node's configuration. DO NOT allow access to untrusted clients.

Access

_19

docker run --rm \

_19

-v /path/to/bootstrap:/bootstrap:ro \

_19

-v /path/to/data:/data:rw \

_19

--name flow-go \

_19

--network host \

_19

--ulimit nofile=8192 \

_19

gcr.io/flow-container-registry/access:<applicable docker tag> \

_19

--nodeid=${FLOW_GO_NODE_ID} \

_19

--bootstrapdir=/bootstrap \

_19

--datadir=/data/protocol \

_19

--secretsdir=/data/secrets \

_19

--execution-data-dir=/data/execution_data \

_19

--rpc-addr=0.0.0.0:9000 \

_19

--http-addr=0.0.0.0:8000 \

_19

--admin-addr=0.0.0.0:9002 \

_19

--collection-ingress-port=9000 \

_19

--script-addr=${FLOW_NETWORK_EXECUTION_NODE} \

_19

--bind 0.0.0.0:3569 \

_19

--loglevel=error

Collection

_15

docker run --rm \

_15

-v /path/to/bootstrap:/bootstrap:ro \

_15

-v /path/to/data:/data:rw \

_15

--name flow-go \

_15

--network host \

_15

--ulimit nofile=8192 \

_15

gcr.io/flow-container-registry/collection:<applicable docker tag> \

_15

--nodeid=${FLOW_GO_NODE_ID} \

_15

--bootstrapdir=/bootstrap \

_15

--datadir=/data/protocol \

_15

--secretsdir=/data/secrets \

_15

--ingress-addr=0.0.0.0:9000 \

_15

--admin-addr=0.0.0.0:9002 \

_15

--bind 0.0.0.0:3569 \

_15

--loglevel=error

Consensus

_14

docker run --rm \

_14

-v /path/to/bootstrap:/bootstrap:ro \

_14

-v /path/to/data:/data:rw \

_14

--name flow-go \

_14

--network host \

_14

--ulimit nofile=8192 \

_14

gcr.io/flow-container-registry/consensus:<applicable docker tag> \

_14

--nodeid=${FLOW_GO_NODE_ID} \

_14

--bootstrapdir=/bootstrap \

_14

--datadir=/data/protocol \

_14

--secretsdir=/data/secrets \

_14

--admin-addr=0.0.0.0:9002 \

_14

--bind 0.0.0.0:3569 \

_14

--loglevel=error

Execution

_17

docker run --rm \

_17

-v /path/to/bootstrap:/bootstrap:ro \

_17

-v /path/to/data:/data:rw \

_17

--name flow-go \

_17

--network host \

_17

--ulimit nofile=500000 \

_17

gcr.io/flow-container-registry/execution:<applicable docker tag> \

_17

--nodeid=${FLOW_GO_NODE_ID} \

_17

--bootstrapdir=/bootstrap \

_17

--datadir=/data/protocol \

_17

--secretsdir=/data/secrets \

_17

--triedir=/data/execution \

_17

--execution-data-dir=/data/execution_data \

_17

--rpc-addr=0.0.0.0:9000 \

_17

--admin-addr=0.0.0.0:9002 \

_17

--bind 0.0.0.0:3569 \

_17

--loglevel=error

For execution nodes, it is recommend to increase the open files limit in your operating system. To do that, add the following to your /etc/security/limits.conf or the equivalent limits.conf for your distribution:

_10

* hard nofile 500000

_10

* soft nofile 500000

_10

root hard nofile 500000

_10

root soft nofile 500000

Restart your machine to apply these changes. To verify that the new limits have been applied, run:

_10

ulimit -n

Verification

_14

docker run --rm \

_14

-v /path/to/bootstrap:/bootstrap:ro \

_14

-v /path/to/data:/data:rw \

_14

--name flow-go \

_14

--network host \

_14

--ulimit nofile=8192 \

_14

gcr.io/flow-container-registry/verification:<applicable docker tag> \

_14

--nodeid=${FLOW_GO_NODE_ID} \

_14

--bootstrapdir=/bootstrap \

_14

--datadir=/data/protocol \

_14

--secretsdir=/data/secrets \

_14

--admin-addr=0.0.0.0:9002 \

_14

--bind 0.0.0.0:3569 \

_14

--loglevel=error

Archive

_18

docker run --rm \

_18

-v /path/to/data:/data:rw \

_18

--network host \

_18

--label=project=flow \

_18

--label=network=mainnet

_18

--label=app=dps

_18

--label=version=v0.26

_18

--name flow-dps gcr.io/flow-container-registry/flow-dps-live:v0.26.1

_18

--address 0.0.0.0:5005

_18

--index /data/index

_18

--bootstrap /data/bootstrap

_18

--checkpoint /data/bootstrap/root.checkpoint

_18

--data /data/protocol

_18

--level debug

_18

--metrics 0.0.0.0:8080

_18

--bucket bucket_name

_18

--seed-address=access_node_address

_18

--seed-key=seed_key

Start the Node

Now that your node is provisioned and configured, it can be started.

warning

Before starting your node, ensure it is registered and authorized.

Ensure you start your node at the appropriate time. See Spork Process for when to start up a node following a spork. See Node Bootstrap for when to start up a newly registered node.

Systemd

1. Check that your runtime-conf.env is at /etc/flow/runtime-conf.env
2. Update your environment variables: source /etc/flow/runtime-conf.env
3. Start your service: sudo systemctl start flow

Verify your Node is Running

Here are a few handy commands that you can use to check if your Flow node is up and running

Systemd

• To get Flow logs: sudo journalctl -u flow-YOUR_ROLE
• To get the status: sudo systemctl status flow

_10

● flow-verification.service - Flow Access Node running with Docker

_10

Loaded: loaded (/etc/systemd/system/flow-verification.service; enabled; vendor preset: enabled)

_10

Active: active (running) since Wed 2020-05-20 18:18:13 UTC; 1 day 6h ago

_10

Process: 3207 ExecStartPre=/usr/bin/docker pull gcr.io/flow-container-registry/verification:${FLOW_GO_NODE_VERSION} (code=exited, status=0/SUCCESS)

_10

Main PID: 3228 (docker)

_10

Tasks: 10 (limit: 4915)

_10

Memory: 33.0M

_10

CGroup: /system.slice/flow-verification.service

_10

└─3228 /usr/bin/docker run --rm -v /var/flow/bootstrap:/bootstrap:ro -v /var/flow/data:/data:rw --rm --name flow-go --network host gcr.io/flow-container-registry/verification:candidate8 --nodeid=489f8a4513d5bd8b8b093108fec00327b683db545b37b4ea9153f61b2c0c49dc --bootstrapdir=/bootstrap --datadir=/data/protocol --alpha=1 --bind 0.0.0.0:3569 --loglevel=error

Docker

• To get Flow logs: sudo docker logs flow-go
• To get the status: sudo docker ps

_10

$ sudo docker ps

_10

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

_10

1dc5d43385b6 gcr.io/flow-container-registry/verification:candidate8 \"/bin/app --nodeid=4…\" 30 hours ago Up 30 hours flow-go

Monitoring and Metrics

This is intended for operators who would like to see what their Flow nodes are currently doing. Head over to Monitoring Node Health to get setup.

Node Status

The metrics for the node should be able to provide a good overview of the status of the node. If we want to get a quick snapshot of the status of the node, and if it's properly participating in the network, you can check the consensus_compliance_finalized_height or consensus_compliance_sealed_height metric, and ensure that it is not zero and strictly increasing.

_10

curl localhost:8080/metrics | grep consensus_compliance_sealed_height

_10

_10

# HELP consensus_compliance_sealed_height the last sealed height

_10

# TYPE consensus_compliance_sealed_height gauge

_10

consensus_compliance_sealed_height 1.132054e+06