🛤 Validator RPC Node Disk Setup

If you have ever run a Solana validator or an RPC node, you know that the hardware is only half the story. The other half is how you lay out the disks. Get the storage setup right, and your node will hum along for months without a second thought. Get it wrong, and you will be chasing I/O bottlenecks at three in the morning while the chain moves on without you.

The requirements for the Solana validator node can be found on the official Solana validator requirements page. Link below:

Solana Validator Requirements

In the current production setup, we use a n2-standard-64 machine which has 128GB of RAM, 64 Core, 8TB of Local SSD NVME. Make sure that it’s NVME, as that is the requirement of the storage device for fast bootstrapping of large ledger data and accounts data. Regular SSDs will work for a while, but when the ledger grows and the chain is under load, the difference becomes very obvious very quickly. After provisioning the VM, we now start with the initial configuration of the VM.

👷‍♂️ Disk Setup

The plan is simple: split the raw NVME drives into three RAID0 groups, each with a specific job. RAID0 gives us the combined speed of all the drives working together, and since this is a node rather than the only copy of your data, the lack of redundancy is an acceptable trade. If a drive dies, you simply rebuild from the network. That is the whole point of a blockchain node anyway.

First, create a RAID0 device using the 24x 375GB Local SSDs. Split the bought 24 Local SSDs into three groups:

12: Transaction Ledger
10: Accounts
2: Logs and Spare Storage

To start creating the RAID0 devices on the transaction ledger you must execute this command:

sudo mdadm --create /dev/md0 --level=0 --raid-devices=12 \
  /dev/nvme0n1 /dev/nvme0n2 /dev/nvme0n3 /dev/nvme0n4 \
  /dev/nvme0n5 /dev/nvme0n6 /dev/nvme0n7 /dev/nvme0n8 \
  /dev/nvme0n9 /dev/nvme0n10 /dev/nvme0n11 /dev/nvme0n12

As you can see on the command flags, we pass the raid devices as 12 because that’s how many NVME drives we are trying to format. On the device name we named the block device /dev/md0 with level 0 RAID setup. After finishing the setup, we now go to setting up the accounts storage:

sudo mdadm --create /dev/md1 --level=0 --raid-devices=10 \
  /dev/nvme0n13 /dev/nvme0n14 /dev/nvme0n15 /dev/nvme0n16 \
  /dev/nvme0n17 /dev/nvme0n18 /dev/nvme0n19 /dev/nvme0n20 \
  /dev/nvme0n21 /dev/nvme0n22

Same command as the upper command, but look closely at the device block name and the number of raid devices. Make sure that if you are using a custom number of NVME drives, you modify the raid devices first before appending the block device names. Lastly, the spare storage:

sudo mdadm --create /dev/md2 --level=0 --raid-devices=2 \
  /dev/nvme0n23 /dev/nvme0n24

This is the last command to add the block device for the spare storage. We now move on to formatting the devices to our preferred storage filesystem type. On our setup we are in favor of using ext4 fs, as it provides better redundancy and journaling.

sudo mkfs.ext4 -F /dev/md0
sudo mkfs.ext4 -F /dev/md1
sudo mkfs.ext4 -F /dev/md2

When the formatting of the devices is done, we now move on to mounting the devices to our servers. Create the mount points which would be stored on the /mnt/disks/.

sudo mkdir -p /mnt/disks/solana-{ledger,account,spare}

Check if the directory is okay by running the ls command on the /mnt/disks/ directory. Also, make sure all the directories read, write and access permissions are okay by running the chmod on the created directories.

sudo chmod a+w /mnt/disks/solana-ledger
sudo chmod a+w /mnt/disks/solana-account
sudo chmod a+w /mnt/disks/solana-spare

The command above will ensure that the correct write permission is on the disk mount points. Then create auto mount on boot by editing the /etc/fstab:

echo UUID=`sudo blkid -s UUID -o value /dev/md0` /mnt/disks/solana-ledger ext4 discard,defaults,nofail 0 2 | sudo tee -a /etc/fstab
echo UUID=`sudo blkid -s UUID -o value /dev/md1` /mnt/disks/solana-account ext4 discard,defaults,nofail 0 2 | sudo tee -a /etc/fstab
echo UUID=`sudo blkid -s UUID -o value /dev/md2` /mnt/disks/solana-spare ext4 discard,defaults,nofail 0 2 | sudo tee -a /etc/fstab

This will create a specific record that will be inserted on the /etc/fstab containing the UUID and mount options. Using the UUID instead of a hardcoded device path is a small detail that saves a lot of pain later, since device names like /dev/md0 can change between reboots while a UUID never does.

Now everything’s done: reboot the server and check if all the mount points have been mounted by running the mount command.

If everything’s good, then you’re good to go to the next tutorial. In case you have to change the RAID0 arrays, remember to run this command on the specific RAID array device to delete it: sudo mdadm -S /dev/md0.

🧠 Final Thoughts

Disk setup is the least glamorous part of running a validator, and also the most important one. Get it right and everything that follows, from downloading snapshots to serving RPC requests, becomes routine. Get it wrong and you will be rebuilding arrays in the middle of the night while the chain moves on without you. Take your time, triple check the drive counts in each mdadm command, and verify your mounts after that first reboot. If everything looks good, you are ready: check out the second part here.