So you’ve seen a cool Dashboard on /r/homelab with graphs and status panels, and now you want one for your lab. Well let me tell you a little secret. It’s actually really easy to set up! This guide series will walk you through how to set up Grafana, Prometheus, InfluxDB, Telegraf, and a few other data services that pair nicely in a homelab setting. This will be a “get your feet wet” guide series and will get you in the door of the fun world of metrics and monitoring. We will be starting from complete scratch, so to follow along make sure you read through the prerequisites below.

The Environment

This guide series assumes you are on Windows 10 or 11 with Windows Terminal, Visual Studio Code, the OpenSSH Client installed and to keep this guide easy to update in the future, that you already have a Ubuntu Server or similar Linux imaged system already running and ready. If you do not know how to install Ubuntu Server, check out this guide from Ubuntu.

Pre-Requisites

• Have an x86 computing device (PC, Server, Workstation, Micro PC, etc.) with Ubuntu Server installed.
  • You can use a Raspberry Pi or other ARM based device. Just make sure to change docker / compose commands to use built for ARM images instead.
• Have an IDE / Text Editor installed on your pc.
  • I recommend Visual Studio Code with the Remote SSH extension installed.
• Access to a good internet connection.
• Have a spare monitor and keyboard handy.
• Your favorite drink and snacks within reach. ( A must for hobbies )

Grafana

If you are not familiar with Grafana, let me give you a quick run down on what it is and is not. Grafana is a data visualization tool that allows you to observe your data using charts, graphs, etc. It is not a replacement for things like Homer / Dashy or other similar dashboard software. Basically if you are wanting to monitor service metrics / logs then Grafana is for you.

Connecting to the Machine

First thing we need to do is hook up Visual Studio Code (VSC) to the machine we will be running Grafana, etc. on. For this we will be using the Remote - SSH extension for VSC. ( If you don’t have the extension installed yet, go ahead and install it. )

1. Open File Explorer and navigate to C:\Users\<username>\.ssh
2. Right click and select “New > Text Document”
3. Name the new file vsc_config and remove the .txt extension.

4. Open Visual Studio Code and click on Extensions in the navigation column on the left.
5. Under “Installed Extensions” find “Remote - SSH” and click on the settings gear then extension setting.
6. In the field labeled “Remote.SSH: Config File” add the location of your SSH config file.

• If you created the file above it will be C:\Users\<username>\.ssh\vsc_config
• If you are using your default ssh config file then the path will be C:\Users\<username>\.ssh\config

7. Close the settings Window
8. Click on the green icon in the lower left
9. In the dropdown at the top click on “Connect Current Window to Host…”
10. Click on ”+ Add New SSH Host…”
11. Enter your connection string for the Grafana host machine.

• e.g. username@192.168.10.1

12. Hit Enter
13. Select your SSH config file from the drop-down
14. Click Connect in the popup notification in the lower right
15. In the new VSC window select “Linux” from the dropdown
16. If you did not include an identity file in your connection string, enter your password in the prompt.
17. Hit Enter to connect to the machine

Setting up the Remote SSH extension for VSC

Installing Docker

To install Docker we first need to add the apt repository for it to Ubuntu’s apt repository index. Run the following commands to install the prerequisites. You can run these commands in the terminal inside VSC since we are connected to our machine over SSH. If the terminal is not showing you can open by clicking on “Terminal” and then “New Terminal” in the toolbar.

# Update the apt index
sudo apt update

# Install pre-requisites for apt to use HTTPS repositories (These are most likely already installed)
sudo apt install \
  ca-certificates \
  curl \
  gnupg \
  lsb-release

Now we need to add Docker’s official GPG key:

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

Next add the official repository for Docker:

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Once the repository is added, we can install the Docker Engine and Docker Compose:

sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin

After Docker installs we can add our user to the Docker group, so we don’t have to use sudo before every Docker command.

sudo usermod -aG docker yourusername

We can check if Docker Engine and Docker Compose were installed properly by running their version commands.

docker version
docker compose version

You should see an output similar to the image below.

While we are working on Docker, let’s update the logging driver to keep our log files from getting huge. Doing this now means we do not have to define it ourselves in the compose files later. To do this we need to edit the daemon.json file located in /etc/docker/. You will need to run this command from a terminal session (can be inside VSC’s terminal).

sudo nano /etc/docker/daemon.json

Then copy the following JSON configuration into the file.

1
{
2
  "log-driver": "json-file",
3
  "log-level": "",
4
  "log-opts": {
5
    "cache-disabled": "false",
6
    "cache-max-file": "5",
7
    "cache-max-size": "20m",
8
    "cache-compress": "true",
9
    "max-file": "2",
10
    "max-size": "10m"
11
  }
12
}

Save and close the file by hitting CTRL + X then y.

Setting up Prometheus

Now that we have Docker installed and configured we can start setting up our services. First up is Prometheus! We will be using Prometheus to gather metrics from our host machine via Node Exporter. First let’s create a directory structure for Prometheus.

sudo mkdir -p /opt/docker/prometheus
cd /opt/docker/prometheus

Next create the docker-compose.yml file and edit it.

touch docker-compose.yml
code docker-compose.yml

Copy the following into the docker-compose.yml file:

1
version: '3'
2
services:
3
  prometheus:
4
    restart: unless-stopped
5
    image: prom/prometheus:latest
6
    volumes:
7
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
8
      - prometheus_data:/prometheus
9
    command:
10
      - '--config.file=/etc/prometheus/prometheus.yml'
11
      - '--storage.tsdb.path=/prometheus'
12
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
13
      - '--web.console.templates=/usr/share/prometheus/consoles'
14
    ports:
15
      - 9090:9090
16

17
volumes:
18
  prometheus_data: {}

Save the file using CTRL+S. You can close it by clicking the X next to the name in the tab bar.

Deploying Node Exporter

Since we are deploying Node Exporter as another Docker container, we need to create the compose file for it. So let’s make a new directory in /opt/docker/prometheus to house it.

mkdir exporters
cd exporters

Next create the docker-compose.yml file and edit it.

touch node-exporter.yml
code node-exporter.yml

Copy the following into the node-exporter.yml file:

1
version: '3'
2
services:
3
  node_exporter:
4
    restart: unless-stopped
5
    image: quay.io/prometheus/node-exporter:latest
6
    command:
7
      - '--path.rootfs=/host'
8
    network_mode: host
9
    pid: host
10
    volumes:
11
      - '/:/host:ro,rslave'

Save the file using CTRL+S. You can close it by clicking the X next to the name in the tab bar. Now deploy Node Exporter:

docker compose -f node-exporter.yml up -d

To verify that it’s working you can run:

curl localhost:9100/metrics

You should see an output like the following:

Configuring Prometheus to Scrape Node Exporter

Now that we have Node Exporter running we can update Prometheus to scrape it for data. Edit the prometheus.yml file with the following:

cd ..
code prometheus.yml

Copy the following into the Prometheus config file:

1
# Set global scrape interval (how long between prometheus data scrapes on exporters)
2
global:
3
  scrape_interval: 15s
4

5
# Define the scrape job(s)
6
scrape_configs:
7
- job_name: node
8
  static_configs:
9
  - targets: ['<your-machine-ip>:9100']

Replace <your-machine-ip> with the IP of your Docker host. Then save the file using CTRL+S. You can close it by clicking the X next to the name in the tab bar.

Deploying Prometheus

To deploy Prometheus, run the following command inside the /opt/docker/prometheus directory.

docker compose up -d

Once deployed, open a browser tab and navigate to http://your-docker-machine-ip:9090/graph. When Prometheus’s query UI loads run the following in the Expression box.

1
rate(node_cpu_seconds_total{mode="system"}[1m])

You should see an output similar to the one pictured below.

Configuring Grafana to Use Prometheus

With Prometheus running and scraping metrics from our Node Exporter, we can now configure Grafana to read that data and display it in a dashboard. So first login to Grafana, then follow the steps below to set up the Prometheus data source and configure the dashboard.

1. Click on the Settings cog
2. Click “Data sources”
3. Click the “Add data source” button
4. Click on “Prometheus” from the list of options
5. Fill out the following info: a. Name your data source (default is Prometheus) b. Enter http://your-machine-ip:9090 for the URL c. Click Save & test
6. After you click “Save & test” you should see a green check mark notification tell you the data source is working.
7. Click the Dashboard button in the left-hand menu
8. Click on “Import”
9. Importing the Dashboard a. Enter 1860 into the “Import via grafana.com” box b. Click Load c. Edit the name of your Dashboard (what will be displayed) d. Select “Prometheus (default)” from the dropdown e. Click Import

After you click Import, you will be taken to your new dashboard! You can now explore the dashboard, as well as the queries each panel is making to Prometheus.

Conclusion

In an effort to keep this guide series easy to follow, I have broken it up into a series of posts. This is just part 1! The next part we will be covering InfluxDB and Telegraf. As always if you run into any issues let me know down below or reach out via email!

Thanks to my good friend Stefan for helping proof this post!