Homelab Notifications with ntfy

Posted Sep 12, 2023 Updated Dec 18, 2023

By Alexander Henderson 16 min read

So in my previous blog update post I mentioned a few projects I was working on for posts on this blog. Well considering this post wasn't on the list it's time to toss the plan! Or did I really have a plan to begin with?

Currently, I have a few homelab notifications being sent to a “private” Discord server. Evident by Logging Fail2Ban to Discord, SSH Login Logging to Discord via Webhooks, GitHub Actions - Automating your Docker Builds with Diun, a few others I was working on, etc. Now, I was going to keep going this route as it does make it extremely convenient since I already have the Discord app on most of my devices. However, I was talking to a good friend of mine, and he brought up a good point.

“My private server isn’t exactly ‘private’ when it comes to someone else managing the data and, the only way to send a notification to Discord is by Webhook or a custom bot. Also, sometimes push notifications to mobile are SUPER delayed.”

So I decided to look into alternatives which lead me to ntfy.sh. The best part? You can self-host it!

What is ntfy.sh?

Examples of some notifications I am piping into my ntfy instance. Picture is the mobile app view

ntfy (pronounced notify) is an HTTP publish / subscribe notification service that allows you to send notifications to your phone or PC. This can be done simply using scripts that can hit ntfy’s REST API or with apps that have direct integrations. The best part about ntfy, (other than being FOSS and having lots of documentation), is that you can self-host it! Which means you can control the data rather than relying on an outside service! Especially if you are rocking an Android phone since you can do instant delivery without Google FBC!

You can use ntfy’s hosted service rather than hosting it yourself as an option. However, you might want to read “the point” above, again..

See docs about iOS instant delivery.

Follow along notes

Before I dive in. There are a few things you will need setup before you can follow along with this post.

A domain with an A record forwarding to your public IP
- Example domain: notify.yourdomain.com
A web server that is port forwarded to listen on 80/443

I recommend Caddy if you want easy certificate management! I will be using Caddy¹² in this post but ntfy has example configs for other web servers.

A host to run ntfy on
- I will be using a host that is running Docker¹², on Ubuntu, as it makes it easy to spin up resources without bloating the host with tons of dependencies.

Setting up the Host

To get started, SSH to your host and install Docker. If you already have docker set up and ready you can skip this section. Run the following commands to install Docker’s dependencies and prepare the host to install Docker.

  
sudo apt update
sudo apt-get install ca-certificates curl gnupg

Next add Docker’s official GPG key.

  
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

Now add Docker’s apt repository to your source list.

  
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Then run apt update to update the sources again.

sudo apt-get update

Finally install Docker.

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

These instructions were pulled from the official Docker install docs. I recommend looking at the official install docs if your host OS is not Debian based.

If you do not want to run sudo in front of every docker command, you can add your user to the docker group.

sudo chmod -aG docker username

Warning from Docker about adding your user to the docker group:

The docker group grants root-level privileges to the user. For details on how this impacts security in your system, see Docker Daemon Attack Surface.

Setting up ntfy’s Configuration

With Docker now setup we can pivot to getting ntfy up and running. Create the directories to house the docker-compose.yml, cache, user.db, and the server.yml config file.

  
sudo mkdir -p /opt/docker/ntfy
sudo mkdir /opt/docker/ntfy/cache
sudo mkdir /opt/docker/ntfy/config
sudo chown -R $USER:$USER /opt/docker

Next create the docker-compose.yml file and edit it.

cd /opt/docker/ntfy
nano docker-compose.yml

  


version: "3"
services:
  server:
    restart: always
    logging:
      driver: "json-file"
      options:
        max-size: "500k"
        max-file: "1"
    image: binwiederhier/ntfy
    command: serve
    environment:
      - TZ=America/New_York
    user: ${UID}:$(id -g)
    volumes:
      - /opt/docker/ntfy/cache:/var/cache/ntfy
      - /opt/docker/ntfy/config:/etc/ntfy
    ports:
      - 80:80

Hit CTRL+ X then Y to save your changes and close nano. Now do the same but for a file called server.yml.

cd config/
nano server.yml

  


default-host: "https://notify.yourdomain.com"
base-url: "https://notify.yourdomain.com"
listen-http: ":80"
cache-file: "/var/cache/ntfy/cache.db"
behind-proxy: true
attachment-cache-dir: "/var/cache/ntfy/attachments"
keepalive-interval: "45s"
auth-file: "/etc/ntfy/user.db"
auth-default-access: "deny-all"
#web-root: "disable"

default-host - This sets the default host for subscribing to notification topics. Since this is self-hosted your notify domain would go here.
base-url - The URL in which you will access / communicate with ntfy. Usually the same as above ^.
listen-http - The port in which the HTTP web server listens on (default = 80). You can add an IP to bind to as well (listen-http: “127.0.0.1:80”)
cache-file - Location of the message cache file.
behind-proxy - If ntfy is deployed behind a proxy like Caddy, nginx, etc., then this needs to be set to true.
attachment-cache-dir - Cache location for message attachments.
keepalive-interval - The keepalive interval message interval in seconds.
auth-file - Location of the user.db authentication file.
auth-default-access - Setting this to “deny-all” will force all topics to be private by default.
web-root - Location of the root directory for ntfy. Setting this to “disable” turns off the web client.

You can see more configuration options in the ntfy docs.

For now, we will leave the web-root enabled to verify ntfy is running correctly. We will disable it later which is why it is commented out in the above config.

Deploying ntfy

Change directory back to the parent directory for ntfy where the docker-compose.yml file is and deploy ntfy.

cd /opt/docker/ntfy
docker compose up -d

If you get an error that the port is already in use, you will have to edit the compose file and change - 80:80 to something different. (example: - 8080:80`). 8080 being the host side port.

Configuring the Web Server

For my environment I use Caddy as my web server / reverse proxy. So this section will be covering how to configure Caddy to serve ntfy with an SSL certificate. If you don’t have Caddy configured already, check out my quick start guide for Caddy with Cloudflare/cloudflared. Otherwise, you can check out the ntfy docs for other web servers / skip this section.

To serve ntfy we need to tell Caddy where to point our notify domain to. To do this edit the Caddyfile and add the following block section.

notify.yourdomain.com, http://notify.yourdomain.com {
  import tls
  log {
    output file /config/logs/notify.log {
      roll_size 10mb
      roll_keep 5
      roll_keep_for 24h
    }
  }
  reverse_proxy ntfy-server-1
  @httpget {
    protocol http
    method GET
    path_regexp ^/([-_a-z0-9]{0,64}$|docs/|static/)
  }
  redir @httpget https://{host}{uri}
}

You will need to change notify.yourdomain.com, http://notify.yourdomain.com to your domain. You will also need to replace ntfy-server-1 to match your container environment or service IP/port. Once added, save and close the file and reload Caddy. If all goes well you can hit your domain, and you will see the web interface for ntfy. It will look like the image below minus the pro upgrade.

I recommend testing your configuration with the Let’s Encrypt staging acme server https://acme-staging-v02.api.letsencrypt.org/directory and watching the log. This will let you know if you have a DNS configuration issue before being rate limited as the production acme server has low retry counts.

Since I have a dedicated docker network I attach containers to for Caddy, I am able to do DNS lookups internally on the container names. This also means I am not exposing service ports to the host so port conflicts are minimized.

Image from ntfy.sh’s web app https://ntfy.sh/app

If you are not using a web server to serve ntfy, you can hit the web UI at the IP:PORT of your machine hosting it.

Securing ntfy

Now that we have ntfy up and being served we can lock it down. First we need to disable the web app (skip this if you want to use the web app) since we won’t be using it. Return to the server.yml file and uncomment the web-root line. Your file should look similar to below.

  


default-host: "https://notify.yourdomain.com"
base-url: "https://notify.yourdomain.com"
listen-http: ":80"
cache-file: "/var/cache/ntfy/cache.db"
behind-proxy: true
attachment-cache-dir: "/var/cache/ntfy/attachments"
keepalive-interval: "45s"
auth-file: "/etc/ntfy/user.db"
auth-default-access: "deny-all"
web-root: "disable"

CTRL+ x then y to save and close the file.

Applying the above changes and refreshing your notify URL will update the page to this

Creating a User in ntfy

Since we already have the auth-file and auth-default-access set, we can skip to adding our first user. To do this we will need to execute into the ntfy container environment and run a few commands. Substitute ntfy-server-1 with the name of your container.

If you do not know your container name you can use docker ps | grep ntfy to list out all running containers and then grep the output for ntfy.

docker exec -it ntfy-server-1 /bin/sh
ntfy user add yourusername

Once inside the container invoke the ntfy cli and add your first user.

Example of creating a user using ntfy cli

ntfy user add nameofuser

Invoking this command with a username will auto prompt you to create a password for the new user. After the user is created we can subscribe it to our first notification topic.

Example of setting user access using ntfy cli

ntfy access testuser ssh_logins wo

wo specifies the access level of the user for the topic ssh_logins. Since this user will be used to send notifications, we need to give it the ability to write to the topic. Hence, the wo = “write-only”. You can do rw for read-write, ro - read-only, and deny as well. You can change access permissions whenever, by re-running the above command with a different access level specified. Now create a second user but give it read-only permissions to the topic. This second user will be used for our mobile app.

You can create new topics by giving a user access to one that does not exist yet.

User Access Tokens

Now at this stage the user can be used to post messages to the topic. Though, to make it even easier we are going to create a access token for the user account. This token can be used in place of the username and password when sending messages. To add an access token run the following inside the ntfy docker container.

Example of creating user access token using ntfy cli

ntfy token add testuser

When you receive the output, copy the token to your notes as we will need it later.

Example of the output of the token add command with the token underlined in red

You can retrieve tokens at anytime by running ntfy token list username.

Type exit to leave the container interface.

Logging SSH login/logouts to ntfy

As an example for this post, I will only be covering ssh login/logout notifications to keep it short. I will include some extra scripts for other services in the blog-files repository. Now, SSH into the host you want to send notifications from and create the script file in /bin. Mark it executable and then edit it.

  


# Create the script file
sudo touch /bin/sshd-login-ntfy
# Mark it as executable
sudo chmod +x /bin/sshd-login-ntfy
# Edit the script file
sudo nano /bin/sshd-login-ntfy

  


#!/bin/bash

# User access token and url to post notification messages
NTFYTOKEN=
NTFYURL=

# If session was opened then post this message with priority level 5
if [ "${PAM_TYPE}" = "open_session" ]; then
  curl -H "Title: ${HOSTNAME} -- ssh login" \
 -u :$NTFYTOKEN \
 -H "Priority: 5" \
 -H "X-Tags: warning" \
 -d "user: ${PAM_USER}
ip/host: ${PAM_RHOST}" \
 $NTFYURL
fi

# If session was closed then post this message with priority level 1
if [ "${PAM_TYPE}" = "close_session" ]; then
  curl -H "Title: ${HOSTNAME} -- ssh logout" \
 -u :$NTFYTOKEN \
 -H "Priority: 1" \
 -d "user: ${PAM_USER}
ip/host: ${PAM_RHOST}" \
 $NTFYURL
fi

You will need to add your token to the NTFYTOKEN= var as well as your notify domain to the NTFYURL=. The URL will be https://notify.yourdomain.com/TOPIC with /TOPIC being the name of the topic you gave your user access to earlier. A completed file will look similar to the example below. Info on message priority and x-tags can be found in the ntfy docs here, and here respectively.

Save and close the file: CTRL + x then y.

  


#!/bin/bash

# User access token and url to post notification messages
NTFYTOKEN=tk_6re713axsggyetxtg15qee1d9614z
NTFYURL=https://notify.yourdomain.com/ssh_logins

# If session was opened then post this message with priority level 5
if [ "${PAM_TYPE}" = "open_session" ]; then
  curl -H "Title: ${HOSTNAME} -- ssh login" \
 -u :$NTFYTOKEN \
 -H "Priority: 5" \
 -H "X-Tags: warning" \
 -d "user: ${PAM_USER}
ip/host: ${PAM_RHOST}" \
 $NTFYURL
fi

# If session was closed then post this message with priority level 1
if [ "${PAM_TYPE}" = "close_session" ]; then
  curl -H "Title: ${HOSTNAME} -- ssh logout" \
 -u :$NTFYTOKEN \
 -H "Priority: 1" \
 -d "user: ${PAM_USER}
ip/host: ${PAM_RHOST}" \
 $NTFYURL
fi

Next we need to edit the sshd file for pam.d.

sudo nano /etc/pam.d/sshd

And drop this into the bottom of the file.

session   optional   pam_exec.so /bin/sshd-login-ntfy

Save and close the file CTRL + x then y.

Configuring the Android Mobile App

You can do this on iOS as well BUT you won’t get instant delivery without relying on a central server. See here for more details.

First download the app from Google Play Store, here (f-droid), and after it installs launch it.

At first launch you will be greeted by a blank screen with zero topics. To set the app up for your self-hosted instance open the settings menu (3 dot menu > settings) and go to “Manage Users” under General. Tap on “Add new user” and in the Service URL field enter the URL you used for your ntfy instance above. The Username & Password fields will be the second user you created earlier. After the fields are filled out tap “ADD USER.” Your user section should then look like the image to the right.

Tap the back arrow (or swipe back) and enter the “Default server” setting and enter your ntfy URL again. Now tap or swipe back to the home screen. To disable the instant delivery notification, that shows in your notification drawer, swipe up to the app switcher and long press on the ntfy app icon. Then tap “App info”. On the app settings page, tap on Notifications and disable the “Subscription Service” notification under “Other.”

Subscribing to a Topic

On the app home screen, tap on the green “+” button and enter the name of the topic you created earlier during the user access step. Then tap “SUBSCRIBE.” If done correctly your new topic should show up on your app’s home screen.

The settings of the subscription can be adjusted by tapping on it and then tap on the 3 dots menu. Then tap on “Subscription Settings” from the menu. Here you can edit notification preferences as well as the icon and display name of the subscription.

Sending a Notification Message

We can send a notification message by either logging out of the host you used to set up ntfy (if you are doing this over ssh) or by logging into a host with the above sshd-login-ntfy script active. Or simply running:

  
curl -H "Title: ${HOSTNAME} -- ssh login" -u :token_here https://notify.yourdomain.com/ssh_logins -d "This is a test message"

Example of notifications in action

Conclusion

There is so much more you can do with ntfy, but I’ll leave that up to the imagination.. Otherwise, this post would be extremely long, haha! I will however, include more examples in the blog-files repository and the docs have great examples on how to send messages as well as services with direct integrations.

As always if you have any questions feel free to reach out via the comments or email! Happy homelabbing!

PS: If you like ntfy and end up using it, make sure to drop a tip to the dev! https://github.com/binwiederhier/ntfy#sponsors

References and other resources:

As I was working on this post Network Chuck posted a video on it. Which I have linked down below as well as another quicker (older) overview of ntfy from Kris Occhipinti.

Video overview by NetworkChuck and Kris Occhipinti

My Caddy docker image includes the Cloudflare DNS plugin if you do not want to build it yourself. ↩ ↩²
You can also use the binary version. Both this and the docker image are built from the source. ↩ ↩²

homelab

This post is licensed under CC BY 4.0 by the author.