Monitoring Nvidia GPUs via Telegraf

The nvidia-smi plugin for Telegraf basically gives you an overview of your GPU usage in the most current iteration in v1.10.4. This “guide” assumes you are using Windows as your host OS. Linux should be fairly easy to get going as long as you know where your nvidia-smi executable is located.

If you do not have Telegraf installed, check out my guides here.

Create a new conf file in telegraf.d folder.

notepad.exe C:\telegraf\telegraf.d\nvidiasmi.conf

Paste the following into the new file and save/close it.

1
# Pulls statistics from nvidia GPUs attached to the host
2
[[inputs.nvidia_smi]]
3
  ## Optional: path to nvidia-smi binary, defaults to $PATH via exec.LookPath
4
  bin_path = "C:\\Program Files\\NVIDIA Corporation\\NVSMI\\nvidia-smi.exe"
5

6
  ## Optional: timeout for GPU polling
7
  timeout = "5s"

Restart Telegraf.

1
net stop telegraf
2
net start telegraf

Once you have verified Telegraf is reporting Nvidia stats, you can start creating your panels in Grafana. Use nvidia-smi from your telegraf data source to build the panels.

Update February 2020

Telegraf recently updated its SMI plugin to include more data retrieval. This new data can be used to create more monitoring panels. Here is a list of the most recent fields that are now returned:

• clocks_current_graphics
• clocks_current_memory
• clocks_current_sm
• clocks_current_video
• encoder_stats_average_fps
• encoder_stats_average_latency
• encoder_stats_session_count
• fan_speed
• memory_free
• memory_total
• memory_used
• pcie_link_gen_current
• pcie_link_width_current
• power_draw
• temperature_gpu
• utilization_gpu
• utilization_memory