Exometer Report via Phoenix Channels

LIVE DEMO: https://exometer-phoenix-channel-demo.herokuapp.com/

Exometer is an incredibly robust package for Erlang to monitor system performance as well as any other type of events one might be interested in (database queries, actions per second, etc). It integrates easily with external statistic collection services such as Graphite through a reporter functionality. By default it includes reporters that send data to StatsD, AMQP messages and other common interfaces.

In this article I will create a custom Exometer reporter module that will send the data through a Phoenix.Channel.

Exometer Report interface

The first thing I needed to do was look at the functions that Exometer expects from its reporters. Looking at a simple tty report implementation that is included in the Exometer library, we can see that there are two essential function calls exometer_init/1 and exometer_report/5, which we need to implement in any custom library. I also included the other functions and modeled them after the mostly benign behavior shown in the example reporter:

defmodule ExometerReportPhoenixChannel do
  @behaviour :exometer_report

  # Initializes the exometer_report with passed params
  # Requires :channel and :app_name options
  def exometer_init(opts) do
    {:ok, opts}
  end

  # Converts the data passed by Exometer and relays it to the channel
  def exometer_report(metric, data_point, extra, value, opts) do
    id = opts[:channel] <> ":" <> name(metric, data_point)

    payload = %{
      value: value,
      extra: extra,
      timestamp: :os.system_time(:milli_seconds)
    }

    Module.concat([opts[:app_name], Endpoint])
    |> apply(:broadcast, [id, "change", payload])

    {:ok, opts}
  end

  # Public function that should be implemented according to
  # https://github.com/Feuerlabs/exometer_core/blob/master/src/exometer_report_tty.erl
  def exometer_call(_, _, opts), do: {:ok, opts}
  def exometer_cast(_, opts), do: {:ok, opts}
  def exometer_info(_, opts), do: {:ok, opts}
  def exometer_newentry(_, opts), do: {:ok, opts}
  def exometer_setopts(_, _, _, opts), do: {:ok, opts}
  def exometer_terminate(_, _), do: nil

  defp name(metric, data_point) do
    Enum.join(metric, "_") <> "_" <> "#{data_point}"
  end
end

exometer_init/1 takes a map of options that need to be defined when you initialize Exometer and define your statistics as well as the reporters and subscribers. To keep the reporter as generic as possible I though it made sense to specify the name of the channel you want to send the stats to. Also the reporter should not be tied to any specific application and as such you should also specify an app_name in the configuration.

exometer_report/5 takes the name of the metric, the specific data_point, some extra content I am still not sure about, the actual value, and the opts map specified during exometer_init/1. Next it concatenates the specified :channel name with the name of the data_point and metric to form a channel:topic that can be used in Phoenix channels. Following that it creates a payload map that includes the value of the metric, whatever might be in extra and a timestamp. Finally it uses the specified :app_name and sends the Endpoint module the broadcast/3 function which will send a “change” event to all subscribers connected to the id channel. The payload gets automatically converted into JSON.

Loading the reporter

As mentioned earlier Exometer is an Erlang package and when you configure it, it expects Erlang conventions. We, however, have built an Elixit module: defmodule ExometerReportPhoenixChannel. So how will an Erlang application load an Elixir module? It does not need to as all Elixir just gets turned into Erlang and prefixed with Elixir. namespace. This means that if you need to reference an Elixir function from Erlang, you just need to apply that prefix. Here is my ecometer_config.exs in which I use this:

polling_interval = 1_000
memory_stats     = ~w(atom binary ets processes total)a

config :exometer,
  predefined:
    [
      {
        ~w(erlang memory)a,
        {:function, :erlang, :memory, [], :proplist, memory_stats},
        []
      },
      {
        ~w(erlang statistics)a,
        {:function, :erlang, :statistics, [:'$dp'], :value, [:run_queue]},
        []
      },
    ],
  reporters:
    [
      "Elixir.ExometerReportPhoenixChannel": [
        app_name: "ExometerPhoenixChannelDemo",
        channel: "stats"
      ],
    ],
  report: [
    subscribers:
      [
        {
          :"Elixir.ExometerReportPhoenixChannel",
          [:erlang, :memory], memory_stats, polling_interval, true
        },
        {
          :"Elixir.ExometerReportPhoenixChannel",
          [:erlang, :statistics], :run_queue, polling_interval, true
        }
      ]
  ]

Note the :"Elixir.ExometerReportPhoenixChannel". You can also see that I have declared my app_name and channel.

Visualizing the data

To graph the data I used the excellent HighCharts library. All I needed to do was hook up the channels so that any incoming data would be automatically included:

import {Socket} from "deps/phoenix/web/static/js/phoenix"

let socket = new Socket("/socket", {params: {token: window.userToken}})

socket.connect()

String.prototype.capitalize = function() {
    return this.charAt(0).toUpperCase() + this.slice(1);
}

$(function () {
  let chart = new Highcharts.Chart({

    chart: {
        renderTo: 'container',
        defaultSeriesType: 'spline'
    },
    title: {
        text: 'Erlang Memory'
    },
    subtitle: {
        text: "Click on label to show/hide series"
    },
    xAxis: {
        type: 'datetime',
        tickPixelInterval: 150,
        maxZoom: 20 * 1000
    },
    yAxis: {
        minPadding: 0.2,
        maxPadding: 0.2,
        title: {
            text: 'Bytes',
            margin: 80
        }
    }
  });

  let chart_1 = new Highcharts.Chart({

    chart: {
        renderTo: 'container_1',
        defaultSeriesType: 'spline'
    },
    title: {
        text: 'Erlang Run queue'
    },
    xAxis: {
        type: 'datetime',
        tickPixelInterval: 150,
        maxZoom: 20 * 1000
    },
    yAxis: {
        minPadding: 0.2,
        maxPadding: 0.2,
        title: {
            text: 'Processes',
            margin: 80
        }
    }
  });

  function join_channel(name, target_chart){
    var channel = socket.channel(name, {})
    channel.join()
      .receive("ok", data => {
        console.log("Joined channel", name)
      })
      .receive("error", resp => {
        console.log("Unable to join topic", name)
      })

    channel.on("change", stat => {
      var series = target_chart.get(name)
      var shift = series.data.length > 15
      var point = [stat.timestamp, stat.value]
      series.addPoint(point, true, shift);
    })
  }

  var topics = ["atom", "binary", "ets", "processes", "total"]

  for(var t of topics){
    var topic = "stats:erlang_memory_" + t
    chart.addSeries({id: topic, name: t.capitalize()})
    join_channel(topic, chart)
  }

  var topic = "stats:erlang_statistics_run_queue"
  chart_1.addSeries({id: topic, name: "Run queue"})
  join_channel(topic, chart_1)

});

Conclusion

You can find a live version of the site here and the source code here