James Routley

I have an IoT button (SONOFF wireless push button) that when pressed I want to save the value in a database for later tracking. In this post - I document my journey yak-shaving my way to get exactly what I’m looking for.

TL;DR

Architecture

Zigbee

While I’ve previously set up a wifi-based smart plug, I have no experience with these buttons. This particular button uses the Zigbee protocol for communication. I also have a Zigbee gateway purchased along with the button (ConBee II). The whole process involves registering the button to the gateway then accessing its actions.

There exists a Python library to interact with Zigbee and this gateway directly, but I didn’t choose to go with this for two reasons. I wanted to write my code in Elixir and I wanted a more robust solution in case I decide to add additional devices.

Digging deeper, this is when I found zigbee2mqtt. It bridges Zigbee to MQTT. As MQTT is just pub-sub under the hood, I thought this would be a good choice. Zigbee2MQTT turned out to be a whole application that allows registering and configuring devices. I connected the Zigbee gateway USB to one of my servers, then brought up the Docker image using Nix OCI containers:

# zigbee.nix

virtualisation.oci-containers = {
  containers = {
    zigbee2mqtt = {
      image = "ghcr.io/koenkk/zigbee2mqtt:2.5.1";
      autoStart = true;
      environment.TZ = "America/New_York";
      volumes = [
	    "/etc/zigbee2mqtt/data:/app/data"
	    "/run/udev:/run/udev:ro"
      ];
      extraOptions = [
	    "--network=host"
	    "--device=/dev/ttyACM0:/dev/ttyACM0"
      ];
    };
  };
};

I didn’t need to put the container on the host network, but in this case it was the quickest way to get up and running. I also passthrough the Zigbee gateway into the container with --device option.

On loading up Zigbee2MQTT, it looks like it can’t act as an MQTT broker, so we need to run our own.

MQTT Broker

From my research, the two options I saw for running an MQTT broker were mosquitto and emqx. Mosquitto seems to be the simpler and leaner of the two, but I saw that emqx offered a Postgres “connector”. Ultimately, I wanted my button to store the clicks in a database so figured this would do the trick!

Bringing up emqx was relatively uneventful. I planned on running the MQTT broker on my Kubernetes cluster and saw that it has a Helm chart. It was nice to be able to fully configure it via YAML, including creating a custom iot client. (They also have a Kubernetes operator for clustering and high availability, but I didn’t need any of that.)

The final config looked something like this:

# values.yaml

ingress:
  dashboard:
    enabled: true
    ingressClassName: nginx
    hosts:
      - dashboard.mqtt.${CLUSTER_DOMAIN_INTERNAL}

extraVolumeMounts:
  - name: bootstrap-clients
    mountPath: /opt/emqx/etc/auth-built-in-db-bootstrap.csv
    subPath: EMQX_CLIENT_BOOTSTRAP_FILE
    readOnly: true

extraVolumes:
  - name: bootstrap-clients
    secret:
      secretName: emqx-prod-secrets

emqxConfig:
  EMQX_AUTHENTICATION__1__ENABLE: true
  EMQX_AUTHENTICATION__1__BACKEND: built_in_database
  EMQX_AUTHENTICATION__1__MECHANISM: password_based
  EMQX_AUTHENTICATION__1__USER_ID_TYPE: username
  EMQX_AUTHENTICATION__1__PASSWORD_HASH_ALGORITHM__NAME: bcrypt

I did run into an issue testing this out. Try as I might, I couldn’t connect to the MQTT broker from outside the cluster! Originally, I had set up ingress for mqtt.example.lan for port 1883 but after some digging, I realized two things:

the mqtt protocol works at the TCP level
ingress-nginx (my Kubernetes ingress of choice) doesn’t play too well proxying anything below HTTP

I found this article about exposing TCP/UDP services outside the cluster. The gist of it is that nginx supports a ConfigMap for TCP and UDP services, with a key-value pair being the port to open and the Service to proxy to. Thankfully, the Helm chart abstracts this to a value and for emqx, it was:

values:
  tcp:
    "1883": "default/emqx:1883"

After setting this up, I was able to test this out locally using the mosquitto-cli and it worked! I configured Zigbee2MQTT to use this broker and it successfully connected.

Elixir App

With the “base” infrastructure up and running, we can press the button and it shows up as a message on the MQTT topic: zigbee2mqtt/button/action. All we need to do now is to hook up our application to this MQTT broker and react to the button press.

The app itself is pretty simple: subscribe to the MQTT topic, press the button, send action to a Postgres database. You would think that something like this already exists, but at least with EMQX, it’s gated behind an enterprise license…

I decided to use Phoenix as the “framework” to build the app, stripped down to the bare minimum. This removes practically the whole frontend and is great for API-only applications:

$ mix phx.new button_app --no-assets --no-html --no-mailer --binary-id

(--binary-id is used to assign UUIDs to DB records and is probably overkill here, but I generally default to that nowadays.)

I used Ecto to create the DB table schema:

# lib/button_app/count.ex

defmodule ButtonApp.Count do
  use Ecto.Schema
  import Ecto.Changeset

  @primary_key {:id, :binary_id, autogenerate: true}
  @foreign_key_type :binary_id
  schema "counts" do
    field :action, :string

    timestamps(type: :utc_datetime)
  end

  @doc false
  def changeset(count, attrs) do
    count
    |> cast(attrs, [:action])
    |> validate_required([:action])
  end
end

The next step was to hook into the MQTT broker. I found there were no good Elixir-native MQTT libraries, but thankfully we can hook into the Erlang ecosystem and there was one by the creators of EMQX: emqtt.

Let’s first create some runtime configuration for MQTT:

# config/runtime.ex

config :button_app, :emqtt,
  host: System.get_env("MQTT_HOST", "mqtt.example.lan"),
  port: 1883,
  username: "iot",
  password: System.get_env("MQTT_CLIENT_PASSWORD"),
  clientid: "button_counter",
  clean_start: false,
  name: :emqtt

config :button_app, :topic, "zigbee2mqtt/button/action"

We can then retrieve this within our module like so:

emqtt_opts = Application.get_env(:button_app, :emqtt)
topic = Application.get_env(:button_app, :topic)

The emqtt documentation says to use a GenServer to run the client and messages come in the form of a {:publish, payload}:

# lib/button_app/subscriber.ex

defmodule ButtonApp.Subscriber do
  use GenServer

  def start_link([]) do
    GenServer.start_link(__MODULE__, [])
  end

  @impl true
  def init([]) do
    emqtt_opts = Application.get_env(:button_app, :emqtt)
    topic = Application.get_env(:button_app, :topic)
    {:ok, pid} = :emqtt.start_link(emqtt_opts)

    state = %{
      topic: topic,
      pid: pid
    }

    {:ok, state, {:continue, :start_emqtt}}
  end

  @impl true
  def handle_continue(:start_emqtt, %{pid: pid, topic: topic} = state) do
    {:ok, _} = :emqtt.connect(pid)
    {:ok, _, _} = :emqtt.subscribe(pid, {topic, 0})
    {:noreply, state}
  end

  @impl true
  def handle_info({:publish, payload}, state) do
    # ButtonApp.add_count(payload) # unimplemented
    IO.inspect(payload, label: "mqtt_payload")
    {:noreply, state}
  end
end

We handle receiving the message via the handle_info call. Running the application and pressing the button, I was able to receive a message!

mqtt_payload: %{action: "single"}

It looks like the “action” is the type of button press, as the button itself can register different types: “single”, “double”, “long”.

Now all we need to do is implement ButtonApp.add_count() and just send it to the database. With Ecto - this is pretty easy:

# button_app.ex
alias ButtonApp.Count

def add_count(payload) do
  %Count{}
  |> Count.changeset(%{action: payload})
  |> Repo.insert
end

I tested this out and I was able to see a new button press registered in my development database. Now let’s ship it!

Deploy it!

Phoenix has a nice mix task that auto-generates a Dockerfile for us.

mix phx.gen.release --docker

The only thing it doesn’t do is auto-migrate the database when deploying, so let’s fix that by adding the migrate script:

# Dockerfile
# ... some other things

CMD ["sh", "-c", "/app/bin/migrate && /app/bin/server"]

We build this image and can now deploy it via a Nix OCI container:

virtualisation.oci-containers = {
  containers = {
    button_app = {
      login = {
	    registry = "https://registry.example.lan";
	    username = "mei";
	    passwordFile = config.sops.secrets.registry_password.path;
      };
      image = "registry.example.lan/registry/button_app:latest";
      environmentFiles = [
	    config.sops.templates.button_app_env.path
      ];
      extraOptions = [
	    "--pull=newer"
	    "--network=host"
      ];
    };
  };
};

I probably could have deployed this to my Kubernetes cluster since it doesn’t need to be on the host network and can talk directly to the MQTT broker in-cluster.

I also run Grafana - so I hooked it up to the database and after a few times clicking the button, we can now do a visualization.

Grafana visualization

Outro

I could have just put all of this on a little box and sent it, but I wanted something that I can expand into for the future if I want to build more apps. I also skipped talking about setting up the database, Docker registry, and Nix secret management because I think they would have distracted from the overall post. Perhaps in the future I can go over those in more detail.