# Temperature Logging: Redux

Previously as I was experimenting with logging the temperature using a Raspberry Pi (to monitor the temperatures experienced by fermenting cider), I noted that the Pi was something of a terrible hack, and it should be possible to do more efficiently with some slightly less common hardware.

I decided that improved version would be interesting to build for use at home, since it’s both kind of fun to collect data like that, and actually knowing the temperature in the house is useful at times. The end result of this project is that I can generate graphs like the one below of conditions around the house:

## Software requirements

My primary requirement for home monitoring of this sort is that it not depend on a proprietary hub (especially not one that depends on an external service that might go away without warning), and I’d also like something that can be integrated with my existing (but minimal) home automation setup that’s based around Home Assistant running on my home server.

Given my main software is open source it should be possible to integrate an arbitrary solution with it, with varying amount of reverse engineering and implementation necessary. Because reverse-engineering services like that is not my idea of fun, it’s much preferable to find something that’s already supported and take advantage of others’ work. While I don’t mind debugging, I don’t want to build an integration from scratch if I don’t need to.

## Hardware selection

As observed last time, the “hub” model for connecting “internet of things” devices to a network seems to be the best choice from a security standpoint- the only externally-visible network device is the hub, which can apply arbitrary security policies to communications between devices and to public networks (in the simplest case, forbidding all communications with public networks). Indeed, recent scholarly work (PDF) suggests systems that work on this model but apply more sophisticated policies to communications passing through the hub.

With that in mind, I decided a Zigbee network for my sensors would be appropriate- the sensors themselves have no ability to talk to public networks because they don’t even run an Internet Protocol stack, and it’s already a fairly common standard for communication. Plus, I was able to get several of the previously-mentioned Xiaomi temperature, humidity and barometric pressure sensors for about $10 each; a quite reasonable cost, given they’re battery powered with very long life and good wireless range. Home assistant already has some support for Zigbee devices; most relevant here seems to be its implementation of the Zigbee Home Automation application standard. Though the documentation isn’t very clear, it supports (or, should support) any radio that communicates with a host processor over a UART interface and speaks either the XBee or EZSP serial protocol. Since the documentation for Home Assistant specifically notes that the Elelabs Zigbee USB adapter is compatible, I bought one of those. Its documentation includes a description of how to configure Home Assistant with it and specifically mentions Xiaomi Aqara devices (which includes the sensors I had selected), so I was confident that radio would meet my needs, though unsure of exactly what protocol was actually used to communicate with the radio over USB at the time I ordered it. ## Experimenting Once I received the Zigbee radio-on-a-usb-stick, I immediately tried to manually drive it using whatever libraries I could use to set up a network and get one of my sensors connected to it. This ended up not working, but I did learn a lot about how the radio adapter is meant to work. For working with it in Python, the Elelabs documentation points to bellows, a library providing EZSP protocol support for the zigpy Zigbee stack. It also includes a command-line interface exposing some basic commands, perfect for the sort of experimentation I wanted to do. Getting connected was easy; I plugged the USB stick into my Linux workstation and it appeared right away as a PL2303 USB-to-serial converter. Between this and noting that bellows implements the EZSP protocol, I inferred that the Elelabs stick is a Silicon Labs EM35x microcontroller running the EmberZNet stack in a network coordinator mode, with a PL2303 exposing a UART over USB so the host can communicate with the microcontroller (and the rest of the network) by speaking EZSP. Having worked that out and made sense of it, I printed out a label for the stick that says what it is (“Elelabs Zigbee USB adapter”) and how to communicate with it (EZSP at 57600 baud) since the stick is completely unmarked otherwise and being able to tell what it does just by looking at it is very helpful. Trying to use the bellows CLI, the status output seemed okay and the NCP was running. In order to connect one of my sensors, I then needed to figure out how to make the sensor join the network after using bellows permit to let new devices join the network. The sensors each came with a little instruction booklet, but it was all in Chinese. With the help of Google Translate, I was able to take photos of it and find the important bit- holding the button on the sensor for about 5 seconds until the LED blinks three times will reset it, at which point it will attempt to join an open network. On trying to run bellows permit prior to resetting a sensor to get it on the network, I encountered an annoying bug- it didn’t seem to do anything, and Python emitted a warning: RuntimeWarning: coroutine 'permit' was never awaited. I dug into that a little more and found the libraries make heavy use of PEP 492 coroutines, and the warning was fairly clear that a function was declared async when it shouldn’t have been (or its coroutine wasn’t then given to an event loop) so the function actually implementing permit never ran. I eventually tracked down the problem, patched it locally and filed a bug which has since been fixed. Having fixed that bug, I continued to try to get a sensor on my toy network but was ultimately (apparently) unsuccessful. I could permit joins and reset the sensor and see debug output indicating something was happening on the network, but never any conclusive messages saying a new device had joined and rather a lot of messages along the lines of “unrecognized message.” I couldn’t tell if it was working or not, so moved on to hooking up Home Assistant. ### Setting up Home Assistant Getting set up with Home Assistant was mostly just a matter of following the guide provided with the USB stick, but using my own knowledge of how to set up the software (not using hassio). Configuring the zha component and pointing it at the right USB path is pretty easy. I did discover that specifying the database_path for the zha component alone is not enough to make it work; if the file doesn’t already exist setup just fails. Simply creating an empty file at the configured path is enough- apparently that file is an sqlite database that zigpy uses to track known devices. Still following the Elelabs document, I spent a bit of time invoking zha.permit and trying to get a sensor online to no apparent success. After a little more searching, I found discussion on the Home Assistant forums and in particular one user suggesting that these particular sensors are somewhat finicky when joining a network. They suggested (and my findings agree) that holding the button on the sensor to reset it, then tapping the button approximately every second for a little while (another 5-10 seconds) will keep it awake long enough to successfully join the network. The keep-awake tapping approach did eventually work, though I also found that Home Assistant sometimes didn’t show a new sensor (or parts of a new sensor, like it might show the temperature but not humidity or pressure) until I restarted it. This might be a bug or a misconfiguration on my part, but it’s minor enough not to worry about. At this point I’ve verified that my software and hardware can all work, so it’s time to set up the permanent configuration. ## Permanent configuration As mentioned above, I run Home Assistant on my Linux home server. Since I was already experimenting on a Linux system, that configuration should be trivial to transfer over, but for one additional desire I had: I want more freedom in where I place the Zigbee radio, in particular not just plugged directly into a free USB port on the server. Putting it in a reasonably central location with other radios (say, near the WiFi router) would be nice. A simple solution might be a USB extension cable, but I didn’t have any of those handy and strewing more wires about the place feels inelegant. My Internet router (a TP-Link Archer C7 running OpenWrt) does have an available USB port though, so I suspected it would be possible to connect the Zigbee radio to the router and make it appear as a serial port on the server. This turned out to be true! ### Serial over network To find the solution for running a serial port over the network, I first searched for existing protocols; it turns out there’s a standard one that’s somewhat commonly used in fancy networking equipment, specified by RFC 2217. RFC 2217 specifies a set of extensions to Telnet allowing serial port configuration (bit rate, data bits, parity, etc) and flow control over Telnet. Having identified a protocol that does what I want, it’s then a matter of finding software that works as a client (assuming I’ll be able to find or write a suitable server). Suitable clients are somewhat tricky however, since from an applicaton perspective UART use on Linux involves making specialized ioctls to the device to configure it, then reading and writing bytes as usual. Making an RFC2217 network serial device appear like a local device would seem to involve writing a kernel driver that exports a new class of RFC2217 device nodes supporting the relevant ioctls- none exists.1 An alternate approach (not using RFC 2217) might be USB/IP, which is supported in mainline Linux and allows a server to bind USB devices physically connected to it to a virtual USB controller that can then be remotely attached to a different physical machine over a network. This seems like a more complex and potentially fragile solution though, so I put that aside after learning of it. Since Linux doesn’t have any kernel-level support for remote serial ports, I needed to search for support at the application level. It turns out bellows uses pyserial to communicate with radios, and pySerial is a quite featureful library- while most users will only ever provide device names like COM1 or /dev/ttyUSB0, it supports a range of more exotic URLs specifying connections, including RFC 2217. So given a suitable server running on a remote machine, I should be able to configure Home Assistant to use a URL like rfc2217://zigbee.local:25 to reach the Zigbee radio. ### Serial server The next step in setting up the Zigbee radio plugged into the router is finding an application that can expose a PL2303 over the network with the RFC 2217 protocol. That turned out to be a short search, where I quickly discovered ser2net which does the job and is already packaged for OpenWRT. Installing it on the router was trivial, though I also needed to be sure the kernel module(s) required to expose the USB-serial port were available: # opkg install kmod-usb-serial-pl2303 ser2net Having installed ser2net, I still had to figure out how to configure it. While the documentation describes its configuration format, I know from experience that configuring servers on OpenWRT is usually done differently (as something of a concession to targeting embedded systems without much storage). I quickly found that the package had installed a sample configuration file at /etc/config/ser2net: config ser2net global option enabled 1 config controlport option enabled 0 option host localhost option port 2000 config default option speed 115200 option databits 8 option parity 'none' option stopbits 1 option rtscts false option local false option remctl true config proxy option enabled 0 option port 5000 option protocol telnet option timeout 0 option device '/dev/ttyAPP0' option baudrate 115200 option databits 8 option parity 'none' option stopbits 1 # option led_tx 'tx' # option led_rx 'rx' option rtscts false option local false option xonxoff false Unfortunately, this configuration doesn’t include any comments so the reader is force to guess the meaning of each option. They mostly correspond to words that appear in the ser2net manual, but I didn’t trust guesses so went digging in the OpenWRT packages source code and found the script responsible for converting /etc/config/ser2net into an actual configuration file when starting ser2net. My initial guess at the configuration I wanted looked something like this: config proxy option enabled 1 option port 5000 option protocol telnet option timeout 0 option device '/dev/ttyUSB0' option baudrate 57600 option remctl true The protocol is specified as telnet because RFC 2217 is a layer on top of telnet (my first guess was that I actually wanted raw until actually reading the RFC and seeing it was a set of telnet extensions), and the device is the device name that I found the Zigbee stick appeared as when plugged into the router.2 Unfortunately, this configuration didn’t work and pyserial gave gack a somewhat perplexing error message: serial.serialutil.SerialException: Remote does not seem to support RFC2217 or BINARY mode [we-BINARY:False(INACTIVE), we-RFC2217:False(REQUESTED)]. Without much visibility into what the serial driver was trying to do, I opted to examine the network traffic with Wireshark. I first attempted to use the text-mode interface (tshark -d tcp.port==5000,telnet -f 'port 5000'), but quickly gave up and switched to the GUI instead. I captured the traffic passing between the server and router, but there was almost nothing! The client (pyserial) was sending some Telnet negotiation messages (DO ECHO, WILL suppress go ahead and COM port control), then nothing happened for a few seconds and the connection closed. Since restarting Home Assistant for every one of these serial tests was quite cumbersome, at this point I checked if pyserial includes any programs suitable for testing connectivity. It happily does, provided in my distribution’s package as miniterm.py. Running miniterm.py rfc2217://c7:5000 failed in the same way, so I had a quicker debugging tool. At this point the problem seems like it’s at the server side, so I stopped the ser2net server on the router and started one in the foreground, with a custom configuration specified on the command line: $ /etc/init.d/ser2net stop
$ser2net -n -d -C '5000:telnet:0:/dev/ttyUSB0:57600 remctl' ser2net[14914]: Unable to create network socket(s) on line 0 While ser2net didn’t outright fail, it did print a concerning error message. Does it work if I change the port it’s listening on? $ ser2net -n -d -C '1234:telnet:0:/dev/ttyUSB0:57600 remctl'

And then running miniterm.py succeeds, leaving me with a terminal I could type into (but didn’t, since I don’t know how to speak EZSP with my keyboard).

$miniterm.py rfc2217://c7:1234 57600 --- Miniterm on rfc2217://c7:1234 57600,8,N,1 --- --- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H --- --- exit --- I discovered after a little digging (netstat -lnp) that miniupnpd was already listening on port 5000 of the router, so changing the port fixes the confusing problem. A different sample port in the ser2net configuration would have prevented such an issue, as would ser2net giving up when it fails to bind to a requested port instead of printing a message and pretending nothing happened. But at least I didn’t have to patch anything to make it work. With ser2net listening on port 2525 instead, Home Assistant can connect to it (hooray!). But it immediately throws a different error: NotImplementedError: write_timeout is currently not supported. I’ve found another bug in a rarely-exercised corner of this software stack, have I? Well, kind of. Finding that error message in the pyserial source, something is trying to set the write timeout to zero and it’s simply not implemented in pyserial for RFC2217 connections. This is ultimately because Home Assistant (as alluded to earlier with bellows and zigpy) is all coroutine-based so it uses pyserial-asyncio to adapt the blocking APIs provided by pyserial to something that works nicely with coroutines running on an event loop. When pyserial-asyncio tries to set non-blocking mode by making the timeout zero, we find it’s not supported. def _reconfigure_port(self): """Set communication parameters on opened port.""" if self._socket is None: raise SerialException("Can only operate on open ports") # if self._timeout != 0 and self._interCharTimeout is not None: # XXX if self._write_timeout is not None: raise NotImplementedError('write_timeout is currently not supported') # XXX While I could probably implement non-blocking support for RFC 2217 in pyserial, that seemed rather difficult and not my idea of fun. So instead I looked for a workaround- if RFC 2217 won’t work, does pyserial support a protocol that will? The answer is of course yes: I can use socket:// for a raw socket connection to the ser2net server. This sacrifices the ability to change UART parameters (format, baud rate, etc) on the fly, but since the USB stick doesn’t support changing parameters on the fly anyway (as far as I can tell), this is no problem. ## Final configuration The ser2net configuration that I’m now using looks like this: config proxy option enabled 1 option port 2525 option protocol raw option timeout 0 option device '/dev/ttyUSB0' option baudrate 57600 option remctl 0 And the relevant stanza in Home Assistant configuration: (The baud rate needs to be specified, but pyserial ignores it for socket:// connections.) zha: usb_path: 'socket://c7:2525' database_path: /srv/homeassistant/.homeassistant/zigbee.db baudrate: 57600 After ensuring the zigbee.db file exists and restarting Home Assistant to reload the configuration, I was able to pair all three sensors by following the procedure defined above: call the permit service in Home Assistant, then reset the sensor by holding the button until its LED blinks three times, then tap the button every second or so for a bit. I did observe some strange behavior on pairing the sensors that made me think they weren’t pairing correctly, like error messages in the log (ERROR (MainThread) [homeassistant.components.sensor] Setup of platform zha is taking longer than 60 seconds. Startup will proceed without waiting any longer.) and some parts of each sensor not appearing (the temperature might be shown, but not humidity or pressure). Restarting Home Assistant after pairing the sensors made everything appear as expected though, so there may be a bug somewhere in there but I can’t be bothered to debug it since there was a very easy workaround. ## Complaining about async I/O It’s rather interesting to me that the major bugs I encountered in trying to set up this system in a slightly unusual configuration were related to asynchronous I/O running in event loops- this is an issue that’s become something of my pet problem, such that I will argue to just about anybody who will listen that asynchronous I/O is usually unnecessary and more difficult to program. That I discovered two separate bugs in the tools that make this work relating to running asynchronous I/O in event loops seems to support that conclusion. If Home Assistant simply spawned threads for components I believe it would simplify individual parts (perhaps at the cost of some slightly more complex low-level communication primitives) and make the system easier to debug. Instead, it runs all of its dependencies in a way they are not well-exercised in, presumably in search of “maximum performance” that seems entirely irrelevant when considering the program’s main function is acting as a hub for a variety of external devices. I have (slowly) been working on distilling all these complaints into a series of essays on the topic, but for now this is a fine opportunity to wave a finger at something that I think is strictly worse because it’s evented. ## Conclusion I’m pretty happy with the sensors and software configuration I have now- the sensors are tiny and unobtrusive, while the software does a fine job of logging data and presenting live readings for my edification. I’d like to also configure a “real” database like InfluxDB to store my sensor readings over arbitrarily long time periods (since Home Assistant doesn’t remember data forever, reasonably so), which shouldn’t be too difficult (it’s supported as a module) but is somewhat unrelated to setting up Zigbee sensors in the first place. Until then, I’m pretty happy with these results despite the fact that I think the developers have made a terrible choice with evented I/O. 1. I did find somebody asking for input on the implementation of exactly that, but it looks like nothing ever came of it. A reply suggesting an application at the master end of a pty (pseudoterminal) suggests an interesting alternate option, but it doesn’t appear to be possible to receive parameter change requests from a pty (though flow control is exposed when running in “packet mode”). [return] 2. I was concerned at the outset that the router might be completely unable to see the Zigbee stick, since apparently the Archer C7 doesn’t include a USB 1.1 OHCI or UHCI controller, so it’s incapable of communicating at all with low-speed devices like keyboards! I’ve heard (but not verified myself) that connecting a USB 2.0 hub will allow the router to communicate with low-speed devices downstream of the hub as a workaround. [return] # Building a terrible 'IoT' temperature logger I had approximately the following exchange with a co-worker a few days ago: Them: “Hey, do you have a spare Raspberry Pi lying around?” Me: [thinks] “..yes, actually.” T: “Do you want to build a temperature logger with Prometheus and a DS18B20+? M: “Uh, okay?” It later turned out that that co-worker had been enlisted by yet another individual to provide a temperature logger for their project of brewing cider, to monitor the temperature during fermentation. Since I had all the hardware at hand (to wit, a Raspberry Pi 2 that I wasn’t using for anything and temperature sensors provided by the above co-worker), I threw something together. It also turned out that the deadline was quite short (brewing began just two days after this initial exchange), but I made it work in time. ## Interfacing the thermometer As noted above, the core of this temperature logger is a DS18B20 temperature sensor. Per the manufacturer: The DS18B20 digital thermometer provides 9-bit to 12-bit Celsius temperature measurements … communicates over a 1-Wire bus that by definition requires only one data line (and ground) for communication with a central microprocessor. … Each DS18B20 has a unique 64-bit serial code, which allows multiple DS18B20s to function on the same 1-Wire bus. Thus, it is simple to use one microprocessor to control many DS18B20s distributed over a large area. Indeed, this is a very easy device to interface with. But even given the svelte hardware needs (power, data and ground signals), writing some code that speaks 1-Wire is not necessarily something I’m interested in. Fortunately, these sensors are very commonly used with the Raspberry Pi, as illustrated by an Adafruit tutorial published in 2013. The Linux kernel provided for the Pi in its default Raspbian (Debian-derived) distribution supports bit-banging 1-Wire over its GPIOs by default, requiring only a device tree overlay to activate it. This is as simple as adding a line to /boot/config.txt to make the machine’s boot loader instruct the kernel to apply a change to the hardware configuration at boot time: dtoverlay=w1-gpio With that configuration, one simply needs to wire the sensor up. The w1-gpio device tree configuration by default uses GPIO 4 on the Pi as the data line, then power and grounds need to be connected and a pull-up resistor added to the data line (since 1-Wire is an open-drain bus). The w1-therm kernel module already understands how to interface with these sensors- meaning I don’t need to write any code to talk to the temperature sensor: Linux can do it all for me! For instance, reading the temperature out in an interactive shell to test, after booting with the 1-Wire overlay enabled: $ modprobe w1-gpio w1-therm
$cd /sys/bus/w1/devices$ ls
28-000004b926f1  w1_bus_master1

## Temperature exporter

Having connected the thermometer to the Pi and set up Prometheus, we now need to glue them together such that Prometheus can read the temperature. The usual way is for Prometheus to make HTTP requests to its known data sources, where the response is formatted such that Prometheus can make sense of the metrics. There is some support for having metrics sources push their values to Prometheus through a bridge (that basically just remembers the values it’s given until they’re scraped), but that seems inelegant given it would require running another program (the bridge) and goes against the how Prometheus is designed to work.

I’ve published the source for the metrics exporter I ended up writing, and will give it a quick description in the remnants of this section.

The easiest solution to providing a service over HTTP is using the http.server module, so that’s what I chose to use. When the program starts up it scans for temperature sensors and stores them. This has a downside of never returning data if a sensor is accidentally disconnected at startup, but detection is fairly slow and only doing it at startup makes it clearer if sensors are accidentally disconnected during operation, since reading them will fail at that point.

#!/usr/bin/env python3

import socketserver
from http.server import HTTPServer, BaseHTTPRequestHandler
from w1thermsensor import W1ThermSensor

SENSORS = W1ThermSensor.get_available_sensors()

The request handler has a method that builds the whole response at once, which is just plain text based on a simple template.

class Exporter(BaseHTTPRequestHandler):
METRIC_HEADER = ('# HELP w1therm_temperature Temperature in Kelvin of the sensor.\n'
'# TYPE w1therm_temperature gauge\n')

def build_exposition(self, sensor_states):
for sensor, temperature in sensor_states.items():
out += 'w1therm_temperature{{id="{}"}} {}\n'.format(sensor, temperature)
return out

do_GET is called by BaseHTTPRequestHandler for all HTTP GET requests to the server. Since this server doesn’t really care what you want (it only exports one thing- metrics), it completely ignores the request and sends back metrics.

    def do_GET(self):
response = self.build_exposition(self.get_sensor_states())
response = response.encode('utf-8')

# We're careful to send a content-length, so keepalive is allowed.
self.protocol_version = 'HTTP/1.1'
self.close_connection = False

self.send_response(200)
self.wfile.write(response)

The http.server API is somewhat cumbersome in that it doesn’t try to handle setting Content-Length on responses to allow clients to keep connections open between requests, but at least in this case it’s very easy to set the Content-Length on the response and correctly implement HTTP 1.1. The Content-Type used here is the one specified by the Prometheus documentation for exposition formats.

The rest of the program is just glue, for the most part. The console_entry_point function is the entry point for the w1therm_prometheus_exporter script specified in setup.py. The network address and port to listen on are taken from the command line, then an HTTP server is started and allowed to run forever.

### As a server

As a Python program with a few non-standard dependencies, installation of this server is not particularly easy. While I could sudo pip install everything and call it sufficient, that’s liable to break unexpectedly if other parts of the system are automatically updated- in particular the Python interpreter itself (though Debian as a matter of policy doesn’t update Python to a different release except as a major update, so it shouldn’t happen without warning). What I’d really like is the ability to build a single standalone program that contains everything in a convenient single-file package, and that’s exactly what PyInstaller can do.

A little bit of wrestling with pyinstaller configuration later (included as the .spec file in the repository), I had successfully built a pretty heavy (5MB) executable containing everything the server needs to run. I placed a copy in /usr/local/bin, for easy accessibility in running it.

I then wrote a simple systemd unit for the temperature server to make it start automatically, installed as /etc/systemd/system/w1therm-prometheus-exporter.service:

[Unit]
Description=Exports 1-wire temperature sensor readings to Prometheus
Documentation=https://bitbucket.org/tari/w1therm-prometheus

[Service]
ExecStart=/usr/local/bin/w1therm-prometheus-exporter localhost 9000
Restart=always

StandardOutput=journal
StandardError=journal

# Standalone binary doesn't need any access beyond its own binary image and
# a tmpfs to unpack itself in.
DynamicUser=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Enable the service, and it will start automatically when the system boots:

systemctl enable w1therm-prometheus-exporter.service

This unit includes rather more protection than is probably very useful, given the machine is single-purpose, but it seems like good practice to isolate the server from the rest of the system as much as possible.

• DynamicUser will make it run as a system user with ID semi-randomly assigned each time it starts so it doesn’t look like anything else on the system for purposes of resource (file) ownership.
• ProtectSystem makes it impossible to write to most of the filesystem, protecting against accidental or malicious changes to system files.
• ProtectHome makes it impossible to read any user’s home directory, preventing information leak from other users.
• PrivateTmp give the server its own private /tmp directory, so it can’t interfere with temporary files created by other things, nor can its be interfered with- preventing possible races which could be exploited.

## Pi connectivity

Having built the HTTP server, I needed a way to get data from it to Prometheus. As discussed earlier, the Raspberry Pi with the sensor is on a WiFi network that doesn’t permit any incoming connections, so how can Prometheus scrape metrics if it can’t connect to the Pi?

One option is to push metrics to Prometheus, using the push gateway. However, I don’t like that option because the push gateway is intended mostly for jobs that run unpredictably, in particular where they can exit without warning. This isn’t true of my sensor server. PushProx provides a rather better solution, wherein clients connect to a proxy which forwards fetches from Prometheus to the relevant client, though I think my ultimate solution is just as effective and simpler.

What I ended up doing is using autossh to open an SSH tunnel at the Prometheus server which connects to the Raspberry Pi’s metrics server. Autossh is responsible for keeping the connection alive, managed by systemd. Code is going to be much more instructive here than a long-form description, so here’s the unit file:

[Unit]
Description=SSH reverse tunnel from %I for Prometheus
After=network-online.target
Wants=network-online.target

[Service]
User=autossh
ExecStart=/usr/bin/autossh -N -p 22 -l autossh -R 9000:localhost:9000 -i /home/autossh/id_rsa %i
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Installed as /etc/systemd/system/autossh-tunnel@.service, this unit file tells systemd that we want to start autossh when the network is online and try to ensure it always stays online. I’ve increased RestartSec from the default 100 milliseconds because I found that even with the dependency on network-online.target, ssh could fail when the system was booting up with DNS lookup failures, then systemd would give up. Increasing the restart time means it takes much longer for systemd to give up, and in the meantime the network actually comes up.

The autossh process itself runs as a system user I created just to run the tunnels (useradd --system -m autossh), and opens a reverse tunnel from port 9000 on the remote host to the same port on the Pi. Authentication is with an SSH key I created on the Pi and added to the Prometheus machine in Google Cloud, so it can log in to the server without any human intervention. Teaching systemd that this should run automatically is a simple enable command away1:

systemctl enable autossh-tunnel@pitemp.example.com

Then it’s just a matter of configuring Prometheus to scrape the sensor exporter. The entire Prometheus config looks like this:

global:
scrape_interval:     15s # By default, scrape targets every 15 seconds.
evaluation_interval: 15s

scrape_configs:
- job_name: 'w1therm'
static_configs:
- targets: ['localhost:9000']

That’s pretty self-explanatory; Prometheus will fetch metrics from port 9000 on the same machine (which is actually an SSH tunnel to the Raspberry Pi), and do so every 15 seconds. When the Pi gets the request for metrics, it reads the temperature sensors and returns their values.

## Data retention

I included InfluxDB in the setup to get arbitrary retention of temperature data- Prometheus is designed primarily for real-time monitoring of computer systems, to alert human operators when things appear to be going wrong. Consequently, in the default configuration Prometheus only retains captured data for a few weeks, and doesn’t provide a convenient way to export data for archival or analysis. While the default retention is probably sufficient for this project’s needs, I wanted better control over how long that data was kept and the ability to save it as long as I liked. So while Prometheus doesn’t offer that control itself, it does support reading and writing data to and from various other databases, including InfluxDB (which I chose only because a package for it is available in Debian without any additional work).

Unfortunately, the version of Prometheus available in Debian right now is fairly old- 1.5.2, where the latest release is 2.2. More problematic, while Prometheus now supports a generic remote read/write API, this was added in version 2.0 and is not yet available in the Debian package. Combined with the lack of documentation (as far as I could find) for the old remote write feature, I was a little bit stuck.

Things ended up working out nicely though- I happened to see flags relating to InluxDB in the Prometheus web UI, which mostly have no default values:

• storage.remote.influxdb-url
• storage.remote.influxdb.database = prometheus
• storage.remote.influxdb.retention-policy
• storage.remote.influxdb.username

These can be specified to Prometheus by editing /etc/defaults/prometheus, which is part of the Debian package for providing the command line arguments to the server without requiring users to directly edit the file that tells the system how to run Prometheus. I ended up with these options there:

ARGS="--storage.local.retention=720h \
--storage.remote.influxdb-url=http://localhost:8086/ \
--storage.remote.influxdb.retention-policy=autogen"

The first option just makes Prometheus keep its data longer than the default, whereas the others tell it how to write data to InfluxDB. I determined where InfluxDB listens for connections by looking at its configuration file /etc/influxdb/influxdb.conf and making a few guesses: a comment in the http section there noted that “these (HTTP endpoints) are the primary mechanism for getting data into and out of InfluxDB” and included the settings bind-address=":8086" and auth-enabled=false, so I guessed (correctly) that telling Prometheus to find InfluxDB at http://localhost:8086/ should be sufficient.

Or, it was almost enough: setting the influxdb-url and restarting Prometheus, it was logging warnings periodically about getting errors back from InfluxDB. Given the influxdb.database settings defaults to prometheus, I (correctly) assumed I needed to create a database. A little browsing of the Influx documentation and a few guesses later, I had done that:

$apt-get install influxdb-client$ influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 1.0.2
InfluxDB shell version: 1.0.2
> CREATE DATABASE prometheus;

Examining the Prometheus logs again, now it was failing and complaining that the specified retention policy didn’t exist. Noting that the Influx documentation for the CREATE DATABASE command mentioned that the autogen retention policy will be used if no other is specified, setting the retention-policy flag to autogen and restarting Prometheus made data start appearing, which I verified by waiting a little while and making a query (guessing a little bit about how I would query a particular metric):

> USE prometheus;
> SELECT * FROM w1therm_temperature LIMIT 10;
name: w1therm_temperature
-------------------------
time                    id              instance        job     value
1532423583303000000     000004b926f1    localhost:9000  w1therm 297.9
1532423598303000000     000004b926f1    localhost:9000  w1therm 297.9
1532423613303000000     000004b926f1    localhost:9000  w1therm 297.9
1532423628303000000     000004b926f1    localhost:9000  w1therm 297.9
1532423643303000000     000004b926f1    localhost:9000  w1therm 297.9
1532423658303000000     000004b926f1    localhost:9000  w1therm 297.9
1532423673303000000     000004b926f1    localhost:9000  w1therm 297.9
1532423688303000000     000004b926f1    localhost:9000  w1therm 297.9
1532423703303000000     000004b926f1    localhost:9000  w1therm 297.9
1532423718303000000     000004b926f1    localhost:9000  w1therm 297.9

## Results

A sample graph of the temperature over two days:

The fermentation temperature is quite stable, with daily variation of less than one degree in either direction from the baseline.

## Refinements

I later improved the temperature server to handle SIGHUP as a trigger to scan for sensors again, which is a slight improvement over restarting it, but not very important because the server is already so simple (and fast to restart).

On reflection, using Prometheus and scraping temperatures is a very strange way to go about solving the problem of logging the temperature (though it has the advantage of using only tools I was already familiar with so it was easy to do quickly). Pushing temperature measurements from the Pi via MQTT would be a much more sensible solution, since that’s a protocol designed specifically for small sensors to report their states. Indeed, there is no shortage of published projects that do exactly that more efficiently than my Raspberry Pi, most of them using ESP8266 microcontrollers which are much lower-power and can still connect to Wi-Fi networks.

### Rambling about IoT security

Getting sensor readings through an MQTT broker and storing them to be able to graph them is not quite as trivial as scraping them with Prometheus, but I suspect there does exist a software package that does most of the work already. If not, I expect a quick and dirty one could be implemented with relative ease.

On the other hand, running a device like that which is internet-connected but is unlikely to ever receive anything remotely looking like a security update seems ill-advised if it’s meant to run for anything but a short amount of time. In that case having the sensor be part of a Zigbee network instead, which does not permit direct internet connectivity and thus avoids the fraught terrain of needing to protect both the device itself from attack and the data transmitted by the device from unauthorized use (eavesdropping) by taking ownership of that problem away from the sensor.

It remains possible to forward messages out to an MQTT broker on the greater internet using some kind of bridge (indeed, this is the system used by many consumer “smart device” platforms, like Philips’ Hue though I don’t think they use MQTT), where individual devices connect only to the Zigbee network, and a more capable bridge is responsible for internet connectivity. The problem of keeping the bridge secure remains, but is appreciably simpler than needing to maintain the security of each individual device in what may be a heterogeneous network.

It’s even possible to get inexpensive off-the-shelf temperature and humidity sensors that connect to Zigbee networks like some sold by Xiaomi, offering much better finish than a prototype-quality one I might be able to build myself, very good battery life, and still capable of operating in a heterogenous Zigbee network with arbitrary other devices (though you wouldn’t know it from the manufacturer’s documentation, since they want consumers to commit to their “platform” exclusively)!

So while my solution is okay in that it works fine with hardware I already had on hand, a much more robust solution is readily available with off-the-shelf hardware and only a little bit of software to glue it together. If I needed to do this again and wanted a solution that doesn’t require my expertise to maintain it, I’d reach for those instead.

1. Hostname changed to an obviously fake one for anonymization purposes. [return]

# Quick and dirty web image optimization

Given a large pile of images that nominally live on a web server, I want to make them smaller and more friendly to serve to clients. This is hardly novel: for example, Google offer detailed advice on reducing the size of images for the web. I have mostly JPEG and PNG images, so, jpegtran and optipng are the tools of choice for bulk lossless compression.

To locate and compress images, I’ll use GNU find and parallel to invoke those tools. For JPEGs I take a simple approach, preserving comment tags and creating a progressive JPEG (which can be displayed at a reduced resolution before the entire image has been downloaded).

find -iname '*.jpg' -printf '%P,%s\n' \
-exec jpegtran -o -progressive -copy comments -outfile {} {} \; \
> delta.csv


Where things get a little more interesting is when I output the name and size of each located file (-printf ...) and store those in a file (> delta.csv).1 This is so I can collect more information about the changes that were made.

## Check the JPEGs

Loading up a Jupyter notebook for quick computations driven with Python, I load the file names and original sizes, adding the current size of each listed file to my dataset.

import os
deltas = []

with open('delta.csv', 'r') as f:
for line in f:
name, _, size = line.partition(',')
size = int(size)
newsize = os.stat(name).st_size
deltas.append((name, size, newsize))

From there, it’s quick work to do some list comprehensions and generate some overall statistics:

original_sizes = [orig for (_, orig, _) in deltas]
final_sizes = [new for (_, _, new) in deltas]
shrinkage = [orig - new for (_, orig, new) in deltas]

pct_total_change = 100 * (sum(original_sizes) -
sum(final_sizes)) / sum(original_sizes)

pct_change = [shrinkage /
orig for (shrinkage, orig) in zip(shrinkage, original_sizes)]
avg_pct_change = 100 * sum(pct_change) / len(pct_change)

print('Total size reduction:', sum(shrinkage),
'bytes ({}%)'.format(round(pct_total_change, 2)))
avg = sum(shrinkage) / len(shrinkage)
print('Average reduction per file:', avg,
'bytes ({}%)'.format(round(avg_pct_change, 2)))

This is by no means good code, but that’s why this short write-up is “quick and dirty”. Over the JPEGs alone, I rewrote 2162 files, saving 8820474 bytes overall; about 8.4 MiB, 8.02% of the original size of all of the files- a respectable savings for exactly zero loss in quality.

## PNGs

I processed the PNGs in a similar fashion, having optipng emit interlaced PNGs which can be displayed at reduced resolution without complete data and try more combinations of compression parameters than it usually would, trading CPU time for possible gains. There was no similar tradeoff with the JPEGs, since apparently jpegtran’s optimization of encoding is entirely deterministic whereas optipng relies on experimental compression of image data with varying parameters to find a minimum size.

Since I observed that many of the PNGs were already optimally-sized and interlacing made them significantly larger (by more than 10% in many cases), I also considered only files that were more than 256 KiB in size. To speed up processing overall, I used parallel to run multiple instances of optipng at once (since it’s a single-threaded program) to better utilize the multicore processors at my disposal.

find -iname '*.png' -size +256k -printf '%P,%s\n' \
-exec parallel optipng -i 1 -o 9 ::: {} + \
> delta.csv


Running the same analysis over this output, I found that interlacing had a significant negative effect on image size. There were 68 inputs larger than 256 KiB, and the size of all of the files increased by 2316732 bytes; 2.2 MiB, nearly 10% of the original size.

A few files had significant size reductions (about 300 KiB in the best case), but the largest increase in size had similar magnitude. Given the overall size increase, the distribution of changes must be skewed towards net increases.

### Try again

Assuming most of the original images were not interlaced (most programs that write PNG don’t interlace unless specifically told to) and recognizing that interlaced PNGs tend to compress worse, I ran this again but without interlacing (-i 0) and selecting all files regardless of size.

The results of this second run were much better: over 3102 files, save 12719421 bytes (12.1 MiB), 15.9% of the original combined size. One file saw a whopping 98% reduction in size, from 234 KB to only 2914 bytes- inspecting that one myself, the original was inefficiently coded 32 bits per pixel (8-bit RGBA), and it was reduced to two bits per pixel. I expect a number of other files had similar but less dramatic transformations. Some were not shrunk at all, but optipng is smart enough to skip rewriting those so it will never make a file larger- the first run was an exception because I asked it to make interlaced images.

## That’s all

I saved about 20 MiB for around 5000 files- not bad. A copy of the notebook I used to do measurements is available (check out nbviewer for an online viewer), useful if you want to do something similar with your own images. I would not recommend doing it to ones that are not meant purely for web viewing, since the optimization process may strip metadata that is useful to preserve.

1. Assumption: none of the file names contain commas, since I’m calling this a CSV (comma-separated values) file. It’s true in this instance, but may not be in others.

[return]

# sax-ng

Over on Cemetech, we’ve long had an embedded chat widget called “SAX” (“Simultaneous Asynchronous eXchange”). It behaves kind of like a traditional shoutbox, in that registered users can use the SAX widget to chat in near-real-time. There is also a bot that relays messages between the on-site widget and an IRC channel, which we call “saxjax”.

The implementation of this, however, was somewhat lacking in efficiency. It was first implemented around mid-2006, and saw essentially no updates until just recently. The following is a good example of how dated the implementation was:

// code for Mozilla, etc
if (window.XMLHttpRequest) {
xmlhttp=new XMLHttpRequest()
xmlhttp.open("GET",url,true)
xmlhttp.send(null)
} else if (window.ActiveXObject) {
// code for IE
xmlhttp=new ActiveXObject("Microsoft.XMLHTTP")
if (xmlhttp) {
xmlhttp.open("GET",url,true)
xmlhttp.send()
}
}


The presence of ActiveXObject here implies it was written at a time when a large fraction of users would have been using Internet Explorer 5 or 6 (the first version of Internet Explorer released which supported the standard form of XMLHttpRequest was version 7).

Around a year ago (that’s how long this post has been a draft for!), I took it upon myself to design and implement a more modern replacement for SAX. This post discusses that process and describes the design of the replacement, which I have called “sax-ng.”

## Legacy SAX

The original SAX implementation, as alluded to above, is based on AJAX polling. On the server, a set of approximately the 30 most recent messages were stored in a MySQL database and a few PHP scripts managed retrieving and modifying messages in the database. This design was a logical choice when initially built, since the web site was running on a shared web host (supporting little more than PHP and MySQL) at the time.

Eventually this design became a problem, as essentially every page containing SAX that is open at any given time regularly polls for new messages. Each poll calls into PHP on the server, which opens a database connection to perform one query. Practically, this means a very large number of database connections being opened at a fairly regular pace. In mid-2012 the connection count reached levels where the shared hosting provider were displeased with it, and requested that we either pay for a more expensive hosting plan or reduce resource usage.

In response, we temporarily disabled SAX, then migrated the web site to a dedicated server provided by OVH, who had opened a new North American datacenter in July. We moved to the dedicated server in August of 2012. This infrastructure change kept the system running, and opened the door to a more sophisticated solution since we gained the ability to run proper server applications.

Meanwhile, the limitations of saxjax (the IRC relay bot) slowly became more evident over time. The implementation was rather ad-hoc, in Python. It used two threads to implement relay, with a dismaying amount of shared state used to relay messages between the two threads. It tended to stop working correctly in case of an error in either thread, be it due to a transient error response from polling the web server for new messages, or an encoding-related exception thrown from the IRC client (since Python 2.x uses bytestrings for most tasks unless specifically told not to, and many string operations (particularly outputting the string to somewhere) can break without warning when used with data that is not 8-bit clean (that is, basically anything that isn’t ASCII).

Practically, this meant that the bot would frequently end up in a state where it would only relay messages one way, or relay none at all. I put some time into making it more robust to these kinds of failures early in 2015, such that some of the time it would manage to catch these errors and outright restart (rather than try to recover from an inconsistent state). Doing so involved some pretty ugly hacks though, which prompted a return to some longtime thoughts on how SAX could be redesigned for greater efficiently and robustness.

## sax-ng

For a long time prior to beginning this work, I frequently (semi-jokingly) suggested XMPP (Jabber) as a solution to the problems with SAX. At a high level this seems reasonable: XMPP is a chat protocol with a number of different implementations available, and is relatively easy to set up as a private chat service.

On the other hand, the feature set of SAX imposes a few requirements which are not inherently available for any given chat service:

1. An HTTP gateway, so clients can run inside a web browser.
2. Group chat, not just one-to-one conversation capability.
3. External authentication (logging in to the web site should permit connection to chat as well).
4. Retrieval of chat history (so a freshly-loaded page can have some amount of chat history shown).

As it turns out, ejabberd enables all of these, with relatively little customization. mod_http_bind provides an HTTP gateway as specified in XEP-0206, mod_muc implements multi-user chat as specified in XEP-0045 which also includes capabilities to send chat history to clients when they connect, and authentication can be handled by an external program which speaks a simple binary protocol and is invoked by ejabberd.

Main implementation of the new XMPP-based system was done in about a week, perhaps 50 hours of concerted work total (though I may be underestimating). I had about a month of “downtime” at the beginning of this past summer, the last week of which was devoted to building sax-ng.

### ejabberd

The first phase involved setting up an instance of ejabberd to support the rest of the system. I opted to run it inside Docker, ideally to make the XMPP server more self-contained and avoid much custom configuration on the server. Conveniently, somebody had already built a Docker configuration for ejabberd with a wealth of configuration switches, so it was relatively easy to set up.

Implementing authentication against the web site was also easy, referring to the protocol description in the ejabberd developers guide. Since this hooks into the website’s authentication system (a highly modified version of phpBB), this script simply connects to the mysql server and runs queries against the database.

Actual authentication is performed with phpBB SIDs (Session ID), rather than a user’s password. It was built this way because the SID and username are stored in a cookie, which is available to a client running in a web browser. This is probably also somewhat more secure than storing a password in the web browser, since the SID is changed regularly so data exposure via some vector cannot compromise a user’s web site password.

Error handling in the authentication script is mostly nonexistent. The Erlang approach to such problems is mostly “restart the component if it fails”, so in case of a problem (of which the only real possibility is a database connection error) ejabberd will restart the authentication script and attempt to carry on. In practice this has proven to be perfectly reliable.

In XMPP MUC (Multi-User Chat), users are free to choose any nickname they wish. For our application, there is really only one room and we wish to enforce that the nickname used in XMPP is the same as a user’s username on the web site. There ends up being no good way in ejabberd to require that a user take a given nickname, but we can ensure that it is impossible to impersonate other users by registering all site usernames as nicknames in XMPP. Registered nicknames may only be used by the user to which they are registered, so the only implementation question is in how to automatically register nicknames.

I ended up writing a small patch to mod_muc_admin, providing an ejabberdctl subcommand to register a nickname. This patch is included in its entirety below.

diff --git a/src/mod_muc_admin.erl b/src/mod_muc_admin.erl
index 9c69628..3666ba0 100644
@@ -15,6 +15,7 @@
start/2, stop/1, % gen_mod API
muc_online_rooms/1,
muc_unregister_nick/1,
+    muc_register_nick/3,
create_room/3, destroy_room/3,
create_rooms_file/1, destroy_rooms_file/1,
rooms_unused_list/2, rooms_unused_destroy/2,
@@ -38,6 +39,9 @@

%% Copied from mod_muc/mod_muc.erl
-record(muc_online_room, {name_host, pid}).
+-record(muc_registered,
+        {us_host = {\{<<"">>, <<"">>}, <<"">>} :: {\{binary(), binary()}, binary()} | '$1', + nick = <<"">> :: binary()}). %%---------------------------- %% gen_mod @@ -73,6 +77,11 @@ commands() -> module = ?MODULE, function = muc_unregister_nick, args = [{nick, binary}], result = {res, rescode}}, + #ejabberd_commands{name = muc_register_nick, tags = [muc], + desc = "Register the nick in the MUC service to the JID", + module = ?MODULE, function = muc_register_nick, + args = [{nick, binary}, {jid, binary}, {domain, binary}], + result = {res, rescode}}, #ejabberd_commands{name = create_room, tags = [muc_room], desc = "Create a MUC room name@service in host", @@ -193,6 +202,16 @@ muc_unregister_nick(Nick) -> error end. +muc_register_nick(Nick, JID, Domain) -> + {jid, UID, Host, _,_,_,_} = jlib:string_to_jid(JID), + F = fun (MHost, MNick) -> + mnesia:write(#muc_registered{us_host=MHost, + nick=MNick}) + end, + case mnesia:transaction(F, [{\{UID, Host}, Domain}, Nick]) of + {atomic, ok} -> ok; + {aborted, _Error} -> error + end. %%---------------------------- %% Ad-hoc commands  It took me a while to work out how exactly to best implement this feature, but considering I had never worked in Erlang before it was reasonably easy. I do suspect some familiarity with Haskell and Rust provided background to more easily understand certain aspects of the language, though. The requirement that I duplicate the muc_registered record (since apparently Erlang provides no way to import records from another file) rubs me the wrong way, though. In practice, then, a custom script traverses the web site database, invoking ejabberdctl to register the nickname for every existing user at server startup and then periodically or on demand when the server is running. ### Web interface The web interface into XMPP was implemented with Strophe.js, communicating with ejabberd via HTTP-bind with the standard support in both the client library and server. The old SAX design served a small amount of chat history with every page load so it was immediately visible without performing any additional requests after page load, but since the web server never receives chat data (it all goes into XMPP directly), this is no longer possible. The MUC specification allows a server to send chat history to clients when they join a room, but that still requires several HTTP round-trips (taking up to several seconds) to even begin receiving old lines. I ended up storing a cache of messages in the browser, which is used to populate the set of displayed messages on initial page load. Whenever a message is received and displayed, its text, sender and a timestamp are added to the local cache. On page load, messages from this cache which are less than one hour old are displayed. The tricky part with this approach is avoiding duplication of lines when messages sent as part of room history already exist, but checking the triple of sender, text and timestamp seems to handle these cases quite reliably. ### webridge The second major feature of SAX is to announce activity on the web site’s bulletin board, such as when people create or reply to threads. Since the entire system was previously managed by code tightly integrated with the bulletin board, a complete replacement of the relevant code was required. In the backend, SAX functionality was implemented entirely in one PHP function, so replacing the implementation was relatively easy. The function’s signature was something like saxSay($type, $who,$what, $where), where type is a magic number indicating what kind of message it is, such as the creation of a new thread, a post in a thread or a message from a user. The interpretation of the other parameters depends on the message type, and tends to be somewhat inconsistent. The majority of that function was a maze of comparisons against the message type, emitting a string which was eventually pushed into the chat system. Rather than attempt to make sense of that code, I decided to replace it with a switch statement over symbolic values (whereas the old code just used numbers with no indication of purpose), feeding simple invocations of sprintf. Finding the purpose of each of the message types was most challenging among that, but it wasn’t terribly difficult as I ended up searching the entire web site source code for references to saxSay and determined the meaning of the types from the caller’s context. To actually feed messages from PHP into XMPP, I wrote a simple relay bot which reads messages from a UNIX datagram socket and repeats them into a MUC room. A UNIX datagram socket was selected because there need not be any framing information in messages coming in (just read a datagram and copy its payload), and this relay should not be accessible to anything running outside the same machine (hence a UNIX socket). The bot is implemented in Python with Twisted, utilizing Twisted’s provided protocol support for XMPP. It is run as a service under twistd, with configuration provided via environment variables because I didn’t want to write anything to handle reading a more “proper” configuration file. When the PHP code calls saxSay, that function connects to a socket with path determined from web site configuration and writes the message into that socket. The relay bot (“webridge”) receives these messages and writes them into MUC. ### saxjax-ng Since keeping a web page open for chatting is not particularly convenient, we also operate a bridge between the SAX chat and an IRC channel called saxjax. The original version of this relay bot was of questionable quality at best: the Python implementation ran two threads, each providing one-way communication though a list. No concurrency primitives, little sanity. Prior to creation of sax-ng I had put some amount of effort in improving the reliability of that system, since an error in either thread would halt all processing of messages in the direction corresponding to the thread in which the error occurred. Given there was essentially no error handling anywhere in the program, this sort of thing happened with dismaying frequency. saxjax-ng is very similar in design to webridge, in that it’s Twisted-based and uses the Twisted XMPP library. On the IRC side, it uses Twisted’s IRC library (shocking!). Both ends of this end up being very robust when combined with the components that provide automatic reconnection and a little bit of custom logic for rotating through a list of IRC servers. Twisted guarantees singlethreaded operation (that’s the whole point; it’s an async event loop), so relaying a message between the two connections is simply a matter of repeating it on the other connection. ## Contact with users This system has been perfectly reliable since deployment, after a few changes. Most notably, the http-bind interface for ejabberd was initially exposed on port 5280 (the default for http-bind). Users behind certain restrictive firewalls can’t connect to that port, so we quickly reconfigured our web server to reverse-proxy to http-bind and solve that problem. Doing so also means the XMPP server doesn’t need its own copy of the server’s SSL certificate. There are still some pieces of the web site that emit messages containing HTML entities in accordance with the old system. The new system.. doesn’t emit HTML entities because that should be the responsibility of something doing HTML presentation (Strong Opinion) and I haven’t bothered trying to find the things that are still emitting HTML-like strings. The reconnect logic on the web client tends to act like it’s received multiples of every message that arrives after it’s tried to reconnect to XMPP, such as when a user puts their computer to sleep and later resumes; the web client tries to detect the lost connection and reopen it, and I think some event handlers are getting duplicated at that point. Haven’t bothered working on a fix for that either. # Conclusion ejabberd is a solid piece of software and not hard to customize. Twisted is a good library for building reliable network programs in Python, but has enough depth that some of its features lack useful documentation so finding what you need and figuring out how to use it can be difficult. This writeup has been languishing for too long so I’m done writing now. # Reverse-engineering Ren'py packages Some time ago (September 3, 2013, apparently), I had just finished reading Analogue: A Hate Story (which I highly recommend, by the way) and was particularly taken with the art. At that point it seems my engineer’s instincts kicked in and it seemed reasonable to reverse-engineer the resource archives to extract the art for my own nefarious purposes. A little examination of the game files revealed a convenient truth: it was built with Ren’Py, a (open-source) visual novel engine written in Python. Python is a language I’m quite familiar with, so the actual task promised to be well within my expertise. ## Code Long story short, I’ve build some rudimentary tools for working with compiled Ren’py data. You can get it from my repository on BitBucket. Technically-inclined readers might also want to follow along in the code while reading. ## Background There are a large number of games designed with Ren’py. It’s an easy tool to get started with and hack on, since the script language is fairly simple and because it’s open-source, more sophisticated users are free to bend it to their will. A few examples of (in my opinion) high-quality things built with the engine: Since visual novels tend to live or die on the combination of art and writing, the ability to examine the assets outside the game environment offers interesting possibilities. Since it was handy, I started my experimentation with Analogue. ## RPA resource archives The largest files distributed with the game were .rpa files, so I investigated those first for finding art. As it turned out, this was exactly the place I needed to look. Start by examining the raw data: $ cd "Analogue A Hate Story/game"
$ls bytecode.rpyb data.rpa dlc1.rpa nd.rpa$ less data.rpa
RPA-3.0 00000000035f5c75 414154bb
<F0><AA><D6>^MީZ<A0><90><FB>^M6<B9><B7>^X<A3><82><F3>F<B0><DF>k8(<BF>ߦx<9C><D5>T
[snip]

There’s an obvious file identifier (RPA-3.0), followed by a couple numbers and a lot of compressed-looking data. The first number turns out to be very close to the total file size, so it’s probably some size or offset field, while the other one looks like some kind of signature.

$python -c 'print(0x35f5c75)' 56581237$ stat -c %s data.rpa
56592058

At this point I simply referred to the Ren’Py source code, rather than waste time experimenting on the data itself. Turns out the first number is the file offset of the index, and the second one is a key used for simple obfuscation of elements of the index (numbers are bitwise exclusive-or’d with the key). The archive index itself is a DEFLATE-compressed block of pickled Python objects. The index maps file names to tuples of offset and block length specifying where within the archive file the data can be found.

With that knowledge in hand, it’s short work to build a decoder for the index data and dump it all to files. This is rpa.py in my tools. Extracting the archives pulls out plenty of images and other media, as well as a number of interesting-looking .rpyb files, which we’ll discuss shortly.

### Cosplay

For a bit of amusement, I exercised my web-programming chops a little and built a standalone web page for playing with the extracted costumes and expressions of *Hyun-ae and *Mute, which I’ve included below. Here’s a link to the bare page for standalone amusement as well.

## Script guts

The basic format of compiled scripts (.rpyb files) is similar to that of resource packages. The entire thing is a tuple of (data, statements), where data is a dictionary of basic metadata and statements is a list of objects representing the script’s code.

The statements in this are just the Ren’py abstract syntax tree, so all the objects come from the renpy.ast module. Unfortunately and as I’ll discuss later, the pickle format makes this representation hard to work with.

The structure of AST members is designed such that each object can have attached bytecode. In practice this appears to never happen in archives. In my investigations of the source, it appears that Ren’py only writes Python bytecode as a performance enhancement, and most of it ends up in bytecode.rpyb. That file appears to provide some sort of bytecode cache that overrides script files in certain situations. For the purposes of reverse-engineering this is fortunate– Python bytecode is documented, but rather more difficult to translate into something human-readable than the source code that is actually present in RPYB archives.

Here’s some of the Act 1 script from Analogue run through the current version of my script decompiler:

## BEGIN Label 'dont_understand', params=None, hide=False
dont_understand:
## <class 'renpy.ast.Show'> -- don't know how to dump this!
## <class 'renpy.ast.Say'> -- don't know how to dump this!
## <class 'renpy.ast.Jump'> -- don't know how to dump this!
## BEGIN Label 'nothing', params=None, hide=False
nothing:
python:
shown_message = None

for block in store.blocks:
for message in block.contents:
if message == _message:
shown_message = str(store.blocks.index(block)+1) + "-" + str(block.contents.index(message)+1)

if shown_message:
gray_out(shown_message)
## <class 'renpy.ast.With'> -- don't know how to dump this!
## <class 'renpy.ast.Show'> -- don't know how to dump this!
## <class 'renpy.ast.With'> -- don't know how to dump this!
## <class 'renpy.ast.Say'> -- don't know how to dump this!


Clearly there are a few things my decompiler needs to learn about. It does, however, handle the more common block elements such as If statements. In any case, the Python code embedded in these scripts tends to be more interesting than the rest (which are mostly just dialogue and display manipulation) for the purposes of reverse-engineering. If you’re more interested in spoiling the game for yourself, it’s not as useful.

A few telling bits of logic from options.rpy:

init -3:
## BEGIN Python
python hide:
## BEGIN PyCode, exec mode, from game/options.rpy
renpy.demo_mode = True
init -1:
## BEGIN Python
python hide:
## BEGIN PyCode, exec mode, from game/options.rpy
config.developer = True

config.screen_width = 1024
if persistent.resolution == None:
persistent.resolution = 2
if persistent.old_resolution == None:
persistent.old_resolution = persistent.resolution

if not persistent.resolution:
config.screen_height = 600
else:
config.screen_height = 640

if renpy.demo_mode:
config.window_title = u"Analogue trial"
else:
config.window_title = u"Analogue: A Hate Story"

config.name = "Analogue"
config.version = "0.0"


The spacing here is interesting; I suspect (but haven’t attempted to verify) that the Ren’py script compiler strips comments since there haven’t been any in all of the scripts I’ve examined, so it’s likely that the unusual empty blocks in the code were comment blocks in a former life.

I’ve yet to dig much into what determines when demo_mode is set, but I doubt it would be difficult to forcibly set (or clear) if one were so inclined. Not that I condone such an action..

A little bit of interesting game-critical logic, also from Analogue (caution: minor spoilers)

store.radiation = 0
store.reactor_enabled = True
def interact_cb():
if store.radiation and store.reactor_enabled:

if (any_read("7-*") or (store.current_character == "mute" and get_m("6-9").enabled)) and store.radiation_levels < 0.65:

config.interact_callbacks.append(interact_cb)

You can get some idea of how specialized Ren’py’s execution environment is from this code. Particularly, store is a magic value injected into the locals of Python script blocks which maps to the RPY persistent variable store, which stores most of the game state. config is a similar magic value providing a handle to the engine configuration.

In this instance, radiation refers to a sort of hidden timer which forces the player to solve a puzzle on expiration (assuming the preconditions have been met), then make a decision which causes the plot to fork depending on that decision. Elsewhere in the code, I found a few developer switches which allow one to display the value of this countdown and reset or force it.

## Conclusions

As the official documentation notes, the process of resource compilation is not very secure but is enough to deter casual copying. I’ve shown here that such a claim is entirely correct, though script decompilation may be somewhat harder than the developers envisioned due to the choice of pickle as a serialization format.

It’s nothing particularly new to me, but a reminder to designers of software: if it runs on your attacker’s system, it can be hacked. It’s not a question of “if”, but instead “how fast”. I was mostly interested in extracting resources with this project, which was quite easy. In that matter, I think the designers of Ren’Py made a good design decision. The compiled archives and scripts are much more robust against accidental modification in the face of curious users than not compiling anything, but the developers do not expend undue effort building something harder to break which would eventually be broken anyway by a sufficiently determined attacker.

### Portability

As I alluded to earlier, the pickle representation makes the Ren’Py AST hard to work with. This is because many of the objects contain references to the engine state, which in turn implies most of the engine needs to be initialized when unpickling the AST. To say the least, this is not easy- engine initialization is not easily separated from game startup.

To illustrate the problem, observe that the Ren’Py developer kit is simply the engine itself packaged with a game of sorts that provides help in getting a new project set up by modifying the included scripts. There simply seems to be no part of the engine that is designed to run without the rest of it running as well.

In experimenting with different products built with Ren’Py, I’ve had to make changes to some combination of the engine itself and my code in order to bootstrap the engine state to a point where the AST can be successfully unpickled. Suffice to say, this has hampered my progress somewhat, and led me to consider slightly different avenues of attack.

The most promising of these would involve a semi-custom unpickler which avoids instantiating actual Ren’Py objects; the only data that need be preserved is the structural information, rather than the many hooks into engine state that are also included in the pickle serialization. Further continuation of this project is likely to take this approach to deserialization.

# GStreamer's playbin, threads and queueing

I’ve been working on a project that uses GStreamer to play back audio files in an automatically-determined order. My implementation uses a playbin, which is nice and easy to use. I had some issues getting it to continue playback on reaching the end of a file, though.

According to the documentation for the about-to-finish signal,

This signal is emitted when the current uri is about to finish. You can set the uri and suburi to make sure that playback continues.

This signal is emitted from the context of a GStreamer streaming thread.

Because I wanted to avoid blocking a streaming thread under the theory that doing so might interrupt playback (the logic in determining what to play next hits external resources so may take some time), my program simply forwarded that message out to be handled in the application’s main thread by posting a message to the pipeline’s bus.

Now, this approach appeared to work, except it didn’t start playing the next URI, and the pipeline never changed state- it was simply wedged. Turns out that you must assign to the uri property from the same thread, otherwise it doesn’t do anything.

Fortunately, it turns out that blocking that streaming thread while waiting for data isn’t an issue (determined by experiment by simply blocking the thread for a while before setting the uri).

# "Four"ier transform

I liked the joke and am familiar enough with the math of working in unusual bases that I felt a need to implement a quick version of this in Python. Code follows.

#!/usr/bin/env python

def fourier(x, b):
"""Attempts to find a fourier version of x, working down from base b.

Returns the fouriest base."""
mostFours = 0
bestBase = -1

for base in range(b, 1, -1):
fours = 0
t = x
while t != 0:
if (t % base) == 4:
fours += 1
t //= base

# Prefer lower bases
if fours >= mostFours:
print(baseconvert(x, base) + "_{0}".format(base))
mostFours = fours
bestBase = base

return bestBase

BASE_CHARS = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"
def baseconvert(x, base):
s = ""
while x != 0:
s += BASE_CHARS[x % base]
x //= base
return ''.join(reversed(s))

if __name__ == '__main__':
from sys import argv, exit
if len(argv) < 2:
print("""Usage: {0} <number>

Computes the "four"ier transform of <number>, printing the optimizations to
reach the "fouriest" form.""".format(argv[0]))
exit(1)

x = int(argv[1])
# Base 36 is the largest sensible base to use
base = fourier(x, 36)

if base == -1:
print("{0} is four-prime!".format(x))

This is Python 3.x code, using explicit integer division. It should work under the 2.x series if you change line 34 to use “/=” rather than “//=”. It can only go up to base 36, because I didn’t want to deal with bases that are hard to represent in reasonable ways. Up to base 64 is an option, but in that case I would have wanted to use MIME base 64, which puts digits at positions 52 through 61, which would be confusing to read. Thus it only supports up to base 36, but could be adjusted with relative east to do larger bases.

Running a few examples:

$python fourier.py 624 HC_36 HT_35 IC_34 IU_33 JG_32 K4_31 143_23 1B4_20 440_12 4444_5$ python fourier.py 65535
1EKF_36
1IHF_35
1MNH_34
1R5U_33
1VVV_32
2661_31
2COF_30
2JQO_29
2RGF_28
38O6_27
3IOF_26
44LA_25
B44F_18
14640_15
4044120_5

\$ python fourier.py 3
3 is four-prime!

A few quirks: it prefers lower bases, so bases that match earlier attempts in fouriness will be printed, despite having equal fouriness. I’ve decided to call values that have no representations containing a ‘4’ character “four-prime”, which is probably going to be a rare occurrence, but the program handles it okay.

Generalization of the algorithm is certainly possible, and basically requires changing the condition on line 14 to match different digit values. For example, a hypothetical “Three”ier transform would replace the ‘4’ on line 14 with a ‘3’.

There’s a rather interesting discussion of the topic over on Reddit, as well as a few other implementations. (Thanks to Merth for pointing those out to me.)

# Rationale

I was reading up on web frameworks available when programming in Haskell earlier today, and I liked the use of domain-specific languages (DSLs) within frameworks such as the routing syntax in Yesod. Compared to how routes are specified in Django (as a similar example that I’m already familiar with), the DSL is both easier to read (because it doesn’t need to be valid code in the hosting language) and faster (since it ends up getting compiled into the application as properly executable code).

A pattern I find myself using rather often in Python projects is to have a small module (usually called config) that encapsulates an INI-style configuration file. It feels like an ugly solution though, since it generally just exports a ConfigParser instance. Combined with consideration of DSLs in Haskell, that got me thinking: what if there were an easier way that made INI configuration files act like Python source such that they could just be imported and have the contents of the file exposed as simple Python types (thus hiding some unnecessary complexity)?

# Implementation

I was aware of Python’s import hook mechanisms, so I figured that it should be a good way to approach this problem, and it ended up being a good excuse to learn more about the import hook mechanism. Thus, the following code provides a way to expose INI-style configuration as Python modules. It should be compatible with Python 3 after changing the import of ConfigParser on line 1 to configparser, but I only tested it on Python 2.7.

import ConfigParser, imp, os, sys

def __init__(self, prefix):
self.prefix = prefix

if name in sys.modules:
return sys.modules[name]

module = imp.new_module(name)
if name == self.prefix:
# 'from config import foo' gets config then config.foo,
# so we need a dummy package.
module.__package__ = name
module.__path__ = []
module.__file__ = __file__
else:
# Try to find a .ini file
module.__package__, _, fname = name.rpartition('.')
fname += '.ini'
module.__file__ = fname
if not os.path.isfile(fname):
raise ImportError("Could not find a .ini file matching " + name)
else:

sys.modules[name] = module
return module

def find_module(self, name, path=None):
if name.startswith(self.prefix):
return self
else:
return None

"""Load ini-style file f into module m."""
cp = ConfigParser.SafeConfigParser()
for section in cp.sections():
setattr(m, section, dict(cp.items(section)))

def init(package='config'):
"""Install the ini import hook for the given virtual package name."""
sys.meta_path.append(INILoader(package))

Most of this code should be fairly easy to follow. The magic of the import hook itself is all in the INILoader class, and exactly how that works is specified in PEP 302.

# Usage

So how do you use this? Basically, you must simply run init(), then any imports from the specified package (config by default) will be resolved from an .ini file rather than an actual Python module. Sections in a file are exposed as dictionaries under the module.

An example is much more informative than the preceding short description, so here’s one. I put the code on my Python path as INIImport.py and created foo.ini with the following contents:

[cat]
sound=meow
[dog]
sound=woof
[cow]
sound=moo


It has three sections, each describing an animal. Now I load up a Python console and use it:

>>> import INIImport
>>> INIImport.init()
>>> from config import foo
>>> foo.cat
{'sound': 'meow'}
>>> foo.dog['sound']
'woof'


This has the same semantics as a normal Python module, so it can be reloaded or aliased just like any other module:

>>> import config.foo
>>> foo == config.foo
True
<module 'config.foo' from 'foo.ini'>


The ability to reload this module is particularly handy, because my normal configuration module approach doesn’t provide an easy way to reload the file.

## Improvements, Limitations

Some addition improvements come to mind if I were to release this experiment as production-quality code. Notably, additional path manipulations for finding .ini files would be useful, such as taking a path argument to init(), supplying a set of directories to search within. Having a way to remove the import hook that it installs would also be good, and straightforward to implement. There’s no way to get all the sections in the module, so it would also be useful to export the sections somehow– perhaps by having the module support the mapping protocol (so all the sections could be retrieved with module.items(), for example).

The main limitation of this scheme is that it has no way to determine the desired type of loaded configuration values, so everything is a string. This is a typical limitation when using the ConfigParser module, but compared to a more robust configuration scheme such as defining objects in a Python file (such as Django does), this might be an annoying loss of expressiveness. The values can always be coerced to the required type when retrieving them, but that’s a bit of unnecessary extra code in whatever uses the configuration.

It may also be useful to provide a way to write configuration back to a file when modifying a config module, but my simplistic implementation makes no attempt at such. Doing so would not be terribly difficult, just involving some wrapper objects to handle member assignment for sections and items, then providing a mechanism for saving the current values back to the original file.

## Postlude

This made for an interesting experiment, and it should be a handy example for how to implement import hooks in Python. You may use this code freely within your own work, but I’d appreciate if you leave a note here that it was useful, and link back to this post.