Considering my backup systems

With the recent news that Crashplan were doing away with their “Home” offering, I had reason to reconsider my choice of online backup backup provider. Since I haven’t written anything here lately and the results of my exploration (plus description of everything else I do to ensure data longevity) might be of interest to others looking to set up backup systems for their own data, a version of my notes from that process follows. The status quo I run a Linux-based home server for all of my long-term storage, currently 15 terabytes of raw storage with btrfs RAID on top. The choice of btrfs and RAID allows me some degree of robustness against local disk failures and accidental damage to data.

Read More…

Reflecting on Breath of the Wild

I’ve been enjoying The Legend of Zelda: Breath of the Wild recently, and reflected some on what makes it interesting to me from a non-gameplay perspective. This document is a version of those musings organized for publication, though perhaps less well organized than my usual writings- I am by no means a skilled critic, but spending longer in composing this would likely just delay its completion to little benefit. Note that at the time of this writing I have not yet completed the game, but there are still some minor spoilers for the early portions of the game and general premise.

Read More…

An illustrated guide to LLVM

At the most recent Rust Sydney meetup (yesterday, “celebrating” Rust’s second birthday) I gave a talk intended to provide an introduction to using LLVM to build compilers, using Rust as the implementing language. The presentations were not recorded which might have been neat, but I’m publishing the slides and notes here for anybody who might find it interesting or useful. It is however not as illustrated as the title may seem to suggest. It’s embedded below, or you can view standalone in your browser or as a PDF, available with or without presenter notes. Navigate with the arrow keys on your keyboard or by swiping.

Read More…

Evaluating async I/O

A common belief among software engineers of a certain temperament seems to be that asynchronous I/O is the only way to achieve good performance in a server application. There is also often confusion around async, since there are several ways in which it can appear: Async I/O; eventing async keywords in languages often using Promises async as a keyword is often useful, though most frequently in the context of evented I/O under the hood. Promises or Futures are very useful when taking advantage of parallelism, but many applications don’t take any advantage of doing that kind of parallelism- they instead adopt async structures and their disadvantages through a cargo-cult assertion that doing so will make the programs fast.

Read More…

Quick and dirty web image optimization

Given a large pile of images that nominally live on a web server, I want to make them smaller and more friendly to serve to clients. This is hardly novel: for example, Google offer detailed advice on reducing the size of images for the web. I have mostly JPEG and PNG images, so, jpegtran and optipng are the tools of choice for bulk lossless compression. To locate and compress images, I’ll use GNU find and parallel to invoke those tools. For JPEGs I take a simple approach, preserving comment tags and creating a progressive JPEG (which can be displayed at a reduced resolution before the entire image has been downloaded).

Read More…

sax-ng

Over on Cemetech, we’ve long had an embedded chat widget called “SAX” (“Simultaneous Asynchronous eXchange”). It behaves kind of like a traditional shoutbox, in that registered users can use the SAX widget to chat in near-real-time. There is also a bot that relays messages between the on-site widget and an IRC channel, which we call “saxjax”. The implementation of this, however, was somewhat lacking in efficiency. It was first implemented around mid-2006, and saw essentially no updates until just recently. The following is a good example of how dated the implementation was: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 // code for Mozilla, etc if (window.

Read More…

Web history archival and WARC management

I’ve been a sort of ‘rogue archivist’ along the lines of the Archive Team for some time, but generally lack the combination of motivation and free time to directly take part in their activities. That said, I do sometimes go on bursts of archival since these things do concern me; it’s just a question of when I’ll get manic enough to be useful and latch onto an archival task as the one to do. An earlier public example is when I mirrored ticalc.org. The historical record contains plenty of instances where people maintained copies of their communications or other documentation which has proven useful to study, and in the digital world the same is likely to be true.

Read More…

Claude Shannon hates this one weird trick!

There was a question posted to /r/AskComputerScience recently: “does this compression scheme look fishy to you?". The algorithm in question, called “press” by its author, makes some.. bold claims in the README: By stringing together 1/0s, using 1’s as negative and a 0 as a positive, I’ve sublimely made a perfect compression. Through outputting into hexadecimal. This shortens <=4096 ‘bits’ into a 2-byte hexadecimal. I know enough to immediately scoff at these claims, and if somebody hadn’t specifically asked about it, I would have left it there and not thought twice. Since somebody asked, however, I thought it would be fun to give this a thorough examination.

Read More…

HodorCSE

Localization of software, while not trivial, is not a particularly novel problem. Where it gets more interesting is in resource-constrained systems, where your ability to display strings is limited by display resolution and memory limitations may make it difficult to include multiple localized copies of any given string in a single binary. All of this is then on top of the usual (admittedly slight in well-designed systems) difficulty in selecting a language at runtime and maintaining reasonably readable code. This all comes to mind following discussion of providing translations of Doors CSE, a piece of software for the TI-84+ Color Silver Edition1 that falls squarely into the “embedded software” category.

Read More…

"A Sufficiently Smart Compiler"

On a bit of a lark today, I decided to see if I could get Spasm running in a web browser via Emscripten. I was successful, but found that something seemed to be optimizing out most of main() such that I had to hack in my own main function that performed the same critical functions and (for the sake of simplicity) hard-coded the relevant command-line options. Looking into the problem a bit further, I observed that not all of main() was being removed; there was one critical line left in. The beginning of the function in source and the generated code were as follows.

Read More…