Benchmarking Ropes: 81,000 times faster than java.lang.String

One of the primary attractions of Java is the incredible richness and robustness of its standard library, the Java Development Kit (JDK). “Basically bug-free” is a prerequisite for any standard library, but the JDK goes above-and-beyond when it comes to how well each of these subsystems is implemented for the general-purpose use case.

A corollary of this is that you should almost never implement your own basic types. Designing and implementing production-grade, high-performance algorithms and data-structures requires its own skillset, and even if you have it, chances are you’ll encounter some distant edge case in the far-off future. Suffice it to say that I’m generally not a fan of “implement your own priority queue” style whiteboard interviews.

But over the years at Sunshower, building infrastructure that’s, well, somewhat outside of the mainstream, I have encountered particular exceptions to this rule and open-sourced them under Sunshower-Arcus. One of the more niche exceptions is a data-structure supporting fast, persistent modifications.

What do we need Ropes for when we have Strings?

My current project is a new, structured infrastructure and deployment language called “Breeze”. As part of that, we need a good IDE supporting code-completion, symbol navigation, etc. I’ll go into Breeze into more detail at some point, but writing parsers for IDEs is somewhat different from writing parsers for other purposes. IDE-quality parsers must be fault-tolerant, which means that when they encounter an error, they must not just “give up”–they should provide helpful hints like “I see this is an assignment–we have these candidate symbols in the current scope” rather than “parse error: expected SYMBOL at LINE 22, COLUMN 15” or something to that effect.

Another aspect of IDE parsers is that the user spends a lot of time modifying sections of a document that are in pathological locations for contiguous, array-based strings. For example, when I add a method to a class in my IDE, each keystroke performs an insertion or deletion somewhere within the document string, usually towards the middle of it. This entails the following for contiguous-memory strings:

  1. Copy the characters before the operation’s start index
  2. Copy the characters after the operation’s end index
  3. Place the operation
  4. Concatenate (1, 3, 2)

For a typical-sized source file (several kB at a minimum), this is excruciatingly slow and generate an unacceptable amount of heap-pressure (creating short-lived objects that must be reclaimed), and so many IDEs use a data-structure called a “gap-buffer” which “freezes” the regions outside of the current edit-operations into contiguous-memory strings and “thaws” the regions within the current edit-operations into a friendlier data-structure like a linked list.

There is an even better data-structure for this sort of workload, one that is actually a reasonable substitute for effectively all string operations: ropes.


A rope is a balanced binary tree (not a binary-search tree) whose leaves are comprised of small contiguous-memory strings. For instance, a Rope constructed from the string “Hello my name is Josiah” may look like this:

│ ├╴rope(5,5)[Hello]
│ └╴rope(3,3)[ my]
│ ├╴rope(3,3)[ na]
│ └╴rope(4,4)[me i]
└╴rope(7,7)[ Josiah]

The “internal” nodes are merely pointers to either other internal nodes, or pointers to contiguous subsequences of characters. The balancing criteria for such a tree is:
1. Each internal node must have two children
2. Each leaf node must be “flat”–contain a contiguous array of characters and no children.

This is a less restrictive balancing than say, AVL balancing (although it is similar), but it does guarantee us a maximum tree height of log2(n), where n is the number of characters in the tree. Each node is identified by its “weight”, or the number of characters contained within its left subtree. For leaf nodes, the weight is the length
of its character array.

Persistence and Concurrency

The JDK designers designed java.lang.String to be immutable for quite a few reasons that span security, intrinsification, caching (interning) and concurrency-safety. There are “mutable” strings in Java such as the java.lang.StringBuilder type which can be more ergonomic and faster for some operations, but ultimately even StringBuilder relies on a contiguous array of memory. Amortized constant-time append operations to StringBuilder require doubling the size of its backing character array each time the array is resized, and StringBuilder is not thread-safe or concurrent (although StringBuffer is).

In the functional programming paradigm, many core data-structures are “tree-like”, and support a feature known as “persistence” to solve this problem. When a persistent data-structure is modified, it retains its structure and returns a “copy” containing the modifications. Many data-structures can share a substantial amount of substructure, reducing the cost of a copy operation. This is the approach to concurrency that our ropes take: modifying a rope will return another rope sharing as much of the original’s substructure as possible (such as leaf nodes). This is a trade-off between performance and concurrency-safety. As we’ll see, this is not as expensive as it might seem.

Finally, Benchmarks!

The benchmarks are run via JMH and are available at

Hardware Configuration:

CPU: Intel Xeon E3-1535M@3.10 GHz
OS: Debian (5.10.0-13-amd64 #1 SMP Debian 5.10.106-1) Virtual Machine with 8 CPUs (2 sockets, 4 cores each), 32 GB RAM. VMWare Workstation 16.2.2

JVM Configuration

OpenJDK Corretto
max heap-size: 8GB
default heap-size: 4GB

This could certainly be tuned to support the operations that Ropes rely on, but benchmarking against defaults seemed to be the most reasonable approach to me.

Each benchmark was conducted on Strings and Ropes constructed from the same byte array ranging in size from 1b to 10mB by factors of 10. Obviously the axes are logarithmic.

String Construction vs. Rope Construction

This benchmark simply tests the amount of time required to construct a String vs. a Rope from the same character array:

Note that the construction time of ropes vs. strings is quite similar up until ~100 bytes. This is because ropes under 199 characters (our default split-size) degenerate to strings. As an aside, the split-size should be prime, and I selected 199 after some microbenchmarks indicated that it provided a good balance between minimizing internal nodes (tree-height) and providing good modification performance.

The ratio of throughput of rope construction vs. string construction is quite consistent once the fallback behavior is no longer relevant:

In any case, constructing a string was never a full order of magnitude faster than constructing a rope for any operand sizes in our benchmarks. This is quite remarkable given the amount of internal structure that ropes possess.

Rope Insertions

The next benchmark I performed is inserting a 100-byte string into the center of Ropes and Strings ranging in length from 1 byte to 10 mB. Object construction is not counted towards this benchmark’s throughput:

This is an operation where Ropes really shine. On the logarithmic scale presented, Ropes decrease in throughput relative to operand size according to an (approximately) constant factor, and overtake Strings in throughput at around the 1 kB mark. Recall that a constant line on a log-scale chart is a logarithmic factor, which we do observe. Furthermore, Strings degrade in performance linearly (linear downward trend) for this operation, until ropes are 81,000x faster for 100Mb operand sizes at which point java.lang.String on my configuration is only capable of 21 operations/second.

Additional Features

The Sunshower rope structure is intended to be a drop-in replacement for java.lang.String. Although it’s still relatively new, it does what you’d expect it to do such as:
1. Works with JDK regular expressions and patterns
2. Is a good candidate for hashtable keys
3. Implements Comparable correctly

In addition, certain operations that Ropes are relatively slow at such as String comparisons, charAt, splitting over non-regex substrings are heavily optimized as we provide high-quality string search algorithms such as tuned Boyer-Moore-Horspool implementations which we’ve compared with competitors such as Knuth-Morris-Pratt, Rabin-Karp, and Raita (which was surprisingly poorly performing).

This rope implementation also provides a block-iteration structure granting efficient in-order access to the leaf nodes of the data-structure.


Thanks for reading! I’ll try to provide additional benchmarks as they’re requested. I’ve only documented several of the benchmarks that I’ve currently run. The benchmark data for the described benchmarks can be found at The benchmarks themselves can be found at:

I hope this post has demonstrated that a quality rope implementation can be a viable contiguous-memory string replacement for most or all operations! This library is fully open-source and available in Maven Central.

Special Thanks

I’d like to thank Reddit user mauganra_it for their help with some implementation details such as recommending the use of java.lang.Strings as the primary leaf-node data-structure as opposed to character arrays. This had some measurable impacts on performance and memory utilization that I hope to discuss soon.

Thoughts on “Testing is an Unsolved Problem”

Beware of bugs in the above code; I have only proved it correct, not tried it
~ Donald Knuth, Notes on the van Emde Boas construction of priority deques: An instructive use of recursion (1977)

I stumbled across Jason Arbon’s excellent and thought-provoking post the other day, “Testing is an Unsolved Problem” and it really got my noodle going. I feel pretty comfortable saying that, theoretical considerations aside, everyone in software finds non-trivial testing a challenging problem (if not a downright pain-in-the-ass).

In a certain sense, anybody can write a program that doesn’t work, and these programs only differ in how broken they really are. Moreover, I think it’s safe to say that most programs are at least partially broken in that same sense. Worse, and to (hopefully accurately) paraphrase the original, it can be hard or impossible to even quantify how broken a program is. That said, I do somewhat disagree that theoretical correctness is the challenging aspect of testing. It turns out that huge, useful classes of algorithms are decidable, and it’s very surprising to see how complex decidable (and even efficiently-decidable) some algorithms can be (with some big caveats).

There’s even a cool, possibly uncoincidental parallel between decidability in formal systems and testing in informal systems, viz. that you can write huge, “complex” programs in a decidable system that loses its decidability when you try to get anything into or out of it. That’s right: introducing IO to a decidable, formal system causes it to lose its decidability. Generalizing this idea motivates the concepts such as monads and effect systems that make functional programmers so insufferable after a few drinks. (The little Yost readability face is shrieking at me right now. I’m a little concerned that it may become corporeal and exact some nightmare readability revenge.)

This mirrors the fact that most of our defects stem from just a few locations:

  1. Getting data to/from users
  2. Getting data to/from integrations
  3. Concurrency

Note that (1) and (2) are effectively the same problem, which mirrors the theoretical “IO” issue. As an aside, we’ve largely been able to eliminate concurrency issues by modeling concurrent operations as a Coffman-Graham schedule. Our implementation is open-source here. This has the added advantage that if we encounter a concurrency issue, we can easily transform it into a single-actor model by linearizing the computation graph.

Anyhoodles, if you made it this far, I’d like to talk about how we really mitigated (1) with Vaadin and Aire::Test.

Keeping it in one process

Whenever you have a client (browser) and a server, you have a distributed system. Automated testing of distributed systems sucks, hard. It sucks so hard that is the first place I’ve ever worked that has developed its own in-house technologies to address the problem. Tools like Selenium/Watir/etc. are fantastic, but to be perfectly honest, I haven’t seen them very effectively used in most orgs. I’ve encountered. Frankly, the it’s pretty hard to write tests that aren’t brittle. Why you ask? In a word, LAYOUT.

Let me show you what I mean. I’m currently working on a new UI that provides an excellent, simple example. Check out that button outlined in blue:

Where is it at? Well, it’s human-navigable location is:

Main > Navbar > Zephyr > Modules > Grid[select] > Right Nav > Module Lifecycle > Stop

Not too bad, but in Selenium that’s gonna probably be (I’m sure they have utilities to simplify this–haven’t used it in a while)


  1. Open Browser
  2. Navigate to Main
  3. Locate Zephyr in top menu
    4 Click Zephyr
  4. Wait for Page Load
  5. Click Modules tab
  6. Wait for Module List to populate
  7. Locate module element
  8. Click module element
  9. Wait for side-nav element to load
  10. Locate button with CSS selector html body div#outlet aire-application-layout aire-panel aire-navigation-bar.vertical aire-drawer.verticalright vaadin-vertical-layout vaadin-menu-bar > vaadin-menu-bar-button > vaadin-context-menu-item > vaadin-context-menu-item > vaadin-button:nth-of-type(2). (This may not be simplest possible selector, but as your pages grow in complexity you generally need to increase your selectivity)
  11. Click button

1. Wait until the module-list contains a stopped module

I mean, in the scheme of things, and with a mature product, team, and methodology, this can be a great way to test stuff. You can reduce the test-development complexity by instituting some selector/UI conventions, refactoring out tests into pages, etc. The problem is that tiny things are going to break you. We change our components’ markup all the time, and in the Selenium model it virtually always breaks something.

Using Aire/Zephyr

What’s an equivalent test look like in Aire and Zephyr?

@Routes(scanClassPackage = ModuleGrid.class)
class ModuleGridTest {
void ensureStoppingAModuleThroughTheStopButtonStopsTheModule(
@Select("vaadin-vertical-layout > aire-drawer")
Drawer drawer, @Context TestContext context, @Autowired Zephyr zephyr) {
val $ = context.downTo(drawer);
val button = $.selectFirst("vaadin-button:nth-of-type(2)", Button.class)
.get(); // throws exception if it's not there
.allMatch(plugin -> plugin.getLifecycle().getState() == State.Active));;
assertEquals(1, zephyr.getPlugins(State.Resolved).size());

Boom. Here’s the tests in our CI:

It’s fully integrated with Spring Test (possibly the most useful middleware I’ve ever encountered), and because it uses Vaadin’s component model rather than the browser’s, it’s insensitive to DOM changes while being sensitive to view changes. Furthermore, it fully integrates with Spring/JTA transactions, etc. allowing you to test another challenging aspect: the data model (but that’s a topic for another time).


If you’ve made it this far, I really appreciate it! I hope it was helpful, and if this is something you’d like your org. to adopt, it’s completely free and open-source. I’d be happy to lend what time I can to assist in adoption. Despite requiring an entire CSS selector engine to implement, velocity is the highest it’s ever been.

Building a Home Cloud with Proxmox: DNS + Terraform


In our last post, we configured a Ceph storage cluster, which we’ll be using as the storage for our virtual machines that we’ll be using to host Kubernetes.

Before we get to that, however, we need to configure our DNS environment. Recall that, in part 2, we configured a Windows server to act as our DNS and DHCP server. For this, you don’t need a Windows Server installation, you just need an RFC 2136-compliant DNS server like PowerDNS. Keep in mind that RFC 2136 has some security vulnerabilities, so don’t use dynamic DNS in a high-security environment or an environment wherein untrusted devices may be on the network.

To provision our DNS names, we’ll be interacting with our DNS server via the Terraform DNS provider. Terraform’s easy to set up, so we won’t be covering that.

Note that this solution requires a fair amount of work in Bash. Other solutions like Chef/Puppet/Ansible/Salt may strictly be better-suited for this sort of work, but since our environment is quite homogeneous (99% Debian), we’ll just script this using basic tools.

A note on Terraform Typing

Terraform’s variable/parameter management system, specifically its inability to share variables between modules can result in some duplication between modules. This isn’t the cleanest, but I don’t feel like it’s a big enough problem to warrant bringing in other tools like Terragrunt. Terraform’s type-system supports a flavor of structural typing wherein you can declare a variable with a subset of another variable’s fields and supply the first variable as the second variable. For instance:

variable "A" {
type = object({
name = string
ip = string
variable "B" {
type = object({
name = string
view raw hosted with ❤ by GitHub


   // module B declares B, and can use a value of type A as the value for b:B
    b = var.a

We’ll be using this feature to share variable values across modules.


Since we’re reserving the upper range of our /24 subnet for MetalLB dynamically-provisioned IP addresses, we’ll configure the worker nodes with static IP addresses assigned from the lower range of our subnet. Since we have a small number of physical machines, we’ll opt for fewer, larger worker-nodes instead of more, smaller worker-nodes. The subnet-ranges we’ll select for these roles are as follows:

  1. 2-10: infrastructure (domain-controller, physical node IPs, etcd VMs, Kubernetes leader nodes)
  2. 10-30: static addresses (worker nodes)
  3. 30-200: DHCP leases for dynamic devices on the network + MetalLB IPs
  4. 200-255: available

To set that up in Windows Server:

  1. Navigate to the Server Manager
  2. From the top-right Tools menu, select DHCP
  3. Right-click on the IPv4 node under your domain-controller host-name and select New Scope
  4. Provide a Start address of <subnet prefix>.30 and an end-address of 200
  5. Continue through the wizard providing your gateway addresses and your exclusions. For the exclusions, select <subnet prefix>-1 through 30.
  6. Finish the wizard and click Create to create your scope.
  7. Right-click on the created scope node in the tree. Select the DNS tab, then enable DNS dynamic updates. Select the Dynamically update DNS records only if requested by the DHCP clients.
  8. Click Ok to save your changes.

Configure your DD-WRT Router

Navigate back to your gateway router. Most capable routers will have an option to either act as a DHCP server (the default), or to act as a DHCP forwarder. For DD-WRT, select the `DHCP Forwarder option and provide the IP address of your DHCP server as the target.

Configure your DNS Server

Go back to your domain controller.

  1. From the Server Manager, select DNS
  2. Locate the Forward Lookup Zone node in the DNS Manager tree that you created (ours was Right-click on the Forward Lookup Zone node and click Properties
  3. In the General Tab, select the Nonsecure and Secure option from the Dynamic updates configuration. Note: This is not suitable for an open-network or one admitting untrusted/unknown devices
  4. Click Ok to apply.

Congrats! We’re now ready to provision our DNS in Terraform.

Terraform Configuration

Note: We’ve provided our Terraform configurations here, so feel free to point them at your infrastructure if you’ve been following along instead of creating your own.

In the directory of your Terraform project ($TFDIR), create a directory called dns.
1. In your dns directory, create 3 files:,, and

Your file should contain:

terraform {
required_providers {
dns = {
source = "hashicorp/dns"
provision DNS entries for each host in the `hosts` list
resource "dns_a_record_set" "virtual_machine_dns" {
for_each = {for vm in var.hosts: => vm}
zone =
name =
addresses = [
provision the DNS A-record for the API server
resource "dns_a_record_set" "api_server_dns" {
addresses = [
zone =
name = var.api_dns
view raw hosted with ❤ by GitHub

Now, in your Terraform project $TFDIR (the parent of the dns directory you’re currently in, create,, and Symlink the dns/ to this directory as via ln -s $(pwd)/ $(pwd)/dns/ to make your DNS variables available to your root configuration.

Parent module

In your $TFDIR/, add the following block:

terraform {
required_version = ">=0.14"
required_providers {
dns = {
source = "hashicorp/dns"
version = "3.0.1"
view raw hosted with ❤ by GitHub

Configure the DNS provider to point to your RFC-2136-compliant DNS server:

provider "dns" {
update {
server = var.dns_server.server
view raw hosted with ❤ by GitHub

Call the dns submodule:

Create DNS entries for virtual machines
module "dns_configuration" {
for_each = var.cluster_nodes
source = "./dns"
dns_server = var.dns_server
hosts = each.value
api_dns = var.api_server
api_domain = var.domain
api_ip = var.load_balancer
view raw hosted with ❤ by GitHub

At this point, we should be ready to actually provide our Terraform variables.

  1. Create a file in your $TFDIR directory called–Terraform will automatically pick up any in the root directory according to the naming convention *.auto.tfvars. Be sure to never check these into source-control. I add the .gitignore rule **/*.tfvars

In your $TFDIR/ file, add the following configurations (substitute with your values):

dns_server = {
// replace with your zone-name (configured above)
zone = "" // note the trailing period here--it's mandatory
// replace with your DNS server's IP
server = ""
// this generates the DNS configuration for ``
api_dns = "kubernetes"
api_domain = ""
api_ip = "" // provide the API you

Finally, we’ll add our Kubernetes cluster DNS configurations:

cluster_nodes = {
etcd_nodes = [
name = "etcd-1"
ip = ""
name = "etcd-2"
ip = ""
name = "etcd-3"
ip = ""
k8s_leaders = [
name = "k8s-leader-1"
ip = ""
name = "k8s-leader-2"
ip = ""
k8s_workers = [
name = "k8s-worker-1"
ip = ""
name = "k8s-worker-2"
ip = ""
name = "k8s-worker-3"
ip = ""

At this point, running terraform apply --auto-approve should quickly generate the DNS entries. You should navigate to your DNS server and refresh the forward lookup zone to confirm that your entries exist.


In this part, we configured DHCP, DNS, and provided a Terrform module that provisions DNS entries for each of the VMs that we’ll provision in subsequent posts. Next time, we’ll create a Terraform module that will provision virtual machines of the correct roles for our Kubernetes cluster. Our final post of the series will show how to actually deploy the Kubernetes cluster–stay tuned!

Building a Home Cloud with Proxmox Part 3: Configuring Ceph and Cloud-Init


Last time, we configured a Windows Server domain controller to handle DNS, DHCP, and ActiveDirectory (our LDAP implementation). In this post, we’ll be configuring the Ceph distributed filesystem which is supported out-of-the-box by Proxmox.

Ceph Storage Types

Ceph supports several storage types, but the types we’re interested in for our purposes are

  1. Filesystem
  2. Block

A Ceph filesystem behaves like a standard filesystem–it can contain files and directories. The advantage to a Ceph filesystem for our purposes is that we can access any files stored within it from any location in the cluster, whereas with local-storage (e.g. local-lvm or whatever), we can only access them on the node they reside upon.

Similarly, a Ceph block device is suitable for container and virtual machine hard-drives, and these drives can be accessed from anywhere in the cluster. This node-agnosticism is important for several reasons:

  1. We’re able to create base images for all our virtual machines and clone them to any node, from any node
  2. We’re able to use Ceph as a persistent volume provider in Kubernetes

In this scheme, your Ceph IO performance is likely to be bottlenecked by your intra-node network performance. Our 10Gbps network connections between nodes seems to perform reasonably well, but is noticeably slower than local storage (we may characterize this at some point, but this series isn’t it).

Drive Configuration

In our setup, we have 4, 4 TB drives on each node. Ceph seems to prefer identical hardware configurations on each node it’s deployed on, so if your nodes differ here, you may want to restrict your Ceph deployment to identical nodes.

Create your Ceph partitions on each node

Careful Ensure that the drives you’re installing Ceph onto aren’t being used by anything–this will erase everything on them. You can colocate Ceph partitions with other partitions on a drive, but that’s not the configuration we’re using here (and how well it performs depends on quite a few factors)

We’re going to use /dev/sdb for our Ceph filesystem on each node. To prepare each drive, open a console into its host and run the following:

  1. fdisk /dev/sdb
  2. d to delete all the partitions. If you have more than one, repeat this process until there are no partitions left.
  3. n to create a new partition
    1 g to create a new GPT partition table
  4. w to write the changes

Your disks are now ready for Ceph!

Install Ceph

On each node, navigate to the left-hand configuration panel, then click on the Ceph node. Initially, you’ll see a message indicating that Ceph is not installed. Select the advanced option and click Install to continue through the wizard.

When you’re presented with the options, ensure that the osd_pool_default_size and osd_pool_default_min_size configurations are set to the number of nodes that you intend to install Ceph on. While the min_size can be less than the pool_size, I have not had good luck with this configuration in my tests.

Once you’ve installed Ceph on each node, navigate to the Monitor node under the Ceph configuration node and create at least 1 monitor and at least 1 manager, depending on your resiliency requirements.

Create an object-storage daemon (OSD)

Our next step is to allocate Ceph the drives we previously provisioned (/dev/sdb). Navigate to the OSD node under the Ceph node, and click create OSD. In our configuration, we use the same disk for both the storage area and the Ceph database (DB) disk. Repeat this process for each node you want to participate in this storage cluster.

Create the Ceph pool

We’re almost there! Navigate to the Pools node under the Ceph node and click create. You’ll be presented with some options, but the options important to us are:

  1. size: set to the number of OSDs you created earlier
  2. min-size: set to the number of OSDs you created earlier

Ceph recommends about 100 placement groups per OSD, but the number must be a power of 2, and the minimum is 8. More placement groups means better reliability in the presence of failures, but it also means replicating more data which may slow things down. Since we’re not managing dozens or hundreds of drives, I opted for fewer placement groups (16).

Click Create–you should now have a Ceph storage pool!

Create your Ceph Block Storage (RBD)

You should now be able to navigate up to the cluster level and click on the storage configuration node.

  1. Click Add and select RBD.
  2. Give it a memorable ID that’s also volume-friendly (lower case, no spaces, only alphanumeric + dashes). We chose ceph-block-storage
  3. Select the cephfs_data pool (or whatever you called it in the previous step)
  4. Select the monitor node you want to use
  5. Ensure that the Nodes value is All (no restrictions) unless you want to configure that.
  6. Click Add!

Congrats! You should now have a Ceph block storage pool!

Create your Ceph directory storage

In the same view (Datacenter > Storage):

  1. Click Add and select CephFS
  2. Give it a memorable ID (same rules as in the previous step), we called ours ceph-fs
  3. Ensure that the content is selected to all the available options (VZDump backup file, ISO image, Container Template, Snippets)
  4. Ensure the Use Proxmox VE managed hyper-converged cephFS option is selected

Click Add–you now have a location you can store VM templates/ISOs in that is accessible to every node participating in the CephFS pool! This is important because it radically simplifies configuration via Terraform, which we’ll be writing about in subsequent posts.

Create a Cloud-Init ready VM Template

We use Debian for virtually everything we do, but these steps should work for Ubuntu as well.

  1. Upload a Debian ISO (we use small installation image) to your CephFS storage
  2. Perform a minimal installation. The things you’ll want to pay attention to here are:
    • Ensure your domain-name is set to whatever domain-name you’ve configured on your domain-controller
    • Only install the following packages: SSH Server and Standard System Utilities
  3. Finish the Debian installation wizard and power on your VM. Make a note of the ID (probably 100 or something) I refer to as VM_ID in subsequent steps
  4. In the Proxmox console for your VM, perform the following steps:
    • apt-get update
    • apt-get install cloud-init net-tools sudo
  5. Once APT is finished installing those packages, edit your sshd_config file at /etc/ssh/sshd_config using your favorite editor (e.g. vi /etc/ssh/sshd_config) and ensure that PermitRootLogin is set to yes from its default of prohibit-password. These are all air-gapped in our environment, so I’m not too worried about the security ramifications of this. If you are, be sure to adjust subsequent configurations to use certificate-based auth.
  6. Save the file and shut down the virtual machine

Now, let’s enable CloudInit!

On any of your nodes joined to the CephFS filesystem:

  1. Open a console
  2. Configure a CloudInit drive:
    • Using the id for the Ceph RBD/block store (referred to as <BLOCK_STORE_ID> we configured above (ours is ceph-block-storage), and the VM_ID from the previous section, create a drive by entering qm set <VM_ID> --ide2 <BLOCK_STORE_ID>:cloudinit. For instance, in our configuration, with VM_ID = 100, this is qm set 100 --ide2 ceph-block-storage:cloudinit. This should complete without errors.
  3. Verify that your VM has a cloud-init drive by navigating to the VM, then selecting the Cloud-Init node. You should see some values there instead of a message indicating Cloud-Init isn’t installed.

You should also verify that this machine is installed on the correct block-storage by attempting to clone it to another node. If everything’s configured properly, cloning a VM to a different node should work seamlessly. At this point, you can convert your VM to a template for subsequent use.


In this post we installed and configured Ceph block and filestores, and also created a Cloud-Init capable VM template. At this point, we’re ready to begin configuring our HA Kubernetes cluster using Terraform from Hashicorp. Our Terraform files will be stored in our public devops Github repository

Building a Home Cloud with Proxmox Part 2: Domain Configuration


Last time I went over how to configure a router equipped with DD-WRT for managing a home cloud, as well as installing the awesome Proxmox virtualization environment. This time, we’ll go over how to configure a Windows domain controller to manage ActiveDirectory profiles, as well as DNS.

General configuration

The end goal is to provision a cluster that’s running a Windows domain controller (for DNS and ActiveDirectory), as well as a Kubernetes cluster. Once we have that we’ll deploy most of our services into the Kubernetes cluster, and we’ll be able to grant uniform access to every object via the domain controller.

Note that multitenancy isn’t a goal for this home cloud, although the domain controller is where we would add that. We could also partition our subnet into more subnets, providing functionality analogous to public cloud providers’ notions of Virtual Private Clouds.

Step 1: Create a Windows Server VM

  1. Ensure you have a suitable storage location:
    1. Open a shell to one of your nodes
    2. Type lsblk–it should show you a tree of your available hard drives and partitions
    3. If you don’t see a partition under your drives, type fdisk. In fdisk <drive>, where <drive> is the drive you want (probably not /dev/sda–that’s likely where Proxmox is installed), create a GPT partition by typing g. Write the changes by entering w
    4. Now you should be able to provision a VM on that drive/partition in the subsequent steps.
  2. Download the Windows Server ISO
  3. Upload the Windows Server ISO to Proxmox:
    1. The Windows server ISO is too large to upload via the GUI, so pick a node (e.g., and scp it up: scp <iso file> root@<node>:/var/lib/vz/template/iso, for example:scp windows_server_2019.iso
    2. Create a new VM
      • Select Windows as the type, the default version should be correct, but verify it’s something like 10/2019/
      • Give it at least 2 cores, 1 socket is fine. The disk should have at least 50GB available.
      • I provided mine with 16 GB RAM, but we have pretty big machines. 4 works fine, and you can change this later.
    3. One of the prompts will be to assign your server to a domain. You should enter the domain that you want to control (ours being

You should be able to access your Windows Server VM now, but it’s not quite ready.

Configure your Domain Controller’s DNS

The first thing we’ll want to do is ensure that our domain controller is at a deterministic location. For this, we’ll want to
1. Assign it a static DHCP lease in our router
1. Assign it a static IP locally
1. Assign it a DNS entry in our router

Assign the DHCP lease

  1. In the Windows VM, open an elevated console and run ipconfig /all. This will show all of your IP configurations, as well as the MAC addresses they’re associated with. Locate the entry for your subnet (ours is, and locate the associated MAC address (format AA:AA:AA:AA:AA:AA, where AA is a 2-letter alphanumeric string).
  2. Once you have your MAC address, log into your router, navigate to the Services/Services subtab, find the Static Leases section, and create an entry whose MAC address is the one you just obtained, and whose IP address is towards the low-end of your subnet (we selected Choose a memorable hostname (we went with and enter that, too.

Save and apply your changes.

Assign the static IP locally

  1. In your Windows VM, navigate to Control Panel > Network and Internet > Network Connections. You’ll probably only have one.
  2. Right-click on it and select Properties. Navigate to Internet Protocol Version 4 (TCP/IPv4)
  3. Select Use the following IP address
    • For IP Address, enter the IP you set in the previous section (Assign the DHCP lease)
    • For Subnet mask, (assuming a /24 network), enter You can look these up here
    • For Default gateway, select your router IP (ours is

For the DNS Server Entries, you can set anything you like (that is a valid DNS server). We like Google’s DNS servers (, Notice that we’re not actually setting our router to be a DNS server, even though it is configured to be one. That’s because we generally want the domain controller to be our DNS server.

Click Ok and restart just to be sure.

Configure your Domain Controller

Your Windows Server VM will need to function as a domain controller (and DNS server) for this to work. After you’ve created the Windows Server VM, log into it as an administrator. You should be presented with the Server Manager Dialog.

  1. From the top-right, click the manage button, select Add Roles and Features. This will bring you to the Roles & Features wizard.
  2. Select Role-based or feature-based installation. Click Next
  3. Select your server. Click Next
  4. Select the following roles
    • Active Directory Domain Services
    • Active Directory Lightweight Directory Services
    • Active Directory Rights Management Services -> Active Directory Rights Management Server
    • DHCP Server
    • DNS Server

Click Install–grab some coffee while Windows does its stuff. It’ll restart at least once.

Configure DNS

Almost done! We’re going to add all the entries that we had previously added in the router to the Domain Controller so that we only need to reference one DNS server: our domain controller. Finally, we’ll join a home computer to the domain.

Add the DNS records

  1. From the Server Manager window, select Tools and click DNS from the dropdown.
  2. Expand the Forward Lookup Zones node in the tree. You should see your domain under there, possibly along with other entries such as _msdcs.<your domain.
  3. Select <your domain>. Ours is
  4. Right-click <your domain>, add new host (A or AAAA)
  5. For each of your Proxmox nodes, add the name (e.g. athena)–the fully-qualified domain name should automatically populate.
    • Provide the IP address of the associated node. If you followed along closely to the previous entry, they’ll be ( up to 1 + number of nodes)
  6. Create an entry for your router such as (e.g. and point it at your gateway IP (e.g.
  7. Create an entry for your controller such as (e.g. and point it at your controller’s static IP.

Save and restart your controller.

Create a ActiveDirectory User

  1. Log back into your domain controller and, from the server manager select the Tools menu, then Active Directory Users and Computers.
  2. From the left-hand navigator tree, expand your domain node and right-click on the Users Sub-node.
  3. Select New, and then User. Fill in your information, and make a note of the username and password, we’ll use it in the next step.

Join a computer to your domain!

On your local computer, navigate to:
1. Control Panel > System and Security > System
1. Under the section Computer name, domain, and workgroup settings, select Change settings. The System Properties dialog should appear.
1. Select the Change button to the right of To rename this computer or change its domain or workgroup, click Change
1. Set your computer name (kitchen, workstation, etc.)
1. Set member of to domain with your domain value (e.g.

Click OK–this can take a bit.

Once that’s done, restart your computer. You should now be able to log in as <DOMAIN>/ (configured in the ActiveDirectory section above)!

Your profile should automatically use your domain controller as its DNS server, so you should be able to ping all of your entries.


Windows Server makes it really easy to install and manage domains, almost enough to justify its steep price-tag. A domain server allows you to control users, groups, permissions, accounts, applications, DNS, etc. in a centralized fashion. Later on, when we configure Kubernetes, we’ll take advantage of Windows Server’s powerful DNS capabilities to automatically provision DNS entries associated with IPs allocated from our cloud subnet and use them to front services deployed into our own high-availability Kubernetes cluster.

If you don’t want to/can’t use Windows Server, you can replicate this functionality using Apache Directory, and PowerDNS. I may do a post on how to setup and configure these, but my primary goal is to move on to the Kubernetes cluster configuration, and managing DNS and LDAP are two relatively complex topics that are greatly simplified by Windows Server.

Building a Home Cloud with Proxmox: Part 1

We’re back!

Whew! As you’re probably already aware, 2020 was an incredibly weird year–but we’re getting back on the wagon and going leaner in 2021! Last year, we’d installed solar power to take advantage of Colorado’s 300+ days of sunshine, and we had some relatively beefy workstations left over from our defense work. At the same time, we noticed that our R&D AWS spend is a little high for our budget right now, so we decided to build a local cloud on top of the wonderful Proxmox virtualization management software. Let’s get started!


To begin with, we reviewed which services from AWS we needed, and came up with the following list:

  1. RDS
  2. Jenkins CI/CD
  3. Kubernetes (We hosted our own with KOPS on AWS–highly recommend it)
  4. Route53
  5. AWS Directory Service
  6. Nexus
  7. Minio

(I haven’t really decided if I want to use OpenShift or not for this cluster yet).

We’re also probably going to end up using MetalLB to allocate load-balancers for our bare-metal(ish) cluster.

Let’s get to it!

Step 0: Hardware configuration

Our setup is 3 HP Z840s with the following configuration:
1. 2TB RAM
1. 24 logical CPUs on 2 sockets
1. 4 4TB Samsung PRO 860 SSDs

For ease-of-use, we also recommend the BYTECC 4 HDMI port KVM switch (for these machines, anyway, YMMV)

Step 1: Configure your Network

  1. Check your network configuration. We have each box hooked up directly to a Linksys AC32000 running DD-WRT.
    1. (Note that these instructions are for DD-WRT–if you don’t use that, you’ll have to find equivalent configurations for your router).
    2. IMPORTANT: Ensure that your network is at least a /24 (256 total IPs)–you’ll end up using more than you expect.
    3. Useful tip: Install your nodes towards the low-end of your subnet. Ours are at 192.168.1.[2, 3, 4] with the gateway
  2. Next, configure your DD-WRT installation to supply DNS over your LAN by:
    1. Navigating to and logging in
    2. Navigate to services from the top level, then ensure the services sub-tab is selected
    3. Select Use NVRAM for client lease DB (this will allow your configuration to survive power-outages and restarts)
    4. Set Used Domain to LAN & WAN
    5. Set your LAN Domain to whatever you want it to be. This is purely local, so it can be anything. We own so we just used that. home.local or whatever should work just fine as well.
    6. Ensure DNSMasq is enabled
    7. Add bash

      to your Additional DNSMasq Options Text box
    8. Save & Apply your configuration

Step 2: Install Proxmox

Proxmox is an open-source alternative to (approximately?) vSphere from VMWare, and we’ve used it for a variety of internal workloads. It’s quite robust and stable, so I encourage you to check it out!

The first thing is to burn a USB drive with the Proxmox ISO. UEFI hasn’t worked well for me on these machines (they’re a little old), so the Rufus configuration I used was:

Partition scheme: MBR
File System: FAT32
Cluster Size: 4096 bytes (default)

I went ahead and checked the Add fixes for old BIOSes option since these are pretty old bioses. Hitting start will ask you if you want to burn the image in ISO image mode or DD image mode. With these 840s, ISO mode didn’t work for me, but DD did. Hit START and grab a coffee!

Once your USB drive is gtg, install Proxmox VE on each of the machines:

  1. Plug your USB drive into the first node in your cluster
  2. Restart/power on your node
  3. Select your USB drive from the boot options. If you followed our instructions for Rufus, it’ll probably be under Legacy instead of UEFI or whatever.
  4. IMPORTANT: Click through the wizard until it asks you which drive/partition you want to install Proxmox into. Select Options and reduce the installation size to about 30GB. I don’t know what the minimum is, but 30GB works great for this setup. This even gives you some space to upload ISOs to the default storage location. (Note that if you don’t select this the installation will consume the entire drive).
  5. Continue on the installation until you see your network configuration.
    1. Select your network (ours is–yours may not be, depending on how you configured your router (above))
    2. Input your node IP (we went with for the first, .3 for the second, etc.)
    3. Add a cool hostname. If you configured your router as we did above, you should be able to input <name>.. For instance, for our 3 nodes we went with, and
  6. Make a note of the username and password you chose for the root. Slap that into Lastpass–you’ll thank yourself later.

Repeat this process for each node you have. Once that’s complete, navigate to any of the nodes at https://<ip>:8006 where <ip> is the IP that you configured in the Proxmox network installation step. Your browser will yell at you that this isn’t a trusted site since it uses a self-signed certificate by default, but that’s OK. Accept the certificate in your browser, then login with the username and password you provided. In the left-hand pane of your Proxmox server, select the top-level Datacenter node, then click Create Cluster. This will take a second, at which point you’ll be able to close the dialog and select Join Information. Copy the contents of the Join Information text-area.

Once you have your Join Information, navigate to each of the other nodes in your cluster. Log in, then select the top level Datacenter node once again. This time, click on the Join Cluster button. Paste the Join Information into the text area, then enter the root password of the first node in the root password text field. In a second or two, you should see 2 cluster nodes under the Datacenter configuration. Repeat this process with all of the nodes you set Proxmox up on!

Configure DNS

You may need to configure your DNS within your router at this point. Click on the Shell button for each node, and run:
1. apt-get update
1. apt-get install net-tools

Then, run ifconfig. You should see something like:

enp1s0: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1500
        ether d8:9d:67:f4:5e:a0  txqueuelen 1000  (Ethernet)
        RX packets 838880  bytes 173824493 (165.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 907011  bytes 170727177 (162.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17  memory 0xef500000-ef520000  

lo: flags=73&lt;UP,LOOPBACK,RUNNING&gt;  mtu 65536
        inet  netmask
        inet6 ::1  prefixlen 128  scopeid 0x10&lt;host&gt;
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 669  bytes 161151 (157.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 669  bytes 161151 (157.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vmbr0: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1500
        inet  netmask  broadcast
        inet6 fe80::da9d:67ff:fef4:5ea0  prefixlen 64  scopeid 0x20&lt;link&gt;
        ether d8:9d:67:f4:5e:a1  txqueuelen 1000  (Ethernet)
        RX packets 834450  bytes 158495280 (151.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 902499  bytes 166814715 (159.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

This is on Your list might look different. Note that under the vmbr0 entry there’s a set of letters/digits to the right of ether (in my case, d8:9d:67:f4:5e:a1). Make a note of these for each node, then go back to your gateway, log in, and navigate to Services once again. The default subtab Services should also be selected–if it’s not, select that.

You’ll see a field called Static Leases–click add. You’ll see a set of 4 fields:
1. MAC Address
1. Hostname
1. IP Address
1. Client Lease Time

For each of the host/MAC addresses you found previously (e.g. -> d8:9d:67:f4:5e:a1), fill the MAC address field with the MAC address, the hostname with the corresponding hostname, and the IP address of the node. Save & Apply Settings. You should be able to visit any of the nodes at their corresponding DNS name. For instance, to access the Proxmox cluster from athena, I can visit


At the end of this exercise, you should have a Proxmox cluster with at least 1 node that’s accessible via DNS on your local cluster!

Why We Wrote Zephyr

Since releasing Zephyr, we’ve been asked by numerous people why we wrote Zephyr instead of sticking to OSGi. Our goal was pretty simple: create an extensible system suitable for SaaS or on-prem.  We looked in our toolbox and knew that we could do this using OSGi, Java, and Spring, and so that’s how it started.

How We Started

First, we wrote our extensible distributed graph reduction machine: Gyre.  This allowed us to describe computations as graphs. It generated a maximally-parallel schedule, did its best to figure out whether to ship a) a computation to data or b) data to a computation or c) both to an underutilized node and executed the schedule.

Then we wrote Anvil, our general-purpose optimization engine that efficiently solved linear and non-linear optimization problems. These were described as Gyre graphs (including how the Gyre could better execute tasks based off of its internal metrics). We deployed Anvil and Gyre together as bundles into an OSGi runtime.  Obviously, Anvil couldn’t operate without Gyre, and so we referenced Gyre services in Anvil.  But Anvil and Gyre themselves were extensible.  We wrote additional solvers and dynamically installed them into Anvil, or wrote different concurrency/distribution/serialization strategies and deployed them into Gyre, and gradually added more and more references.

Then we wrote Troposphere, our deployment engine. Troposphere would execute its tasks on Gyre, and Anvil would optimize them. Troposphere would define types of tasks, and we exported them as requirements to be satisfied by capabilities. (For example, Troposphere would define a “discovery” task, and an AWS EC2 plugin would fulfill that capability.)

Handling OSGi with Spring

Being a small team, we pretty much only used one actual framework (Spring), so we deployed yet another bundle containing only the Spring classpath, to be depended on by any bundle that required it.  We initially used bnd to generate our package import/export statements in our manifest, and pulled in the bnd Gradle plugin as part of the build, but the reality was that if a plugin depended on Troposphere, then it pretty much always depended on Gyre, Anvil, and Spring.

If Anvil contains a service-reference to Gyre, and Troposphere contains one to Anvil, you get the correct start-order.  But if you stop Gyre while Troposphere is running?  Well, that’s a stale reference, and Troposphere needs to handle it, which means refactoring Troposphere and Gyre to use service factories, prototype service factories, or whatever else.

But we just wanted to write Spring and Java.  To really use Spring in an OSGi-friendly way, you have to use Blueprints, and now you’re back to writing XML in addition to all of the OSGi-y things you’re doing in your code. The point isn’t that OSGi’s way doesn’t work — it does. These are solid technologies written by smart people. The point is that introduces a lot of additional complexity, and you’re forced to really understand both Spring and OSGi to be productive when Spring is the only framework that’s actually providing value (in the form of features) to your users because the extensibility component (OSGi) is a management concern.

What Zephyr gets us that OSGi didn’t


We’re big fans of unit tests, and we write a lot of them.  Ideally, if you’re sure components A and B both work, then the combination of A and B should work.  The reality is that sometimes they don’t for a huge variety of reasons. For example, for us, using any sort of concurrency mechanism outside of Gyre could severely bork Gyre, which could and did bamboozle dozens of plugins. We’re small enough that we could just set a pattern and decree that hey, that is the pattern, and catch violations in reviews or PMD rules. But once again, we just wanted to write integration tests and we wanted to use Spring Test to do it.

With OSGi, you can create projects whose test classpath matches the deployment classpath (although statically), and we did.  We also wrote harnesses and simulations that would set up OSGi and deploy plugins from Maven, etc., and it all worked. But it was still complex, and it wasn’t just Spring Test. This was, and continues to be, a big source of pain for us.  The fact of the matter is that, once again, Spring was providing the developer benefit and OSGi was introducing complexity.

Quick Startup/Shutdown Times

We use a lot of Spring’s features and perform DB migrations in a variety of plugins — not an unusual use case.  A plugin might only take a few seconds to start, but amortized over dozens of plugins, startup time became pretty noticeable.  There are some ways to configure parallel bundle lifecycle, but they’re pretty esoteric, sometimes implementation-dependent, and always require additional metadata or code. With Zephyr, we get parallel deployments out-of-the-box and as the default, reducing startup times from 30+ seconds to 5 or so.

Remote Plugins

One of our requirements is the ability to run plugins whose processes and lifecycles reside outside of Zephyr’s JVM. OSGi (understandably) wasn’t designed to support this, but Zephyr was.

Getting it right with Zephyr

We spent about two years wrangling OSGi and Spring, by turns coping with these and other problems either in code or operations. It was generally successful, but there was always an understanding that we were paying a high price in terms of time and complexity. After the first dozen or so plugins, we’d really come to understand what we wanted from a plugin framework.

To boot, we are pretty good at graph processing, and it had been clear to us for a while that the plugin management issues we were continually encountering were graph problems. Classpath dependency issues could be easily understood through the transitive closure of a plugin, and most of our plugins had the same transitive closure. Even if they didn’t, that was the disjoint-subgraph problem and we could easily cope with that. Correct parallel start schedules were easily found and correctly executed by Coffman-Graham scheduling, and we could tweak all of these subgraphs through subgraph-induction under a property.  Transitive reduction allowed us to easily and transparently avoid problems caused by non-idempotent plugin management operations.

Once we’d implemented those, we discovered that a lot of the problems we struggled with just went away. Required services could never become stale, and optional services just came and went.  A lot of the OSGi-Spring integration code we’d written became dramatically simpler, and we could provide simple but powerful Spring Test extensions that felt very natural.

What’s Next

But we’re not stopping with Spring: Zephyr can support any platform and any JVM language, and we’re planning on creating support for Clojure, Kotlin, and Scala initially as installable runtimes. We’re investigating NodeJS support via Graal and should have some announcements about that in the new year. Spring is already supported, and we hope to add Quarkus and Dropwizard soon. And keep in mind that these integrations should require little or no knowledge of Zephyr at all.

We’re also in the process of open-sourcing a beautiful management UI, a powerful repository, and a host of other goodies — stay tuned!

Why Right Sizing Instances is Not Nonsense

I like Corey Quinn — his newsletter and blog make some good points, but his recent post, Right Sizing Your Instances Is Nonsense, is a little off base.  I encourage you to read it in its entirety.

“Right Sizing means upgrading to latest-gen”

Corey makes the argument that upgrading an m3.2xlarge to a m5.2xlarge for a savings of 28% is the correct course of action.  We have a user with > 30 m3.2xlarge instances whose CPU utilization is typically in the low digits, but which spikes to 60+% periodically.  Whatever, workloads rarely crash because of insufficient CPU — they do, however, frequently crash because of insufficient memory.  In this case, their memory utilization has never exceeded 50%.

Our optimizations, which account for this and other utilization requirements, indicate that the “best fit” for their workload is in fact an r5.large, which saves them ~75%.  In this case, for their region, the calculation is:

  1. m3.2xlarge * 0.532000/hour * 730 hours/month * 30 = $11,650.80/month
  2. r5.large * 0.126000/hour * 730 hours/month * 30 = $2759.40

The approximate monthly difference is $8891.40/month

Now, these assume on-demand instances, and reserved instances can save you a substantial amount (29% in this case at $0.380 per instance/hour), but you’re locked in for at least a year and you’re still overpaying by 320%.

“An ‘awful lot of workloads are legacy’ -> Legacy workloads can’t be migrated”

So, this one’s a little harder to tackle just because “an awful lot” doesn’t correspond to a proportion, but let’s assume it means “100%” just to show how wrong this is according to the points he adduces:

Memory Consumption in Spring and Guice

The heart of the cloud management platform is a distributed virtual machine that we call Gyre. Developers can extend Gyre through Sunshower’s plugin system, and one of the nicer features that Gyre plugins provide is a dependency-injection context populated with your task’s dependencies, as well as extension-point fulfillments exported by your plugin. For example, take (part) of our DigitalOcean plugin:

The details are beyond the scope of this post, but basically Gyre will accept an intermediate representation (IR) from one of its front-ends and rewrite it into a set of typed tuples whose values are derived from fields with certain annotations. The task above will be rewritten into (ComputeTask, Vertex, TaskResolver, TaskName, Credential). This tuple-format is convenient for expressing type-judgements, especially on the composite and recursive types that constitute Gyre programs.

Unit-testing Aurelia with Jest + JSDOM + TypeScript + Pug

All source can be found at: Aurelia-Aire

Testing UI components is one of the more difficult parts of QA. In Stratosphere (one of the key components of our cloud management platform), we’d been using Karma + Jasmine, but the browser component had been complicated by the fact that providing a DOM to tests in a fast, portable manner subject to memory and speed constraints can be pretty challenging. Sure, initially we did the PhantomJS thing, then Chrome Headless came out, but writing UI tests just never really felt natural.

Then, last week, we decided to open-source our component framework, Aire, built on top of UIKit+Aurelia, and that created an exigent need to fix some of the things we’d been limping along with, most importantly testing. The success of OSS projects depends on quite a few things, but I consider providing a simple way to get contributors up-and-running critical.

Simple set-up

Internally, Aurelia uses an abstraction layer (Aurelia PAL) instead of directly referencing the browser’s DOM. Aurelia will (in principle) run on any reasonable implementation of PAL. Aurelia provides a partial implementation OOTB, Aurelia/pal-nodejs, that will enable to (mostly) run your application inside of NodeJS.

Project Structure

Our project structure is pretty simple: we keep all our components and tests under a single directory, src:

├── build
│   └── paths.js
├── gulpfile.js
├── index.html
├── jest.config.js
├── jspm.config.js
├── package.json
├── package-lock.json
├── src
│   ├── main
│   │   ├── aire.ts
│   │   ├── application
│   │   ├── button
│   │   ├── card
│   │   ├── core
│   │   ├── core.ts
│   │   ├── dropdown
│   │   ├── events.ts
│   │   ├── fab
│   │   ├── form
│   │   ├── icon
│   │   ├── init
│   │   ├── init.ts
│   │   ├── loader
│   │   ├── nav
│   │   ├── navbar
│   │   ├── offcanvas
│   │   ├── page
│   │   ├── search
│   │   ├── table
│   │   ├── tabs
│   │   └── widget
│   └── test
│   │   ├── button
│   │   ├── core
│   │   ├── init
│   │   ├── render.ts
│   │   ├── setup.ts
│   │   └── tabs


At the top of the tree you’ll notice jest.config.js, the contents of which look like this:

Basically, we tell Jest to look under src for everything. ts-jest will automatically look for your Typescript compiler configuration, tsconfig.js in its current directory, so there’s no need to specify that.

Our tsconfig is pretty standard for Aurelia projects:


If you just copy and paste our tsconfig.json and jest.config.js files while following the outlined directory structure, everything will Just Work (don’t forget to npm i -D the appropriate Jest and Aurelia packages.)

At this point, you can use aurelia-test to write tests a la:




Now, you can run your tests with npx jest:

aire@1.0.0 test /home/josiah/dev/src/
npx jest

PASS src/test/button/button.spec.ts
PASS src/test/tabs/tab-panel.spec.ts
PASS src/test/init/init.spec.ts
PASS src/test/core/dom.spec.ts

Test Suites: 4 passed, 4 total
Tests: 12 passed, 12 total
Snapshots: 0 total
Time: 3.786s
Ran all test suites.

Enabling support for complex DOM operations

That wasn’t too bad, was it? Well, the problem we encountered was that we use the excellent UIKit framework, and they obviously depend pretty heavily on the DOM. Any reference in Aire to UIKit’s Javascript would fail with a ReferenceError: is not defined error. Moreover, if we changed the Jest environment from node to jsdom, we’d encounter a variety of errors along the lines of TypeError: Failed to execute 'appendChild' on 'Node': parameter 1 is not of type 'Node' which I suspect were caused by pal-nodejs creating DOM elements via its own jsdom dependency while Jest was performing DOM operations using its jsdom dependency. In any case, the solution turned out to be to define a single, global jsdom by importing jsdom-global. Once we discovered this, we encountered other issues with browser-environment types and operations not being defined, but this setup.js resolved them:

At this point we could successfully test Aurelia + UIKit in NodeJS.

The final piece

All of our component views are developed in Pug, and I didn’t like that we’d be developing in Pug but testing using HTML. The solution to this turned out to be pretty simple: adding Pug as a development dependency and creating a pretty simple helper function:

With that final piece in place, our test looks like:


The benefits of writing tests in this way became apparent the moment we cloned the project into a new environment and just ran them via npm run test or our IDE’s. They’re fast, don’t require any environmental dependencies (e.g. browsers), and allow you to run and debug them seamlessly from the comfort of your IDE. But, perhaps most importantly, these are fun to write!