Building a Home Cloud with Proxmox: DNS + Terraform

Overview

In our last post, we configured a Ceph storage cluster, which we’ll be using as the storage for our virtual machines that we’ll be using to host Kubernetes.

Before we get to that, however, we need to configure our DNS environment. Recall that, in part 2, we configured a Windows server to act as our DNS and DHCP server. For this, you don’t need a Windows Server installation, you just need an RFC 2136-compliant DNS server like PowerDNS. Keep in mind that RFC 2136 has some security vulnerabilities, so don’t use dynamic DNS in a high-security environment or an environment wherein untrusted devices may be on the network.

To provision our DNS names, we’ll be interacting with our DNS server via the Terraform DNS provider. Terraform’s easy to set up, so we won’t be covering that.

Note that this solution requires a fair amount of work in Bash. Other solutions like Chef/Puppet/Ansible/Salt may strictly be better-suited for this sort of work, but since our environment is quite homogeneous (99% Debian), we’ll just script this using basic tools.

A note on Terraform Typing

Terraform’s variable/parameter management system, specifically its inability to share variables between modules can result in some duplication between modules. This isn’t the cleanest, but I don’t feel like it’s a big enough problem to warrant bringing in other tools like Terragrunt. Terraform’s type-system supports a flavor of structural typing wherein you can declare a variable with a subset of another variable’s fields and supply the first variable as the second variable. For instance:

variable "A" {
type = object({
name = string
ip = string
})
}
variable "B" {
type = object({
name = string
})
}
view raw variables.tf hosted with ❤ by GitHub

Usage:

   // module B declares B, and can use a value of type A as the value for b:B
    b = var.a

We’ll be using this feature to share variable values across modules.

Setup

Since we’re reserving the upper range of our /24 subnet for MetalLB dynamically-provisioned IP addresses, we’ll configure the worker nodes with static IP addresses assigned from the lower range of our subnet. Since we have a small number of physical machines, we’ll opt for fewer, larger worker-nodes instead of more, smaller worker-nodes. The subnet-ranges we’ll select for these roles are as follows:

  1. 2-10: infrastructure (domain-controller, physical node IPs, etcd VMs, Kubernetes leader nodes)
  2. 10-30: static addresses (worker nodes)
  3. 30-200: DHCP leases for dynamic devices on the network + MetalLB IPs
  4. 200-255: available

To set that up in Windows Server:

  1. Navigate to the Server Manager
  2. From the top-right Tools menu, select DHCP
  3. Right-click on the IPv4 node under your domain-controller host-name and select New Scope
  4. Provide a Start address of <subnet prefix>.30 and an end-address of 200
  5. Continue through the wizard providing your gateway addresses and your exclusions. For the exclusions, select <subnet prefix>-1 through 30.
  6. Finish the wizard and click Create to create your scope.
  7. Right-click on the created scope node in the tree. Select the DNS tab, then enable DNS dynamic updates. Select the Dynamically update DNS records only if requested by the DHCP clients.
  8. Click Ok to save your changes.

Configure your DD-WRT Router

Navigate back to your gateway router. Most capable routers will have an option to either act as a DHCP server (the default), or to act as a DHCP forwarder. For DD-WRT, select the `DHCP Forwarder option and provide the IP address of your DHCP server as the target.

Configure your DNS Server

Go back to your domain controller.

  1. From the Server Manager, select DNS
  2. Locate the Forward Lookup Zone node in the DNS Manager tree that you created (ours was sunshower.io. Right-click on the Forward Lookup Zone node and click Properties
  3. In the General Tab, select the Nonsecure and Secure option from the Dynamic updates configuration. Note: This is not suitable for an open-network or one admitting untrusted/unknown devices
  4. Click Ok to apply.

Congrats! We’re now ready to provision our DNS in Terraform.

Terraform Configuration

Note: We’ve provided our Terraform configurations here, so feel free to point them at your infrastructure if you’ve been following along instead of creating your own.

In the directory of your Terraform project ($TFDIR), create a directory called dns.
1. In your dns directory, create 3 files: main.tf, variables.tf, and outputs.tf

Main.tf

Your main.tf file should contain:

terraform {
required_providers {
dns = {
source = "hashicorp/dns"
}
}
}
/**
provision DNS entries for each host in the `hosts` list
*/
resource "dns_a_record_set" "virtual_machine_dns" {
for_each = {for vm in var.hosts: vm.name => vm}
zone = var.dns_server.zone
name = each.value.name
addresses = [
each.value.ip
]
}
/**
provision the DNS A-record for the API server
*/
resource "dns_a_record_set" "api_server_dns" {
addresses = [
var.api_ip
]
zone = var.dns_server.zone
name = var.api_dns
}
view raw main.tf hosted with ❤ by GitHub

Now, in your Terraform project $TFDIR (the parent of the dns directory you’re currently in, create main.tf, outputs.tf, and variables.tf. Symlink the dns/variables.tf to this directory as dns-variables.tf via ln -s $(pwd)/vm-variables.tf $(pwd)/dns/variables.tf to make your DNS variables available to your root configuration.

Parent module

In your $TFDIR/main.tf, add the following block:

terraform {
required_version = ">=0.14"
required_providers {
dns = {
source = "hashicorp/dns"
version = "3.0.1"
}
}
}
view raw main.tf hosted with ❤ by GitHub

Configure the DNS provider to point to your RFC-2136-compliant DNS server:

provider "dns" {
update {
server = var.dns_server.server
}
}
view raw dns-config.tf hosted with ❤ by GitHub

Call the dns submodule:

/**
Create DNS entries for virtual machines
*/
module "dns_configuration" {
for_each = var.cluster_nodes
source = "./dns"
dns_server = var.dns_server
hosts = each.value
api_dns = var.api_server
api_domain = var.domain
api_ip = var.load_balancer
}
view raw main.tf hosted with ❤ by GitHub

At this point, we should be ready to actually provide our Terraform variables.

  1. Create a file in your $TFDIR directory called configuration.auto.tfvars–Terraform will automatically pick up any in the root directory according to the naming convention *.auto.tfvars. Be sure to never check these into source-control. I add the .gitignore rule **/*.tfvars

In your $TFDIR/configuration.auto.tfvars file, add the following configurations (substitute with your values):

dns_server = {
// replace with your zone-name (configured above)
zone = "sunshower.cloud." // note the trailing period here--it's mandatory
// replace with your DNS server's IP
server = "192.168.1.2"
}
// this generates the DNS configuration for `kubernetes.sunshower.cloud`
api_dns = "kubernetes"
api_domain = "sunshower.cloud"
api_ip = "192.168.1.10" // provide the API you

Finally, we’ll add our Kubernetes cluster DNS configurations:

cluster_nodes = {
etcd_nodes = [
{
name = "etcd-1"
ip = "192.168.1.5"
},
{
name = "etcd-2"
ip = "192.168.1.6"
},
{
name = "etcd-3"
ip = "192.168.1.7"
}
],
k8s_leaders = [
{
name = "k8s-leader-1"
ip = "192.168.1.8"
},
{
name = "k8s-leader-2"
ip = "192.168.1.9"
}
],
k8s_workers = [
{
name = "k8s-worker-1"
ip = "192.168.1.11"
},
{
name = "k8s-worker-2"
ip = "192.168.1.12"
},
{
name = "k8s-worker-3"
ip = "192.168.1.13"
}
]
}

At this point, running terraform apply --auto-approve should quickly generate the DNS entries. You should navigate to your DNS server and refresh the forward lookup zone to confirm that your entries exist.

Conclusion

In this part, we configured DHCP, DNS, and provided a Terrform module that provisions DNS entries for each of the VMs that we’ll provision in subsequent posts. Next time, we’ll create a Terraform module that will provision virtual machines of the correct roles for our Kubernetes cluster. Our final post of the series will show how to actually deploy the Kubernetes cluster–stay tuned!

Building a Home Cloud with Proxmox Part 3: Configuring Ceph and Cloud-Init

Overview

Last time, we configured a Windows Server domain controller to handle DNS, DHCP, and ActiveDirectory (our LDAP implementation). In this post, we’ll be configuring the Ceph distributed filesystem which is supported out-of-the-box by Proxmox.

Ceph Storage Types

Ceph supports several storage types, but the types we’re interested in for our purposes are

  1. Filesystem
  2. Block

A Ceph filesystem behaves like a standard filesystem–it can contain files and directories. The advantage to a Ceph filesystem for our purposes is that we can access any files stored within it from any location in the cluster, whereas with local-storage (e.g. local-lvm or whatever), we can only access them on the node they reside upon.

Similarly, a Ceph block device is suitable for container and virtual machine hard-drives, and these drives can be accessed from anywhere in the cluster. This node-agnosticism is important for several reasons:

  1. We’re able to create base images for all our virtual machines and clone them to any node, from any node
  2. We’re able to use Ceph as a persistent volume provider in Kubernetes

In this scheme, your Ceph IO performance is likely to be bottlenecked by your intra-node network performance. Our 10Gbps network connections between nodes seems to perform reasonably well, but is noticeably slower than local storage (we may characterize this at some point, but this series isn’t it).

Drive Configuration

In our setup, we have 4, 4 TB drives on each node. Ceph seems to prefer identical hardware configurations on each node it’s deployed on, so if your nodes differ here, you may want to restrict your Ceph deployment to identical nodes.

Create your Ceph partitions on each node

Careful Ensure that the drives you’re installing Ceph onto aren’t being used by anything–this will erase everything on them. You can colocate Ceph partitions with other partitions on a drive, but that’s not the configuration we’re using here (and how well it performs depends on quite a few factors)

We’re going to use /dev/sdb for our Ceph filesystem on each node. To prepare each drive, open a console into its host and run the following:

  1. fdisk /dev/sdb
  2. d to delete all the partitions. If you have more than one, repeat this process until there are no partitions left.
  3. n to create a new partition
    1 g to create a new GPT partition table
  4. w to write the changes

Your disks are now ready for Ceph!

Install Ceph

On each node, navigate to the left-hand configuration panel, then click on the Ceph node. Initially, you’ll see a message indicating that Ceph is not installed. Select the advanced option and click Install to continue through the wizard.

When you’re presented with the options, ensure that the osd_pool_default_size and osd_pool_default_min_size configurations are set to the number of nodes that you intend to install Ceph on. While the min_size can be less than the pool_size, I have not had good luck with this configuration in my tests.

Once you’ve installed Ceph on each node, navigate to the Monitor node under the Ceph configuration node and create at least 1 monitor and at least 1 manager, depending on your resiliency requirements.

Create an object-storage daemon (OSD)

Our next step is to allocate Ceph the drives we previously provisioned (/dev/sdb). Navigate to the OSD node under the Ceph node, and click create OSD. In our configuration, we use the same disk for both the storage area and the Ceph database (DB) disk. Repeat this process for each node you want to participate in this storage cluster.

Create the Ceph pool

We’re almost there! Navigate to the Pools node under the Ceph node and click create. You’ll be presented with some options, but the options important to us are:

  1. size: set to the number of OSDs you created earlier
  2. min-size: set to the number of OSDs you created earlier

Ceph recommends about 100 placement groups per OSD, but the number must be a power of 2, and the minimum is 8. More placement groups means better reliability in the presence of failures, but it also means replicating more data which may slow things down. Since we’re not managing dozens or hundreds of drives, I opted for fewer placement groups (16).

Click Create–you should now have a Ceph storage pool!

Create your Ceph Block Storage (RBD)

You should now be able to navigate up to the cluster level and click on the storage configuration node.

  1. Click Add and select RBD.
  2. Give it a memorable ID that’s also volume-friendly (lower case, no spaces, only alphanumeric + dashes). We chose ceph-block-storage
  3. Select the cephfs_data pool (or whatever you called it in the previous step)
  4. Select the monitor node you want to use
  5. Ensure that the Nodes value is All (no restrictions) unless you want to configure that.
  6. Click Add!

Congrats! You should now have a Ceph block storage pool!

Create your Ceph directory storage

In the same view (Datacenter > Storage):

  1. Click Add and select CephFS
  2. Give it a memorable ID (same rules as in the previous step), we called ours ceph-fs
  3. Ensure that the content is selected to all the available options (VZDump backup file, ISO image, Container Template, Snippets)
  4. Ensure the Use Proxmox VE managed hyper-converged cephFS option is selected

Click Add–you now have a location you can store VM templates/ISOs in that is accessible to every node participating in the CephFS pool! This is important because it radically simplifies configuration via Terraform, which we’ll be writing about in subsequent posts.

Create a Cloud-Init ready VM Template

We use Debian for virtually everything we do, but these steps should work for Ubuntu as well.

  1. Upload a Debian ISO (we use small installation image) to your CephFS storage
  2. Perform a minimal installation. The things you’ll want to pay attention to here are:
    • Ensure your domain-name is set to whatever domain-name you’ve configured on your domain-controller
    • Only install the following packages: SSH Server and Standard System Utilities
  3. Finish the Debian installation wizard and power on your VM. Make a note of the ID (probably 100 or something) I refer to as VM_ID in subsequent steps
  4. In the Proxmox console for your VM, perform the following steps:
    • apt-get update
    • apt-get install cloud-init net-tools sudo
  5. Once APT is finished installing those packages, edit your sshd_config file at /etc/ssh/sshd_config using your favorite editor (e.g. vi /etc/ssh/sshd_config) and ensure that PermitRootLogin is set to yes from its default of prohibit-password. These are all air-gapped in our environment, so I’m not too worried about the security ramifications of this. If you are, be sure to adjust subsequent configurations to use certificate-based auth.
  6. Save the file and shut down the virtual machine

Now, let’s enable CloudInit!

On any of your nodes joined to the CephFS filesystem:

  1. Open a console
  2. Configure a CloudInit drive:
    • Using the id for the Ceph RBD/block store (referred to as <BLOCK_STORE_ID> we configured above (ours is ceph-block-storage), and the VM_ID from the previous section, create a drive by entering qm set <VM_ID> --ide2 <BLOCK_STORE_ID>:cloudinit. For instance, in our configuration, with VM_ID = 100, this is qm set 100 --ide2 ceph-block-storage:cloudinit. This should complete without errors.
  3. Verify that your VM has a cloud-init drive by navigating to the VM, then selecting the Cloud-Init node. You should see some values there instead of a message indicating Cloud-Init isn’t installed.

You should also verify that this machine is installed on the correct block-storage by attempting to clone it to another node. If everything’s configured properly, cloning a VM to a different node should work seamlessly. At this point, you can convert your VM to a template for subsequent use.

Conclusion

In this post we installed and configured Ceph block and filestores, and also created a Cloud-Init capable VM template. At this point, we’re ready to begin configuring our HA Kubernetes cluster using Terraform from Hashicorp. Our Terraform files will be stored in our public devops Github repository

Building a Home Cloud with Proxmox Part 2: Domain Configuration

Overview

Last time I went over how to configure a router equipped with DD-WRT for managing a home cloud, as well as installing the awesome Proxmox virtualization environment. This time, we’ll go over how to configure a Windows domain controller to manage ActiveDirectory profiles, as well as DNS.

General configuration

The end goal is to provision a cluster that’s running a Windows domain controller (for DNS and ActiveDirectory), as well as a Kubernetes cluster. Once we have that we’ll deploy most of our services into the Kubernetes cluster, and we’ll be able to grant uniform access to every object via the domain controller.

Note that multitenancy isn’t a goal for this home cloud, although the domain controller is where we would add that. We could also partition our subnet into more subnets, providing functionality analogous to public cloud providers’ notions of Virtual Private Clouds.

Step 1: Create a Windows Server VM

  1. Ensure you have a suitable storage location:
    1. Open a shell to one of your nodes
    2. Type lsblk–it should show you a tree of your available hard drives and partitions
    3. If you don’t see a partition under your drives, type fdisk. In fdisk <drive>, where <drive> is the drive you want (probably not /dev/sda–that’s likely where Proxmox is installed), create a GPT partition by typing g. Write the changes by entering w
    4. Now you should be able to provision a VM on that drive/partition in the subsequent steps.
  2. Download the Windows Server ISO
  3. Upload the Windows Server ISO to Proxmox:
    1. The Windows server ISO is too large to upload via the GUI, so pick a node (e.g. athena.sunshower.io), and scp it up: scp <iso file> root@<node>:/var/lib/vz/template/iso, for example:scp windows_server_2019.iso root@athena.sunshower.io/var/lib/vz/template/iso/
    2. Create a new VM
      • Select Windows as the type, the default version should be correct, but verify it’s something like 10/2019/
      • Give it at least 2 cores, 1 socket is fine. The disk should have at least 50GB available.
      • I provided mine with 16 GB RAM, but we have pretty big machines. 4 works fine, and you can change this later.
    3. One of the prompts will be to assign your server to a domain. You should enter the domain that you want to control (ours being sunshower.io).

You should be able to access your Windows Server VM now, but it’s not quite ready.

Configure your Domain Controller’s DNS

The first thing we’ll want to do is ensure that our domain controller is at a deterministic location. For this, we’ll want to
1. Assign it a static DHCP lease in our router
1. Assign it a static IP locally
1. Assign it a DNS entry in our router

Assign the DHCP lease

  1. In the Windows VM, open an elevated console and run ipconfig /all. This will show all of your IP configurations, as well as the MAC addresses they’re associated with. Locate the entry for your subnet (ours is 192.168.1.0/24), and locate the associated MAC address (format AA:AA:AA:AA:AA:AA, where AA is a 2-letter alphanumeric string).
  2. Once you have your MAC address, log into your router, navigate to the Services/Services subtab, find the Static Leases section, and create an entry whose MAC address is the one you just obtained, and whose IP address is towards the low-end of your subnet (we selected 192.168.1.5). Choose a memorable hostname (we went with control.sunshower.io) and enter that, too.

Save and apply your changes.

Assign the static IP locally

  1. In your Windows VM, navigate to Control Panel > Network and Internet > Network Connections. You’ll probably only have one.
  2. Right-click on it and select Properties. Navigate to Internet Protocol Version 4 (TCP/IPv4)
  3. Select Use the following IP address
    • For IP Address, enter the IP you set in the previous section (Assign the DHCP lease)
    • For Subnet mask, (assuming a /24 network), enter 255.255.255.0. You can look these up here
    • For Default gateway, select your router IP (ours is 192.168.1.1)

For the DNS Server Entries, you can set anything you like (that is a valid DNS server). We like Google’s DNS servers (8.8.8.8, 8.8.4.4). Notice that we’re not actually setting our router to be a DNS server, even though it is configured to be one. That’s because we generally want the domain controller to be our DNS server.

Click Ok and restart just to be sure.

Configure your Domain Controller

Your Windows Server VM will need to function as a domain controller (and DNS server) for this to work. After you’ve created the Windows Server VM, log into it as an administrator. You should be presented with the Server Manager Dialog.

  1. From the top-right, click the manage button, select Add Roles and Features. This will bring you to the Roles & Features wizard.
  2. Select Role-based or feature-based installation. Click Next
  3. Select your server. Click Next
  4. Select the following roles
    • Active Directory Domain Services
    • Active Directory Lightweight Directory Services
    • Active Directory Rights Management Services -> Active Directory Rights Management Server
    • DHCP Server
    • DNS Server

Click Install–grab some coffee while Windows does its stuff. It’ll restart at least once.

Configure DNS

Almost done! We’re going to add all the entries that we had previously added in the router to the Domain Controller so that we only need to reference one DNS server: our domain controller. Finally, we’ll join a home computer to the domain.

Add the DNS records

  1. From the Server Manager window, select Tools and click DNS from the dropdown.
  2. Expand the Forward Lookup Zones node in the tree. You should see your domain under there, possibly along with other entries such as _msdcs.<your domain.
  3. Select <your domain>. Ours is sunshower.io
  4. Right-click <your domain>, add new host (A or AAAA)
  5. For each of your Proxmox nodes, add the name (e.g. athena)–the fully-qualified domain name should automatically populate.
    • Provide the IP address of the associated node. If you followed along closely to the previous entry, they’ll be (192.168.1.2 up to 1 + number of nodes)
  6. Create an entry for your router such as gateway.my.domain (e.g. gateway.sunshower.io) and point it at your gateway IP (e.g. 192.168.1.1)
  7. Create an entry for your controller such as control.my.domain (e.g. control.sunshower.io) and point it at your controller’s static IP.

Save and restart your controller.

Create a ActiveDirectory User

  1. Log back into your domain controller and, from the server manager select the Tools menu, then Active Directory Users and Computers.
  2. From the left-hand navigator tree, expand your domain node and right-click on the Users Sub-node.
  3. Select New, and then User. Fill in your information, and make a note of the username and password, we’ll use it in the next step.

Join a computer to your domain!

On your local computer, navigate to:
1. Control Panel > System and Security > System
1. Under the section Computer name, domain, and workgroup settings, select Change settings. The System Properties dialog should appear.
1. Select the Change button to the right of To rename this computer or change its domain or workgroup, click Change
1. Set your computer name (kitchen, workstation, etc.)
1. Set member of to domain with your domain value (e.g. sunshower.io).

Click OK–this can take a bit.

Once that’s done, restart your computer. You should now be able to log in as <DOMAIN>/ (configured in the ActiveDirectory section above)!

Your profile should automatically use your domain controller as its DNS server, so you should be able to ping all of your entries.

Conclusion

Windows Server makes it really easy to install and manage domains, almost enough to justify its steep price-tag. A domain server allows you to control users, groups, permissions, accounts, applications, DNS, etc. in a centralized fashion. Later on, when we configure Kubernetes, we’ll take advantage of Windows Server’s powerful DNS capabilities to automatically provision DNS entries associated with IPs allocated from our cloud subnet and use them to front services deployed into our own high-availability Kubernetes cluster.

If you don’t want to/can’t use Windows Server, you can replicate this functionality using Apache Directory, and PowerDNS. I may do a post on how to setup and configure these, but my primary goal is to move on to the Kubernetes cluster configuration, and managing DNS and LDAP are two relatively complex topics that are greatly simplified by Windows Server.

Building a Home Cloud with Proxmox: Part 1

We’re back!

Whew! As you’re probably already aware, 2020 was an incredibly weird year–but we’re getting back on the wagon and going leaner in 2021! Last year, we’d installed solar power to take advantage of Colorado’s 300+ days of sunshine, and we had some relatively beefy workstations left over from our defense work. At the same time, we noticed that our R&D AWS spend is a little high for our budget right now, so we decided to build a local cloud on top of the wonderful Proxmox virtualization management software. Let’s get started!

Structure

To begin with, we reviewed which services from AWS we needed, and came up with the following list:

  1. RDS
  2. Jenkins CI/CD
  3. Kubernetes (We hosted our own with KOPS on AWS–highly recommend it)
  4. Route53
  5. AWS Directory Service
  6. Nexus
  7. Minio

(I haven’t really decided if I want to use OpenShift or not for this cluster yet).

We’re also probably going to end up using MetalLB to allocate load-balancers for our bare-metal(ish) cluster.

Let’s get to it!

Step 0: Hardware configuration

Our setup is 3 HP Z840s with the following configuration:
1. 2TB RAM
1. 24 logical CPUs on 2 sockets
1. 4 4TB Samsung PRO 860 SSDs

For ease-of-use, we also recommend the BYTECC 4 HDMI port KVM switch (for these machines, anyway, YMMV)

Step 1: Configure your Network

  1. Check your network configuration. We have each box hooked up directly to a Linksys AC32000 running DD-WRT.
    1. (Note that these instructions are for DD-WRT–if you don’t use that, you’ll have to find equivalent configurations for your router).
    2. IMPORTANT: Ensure that your network is at least a /24 (256 total IPs)–you’ll end up using more than you expect.
    3. Useful tip: Install your nodes towards the low-end of your subnet. Ours are at 192.168.1.[2, 3, 4] with the gateway 192.168.1.1.
  2. Next, configure your DD-WRT installation to supply DNS over your LAN by:
    1. Navigating to 192.168.1.1 and logging in
    2. Navigate to services from the top level, then ensure the services sub-tab is selected
    3. Select Use NVRAM for client lease DB (this will allow your configuration to survive power-outages and restarts)
    4. Set Used Domain to LAN & WAN
    5. Set your LAN Domain to whatever you want it to be. This is purely local, so it can be anything. We own sunshower.io so we just used that. home.local or whatever should work just fine as well.
    6. Ensure DNSMasq is enabled
    7. Add bash
      local=/home/
      expand-hosts
      domain-needed
      bogus-priv
      addn-hosts=/jffs/etc/hosts.home
      strict-order

      to your Additional DNSMasq Options Text box
    8. Save & Apply your configuration

Step 2: Install Proxmox

Proxmox is an open-source alternative to (approximately?) vSphere from VMWare, and we’ve used it for a variety of internal workloads. It’s quite robust and stable, so I encourage you to check it out!

The first thing is to burn a USB drive with the Proxmox ISO. UEFI hasn’t worked well for me on these machines (they’re a little old), so the Rufus configuration I used was:

Partition scheme: MBR
File System: FAT32
Cluster Size: 4096 bytes (default)

I went ahead and checked the Add fixes for old BIOSes option since these are pretty old bioses. Hitting start will ask you if you want to burn the image in ISO image mode or DD image mode. With these 840s, ISO mode didn’t work for me, but DD did. Hit START and grab a coffee!

Once your USB drive is gtg, install Proxmox VE on each of the machines:

  1. Plug your USB drive into the first node in your cluster
  2. Restart/power on your node
  3. Select your USB drive from the boot options. If you followed our instructions for Rufus, it’ll probably be under Legacy instead of UEFI or whatever.
  4. IMPORTANT: Click through the wizard until it asks you which drive/partition you want to install Proxmox into. Select Options and reduce the installation size to about 30GB. I don’t know what the minimum is, but 30GB works great for this setup. This even gives you some space to upload ISOs to the default storage location. (Note that if you don’t select this the installation will consume the entire drive).
  5. Continue on the installation until you see your network configuration.
    1. Select your network (ours is 192.168.1.1/24–yours may not be, depending on how you configured your router (above))
    2. Input your node IP (we went with 192.168.1.2 for the first, .3 for the second, etc.)
    3. Add a cool hostname. If you configured your router as we did above, you should be able to input <name>.. For instance, for our 3 nodes we went with athena.sunshower.io, demeter.sunshower.io and calypso.sunshower.io.
  6. Make a note of the username and password you chose for the root. Slap that into Lastpass–you’ll thank yourself later.

Repeat this process for each node you have. Once that’s complete, navigate to any of the nodes at https://<ip&gt;:8006 where <ip> is the IP that you configured in the Proxmox network installation step. Your browser will yell at you that this isn’t a trusted site since it uses a self-signed certificate by default, but that’s OK. Accept the certificate in your browser, then login with the username and password you provided. In the left-hand pane of your Proxmox server, select the top-level Datacenter node, then click Create Cluster. This will take a second, at which point you’ll be able to close the dialog and select Join Information. Copy the contents of the Join Information text-area.

Once you have your Join Information, navigate to each of the other nodes in your cluster. Log in, then select the top level Datacenter node once again. This time, click on the Join Cluster button. Paste the Join Information into the text area, then enter the root password of the first node in the root password text field. In a second or two, you should see 2 cluster nodes under the Datacenter configuration. Repeat this process with all of the nodes you set Proxmox up on!

Configure DNS

You may need to configure your DNS within your router at this point. Click on the Shell button for each node, and run:
1. apt-get update
1. apt-get install net-tools

Then, run ifconfig. You should see something like:

enp1s0: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1500
        ether d8:9d:67:f4:5e:a0  txqueuelen 1000  (Ethernet)
        RX packets 838880  bytes 173824493 (165.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 907011  bytes 170727177 (162.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17  memory 0xef500000-ef520000  

lo: flags=73&lt;UP,LOOPBACK,RUNNING&gt;  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10&lt;host&gt;
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 669  bytes 161151 (157.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 669  bytes 161151 (157.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vmbr0: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1500
        inet 192.168.1.2  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::da9d:67ff:fef4:5ea0  prefixlen 64  scopeid 0x20&lt;link&gt;
        ether d8:9d:67:f4:5e:a1  txqueuelen 1000  (Ethernet)
        RX packets 834450  bytes 158495280 (151.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 902499  bytes 166814715 (159.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

This is on athena.sunshower.io. Your list might look different. Note that under the vmbr0 entry there’s a set of letters/digits to the right of ether (in my case, d8:9d:67:f4:5e:a1). Make a note of these for each node, then go back to your gateway, log in, and navigate to Services once again. The default subtab Services should also be selected–if it’s not, select that.

You’ll see a field called Static Leases–click add. You’ll see a set of 4 fields:
1. MAC Address
1. Hostname
1. IP Address
1. Client Lease Time

For each of the host/MAC addresses you found previously (e.g. athena.sunshower.io -> d8:9d:67:f4:5e:a1), fill the MAC address field with the MAC address, the hostname with the corresponding hostname, and the IP address of the node. Save & Apply Settings. You should be able to visit any of the nodes at their corresponding DNS name. For instance, to access the Proxmox cluster from athena, I can visit https://athena.sunshower.io:8006

Conclusion

At the end of this exercise, you should have a Proxmox cluster with at least 1 node that’s accessible via DNS on your local cluster!

Why We Wrote Zephyr

Since releasing Zephyr, we’ve been asked by numerous people why we wrote Zephyr instead of sticking to OSGi. Our goal was pretty simple: create an extensible system suitable for SaaS or on-prem.  We looked in our toolbox and knew that we could do this using OSGi, Java, and Spring, and so that’s how it started.

How We Started

First, we wrote our extensible distributed graph reduction machine: Gyre.  This allowed us to describe computations as graphs. It generated a maximally-parallel schedule, did its best to figure out whether to ship a) a computation to data or b) data to a computation or c) both to an underutilized node and executed the schedule.

Then we wrote Anvil, our general-purpose optimization engine that efficiently solved linear and non-linear optimization problems. These were described as Gyre graphs (including how the Gyre could better execute tasks based off of its internal metrics). We deployed Anvil and Gyre together as bundles into an OSGi runtime.  Obviously, Anvil couldn’t operate without Gyre, and so we referenced Gyre services in Anvil.  But Anvil and Gyre themselves were extensible.  We wrote additional solvers and dynamically installed them into Anvil, or wrote different concurrency/distribution/serialization strategies and deployed them into Gyre, and gradually added more and more references.

Then we wrote Troposphere, our deployment engine. Troposphere would execute its tasks on Gyre, and Anvil would optimize them. Troposphere would define types of tasks, and we exported them as requirements to be satisfied by capabilities. (For example, Troposphere would define a “discovery” task, and an AWS EC2 plugin would fulfill that capability.)

Handling OSGi with Spring

Being a small team, we pretty much only used one actual framework (Spring), so we deployed yet another bundle containing only the Spring classpath, to be depended on by any bundle that required it.  We initially used bnd to generate our package import/export statements in our manifest, and pulled in the bnd Gradle plugin as part of the build, but the reality was that if a plugin depended on Troposphere, then it pretty much always depended on Gyre, Anvil, and Spring.

If Anvil contains a service-reference to Gyre, and Troposphere contains one to Anvil, you get the correct start-order.  But if you stop Gyre while Troposphere is running?  Well, that’s a stale reference, and Troposphere needs to handle it, which means refactoring Troposphere and Gyre to use service factories, prototype service factories, or whatever else.

But we just wanted to write Spring and Java.  To really use Spring in an OSGi-friendly way, you have to use Blueprints, and now you’re back to writing XML in addition to all of the OSGi-y things you’re doing in your code. The point isn’t that OSGi’s way doesn’t work — it does. These are solid technologies written by smart people. The point is that introduces a lot of additional complexity, and you’re forced to really understand both Spring and OSGi to be productive when Spring is the only framework that’s actually providing value (in the form of features) to your users because the extensibility component (OSGi) is a management concern.

What Zephyr gets us that OSGi didn’t

Testability

We’re big fans of unit tests, and we write a lot of them.  Ideally, if you’re sure components A and B both work, then the combination of A and B should work.  The reality is that sometimes they don’t for a huge variety of reasons. For example, for us, using any sort of concurrency mechanism outside of Gyre could severely bork Gyre, which could and did bamboozle dozens of plugins. We’re small enough that we could just set a pattern and decree that hey, that is the pattern, and catch violations in reviews or PMD rules. But once again, we just wanted to write integration tests and we wanted to use Spring Test to do it.

With OSGi, you can create projects whose test classpath matches the deployment classpath (although statically), and we did.  We also wrote harnesses and simulations that would set up OSGi and deploy plugins from Maven, etc., and it all worked. But it was still complex, and it wasn’t just Spring Test. This was, and continues to be, a big source of pain for us.  The fact of the matter is that, once again, Spring was providing the developer benefit and OSGi was introducing complexity.

Quick Startup/Shutdown Times

We use a lot of Spring’s features and perform DB migrations in a variety of plugins — not an unusual use case.  A plugin might only take a few seconds to start, but amortized over dozens of plugins, startup time became pretty noticeable.  There are some ways to configure parallel bundle lifecycle, but they’re pretty esoteric, sometimes implementation-dependent, and always require additional metadata or code. With Zephyr, we get parallel deployments out-of-the-box and as the default, reducing startup times from 30+ seconds to 5 or so.

Remote Plugins

One of our requirements is the ability to run plugins whose processes and lifecycles reside outside of Zephyr’s JVM. OSGi (understandably) wasn’t designed to support this, but Zephyr was.

Getting it right with Zephyr

We spent about two years wrangling OSGi and Spring, by turns coping with these and other problems either in code or operations. It was generally successful, but there was always an understanding that we were paying a high price in terms of time and complexity. After the first dozen or so plugins, we’d really come to understand what we wanted from a plugin framework.

To boot, we are pretty good at graph processing, and it had been clear to us for a while that the plugin management issues we were continually encountering were graph problems. Classpath dependency issues could be easily understood through the transitive closure of a plugin, and most of our plugins had the same transitive closure. Even if they didn’t, that was the disjoint-subgraph problem and we could easily cope with that. Correct parallel start schedules were easily found and correctly executed by Coffman-Graham scheduling, and we could tweak all of these subgraphs through subgraph-induction under a property.  Transitive reduction allowed us to easily and transparently avoid problems caused by non-idempotent plugin management operations.

Once we’d implemented those, we discovered that a lot of the problems we struggled with just went away. Required services could never become stale, and optional services just came and went.  A lot of the OSGi-Spring integration code we’d written became dramatically simpler, and we could provide simple but powerful Spring Test extensions that felt very natural.

What’s Next

But we’re not stopping with Spring: Zephyr can support any platform and any JVM language, and we’re planning on creating support for Clojure, Kotlin, and Scala initially as installable runtimes. We’re investigating NodeJS support via Graal and should have some announcements about that in the new year. Spring is already supported, and we hope to add Quarkus and Dropwizard soon. And keep in mind that these integrations should require little or no knowledge of Zephyr at all.

We’re also in the process of open-sourcing a beautiful management UI, a powerful repository, and a host of other goodies — stay tuned!

Why Right Sizing Instances is Not Nonsense

I like Corey Quinn — his newsletter and blog make some good points, but his recent post, Right Sizing Your Instances Is Nonsense, is a little off base.  I encourage you to read it in its entirety.

“Right Sizing means upgrading to latest-gen”

Corey makes the argument that upgrading an m3.2xlarge to a m5.2xlarge for a savings of 28% is the correct course of action.  We have a user with > 30 m3.2xlarge instances whose CPU utilization is typically in the low digits, but which spikes to 60+% periodically.  Whatever, workloads rarely crash because of insufficient CPU — they do, however, frequently crash because of insufficient memory.  In this case, their memory utilization has never exceeded 50%.

Our optimizations, which account for this and other utilization requirements, indicate that the “best fit” for their workload is in fact an r5.large, which saves them ~75%.  In this case, for their region, the calculation is:

  1. m3.2xlarge * 0.532000/hour * 730 hours/month * 30 = $11,650.80/month
  2. r5.large * 0.126000/hour * 730 hours/month * 30 = $2759.40

The approximate monthly difference is $8891.40/month

Now, these assume on-demand instances, and reserved instances can save you a substantial amount (29% in this case at $0.380 per instance/hour), but you’re locked in for at least a year and you’re still overpaying by 320%.

“An ‘awful lot of workloads are legacy’ -> Legacy workloads can’t be migrated”

So, this one’s a little harder to tackle just because “an awful lot” doesn’t correspond to a proportion, but let’s assume it means “100%” just to show how wrong this is according to the points he adduces:

Memory Consumption in Spring and Guice

The heart of the Sunshower.io cloud management platform is a distributed virtual machine that we call Gyre. Developers can extend Gyre through Sunshower’s plugin system, and one of the nicer features that Gyre plugins provide is a dependency-injection context populated with your task’s dependencies, as well as extension-point fulfillments exported by your plugin. For example, take (part) of our DigitalOcean plugin:

The details are beyond the scope of this post, but basically Gyre will accept an intermediate representation (IR) from one of its front-ends and rewrite it into a set of typed tuples whose values are derived from fields with certain annotations. The task above will be rewritten into (ComputeTask, Vertex, TaskResolver, TaskName, Credential). This tuple-format is convenient for expressing type-judgements, especially on the composite and recursive types that constitute Gyre programs.

Unit-testing Aurelia with Jest + JSDOM + TypeScript + Pug

All source can be found at: Aurelia-Aire

Testing UI components is one of the more difficult parts of QA. In Stratosphere (one of the key components of our cloud management platform), we’d been using Karma + Jasmine, but the browser component had been complicated by the fact that providing a DOM to tests in a fast, portable manner subject to memory and speed constraints can be pretty challenging. Sure, initially we did the PhantomJS thing, then Chrome Headless came out, but writing UI tests just never really felt natural.

Then, last week, we decided to open-source our component framework, Aire, built on top of UIKit+Aurelia, and that created an exigent need to fix some of the things we’d been limping along with, most importantly testing. The success of OSS projects depends on quite a few things, but I consider providing a simple way to get contributors up-and-running critical.

Simple set-up

Internally, Aurelia uses an abstraction layer (Aurelia PAL) instead of directly referencing the browser’s DOM. Aurelia will (in principle) run on any reasonable implementation of PAL. Aurelia provides a partial implementation OOTB, Aurelia/pal-nodejs, that will enable to (mostly) run your application inside of NodeJS.

Project Structure

Our project structure is pretty simple: we keep all our components and tests under a single directory, src:


aire
├── build
│   └── paths.js
├── gulpfile.js
├── index.html
├── jest.config.js
├── jspm.config.js
├── jspm.dev.js
├── package.json
├── package-lock.json
├── src
│   ├── main
│   │   ├── aire.ts
│   │   ├── application
│   │   ├── button
│   │   ├── card
│   │   ├── core
│   │   ├── core.ts
│   │   ├── dropdown
│   │   ├── events.ts
│   │   ├── fab
│   │   ├── form
│   │   ├── icon
│   │   ├── init
│   │   ├── init.ts
│   │   ├── loader
│   │   ├── nav
│   │   ├── navbar
│   │   ├── offcanvas
│   │   ├── page
│   │   ├── search
│   │   ├── table
│   │   ├── tabs
│   │   └── widget
│   └── test
│   │   ├── button
│   │   ├── core
│   │   ├── init
│   │   ├── render.ts
│   │   ├── setup.ts
│   │   └── tabs

...etc

At the top of the tree you’ll notice jest.config.js, the contents of which look like this:

Basically, we tell Jest to look under src for everything. ts-jest will automatically look for your Typescript compiler configuration, tsconfig.js in its current directory, so there’s no need to specify that.

Our tsconfig is pretty standard for Aurelia projects:

Test

If you just copy and paste our tsconfig.json and jest.config.js files while following the outlined directory structure, everything will Just Work (don’t forget to npm i -D the appropriate Jest and Aurelia packages.)

At this point, you can use aurelia-test to write tests a la:

hello.html

hello.ts

hello.test.ts

Now, you can run your tests with npx jest:

aire@1.0.0 test /home/josiah/dev/src/github.com/sunshower/aurelia-aire/aire
npx jest

PASS src/test/button/button.spec.ts
PASS src/test/tabs/tab-panel.spec.ts
PASS src/test/init/init.spec.ts
PASS src/test/core/dom.spec.ts

Test Suites: 4 passed, 4 total
Tests: 12 passed, 12 total
Snapshots: 0 total
Time: 3.786s
Ran all test suites.

Enabling support for complex DOM operations

That wasn’t too bad, was it? Well, the problem we encountered was that we use the excellent UIKit framework, and they obviously depend pretty heavily on the DOM. Any reference in Aire to UIKit’s Javascript would fail with a ReferenceError: is not defined error. Moreover, if we changed the Jest environment from node to jsdom, we’d encounter a variety of errors along the lines of TypeError: Failed to execute 'appendChild' on 'Node': parameter 1 is not of type 'Node' which I suspect were caused by pal-nodejs creating DOM elements via its own jsdom dependency while Jest was performing DOM operations using its jsdom dependency. In any case, the solution turned out to be to define a single, global jsdom by importing jsdom-global. Once we discovered this, we encountered other issues with browser-environment types and operations not being defined, but this setup.js resolved them:

At this point we could successfully test Aurelia + UIKit in NodeJS.

The final piece

All of our component views are developed in Pug, and I didn’t like that we’d be developing in Pug but testing using HTML. The solution to this turned out to be pretty simple: adding Pug as a development dependency and creating a pretty simple helper function:

With that final piece in place, our test looks like:

Conclusion

The benefits of writing tests in this way became apparent the moment we cloned the project into a new environment and just ran them via npm run test or our IDE’s. They’re fast, don’t require any environmental dependencies (e.g. browsers), and allow you to run and debug them seamlessly from the comfort of your IDE. But, perhaps most importantly, these are fun to write!

Gratitude as a Hedge

I think everyone says that building a startup is hard, but what they generally fail to convey is how hard it actually is. I left my previous position on March 1st of this year, and since then it seems like it’s been an unpunctuated litany of 14-hour days.  There’s also this notion (that I had) that when you found a company, you’ll mostly be working on projects that are interesting to you. I mean, it is your company, right?  And you do generally get to choose what that entails, right? And that that engagement will be prophylactic against the darkness.

That’s pretty heckin’ far from the truth in every way. The problems we’re solving are incredibly fascinating: AWS cost optimization, cloud resource management, and EC2 management are all issues I’m excited to be working on. But in the daily grind of solving those problems, about 10% of the code in Sunshower is what I would consider to be “strongly interesting”–the rest is fairly standard enterprise Java, Go, and Typescript.  In other words, Sunshower is 10% distributed graph reduction machine and 90% infrastructure built to interact and support that. It’s actually about twice as bad as that, since fully 55% of the code is tests and testing infrastructure.  And this is to say nothing of setting up continuous delivery, writing deployment manifests, replacing failed hard drives, configuring VPNs, signing documents, worrying about corporate and legal stuff…the list goes on.

That proportion of fun-work to work-work largely tracks my experience at companies of all sizes, with the primary differences being that the pay is much worse at your own startup, and that nobody’s really skeptical of a software project that is already profitable.  The cocktail of doubt, overwork, and scarcity is strong and bitter, and there’s only so much of it you can drink.

In October I got pretty sick and have only just really recovered, and that really pushed me to a place that the kids probably call “too extra.”  You can tell when you get there because everything’s distorted and whack–speed-bumps become cliffs and the little nuggets of interesting work you sprinkle throughout your day lose their luster. But the worst part is that it’s actually pretty hard to tell when you’re there.  I only realized I wasn’t in a great spot recently, and had troubles seeing a way out.

Lisa, my amazing wife and co-founder, really helped with the insight that gratitude is really what will get you out of dark spots.  Gratitude for family, for the people who throw their lot in with yours, for the opportunity to even try, for health, and even a warm breakfast and a colorful sunrise (fingers crossed).  And so I’m trying that and it’s really working.

 

 

 

 

Github Pages with TLS and a Backend

When you’re spending your days (and nights) developing the best cloud management platform possible, blogging consistently is hard. That said, we’re redoubling our efforts to get content out semi-regularly, even if it’s just to post something simple and (hopefully) helpful. To that end, I’d like to discuss setting up Github pages with a backend.

The Problem

 

Github.com allows you to host static content via its pages feature.  This is fantastic because it makes it trivial to create, deploy, and update a website.  This is how https://sunshower.io is hosted.

But what if you wanted to interact with, say, a database, to track signups?  Furthermore, what if you wanted to do it all over TLS?  This tutorial presumes that your registrar is AWS and your DNS is configured through Route53.

 

Set up Github DNS Alias Records For Github Pages

This one’s easy: In your Route53 Hosted Zone, create an A record that points to Github Page’s DNS servers:

>185.199.110.153

> 185.199.109.153

> 185.199.108.153

> 185.199.111.153

 

 

Then, check in a file in your pages repository under /docs called CNAME containing your DNS name (e.g. sunshower.io)

 

Push that sucker to master and you should have a bouncing baby site!

 

Publish your backend with the correct CORS Preflights

We pretty much just have a plugin for Sunshower.io that registers unactivated users.  Create an EC2 webserver/Lambda function/whatever to handle your requests.  The thing you have to note here is that your backend will have to support preflight requests.  A preflight request is an OPTIONS request that your server understands, and responds with the set of cross-origin resource sharing (CORS​ ) that your backend understands.

This is because your page, hosted at a Github IP, will be making requests to Amazon IPs, even though both are subdomains beneath your top-level domain.  For a JAX-RS service at, say, https://mysite.com:8443/myservice, you will need two methods:

 

@Path("myservice")
@Produces({
MediaType.APPLICATION_JSON,
MediaType.APPLICATION_XML,
})
@Consumes({
MediaType.APPLICATION_JSON,
MediaType.APPLICATION_XML,
})
public interface SignupEndpoint {

@POST
RegistrationConfirmationElement signup(RegistrationRequestElement request);
@OPTIONS
Response getOptions();
}

Note that there must be an @OPTIONS method for each actual conversation method that you want to interact with (e.g. the @POST method here).  What will happen is that a preflight will be made to the same path as the request, and the server will respond with whether that request is allowed.  You can widen the scope of @OPTIONS responses, but you should have a pretty good reason for doing so.

 

The actual @OPTIONS method will look something like:


@Override
public Response getOptions() {
return Response.status(Response.Status.OK)
.allow("POST")
.header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Methods", "POST, OPTIONS")
.header("Access-Control-Max-Age", 1000)
.header("Access-Control-Allow-Headers", "origin, x-csrftoken, content-type, accept")
.build();
}

where the allow value and Access-Control-Allow-Methods values align with the request type of the actual backend methods.

 

Set up a SSL-enabled ELB

The first thing to do here is to create a certificate using AWS’s wonderful Certificate Manager Service.

Request a certificate by navigating to Certificate Manager > Request a Certificate in the AWS Management Console.  Request a public certificate, and then request it for the fully-qualified name that you want your backend to appear under (e.g. backend.sunshower.io).  If you have access to any of the follow e-mail addresses at your domain:

  • administrator@your_domain_name
  • hostmaster@your_domain_name
  • postmaster@your_domain_name
  • webmaster@your_domain_name
  • admin@your_domain_name

then select e-mail verification, otherwise select DNS verification and proceed according to the instructions provided by AWS.

 

Once you’ve verified your certificate, create an elastic load balancer on whatever availability zones your backend spans.  If you only have one instance in one availability zone (shame shame!), add that availability zone first.  Create a listener targeting the port that your actual backend is listening on, and add a security group with an ingress port that corresponds to the port that you want your public-facing backend to listen on (for instance, if you want your backend to respond to requests to https://backend.sunshower.io, configure your ingress to be 443.

From there, configure a target that points to your actual backend. If your backend is hosted on EC2 instances, select the instance type for the target, otherwise select ip.  Reference the IP/instances of your backend and create the ELB.

 

Configure a DNS A Record for your ELB

The last thing we need to do is create an A Record that points to your ELB.  If, previously, you requested a TLS certificate for backend.mysite.com, you’ll want to create an A-record whose name is backend, alias=true, with an alias target that is the ELB you created in the previous step.  Save that sucker and you are good to go!