DevOps Without the DevOps Part 4: Finally, Containers!

In previous segments, we discussed how to collect project dependencies, and use Maven, Gradle, and the Spring Maven Gradle plugin to organize your project dependencies in a maintainable and traceable fashion. In this post, we’re going to take that setup and create a clean, reproducible build using Docker.

Understanding Containerization

A containerized application’s lifecycle is composed of several steps:

  1. Defining the container
  2. Building the container
  3. Tagging the container
  4. Pushing the container to a registry
  5. Running the container

We’ll be using Docker as our containerization technology, but it’s not the only option. Ensure that it’s installed for your platform before you continue.

Note that these instructions are intended for Linux-based machines. Containers differ from VMs in a very important way, namely that containers share the OS kernel of their container host.

What this means is that, say that my host operating system is Debian, based on Linux Kernel x86_64. Even if my container uses a different OS base (say, Ubuntu), the container and the host should be kernel-compatible (so, no Linux Kernel 3.10.x-ARM, for instance.)

These considerations are rarely an issue for many software projects. We use Debian-based containers because they’re similar to our dev environments and tooling, but Alpine is a really good option for many projects and has some advantages in production (e.g. resource utilization is anecdotally better.)

Step 1: Defining the Container

In the same way that we identified the compile and runtime dependencies of our project in the first series, let’s think about what we need to build the project. On my development system, I typically build a project by building and publishing the project’s Bill-Of-Materials POMs (mvn clean install -f bom), then by building the project’s artifacts with Gradle and publishing them to my local Maven repository (gradle clean build publishToMavenLocal, or, with Gradle’s nifty shortcuts gradle cle b pTML). So, I obviously need Gradle and Maven installed in my container.

So, what we’re going to do here is:

  1. Install the correct version of Java
  2. Download Maven from the Maven project site and install it into the container
  3. Download Gradle from the Maven project site and install it into the container

Let’s see if we can get rid of any of these steps by selecting the correct base container. Searching Docker hub, I see that Oracle provides JDK images. openjdk:8u141-jdk

My Dockerfile becomes simply:

FROM openjdk:8u141-jdk
ENTRYPOINT /bin/bash

Breaking down these instructions:

FROM openjdk:8u141-jdk says, “ask the local Docker registry to find an image with ID openjdk:8u141-jdk and derive from that. If you can’t find that image locally, reach out to and see if it’s there`

Building the container pulls the base image, then executes all of the commands in the Dockerfile, which produces a new container:

Sending build context to Docker daemon  220.7kB
Step 1/2 : FROM openjdk:8u141-jdk
8u141-jdk: Pulling from library/openjdk
3e17c6eae66c: Already exists 
74d44b20f851: Already exists 
a156217f3fa4: Already exists 
4a1ed13b6faa: Already exists 
77980e5d0a6d: Already exists 
5458607a81d3: Already exists 
e34cf8338f42: Already exists 
2f3d3da5c56e: Already exists 
2ade7a861e3f: Already exists 
Digest: sha256:4b0c879909b729d67d13e5004f5564df85a5f9c1c3820c13e41151edf1f1b1c0
Status: Downloaded newer image for openjdk:8u141-jdk
 ---> 74c95c985a85
Step 2/2 : ENTRYPOINT /bin/bash
 ---> Running in a4bd5943abcd
Removing intermediate container a4bd5943abcd
 ---> b1c22add1692
Successfully built b1c22add1692 # <-- IMPORTANT, this is your container ID, referenced as $CID.

We can check this new container out by running docker run -it --rm $CID which will drop you into a shell that looks something like:


Now, most base containers don’t have many programs installed. The JDK base containers do have Java, which is the important one.


Now, we can install Maven and Gradle really quickly.

Installing the Prerequisites

I like wget for simple downloads, but curl would work just as well. But we need to install Git anyway, so add:

RUN apt-get update
RUN apt-get install -y git-core wget

To your Dockerfile and build:

Sending build context to Docker daemon  220.7kB
Step 1/4 : FROM openjdk:8u141-jdk
 ---> 74c95c985a85
Step 2/4 : RUN apt-get update
 ---> Running in 2bfa3d396ac6
Get:1 stretch/updates InRelease [94.3 kB]
Ign:2 stretch InRelease
Get:3 stretch-updates InRelease [91.0 kB]
Get:4 stretch Release [118 kB]
Get:5 stretch Release.gpg [2434 B]
Get:6 stretch-updates/main amd64 Packages [12.1 kB]
Get:7 stretch/main amd64 Packages [9530 kB]
Get:8 stretch/updates/main amd64 Packages [468 kB]
Fetched 10.3 MB in 1s (5807 kB/s)
Reading package lists...
Removing intermediate container 2bfa3d396ac6
 ---> 4984ada9d0c8
Step 3/4 : RUN apt-get install -y git-core wget
 ---> Running in 6e6a79b1c1ab
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
The following packages will be upgraded:
1 upgraded, 1 newly installed, 0 to remove and 64 not upgraded.
Need to get 801 kB of archives.
After this operation, 8192 B of additional disk space will be used.
Get:1 stretch/main amd64 wget amd64 1.18-5+deb9u1 [800 kB]
Get:2 stretch/main amd64 git-core all 1:2.11.0-3+deb9u2 [1410 B]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 801 kB in 0s (3242 kB/s)
(Reading database ... 29522 files and directories currently installed.)
Preparing to unpack .../wget_1.18-5+deb9u1_amd64.deb ...
Unpacking wget (1.18-5+deb9u1) over (1.18-5) ...
Selecting previously unselected package git-core.
Preparing to unpack .../git-core_1%3a2.11.0-3+deb9u2_all.deb ...
Unpacking git-core (1:2.11.0-3+deb9u2) ...
Setting up wget (1.18-5+deb9u1) ...
Setting up git-core (1:2.11.0-3+deb9u2) ...
Removing intermediate container 6e6a79b1c1ab
 ---> 67169937ccdd
Step 4/4 : ENTRYPOINT ["/bin/bash"]
 ---> Running in cd4e07152d23
Removing intermediate container cd4e07152d23
 ---> 2c1ab0b17981
Successfully built 2c1ab0b17981

Now, your container will have both wget and git installed.

Setting Environment Variables

If you define an ENV variable in Docker, the value of that ENV can either be passed into the container, or you can specify a default value (or both). We want to be able to reference (and change) both the Gradle version and the Maven version so that if we want to upgrade either, we just pass in new versions when we’re building the container and viola!

# Environment Variables
ENV BASE_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Now, when we run the container, we have access to those environment variables:


Install Gradle

RUN mkdir -p /opt/build/tools/gradle #Create directory for gradle
RUN wget$ -O /opt/build/tools/  # Download gradle from 
RUN unzip -d /opt/build/tools/gradle /opt/build/tools/ # Unzip gradle 
ENV GRADLE_HOME=/opt/build/tools/gradle/gradle-$GRADLE_VERSION/bin #Export gradle location as GRADLE_HOME

Now, one really sweet thing about Docker is that, each of these commands defines a new layer. If the textual value of the command that builds a layer doesn’t change, then that layer is retrieved from the layer cache upon subsequent builds. What this means is that the URL for the Gradle download could change, and that wouldn’t break our container.

Install Maven

RUN mkdir -p /opt/build/tools/maven
RUN wget$MAVEN_VERSION/binaries/apache-maven-$ \
-O /opt/build/tools/
RUN unzip -d /opt/build/tools/maven /opt/build/tools/
ENV MAVEN_HOME=/opt/build/tools/maven/apache-maven-$MAVEN_VERSION/bin

Once these have all executed, we have a container with Maven, Gradle, and Git installed and ready to use! In the next post, I’ll discuss how to store credentials securely and pass them into the container so that we can pull our project from source-control and build it.

Of course, if you don’t want to maintain your own, this base image is available on docker hub as sunshower/sunshower-base:latest

Server-Sent Events with Undertow, Spring 5, and Resteasy 4

Resteasy 3.5 introduced Server-Sent Events (SSE), and there weren’t any good resources for showing how to get it up-and-running, so I thought I’d put together a quick how-to guide.

Add your dependencies

Up your org.jboss.resteasy:resteasy-jaxrs version to 4.0.0.Beta2. This should automatically pull in a JAX-RS API version 2.1 unless you’ve specified a version of JAX-RS directly. If you have, upgrade to 2.1. This will provide access to the SSE component of JAX-RS 2.1 (SseEventSink, Sse, etc.).

Define an SSE method

public interface TestService {

  void subscribe(@PathParam("id") String id, @Context SseEventSink sink, @Context Sse sse);// It's ok to put JAX-RS annotations on the interface.  Recommended, in fact.

  TestEntity save(TestEntity testEntity);

  String call(@PathParam("value") String input);

Implement that biz:

  public void subscribe(String id, SseEventSink sink, Sse sse) {
        new Thread(
            () -> {
              try {
                        .data(String.class, "starting domain " + id + " ...")
                sink.send(sse.newEvent("domain-progress", "50%"));
                sink.send(sse.newEvent("domain-progress", "60%"));
                sink.send(sse.newEvent("domain-progress", "70%"));
                sink.send(sse.newEvent("domain-progress", "99%"));
                sink.send(sse.newEvent("domain-progress", "Done."))
                        (Object obj) -> {
              } catch (final InterruptedException e) {

Write a test-case:

  void ensureSseWorks() throws InterruptedException {

    ResteasyWebTarget path =
        ((ResteasyWebTarget) webTarget).path(TestService.class).path("1/events");
    SseEventSource source =, TimeUnit.SECONDS).build();
    try (SseEventSource s = source) {
          e -> {

Experience failure

Failure 1

If you’ve done all that, it won’t work. The first error you’ll get is that your @Context SseEventSink is null. Unhork yourself by adding

  public SseEventSinkInterceptor sseEventSinkInterceptor() {
    return new SseEventSinkInterceptor();

to your Spring configuration.

Failure 2

The second failure you’ll experience is on the client-side: No MessageBodyReader for “text/event-stream”. This is fixed by adding a

  public SseEventProvider sseEventOutputProvider() {
    return new SseEventProvider ();

to your Spring client configuration.

Experience success

Ahh, delicious success

starting domain 1 ...


Plugging Sunshower again, you can get all this goodness by annotating your test-class with if you have io.sunshower.test:test-ws:1.0.0-SNAPSHOT as a dependency.

Unit-Testing Complex External Dependencies

The question of how to unit-test complex external dependencies arises pretty frequently. This is near-and-dear to our hearts because we have many of them. Mocking out complex responses is tedious and error-prone, so I’ll tell you what we do instead.

1: The setup

Stratosphere has an contract for obtaining instances from a cloud service provider, viz.,

public interface ListInstancesOperation extends ProviderOperation<List<Instances>> {
      public List<Instance> perform();

      public Provider getProvider(); 
      public Secret() getSecret();


perform() will typically interact with the provider service endpoint. The body of perform is quite simple:

  public List<Instance> perform() {
    AmazonEC2 client = createClient();

    DescribeInstancesResult result = client.describeInstances();
    return toInstances(result.getReservations());

This doesn’t look like a super-testable method, so what do we do?

2: Get the actual response and write it to a file

Before you write any code that interacts with an external dependency, you have to understand how that dependency behaves. I recommend keeping a set of IAM credentials in ~/.aws/credentials for testing. Once you do that, actually perform the request and see what you get back.


Now, we serialize the response using Java’s default serialization mechanism to a file that we check into source-control.

DescribeInstancesResult result = client.describeInstances();
Objects.write(result, relativeToRoot("src/test/resources/ec2/list-instances.obj")); // our utilities for writing an object using serialization.

3: Mock it real good

Recall that our method under test had 3 statements. One to create the client, one to make the request to the client, and one to map the results. The client’s method under test is public, so we can mock that. We gave createClient default visibility so that we could mock that while not exposing it as part of our Operation API, and then we put all our actual logic into a private method, whose behavior we want to test.

We can now set up our tests to mock out the external operation with a real result:

  private Secret secret;
  private AmazonEC2Client client;
  private EC2ListInstancesOperation operation;

  void setUp() {
    secret = new Secret();
    operation = new EC2ListInstancesOperation(secret, "us-west-2", (AWS) AWS.getInstance());
    operation = spy(operation);

    client = mock(AmazonEC2Client.class);
        .willReturn("src/test/resources/ec2/list-instances.obj", true));

And test it as follows:

  void ensureInstanceFirewallsAreCorrect() {
    List<Instance> perform = operation.perform();
    Instance instance = perform.get(0);
    assertThat(instance.getFirewalls().size(), is(1));
    Firewall firewall = instance.getFirewalls().iterator().next();
    assertThat(firewall.getName(), is("launch-wizard-2"));
    assertThat(firewall.getSecured().contains(instance), is(true));

This approach works pretty well for external dependencies that don’t change frequently (like the public APIs of large cloud service providers), but less well for external dependencies that do. What are some approaches you use?

Mapping Many Properties Quickly

One of the challenges in writing abstraction software like Sunshower is that we have to map a ton of vendor-specific properties into our data-model. I had considered writing a mapping language to transform, say, an EC2 Instance or Azure VM into one of our generic Sunshower compute instances, but decided against it because we just dump all the generic vendor properties like spotInstanceRequestId into properties on our internal model analog, and writing a compiler to do that is for big companies.

But how to avoid individually mapping each darn property by hand? Writing code like:

        new Property(

for each of exactly 100 hojillion properties across AWS’s and Azure’s and GCE’s and etc. data model is a breeding ground for ennui and bugs.

Attempt 1

Java 7 introduced the Beans API which makes introspecting Java beans super easy. There’s also a little-known feature of Java Matcher.replaceAll/First that allows you to reference a regular-expression capture group in the replacement string, so I whipped up:

PropertyDescriptor[] propertyDescriptors =
        Introspector.getBeanInfo(, Object.class)

                t -> {
                  Property.Type type = resolveType(t.getPropertyType());
                  if (type == null) {
                    return Stream.empty();
                  } else {

                    Object value = ReflectionUtils.invokeMethod(t.getReadMethod(), ec2instance);

                    return Stream.of(
                        new Property(
                            t.getDisplayName().replaceAll("(.)(\\p{Upper})", "$1-$2").toLowerCase(),
                                + t.getDisplayName()
                                    .replaceAll("(.)(\\p{Upper})", "$1-$2")
                            value == null ? null : String.valueOf(value)));

Which got me what I wanted:

            t -> {
                      "Name: %s, key: %s, value: %s", t.getName(), t.getKey(), t.getValue()));
Name: role, key: role, value: io.sunshower.stratosphere.core.topology.model.Instance
Name: aws.i18n.architecture, key: architecture, value: x86_64
Name: aws.i18n.client-token, key: client-token, value: sunsh-WebSe-1KKFOLRIA853V
Name: aws.i18n.ebs-optimized, key: ebs-optimized, value: false
Name: aws.i18n.ena-support, key: ena-support, value: true
Name: aws.i18n.hypervisor, key: hypervisor, value: xen
Name: aws.i18n.image-id, key: image-id, value: ami-39595240
Name: aws.i18n.instance-id, key: instance-id, value: i-05b1e0984260d51dd
Name: aws.i18n.instance-lifecycle, key: instance-lifecycle, value: null
Name: aws.i18n.instance-type, key: instance-type, value: t2.micro
Name: aws.i18n.kernel-id, key: kernel-id, value: null
Name: aws.i18n.key-name, key: key-name, value: sunshower-io
Name: aws.i18n.platform, key: platform, value: null
Name: aws.i18n.private-dns-name, key: private-dns-name, value:
Name: aws.i18n.private-ip-address, key: private-ip-address, value:

But it wasn’t very pretty or maintainable. So, refactoring:

Attempt 2

public class Properties {

  public static <T> void map(
      Class<T> type,
      T instance,
      Class<?> bound,
      PropertyAwareObject<?> target,
      PropertyMappingConfiguration cfg)
      throws IntrospectionException {

    PropertyDescriptor[] propertyDescriptors =
        Introspector.getBeanInfo(type, bound).getPropertyDescriptors();

            .flatMap(t ->, instance))

public class ProviderPropertyMappingConfiguration implements PropertyMappingConfiguration {

  private final String prefix;

  public ProviderPropertyMappingConfiguration(String prefix) {
    this.prefix = prefix;

  public Property.Type resolveType(PropertyDescriptor descriptor) {

    Class<?> propertyType = descriptor.getPropertyType();

    if (Boolean.class.equals(propertyType)) {
      return Property.Type.Boolean;
    if (isIntegral(propertyType)) {
      return Property.Type.Integer;
    if (String.class.equals(propertyType)) {
      return Property.Type.String;
    return null;

  public boolean accept(PropertyDescriptor propertyDescriptor) {
    Class<?> propertyType = propertyDescriptor.getPropertyType();
    return Boolean.class.equals(propertyType)
        || String.class.equals(propertyType)
        || isIntegral(propertyType);

  public String mapKeyName(PropertyDescriptor descriptor) {
    return descriptor.getDisplayName().replaceAll("(.)(\\p{Upper})", "$1-$2").toLowerCase();

  public String mapName(PropertyDescriptor descriptor) {
    return prefix
        + ".i18n."
        + descriptor.getDisplayName().replaceAll("(.)(\\p{Upper})", "$1-$2").toLowerCase();

  public String mapValue(PropertyDescriptor propertyDescriptor, Object instance) {
    Object result = ReflectionUtils.invokeMethod(propertyDescriptor.getReadMethod(), instance);
    return result == null ? null : String.valueOf(result);

  private boolean isIntegral(Class<?> propertyType) {
    return Integer.class.equals(propertyType)
        || int.class.equals(propertyType)
        || long.class.equals(propertyType)
        || Long.class.equals(propertyType);

  public <T> Stream<Property> map(PropertyDescriptor propertyDescriptor, T instance) {
    Property.Type type = resolveType(propertyDescriptor);
    return type == null
        ? Stream.empty()
        : Stream.of(
            new Property(
                mapValue(propertyDescriptor, instance)));

Allowing us to easily map any properties:

Name: role, key: role, value: io.sunshower.stratosphere.core.topology.model.Instance
Name: aws.i18n.ami-launch-index, key: ami-launch-index, value: 0
Name: aws.i18n.architecture, key: architecture, value: x86_64
Name: aws.i18n.client-token, key: client-token, value: sunsh-WebSe-1KKFOLRIA853V
Name: aws.i18n.ebs-optimized, key: ebs-optimized, value: false
Name: aws.i18n.ena-support, key: ena-support, value: true
Name: aws.i18n.hypervisor, key: hypervisor, value: xen

DevOps without the DevOps part 3: Don’t Buy the Hype

I’ve been asked whether this series really pertains to DevOps. The criticism, as I interpret it, is that DevOps is a conceptual framework for creating high-functioning development teams, whereas the concepts and processes outlined here are related to creating a build with specific tools.

The point at hand seems to be that DevOps as it’s practiced by its adherents and evangelists provides a universal template for whatever ails your development organization. In the DevOps utopia, managers, PMs, engineers, and QA all coexist in complete harmony to produce reliably phenomenal software on time and within the budget. Only, that’s not right, engineers and QA are coalesced in this other interpretation. And managers and PMs? Isn’t that a lot of overhead for healthy, DevOps practicing shops? In this construction, DevOps can become whatever you want it to be, or feel like it should be, because it has no grounding in actual processes.

What I’m getting at is that DevOps can’t be some abstract “template” for “building software factories.” There’s no such thing. What there is is a collective knowledge built of decades of trial and error, and everyone who tells you otherwise is trying to sell you something that you probably don’t need. Sure, containers are great. Kubernetes is great. CI/CD software is probably pretty solid. But an uncomfortable truth is that you can get every bit as good of results with Make, Cron, and Bash because the important thing in writing software is the knowledge of how it all fits together.

And that’s what this series is about: taking common tools and sharing our experience with them and what works well for us, and what doesn’t. You can derive some pretty generalizable truths from this process, like “You need a sane dependency-management process. Here’s what one looks like. There are other valid processes. The important thing is to have one.”

Unmarshalling generic properties with custom logic in JAXB.

I’m not entirely sure the idea behind this post will ever be useful to anyone, and certain as only a reformed sinner can be that it is not wise. Originally we had a polymorphic JAXB Property type, defined as:

public class PropertyElement<U, T extends PropertyElement<U, T>> extends AbstractElement<T> {

  @XmlAttribute String key;

  @XmlAttribute String name;

  @XmlAnyElement(lax = true)
  private U value;


Where we wanted private U value to be overridable by any subclass, provided that the actual runtime type of U exposed a public, static valueOf method that accepted a string and returned the value represented by the string. We did this because it worked for every Java primitive, plus most of the value types that I could think of. Naturally, this didn’t really work well with anything, so we just created an enumeration of possible types and forsook the notion of dynamically registering new property elements.

Abandoning this was a slightly bitter pill to swallow as it somewhat constrained the extensibility of Sunshower’s core data-model. But meh I tell you, because I spent hours fruitlessly and foolishly attempting to overcome the profound limitations of Java’s erased generics, only for my folly to haunt me for weeks longer. However, should some soul wish to soldier past, here’s what we did:

public class PropertyElement<U, T extends PropertyElement<U, T>> extends AbstractElement<T> {

  @XmlAnyElement(lax = true)
  private U value;

  private static transient Method valueOf;

I can hear some distant screaming. Possibly it’s me. That is verily a reference to a java.lang.reflect.Method It gets worse. The excellent EclipseLink MOXy library which we prefer over Jackson due to, you know, standards, allows us to define a void afterUnmarshal(Unmarshaller u, Object parent) method, which is invoked after MOXy has finished unmarshalling the object at hand. We figured that we didn’t need to write the actual value since, at runtime, its type is known to MOXy and JAX-RS, and sure enough we didn’t. Reading the object, on the other hand…(hangs head in shame):

  protected void doUnmarshal() {
    try {
      if (valueOf != null) {
        this.value = (U) valueOf.invoke(this, ((XMLRoot) value).getObject());
    } catch (Exception ex) {
      if(ex instanceof RuntimeException) { 
          throw ex;
      throw new RuntimeException(ex);

And we detected the actual type of the argument by introspecting the current type:

  public PropertyElement() {
    this.valueOf = configure(getClass());

 protected static Method configure(Class<?> type) {
    if (valueOf != null) {
      return valueOf;
    final ParameterizedType superclass = (ParameterizedType) type.getGenericSuperclass();
    final Object vt = superclass.getActualTypeArguments()[0];

    if (vt.getClass().equals(Class.class)) {
      Class<?> valueType = (Class<?>) vt;
      return Stream.of(valueType.getDeclaredMethods())
          .filter(t -> t.getName().equals("valueOf") && t.getParameterCount() == 1)
              t -> {
                final Class<?>[] ptypes = t.getParameterTypes();
                final Class<?> ptype = ptypes[0];
                return String.class.equals(ptype) || Object.class.equals(ptype);
              () ->
                  new IllegalArgumentException(
                      "Type does not supply a public, static method valueOf() accepting a string"));
    return null;

In a way, I’m sort of proud of this monster. Dr. Frankenstein would probably understand. On the other hand, our sins surely caught us, prompting a reasonable refactor to exorcise this demon.

Fix your indexes, homeslice

Database performance is the principal thing; therefore, undonk your indexes.
~Probably Confucius

Working with databases is one of the worst parts of building applications. There’s the setup, then there’s migrations, then there’s security, then there’s testing and on and on and on until you just can’t anymore. But, eventually, someone cobbles together a solution that encompasses at least a few of these things, and then you never touch it. Ever. It is forbidden to you. You will ruin everything in the application if you change even one thing. Everything will die.

But one day, something goes wrong. You’re getting transaction timeouts. Users are complaining that it’s taking 40 minutes to log into your UI (this happened once). Your application can’t handle all the data your users are feeding it.

So, someone profiles it and finds that it’s not your application, it’s the database. I mean, it’s pretty much always the database unless Keith is overusing hashmaps and locks again, but discovering which part is pretty tricky. Is it indexes? Is it disks? Is it the transaction log? Is it locks? Is it all of them? Who knows?

Fixing the code is at least a multi-month proposition that requires some poor sap go back and look at the code that interacts with your database, potentially donking the whole biz for everyone, not just this customer. The people who can actually fix it are generally not the ones who volunteer for that hell, and the people who volunteer will usually just make it worse.

So, you do the sensible thing and buy a bigger database server with faster disks. It helps for a while, then it just falls over again. You’ve got to fix this.

So, you perform an investigation, and discover that you’re using either a database sequence ID, or a UUID, or a hash function (I have seen this), and here’s the choose-your-own-misadventure part.

Database sequence ID

Your data is insertion-ordered, which is good for indexing. Heck, you can even use BRIN indexing, but you’re making a round-trip to the database for every insertion, and there’s a network round-trip and either a lock or a single thread somewhere in there. Usually there’s No escaping that.


Insertions are random. You got page-splits homeskillet. Like, everywhere. And these aren’t cheap. Your average page density is like, 0.0000000001%. Your index is like a million-page phonebook with one number per page. Oh, yeah, and you gotta maintain that B-Tree or whatever on disk because that sucker is way too big to fit into memory. Additional disk hits, baby! No BRINs for you!

Not to mention you’re probably storing the value as a string if my experience with these things is generalizable, which means you’re probably using 36 bytes/ID instead of the 16 actually required. Plus your IDs have a character set that they don’t need, and god help you if that default changes between versions or someone changes it.

Cryptographic hash (content-based addressing)

This one has all the problems of a UUID, plus some. Like, I’ve seen 128-byte (that is truly “byte” because they’re storing it as a string) MD5sum IDs used in large systems.

Ok, ok. So you stored them as a byte array, that only eliminates the size problem.

But here’s the kicker: If you’re using one of these techniques, your implementation will probably be exchanged for another of the problematic ones I just mentioned. Hax.

Use Flake! (Just do it)

At Sunshower, being ourselves the suckers who will eventually have to undonk our database, we decided to get ahead of the problem. The first post we ever did here was about Flake IDs, and now we have a well-tested, high-quality, MIT-licensed implementation that you can just use. If you’re already using UUIDs stored as byte arrays, drop this biz right in.

Step 1: Add our common library to your dependencies

  1. Make sure Nexus Snapshots are enabled by adding to your Maven repositories

  2. Add io.sunshower.persist:persist-api:<version> (version is currently 1.0.0-SNAPSHOT)–we’ll get a release soon (TM), but this API is totes stable.

  3. Create you a Flake Sequence. I’d recommend 1 per table for very high-scale systems since you can only generate 10,000/sequence/second. Pretty simple:

import io.sunshower.common.Identifier;
import io.sunshower.persist.Identifiers;
import io.sunshower.persist.Sequence;

Sequence<Identifier> sequence = Identifiers.newSequence(true); //'false' would have the sequence API throw an exception if you requested more IDs than you could generate in a given timespan, ~10k/sec/sequence.  Otherwise, the API blocks until the counter resets.

Then, use it however. For instance, to use it with JPA/Hibernate:

public class MyFlakeEntity {
private byte[] id;


In your database schema, just store them as a byte-array. We use bytea in Postgres which incurs 1 additional byte of overhead per row. Meh.

Swank ACLs with Flake IDs

If you really want declarative ACL goodness coupled with delicious DB goodness, and you’re using Spring Security, pull in our service-security library at io.sunshower.service:service-core:<version> (still 1.0.0-SNAPSHOT), then add these to your Spring configuration:

prePostEnabled = true, 
jsr250Enabled = true, 
securedEnabled = true

public class MySecurityConfiguration { // whatever your configuration is here

  public MutableAclService jdbcAclService(
      JdbcTemplate template, LookupStrategy lookupStrategy, AclCache aclCache) {
    return new IdentifierJdbcMutableAclService(template, lookupStrategy, aclCache, "<SCHEMA>");

  public AclCache aclCache(
      @Named("caches:spring:acl") Cache cache, // replace with your own cache.  This can just be a concurrent hashmap implementation.  We like Ignite
      PermissionGrantingStrategy permissionGrantingStrategy,
      AclAuthorizationStrategy aclAuthorizationStrategy) {
    return new SpringCacheBasedAclCache(
        cache, permissionGrantingStrategy, aclAuthorizationStrategy);

  public LookupStrategy aclLookupStrategy(
      DataSource dataSource,
      AclCache aclCache,
      AclAuthorizationStrategy aclAuthorizationStrategy,
      PermissionGrantingStrategy permissionGrantingStrategy) {
    return new IdentifierEnabledLookupStrategy(
        "<SCHEMA>", dataSource, aclCache, aclAuthorizationStrategy, permissionGrantingStrategy);

  public AclAuthorizationStrategy aclAuthorizationStrategy(GrantedAuthority role) {
    return new MultitenantedAclAuthorizationStrategy(role);

  public PermissionGrantingStrategy permissionGrantingStrategy(AuditLogger logger) {
    return new DefaultPermissionGrantingStrategy(logger);

Then, drop this schema into your migrations:

  id        BYTEA        NOT NULL PRIMARY KEY,
  principal BOOLEAN      NOT NULL,
  sid       VARCHAR(100) NOT NULL,
  CONSTRAINT unique_uk_1 UNIQUE (sid, principal)

  class VARCHAR(100) NOT NULL,
  CONSTRAINT unique_uk_2 UNIQUE (class)

CREATE TABLE <SCHEMA>.acl_object_identity (
  id                 BYTEA PRIMARY KEY,
  object_id_class    BYTEA   NOT NULL,
  object_id_identity BYTEA   NOT NULL,
  parent_object      BYTEA,
  owner_sid          BYTEA,
  entries_inheriting BOOLEAN NOT NULL,
  CONSTRAINT unique_uk_3 UNIQUE (object_id_class, object_id_identity),
  CONSTRAINT foreign_fk_1 FOREIGN KEY (parent_object) REFERENCES <SCHEMA>.acl_object_identity (id),
  CONSTRAINT foreign_fk_2 FOREIGN KEY (object_id_class) REFERENCES <SCHEMA>.acl_class (id),
  CONSTRAINT foreign_fk_3 FOREIGN KEY (owner_sid) REFERENCES <SCHEMA>.acl_sid (id)

  id                  BYTEA PRIMARY KEY,
  acl_object_identity BYTEA   NOT NULL,
  ace_order           INT     NOT NULL,
  sid                 BYTEA   NOT NULL,
  mask                INTEGER NOT NULL,
  granting            BOOLEAN NOT NULL,
  audit_success       BOOLEAN NOT NULL,
  audit_failure       BOOLEAN NOT NULL,
  CONSTRAINT unique_uk_4 UNIQUE (acl_object_identity, ace_order),
  CONSTRAINT foreign_fk_4 FOREIGN KEY (acl_object_identity)
  REFERENCES <SCHEMA>.acl_object_identity (id),

And you can totally use Spring Security annotation-driven security! For instance:

  @PreAuthorize("hasPermission(#id, 'io.sunshower.stratosphere.core.vault.model.Secret', 'DELETE')")
  public Secret delete(Identifier id) {
    Secret s = super.delete(id);
    return s;

Also, if you want to use your ACLs in JPQL (or HQL or whatever), we’ve mapped your entities for you. Pull in io.sunshower.core:core-api:1.0.0-SNAPSHOT and you’ll find the following classes:

  1. io.sunshower.model.core.auth.ObjectIdentity
  2. io.sunshower.model.core.auth.SecuredObject
  3. io.sunshower.model.core.auth.SecurityIdentity

If you need multitenancy and security groups (RBAC), that’s a topic for another post, but we have that, too.

To grant a set of permissions:

  public <T extends Persistable> void grantWithCurrentSession(
      Class<T> type, T instance, Permission... permissions) {
    final ObjectIdentity oid = new ObjectIdentityImpl(type, instance.getId());
    Sid sid = new PrincipalSid(session.getUsername());
    MutableAcl acl;
    try {
      acl = (MutableAcl) aclService.readAclById(oid);
    } catch (NotFoundException ex) {
      acl = ((MutableAclService) aclService).createAcl(oid);
    for (Permission permission : permissions) {
      acl.insertAce(acl.getEntries().size(), permission, sid, true);
    ((MutableAclService) aclService).updateAcl(acl);

To query all of the objects belonging to a user:

select e from Entity e
join e.identity oid
where oid.owner.username = :username;

Which simply requires the mapping:

  @OneToOne(fetch = FetchType.LAZY)
  @JoinColumn(name = "id", insertable = false, updatable = false)
  private ObjectIdentity identity;

  public ObjectIdentity getIdentity() {
    return identity;

In summary, you don’t have to choose between cool features and robust ACL/RBAC support and database performance with Sunshower. We’re happy to do some of that heavy lifting–and it’s all free!

Aurelia @containerless and custom events

One thing that just got me about [Aurelia’s] @customElement used in conjunction with @containerless is that it doesn’t propagate events.

For instance, I had tried to use:

export class TagPanel {
     el: HTMLElement;
     dispatch() : void {
        let e = createEvent('saved',;

With view markup (Pug)

   button(ref="el", click.delegate="dispatch()") 

But I noticed that the event wasn’t propagating. It turns out that the correct thing to do in this case is to inject the DOM element into your view-model, and then dispatch the event from there:

export class TagPanel {

     constructor(private el: Element) {

     dispatch() : void {
        let e = createEvent('saved',;

And don’t reference el from your view:


DevOps without the Devops part 2: The Structure

I’m back! We’ve decided to go head with Sunshower full time, so expect updates much more regularly here!

Last time, we looked at getting a simple build dockerized. Using the Go platform made this pretty simple for us from a dependency perspective, but a lot of you are using a dependency resolution tool like Gradle, Maven, Crate, or Ivy. This post will detail how to configure Maven and Gradle so that your dependencies are manageable and consolidated–a necessary prerequisite for any sane build/release process.

The Base Project has quite a few individual projects, each of which having at least several sub-projects. The first project that we need to build is sunshower-devops. This project contains

  1. Docker container definitions
  2. Bill-of-material POMs that are used by each of the projects
  3. Various scripts bundled with our Docker images.

Recall that last time, the first thing I recommended was for you to aggregate all of your dependencies. We needed that information because it allows us to build a bill-of-materials for our project. This is important because it allows us to understand clearly what our project pulls in. This in turn enables us to manage our dependencies in a revisionable and deterministic fashion. Let’s look at what one of our bills-of-material POMs looks like:

<project xmlns=""



  <name>Sunshower Persistence Dependencies</name>


... other properties

... Other dependencies


Basically, this is just a standard Maven POM file with a structure that is convenient for declaring dependencies. The first thing to note is that dependencies are declared within a <dependencyManagement> tag. This means that POM files that inherit from this, or import it, will not automatically depend on the dependencies declared within, only that if they explicitly declare a dependency from this POM, they will inherit its configuration as it appears in this declaration. For instance, if I import sunshower-env:persist-dependencies, then if I declare org.hibernate:hibernate-entitymanager in my importing POM, I will get the ${hibernate.version} version declared in persist-dependencies without having to redeclare it.

Basically, what we’re going for is this:

  1. We create bill-of-material (BOM) POMs for each category of dependency. This is optional, but I like it because these suckers can get huge otherwise.

  2. If we have commonality between our BOM POMs (and we will), we pull it up into an aggregator POM.

  3. We import each of our BOM POMs into our parent pom (sunshower-parent)

  4. Every subproject in our system will have its own BOM POM that derives from sunshower-parent

  5. Each gradle file for each project uses the spring-maven-gradle plugin to import its BOM pom
  6. Viola! If we add a dependency, that addition is recorded in Git. We can see exactly what we’re pulling in for any release (and go back to a previous POM if we need to)



Now, say I want to use hibernate-entitymanager in sunshower-base:persist:

<project xmlns="" xmlns:xsi=""


    <name> Imported Bill-Of-Materials</name>




Then, I simply import that into my build.gradle file for whichever project depends on hibernate-entitymanager

    dependencyManagement {
        imports {

Now, in that project (or any subproject thereof), I can just add hibernate-entitymanager to the dependencies block:

dependencies {
    implementation 'org.hibernate:hibernate-entitymanager'


While this may seem like overkill, I like it because it scales quite well. It’s easy to audit (assuming you enforce the process), maintainable (dependencies are grouped together sensibly), and forces you to think about what you’re bringing in. Sometimes incompatibilities can be prevented simply by looking through the dependency lists and determining whether two versions are compatible. Finally, it gives a consistent view of the world to everyone in the project: if everyone contributing to the project follows the rules, you won’t get one component consuming one version of a dependency, and another component consuming another, which is a common source of bad builds IME.

DevOps without the DevOps part 1: The Plan

Forget DevOps

The point of this blog series is to cut through the hype and jargon and get to the pith of DevOps, and there is no better way to do that than with concrete examples.

The Problem

Before you go looking for solutions, it’s helpful to understand what the problem is. And what the problem is that, somewhere in the deep past, your foundational engineers started writing software. They got a requirement, and they:

  • Selected a language
  • Selected a build tool
  • Selected a revision control system
  • Started writing code

That’s pretty much it. Life was good–the software probably did its job (which is why you’re in business, right?) Then, requirements got added, those engineers left and new ones joined, and they:

  • Wrote new functionality (frequently in other languages)
    — Added new build tools and processes and artifacts
  • Wrote a bunch of helper scripts at 1:00 in the morning on a customer call that got integrated into the product
  • Added support for new platforms and operating systems
  • Wrote a bunch of QA and automation scripts
  • etc.

And now it’s no longer an easy task to even build the project. In fact, if you’re doing anything Enterprisey (TM), I would bet good money that fewer than 10% of your team understands how to build and package your product, much less run your automated system tests, etc. And I promise you that no one making purchasing decisions for DevOps solutions understands what your process is. So, let’s figure out how to figure out what your problem is.


A typical pipeline addresses the following steps (current in bold)

  • Build
  • Package
  • Release

Now, your process can vary quite a bit. This series isn’t intended to teach or advocate a particular methodology, and it’s not prescriptive. There are frequently many sub-steps to each of these steps.

Step 1: Understand your Project’s Build Structure

You’re going to have to go beyond “well, we use Java and Lua and Make and Gradle.” You’re going to want to start by enumerating every single component of your project. Every last one. This will create a set of projects that we will call P (for projects). Draw them out, but give yourself plenty of space.

Here’s mine for Sunshower components:

Now, yours doesn’t have to look like this. You could get away with just creating a list, but I’d prefer that you drew it out: it’s a good exercise, it provides a lot of hints as to how things fit together. Now, understand that this is a pretty small project with just a few dependencies. Your project could take weeks to diagram out correctly and comprehensively–that’s ok. You’ll save time and money in the long run. Draw it out, but don’t get hung up on anything like UML or BPML–just try to visually understand the pieces of your project.

Information that should be present

  • All of the tools on all of the platforms you need to support
    — For instance, if you need to build on Windows Server 2012 and Ubuntu 14.04, and your build-tool is Make, you’ll need (for instance)
    — MinGW on Windows and devtools on Ubuntu. I would just draw them out as separate tool boxes on your diagram. It’ll help
  • Source information for your dependencies
    — You may have to build behind a firewall. Make sure that your description of build environments includes what needs to be accessed and how
  • If you’re using managed dependencies (via Ivy, Maven, Gradle, or Go), don’t necessarily worry about enumerating them here. It’ll clutter things up
    — Unless: you need access to a private source for your dependencies (e.g. you’ve purchased a subscription to a library that requires authenticated access). In which case:
    — Be sure to indicate somewhere on here how to access private sources (e.g. settings.xml for Maven)
  • Specific versions for tools. This is really important. Don’t assume that different versions of tooling will produce the same results.

Some words to the wise:

  • Do not buy and products or services at this point. You’ll get something that you’ll never implement correctly or completely.
  • Get one of your engineers to do this. If you don’t have one who can, be absolutely certain that a consultant can build one out for you
  • This isn’t the whole story by any stretch of the imagination. In subsequent posts, we’ll cover how to ensure that everything gets built
  • Be sure that this is correct. Try it out for yourself and verify that you can at least build everything (don’t worry about getting it running or packaged or installed at this point)
  • Institute a process wherein any changes to the project are reflected in this diagram. If this diagram is changing too frequently, you need to understand why. Too much churn on this diagram past a few weeks into the start of a project is a serious warning sign.

Step 2: Create a Template for Building your Project

This is step zero in terms of automation IMO. Sure, build tools like Gradle “automate” builds in that they can be used for many steps described in this series, but we’re only considering the “compile my component and generate some artifacts” portions of their capabilities. You will almost certainly be using your build tools to automate other tasks, but don’t worry about that now. It’s going to be a lot harder to implement a good automated workflow if you don’t clearly distinguish between types of tasks and how you (you dear reader, and your org) perform them.

The goal of this step is to create a template for getting properly configured build environments. This means that

  1. They will have all of the tools enumerated in your tool boxes in your diagrams
  2. They will have the correct versions of those tools
  3. They will have access to the resources (connectivity to Github, credentials to Maven repositories, etc.) required to build your project

Let’s look at some tools that can help us out here.

There are quite a few more, and each of these tools can do more than allow you to create build templates, but we’re only concerned with creating build templates at this stage. These can all be made to work with subsequent stages in your process, and some of them can use others to create templates (e.g Vagrant can generate OVAs). This series will only cover Docker, contact me or leave a comment if you want me to show you how to do the same thing in any of the others.

Example 1: Translating A Tools Box into a Docker Container

Let’s start with the tools box for Updraft:

Before we get started, there’s a bit of a chicken-and-egg situation we need to discuss here. It might seem natural to:

  1. Install Go in our container
  2. Install Git in our container
  3. Checkout Updraft into our container
  4. Build it, etc.

But this introduces a bit of a problem in that there’s not an easy, clean way to check out and build a specific version of Updraft. So, instead, what we’ll do is we’ll require the actual build machines (VM images or bare metal) to have

  1. Docker installed
  2. Git installed

Then, we’ll check out the desired version of the project from Github, build the container using Docker, then build Updraft inside the container. This allows us to easily build different versions of Updraft reliably and reproducibly. I promise that you won’t need much more than this on any actual instance of a Docker host.

So, having installed Docker and Git, let’s add our Dockerfile. A lot of people like to keep their Dockerfiles at the root of their projects, but I like to keep mine in a “docker” subdirectory in case I need more than one:

| etc.
│   │   └── error_code.go
│   └── utils
│   ├── uuid.go
│   └── uuid_test.go
├── docker
│   ├── Dockerfile
│   └──

├── front
│   └── parser
│   ├── abstract_parser.go
│   ├── base_source.go
│   ├── base_source_test.go
│   ├── parser_message_listener.go
│   └── parser_test.go
| etc.

Note It’ll probably take a few tries to get your Dockerfile right. You can build and run the container in one (chained) command thuslike:

docker build -t "updraft" -f docker/Dockerfile . && docker run -it --rm --name updraft updraft 

Or, if you like, two commands:

docker build -t "updraft" -f docker/Dockerfile . 
docker run -it --rm --name updraft updraft 

Each run will re-create the container in the new state and drop you into a bash shell inside the container, where you can execute more commands until you get it right. Remember to add each command necessary to build your project to your Dockerfile as a RUN statement (so that it becomes a layer in your container).


FROM golang:1.8 ## Super easy Go base image (first dependency in our tools box)
WORKDIR /go/src/updraft ## current working directory inside the container
COPY . .  ## Copy the contents of the current directory into WORKDIR
RUN go get ./... ## Get updraft's dependencies (dependencies in our dependencies box)
RUN go build -o out/ucc ucc/main.go  ## Build updraft and output it into $WORKDIR/out

And viola! Running that all together produces:

➜  updraft git:(master) ✗ docker build -t "updraft" -f docker/Dockerfile . && docker run -it --rm --name updraft updraft
Sending build context to Docker daemon  33.59MB
Step 1/5 : FROM golang:1.8
 ---> 0e070ede84f7
Step 2/5 : WORKDIR /go/src/updraft
 ---> Using cache
 ---> 4d23d64b3b23
Step 3/5 : COPY . .
 ---> 8a9a904e4971
Removing intermediate container 996472d595ff
Step 4/5 : RUN go get ./...
 ---> Running in 919addd0c758
 ---> 1608bbf6258f
Removing intermediate container 919addd0c758
Step 5/5 : RUN go build -o out/ucc ucc/main.go
 ---> Running in a0c9e2bd2112
 ---> d2d8855e56ae
Removing intermediate container a0c9e2bd2112
Successfully built d2d8855e56ae
Successfully tagged updraft:latest
root@caa586b9157c:/go/src/updraft# ls  backends  common  docker  front  middle  out  pascal  ucc

root@caa586b9157c:/go/src/updraft# cd out
root@caa586b9157c:/go/src/updraft/out# ./ucc -h

            Updraft is a modern compiler
            framework and collection written in
            pure Go for portability, speed,
            and embeddability

  ucc [flags]
  ucc [command]

Available Commands:
  help        Help about any command
  trace       trace various aspects of configuration and execution

      --config string     config file (default is $HOME/.ucc.yaml)
  -e, --execute string    select the execution model (default "interpreter")
  -f, --file string       file(s) to compile
  -h, --help              help for ucc
  -l, --language string   select the target language (default "pascal")

Use "ucc [command] --help" for more information about a command.

Deploy to Dockerhub

Now that we’ve successfully built our image, let’s publish it so that anyone can use it:

➜  updraft git:(master) ✗ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to to create one.
Username (jhaswell): 
Login Succeeded
➜  updraft git:(master) ✗ docker tag updraft sunshower/updraft
➜  updraft git:(master) ✗ docker push sunshower/updraft
The push refers to a repository []
34f0b9f7c17e: Pushed 
ae2fc6d3daac: Pushed 
9fabb269895b: Pushed 
233937e534e9: Pushed 
f847715687e8: Mounted from library/golang 
056e99358ddc: Mounted from library/golang 
237b592486ad: Mounted from library/golang 
86ca4e4dcab9: Mounted from library/golang 
44b57351135e: Mounted from library/golang 
00b029f9aa09: Mounted from library/golang 
18f9b4e2e1bc: Mounted from library/golang 
latest: digest: sha256:089e2aa54db28ce35428e8a37cb46f51461def857d812aa14f1060030be384ba size: 2635
➜  updraft git:(master) ✗ 

And that’s it! Next time we’ll look at how to automate this process so that it happens with every checkin (introducing new tools) and how to make your artifacts available to everyone via the web and other means.