Unit-Testing Complex External Dependencies

The question of how to unit-test complex external dependencies arises pretty frequently. This is near-and-dear to our hearts because we have many of them. Mocking out complex responses is tedious and error-prone, so I’ll tell you what we do instead.

1: The setup

Stratosphere has an contract for obtaining instances from a cloud service provider, viz.,

public interface ListInstancesOperation extends ProviderOperation<List<Instances>> {
      public List<Instance> perform();

      public Provider getProvider(); 
      public Secret() getSecret();


perform() will typically interact with the provider service endpoint. The body of perform is quite simple:

  public List<Instance> perform() {
    AmazonEC2 client = createClient();

    DescribeInstancesResult result = client.describeInstances();
    return toInstances(result.getReservations());

This doesn’t look like a super-testable method, so what do we do?

2: Get the actual response and write it to a file

Before you write any code that interacts with an external dependency, you have to understand how that dependency behaves. I recommend keeping a set of IAM credentials in ~/.aws/credentials for testing. Once you do that, actually perform the request and see what you get back.


Now, we serialize the response using Java’s default serialization mechanism to a file that we check into source-control.

DescribeInstancesResult result = client.describeInstances();
Objects.write(result, relativeToRoot("src/test/resources/ec2/list-instances.obj")); // our utilities for writing an object using serialization.

3: Mock it real good

Recall that our method under test had 3 statements. One to create the client, one to make the request to the client, and one to map the results. The client’s method under test is public, so we can mock that. We gave createClient default visibility so that we could mock that while not exposing it as part of our Operation API, and then we put all our actual logic into a private method, whose behavior we want to test.

We can now set up our tests to mock out the external operation with a real result:

  private Secret secret;
  private AmazonEC2Client client;
  private EC2ListInstancesOperation operation;

  void setUp() {
    secret = new Secret();
    operation = new EC2ListInstancesOperation(secret, "us-west-2", (AWS) AWS.getInstance());
    operation = spy(operation);

    client = mock(AmazonEC2Client.class);
        .willReturn(Objects.read("src/test/resources/ec2/list-instances.obj", true));

And test it as follows:

  void ensureInstanceFirewallsAreCorrect() {
    List<Instance> perform = operation.perform();
    Instance instance = perform.get(0);
    assertThat(instance.getFirewalls().size(), is(1));
    Firewall firewall = instance.getFirewalls().iterator().next();
    assertThat(firewall.getName(), is("launch-wizard-2"));
    assertThat(firewall.getSecured().contains(instance), is(true));

This approach works pretty well for external dependencies that don’t change frequently (like the public APIs of large cloud service providers), but less well for external dependencies that do. What are some approaches you use?

Mapping Many Properties Quickly

One of the challenges in writing abstraction software like Sunshower is that we have to map a ton of vendor-specific properties into our data-model. I had considered writing a mapping language to transform, say, an EC2 Instance or Azure VM into one of our generic Sunshower compute instances, but decided against it because we just dump all the generic vendor properties like spotInstanceRequestId into properties on our internal model analog, and writing a compiler to do that is for big companies.

But how to avoid individually mapping each darn property by hand? Writing code like:

        new Property(

for each of exactly 100 hojillion properties across AWS’s and Azure’s and GCE’s and etc. data model is a breeding ground for ennui and bugs.

Attempt 1

Java 7 introduced the Beans API which makes introspecting Java beans super easy. There’s also a little-known feature of Java Matcher.replaceAll/First that allows you to reference a regular-expression capture group in the replacement string, so I whipped up:

PropertyDescriptor[] propertyDescriptors =
        Introspector.getBeanInfo(com.amazonaws.services.ec2.model.Instance.class, Object.class)

                t -> {
                  Property.Type type = resolveType(t.getPropertyType());
                  if (type == null) {
                    return Stream.empty();
                  } else {

                    Object value = ReflectionUtils.invokeMethod(t.getReadMethod(), ec2instance);

                    return Stream.of(
                        new Property(
                            t.getDisplayName().replaceAll("(.)(\\p{Upper})", "$1-$2").toLowerCase(),
                                + t.getDisplayName()
                                    .replaceAll("(.)(\\p{Upper})", "$1-$2")
                            value == null ? null : String.valueOf(value)));

Which got me what I wanted:

            t -> {
                      "Name: %s, key: %s, value: %s", t.getName(), t.getKey(), t.getValue()));
Name: role, key: role, value: io.sunshower.stratosphere.core.topology.model.Instance
Name: aws.i18n.architecture, key: architecture, value: x86_64
Name: aws.i18n.client-token, key: client-token, value: sunsh-WebSe-1KKFOLRIA853V
Name: aws.i18n.ebs-optimized, key: ebs-optimized, value: false
Name: aws.i18n.ena-support, key: ena-support, value: true
Name: aws.i18n.hypervisor, key: hypervisor, value: xen
Name: aws.i18n.image-id, key: image-id, value: ami-39595240
Name: aws.i18n.instance-id, key: instance-id, value: i-05b1e0984260d51dd
Name: aws.i18n.instance-lifecycle, key: instance-lifecycle, value: null
Name: aws.i18n.instance-type, key: instance-type, value: t2.micro
Name: aws.i18n.kernel-id, key: kernel-id, value: null
Name: aws.i18n.key-name, key: key-name, value: sunshower-io
Name: aws.i18n.platform, key: platform, value: null
Name: aws.i18n.private-dns-name, key: private-dns-name, value: ip-172-31-12-114.us-west-2.compute.internal
Name: aws.i18n.private-ip-address, key: private-ip-address, value:

But it wasn’t very pretty or maintainable. So, refactoring:

Attempt 2

public class Properties {

  public static <T> void map(
      Class<T> type,
      T instance,
      Class<?> bound,
      PropertyAwareObject<?> target,
      PropertyMappingConfiguration cfg)
      throws IntrospectionException {

    PropertyDescriptor[] propertyDescriptors =
        Introspector.getBeanInfo(type, bound).getPropertyDescriptors();

            .flatMap(t -> cfg.map(t, instance))

public class ProviderPropertyMappingConfiguration implements PropertyMappingConfiguration {

  private final String prefix;

  public ProviderPropertyMappingConfiguration(String prefix) {
    this.prefix = prefix;

  public Property.Type resolveType(PropertyDescriptor descriptor) {

    Class<?> propertyType = descriptor.getPropertyType();

    if (Boolean.class.equals(propertyType)) {
      return Property.Type.Boolean;
    if (isIntegral(propertyType)) {
      return Property.Type.Integer;
    if (String.class.equals(propertyType)) {
      return Property.Type.String;
    return null;

  public boolean accept(PropertyDescriptor propertyDescriptor) {
    Class<?> propertyType = propertyDescriptor.getPropertyType();
    return Boolean.class.equals(propertyType)
        || String.class.equals(propertyType)
        || isIntegral(propertyType);

  public String mapKeyName(PropertyDescriptor descriptor) {
    return descriptor.getDisplayName().replaceAll("(.)(\\p{Upper})", "$1-$2").toLowerCase();

  public String mapName(PropertyDescriptor descriptor) {
    return prefix
        + ".i18n."
        + descriptor.getDisplayName().replaceAll("(.)(\\p{Upper})", "$1-$2").toLowerCase();

  public String mapValue(PropertyDescriptor propertyDescriptor, Object instance) {
    Object result = ReflectionUtils.invokeMethod(propertyDescriptor.getReadMethod(), instance);
    return result == null ? null : String.valueOf(result);

  private boolean isIntegral(Class<?> propertyType) {
    return Integer.class.equals(propertyType)
        || int.class.equals(propertyType)
        || long.class.equals(propertyType)
        || Long.class.equals(propertyType);

  public <T> Stream<Property> map(PropertyDescriptor propertyDescriptor, T instance) {
    Property.Type type = resolveType(propertyDescriptor);
    return type == null
        ? Stream.empty()
        : Stream.of(
            new Property(
                mapValue(propertyDescriptor, instance)));

Allowing us to easily map any properties:

Name: role, key: role, value: io.sunshower.stratosphere.core.topology.model.Instance
Name: aws.i18n.ami-launch-index, key: ami-launch-index, value: 0
Name: aws.i18n.architecture, key: architecture, value: x86_64
Name: aws.i18n.client-token, key: client-token, value: sunsh-WebSe-1KKFOLRIA853V
Name: aws.i18n.ebs-optimized, key: ebs-optimized, value: false
Name: aws.i18n.ena-support, key: ena-support, value: true
Name: aws.i18n.hypervisor, key: hypervisor, value: xen

DevOps without the DevOps part 3: Don’t Buy the Hype

I’ve been asked whether this series really pertains to DevOps. The criticism, as I interpret it, is that DevOps is a conceptual framework for creating high-functioning development teams, whereas the concepts and processes outlined here are related to creating a build with specific tools.

The point at hand seems to be that DevOps as it’s practiced by its adherents and evangelists provides a universal template for whatever ails your development organization. In the DevOps utopia, managers, PMs, engineers, and QA all coexist in complete harmony to produce reliably phenomenal software on time and within the budget. Only, that’s not right, engineers and QA are coalesced in this other interpretation. And managers and PMs? Isn’t that a lot of overhead for healthy, DevOps practicing shops? In this construction, DevOps can become whatever you want it to be, or feel like it should be, because it has no grounding in actual processes.

What I’m getting at is that DevOps can’t be some abstract “template” for “building software factories.” There’s no such thing. What there is is a collective knowledge built of decades of trial and error, and everyone who tells you otherwise is trying to sell you something that you probably don’t need. Sure, containers are great. Kubernetes is great. CI/CD software is probably pretty solid. But an uncomfortable truth is that you can get every bit as good of results with Make, Cron, and Bash because the important thing in writing software is the knowledge of how it all fits together.

And that’s what this series is about: taking common tools and sharing our experience with them and what works well for us, and what doesn’t. You can derive some pretty generalizable truths from this process, like “You need a sane dependency-management process. Here’s what one looks like. There are other valid processes. The important thing is to have one.”

Unmarshalling generic properties with custom logic in JAXB.

I’m not entirely sure the idea behind this post will ever be useful to anyone, and certain as only a reformed sinner can be that it is not wise. Originally we had a polymorphic JAXB Property type, defined as:

public class PropertyElement<U, T extends PropertyElement<U, T>> extends AbstractElement<T> {

  @XmlAttribute String key;

  @XmlAttribute String name;

  @XmlAnyElement(lax = true)
  private U value;


Where we wanted private U value to be overridable by any subclass, provided that the actual runtime type of U exposed a public, static valueOf method that accepted a string and returned the value represented by the string. We did this because it worked for every Java primitive, plus most of the value types that I could think of. Naturally, this didn’t really work well with anything, so we just created an enumeration of possible types and forsook the notion of dynamically registering new property elements.

Abandoning this was a slightly bitter pill to swallow as it somewhat constrained the extensibility of Sunshower’s core data-model. But meh I tell you, because I spent hours fruitlessly and foolishly attempting to overcome the profound limitations of Java’s erased generics, only for my folly to haunt me for weeks longer. However, should some soul wish to soldier past, here’s what we did:

public class PropertyElement<U, T extends PropertyElement<U, T>> extends AbstractElement<T> {

  @XmlAnyElement(lax = true)
  private U value;

  private static transient Method valueOf;

I can hear some distant screaming. Possibly it’s me. That is verily a reference to a java.lang.reflect.Method It gets worse. The excellent EclipseLink MOXy library which we prefer over Jackson due to, you know, standards, allows us to define a void afterUnmarshal(Unmarshaller u, Object parent) method, which is invoked after MOXy has finished unmarshalling the object at hand. We figured that we didn’t need to write the actual value since, at runtime, its type is known to MOXy and JAX-RS, and sure enough we didn’t. Reading the object, on the other hand…(hangs head in shame):

  protected void doUnmarshal() {
    try {
      if (valueOf != null) {
        this.value = (U) valueOf.invoke(this, ((XMLRoot) value).getObject());
    } catch (Exception ex) {
      if(ex instanceof RuntimeException) { 
          throw ex;
      throw new RuntimeException(ex);

And we detected the actual type of the argument by introspecting the current type:

  public PropertyElement() {
    this.valueOf = configure(getClass());

 protected static Method configure(Class<?> type) {
    if (valueOf != null) {
      return valueOf;
    final ParameterizedType superclass = (ParameterizedType) type.getGenericSuperclass();
    final Object vt = superclass.getActualTypeArguments()[0];

    if (vt.getClass().equals(Class.class)) {
      Class<?> valueType = (Class<?>) vt;
      return Stream.of(valueType.getDeclaredMethods())
          .filter(t -> t.getName().equals("valueOf") && t.getParameterCount() == 1)
              t -> {
                final Class<?>[] ptypes = t.getParameterTypes();
                final Class<?> ptype = ptypes[0];
                return String.class.equals(ptype) || Object.class.equals(ptype);
              () ->
                  new IllegalArgumentException(
                      "Type does not supply a public, static method valueOf() accepting a string"));
    return null;

In a way, I’m sort of proud of this monster. Dr. Frankenstein would probably understand. On the other hand, our sins surely caught us, prompting a reasonable refactor to exorcise this demon.

Fix your indexes, homeslice

Database performance is the principal thing; therefore, undonk your indexes.
~Probably Confucius

Working with databases is one of the worst parts of building applications. There’s the setup, then there’s migrations, then there’s security, then there’s testing and on and on and on until you just can’t anymore. But, eventually, someone cobbles together a solution that encompasses at least a few of these things, and then you never touch it. Ever. It is forbidden to you. You will ruin everything in the application if you change even one thing. Everything will die.

But one day, something goes wrong. You’re getting transaction timeouts. Users are complaining that it’s taking 40 minutes to log into your UI (this happened once). Your application can’t handle all the data your users are feeding it.

So, someone profiles it and finds that it’s not your application, it’s the database. I mean, it’s pretty much always the database unless Keith is overusing hashmaps and locks again, but discovering which part is pretty tricky. Is it indexes? Is it disks? Is it the transaction log? Is it locks? Is it all of them? Who knows?

Fixing the code is at least a multi-month proposition that requires some poor sap go back and look at the code that interacts with your database, potentially donking the whole biz for everyone, not just this customer. The people who can actually fix it are generally not the ones who volunteer for that hell, and the people who volunteer will usually just make it worse.

So, you do the sensible thing and buy a bigger database server with faster disks. It helps for a while, then it just falls over again. You’ve got to fix this.

So, you perform an investigation, and discover that you’re using either a database sequence ID, or a UUID, or a hash function (I have seen this), and here’s the choose-your-own-misadventure part.

Database sequence ID

Your data is insertion-ordered, which is good for indexing. Heck, you can even use BRIN indexing, but you’re making a round-trip to the database for every insertion, and there’s a network round-trip and either a lock or a single thread somewhere in there. Usually there’s No escaping that.


Insertions are random. You got page-splits homeskillet. Like, everywhere. And these aren’t cheap. Your average page density is like, 0.0000000001%. Your index is like a million-page phonebook with one number per page. Oh, yeah, and you gotta maintain that B-Tree or whatever on disk because that sucker is way too big to fit into memory. Additional disk hits, baby! No BRINs for you!

Not to mention you’re probably storing the value as a string if my experience with these things is generalizable, which means you’re probably using 36 bytes/ID instead of the 16 actually required. Plus your IDs have a character set that they don’t need, and god help you if that default changes between versions or someone changes it.

Cryptographic hash (content-based addressing)

This one has all the problems of a UUID, plus some. Like, I’ve seen 128-byte (that is truly “byte” because they’re storing it as a string) MD5sum IDs used in large systems.

Ok, ok. So you stored them as a byte array, that only eliminates the size problem.

But here’s the kicker: If you’re using one of these techniques, your implementation will probably be exchanged for another of the problematic ones I just mentioned. Hax.

Use Flake! (Just do it)

At Sunshower, being ourselves the suckers who will eventually have to undonk our database, we decided to get ahead of the problem. The first post we ever did here was about Flake IDs, and now we have a well-tested, high-quality, MIT-licensed implementation that you can just use. If you’re already using UUIDs stored as byte arrays, drop this biz right in.

Step 1: Add our common library to your dependencies

  1. Make sure Nexus Snapshots are enabled by adding https://oss.sonatype.org/content/repositories/snapshots to your Maven repositories

  2. Add io.sunshower.persist:persist-api:<version> (version is currently 1.0.0-SNAPSHOT)–we’ll get a release soon (TM), but this API is totes stable.

  3. Create you a Flake Sequence. I’d recommend 1 per table for very high-scale systems since you can only generate 10,000/sequence/second. Pretty simple:

import io.sunshower.common.Identifier;
import io.sunshower.persist.Identifiers;
import io.sunshower.persist.Sequence;

Sequence<Identifier> sequence = Identifiers.newSequence(true); //'false' would have the sequence API throw an exception if you requested more IDs than you could generate in a given timespan, ~10k/sec/sequence.  Otherwise, the API blocks until the counter resets.

Then, use it however. For instance, to use it with JPA/Hibernate:

public class MyFlakeEntity {
private byte[] id;


In your database schema, just store them as a byte-array. We use bytea in Postgres which incurs 1 additional byte of overhead per row. Meh.

Swank ACLs with Flake IDs

If you really want declarative ACL goodness coupled with delicious DB goodness, and you’re using Spring Security, pull in our service-security library at io.sunshower.service:service-core:<version> (still 1.0.0-SNAPSHOT), then add these to your Spring configuration:

prePostEnabled = true, 
jsr250Enabled = true, 
securedEnabled = true

public class MySecurityConfiguration { // whatever your configuration is here

  public MutableAclService jdbcAclService(
      JdbcTemplate template, LookupStrategy lookupStrategy, AclCache aclCache) {
    return new IdentifierJdbcMutableAclService(template, lookupStrategy, aclCache, "<SCHEMA>");

  public AclCache aclCache(
      @Named("caches:spring:acl") Cache cache, // replace with your own cache.  This can just be a concurrent hashmap implementation.  We like Ignite
      PermissionGrantingStrategy permissionGrantingStrategy,
      AclAuthorizationStrategy aclAuthorizationStrategy) {
    return new SpringCacheBasedAclCache(
        cache, permissionGrantingStrategy, aclAuthorizationStrategy);

  public LookupStrategy aclLookupStrategy(
      DataSource dataSource,
      AclCache aclCache,
      AclAuthorizationStrategy aclAuthorizationStrategy,
      PermissionGrantingStrategy permissionGrantingStrategy) {
    return new IdentifierEnabledLookupStrategy(
        "<SCHEMA>", dataSource, aclCache, aclAuthorizationStrategy, permissionGrantingStrategy);

  public AclAuthorizationStrategy aclAuthorizationStrategy(GrantedAuthority role) {
    return new MultitenantedAclAuthorizationStrategy(role);

  public PermissionGrantingStrategy permissionGrantingStrategy(AuditLogger logger) {
    return new DefaultPermissionGrantingStrategy(logger);

Then, drop this schema into your migrations:

  id        BYTEA        NOT NULL PRIMARY KEY,
  principal BOOLEAN      NOT NULL,
  sid       VARCHAR(100) NOT NULL,
  CONSTRAINT unique_uk_1 UNIQUE (sid, principal)

  class VARCHAR(100) NOT NULL,
  CONSTRAINT unique_uk_2 UNIQUE (class)

CREATE TABLE <SCHEMA>.acl_object_identity (
  id                 BYTEA PRIMARY KEY,
  object_id_class    BYTEA   NOT NULL,
  object_id_identity BYTEA   NOT NULL,
  parent_object      BYTEA,
  owner_sid          BYTEA,
  entries_inheriting BOOLEAN NOT NULL,
  CONSTRAINT unique_uk_3 UNIQUE (object_id_class, object_id_identity),
  CONSTRAINT foreign_fk_1 FOREIGN KEY (parent_object) REFERENCES <SCHEMA>.acl_object_identity (id),
  CONSTRAINT foreign_fk_2 FOREIGN KEY (object_id_class) REFERENCES <SCHEMA>.acl_class (id),
  CONSTRAINT foreign_fk_3 FOREIGN KEY (owner_sid) REFERENCES <SCHEMA>.acl_sid (id)

  id                  BYTEA PRIMARY KEY,
  acl_object_identity BYTEA   NOT NULL,
  ace_order           INT     NOT NULL,
  sid                 BYTEA   NOT NULL,
  mask                INTEGER NOT NULL,
  granting            BOOLEAN NOT NULL,
  audit_success       BOOLEAN NOT NULL,
  audit_failure       BOOLEAN NOT NULL,
  CONSTRAINT unique_uk_4 UNIQUE (acl_object_identity, ace_order),
  CONSTRAINT foreign_fk_4 FOREIGN KEY (acl_object_identity)
  REFERENCES <SCHEMA>.acl_object_identity (id),

And you can totally use Spring Security annotation-driven security! For instance:

  @PreAuthorize("hasPermission(#id, 'io.sunshower.stratosphere.core.vault.model.Secret', 'DELETE')")
  public Secret delete(Identifier id) {
    Secret s = super.delete(id);
    return s;

Also, if you want to use your ACLs in JPQL (or HQL or whatever), we’ve mapped your entities for you. Pull in io.sunshower.core:core-api:1.0.0-SNAPSHOT and you’ll find the following classes:

  1. io.sunshower.model.core.auth.ObjectIdentity
  2. io.sunshower.model.core.auth.SecuredObject
  3. io.sunshower.model.core.auth.SecurityIdentity

If you need multitenancy and security groups (RBAC), that’s a topic for another post, but we have that, too.

To grant a set of permissions:

  public <T extends Persistable> void grantWithCurrentSession(
      Class<T> type, T instance, Permission... permissions) {
    final ObjectIdentity oid = new ObjectIdentityImpl(type, instance.getId());
    Sid sid = new PrincipalSid(session.getUsername());
    MutableAcl acl;
    try {
      acl = (MutableAcl) aclService.readAclById(oid);
    } catch (NotFoundException ex) {
      acl = ((MutableAclService) aclService).createAcl(oid);
    for (Permission permission : permissions) {
      acl.insertAce(acl.getEntries().size(), permission, sid, true);
    ((MutableAclService) aclService).updateAcl(acl);

To query all of the objects belonging to a user:

select e from Entity e
join e.identity oid
where oid.owner.username = :username;

Which simply requires the mapping:

  @OneToOne(fetch = FetchType.LAZY)
  @JoinColumn(name = "id", insertable = false, updatable = false)
  private ObjectIdentity identity;

  public ObjectIdentity getIdentity() {
    return identity;

In summary, you don’t have to choose between cool features and robust ACL/RBAC support and database performance with Sunshower. We’re happy to do some of that heavy lifting–and it’s all free!

Aurelia @containerless and custom events

One thing that just got me about [Aurelia’s] @customElement used in conjunction with @containerless is that it doesn’t propagate events.

For instance, I had tried to use:

export class TagPanel {
     el: HTMLElement;
     dispatch() : void {
        let e = createEvent('saved', this.property);

With view markup (Pug)

   button(ref="el", click.delegate="dispatch()") 

But I noticed that the event wasn’t propagating. It turns out that the correct thing to do in this case is to inject the DOM element into your view-model, and then dispatch the event from there:

export class TagPanel {

     constructor(private el: Element) {

     dispatch() : void {
        let e = createEvent('saved', this.property);

And don’t reference el from your view:


DevOps without the Devops part 2: The Structure

I’m back! We’ve decided to go head with Sunshower full time, so expect updates much more regularly here!

Last time, we looked at getting a simple build dockerized. Using the Go platform made this pretty simple for us from a dependency perspective, but a lot of you are using a dependency resolution tool like Gradle, Maven, Crate, or Ivy. This post will detail how to configure Maven and Gradle so that your dependencies are manageable and consolidated–a necessary prerequisite for any sane build/release process.

The Base Project

Sunshower.io has quite a few individual projects, each of which having at least several sub-projects. The first project that we need to build is sunshower-devops. This project contains

  1. Docker container definitions
  2. Bill-of-material POMs that are used by each of the sunshower.io projects
  3. Various scripts bundled with our Docker images.

Recall that last time, the first thing I recommended was for you to aggregate all of your dependencies. We needed that information because it allows us to build a bill-of-materials for our project. This is important because it allows us to understand clearly what our project pulls in. This in turn enables us to manage our dependencies in a revisionable and deterministic fashion. Let’s look at what one of our bills-of-material POMs looks like:

<project xmlns="http://maven.apache.org/POM/4.0.0"



  <name>Sunshower Persistence Dependencies</name>


... other properties

... Other dependencies


Basically, this is just a standard Maven POM file with a structure that is convenient for declaring dependencies. The first thing to note is that dependencies are declared within a <dependencyManagement> tag. This means that POM files that inherit from this, or import it, will not automatically depend on the dependencies declared within, only that if they explicitly declare a dependency from this POM, they will inherit its configuration as it appears in this declaration. For instance, if I import sunshower-env:persist-dependencies, then if I declare org.hibernate:hibernate-entitymanager in my importing POM, I will get the ${hibernate.version} version declared in persist-dependencies without having to redeclare it.

Basically, what we’re going for is this:

  1. We create bill-of-material (BOM) POMs for each category of dependency. This is optional, but I like it because these suckers can get huge otherwise.

  2. If we have commonality between our BOM POMs (and we will), we pull it up into an aggregator POM.

  3. We import each of our BOM POMs into our parent pom (sunshower-parent)

  4. Every subproject in our system will have its own BOM POM that derives from sunshower-parent

  5. Each gradle file for each project uses the spring-maven-gradle plugin to import its BOM pom
  6. Viola! If we add a dependency, that addition is recorded in Git. We can see exactly what we’re pulling in for any release (and go back to a previous POM if we need to)



Now, say I want to use hibernate-entitymanager in sunshower-base:persist:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">


    <name>Sunshower.io Imported Bill-Of-Materials</name>




Then, I simply import that into my build.gradle file for whichever project depends on hibernate-entitymanager

    dependencyManagement {
        imports {

Now, in that project (or any subproject thereof), I can just add hibernate-entitymanager to the dependencies block:

dependencies {
    implementation 'org.hibernate:hibernate-entitymanager'


While this may seem like overkill, I like it because it scales quite well. It’s easy to audit (assuming you enforce the process), maintainable (dependencies are grouped together sensibly), and forces you to think about what you’re bringing in. Sometimes incompatibilities can be prevented simply by looking through the dependency lists and determining whether two versions are compatible. Finally, it gives a consistent view of the world to everyone in the project: if everyone contributing to the project follows the rules, you won’t get one component consuming one version of a dependency, and another component consuming another, which is a common source of bad builds IME.

DevOps without the DevOps part 1: The Plan

Forget DevOps

The point of this blog series is to cut through the hype and jargon and get to the pith of DevOps, and there is no better way to do that than with concrete examples.

The Problem

Before you go looking for solutions, it’s helpful to understand what the problem is. And what the problem is that, somewhere in the deep past, your foundational engineers started writing software. They got a requirement, and they:

  • Selected a language
  • Selected a build tool
  • Selected a revision control system
  • Started writing code

That’s pretty much it. Life was good–the software probably did its job (which is why you’re in business, right?) Then, requirements got added, those engineers left and new ones joined, and they:

  • Wrote new functionality (frequently in other languages)
    — Added new build tools and processes and artifacts
  • Wrote a bunch of helper scripts at 1:00 in the morning on a customer call that got integrated into the product
  • Added support for new platforms and operating systems
  • Wrote a bunch of QA and automation scripts
  • etc.

And now it’s no longer an easy task to even build the project. In fact, if you’re doing anything Enterprisey (TM), I would bet good money that fewer than 10% of your team understands how to build and package your product, much less run your automated system tests, etc. And I promise you that no one making purchasing decisions for DevOps solutions understands what your process is. So, let’s figure out how to figure out what your problem is.


A typical pipeline addresses the following steps (current in bold)

  • Build
  • Package
  • Release

Now, your process can vary quite a bit. This series isn’t intended to teach or advocate a particular methodology, and it’s not prescriptive. There are frequently many sub-steps to each of these steps.

Step 1: Understand your Project’s Build Structure

You’re going to have to go beyond “well, we use Java and Lua and Make and Gradle.” You’re going to want to start by enumerating every single component of your project. Every last one. This will create a set of projects that we will call P (for projects). Draw them out, but give yourself plenty of space.

Here’s mine for Sunshower components:

Now, yours doesn’t have to look like this. You could get away with just creating a list, but I’d prefer that you drew it out: it’s a good exercise, it provides a lot of hints as to how things fit together. Now, understand that this is a pretty small project with just a few dependencies. Your project could take weeks to diagram out correctly and comprehensively–that’s ok. You’ll save time and money in the long run. Draw it out, but don’t get hung up on anything like UML or BPML–just try to visually understand the pieces of your project.

Information that should be present

  • All of the tools on all of the platforms you need to support
    — For instance, if you need to build on Windows Server 2012 and Ubuntu 14.04, and your build-tool is Make, you’ll need (for instance)
    — MinGW on Windows and devtools on Ubuntu. I would just draw them out as separate tool boxes on your diagram. It’ll help
  • Source information for your dependencies
    — You may have to build behind a firewall. Make sure that your description of build environments includes what needs to be accessed and how
  • If you’re using managed dependencies (via Ivy, Maven, Gradle, or Go), don’t necessarily worry about enumerating them here. It’ll clutter things up
    — Unless: you need access to a private source for your dependencies (e.g. you’ve purchased a subscription to a library that requires authenticated access). In which case:
    — Be sure to indicate somewhere on here how to access private sources (e.g. settings.xml for Maven)
  • Specific versions for tools. This is really important. Don’t assume that different versions of tooling will produce the same results.

Some words to the wise:

  • Do not buy and products or services at this point. You’ll get something that you’ll never implement correctly or completely.
  • Get one of your engineers to do this. If you don’t have one who can, be absolutely certain that a consultant can build one out for you
  • This isn’t the whole story by any stretch of the imagination. In subsequent posts, we’ll cover how to ensure that everything gets built
  • Be sure that this is correct. Try it out for yourself and verify that you can at least build everything (don’t worry about getting it running or packaged or installed at this point)
  • Institute a process wherein any changes to the project are reflected in this diagram. If this diagram is changing too frequently, you need to understand why. Too much churn on this diagram past a few weeks into the start of a project is a serious warning sign.

Step 2: Create a Template for Building your Project

This is step zero in terms of automation IMO. Sure, build tools like Gradle “automate” builds in that they can be used for many steps described in this series, but we’re only considering the “compile my component and generate some artifacts” portions of their capabilities. You will almost certainly be using your build tools to automate other tasks, but don’t worry about that now. It’s going to be a lot harder to implement a good automated workflow if you don’t clearly distinguish between types of tasks and how you (you dear reader, and your org) perform them.

The goal of this step is to create a template for getting properly configured build environments. This means that

  1. They will have all of the tools enumerated in your tool boxes in your diagrams
  2. They will have the correct versions of those tools
  3. They will have access to the resources (connectivity to Github, credentials to Maven repositories, etc.) required to build your project

Let’s look at some tools that can help us out here.

There are quite a few more, and each of these tools can do more than allow you to create build templates, but we’re only concerned with creating build templates at this stage. These can all be made to work with subsequent stages in your process, and some of them can use others to create templates (e.g Vagrant can generate OVAs). This series will only cover Docker, contact me or leave a comment if you want me to show you how to do the same thing in any of the others.

Example 1: Translating A Tools Box into a Docker Container

Let’s start with the tools box for Updraft:

Before we get started, there’s a bit of a chicken-and-egg situation we need to discuss here. It might seem natural to:

  1. Install Go in our container
  2. Install Git in our container
  3. Checkout Updraft into our container
  4. Build it, etc.

But this introduces a bit of a problem in that there’s not an easy, clean way to check out and build a specific version of Updraft. So, instead, what we’ll do is we’ll require the actual build machines (VM images or bare metal) to have

  1. Docker installed
  2. Git installed

Then, we’ll check out the desired version of the project from Github, build the container using Docker, then build Updraft inside the container. This allows us to easily build different versions of Updraft reliably and reproducibly. I promise that you won’t need much more than this on any actual instance of a Docker host.

So, having installed Docker and Git, let’s add our Dockerfile. A lot of people like to keep their Dockerfiles at the root of their projects, but I like to keep mine in a “docker” subdirectory in case I need more than one:

| etc.
│   │   └── error_code.go
│   └── utils
│   ├── uuid.go
│   └── uuid_test.go
├── docker
│   ├── Dockerfile
│   └── Dockerfile.windows

├── front
│   └── parser
│   ├── abstract_parser.go
│   ├── base_source.go
│   ├── base_source_test.go
│   ├── parser_message_listener.go
│   └── parser_test.go
| etc.

Note It’ll probably take a few tries to get your Dockerfile right. You can build and run the container in one (chained) command thuslike:

docker build -t "updraft" -f docker/Dockerfile . && docker run -it --rm --name updraft updraft 

Or, if you like, two commands:

docker build -t "updraft" -f docker/Dockerfile . 
docker run -it --rm --name updraft updraft 

Each run will re-create the container in the new state and drop you into a bash shell inside the container, where you can execute more commands until you get it right. Remember to add each command necessary to build your project to your Dockerfile as a RUN statement (so that it becomes a layer in your container).


FROM golang:1.8 ## Super easy Go base image (first dependency in our tools box)
WORKDIR /go/src/updraft ## current working directory inside the container
COPY . .  ## Copy the contents of the current directory into WORKDIR
RUN go get ./... ## Get updraft's dependencies (dependencies in our dependencies box)
RUN go build -o out/ucc ucc/main.go  ## Build updraft and output it into $WORKDIR/out

And viola! Running that all together produces:

➜  updraft git:(master) ✗ docker build -t "updraft" -f docker/Dockerfile . && docker run -it --rm --name updraft updraft
Sending build context to Docker daemon  33.59MB
Step 1/5 : FROM golang:1.8
 ---> 0e070ede84f7
Step 2/5 : WORKDIR /go/src/updraft
 ---> Using cache
 ---> 4d23d64b3b23
Step 3/5 : COPY . .
 ---> 8a9a904e4971
Removing intermediate container 996472d595ff
Step 4/5 : RUN go get ./...
 ---> Running in 919addd0c758
 ---> 1608bbf6258f
Removing intermediate container 919addd0c758
Step 5/5 : RUN go build -o out/ucc ucc/main.go
 ---> Running in a0c9e2bd2112
 ---> d2d8855e56ae
Removing intermediate container a0c9e2bd2112
Successfully built d2d8855e56ae
Successfully tagged updraft:latest
root@caa586b9157c:/go/src/updraft# ls
README.md  backends  common  docker  front  middle  out  pascal  ucc

root@caa586b9157c:/go/src/updraft# cd out
root@caa586b9157c:/go/src/updraft/out# ./ucc -h

            Updraft is a modern compiler
            framework and collection written in
            pure Go for portability, speed,
            and embeddability

  ucc [flags]
  ucc [command]

Available Commands:
  help        Help about any command
  trace       trace various aspects of configuration and execution

      --config string     config file (default is $HOME/.ucc.yaml)
  -e, --execute string    select the execution model (default "interpreter")
  -f, --file string       file(s) to compile
  -h, --help              help for ucc
  -l, --language string   select the target language (default "pascal")

Use "ucc [command] --help" for more information about a command.

Deploy to Dockerhub

Now that we’ve successfully built our image, let’s publish it so that anyone can use it:

➜  updraft git:(master) ✗ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username (jhaswell): 
Login Succeeded
➜  updraft git:(master) ✗ docker tag updraft sunshower/updraft
➜  updraft git:(master) ✗ docker push sunshower/updraft
The push refers to a repository [docker.io/sunshower/updraft]
34f0b9f7c17e: Pushed 
ae2fc6d3daac: Pushed 
9fabb269895b: Pushed 
233937e534e9: Pushed 
f847715687e8: Mounted from library/golang 
056e99358ddc: Mounted from library/golang 
237b592486ad: Mounted from library/golang 
86ca4e4dcab9: Mounted from library/golang 
44b57351135e: Mounted from library/golang 
00b029f9aa09: Mounted from library/golang 
18f9b4e2e1bc: Mounted from library/golang 
latest: digest: sha256:089e2aa54db28ce35428e8a37cb46f51461def857d812aa14f1060030be384ba size: 2635
➜  updraft git:(master) ✗ 

And that’s it! Next time we’ll look at how to automate this process so that it happens with every checkin (introducing new tools) and how to make your artifacts available to everyone via the web and other means.

Introducing Anvil

Go is a good language (link to my thoughts impending). I like its philosophy, which is that software is a complex enough without your tools introducing more complexity. In my 10 or so years of writing software professionally (sometimes software with demanding concurrency and performance requirements), I’ve found that you can get away with a good resizable array and hashtable implementation for 99.9% of the problems out there. If you have some nice syntactic jujubees over the array, you get your stacks and your queues for free, which is what Go opted for.

Another thing that I mostly like about Go’s hashtable (map) implementation is that it’s restrictive as to what you can use for keys. I can’t tell you how many bugs I’ve seen introduced in Java projects from someone implementing hashCode and equals methods incorrectly, or forgetting to implement one of them when they do the other. In fact, hashing is such a misunderstood topic that I have no doubt that omitting hash-based structures from the C++ standard until C++11 was in part due to that. In summary, Go made the right choice in restricting their standard container library and building it into the language–mostly.


Online sorting

It’s a bit baffling that there is no sorted map or set. The online sorting provided by a good red-black tree implementation is frequently useful, especially when you’re creating networked middleware or (in my case), online algorithms for rewriting trees and graphs.

Iteration and Access

The idiom for iteration in go is the range operator. The only arguments that the range operator accepts are native Go types (chan, map, array), so the language does not help you in creating abstractions for something fundamental like iteration. While this is by design, it can in many cases hinder good software design. Maybe there’s a super-hot portion of an algorithm wherein iteration over the default implementation of a map just isn’t cutting it (hash tables require that you iterate over their entire backing store in the worst case, which can be substantially larger than the number of items they contain). Whatever your case may be, when you encounter a limitation in the core Go container, you must

  • Implement a new container that meets your requirement
  • Adapt the algorithm to suit your needs

I give you Anvil

Oddly, I do not notice any widely-used or maintained collections collections for Go. For Updraft, I had already had to implement a red-black tree. I then had a situation wherein an Updraft user could legitimately need different hashing characteristics. Finally, not being able to use an IR node as a key turned out to be a pain. So, I decided to implement my own collections framework with the intent that it should:

  • Be simple and similar to existing collections frameworks
  • Be Correct
  • Be Idiomatic
  • Be Consistent
  • Be Literate
  • Get the small things like serialization right

So far, the interfaces are still in flux and in a state of partial implementation, but I have implementations for a sorted map (RB tree) and an unordered map (linear probing hash table) that can be used as follows:

Map Creation

Unordered linear-probing hashmap

    hash := anvil.

Sorted map using red-black balancing

    rbtree := anvil.Maps().Sorted(anvil.Trees.RedBlack).OrderBy(cmp)


Stateful iterator (very fast, available on all collections)

    for iter := map.Iterator(); iter.HasNext(); {
        value := iter.Next()

Range iterator (slower than stateful iterator, but fast enough for most purposes, available on all collections)

    for value := range m.Range() {

The collections are inspired by the JCF and .Net collections classes, but have a distinct Go flavor.


One of these days when I’m not getting married and starting a new job, I’ll post full benchmarks of everything relative to builtin types. My preliminary impression is that they are competitive with the corresponding default implementations (I was actually pleasantly surprised that my linear probing hashmap is virtually indistinguishable in terms of insertion and lookup performance from the default implementation for certain keys (strings) and load-factors (.85)). As the collections mature, they’ll get even faster.

HP Z-Book G4 Review

I purchased an HP Z-Book G3 17 last year–Xeon processor, 64 GB of RAM, it is a true beast. Unfortunately, despite its power, it was plagued with problems like:

  • Not working with its docking station
  • Requiring a new, awful type of docking station
  • Ultra flaky drivers

Docking Station

Let’s start here because nothing quite makes me want to go Office Space on a device like repeated failures to sleep and wake up properly. I mean, to begin with, the laptop ships with Thunderbolt security enabled, which results in the computer not detecting its own docking station by default.

Sure, it’s a trip to the BIOS to fix, but really? I mean, when you’re sitting around repeatedly installing drivers and unplugging/plugging in the docking station waiting for that sweet, sweet “beep boop” chime of detection and reading through troubleshooting guides that mention this in exactly zero places, it’s hard not to feel bitter about expensive office hardware that was specifically designed to work together not working together.

But that’s not the worst of it. I guess I’m feeling my age or something, but the old docking stations worked fine. I’m, at this moment, gazing wistfully at the old HP Advanced Docking station with its 6 USB ports, SATA, etc. etc. ports, and the fact that it Just Worked asking myself “why?” To be fair, the new Thunderbolt has 4 USB ports and 2 Display Ports requiring you to only purchase 2 HDMI/DP adapters if you want to use it with 96% of the monitors available, but that presupposes that 1.) you got it to work and 2.) it continues to work. Which it mostly doesn’t.

Finally, you’d think that the Thunderbolt 3 Docking station connector would be power + garden-variety Thunderbolt cable. It’s somehow not, and it’s about 18 inches long and pretty damn inflexible, so you get wiring scenarios like this:


I’ve tried most variations and have reached the conclusion that HP did this specifically to hurt me personally.


My morning ritual is this:


And yes, all the drivers and firmware are up-to-date, I’ve contacted support, etc. etc.

So what I did was

Buy a G4. The experience with the G3 was so cartoonishly bad that I thought to myself “there is no way that they will do this again. They will fix it and then I will have a functioning and powerful laptop.” Not only was this incorrect, it was horribly incorrect. How could I be so wrong?