One thing that just got me about [Aurelia’s] @customElement used in conjunction with @containerless is that it doesn’t propagate events.

For instance, I had tried to use:

//view-model
@containerless
@customElement('tag-panel')
export class TagPanel {
     el: HTMLElement;
     //etc
     dispatch() : void {
        let e = createEvent('saved', this.property);
        this.el.dispatchEvent(e);
     }
}

With view markup (Pug)

template
   button(ref="el", click.delegate="dispatch()") 

But I noticed that the event wasn’t propagating. It turns out that the correct thing to do in this case is to inject the DOM element into your view-model, and then dispatch the event from there:

//view-model
@containerless
@inject(Element)
@customElement('tag-panel')
export class TagPanel {

     constructor(private el: Element) {
     }

     dispatch() : void {
        let e = createEvent('saved', this.property);
        this.el.dispatchEvent(e);
     }
}

And don’t reference el from your view:

template
   button(click.delegate="dispatch()") 

I’m back! We’ve decided to go head with Sunshower full time, so expect updates much more regularly here!

Last time, we looked at getting a simple build dockerized. Using the Go platform made this pretty simple for us from a dependency perspective, but a lot of you are using a dependency resolution tool like Gradle, Maven, Crate, or Ivy. This post will detail how to configure Maven and Gradle so that your dependencies are manageable and consolidated–a necessary prerequisite for any sane build/release process.

The Base Project

Sunshower.io has quite a few individual projects, each of which having at least several sub-projects. The first project that we need to build is sunshower-devops. This project contains

1. Docker container definitions
2. Bill-of-material POMs that are used by each of the sunshower.io projects
3. Various scripts bundled with our Docker images.

Recall that last time, the first thing I recommended was for you to aggregate all of your dependencies. We needed that information because it allows us to build a bill-of-materials for our project. This is important because it allows us to understand clearly what our project pulls in. This in turn enables us to manage our dependencies in a revisionable and deterministic fashion. Let’s look at what one of our bills-of-material POMs looks like:

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                      http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>io.sunshower.env</groupId>
  <artifactId>persist-dependencies</artifactId>
  <version>1.0.0-SNAPSHOT</version>
  <packaging>pom</packaging>

  <parent>
    <groupId>io.sunshower.env</groupId>
    <artifactId>env-aggregator</artifactId>
    <relativePath>../pom.xml</relativePath>
    <version>1.0.0-SNAPSHOT</version>
  </parent>

  <name>Sunshower Persistence Dependencies</name>

  <scm>
    <url>https://github.com/sunshower-io/sunshower.io</url>
  </scm>

  <properties>
    <hibernate.version>5.1.10.Final</hibernate.version>
... other properties
  </properties>


  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.hibernate</groupId>
        <artifactId>hibernate-entitymanager</artifactId>
        <version>${hibernate.version}</version>
      </dependency>
... Other dependencies
  </dependencyManagement>


</project>

Basically, this is just a standard Maven POM file with a structure that is convenient for declaring dependencies. The first thing to note is that dependencies are declared within a <dependencyManagement> tag. This means that POM files that inherit from this, or import it, will not automatically depend on the dependencies declared within, only that if they explicitly declare a dependency from this POM, they will inherit its configuration as it appears in this declaration. For instance, if I import sunshower-env:persist-dependencies, then if I declare org.hibernate:hibernate-entitymanager in my importing POM, I will get the ${hibernate.version} version declared in persist-dependencies without having to redeclare it.

Basically, what we’re going for is this:

1. We create bill-of-material (BOM) POMs for each category of dependency. This is optional, but I like it because these suckers can get huge otherwise.

2. If we have commonality between our BOM POMs (and we will), we pull it up into an aggregator POM.

3. We import each of our BOM POMs into our parent pom (sunshower-parent)

4. Every subproject in our system will have its own BOM POM that derives from sunshower-parent

5. Each gradle file for each project uses the spring-maven-gradle plugin to import its BOM pom
6. Viola! If we add a dependency, that addition is recorded in Git. We can see exactly what we’re pulling in for any release (and go back to a previous POM if we need to)

Visually:

Now, say I want to use hibernate-entitymanager in sunshower-base:persist:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>io.sunshower.base</groupId>
        <artifactId>bom</artifactId>
        <version>1.0.0-SNAPSHOT</version>
        <relativePath>../</relativePath>
    </parent>
    <groupId>io.sunshower.base</groupId>
    <artifactId>bom-imported</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>pom</packaging>

    <name>Sunshower.io Imported Bill-Of-Materials</name>
    <url>http://www.sunshower.io</url>

    <properties>
        <env.version>1.0.0-SNAPSHOT</env.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.hibernate</groupId>
            <artifactId>hibernate-entitymanager</artifactId>
        </dependency>
    </dependencies>

</project>

Then, I simply import that into my build.gradle file for whichever project depends on hibernate-entitymanager

    dependencyManagement {
        imports {
            mavenBom("io.sunshower.base:bom-imported:${version}")
        }
    }

Now, in that project (or any subproject thereof), I can just add hibernate-entitymanager to the dependencies block:

dependencies {
    implementation 'org.hibernate:hibernate-entitymanager'
}

Conclusion

While this may seem like overkill, I like it because it scales quite well. It’s easy to audit (assuming you enforce the process), maintainable (dependencies are grouped together sensibly), and forces you to think about what you’re bringing in. Sometimes incompatibilities can be prevented simply by looking through the dependency lists and determining whether two versions are compatible. Finally, it gives a consistent view of the world to everyone in the project: if everyone contributing to the project follows the rules, you won’t get one component consuming one version of a dependency, and another component consuming another, which is a common source of bad builds IME.

Forget DevOps

The point of this blog series is to cut through the hype and jargon and get to the pith of DevOps, and there is no better way to do that than with concrete examples.

The Problem

Before you go looking for solutions, it’s helpful to understand what the problem is. And what the problem is that, somewhere in the deep past, your foundational engineers started writing software. They got a requirement, and they:

• Selected a language
• Selected a build tool
• Selected a revision control system
• Started writing code

That’s pretty much it. Life was good–the software probably did its job (which is why you’re in business, right?) Then, requirements got added, those engineers left and new ones joined, and they:

• Wrote new functionality (frequently in other languages)
  — Added new build tools and processes and artifacts
• Wrote a bunch of helper scripts at 1:00 in the morning on a customer call that got integrated into the product
• Added support for new platforms and operating systems
• Wrote a bunch of QA and automation scripts
• etc.

And now it’s no longer an easy task to even build the project. In fact, if you’re doing anything Enterprisey (TM), I would bet good money that fewer than 10% of your team understands how to build and package your product, much less run your automated system tests, etc. And I promise you that no one making purchasing decisions for DevOps solutions understands what your process is. So, let’s figure out how to figure out what your problem is.

Steps:

A typical pipeline addresses the following steps (current in bold)

• Build
• Package
• Release

Now, your process can vary quite a bit. This series isn’t intended to teach or advocate a particular methodology, and it’s not prescriptive. There are frequently many sub-steps to each of these steps.

Step 1: Understand your Project’s Build Structure

You’re going to have to go beyond “well, we use Java and Lua and Make and Gradle.” You’re going to want to start by enumerating every single component of your project. Every last one. This will create a set of projects that we will call P (for projects). Draw them out, but give yourself plenty of space.

Here’s mine for Sunshower components:

Now, yours doesn’t have to look like this. You could get away with just creating a list, but I’d prefer that you drew it out: it’s a good exercise, it provides a lot of hints as to how things fit together. Now, understand that this is a pretty small project with just a few dependencies. Your project could take weeks to diagram out correctly and comprehensively–that’s ok. You’ll save time and money in the long run. Draw it out, but don’t get hung up on anything like UML or BPML–just try to visually understand the pieces of your project.

Information that should be present

• All of the tools on all of the platforms you need to support
  — For instance, if you need to build on Windows Server 2012 and Ubuntu 14.04, and your build-tool is Make, you’ll need (for instance)
  — MinGW on Windows and devtools on Ubuntu. I would just draw them out as separate tool boxes on your diagram. It’ll help
• Source information for your dependencies
  — You may have to build behind a firewall. Make sure that your description of build environments includes what needs to be accessed and how
• If you’re using managed dependencies (via Ivy, Maven, Gradle, or Go), don’t necessarily worry about enumerating them here. It’ll clutter things up
  — Unless: you need access to a private source for your dependencies (e.g. you’ve purchased a subscription to a library that requires authenticated access). In which case:
  — Be sure to indicate somewhere on here how to access private sources (e.g. settings.xml for Maven)
• Specific versions for tools. This is really important. Don’t assume that different versions of tooling will produce the same results.

Some words to the wise:

• Do not buy and products or services at this point. You’ll get something that you’ll never implement correctly or completely.
• Get one of your engineers to do this. If you don’t have one who can, be absolutely certain that a consultant can build one out for you
• This isn’t the whole story by any stretch of the imagination. In subsequent posts, we’ll cover how to ensure that everything gets built
• Be sure that this is correct. Try it out for yourself and verify that you can at least build everything (don’t worry about getting it running or packaged or installed at this point)
• Institute a process wherein any changes to the project are reflected in this diagram. If this diagram is changing too frequently, you need to understand why. Too much churn on this diagram past a few weeks into the start of a project is a serious warning sign.

Step 2: Create a Template for Building your Project

This is step zero in terms of automation IMO. Sure, build tools like Gradle “automate” builds in that they can be used for many steps described in this series, but we’re only considering the “compile my component and generate some artifacts” portions of their capabilities. You will almost certainly be using your build tools to automate other tasks, but don’t worry about that now. It’s going to be a lot harder to implement a good automated workflow if you don’t clearly distinguish between types of tasks and how you (you dear reader, and your org) perform them.

The goal of this step is to create a template for getting properly configured build environments. This means that

1. They will have all of the tools enumerated in your tool boxes in your diagrams
2. They will have the correct versions of those tools
3. They will have access to the resources (connectivity to Github, credentials to Maven repositories, etc.) required to build your project

Let’s look at some tools that can help us out here.

• Ansible
• Chef
• Puppet
• Salt
• VMs (Open Virtualization Format–OVA): VMWare, VirtualBox, etc.
• Docker
• Vagrant
• …etc. There are tons.

There are quite a few more, and each of these tools can do more than allow you to create build templates, but we’re only concerned with creating build templates at this stage. These can all be made to work with subsequent stages in your process, and some of them can use others to create templates (e.g Vagrant can generate OVAs). This series will only cover Docker, contact me or leave a comment if you want me to show you how to do the same thing in any of the others.

Example 1: Translating A Tools Box into a Docker Container

Let’s start with the tools box for Updraft:

Before we get started, there’s a bit of a chicken-and-egg situation we need to discuss here. It might seem natural to:

1. Install Go in our container
2. Install Git in our container
3. Checkout Updraft into our container
4. Build it, etc.

But this introduces a bit of a problem in that there’s not an easy, clean way to check out and build a specific version of Updraft. So, instead, what we’ll do is we’ll require the actual build machines (VM images or bare metal) to have

1. Docker installed
2. Git installed

Then, we’ll check out the desired version of the project from Github, build the container using Docker, then build Updraft inside the container. This allows us to easily build different versions of Updraft reliably and reproducibly. I promise that you won’t need much more than this on any actual instance of a Docker host.

So, having installed Docker and Git, let’s add our Dockerfile. A lot of people like to keep their Dockerfiles at the root of their projects, but I like to keep mine in a “docker” subdirectory in case I need more than one:

| etc.
│   │   └── error_code.go
│   └── utils
│   ├── uuid.go
│   └── uuid_test.go
├── docker
│   ├── Dockerfile
│   └── Dockerfile.windows
├── front
│   └── parser
│   ├── abstract_parser.go
│   ├── base_source.go
│   ├── base_source_test.go
│   ├── parser_message_listener.go
│   └── parser_test.go
| etc.

Note It’ll probably take a few tries to get your Dockerfile right. You can build and run the container in one (chained) command thuslike:

 
docker build -t "updraft" -f docker/Dockerfile . && docker run -it --rm --name updraft updraft 

Or, if you like, two commands:

 
docker build -t "updraft" -f docker/Dockerfile . 
docker run -it --rm --name updraft updraft 

Each run will re-create the container in the new state and drop you into a bash shell inside the container, where you can execute more commands until you get it right. Remember to add each command necessary to build your project to your Dockerfile as a RUN statement (so that it becomes a layer in your container).

updraft/docker/Dockerfile

FROM golang:1.8 ## Super easy Go base image (first dependency in our tools box)
WORKDIR /go/src/updraft ## current working directory inside the container
COPY . .  ## Copy the contents of the current directory into WORKDIR
RUN go get ./... ## Get updraft's dependencies (dependencies in our dependencies box)
RUN go build -o out/ucc ucc/main.go  ## Build updraft and output it into $WORKDIR/out

And viola! Running that all together produces:

➜  updraft git:(master) ✗ docker build -t "updraft" -f docker/Dockerfile . && docker run -it --rm --name updraft updraft
Sending build context to Docker daemon  33.59MB
Step 1/5 : FROM golang:1.8
 ---> 0e070ede84f7
Step 2/5 : WORKDIR /go/src/updraft
 ---> Using cache
 ---> 4d23d64b3b23
Step 3/5 : COPY . .
 ---> 8a9a904e4971
Removing intermediate container 996472d595ff
Step 4/5 : RUN go get ./...
 ---> Running in 919addd0c758
 ---> 1608bbf6258f
Removing intermediate container 919addd0c758
Step 5/5 : RUN go build -o out/ucc ucc/main.go
 ---> Running in a0c9e2bd2112
 ---> d2d8855e56ae
Removing intermediate container a0c9e2bd2112
Successfully built d2d8855e56ae
Successfully tagged updraft:latest
root@caa586b9157c:/go/src/updraft# ls
README.md  backends  common  docker  front  middle  out  pascal  ucc

root@caa586b9157c:/go/src/updraft# cd out
root@caa586b9157c:/go/src/updraft/out# ./ucc -h

            Updraft is a modern compiler
            framework and collection written in
            pure Go for portability, speed,
            and embeddability

Usage:
  ucc [flags]
  ucc [command]

Available Commands:
  help        Help about any command
  trace       trace various aspects of configuration and execution

Flags:
      --config string     config file (default is $HOME/.ucc.yaml)
  -e, --execute string    select the execution model (default "interpreter")
  -f, --file string       file(s) to compile
  -h, --help              help for ucc
  -l, --language string   select the target language (default "pascal")

Use "ucc [command] --help" for more information about a command.
root@caa586b9157c:/go/src/updraft/out# 

Deploy to Dockerhub

Now that we’ve successfully built our image, let’s publish it so that anyone can use it:

➜  updraft git:(master) ✗ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username (jhaswell): 
Password: 
Login Succeeded
➜  updraft git:(master) ✗ docker tag updraft sunshower/updraft
➜  updraft git:(master) ✗ docker push sunshower/updraft
The push refers to a repository [docker.io/sunshower/updraft]
34f0b9f7c17e: Pushed 
ae2fc6d3daac: Pushed 
9fabb269895b: Pushed 
233937e534e9: Pushed 
f847715687e8: Mounted from library/golang 
056e99358ddc: Mounted from library/golang 
237b592486ad: Mounted from library/golang 
86ca4e4dcab9: Mounted from library/golang 
44b57351135e: Mounted from library/golang 
00b029f9aa09: Mounted from library/golang 
18f9b4e2e1bc: Mounted from library/golang 
latest: digest: sha256:089e2aa54db28ce35428e8a37cb46f51461def857d812aa14f1060030be384ba size: 2635
➜  updraft git:(master) ✗ 

And that’s it! Next time we’ll look at how to automate this process so that it happens with every checkin (introducing new tools) and how to make your artifacts available to everyone via the web and other means.

Go is a good language (link to my thoughts impending). I like its philosophy, which is that software is a complex enough without your tools introducing more complexity. In my 10 or so years of writing software professionally (sometimes software with demanding concurrency and performance requirements), I’ve found that you can get away with a good resizable array and hashtable implementation for 99.9% of the problems out there. If you have some nice syntactic jujubees over the array, you get your stacks and your queues for free, which is what Go opted for.

Another thing that I mostly like about Go’s hashtable (map) implementation is that it’s restrictive as to what you can use for keys. I can’t tell you how many bugs I’ve seen introduced in Java projects from someone implementing hashCode and equals methods incorrectly, or forgetting to implement one of them when they do the other. In fact, hashing is such a misunderstood topic that I have no doubt that omitting hash-based structures from the C++ standard until C++11 was in part due to that. In summary, Go made the right choice in restricting their standard container library and building it into the language–mostly.

Limitations

Online sorting

It’s a bit baffling that there is no sorted map or set. The online sorting provided by a good red-black tree implementation is frequently useful, especially when you’re creating networked middleware or (in my case), online algorithms for rewriting trees and graphs.

Iteration and Access

The idiom for iteration in go is the range operator. The only arguments that the range operator accepts are native Go types (chan, map, array), so the language does not help you in creating abstractions for something fundamental like iteration. While this is by design, it can in many cases hinder good software design. Maybe there’s a super-hot portion of an algorithm wherein iteration over the default implementation of a map just isn’t cutting it (hash tables require that you iterate over their entire backing store in the worst case, which can be substantially larger than the number of items they contain). Whatever your case may be, when you encounter a limitation in the core Go container, you must

• Implement a new container that meets your requirement
• Adapt the algorithm to suit your needs

I give you Anvil

Oddly, I do not notice any widely-used or maintained collections collections for Go. For Updraft, I had already had to implement a red-black tree. I then had a situation wherein an Updraft user could legitimately need different hashing characteristics. Finally, not being able to use an IR node as a key turned out to be a pain. So, I decided to implement my own collections framework with the intent that it should:

• Be simple and similar to existing collections frameworks
• Be Correct
• Be Idiomatic
• Be Consistent
• Be Literate
• Get the small things like serialization right

So far, the interfaces are still in flux and in a state of partial implementation, but I have implementations for a sorted map (RB tree) and an unordered map (linear probing hash table) that can be used as follows:

Map Creation

Unordered linear-probing hashmap

    hash := anvil.
        Maps().
        Hashes(anvil.Hashing.LinearProbing).
        DefaultLoadFactor().
        InitialCapacity(10).
        KeyedBy(strings.DefaultHash)

Sorted map using red-black balancing

    rbtree := anvil.Maps().Sorted(anvil.Trees.RedBlack).OrderBy(cmp)

Iteration

Stateful iterator (very fast, available on all collections)

    for iter := map.Iterator(); iter.HasNext(); {
        value := iter.Next()
        //etc
    }

Range iterator (slower than stateful iterator, but fast enough for most purposes, available on all collections)

    for value := range m.Range() {
        //etc
    }

The collections are inspired by the JCF and .Net collections classes, but have a distinct Go flavor.

Performance

One of these days when I’m not getting married and starting a new job, I’ll post full benchmarks of everything relative to builtin types. My preliminary impression is that they are competitive with the corresponding default implementations (I was actually pleasantly surprised that my linear probing hashmap is virtually indistinguishable in terms of insertion and lookup performance from the default implementation for certain keys (strings) and load-factors (.85)). As the collections mature, they’ll get even faster.

Hi folks! Been a while–just started my (paying) job a few weeks ago, and haven’t had much time to work on Sunshower. I took some time last weekend to implement expression parsing for the Pascal frontend and work on the IR (intermediate representation), so we’re still making some good progress.

You can grab the source over at https://github.com/sunshower-io/updraft

Building

It’s simple to build Updraft. It doesn’t have any external dependencies, so just clone that repo and run go build -o ucc main.go. None of the backends are fleshed out or integrated yet, but you can see some cool things about the structure of your programs.

Supported syntax structures

Right now, things are pretty basic (hahaha, nah, they’re Pascal, which is almost as bad). Pascal’s pretty easy to parse, and so it makes for a good straw-man language. It also supports most of the features that our backends and IRs will require, so using it to punch through the entire compiler stack won’t leave us in the lurch with more modern languages. Right now, the Pascal front-end supports:

Comments

{this is a comment}

{this
   is 
another
comment}

Primitive Types

{string}
str := 'string';
{integers (64 bit)}
a := 432123421341232515;

{floating-point}
c := 2.001e-6;

Compound statements

BEGIN
{ statement list }
END.

Assignments

a := 1;
b := a + a;

Arithmetic Expressions

c := 4 * 2 + 16 / 90 + (200 * (300 + 400) + 500);
a := 2 * (c - 16);

But what we’re demoing this week is some cool trace features.

Feature 1: Pluggable front-ends

While only Pascal’s currently supported, if you look into UCC (Updraft Compiler Collection)’s entry-point, you’ll see that, by using Convection, our Go dependency-injection framework, we only load the compiler toolset for a given language:

func (t *TraceConfiguration) TraceCommand(
        cmd *root.RootCommand,
        c compiler.Compiler, // yaaay.  Injected =)
        cfg *root.RootConfiguration,
) {
    //etc.
}

which keeps startup fast and memory-consumption lean. Eventually, we might support dynamically-linkable front-ends, but since Updraft is intended to be embedded, I doubt it’ll become a problem.

Feature 2: Tracing

UCC contains some pretty legit tracing facilities already, including:

Unresolved symbols:

{test.pas} 
{hello world}

BEGIN
        a := 1;
        b := 200;
        c := 1 * 20 + a;
        e := 1;
        d := c + a + e + nothere;


END.

When run with -st (-s -> summary, t -> tokens) (–only available in trace mode)(actual unresolved symbols during compilation will be reported after lexical analysis)

Results in:

ERROR{type:'syntax', token: TOKEN{text: nothere, line: 8, col: 17, type: Token('IDENTIFIER'), value: '%!s(<nil>)'}

                   3 source lines.
                   0 syntax errors.
0 seconds total parsing time

Symbol table dumps

Adding the -y (./ucc trace -e interpreter -f example.pas -l pascal -sty)
flag for sYmbols on the same file results in:

ERROR{type:'syntax', token: TOKEN{text: nothere, line: 8, col: 17, type: Token('IDENTIFIER'), value: '%!s(<nil>)'}
===== SYMBOL TABLE =====
Identifier          Line Numbers
----------          ------------
a                    004 006 008 
b                    005 
c                    006 008 
d                    008 
e                    007 008 
nothere              008 


                   3 source lines.
                   0 syntax errors.
0 seconds total parsing time

Note that nothere still got a symbol-table entry. That’s because some languages allow forward declarations, and so we’ll report it to the lexer, but subsequent phases could opt to ignore them.

IR dumps

The latest cool feature is the ability to dump the entire IR into JSON (or other formats if you choose). Optimization passes will rewrite this structure, and you can dump the IR out after any number of optimization passes you like. The reducer-interpreter backend will support tracing the IR as it is dynamically rewritten during program execution.

The IR of the above program can be shown (in conjunction with everything else) via:
./ucc trace -e interpreter -f example.pas -l pascal -styp --format json

Which results in everything above, plus:

{
 "node":{
    "type":"compound", 
    "depth":1,
    "value":"<nil>",
    "text":"a",
    "value":"%!s(<nil>)",
    "line":4,
    "position":1,
    "children":[
     {
      "node":{
         "type":"Assign", 
         "depth":6,
         "value":"<nil>",
         "text":"a",
         "value":"%!s(<nil>)",
         "line":4,
         "position":1,
         "children":[
          {
           "node":{
              "type":"var", 
              "depth":11,
              "value":"<nil>",
              "text":":=",
              "value":"%!s(<nil>)",
              "line":4,
              "position":3
           }
          },
          {
           "node":{
              "type":"int64", 
              "depth":11,
              "value":"1",
              "text":"",
              "value":"%!s(int64=1)",
              "line":4,
              "position":6
           }
          }
         ]
      }
     },
     {
      "node":{
         "type":"Assign", 
         "depth":6,
         "value":"<nil>",
         "text":"b",
         "value":"%!s(<nil>)",
         "line":5,
         "position":1,
         "children":[
          {
           "node":{
              "type":"var", 
              "depth":11,
              "value":"<nil>",
              "text":":=",
              "value":"%!s(<nil>)",
              "line":5,
              "position":3
           }
          },
          {
           "node":{
              "type":"int64", 
              "depth":11,
              "value":"200",
              "text":"",
              "value":"%!s(int64=200)",
              "line":5,
              "position":6
           }
          }
         ]
      }
     },
     {
      "node":{
         "type":"Assign", 
         "depth":6,
         "value":"<nil>",
         "text":"c",
         "value":"%!s(<nil>)",
         "line":6,
         "position":1,
         "children":[
          {
           "node":{
              "type":"var", 
              "depth":11,
              "value":"<nil>",
              "text":":=",
              "value":"%!s(<nil>)",
              "line":6,
              "position":3
           }
          },
          {
           "node":{
              "type":"+", 
              "depth":11,
              "value":"<nil>",
              "text":"+",
              "value":"%!s(<nil>)",
              "line":6,
              "position":13,
              "children":[
               {
                "node":{
                   "type":"*", 
                   "depth":16,
                   "value":"<nil>",
                   "text":"*",
                   "value":"%!s(<nil>)",
                   "line":6,
                   "position":8,
                   "children":[
                    {
                     "node":{
                        "type":"int64", 
                        "depth":21,
                        "value":"1",
                        "text":"",
                        "value":"%!s(int64=1)",
                        "line":6,
                        "position":6
                     }
                    },
                    {
                     "node":{
                        "type":"int64", 
                        "depth":21,
                        "value":"20",
                        "text":"",
                        "value":"%!s(int64=20)",
                        "line":6,
                        "position":10
                     }
                    }
                   ]
                }
               },
               {
                "node":{
                   "type":"var", 
                   "depth":16,
                   "value":"<nil>",
                   "text":"a",
                   "value":"%!s(<nil>)",
                   "line":6,
                   "position":15
                }
               }
              ]
           }
          }
         ]
      }
     },
     {
      "node":{
         "type":"Assign", 
         "depth":6,
         "value":"<nil>",
         "text":"e",
         "value":"%!s(<nil>)",
         "line":7,
         "position":1,
         "children":[
          {
           "node":{
              "type":"var", 
              "depth":11,
              "value":"<nil>",
              "text":":=",
              "value":"%!s(<nil>)",
              "line":7,
              "position":3
           }
          },
          {
           "node":{
              "type":"int64", 
              "depth":11,
              "value":"1",
              "text":"",
              "value":"%!s(int64=1)",
              "line":7,
              "position":6
           }
          }
         ]
      }
     },
     {
      "node":{
         "type":"Assign", 
         "depth":6,
         "value":"<nil>",
         "text":"d",
         "value":"%!s(<nil>)",
         "line":8,
         "position":1,
         "children":[
          {
           "node":{
              "type":"var", 
              "depth":11,
              "value":"<nil>",
              "text":":=",
              "value":"%!s(<nil>)",
              "line":8,
              "position":3
           }
          },
          {
           "node":{
              "type":"+", 
              "depth":11,
              "value":"<nil>",
              "text":"+",
              "value":"%!s(<nil>)",
              "line":8,
              "position":16,
              "children":[
               {
                "node":{
                   "type":"+", 
                   "depth":16,
                   "value":"<nil>",
                   "text":"+",
                   "value":"%!s(<nil>)",
                   "line":8,
                   "position":12,
                   "children":[
                    {
                     "node":{
                        "type":"+", 
                        "depth":21,
                        "value":"<nil>",
                        "text":"+",
                        "value":"%!s(<nil>)",
                        "line":8,
                        "position":8,
                        "children":[
                         {
                          "node":{
                             "type":"var", 
                             "depth":26,
                             "value":"<nil>",
                             "text":"c",
                             "value":"%!s(<nil>)",
                             "line":8,
                             "position":6
                          }
                         },
                         {
                          "node":{
                             "type":"var", 
                             "depth":26,
                             "value":"<nil>",
                             "text":"a",
                             "value":"%!s(<nil>)",
                             "line":8,
                             "position":10
                          }
                         }
                        ]
                     }
                    },
                    {
                     "node":{
                        "type":"var", 
                        "depth":21,
                        "value":"<nil>",
                        "text":"e",
                        "value":"%!s(<nil>)",
                        "line":8,
                        "position":14
                     }
                    }
                   ]
                }
               },
               {
                "node":{
                   "type":"var", 
                   "depth":16,
                   "value":"<nil>",
                   "text":"nothere",
                   "value":"%!s(<nil>)",
                   "line":8,
                   "position":17
                }
               }
              ]
           }
          }
         ]
      }
     }
    ]
 }
}

One of the cooler features of UCC is the ability to dynamically insert or substitute IR nodes. For instance, an early language I plan on supporting is Groovy, and it would be relatively easy to append additional nodes that convey type information about a given definition even if that type information isn’t available at compile-time. Non-compatible type-reassignments could remove that type-information, while covariant assignments could specialize it further.

I’ve been extending JUnit to provide project-specific testing infrastructure for quite some time, and have always been disappointed with the extension mechanisms prior to JUnit 5.

To begin with, JUnit 4’s primary extension mechanism was the actual test runner. If you wanted to mixin multiple extensions, you were in for a hassle. For one test-infrastructure-project, I would have the runner dynamically subclass the test case and add behaviors based on the infrastructure-specific annotations that were present. This worked really well, except that most IDEs I tried would not run the tests correctly at the method level, so you’d be left re-running the entire class.

On another project, I jumped through all manner of hoops to avoid this. Spring-test alleviated many of the problems with their TestExecutionLisiteners, but not all, and it was less lovely than subclassing in use. The result worked quite well, but it was pretty excruciating to write. Now that we’re rewriting everything from scratch, I’m implementing our testing infrastructure on JUnit 5 and Spring 5, and JUnit 5 makes this awesome!

How’s it work?

JUnit 5 provides a new extension mechanism called, well, an Extension. Extension is a marker-interface with specializations for each portion of the test lifecycle. For instance, the interface BeforeAll will equip you with a beforeAll method which is called with the container extension context, providing, among other things, the test class.

For testing purposes, we need to inject and override properties pretty routinely. Spring 5’s YAML support isn’t there when used with @TestPropertySource, so I thought I’d whip up my own (full source)

Then, to use it (given a resource-structure) suchlike:

Properties Extension Usage

@RunWith(JUnitPlatform.class)
@ExtendWith({
        SpringExtension.class,
        PropertiesExtension.class
})
@Properties({
        "classpath:classes/frap.yml"
})
@ContextConfiguration(classes = PropertiesExtensionConfiguration.class)
public class PropertiesExtensionTest {

    @Inject
    private Environment environment;

    @Value("${classes.properties.frap}")
    String frap;


    @Test
    public void ensureInjectingPropertiesWorks() {
        assertThat(frap, is("lolwat"));

    }

    @Test
    public void ensureLoadingYamlAtClassLevelWorks() {
        String prop = environment.getProperty("classes.properties.frap");
        assertThat(prop, is("lolwat"));
    }

    @Test
    @Properties("classpath:methods/frap.yml")
    public void ensureOverridingOnMethodWorksWithInjection() {
        String prop = environment.getProperty("classes.properties.frap");
        assertThat(prop, is("overridden"));
    }

    @Test
    @Properties("classpath:methods/frap.yml")
    public void ensureOverridingOnMethodWorks() {
        assertThat(frap, is("overridden"));
    }

}

‘Morning folks,

Today’s blog will be a bit out-of-order (there’s a subsequent blog that I haven’t finished that structurally precedes this one), but I hope it won’t throw too much wire into the hay-baler.

Persistence IDs

The problem we are trying to solve is how to identify objects within a system. Most of the time, we think of object identity as having two components:

1. Object type and
2. Some unique name or tag for a given object instance

Selecting the correct persistence ID scheme for your application can be quite a task, and people generally don’t give it much thought at the outset. Later on, when the initial ID scheme isn’t ideal for the application, developers have to perform an ID type migration, which for live systems is frequently one of the nastier modifications to make.

About 80% of the applications that I’ve seen and worked with just use sequential, database-generated integral IDs, and that mostly works for them. There are well documented problems with them, so we’ll avoid them altogether (except under very specific circumstances, which will be discussed on a case-by-case basis). They do have one desirable property that we’ll discuss below, viz. that they’re sorted.

UUIDs

Now that database-generated sequential IDs are out, what about UUIDs? It’s a good question, and I’ve used and seen used UUIDs quite successfully in quite a few applications. But we’re not using them in Sunshower for the following reasons:

1. They’re pretty ugly. I mean, URLs like sunshower.io/orchestrations/123e4567-e89b-12d3-a456-426655440000/deployments/123e4567-e89b-12d3-a456-426655440000 aren’t great. We previously base-58 encoded our UUIDs to produce prettier URLs along the lines of sunshower.io/orchestrations/11W7CuKyzdu7FGXEVQvK/deployments/11W7CuKz27Y9ePpV2ju9. One of the problems that we encountered was that having different string representations of IDs inside and outside our application made debugging a less straightforward than it needed to be.
2. They’re pretty inconsistent across different databases and workloads.
   — For write-intensive workloads, UUIDs as primary keys are a poor choice if you don’t have an auxiliary clustering index (which requires that you maintain 2 indexes per table, at least). Insertions into database pages will happen at random locations, and you’ll incur jillions of unnecessary page-splits in high-volume scenarios. On the other hand, adding the additional clustering index will incur additional overhead to writes.
   — Index scans that don’t include the clustering index can perform poorly because the data are spread out all over the disk.

So, is there a better way?

How about Flake?

Twitter back in the day encountered similar issues with ID selection, so they designed the Snowflake ID scheme. There is a 128-bit extension that minimizes the need for node-coordination, which is desirable in our case (especially since we were willing to tolerate 128-bit IDs for UUIDs). The layout of the ID is as follows:

1. The first 64 bits are a timestamp (for a single-node without modifications to the system clock, monotonically increasing).
2. 48 random bits (usually a MAC address of a network interface, other schemes could be used)
3. A 16-bit monotonically-increasing sequence that is reset every time the system-clock ticks forward. This is important because it places an upper limit on the number of IDs that can safely be generated in a given time-period (65,535/second). My implementation provides back-pressure, but this can cause undesirable behavior (contention) in very high-volume scenarios. To put this in perspective, Twitter accommodates an average of 6,000 Tweets/second, but even this would only consume about 10% of the bandwidth of our ID generation for a single node.

Our implementation

full source

I’m sorry. I’m pretty old-school. I like Spring and JPA (and even EJB!). Things like JPQL and transparent mapping between objects and their storage representations (e.g. tables) are important to me. I also super-like transactions, and I really really like declarative transaction management. Why? Because not having these things places a very high burden on development teams, and in my experience, reduces testability, frequently dramatically. Another requirement is that we be able to easily serialize IDs to a variety of formats, so we’ll make our ID JAXB-enabled. Here’s the important parts of the Identifier class:

//not using @Embeddable because we will create a custom Hibernate type for this--that way we can use the same annotations for everything
@XmlRootElement(name = "id")
@XmlAccessorType(XmlAccessType.NONE)
public class Identifier implements
        Comparable<Identifier>,
        Serializable {

    static final transient Encoding base58 = Base58.getInstance(
            Default
    );


    @XmlAttribute
    @XmlJavaTypeAdapter(Base58ByteArrayConverter.class)
    private byte[] value;

    protected Identifier() {

    }

    Identifier(byte[] value) {
        if(value == null || value.length != 16) {
            throw new IllegalArgumentException(
                    \"Argument cannot possibly be a valid identifier\"
            );
        }
        this.value = value;
    }

// other stuff
}

Now, ideally, we would be able to make value final. If an ID is created from thread A and somehow immediately accessed from thread B, final would guarantee that thread A and thread B would always agree on the value of value. Since neither JAXB nor JPA really work with final fields, we can’t really do that. We could partially fix value‘s publication by marking it volatile, but there are downsides to that as well. The solution that I’m opting for is protecting the creation of Identifiers by forcing the creation of IDs to occur within a sequence (note the protected and package-protected constructors of Identifier):

public interface Sequence<ID extends Serializable> {
    ID next();
}

with a Flake ID sequence (full source: [](Flake ID))

    @Override
    public Identifier next() {
        synchronized (sequenceLock) {
            increment();
            ByteBuffer sequenceBytes =
                    ByteBuffer.allocate(ID_SIZE);
            return new Identifier(
                    sequenceBytes
                            .putLong(currentTime)
                            .put(seed)
                            .putShort((short) sequence).array()
            );
        }
    }

Now, we’re guaranteed that sequences can be shared across threads, and we have several options:

1. Each entity-type could be assigned its own sequence
2. Entities can share sequences

We don’t really care too much about ID collisions across tables, and we can generate a ton of IDs quickly for a given sequence, so we’ll just default to sharing a sequence for entities:

@MappedSuperclass
@XmlDiscriminatorNode(\"@type\")
public class AbstractEntity extends
        SequenceIdentityAssignedEntity<Identifier> {

    static final transient Sequence<Identifier> DEFAULT_SEQUENCE;

    static {
        DEFAULT_SEQUENCE = Identifiers.newSequence(true);
    }

    @Id
    @XmlID
    @XmlJavaTypeAdapter(IdentifierAdapter.class)
    private Identifier id;


    protected AbstractEntity() {
        super(DEFAULT_SEQUENCE);
    }


    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (!(o instanceof AbstractEntity)) return false;
        if (!super.equals(o)) return false;

        AbstractEntity that = (AbstractEntity) o;

        return id != null ? id.equals(that.id) : that.id == null;
    }

    @Override
    public int hashCode() {
        int result = super.hashCode();
        result = 31 * result + (id != null ? id.hashCode() : 0);
        return result;
    }


    @Override
    public String toString() {
        return String.format(\"%s{\" +
                \"id=\" + id +
                '}', getClass());
    }
}

MOXy only allows you to use String @XmlID values, so we need to transform our IDs to strings (hence the @XmlJavaTypeAdapter)

In the next blog post, we’ll demonstrate how to make Identifiers JPA native types!

Hello and welcome to sunshower.io’s official blog! Josiah here excited to tell you about who we are and where we are going. Over the past 16 months, I’ve been working as the CTO of a startup. Unfortunately, my co-founders and I were not able to align our visions for the product and the project, so we decided to part ways. I had hoped that we would be able to reach an agreement to open-source the project, but I doubt that will ever fully materialize. In any case, I think it’s best to start a new project with a clean slate. This will ensure that the project is and always remains free of any legal or conceptual constraints, and I believe that with a more cohesive vision, we will be able to deliver a better product. With that in mind, let’s talk about what it is!

The goals of sunshower.io are to provide a robust, streamlined platform for Provisioning-as-a-Service (PaaS, not to be confused with Platform-as-a-Service) and Configuration-as-a-Service, so let’s talk about what we mean.

Configuration-as-a-Service

I had initially envisioned our project to be peer-to-peer, secure Git. The goal was to be able to commit arbitrary files (including very large binaries, helpful for installers), indicate that some files should be treated as templates (i.e. filtered through various data-sources available to the system, such as our key-value store, system properties, etc.), and then “push” out the repository to collections of systems. Failures could be handled according to several strategies (up to and including reverting the entirety of the push across the cluster), and you could seamlessly revert to a previous commit. Sunshower.io will not attempt to handle as many enterprise cases initially. At least, if we do, we’ll delegate down to the capabilities of the orchestration provider (e.g. Docker Swarm or Kubernetes). One of the primary goals of sunshower is to abstract the process of deployment away from the technologies used for the deployment, and so we’ll have to think carefully about how we go about it: not every orchestration provider is going to tackle the same problems the same ways.

Provisioning-as-a-Service

The other domain that we thought a lot about tackling was provisioning. Provisioning infrastructure elements such as virtual machines and security groups is conceptually the same across all the cloud providers, and so we would like to be able to take the same orchestration and deploy it to different clouds without any changes. Of course, things like our default sizing tiers won’t generally be adequate for many users needs, but there are sufficient similarities between most CSP provider instance-types that translating between them is frequently possible. In the cases where it’s not, overriding at deployment is always a possibility.

Thanks again for taking some time to learn more about us! Expect updates here frequently, and we’ll get this out the door ASAP. You can always check our progress out on Github.