I think everyone says that building a startup is hard, but what they generally fail to convey is how hard it actually is. I left my previous position on March 1st of this year, and since then it seems like it’s been an unpunctuated litany of 14-hour days.  There’s also this notion (that I had) that when you found a company, you’ll mostly be working on projects that are interesting to you. I mean, it is your company, right?  And you do generally get to choose what that entails, right? And that that engagement will be prophylactic against the darkness.

That’s pretty heckin’ far from the truth in every way.  About 10% of the code in Sunshower is what I would consider to be “strongly interesting”–the rest is fairly standard enterprise Java, Go, and Typescript.  In other words, Sunshower is 10% distributed graph reduction machine and 90% infrastructure built to interact and support that. It’s actually about twice as bad as that, since fully 55% of the code is tests and testing infrastructure.  And this is to say nothing of setting up continuous delivery, writing deployment manifests, replacing failed hard drives, configuring VPNs, signing documents, worrying about corporate and legal stuff…the list goes on.

That proportion of fun-work to work-work largely tracks my experience at companies of all sizes, with the primary differences being that the pay is much worse at your own startup, and that nobody’s really skeptical of a software project that is already profitable.  The cocktail of doubt, overwork, and scarcity is strong and bitter, and there’s only so much of it you can drink.

In October I got pretty sick and have only just really recovered, and that really pushed me to a place that the kids probably call “too extra.”  You can tell when you get there because everything’s distorted and whack–speed-bumps become cliffs and the little nuggets of interesting work you sprinkle throughout your day lose their luster. But the worst part is that it’s actually pretty hard to tell when you’re there.  I only realized I wasn’t in a great spot recently, and had troubles seeing a way out.

Lisa, my amazing wife and co-founder, really helped with the insight that gratitude is really what will get you out of dark spots.  Gratitude for family, for the people who throw their lot in with yours, for the opportunity to even try, for health, and even a warm breakfast and a colorful sunrise (fingers crossed).  And so I’m trying that and it’s really working.

 

 

 

 

As a kid, I remember having my mom drop me off at the library so I could rummage through the card catalog. I still have warm-and-fuzzy nostalgic feelings about spending hours pulling books off the shelves, checking the indexes, and finding exactly the pieces of information I needed. It’s a nice memory, but would I trade away the Internet to get another chance at flipping through musty drawers of tiny typewritten cards? No way. You’d have to be bonkers to go backwards and do research like that. I’d rather sit at home in my pajamas and access hundreds of sources online in seconds, thanks very much.

Innovation is a funny thing. We get so accustomed to doing something one way, that it never occurs to us that it could be different. Better. Think about all of the things we used to consider perfectly fine until we figured out a better way:

• Driving to the store to rent a movie (RIP, Blockbuster)
• Placing a personal ad in the newspaper
• Pulling over to make call from a pay phone
• Planning a route using a tri-fold city map
• Sending a fax
• Looking numbers up in the Yellow Pages
• Listening to music on a Walkman
• Checking out what’s on tonight in the TV Guide
• Using a typewriter with a bottle of Wite Out handy
• Checking spelling using a dictionary
• Looking facts up in an encyclopedia
• Keeping business contacts in a Rolodex
• Calling the theater for showtimes
• Making a mixtape

At one point, someone asked: Why isn’t there a better way? Then they changed the game.

That’s what we did for the cloud at Sunshower.io.

People used to think it was okay to be surprised by their cloud bill, or to spend hours, sometimes days, sometimes months, deploying applications. We used to find it totally acceptable to ask developers to become experts in how to deploy their software, while still … you know … finding the time to actually write said software. Not anymore. Sunshower.io has all the tools you need to work with the cloud faster, simpler, and more efficiently than ever before.

Why take a boat across the ocean when you could fly? You have stuff to do, deadlines to meet. Check out our beta on DigitalOcean to see how we can show you a better way.

Blogging consistently is hard, but we’re redoubling our efforts to get content out semi-regularly, even if it’s just to post something simple and (hopefully) helpful. To that end, I’d like to discuss setting up Github pages with a backend.

The Problem

 

Github.com allows you to host static content via its pages feature.  This is fantastic because it makes it trivial to create, deploy, and update a website.  This is how https://sunshower.io is hosted.

But what if you wanted to interact with, say, a database, to track signups?  Furthermore, what if you wanted to do it all over TLS?  This tutorial presumes that your registrar is AWS and your DNS is configured through Route53.

 

Set up Github DNS Alias Records For Github Pages

This one’s easy: In your Route53 Hosted Zone, create an A record that points to Github Page’s DNS servers:

>185.199.110.153

> 185.199.109.153

> 185.199.108.153

> 185.199.111.153

 

 

Then, check in a file in your pages repository under /docs called CNAME containing your DNS name (e.g. sunshower.io)

 

Push that sucker to master and you should have a bouncing baby site!

 

Publish your backend with the correct CORS Preflights

We pretty much just have a plugin for Sunshower.io that registers unactivated users.  Create an EC2 webserver/Lambda function/whatever to handle your requests.  The thing you have to note here is that your backend will have to support preflight requests.  A preflight request is an OPTIONS request that your server understands, and responds with the set of cross-origin resource sharing (CORS​ ) that your backend understands.

This is because your page, hosted at a Github IP, will be making requests to Amazon IPs, even though both are subdomains beneath your top-level domain.  For a JAX-RS service at, say, https://mysite.com:8443/myservice, you will need two methods:

 

@Path("myservice")
@Produces({
MediaType.APPLICATION_JSON,
MediaType.APPLICATION_XML,
})
@Consumes({
MediaType.APPLICATION_JSON,
MediaType.APPLICATION_XML,
})
public interface SignupEndpoint {

@POST
RegistrationConfirmationElement signup(RegistrationRequestElement request);
@OPTIONS
Response getOptions();
}

Note that there must be an @OPTIONS method for each actual conversation method that you want to interact with (e.g. the @POST method here).  What will happen is that a preflight will be made to the same path as the request, and the server will respond with whether that request is allowed.  You can widen the scope of @OPTIONS responses, but you should have a pretty good reason for doing so.

 

The actual @OPTIONS method will look something like:

@Override
public Response getOptions() {
return Response.status(Response.Status.OK)
.allow("POST")
.header("Access-Control-Allow-Origin", "*")
.header("Access-Control-Allow-Methods", "POST, OPTIONS")
.header("Access-Control-Max-Age", 1000)
.header("Access-Control-Allow-Headers", "origin, x-csrftoken, content-type, accept")
.build();
}

where the allow value and Access-Control-Allow-Methods values align with the request type of the actual backend methods.

 

Set up a SSL-enabled ELB

The first thing to do here is to create a certificate using AWS’s wonderful Certificate Manager Service.

Request a certificate by navigating to Certificate Manager > Request a Certificate in the AWS Management Console.  Request a public certificate, and then request it for the fully-qualified name that you want your backend to appear under (e.g. backend.sunshower.io).  If you have access to any of the follow e-mail addresses at your domain:

• administrator@your_domain_name
• hostmaster@your_domain_name
• postmaster@your_domain_name
• webmaster@your_domain_name
• admin@your_domain_name

then select e-mail verification, otherwise select DNS verification and proceed according to the instructions provided by AWS.

 

Once you’ve verified your certificate, create an elastic load balancer on whatever availability zones your backend spans.  If you only have one instance in one availability zone (shame shame!), add that availability zone first.  Create a listener targeting the port that your actual backend is listening on, and add a security group with an ingress port that corresponds to the port that you want your public-facing backend to listen on (for instance, if you want your backend to respond to requests to https://backend.sunshower.io, configure your ingress to be 443.

From there, configure a target that points to your actual backend. If your backend is hosted on EC2 instances, select the instance type for the target, otherwise select ip.  Reference the IP/instances of your backend and create the ELB.

 

Configure a DNS A Record for your ELB

The last thing we need to do is create an A Record that points to your ELB.  If, previously, you requested a TLS certificate for backend.mysite.com, you’ll want to create an A-record whose name is backend, alias=true, with an alias target that is the ELB you created in the previous step.  Save that sucker and you are good to go!

 

 

 

We’re thrilled to announce a preview launch of Sunshower.io. I’d like to take a little time to explain what we do!

Beautifully Simple Multicloud Management

The first thing we do is provide a simple, unified interface for managing your public clouds. This means that you provide some read-only access to your cloud, which we store securely in our vault, and then we go discover whatever infrastructure is in your cloud(s) and manage and organize it for you.

Let me step through a quick example:

Discover your Resources (aka “Systems”)

The first thing we need to do is to discover your resources. Upon logging into Sunshower.io, you’ll be presented with the System Discovery Wizard. A System (i.e. Weather System) is the set of all infrastructure associated with a set of cloud accounts. So, if you have:

• 1 Azure Scale Set with 4 active members that is your development cluster (azure-dev)
• 1 Azure Scale Set with 20 active members that is your production cluster (azure-prod)
• 1 AWS Autoscaling Group with 10 active members that is your AWS dev cluster (aws-dev)
• 1 AWS Autoscaling Group with 30 active members that is your AWS production cluster (aws-prod)

Then you will create a system with at least 2 credentials, one for AWS, the other for Azure.

You can add as many accounts from as many cloud providers as you want. Each of the cloud providers is implemented as a (relatively) simple plugin, and if you add a new plugin (e.g. Google Cloud), you’ll be able to add credentials for that cloud, too.

Once all your accounts have been added, we’ll go through and perform the actual discovery. Once that completes, you’ll be presented with a topological overview of your cloud infrastructure:

The group color is used in the topology view as the color between edges of nodes, and in the geography view to color connections between regions:

And yes, you can totally spin the globe!

Use Your Groups

Grouping is pretty fundamental to Sunshower.io:

• Access control is based on groups
• Management operations can be performed on entire groups (yes, you can spin down every node in a cluster by stopping its group)
• Deployments are based on groups

For instance, you can SSH into an individual machine or an entire group. It’s pretty typical to have a ton of identical machines, so executing the same series of commands produces identical output from each machine. We de-duplicate the output and provide you with the results. Just store your private key in our vault (the actual key is stored in the excellent HashiCorp Vault). But, hey, tailing logs across a bunch of machines has never been so easy!

Visual Deployments (aka “Strata”)

One feature that we’re really excited about is visual modeling of deployments. Basically, you start off with a series of commands (e.g. shell commands), and you compose them together:

From there, you can select a deployer format (e.g. Docker or Packer or even just the userdata section of an AMI, whatever you have plugins installed for) and viola! You have a deployment that you can share with coworkers, publish to everyone, or share with a specific group. For instance, you might create a Stratus that

• Installs Java
• Installs NodeJS
• Installs Gulp

And you can generate a Dockerfile for it, a Packer file, an Azure Image, or an AMI without changing a thing!

Visual Infrastructure Modeling for Systems

But let’s say you don’t have any existing infrastructure and you want to create some. Now, you can log into your various cloud providers’ consoles and spin up whatever you need, but what if you want to create some infrastructure and try it out across clouds? Enter our visual infrastructure modeling for Systems. You can quickly model your infrastructure and deploy it out to any supported cloud (below is the cluster structure for Sunshower.io’s deployment)

You can also export the model to a variety of provisioners like AWS CloudFormation, Azure Resource Manager, or Hashicorp Terraform. We’re currently figuring out what generating Kubernetes manifests looks like, but you’ll be able to do that soon too.

Finally, Anvil for Optimization

All of the data that we collect about your infrastructure we use to build a model of your tasks. If you think about purchased infrastructure as a shipping container, then it makes sense to purchase the smallest shipping container that can fit all of your packages. Anvil extends this analogy by allowing you to define new dimensions for your packages (think 5 or more), and we’ll figure out the smallest shipping container with those dimensions possible across any infrastructure for which there’s an infrastructure plugin. For instance, here’s an example of us spinning up a suboptimal configuration of resources and running Anvil on it:

It yields a much more compact (dense) configuration for your infrastructure. In fact, it typically yields an optimal configuration. You can even configure it to model packages based off of their greatest historical dimensions (like peak hours) so that you’ll never under-provision again, even while saving a substantial amount on your infrastructure.

Conclusion

Thanks for sticking around! I wanted to provide a list of Sunshower.io’s current features to give everyone a better idea as to how it’s used and what it can do for you. Many people will only need one or two of the features, and we’d like to get some feedback as to which might be the most valuable for your organization so that we can get them to you ASAP.

At Sunshower, we’ve been happily using Docker Compose and Docker Swarm for development and deployment respectively. These technologies make it a snap to build and deploy code, and the effort involved in setting them up is quickly offset by their utility.

We’ll continue to use Compose for development, but for better or worse, the industry has spoken and declared Kubernetes the winner of the container orchestration wars. And by this I mean that Swarm is not offered as a service by any of the major CSPs, but each of them either have or are working towards offering turnkey Kubernetes offerings. So, no sense in swimming upstream. We decided to create a Kubernetes deployment for Sunshower. However, a lot of the Kubernetes public cloud offerings are relatively expensive, which is a barrier for a self-funded startup like Sunshower. Fortunately, we got a very generous donation of hardware that I deployed OpenShift to.

Kubernetes vs. OpenShift

OpenShift is pretty much Kubernetes with some extras that make it extra attractive if you’re going to manage your own infrastructure. We’re using it because it also supplies builds.

Getting Started

Configuring your infrastructure

Configuring your infrastructure is the most tedious and error-prone part of this exercise, but if you don’t get it right, it will bite you.

Infrastructure step 1: Create a base VM image

Download a CentOS ISO (minimal is fine) to ISO_PATH where ISO_PATH is some accessible location on your local hard drive. I was not able to get uploads to work with the ESXi web client, so you’ll need to use the older ESXi thick client. Select the following options:

1. Hardware Compatibility (only if you’re interacting with ESXi through VMWare Workstation): Workstation 11.x
2. Installer disc image (iso): ISO_PATH
3. Virtual Machine Name: centos-base (or whatever)
4. Processors: (at least 2 are required, whether that’s 2 processors, 1 core/processor or whatever)
5. 8GB memory
6. Network type: Bridged !important
7. Defaults for I/O Controller types, disk type, select a disk
8. Disk capacity: 50 GB

Once your VM is up and running, run:

yum install update 
reboot now

When your VM comes back up, install some basics:

yum install -y open-vm-tools git docker wget 
systemctl stop network #if you're ssh\'d in, you'll lose access.  Do this through the VMWare console
chkconfig network off
chkconfig NetworkManager on
systemctl NetworkManager start
nmcli dev connect ens33 # You might have a different bridged interface name--check by running nmcli

Infrastructure step 2: Create the VM inventory

This is a bit of a chore since ESXi doesn’t even let you effing clone a VM. If you’re using Workstation, I’d recommend creating all the clones locally, then upload them to the ESXi host.

If you’re not using VMWare Workstation, manually clone each VM by:

1. Creating the base VM (previous step)
2. Browse the datastore you want to clone a VM to, create a new folder with the VM’s name (e.g. openshift-cluster-manager)
3. Copy every file from the base VM’s directory to the new folder except for the log files

For our installation, we’ll have 1 master and 3 workers. If you need HA, you need 3 or 5 masters and however many workers. If you need a production-ready cluster with dozens or hundreds of workers, I do consult =).

Infrastructure step 2: assign static IPs

If you’re using OpenStack/AWS/vSphere, or are running your own DNS server, this is an optional step. Since ESXi does not have any available mechanism for dynamically deploying new virtual machines, your installation will be pretty static, so pick a naming convention, pick an adequate network size, and assign each cluster-node’s interface MAC to an IP. This obviously doesn’t scale well, but eh, it’s fine for a local cluster. One day I’ll show you how to create something a little less hands-on with OpenStack.

Install OpenShift using the convenient installer script provided by Grant Shipley

Select an initiator node. This node will have the role of APIserver, so select one of your masters (or your only master). Clone (Grant Shipley’s installer script)[https://github.com/gshipley/installcentos] with

git clone https://github.com/gshipley/installcentos

Install it with:

echo "console.yourdomain.com" >> /etc/hosts

export DOMAIN=yourdomain.com
export USERNAME=<username> #maybe administrator or something
export PASSWORD=<some super strong password>
cd installcentos && ./install-openshift.sh

This takes a while. Go get some coffee. Take a walk. If your installation node does not have a static IP, it will probably change due to the network reconfiguration this step performs and hork your installation.

If it completes successfully, visit console.yourdomain.com and you should see:

Install your cluster nodes

From your initiator node, copy your ssh key to each of the nodes you want to add to the cluster:

hosts=(os1.sunshower.io
os2.sunshower.io
os3.sunshower.io)

for i in "${hosts[@]}" do
    ssh-copy-id root@$(i)
done;

Otherwise you’ll be typing a lot of passwords.

On the initiator node, edit /etc/ansible/hosts:

[OSEv3:children]
masters
nodes
new_nodes
etcd

[OSEv3:vars]
openshift_deployment_type=origin  # if you bought enterprise, this would be enterprise
os_sdn_network_plugin_name=redhat/openshift-ovs-multitenant  #Important!  The installer defaults to openshift-ovs-subnet for the nodes, but the master is running multitenant.  The openshift node process will fail to start without this
os_firewall_use_firewalld=true  #iptables sucks
osm_cluster_network_cidr=10.0.0.0/24  #change to whatever your network is
openshift_metrics_install_metrics=true  #optional

[masters]
console.sunshower.io  #or whatever your current hostname is

[etcd]
console.sunshower.io  # production installations would have several of these

[new_nodes]
os1.sunshower.io        openshift_schedulable=true  # All the DNS names of your nodes.  Make sure that they're either in your /etc/hosts file or your DNS server is correctly configured
os2.sunshower.io        openshift_schedulable=true
os3.sunshower.io        openshift_schedulable=true

Then, run:

ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/openshift-node/scaleup.yml. These playbooks are installed by atomic-openshift-utils. After about 10 minutes, the installation process should complete and you should be able to run:

<br />oc get nodes
NAME        STATUS    ROLES            AGE       VERSION
10.0.0.10   Ready     compute,master   1h        v1.9.1+a0ce1bc657
10.0.0.4    Ready     <none>           48m       v1.9.1+a0ce1bc657
10.0.0.5    Ready     <none>           48m       v1.9.1+a0ce1bc657
10.0.0.6    Ready     <none>           47m       v1.9.1+a0ce1bc657
[root@localhost ~]# kubectl get nodes
NAME        STATUS    ROLES            AGE       VERSION
10.0.0.10   Ready     compute,master   1h        v1.9.1+a0ce1bc657
10.0.0.4    Ready     <none>           48m       v1.9.1+a0ce1bc657
10.0.0.5    Ready     <none>           48m       v1.9.1+a0ce1bc657
10.0.0.6    Ready     <none>           48m       v1.9.1+a0ce1bc657

EDIT: WordPress’s Markdown escaping is pretty broken–sorry for all the HTML escape codes–I’ll fix them when I get a moment.

In previous segments, we discussed how to collect project dependencies, and use Maven, Gradle, and the Spring Maven Gradle plugin to organize your project dependencies in a maintainable and traceable fashion. In this post, we’re going to take that setup and create a clean, reproducible build using Docker.

Understanding Containerization

A containerized application’s lifecycle is composed of several steps:

1. Defining the container
2. Building the container
3. Tagging the container
4. Pushing the container to a registry
5. Running the container

We’ll be using Docker as our containerization technology, but it’s not the only option. Ensure that it’s installed for your platform before you continue.

Note that these instructions are intended for Linux-based machines. Containers differ from VMs in a very important way, namely that containers share the OS kernel of their container host.

What this means is that, say that my host operating system is Debian, based on Linux Kernel 4.9.0.6-amd64 x86_64. Even if my container uses a different OS base (say, Ubuntu), the container and the host should be kernel-compatible (so, no Linux Kernel 3.10.x-ARM, for instance.)

These considerations are rarely an issue for many software projects. We use Debian-based containers because they’re similar to our dev environments and tooling, but Alpine is a really good option for many projects and has some advantages in production (e.g. resource utilization is anecdotally better.)

Step 1: Defining the Container

In the same way that we identified the compile and runtime dependencies of our project in the first series, let’s think about what we need to build the project. On my development system, I typically build a project by building and publishing the project’s Bill-Of-Materials POMs (mvn clean install -f bom), then by building the project’s artifacts with Gradle and publishing them to my local Maven repository (gradle clean build publishToMavenLocal, or, with Gradle’s nifty shortcuts gradle cle b pTML). So, I obviously need Gradle and Maven installed in my container.

So, what we’re going to do here is:

1. Install the correct version of Java
2. Download Maven from the Maven project site and install it into the container
3. Download Gradle from the Maven project site and install it into the container

Let’s see if we can get rid of any of these steps by selecting the correct base container. Searching Docker hub, I see that Oracle provides JDK images. openjdk:8u141-jdk

My Dockerfile becomes simply:

FROM openjdk:8u141-jdk
ENTRYPOINT /bin/bash

Breaking down these instructions:

FROM openjdk:8u141-jdk says, “ask the local Docker registry to find an image with ID openjdk:8u141-jdk and derive from that. If you can’t find that image locally, reach out to hub.docker.com and see if it’s there`

Building the container pulls the base image, then executes all of the commands in the Dockerfile, which produces a new container:

Sending build context to Docker daemon  220.7kB
Step 1/2 : FROM openjdk:8u141-jdk
8u141-jdk: Pulling from library/openjdk
3e17c6eae66c: Already exists 
74d44b20f851: Already exists 
a156217f3fa4: Already exists 
4a1ed13b6faa: Already exists 
77980e5d0a6d: Already exists 
5458607a81d3: Already exists 
e34cf8338f42: Already exists 
2f3d3da5c56e: Already exists 
2ade7a861e3f: Already exists 
Digest: sha256:4b0c879909b729d67d13e5004f5564df85a5f9c1c3820c13e41151edf1f1b1c0
Status: Downloaded newer image for openjdk:8u141-jdk
 ---> 74c95c985a85
Step 2/2 : ENTRYPOINT /bin/bash
 ---> Running in a4bd5943abcd
Removing intermediate container a4bd5943abcd
 ---> b1c22add1692
Successfully built b1c22add1692 # <-- IMPORTANT, this is your container ID, referenced as $CID.

We can check this new container out by running docker run -it --rm $CID which will drop you into a shell that looks something like:

Now, most base containers don’t have many programs installed. The JDK base containers do have Java, which is the important one.

Now, we can install Maven and Gradle really quickly.

Installing the Prerequisites

I like wget for simple downloads, but curl would work just as well. But we need to install Git anyway, so add:

RUN apt-get update
RUN apt-get install -y git-core wget

To your Dockerfile and build:

Sending build context to Docker daemon  220.7kB
Step 1/4 : FROM openjdk:8u141-jdk
 ---> 74c95c985a85
Step 2/4 : RUN apt-get update
 ---> Running in 2bfa3d396ac6
Get:1 http://security.debian.org stretch/updates InRelease [94.3 kB]
Ign:2 http://deb.debian.org/debian stretch InRelease
Get:3 http://deb.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:4 http://deb.debian.org/debian stretch Release [118 kB]
Get:5 http://deb.debian.org/debian stretch Release.gpg [2434 B]
Get:6 http://deb.debian.org/debian stretch-updates/main amd64 Packages [12.1 kB]
Get:7 http://deb.debian.org/debian stretch/main amd64 Packages [9530 kB]
Get:8 http://security.debian.org stretch/updates/main amd64 Packages [468 kB]
Fetched 10.3 MB in 1s (5807 kB/s)
Reading package lists...
Removing intermediate container 2bfa3d396ac6
 ---> 4984ada9d0c8
Step 3/4 : RUN apt-get install -y git-core wget
 ---> Running in 6e6a79b1c1ab
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  git-core
The following packages will be upgraded:
  wget
1 upgraded, 1 newly installed, 0 to remove and 64 not upgraded.
Need to get 801 kB of archives.
After this operation, 8192 B of additional disk space will be used.
Get:1 http://deb.debian.org/debian stretch/main amd64 wget amd64 1.18-5+deb9u1 [800 kB]
Get:2 http://deb.debian.org/debian stretch/main amd64 git-core all 1:2.11.0-3+deb9u2 [1410 B]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 801 kB in 0s (3242 kB/s)
(Reading database ... 29522 files and directories currently installed.)
Preparing to unpack .../wget_1.18-5+deb9u1_amd64.deb ...
Unpacking wget (1.18-5+deb9u1) over (1.18-5) ...
Selecting previously unselected package git-core.
Preparing to unpack .../git-core_1%3a2.11.0-3+deb9u2_all.deb ...
Unpacking git-core (1:2.11.0-3+deb9u2) ...
Setting up wget (1.18-5+deb9u1) ...
Setting up git-core (1:2.11.0-3+deb9u2) ...
Removing intermediate container 6e6a79b1c1ab
 ---> 67169937ccdd
Step 4/4 : ENTRYPOINT ["/bin/bash"]
 ---> Running in cd4e07152d23
Removing intermediate container cd4e07152d23
 ---> 2c1ab0b17981
Successfully built 2c1ab0b17981

Now, your container will have both wget and git installed.

Setting Environment Variables

If you define an ENV variable in Docker, the value of that ENV can either be passed into the container, or you can specify a default value (or both). We want to be able to reference (and change) both the Gradle version and the Maven version so that if we want to upgrade either, we just pass in new versions when we’re building the container and viola!

# Environment Variables
ENV PROJECT_NAME workspace
ENV GRADLE_VERSION 4.3.1
ENV MAVEN_VERSION 3.5.2
ENV BASE_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Now, when we run the container, we have access to those environment variables:

echo $GRADLE_VERSION
4.3.1

Install Gradle

RUN mkdir -p /opt/build/tools/gradle #Create directory for gradle
RUN wget https://services.gradle.org/distributions/gradle-$GRADLE_VERSION-bin.zip -O /opt/build/tools/gradle.zip  # Download gradle from gradle.org 
RUN unzip -d /opt/build/tools/gradle /opt/build/tools/gradle.zip # Unzip gradle 
ENV GRADLE_HOME=/opt/build/tools/gradle/gradle-$GRADLE_VERSION/bin #Export gradle location as GRADLE_HOME

Now, one really sweet thing about Docker is that, each of these commands defines a new layer. If the textual value of the command that builds a layer doesn’t change, then that layer is retrieved from the layer cache upon subsequent builds. What this means is that the URL for the Gradle download could change, and that wouldn’t break our container.

Install Maven

RUN mkdir -p /opt/build/tools/maven
RUN wget http://www-eu.apache.org/dist/maven/maven-3/$MAVEN_VERSION/binaries/apache-maven-$MAVEN_VERSION-bin.zip \
-O /opt/build/tools/maven.zip
RUN unzip -d /opt/build/tools/maven /opt/build/tools/maven.zip
ENV MAVEN_HOME=/opt/build/tools/maven/apache-maven-$MAVEN_VERSION/bin

Once these have all executed, we have a container with Maven, Gradle, and Git installed and ready to use! In the next post, I’ll discuss how to store credentials securely and pass them into the container so that we can pull our project from source-control and build it.

Of course, if you don’t want to maintain your own, this base image is available on docker hub as sunshower/sunshower-base:latest

Resteasy 3.5 introduced Server-Sent Events (SSE), and there weren’t any good resources for showing how to get it up-and-running, so I thought I’d put together a quick how-to guide.

Add your dependencies

Up your org.jboss.resteasy:resteasy-jaxrs version to 4.0.0.Beta2. This should automatically pull in a JAX-RS API version 2.1 unless you’ve specified a version of JAX-RS directly. If you have, upgrade to 2.1. This will provide access to the SSE component of JAX-RS 2.1 (SseEventSink, Sse, etc.).

Define an SSE method

@Path("test")
@Produces({MediaType.APPLICATION_JSON})
@Consumes({MediaType.APPLICATION_JSON})
public interface TestService {

  @GET
  @Path("{id}/events")
  @Produces(MediaType.SERVER_SENT_EVENTS)
  void subscribe(@PathParam("id") String id, @Context SseEventSink sink, @Context Sse sse);// It's ok to put JAX-RS annotations on the interface.  Recommended, in fact.

  @POST
  @Path("test")
  TestEntity save(TestEntity testEntity);

  @GET
  @Path("{value}")
  @Produces({MediaType.TEXT_PLAIN})
  String call(@PathParam("value") String input);
}

Implement that biz:

  public void subscribe(String id, SseEventSink sink, Sse sse) {
    service.execute(
        new Thread(
            () -> {
              try {
                sink.send(
                    sse.newEventBuilder()
                        .name("domain-progress")
                        .data(String.class, "starting domain " + id + " ...")
                        .build());
                Thread.sleep(200);
                sink.send(sse.newEvent("domain-progress", "50%"));
                Thread.sleep(200);
                sink.send(sse.newEvent("domain-progress", "60%"));
                Thread.sleep(200);
                sink.send(sse.newEvent("domain-progress", "70%"));
                Thread.sleep(200);
                sink.send(sse.newEvent("domain-progress", "99%"));
                Thread.sleep(200);
                sink.send(sse.newEvent("domain-progress", "Done."))
                    .thenAccept(
                        (Object obj) -> {
                          sink.close();
                        });
              } catch (final InterruptedException e) {
                e.printStackTrace();
              }
            }));
  }

Write a test-case:

 @Test
  void ensureSseWorks() throws InterruptedException {

    ResteasyWebTarget path =
        ((ResteasyWebTarget) webTarget).path(TestService.class).path("1/events");
    SseEventSource source =
        SseEventSource.target(path).reconnectingEvery(10, TimeUnit.SECONDS).build();
    try (SseEventSource s = source) {
      System.out.println("a");
      s.register(
          e -> {
            System.out.println("d");
            System.out.println(e.readData(String.class));
            System.out.println("e");
          },
              System.out::println);
      System.out.println("b");
      s.open();
      System.out.println("c");
      Thread.sleep(1000);
    }
  }

Experience failure

Failure 1

If you’ve done all that, it won’t work. The first error you’ll get is that your @Context SseEventSink is null. Unhork yourself by adding

  @Bean
  public SseEventSinkInterceptor sseEventSinkInterceptor() {
    return new SseEventSinkInterceptor();
  }

to your Spring configuration.

Failure 2

The second failure you’ll experience is on the client-side: No MessageBodyReader for “text/event-stream”. This is fixed by adding a

  @Bean
  public SseEventProvider sseEventOutputProvider() {
    return new SseEventProvider ();
  }

to your Spring client configuration.

Experience success

Ahh, delicious success

a
b
c
d
starting domain 1 ...
e
d
50%
e
d
60%
e
d
70%
e
d
99%
e
d
Done.
e

Sunshower-Test

Plugging Sunshower again, you can get all this goodness by annotating your test-class with @io.sunshower.test.ws.EnableJAXRS if you have io.sunshower.test:test-ws:1.0.0-SNAPSHOT as a dependency.

The question of how to unit-test complex external dependencies arises pretty frequently. This is near-and-dear to our hearts because we have many of them. Mocking out complex responses is tedious and error-prone, so I’ll tell you what we do instead.

1: The setup

Stratosphere has an contract for obtaining instances from a cloud service provider, viz.,

public interface ListInstancesOperation extends ProviderOperation<List<Instances>> {
      @Override
      public List<Instance> perform();

      @Override
      public Provider getProvider(); 
     
      @Override
      public Secret() getSecret();
//...etc.

}

perform() will typically interact with the provider service endpoint. The body of perform is quite simple:

  public List<Instance> perform() {
    AmazonEC2 client = createClient();

    DescribeInstancesResult result = client.describeInstances();
    return toInstances(result.getReservations());
  }

This doesn’t look like a super-testable method, so what do we do?

2: Get the actual response and write it to a file

Before you write any code that interacts with an external dependency, you have to understand how that dependency behaves. I recommend keeping a set of IAM credentials in ~/.aws/credentials for testing. Once you do that, actually perform the request and see what you get back.

Now, we serialize the response using Java’s default serialization mechanism to a file that we check into source-control.

DescribeInstancesResult result = client.describeInstances();
Objects.write(result, relativeToRoot("src/test/resources/ec2/list-instances.obj")); // our utilities for writing an object using serialization.

3: Mock it real good

Recall that our method under test had 3 statements. One to create the client, one to make the request to the client, and one to map the results. The client’s method under test is public, so we can mock that. We gave createClient default visibility so that we could mock that while not exposing it as part of our Operation API, and then we put all our actual logic into a private method, whose behavior we want to test.

We can now set up our tests to mock out the external operation with a real result:

  private Secret secret;
  private AmazonEC2Client client;
  private EC2ListInstancesOperation operation;

  @BeforeEach
  void setUp() {
    secret = new Secret();
    operation = new EC2ListInstancesOperation(secret, "us-west-2", (AWS) AWS.getInstance());
    operation = spy(operation);

    client = mock(AmazonEC2Client.class);
    given(client.describeInstances())
        .willReturn(Objects.read("src/test/resources/ec2/list-instances.obj", true));
    given(operation.createClient()).willReturn(client);
  }

And test it as follows:

 @Test
  void ensureInstanceFirewallsAreCorrect() {
    List<Instance> perform = operation.perform();
    Instance instance = perform.get(0);
    assertThat(instance.getFirewalls().size(), is(1));
    Firewall firewall = instance.getFirewalls().iterator().next();
    assertThat(firewall.getName(), is("launch-wizard-2"));
    assertThat(firewall.getSecured().contains(instance), is(true));
  }

This approach works pretty well for external dependencies that don’t change frequently (like the public APIs of large cloud service providers), but less well for external dependencies that do. What are some approaches you use?

One of the challenges in writing abstraction software like Sunshower is that we have to map a ton of vendor-specific properties into our data-model. I had considered writing a mapping language to transform, say, an EC2 Instance or Azure VM into one of our generic Sunshower compute instances, but decided against it because we just dump all the generic vendor properties like spotInstanceRequestId into properties on our internal model analog, and writing a compiler to do that is for big companies.

But how to avoid individually mapping each darn property by hand? Writing code like:

    instance.addProperty(
        new Property(
            Property.Type.String,
            "ami-launch-index",
            "aws.i18n.ami-launch-index",
            String.valueOf(ec2instance.getAmiLaunchIndex())));

for each of exactly 100 hojillion properties across AWS’s and Azure’s and GCE’s and etc. data model is a breeding ground for ennui and bugs.

Attempt 1

Java 7 introduced the Beans API which makes introspecting Java beans super easy. There’s also a little-known feature of Java Matcher.replaceAll/First that allows you to reference a regular-expression capture group in the replacement string, so I whipped up:

PropertyDescriptor[] propertyDescriptors =
        Introspector.getBeanInfo(com.amazonaws.services.ec2.model.Instance.class, Object.class)
            .getPropertyDescriptors();

    instance.setProperties(
        Stream.of(propertyDescriptors)
            .flatMap(
                t -> {
                  Property.Type type = resolveType(t.getPropertyType());
                  if (type == null) {
                    return Stream.empty();
                  } else {

                    Object value = ReflectionUtils.invokeMethod(t.getReadMethod(), ec2instance);

                    return Stream.of(
                        new Property(
                            type,
                            t.getDisplayName().replaceAll("(.)(\\p{Upper})", "$1-$2").toLowerCase(),
                            "aws.i18n."
                                + t.getDisplayName()
                                    .replaceAll("(.)(\\p{Upper})", "$1-$2")
                                    .toLowerCase(),
                            value == null ? null : String.valueOf(value)));
                  }
                })
            .collect(Collectors.toList()));

Which got me what I wanted:

    instance
        .getProperties()
        .forEach(
            t -> {
              System.out.println(
                  String.format(
                      "Name: %s, key: %s, value: %s", t.getName(), t.getKey(), t.getValue()));
            });
  }

Name: role, key: role, value: io.sunshower.stratosphere.core.topology.model.Instance
Name: aws.i18n.architecture, key: architecture, value: x86_64
Name: aws.i18n.client-token, key: client-token, value: sunsh-WebSe-1KKFOLRIA853V
Name: aws.i18n.ebs-optimized, key: ebs-optimized, value: false
Name: aws.i18n.ena-support, key: ena-support, value: true
Name: aws.i18n.hypervisor, key: hypervisor, value: xen
Name: aws.i18n.image-id, key: image-id, value: ami-39595240
Name: aws.i18n.instance-id, key: instance-id, value: i-05b1e0984260d51dd
Name: aws.i18n.instance-lifecycle, key: instance-lifecycle, value: null
Name: aws.i18n.instance-type, key: instance-type, value: t2.micro
Name: aws.i18n.kernel-id, key: kernel-id, value: null
Name: aws.i18n.key-name, key: key-name, value: sunshower-io
Name: aws.i18n.platform, key: platform, value: null
Name: aws.i18n.private-dns-name, key: private-dns-name, value: ip-172-31-12-114.us-west-2.compute.internal
Name: aws.i18n.private-ip-address, key: private-ip-address, value: 172.31.12.114
...etc.

But it wasn’t very pretty or maintainable. So, refactoring:

Attempt 2

public class Properties {

  public static <T> void map(
      Class<T> type,
      T instance,
      Class<?> bound,
      PropertyAwareObject<?> target,
      PropertyMappingConfiguration cfg)
      throws IntrospectionException {

    PropertyDescriptor[] propertyDescriptors =
        Introspector.getBeanInfo(type, bound).getPropertyDescriptors();

    target.setProperties(
        Stream.of(propertyDescriptors)
            .filter(cfg::accept)
            .flatMap(t -> cfg.map(t, instance))
            .collect(Collectors.toList()));
  }
}

public class ProviderPropertyMappingConfiguration implements PropertyMappingConfiguration {

  private final String prefix;

  public ProviderPropertyMappingConfiguration(String prefix) {
    this.prefix = prefix;
  }

  @Override
  public Property.Type resolveType(PropertyDescriptor descriptor) {

    Class<?> propertyType = descriptor.getPropertyType();

    if (Boolean.class.equals(propertyType)) {
      return Property.Type.Boolean;
    }
    if (isIntegral(propertyType)) {
      return Property.Type.Integer;
    }
    if (String.class.equals(propertyType)) {
      return Property.Type.String;
    }
    return null;
  }

  @Override
  public boolean accept(PropertyDescriptor propertyDescriptor) {
    Class<?> propertyType = propertyDescriptor.getPropertyType();
    return Boolean.class.equals(propertyType)
        || String.class.equals(propertyType)
        || isIntegral(propertyType);
  }

  @Override
  public String mapKeyName(PropertyDescriptor descriptor) {
    return descriptor.getDisplayName().replaceAll("(.)(\\p{Upper})", "$1-$2").toLowerCase();
  }

  @Override
  public String mapName(PropertyDescriptor descriptor) {
    return prefix
        + ".i18n."
        + descriptor.getDisplayName().replaceAll("(.)(\\p{Upper})", "$1-$2").toLowerCase();
  }

  @Override
  public String mapValue(PropertyDescriptor propertyDescriptor, Object instance) {
    Object result = ReflectionUtils.invokeMethod(propertyDescriptor.getReadMethod(), instance);
    return result == null ? null : String.valueOf(result);
  }

  private boolean isIntegral(Class<?> propertyType) {
    return Integer.class.equals(propertyType)
        || int.class.equals(propertyType)
        || long.class.equals(propertyType)
        || Long.class.equals(propertyType);
  }

  @Override
  public <T> Stream<Property> map(PropertyDescriptor propertyDescriptor, T instance) {
    Property.Type type = resolveType(propertyDescriptor);
    return type == null
        ? Stream.empty()
        : Stream.of(
            new Property(
                type,
                mapKeyName(propertyDescriptor),
                mapName(propertyDescriptor),
                mapValue(propertyDescriptor, instance)));
  }
}

Allowing us to easily map any properties:

Name: role, key: role, value: io.sunshower.stratosphere.core.topology.model.Instance
Name: aws.i18n.ami-launch-index, key: ami-launch-index, value: 0
Name: aws.i18n.architecture, key: architecture, value: x86_64
Name: aws.i18n.client-token, key: client-token, value: sunsh-WebSe-1KKFOLRIA853V
Name: aws.i18n.ebs-optimized, key: ebs-optimized, value: false
Name: aws.i18n.ena-support, key: ena-support, value: true
Name: aws.i18n.hypervisor, key: hypervisor, value: xen
...etc.

I’ve been asked whether this series really pertains to DevOps. The criticism, as I interpret it, is that DevOps is a conceptual framework for creating high-functioning development teams, whereas the concepts and processes outlined here are related to creating a build with specific tools.

The point at hand seems to be that DevOps as it’s practiced by its adherents and evangelists provides a universal template for whatever ails your development organization. In the DevOps utopia, managers, PMs, engineers, and QA all coexist in complete harmony to produce reliably phenomenal software on time and within the budget. Only, that’s not right, engineers and QA are coalesced in this other interpretation. And managers and PMs? Isn’t that a lot of overhead for healthy, DevOps practicing shops? In this construction, DevOps can become whatever you want it to be, or feel like it should be, because it has no grounding in actual processes.

What I’m getting at is that DevOps can’t be some abstract “template” for “building software factories.” There’s no such thing. What there is is a collective knowledge built of decades of trial and error, and everyone who tells you otherwise is trying to sell you something that you probably don’t need. Sure, containers are great. Kubernetes is great. CI/CD software is probably pretty solid. But an uncomfortable truth is that you can get every bit as good of results with Make, Cron, and Bash because the important thing in writing software is the knowledge of how it all fits together.

And that’s what this series is about: taking common tools and sharing our experience with them and what works well for us, and what doesn’t. You can derive some pretty generalizable truths from this process, like “You need a sane dependency-management process. Here’s what one looks like. There are other valid processes. The important thing is to have one.”