At Sunshower, we’ve been happily using Docker Compose and Docker Swarm for development and deployment respectively. These technologies make it a snap to build and deploy code, and the effort involved in setting them up is quickly offset by their utility.
We’ll continue to use Compose for development, but for better or worse, the industry has spoken and declared Kubernetes the winner of the container orchestration wars. And by this I mean that Swarm is not offered as a service by any of the major CSPs, but each of them either have or are working towards offering turnkey Kubernetes offerings. So, no sense in swimming upstream. We decided to create a Kubernetes deployment for Sunshower. However, a lot of the Kubernetes public cloud offerings are relatively expensive, which is a barrier for a self-funded startup like Sunshower. Fortunately, we got a very generous donation of hardware that I deployed OpenShift to.
Kubernetes vs. OpenShift
OpenShift is pretty much Kubernetes with some extras that make it extra attractive if you’re going to manage your own infrastructure. We’re using it because it also supplies builds.
Getting Started
Configuring your infrastructure
Configuring your infrastructure is the most tedious and error-prone part of this exercise, but if you don’t get it right, it will bite you.
Infrastructure step 1: Create a base VM image
Download a CentOS ISO (minimal is fine) to ISO_PATH
where ISO_PATH
is some accessible location on your local hard drive. I was not able to get uploads to work with the ESXi web client, so you’ll need to use the older ESXi thick client. Select the following options:
- Hardware Compatibility (only if you’re interacting with ESXi through VMWare Workstation): Workstation 11.x
- Installer disc image (iso):
ISO_PATH
- Virtual Machine Name: centos-base (or whatever)
- Processors: (at least 2 are required, whether that’s 2 processors, 1 core/processor or whatever)
- 8GB memory
- Network type: Bridged !important
- Defaults for I/O Controller types, disk type, select a disk
- Disk capacity: 50 GB
Once your VM is up and running, run:
yum install update reboot now
When your VM comes back up, install some basics:
yum install -y open-vm-tools git docker wget systemctl stop network #if you're ssh\'d in, you'll lose access. Do this through the VMWare console chkconfig network off chkconfig NetworkManager on systemctl NetworkManager start nmcli dev connect ens33 # You might have a different bridged interface name--check by running nmcli
Infrastructure step 2: Create the VM inventory
This is a bit of a chore since ESXi doesn’t even let you effing clone a VM. If you’re using Workstation, I’d recommend creating all the clones locally, then upload them to the ESXi host.
If you’re not using VMWare Workstation, manually clone each VM by:
- Creating the base VM (previous step)
- Browse the datastore you want to clone a VM to, create a new folder with the VM’s name (e.g. openshift-cluster-manager)
- Copy every file from the base VM’s directory to the new folder except for the log files
For our installation, we’ll have 1 master and 3 workers. If you need HA, you need 3 or 5 masters and however many workers. If you need a production-ready cluster with dozens or hundreds of workers, I do consult =).
Infrastructure step 2: assign static IPs
If you’re using OpenStack/AWS/vSphere, or are running your own DNS server, this is an optional step. Since ESXi does not have any available mechanism for dynamically deploying new virtual machines, your installation will be pretty static, so pick a naming convention, pick an adequate network size, and assign each cluster-node’s interface MAC to an IP. This obviously doesn’t scale well, but eh, it’s fine for a local cluster. One day I’ll show you how to create something a little less hands-on with OpenStack.
Install OpenShift using the convenient installer script provided by Grant Shipley
Select an initiator node. This node will have the role of APIserver, so select one of your masters (or your only master). Clone (Grant Shipley’s installer script)[https://github.com/gshipley/installcentos] with
git clone https://github.com/gshipley/installcentos
Install it with:
echo "console.yourdomain.com" >> /etc/hosts export DOMAIN=yourdomain.com export USERNAME=<username> #maybe administrator or something export PASSWORD=<some super strong password> cd installcentos && ./install-openshift.sh
This takes a while. Go get some coffee. Take a walk. If your installation node does not have a static IP, it will probably change due to the network reconfiguration this step performs and hork your installation.
If it completes successfully, visit console.yourdomain.com
and you should see:
Install your cluster nodes
From your initiator node, copy your ssh key to each of the nodes you want to add to the cluster:
hosts=(os1.sunshower.io os2.sunshower.io os3.sunshower.io) for i in "${hosts[@]}" do ssh-copy-id root@$(i) done;
Otherwise you’ll be typing a lot of passwords.
On the initiator node, edit /etc/ansible/hosts
:
[OSEv3:children] masters nodes new_nodes etcd [OSEv3:vars] openshift_deployment_type=origin # if you bought enterprise, this would be enterprise os_sdn_network_plugin_name=redhat/openshift-ovs-multitenant #Important! The installer defaults to openshift-ovs-subnet for the nodes, but the master is running multitenant. The openshift node process will fail to start without this os_firewall_use_firewalld=true #iptables sucks osm_cluster_network_cidr=10.0.0.0/24 #change to whatever your network is openshift_metrics_install_metrics=true #optional [masters] console.sunshower.io #or whatever your current hostname is [etcd] console.sunshower.io # production installations would have several of these [new_nodes] os1.sunshower.io openshift_schedulable=true # All the DNS names of your nodes. Make sure that they're either in your /etc/hosts file or your DNS server is correctly configured os2.sunshower.io openshift_schedulable=true os3.sunshower.io openshift_schedulable=true
Then, run:
ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/openshift-node/scaleup.yml
. These playbooks are installed by atomic-openshift-utils
. After about 10 minutes, the installation process should complete and you should be able to run:
<br />oc get nodes NAME STATUS ROLES AGE VERSION 10.0.0.10 Ready compute,master 1h v1.9.1+a0ce1bc657 10.0.0.4 Ready <none> 48m v1.9.1+a0ce1bc657 10.0.0.5 Ready <none> 48m v1.9.1+a0ce1bc657 10.0.0.6 Ready <none> 47m v1.9.1+a0ce1bc657 [root@localhost ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION 10.0.0.10 Ready compute,master 1h v1.9.1+a0ce1bc657 10.0.0.4 Ready <none> 48m v1.9.1+a0ce1bc657 10.0.0.5 Ready <none> 48m v1.9.1+a0ce1bc657 10.0.0.6 Ready <none> 48m v1.9.1+a0ce1bc657
EDIT: WordPress’s Markdown escaping is pretty broken–sorry for all the HTML escape codes–I’ll fix them when I get a moment.
1 comment