HomeOps

Sync Unifi with Hiera

Thu, 03 Dec 2020 00:00:00 +0000

In my previous post I walked you through configuring DHCP with hiera yaml files. These entries allow you to reserve IPs for nodes. These IPs can then also be tied to DNS entries. However I always strive to integrate all services I run. One of those core services is a Unifi Controller. Unifi’s web interface is great for viewing current Wifi clients and also to troubleshoot and track down connectivity issues. However when a system doesn’t advertise its name, you are presented with a MAC address or in the case of some IOT devices a random string. Vendor lookups don’t always help given the same chipset being used on multiple platforms. In this post I’ll show you how to sync Unifi with a yaml file.

Versions tested
Software	Version	OS
ruby	2.7.0p0	ubuntu
Unifi	5.6.40	ubuntu

I have some older Unifi hardware mixed in with newer. This older version supports both.

I run my unifi controller on my synology. I split the controller (Web interface) and its database into separate containers. This is useful for upgrades and migrating the container to different hosts and platforms over time. This also makes it easier for me to connect to the mongo api and rewrite aliases that unifi uses. Lets look at an example of blank node.

This node doesn’t add its hostname to DHCP however I know what it is when looking though my DHCP reseverations. It is the dog treat dispenser that doubles as one of the security cameras for the Synology.

   furbo:
     mac: '38:a2:8c:xx:xx:xx'
     ip: '192.168.53.167'

The DHCP reservation is working due to the puppet code, but the synology is unable to lookup this data on its own. While its possible to import this via CSVs I want this to happen automatically do this as I add new nodes. To do this first I write a script that will read a yaml file on disk and update the names based on the format above.

Push aliases to Unifi

sync_unif_names.rb

#!/usr/bin/env ruby
 require 'mongo'
 require 'yaml'

 begin
   dhcp_hosts = YAML.load_file('/etc/dhcp.yaml')['dhcp_hosts']
   puts dhcp_hosts.inspect
   mac_to_name = Hash.new
   dhcp_hosts.each do |k,h|
     puts "Processing #{k}"
     mac_to_name[h['mac'].downcase] = k
   end
   db = Mongo::Client.new(['synology.homeops.tech:27117'],:database => 'ace')
   db.collections.each do |collection|
     if collection.name == 'user'
       collection.find().each do |doc|
         unless doc['mac'].nil?
           if mac_to_name.keys.include?(doc['mac'].downcase)
             puts "Found: #{mac_to_name[doc['mac']]}"
             puts doc['hostname'] unless doc['hostname'].nil?
             puts doc['oui'] unless doc['oui'].nil?
             collection.update_one({ _id: doc['_id'] }, { '$set' => { name: mac_to_name[doc['mac']] }})
           end
         end
       end
     end
   end
 rescue StandardError => err
   puts('Error: ')
   puts(err)
 end

I can copy my dhcp.yaml to /etc and run the script and it will sync the names.

Automating the Sync

While its nice to be able to update the Unifi interface en masse, Ideally we should be able to this automatically as the YAML file in hiera is updated. We can do that with an refresh in Puppet. On the DHCP server (or any node) we apply the following code.

   package {'mongo':
     ensure   => 'present',
     provider => 'gem',
   }

   file {'/usr/local/bin/sync_names.rb':
     ensure => 'present',
     owner  => 'root',
     mode   => '0700',
     source => 'puppet:///modules/profile/sync_names.rb',
     require => Package['monogo'],
   }

   $hosts = hiera('dhcp_hosts')

   file {'/etc/dhcp.yaml':
     ensure  => 'present',
     owner   => 'root',
     mode    => '0700',
     content => inline_template('<%= @{ dhcp_hosts: hosts }.to_yaml %>'),
     notify  => Exec['/usr/local/bin/sync_names.rb'],
   }

   exec { '/usr/local/bin/sync_names.rb':
     refreshonly => true,
     path        => $::path,
   }

Now when I add a new node to my DHCP server, the same puppet run will trigger the script to run. This happens because the local YAML file’s content will change and puppet will issue a refresh notification to the exec.

Using Terraform with Proxmox

Fri, 16 Oct 2020 00:00:00 +0000

In my previous post on using Razor with Cloud Init I was able to get newly created VMs to PXE boot to razor and automatically install debian. This allows me to quickly deploy VMs for testing and hosting services. However to create these VMs still required clicking though multiple pages of settings in the Proxmox UI. Given I use terraform so much as work I wanted to use it in my home lab. I this quick post I will show you some example code to provision a VM in proxmox with terraform. New to Terraform, checkout Learn.

Versions tested
Software	Version	OS
terraform	v0.14.5	Ubuntu
Ubuntu	20.04	Ubuntu

One great things about the terraform ecosystem is like other tools such as Puppet it has a provider architecture. This plugin system allows users to add resources to terraform that can then be use along side built-in ones. You could for instance use terraform to spin up a vault node and then use the vault node to provision the configuration. The same could happen with the Gitlab provider.

Installing the Provider

versions.tf

 terraform {
   required_providers {
     proxmox = {
       source  = "Telmate/proxmox"
       version = "2.6.7"
     }
   }
 }

In the most recent version of terraform you specify the provider version by convention in the versions.tf file. This is akin to something like a Puppetfile or a Git Submodule. It allows you to pin a version of this provider so that you and your team don’t have compatibility issues across invocations.

Provisioning a Bare Metal Node

main.tf

resource "proxmox_vm_qemu" "metrics" {
   name = "metrics"
   desc = "Graphite, Grafana and Collectd"
 
   network {
     model    = "virtio"
     bridge   = "vmbr0"
     firewall = true
   }
 
   onboot = true
 
   os_type = "cloud-init"
 
   target_node = "proxmox"
   boot        = "order=net0;scsi0"
   iso = "none"
 
   cores   = "2"
   sockets = "1"
   vcpus   = "0"
   cpu     = "host"
   memory  = "2048"
   scsihw  = "virtio-scsi-pci"
 
 
   disk {
     type    = "scsi"
     size    = "64G"
     format  = "qcow2"
     storage = "mute"
   }
 
}

Here is an example of Provisioning the Graphana node covered in this previous post. Lets break down some of the important bits here.

...
  boot        = "order=net0;scsi0"
... 

As I mentioned in the Razor with Cloud Init ,razor server normally expects network booting to always happen. That includes after a node has been provisioned. This allows razor to centrally reinstall the node.

...
    disk {
      type    = "scsi"
      size    = "64G"
      format  = "qcow2"
      storage = "mute"
    }
...

This configures the virtual scsi interface that will correspond to the /dev/sda entry in your preseed.

At the time of this writing the format ends up being raw instead of qcow. Known Bug

If you wanted to simply boot from an iso here you would add:

...
  boot        = "legacy=ncd"
  iso         = "synology:iso/debian-10.7.0-amd64-netinst.iso"
...

This is an exmample of an iso placed in the templates directory of the proxmox share.

Cloud-init

...
os_type = "cloud-init"
...

We specify cloud-init here as in the razor example we are not creating this system from a packer image. However that is possible to do with templates in proxmox.

Ideally we would be able to add the following cloud-init config information

     sshkeys = <<EOF
       ssh-rsa AAAA...
   EOF

However the provider has a heavy handed error message that you cant use cloud-init without templates. Once this is resolved we should be able to use this as a cloud-drive to have proxmox merge configurations with our Razor Cloud Init configuration. This is the bug that covers that issue. I might expierement with forking the provider to get it working in the mean time.

As a side note, this should eventually allow us to tie into Terrform’s ability to generate cloud-init multi part files.

 Source the Cloud Init Config file
  template  = file("${path.module}/templates/cloud.cfg")

  vars = {
    ssh_key = file("/home/acidprime/.ssh/id_rsa.pub")
    domain = "homeops.tech"
  }
}

data "cloudinit_config" "homeops" {
  gzip = false
  base64_encode = false

  part {
    content_type = "text/x-shellscript"
    content = "baz"
    filename = "init.sh"
  }
  # Main cloud-config configuration file.
  part {
    filename     = "init.cfg"
    content_type = "text/cloud-config"
    content      = "${data.template_file.script.rendered}"
  }
}

I will update this post once that bug is updated.

Using Synology as a ZeroTier Bridge

Wed, 15 Jul 2020 00:00:00 +0000

I was introduced to Zero tier a few years ago. I find it pretty solid and the free plan works well as a sort of reverse NAT style VPN’esk solution. Each node gets a multi homed IP that’s always available. This means I can ssh into my laptop no matter where it is, or have it backup to an internal server consistently. You don’t have to open any ports up as your routing traffic through their servers. Additionally you don’t need a static IP to make this work.

Versions tested
Software	Version	OS
DSM	DSM 6.2.2-24922 Update 6	Synology

You need at least one node that’s consistently online to route the traffic from Zero Tier to your devices. You can buy stand alone units for this but I often use my Synology for central services like this. To get started you will need the following:

Sign up for account: https://my.zerotier.com/
Download correct Synology Package for your arch https://download.zerotier.com/dist/synology/

Use the Github Wiki to find the right arch

Install Package

Using the Package Center add the package via the Manual Install Button

Join the Zero Tier network

Grab your network ID from https://my.zerotier.com/ and launch the Zero Tier app you installed via Package Center.

Enable Routing & Nat

/etc/sysctl.conf

net.ipv4.conf.all.forwarding=1
net.ipv4.conf.default.forwarding=1
net.ipv6.conf.all.forwarding=1
net.ipv6.conf.default.forwarding=1

ssh into your Synology and edit the sysctl configuration. Then apply your changes

$ sysctl --system

This needs to be automated but I haven’t decided from a gem installed puppet or hacked ansible for managing my synology yet.

Enable IP Tables Rules

sudo -s
/sbin/iptables -t nat -A POSTROUTING -o bond0 -j MASQUERADE
/sbin/iptables -A FORWARD -i bond0 -o eth50 -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -A FORWARD -i eth50 -o bond0 -j ACCEPT

Where is bond0 is your LAN interface and eth50 is your zerotier interface

I have a bond as I use LACP to bond my 4 Ethernet ports together on my DS1515+

Persistance

These need to be set at reboot to stay persistent so you should add them to /etc/rc.local

Configure Bridging

Add your synology to your network and Allow Ethernet bridging. Create a managed route as shown above.

Zero Tier on Phone

Now that you have an ethernet bridge you can install the Zero Tier app on your phone and join the network.

As you can see I can connect to my arduino project that allows me to lock my car doors at night.

If you followed my DNS Post then you can actually use split DNS here as well by allowing this subnet to make calls to your named service.

Using Razor Server with Cloud-Init

Wed, 15 Jul 2020 00:00:00 +0000

Bare metal imaging is certainly not the rage anymore. Tools like packer allow us to walk the line between golden images and infrastructure as code. I really like packer but I often want more tools in my belt. In a home lab I want to be able to install physical machines with ease the way I install virtual ones at work. Looking at the commit history of razor you can see that its not something being actively updated that much anymore. That said, its actually very versatile software when you get know it. Its light weight and adding modern tools to the mix makes it work quite well. Its lack of use can also be attributed to it being difficult to get started with. Hopefully this post can help.

Versions tested
Software	Version	OS
puppet	6.21.0	centos
centos	7.9.2009	centos

I install razor side by side my Puppet server which is on the Centos Box

Puppetfile

mod 'razor',
  :git => 'https://github.com/acidprime/puppet-razor.git',
  :ref => 'master'

mod 'wget',
  :git => 'https://github.com/voxpupuli/puppet-wget.git',
  :ref => 'v2.0.1'

mod 'java',
  :git => 'https://github.com/puppetlabs/puppetlabs-java.git',
  :ref => 'v6.5.0'

mod 'postgresql',
  :git => 'https://github.com/puppetlabs/puppetlabs-postgresql.git',
  :ref => 'v6.5.0'
 

Installing Razor Server

   include java
   class { 'postgresql::server':
     ip_mask_allow_all_users    => '0.0.0.0/0',
     postgres_password          => 'razor',
   }
 
   include 'wget'
 
   postgresql::server::db { 'razor':
     user     => 'razor',
     password => postgresql_password('razor', 'secret'),
   }
 
   file {'/etc/razor':
     owner  => 'razor',
     ensure => 'directory',
   }
 
   file {'/opt/razor':
     owner  => 'razor',
     ensure => 'directory',
   }
 
   file {'/var/lib/razor/repo-store':
     ensure => 'directory',
     owner  => 'razor',
   }
 
   file {'/etc/puppetlabs/razor-server/config.yaml':
     ensure => 'link',
     source => '/etc/razor/config.yaml',
   }
 
   class { 'razor':
     database_hostname   => '127.0.0.1',
     compile_microkernel => false,
     database_password   => 'secret',
     enable_tftp         => false,
   }

I chose razor server, because its actually light weight. Systems like foreman want to take over multiple parts of your infrastructure for their narrative to work. In my case I had two main goals.

PXE boot new VMs in proxmox
PXE boot my Intel NUC

Its very possible you have never heard of razor before. The basic concept is very intuitive once you start using it. Once you have a razor server, the PXE booted server first boots into an intermediary “micro” kernel which is essentially a tiny Centos7 box. This micro kernel, reports back on the various details of the machine its booted from. These “facts” are collected using Puppet’s facter tool. The razor server then can set the next boot to be one of installation or pass through. In other words on say my Intel NUC I configure to always network boot. This allows razor to check if the box should be installed or perhaps reinstalled. Razor even has IPMI support for proper server hardware to reboot a new box.

I tried to get the IPMI libvirt tools to work with Proxmox only to find that proxmox doesnt support them. Not a huge deal in my case as I often can reboot a node easily.

To reinstall a node type

razor reinstall-node node30

In my previous post I walked you through installing ISC-DHCPD. Those DHCP/DNS/TFTP servers allow me to PXE boot machines in my network with razor. This exposes the Web URLs that razor uses to advance the node on to the next step in the process of being installed.

Node Protection

razor::razor_yaml_setting { "all/checkin_interval":
  value => "15"
}

razor::razor_yaml_setting { "all/protect_new_nodes":
  value => false
}

By default Razor protects new nodes (as to not erase them!). I actually don’t want that protection here as I want to be able to spin a new node up with terraform. I’m also setting the checkin interval here as its required in the template.

Razor Repo

   razor_repo { 'debian-buster-amd64-netboot':
     ensure  => 'present',
     iso_url => "https://deb.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/mini.iso",
     task    => 'debian',
   }

In this example we configure a repo for the debian netboot mini iso. One of the first things I found confusing about razor was new terms it uses for common imaging steps. Some like repo more or less make sense over time.

Razor Tags

   razor_tag { 'virtual':
     ensure => 'present',
     rule   => ['=', ['fact', 'is_virtual'], true]
   }
 
   razor_tag { 'proxmox':
     ensure => 'present',
     rule   => ['=', ['fact', 'macaddress'], "54:b2:03:1c:0c:bd"]
   }

A tag in razor is a rule set that will be applied to a node. In these examples I am creating a tag virtual that will be applied many times to all virtual nodes. I’m also creating a tag proxmox that will likely only ever be applied once. This tag is matched only by the mac address. However I can modify it to match multiple mac addresses if I want to install multiple servers.

This is sort of the tribble style of deployment as my razor server runs on a proxmox node which in turn can create other proxmox nodes in the cluster.

Razor Broker

   razor_broker { 'cloud-init':
     ensure        => 'present',
     broker_type   => 'cloud-init',
     configuration => hiera('cloud-init'),
   }
   razor::broker {'cloud-init':
     module => 'profile',
     root => '/opt/puppetlabs/server/apps/razor-server/share/razor-server/brokers',
   }

A razor broker is essentially a script that runs after first boot. While in ye’olden times that was likely rc.local, since systemd we would customize our own script. In my case while I am not adverse to maintaining custom scripts, I wanted to use common tools I use in my day job. Cloud-init is the defacto standard for this sort of first boot config. It even supports puppet’s advanced install features.

The code above references a custom broker (see below). Lets break it down.

...
configuration => hiera('cloud-init'),
...

This line allows us to load out cloud-cfg which is the configuration cloud-init uses to configure its’ self on first boot, from hiera. This is nice as most examples on the internet are going to copy and paste-able. Albeit with a little more indentation given its nested under the cloud-init key. We don’t get many of the other benefits of hiera here as its ability to merge data is not relevant for this use case. That’s because this hiera call is evaluated in the context of the razor-server and not he respective node. That shouldn’t really be an issue given cloud-init itself can pull in multiple sources of data from multiple types of input.

The razor::broker is copying our custom broker from our module, profile to its final location on the razor server. Before we disect that broker (below) lets tie everything together.

Tiny bit hand wavey here as the puppet config needs to be per node, but I do that based on the certname rather then the csr attributes currently.

Razor Policies

 razor_policy { 'install_debian_on_virtual':
     ensure        => 'present',
     repo          => 'debian-buster-amd64-netboot',
     task          => 'buster',
     broker        => 'cloud-init',
     hostname      => 'node${id}.homeops.tech',
     root_password => 'pveroot',
     max_count     =>  100,
     node_metadata => {"os_version" => "buster"},
     tags          => ['virtual'],
   }
 
   razor_policy { 'proxmox_install':
     ensure        => 'present',
     repo          => 'debian-buster-amd64-netboot',
     task          => 'buster',
     broker        => 'cloud-init',
     hostname      => 'proxmox.homeops.tech',
     root_password => 'pveroot',
     max_count     =>  1,
     node_metadata => {"os_version" => "buster"},
     tags          => ['proxmox'],
   }

Once you understand the individual building blocks of tags,repo and broker a policy should help solidify how they fit together. Simply put a policy is the only actionable item of the lot. Its code references the tag to line up the matching node with the correct repo and broker.

In the examples above, we are using our two tags to create one dynamic policy and one static policy. The install_debian_on_virtual will run 100 times on as many VMs as we throw at it. The proxmox_install will run once. The hostnames on the virtual machines will be dynamic with ${id} being interpolated into the current node number. The proxmox hostname is static.

And with that we can now boot any new VM in proxmox right to the install screen. Further when I start my Intel NUC up it does the same. Wait for it…

Cloud-init Broker.

While its not required to use cloud-init, you might be surprised how under utilized the tool normally is. I use puppet to configure my machines and generally that means I want to keep any initial configuration minimum. However be careful to use the right tools at the right time. Puppet when it works, is great but what happens if you are troubleshooting puppet itself. Having a barebones base configuration with cloud-init sounds like a great plan. Given both puppet and the cloud-init broker can read from hiera, you don’t even in practice have to worry about duplication.

So here is what a bare bones cloud-init configuration can look like

 #cloud-config
 cloud-init:
   # Initial User Configuration
   groups:
     - homeops 
 
   users:
     - default
     - name: homeops 
       gecos: homeops.tech
       shell: /bin/bash
       primary_group: homeops
       sudo: ALL=(ALL) NOPASSWD:ALL
       groups: users, admin
       lock_passwd: false
       ssh_authorized_keys:
         - ssh-rsa AAAAB3Nz...
 
   # If this is set, 'root' will not be able to ssh in and they
   # will get a message to login instead as the above $user (ubuntu)
   disable_root: false
 
   # This will cause the set+update hostname module to not operate (if true)
   preserve_hostname: false
 
   rsyslog:
       remotes:
           log_serv: "@synology.homeops.tech:514"
 
   # Downloads the golang package
   packages:
     - jq
     - curl
     - git
     - qemu-guest-agent
   # Add apt repositories
   apt_sources:
     - source: deb http://apt.puppetlabs.com buster puppet6
       keyid: 'D6811ED3ADEEB8441AF5AA8F4528B6CD9E61EF26'
       filename: puppet6.list
 
   # Run 'apt-get update' on first boot
   apt_update: true
 
   puppet:
     install: true
     package_name: 'puppet-agent'
     conf_file: '/etc/puppetlabs/puppet/puppet.conf'
     csr_attributes_path: '/etc/puppetlabs/puppet/csr_attributes.yaml'
     conf:
       agent:
         server: "puppet.homeops.tech"
         certname: "%f"
     csr_attributes:
         custom_attributes:
             1.2.840.113549.1.9.7: 342thbjkt82094y0uthhor289jnqthpc2290
         extension_requests:
             pp_preshared_key: 342thbjkt82094y0uthhor289jnqthpc2290
 
   timezone: 'UTC'
 
   manage_resolv_conf: true
 
   resolv_conf:
     nameservers: ['192.168.53.60', '192.168.53.70']
     searchdomains:
       - homeops.tech
     domain: homeops.tech
     options:
       rotate: true
       timeout: 1
 
   datasource_list: [ ConfigDrive, None ]
   datasource:
     ConfigDrive:
       dsmode: local

So as you can see , this cloud-init handles the the basics of creating a backup account, and then bootstraps puppet. The csr_attributes can be use to then allow autosigning and classification of the node in puppet. I enable the ConfigDrive data source here as proxmox actually supports cloud-init out of the box. This allows an ISO image to contain extra node specific data such as the IP address of the node. I will be expierementing with using the SD card slot on my servers to bootstrap some of this local data from my servers in the future.

Again a little hand wavy here with the csr_attributes as they can’t be node specific. Thats v2 and can be fixed with cloud-drives and such.

Cloud-init Broker

You can check out the gist for the configuration yaml for the broker. However lets take a look at what the install.erb script looks like. This script is essentially the payload for the cloud-init process. It will create a cloud-config and then run the final boot steps for cloud init. Assuming everything was successful it will phone home to razor and mark the node as installed. This by the way showcases one of the powers of razor, which is to interpolate ruby templates on files by adding metadata and node data dyanmically.

Buster Task

In my example I am installing debian buster. At the time of this writing the razor server rather predictably only had a “task” for wheezy. Updating for buster was a minor pain however it also works with the cloud-init broker to enable a new systemd item in liue of the rc.local file. You can find a full buster task in this gist.

A few notes here on this preseed

<% if node.facts['is_virtual'] == true %>
d-i partman-auto/disk string /dev/sda
<% else %>
d-i partman-auto/disk string /dev/nvme0n1
<% end %>

The virtual machines I use in proxmox are scsi emulated storage controllers. The Intel NUC on the other hand is an M.2 Nvme.

sh -c 'set -- $(vgs --rows --noheadings | head -n 1); for vg in "$@"; do swapoff "/dev/$vg/swap"; vgremove -f "$vg"; done; set -- $(pvs --rows --noheadings | head -n 1); for pv in "$@"; do pvremove -f "$pv"; done'

This little script is used to remove the volume group as its name conflicts given I have static DHCP reservations. In other words the hostname is always the same in Netboot or Regular boot.

The new systemd item is simply a runner for the postinstall script that calls cloud-init

Cloud-init seems well documented in some ways and not in others. My initial assumption was that I would not have to run these commands if I simply enabled the systemd items for cloud-init. However in testing that didn’t work and so the script is essentially running cloud-init.

The updated task and the custom broker above are essentially modified versions of the ones you find here. I might add these to my fork in the future, but at the moment as they are site specific I add them to my profile modules’ files directory.

Managing DHCP with Puppet

Wed, 15 Jul 2020 00:00:00 +0000

In my previous post on managing DNS with Puppet I walked you through the code to configure an internal DNS domain with puppet. In this post I will cover setting up a DHCP server that supports IP reservations and has failover configured. I use this with two raspberry pis that also provide my DNS. While DHCP might often be ran off your router, in a home lab you might want to run DHCP in a separate VLAN for testing things out like PXE booting.

Versions tested
Software	Version	OS
puppet	6.21.0	ubuntu
dhcpd	isc-dhcpd-4.4.1	ubuntu
ubuntu	20.04	server

I run this all on a set of two raspberry pis’

Puppetfile

mod 'dhcp',
  :git => 'git@github.com:puppetlabs/puppetlabs-dhcp.git',
  :ref => 'b5925938188787faad99fdb52f294796e527a3d1'

mod 'nodejs',
  :git => 'https://github.com/voxpupuli/puppet-nodejs.git',
  :tag => 'v8.1.0'

mod 'staging',
  :git    => 'https://github.com/nanliu/puppet-staging.git',
  :commit => 'b466d93f8deb0ed4d9762a17c3c38f356aa833ee'

Installing DHCPD

  class { '::dhcp':
    nameservers   => ['192.168.53.60','192.168.53.70'],
    ntpservers    => ['time.homeops.tech'],
    interfaces    => ['eth0'],
    dnsdomain     => ['homeops.tech'],
    extra_config  => @("EOS")
      option space APC;
      option APC.cookie code 1 = string;
      class "vendor-id" {
           match option vendor-class-identifier;
      }
      subclass "vendor-id" "APC" {
           vendor-option-space APC;
           option APC.cookie "1APC";
      }
      set vendor-string = option vendor-class-identifier;
      if exists user-class and option user-class = "iPXE" {
          filename "bootstrap.ipxe";
      } else {
          filename "undionly.kpxe";
      }
      next-server $::ipaddress;
      | EOS
  }

Here we are installing isc-dhcpd , which is the most common dhcp deamon on embedded systems. If your wireless gear is based on openwrt or if your networking gear is based on linux you might have this service already running. Given I have DNS running on my raspberry pis it only makes sense to add DHCP. This allows us in this example to configure DHCP options like PXE boot servers and the APC cookie configuration on my batter backups.

Setting up failover.

 # Hiera call is needed here as they are two different servers
 class { 'dhcp::failover':
   role         => hiera('dhcp::failover::role'),
   peer_address => hiera('dhcp::failover::peer_address'),
 }

I have two rasperry pis and unlike DNS that has its own client side failure abilities DHCP requires a server side configuration for failure protection. This allows me to reboot,reimage or reconfigure my running DHCP servers without bumping Netflix offline.

in my experience DHCP failover works best with reservations. In the event of a failure the pool is cut in half , however those nodes with reservations will continue to receives their IPs. This keeps me honest in updating the yaml I use to manage this.

This hiera lookup is essentially just a per node configuration. In the example hierarchy we setup in configuring DNS we added this line:

'nodes/%{::trusted.certname}'

I have two DHCP servers hank.homeops.tech and dean.homeops.tech So my yaml configuration is stored in my data directory as follows

nodes/
├── dean.wallcity.org.yaml
├── hank.wallcity.org.yaml
└── puppet.wallcity.org.yaml

Hank is the primary so that servers configuration is:

---
dhcp::failover::role: 'primary'
dhcp::failover::peer_address: '192.168.53.70'

Dean is secondary/failover node and its configuration is:

---
dhcp::failover::role: 'secondary'
dhcp::failover::peer_address: '192.168.53.60'

While there are many ways of making this information dynamic, you likely want to keep things simple given these servers are some of the first needed to get things rolling in your lab.

Configuring an IP Pool

dhcp::pool{ 'homeops.tech':
  network => '192.168.53.0',
  mask    => '255.255.255.0',
  range   => ['192.168.53.165 192.168.53.200'],
  gateway => '192.168.53.1',
}

Here we create a range of IPs form 165 to 200. The DHCP reservations should be outside this range.

Keep in mind this pool is cut in half during failover.

Load Static Reservations

  $dhcp_hosts = hiera('dhcp_hosts')

  create_resources('dhcp::host',$dhcp_hosts)

Here we load in the DHCP hosts from hiera. This allows us (like DNS) to manage this with YAML files. In my case I create a file named dhcp.yaml in the hierarchy to contain these entries. This data can then be consumed by many other service , like icinga and razor-server.

Note: I need to upgrade this for completeness to the .each syntax in Puppet.

Adding static entries is as simple as adding entries to the hash:

dhcp.yaml

---
dhcp_hosts:
  aurora:
    mac: '14:B3:2F:0B:F1:15'
    ip: '192.168.53.16' 
  echo_dot:
    mac: '44:65:0D:49:F9:47'
    ip: '192.168.53.51'

Monitoring with Monit

  class { 'monit':
    httpd          => true,
    httpd_address  => $::ipaddress,
    httpd_password => '$uper$secret',
  }

  monit::check { 'dhcpd':
    content => 'check process isc-dhcp-server with pidfile /run/dhcp-server/dhcpd.pid 
    start program = "/etc/init.d/isc-dhcp-server start"
    stop  program = "/etc/init.d/isc-dhcp-server stop"
    if failed host 127.0.0.1 port 67 type udp then alert
  ',
  }

Because of the criticality of this service I want to make sure it is always running, as we saw with DNS we can add monit. We can run this on the local box as well to have status page for the services.

Installing Glass (DHCP Web interface)

  class { '::nodejs':
    manage_package_repo       => true,
    nodejs_dev_package_ensure => 'present',
    npm_package_ensure        => 'present',
  }

  package { 'forever':
    ensure   => 'present',
    provider => 'npm',
  }

  vcsrepo { '/opt/glass-isc-dhcp':
    ensure   => present,
    provider => git,
    source   => 'https://github.com/Akkadius/glass-isc-dhcp.git',
    revision => '10102a670909f30f7a471837826cf863b18ba0b0',
    notify   => Exec['install'],
  }

  exec {'install':
    command =>  'chmod u+x ./bin/ -R && chmod u+x *.sh && mkdir -p logs && npm install',
    cwd     => '/opt/glass-isc-dhcp',
    refreshonly => true,
    path        => '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
  }

  cron { 'glass_startup':
    ensure  => 'present',
    command => 'cd /opt/glass-isc-dhcp && /usr/local/bin/forever --minUptime 10000 --spinSleepTime 10000 -a -o ./logs/glass-process.log -e ./logs/glass-error.log ./bin/www',
    special => 'reboot',
    target  => 'root',
    user    => 'root',
    require => Package['forever'],
  }

In configuring the raspberry pis you might miss having the standard web interface that comes with most home routers version of DHCP. This can be helpful for finding MAC addresses on opaque machines, like smart speakers. I Have been using the Glass DHCP web interface for a while now and while not every feature is useful,given I’m managing the configuration with puppet, I find the overview page useful for viewing the lease files. There is no official installer for this tool but a little puppet code goes along way here.

Optional: Enable PXE & TFTP Services

package {'tftp':}

  file { '/var/lib/tftpboot':
    owner  => 'tftp',
    group  =>  'tftp',
    ensure => directory,
    mode   => '0755'
  }

  class { 'tftp':
    directory => '/var/lib/tftpboot',
    address   => $::ipaddress,
    options   => '--verbose --permissive --secure --timeout 120',
    inetd     =>  false,
  }

  # The iPXE kernel will fetch a bootstrap.ipxe file, that then aims a chain
  # loader at the Razor server to determine how to proceed.  For instance, new
  # machines are simply told to load a microkernel.
  # The bootstrap is static, but Razor likes to be the one to craft it.
  staging::file { 'bootstrap.ipxe':
    target      => '/var/lib/tftpboot/bootstrap.ipxe',
    source      => "http://razor.homeops.tech:8150/api/microkernel/bootstrap?nic_max=1&http_port=8150",
    curl_option => '--insecure',
    require     => [ File['/var/lib/tftpboot'] ],
  } ->

  tftp::file { 'bootstrap.ipxe':
    ensure => file,
    mode   =>  '0755',
    source => "/var/lib/tftpboot/bootstrap.ipxe",
  }

  # undionly.kpxe
  wget::fetch { 'http://boot.ipxe.org/undionly.kpxe':
    destination => "/var/lib/tftpboot/undionly.kpxe",
  } ->

  tftp::file { 'undionly.kpxe':
    ensure => file,
    mode   =>  '0755',
    source => "/var/lib/tftpboot/undionly.kpxe",
  }

While this not strictly needed in every environment, I find the inclusion of TFT and PXE to be generally connected to my DHCP ecosystem.

...
      if exists user-class and option user-class = "iPXE" {
          filename "bootstrap.ipxe";
      } else {
          filename "undionly.kpxe";
      }
      next-server $::ipaddress;
...

As we saw earlier we setup the DHCP options to point to the needed files to PXE boot a razor-server implementation.

Note that you need to comment out this section unless you have setup razor-server. Watch here for a future razor article

Managing DNS with Puppet

Sat, 06 Jun 2020 00:00:00 +0000

One of the more important things in configuration management is DNS. In home labs we often don’t have DNS out of box. Some folks do use pi-holes but often don’t configure custom domains. I often use static reservations for all my IOT devices. This means all devices on my network can use DNS names to configure and talk to each other with those respective static’ish IPs. In this post I will show you how I use Puppet to setup an internal DNS domain.

Versions tested
Software	Version	OS
puppet	6.21.0	ubuntu
bind	9.16.1-Ubuntu	ubuntu
ubuntu	20.04	server

I run this all on a set of two raspberry pis’

Puppetfile

mod 'dns',
  :git => 'git://github.com/amateo/puppet-dns.git',
  :commit => '15805a8a6577bea6b3dfab1d8951369c925b5e6a'

mod 'monit',
  :git => 'https://github.com/acidprime/puppet-monit.git',
  :commit => 'f94712677271ccab0fc990478e31a7f37cb9791d'

mod 'collectd',
  :git => 'git@github.com:pdxcat/puppet-module-collectd.git',
  :tag => 'v12.2.0'

mod 'hiera',
  :git => 'https://github.com/hunner/puppet-hiera.git'
  :tag => 'v2.1.1'

mod 'resolv_conf',
  :git => 'git@github.com:saz/puppet-resolv_conf.git',
  :tag => 'v3.0.5'

mod 'systemd',
  :git => 'https://github.com/camptocamp/puppet-systemd.git',
  :ref => '2.12.0'

mod 'timezone',
  :git => 'https://github.com/kogitoapp/puppet-timezone.git',
  :ref => 'v2.1.1'

Using a monit fork here as the original author doesn’t update OS version quickly enough for me.

Configuring NTP

Given modern requirements like DNSSec , time sync is an about necessity for dns servers.

class{'systemd':
  manage_timesyncd    => true,
  ntp_server          => ['0.pool.ntp.org', '1.pool.ntp.org'],
  fallback_ntp_server => ['2.pool.ntp.org', '4.pool.ntp.org'],
}

class { 'timezone':
    timezone => 'UTC',
}

class { 'ntp':
  servers    => [ ''0.pool.ntp.org,'1.pool.ntp.org' ],
  autoupdate => true,
}

package {'ntpdate':}

While this goes most of the way, there is a chicken before the egg scenerio here, that ntp requires DNS. I work around that on boot with the systemd implementation. See below.

Installing Bind

  $dns_domain = 'homeops.tech'


  class {'dns::server':
   enable_default_zones => false,
  }

  dns::server::options { '/etc/bind/named.conf.options':
    check_names_master     => 'fail',
    check_names_slave      => 'warn',
    dnssec_enable          => false,
    forwarders             => [ '205.171.3.65', '205.171.2.65' ],
    statistic_channel_ip   => '127.0.0.1',
    statistic_channel_port => 8053,
  }

The code above is an example of configuring bind9 without the default zones. I’m setting this server up with forwarders as I find that media streaming and such seems to work best with the content distro networks when I use my ISP’s DNS.

I’m making an anecdotal assumption here as its hard to verify this behavior but seems like the most likely conclusion

Adding DNS Zones

 dns::zone { $dns_domain:
    soa => $::fqdn,
    soa_email => "root.${::fqdn}",
    nameservers => [$::hostname],
  }

  # Reverse Zone
  dns::zone { '53.168.192.IN-ADDR.ARPA':
    soa => $::fqdn,
    soa_email => "root.${::fqdn}",
    nameservers => [$::hostname],
  }

  # Apex record e.g. homeops.tech vs www.homeops.tech
  dns::record { "apex_alias${dns_domain}":
    zone   => $dns_domain,
    host   => '@',
    record => 'A',
    data   => '192.168.53.222'
  }

Here we create a new DNS zone , and set the nameservers to the hostname of the machine.

192.168.22.222 is the address of a blob server that runs the rendered for my website before I push it to the web

Configuring records

Now that we have out DNS zones configured we can add record. While you could do this in code, I find the amount of DNS records over time mean something like yaml is better. Further it also means you can load this data set into other tools e.g. add automatic testing with icinga. Stay tuned for a future article on icingaweb2.

First lets setup Hiera. Hiera is a data ingestion engine builtin to puppet. If you just setup your Puppet Server using my previous article then you need to manage hiera’s configuration.

  class { '::hiera':
    backends     => [
      'yaml',
    ],
    datadir      => '/etc/puppetlabs/code/environments/%{environment}/data',
    hierarchy    => [
      'nodes/%{::trusted.certname}',
      'environments/%{environment}',
      'dhcp',
      'dns',
      'common',
    ],
  }

Bootstrapping puppet can have some chicken before the egg scenarios.

This is an example configuration you can apply to your Puppet Master to configure hiera with support for a file called dns.yaml in your data directory of your control repo. The most important idea here is that hiera is using yaml files that are dynamically name e.g. %{::trusted.certname} or statically named e.g. dns. For our purposes we will mostly be using static entries for our DNS records.

Lets add code to pull DNS records in from hiera:

  # Resource Defaults
  Dns::Record::A {
    zone => $dns_domain,
    ptr  => true,
  }

  Dns::Record::Cname {
    zone => $dns_domain,
  }

  hiera('a_records').each |$key, $value| {
    dns::record::a { $key:
      data => "${value['data']}",
    }
  }

  hiera('cname_records').each |$key, $value| {
    dns::record::cname { $key:
      data => "${value['data']}"
    }
  }

This code snippet will iterate over the keys in hiera. You can create more then A and CNAME but for my purposes, these and the automatically created PTR records normally are enough. Lets look at what our dns.yaml file looks like in this configuration:

dns.yaml

---
# DNS Records
cname_records:
  grafana:
    data: 'proxmox.homeops.tech'
  www:
    data: 'proxmox.homeops.tech'
a_records:
  proxmox:
    data: '192.168.53.38'
    tag: ['linux']
  alexa:
    data: '192.168.53.206'
    tag: ['iot']

The nice thing about this configuration is that to add DNS records you now simply edit the yaml.

You can safely ignore the tag key here, it will be used in future articles to classify these entries for monitoring.

Installing useful tools

  package {'dnstop':
    ensure => present,
  }

Monitoring & Reporting

  monit::check { 'bind9':
    content => 'check process named with pidfile /var/run/named/named.pid 
    start program = "/etc/init.d/named start"
    stop  program = "/etc/init.d/named stop"
    if failed host 127.0.0.1 port 53 type tcp protocol dns then restart
    if failed host 127.0.0.1 port 53 type udp protocol dns then restart
  ',
  }

While systemd is all the rage, I like pants and suspenders when it comes to critical services.

class { 'collectd::plugin::bind':
  url    => 'http://localhost:8053/',
}

We can also load the metrics into collect/graphite/graphana. This is configured above in the options:

...
   statistic_channel_ip   => '127.0.0.1',
   statistic_channel_port => 8053,
...

You can checkout my article on setting up Graphite/Grafana/Collectd here

Split DNS on VPN

  dns::acl { 'trusted':
    ensure => present,
    data   => [ '192.168.53.221/32', '192.168.54.0/24', ]
  }

I use zerotier for my “vpn” in addition to swan. I need to allow these clients to access the DNS server. in this configuration I setup the hiera range of my subnet (vpn) and the .54 address space I use with zerotier.

Configure resolv.conf

class { 'resolv_conf':
   nameservers => ['127.0.0.1', '192.168.53.60', '192.168.53.70'],
   domainname  => $dns_domain,
}

Once we have tested the DNS server we can set it to perform resolutions with itself.

Systemd Hacks

Given i normally use raspberry pi’s I can’t rely on timesync working out of the box. I work around that by waiting until after boot (and the network is online) to sync with an IP address of a known NTP server.

 package {'ntpdate':
   ensure => present,
 }

 # Setup the users custom shell as pass through to docker
  file {"/usr/local/bin/atboot.sh":
    ensure  => file,
    owner   => root,
    mode    => '0755',
    content =>   @("SHELL"/L)
    #!/bin/bash -x
    ntpdate -u 209.114.111.1 &&
      service bind9 restart &&
        dig google.com @127.0.0.1
    | SHELL
  } 

  $_timer = @(EOT)
  [Unit]
  Description=Run DNS checks at startup
  Wants=network-online.target
  After=network-online.target
  [Timer]
  OnBootSec=0
  OnUnitActiveSec=1w 
  Unit=timesync.service
  EOT
  
  $_service = @(EOT)
  [Service]
  Type=oneshot
  ExecStartPre=/bin/sh -c 'until ping -c 1 209.114.111.1 ; do sleep 1; done;'
  ExecStart=/usr/local/bin/atboot.sh
  TimeoutSec=0
  StandardOutput=journal+console
  StandardError=journal+console
  EOT
  
  systemd::timer{'atboot.timer':
    timer_content   => $_timer,
    service_unit    => 'timesync.service',
    service_content => $_service,
    active          => true,
    enable          => true,
  }

This should run and make sure during a reboot or outage that your server syncs time even when DNS is broken

● timesync.service
     Loaded: loaded (/etc/systemd/system/timesync.service; static; vendor preset: enabled)
     Active: inactive (dead) since Tue 2021-02-16 12:27:00 PST; 31s ago
TriggeredBy: ● atboot.timer
    Process: 248739 ExecStartPre=/bin/sh -c until ping -c 1 209.114.111.1 ; do sleep 1; done; (code=exited, status=0/SUCCESS)
    Process: 248742 ExecStart=/usr/local/bin/atboot.sh (code=exited, status=0/SUCCESS)
   Main PID: 248742 (code=exited, status=0/SUCCESS)

Feb 16 12:27:00 dns1.homeops.tech atboot.sh[248941]: ;; QUESTION SECTION:
Feb 16 12:27:00 dns1.homeops.tech atboot.sh[248941]: ;google.com.                        IN        A
Feb 16 12:27:00 dns1.homeops.tech atboot.sh[248941]: ;; ANSWER SECTION:
Feb 16 12:27:00 dns1.homeops.tech atboot.sh[248941]: google.com.                89        IN        A        172.217.3.206
Feb 16 12:27:00 dns1.homeops.tech atboot.sh[248941]: ;; Query time: 19 msec
Feb 16 12:27:00 dns1.homeops.tech atboot.sh[248941]: ;; SERVER: 127.0.0.1#53(127.0.0.1)
Feb 16 12:27:00 dns1.homeops.tech atboot.sh[248941]: ;; WHEN: Tue Feb 16 12:27:00 PST 2021
Feb 16 12:27:00 dns1.homeops.tech atboot.sh[248941]: ;; MSG SIZE  rcvd: 83
Feb 16 12:27:00 dns1.homeops.tech systemd[1]: timesync.service: Succeeded.
Feb 16 12:27:00 dns1.homeops.tech systemd[1]: Finished timesync.service.

You should see similar output to this, if your systemd item fails you can monitor its failures to determine if DNS is down due to timesync issues.

Graphite, Grafana and Collectd

Wed, 20 May 2020 00:00:00 +0000

I recently moved my existing esxi hypervisors to ProxMox to have more control then the esxi shell allowed. This allows me to run puppet natively on the hyper visor host. As part of this upgrade I installed an old SSD and ram into my 2011 Mac Mini. While doing so I broke the small traces that attach the cooling fan. After some research I found many people have done the same and just leave it fanless. This presented me with a great opportunity to graph the the various temp sensors as I tried different cooling methods.

Versions tested
Software	Version	OS
puppet	6.21.0	Debian
Debian	10.7	buster

Puppetfile

mod 'graphite',
  :git => 'git@github.com:echocat/puppet-graphite.git',
  :ref => 'v8.0.0'

mod 'grafana',
  :git => 'git@github.com:echocat/puppet-grafana.git',
  :ref => 'v1.2.0'

mod 'elasticsearch',
  :git => 'https://github.com/elasticsearch/puppet-elasticsearch.git',
  :ref => '7.0.0'

mod 'elastic_stack',
  :git => 'https://github.com/elastic/puppet-elastic-stack.git',
  :ref => '7.0.0'

mod 'apache',
  :git => 'https://github.com/puppetlabs/puppetlabs-apache',
  :ref => '1.5.0'

mod 'collectd',
  :git => 'git@github.com:pdxcat/puppet-module-collectd.git',
  :ref => 'v12.2.0'

New to puppet? Checkout my how to on installing a puppetserver

We can install the packages needed for graphite to configure and store data.

Installing Graphite

package{ 'graphite-carbon':
  ensure => 'present',
}

class { 'graphite':
    gr_apache_24               => true,
    gr_web_cors_allow_from_all => true,
    secret_key                 => '$uper$ecret?',
    gr_web_server           => 'none',
    gr_web_user             => 'www-data',
    gr_web_group             => 'www-data',
    gr_disable_webapp_cache => true,
    gr_storage_schemas      => [
    {
      name       => 'carbon',
      pattern    => '^carbon\.',
      retentions => '1m:90d'
    },
    {
      name       => 'default',
      pattern    => '.*',
      retentions => '1m:30m,1m:1d,5m:2y'
    },
    {
      name       => 'collectd',
      pattern    => '^collectd.*',
      retentions => '1m:30m,1m:1d,5m:2y'
    }
  ],
  }

Here we can install Graphite and configure it with a storage scheme for collectd. The important bits here are gr_web_server => 'none', which allows us to configure graphite to be hosted with apache.

Setting up a Webserver

As both Graphite and Grafana can use apache lets start with a default installation.

include '::apache'

package {'libffi-dev':
  ensure => 'present',
}

Setting up Graphite Vhost

Once apache is running we can setup a vhost for Graphite as follows

apache::vhost { "graphite.${::domain}":
    port    => '80',
    docroot => '/opt/graphite/webapp',
    wsgi_application_group      => '%{GLOBAL}',
    wsgi_daemon_process         => 'graphite',
    wsgi_daemon_process_options => {
      processes          => '5',
      threads            => '5',
      display-name       => '%{GROUP}',
      inactivity-timeout => '120',
    },
    wsgi_import_script          => '/opt/graphite/conf/graphite.wsgi',
    wsgi_import_script_options  => {
      process-group     => 'graphite',
      application-group => '%{GLOBAL}'
    },
    aliases => [
         { 
           alias            => '/content/',
           path             => '/opt/graphite/webapp/content/',
         },
         { 
           alias            => '/static/',
           path             => '/opt/graphite/static/',
         },
   ],
    wsgi_process_group          => 'graphite',
    wsgi_script_aliases         => {
      '/' => '/opt/graphite/conf/graphite.wsgi'
    },
    headers => [
      'set Access-Control-Allow-Origin "*"',
      'set Access-Control-Allow-Methods "GET, OPTIONS, POST"',
      'set Access-Control-Allow-Headers "origin, authorization, accept"',
    ],
    directories => [
      {
        path => '/media/',
        order => 'deny,allow',
        allow => 'from all'
      },
      {
        path => '/opt/graphite/static/',
        order => 'deny,allow',
        allow => 'from all'
      },
      {
        path => '/opt/graphite/webapp/content/',
        order => 'deny,allow',
        allow => 'from all'
      },
    ]
  }

This is more or less the example in the modules README with a small tweak:

...
{ 
  alias            => '/static/',
  path             => '/opt/graphite/static/',
},
...

This undocumented line is the most important as otherwise you are greeted with a blank site with only headers.

  file {"/opt/graphite/conf/graphite.wsgi":
    ensure => 'present',
    source => "/opt/graphite/conf/graphite.wsgi.example"
  }

We now can copy the example .wsgi file

Setup

We now need to init the graphite django configuration. I should show you how to do this with an exec here, but honestly I often do this manually on first build with the following commands

sudo -H PYTHONPATH=/opt/graphite/webapp django-admin.py migrate --settings=graphite.settings --run-syncdb
sudo -H PYTHONPATH=/opt/graphite/webapp django-admin.py createsuperuser --settings=graphite.settings
sudo -H PYTHONPATH=/opt/graphite/webapp django-admin.py collectstatic --noinput --settings=graphite.settings

You should be able to verify the installation by sending some example metrics too it:

PORT=2003
SERVER=graphite.homeops.tech
echo "local.random.diceroll 4 `date +%s`" | nc -q0 ${SERVER} ${PORT}

Installing ElasticSearch

To store the graphs in grafana we need an elasticsearch deployment. I actually don’t use this as I normally export the json and have puppet manage it so it can be redeployed at will. The following however is an example of getting elasticsearch up and running.

class {'elasticsearch':
  config       => {
    'cluster.name'           => 'elk',
    'node.name'              => $::fqdn,
    'network.host'           => '0.0.0.0',
    'discovery.type'         => 'single-node',
    'http.cors.allow-origin' => '*',
    'http.cors.enabled'      => 'true',
    'xpack.security.enabled' => 'false',
    'xpack.ml.enabled'       => 'false',
    'node.master' => 'true',
    'node.data' => true,
    'node.ingest' => true,
    'action.auto_create_index' => true,
  },
  restart_on_change => true,
  manage_repo  => true,
  init_defaults => {
      'ES_USER' => 'elasticsearch',
      'ES_GROUP' => 'elasticsearch',
      'MAX_OPEN_FILES' => '65535',
      'MAX_LOCKED_MEMORY' => 'unlimited',
      'JAVA_HOME' => '/usr',
      },
}

es_instance_conn_validator { 'elk' :
  server => 'localhost',
  port   => '9200',
}

I use this elasticsearch for an ELK stack (look forward a future article!) so thats why I call it elk here.

You should be able to verify the installation with the following:

curl -X GET "localhost:9200/_cluster/health?wait_for_status=yellow&timeout=50s&pretty"

{
  "cluster_name" : "elk",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 7,
  "active_shards" : 7,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 1,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 87.5
}

Single shard deployments will show yellow not green/

Installing Grafana

class {'grafana':
  graphite_host      => "graphite.${::domain}",
  elasticsearch_host => "localhost",
  elasticsearch_port => 9200,
}

I’m using an older version of grafana as I have some older dashboards I don’t want to update.

Dashboards

# Collectd Dashboard
file {'/opt/grafana/app/dashboards/':
  ensure  =>  directory,
  source  => 'puppet:///modules/profile/grafana/',
  recurse => true,
}

As mentioned, while its neat that grafana can store its data in ElasticSearch. I don’t actually use that. When I build a dashboard I export it and save it in the profile/files/grafana directory of my module. This the is recusively copied into grafana’s web directory. This means when I build or rebuild my grafana deployments I don’t have to import elasticsearch data. It also means I can use templates to build dynamic dashboards that are copied in to /opt/grafana/app/dashboards/. These are then available using the following URL e.g. for collectd.json : http://grafana.homeops.tech/#/dashboard/file/collectd.json

Setting up collectd

As I mentioned I wanted to monitor the temperature of the 2011 mac mini server I use as the Proxmox virtualisation box. Its my backup box and so I wanted to see if any of the USB fans I had laying around the house would actually work to cool it. To start graphing the temperature I needed to use the sensors package on debian/proxmox.

This is as simple as appling this configuration to that node:

class {'collectd::plugin::sensors':
  manage_package => true,
}

This installs the sensors package. I also for good measure (and maybe required) ran the following to detect all sensors on the system.

sudo sensors-detect
sudo sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +53.0°C  (high = +86.0°C, crit = +100.0°C)
Core 0:        +51.0°C  (high = +86.0°C, crit = +100.0°C)
Core 1:        +53.0°C  (high = +86.0°C, crit = +100.0°C)
Core 2:        +46.0°C  (high = +86.0°C, crit = +100.0°C)
Core 3:        +49.0°C  (high = +86.0°C, crit = +100.0°C)
...

I was pleasantly surprised to find most supported by the package.

While collectd doesn’t resolve them , a long term plan is to rewrite the json using this table.

Create the dashboard is simple as creating a dynamic entry for the node as show above.

As luck would have it the only USB fans I had was a gift that someone gave me years ago. It was a LED fan that I had since lost the windows-only install CD for. Given the state of the mini it seemed appropriate.

Interestingly enough even at high load the fan managed to cool the SMC temp sensors ( and the CPU sensors) below the high/crit values.

Suffice to say I have replacement server (intel NUC) on order , however I may keep this mini around given its working. Proxmox supports a cluster with my APC PDU where it would only power on in the event of failure.

I’ll write that up if I get it working.

Run Terraform In Docker

Tue, 21 Apr 2020 00:00:00 +0000

In my previous post on compiling go in docker I noted that it had many advantages. This is also true for tools like terraform. One of the major advantages of say Terraform Cloud is the fact that you can pin or change your terraform version. Terraform like many configuration languages is not backwards compatible. Upgrading code , while easier then say puppet , can be complex when you have multiple state files. Being able to selectively upgrade one terraform configuration while leaving the others pinned allows for more flexibility. Using docker means you don’t have to rely on your local machines copy or version compatibility across your team.

Versions tested
Software	Version	OS
docker	18.0.9	Ubuntu
terraform	12,13,14	Ubuntu

Dockerfile

The main thing to notice here is the ARG version which allows you to pin your docker version. We do a multi-stage build here to copy terraform out of the upstream container.

This dockerfile runs terraform after init. It can take any argument you pass to the regular cli. I have a couple other additions which allow terraform to itself use docker but thats left to the reader (or another future article!).

backend.tvars

In this example I’m using s3 storage for state.

variables.sh

You can send vars in anyway you would like, however in my case I might need some “active” vars like the current commit. This variables file works as a config file for the terraform stack. It is actually the only file you would need to update across multiple repos as well.

functions.sh

This script uses functions to override the local invocations of the terraform command.

Lets break this down…

AWS Credentials

     --network="credentials_network" \
     ...
     --build-arg aws_container_credentials_relative_uri="${AWS_CONTAINER_CREDENTIALS_RELATIVE_URI:?}"
     ...
     --env AWS_CONTAINER_CREDENTIALS_RELATIVE_URI \

When you first configure terraform you use terraform init. This downloads the plugins and potentially creates s3 objects. In the case you store your state remotely is say s3, you also need access to AWS credentials to access that s3 bucket. Because docker only has access to what you say you need to pass in any credentials. You could at this point simply pass the AWS credentials in as build args, however that’s decently insecure if your not using short lived credentials.

Our solution is to use the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI var. This is a standard variable that is used by most of the AWS SDKs. As terraform is simply a wrapper around the golang aws sdk it also supports this. This URL is used to query the link local address 169.254.170.2 for credentials. This way the creds are handed directly to the sdk and not saved in the container. The requirements for getting this up and running are to use amazon-ecs-local-container-endpoints container. There are multiple articles that cover this far better then I would. You can find the two most relevent here and here

This variable is passed either through a build-arg directly but ends up simply being an environmental variable thats present at both terraform init and terraform apply

You may note that I’m not using DOCKER_BUILDKIT in the build step of this container. Unfortunately at the time of this writing docker buildkit does not support attaching a network like the current implementation does. This is I believe scoped but not currently working.

Terraform variables

export TF_VAR_git_commit=$(git rev-parse --short HEAD)
...
for var in ${!TF_VAR*} ; do
  echo "${var}=${!var}" >>.env
done
...
--env-file .env \
...
unset -f ${!TF_VAR*}

These snippets allow you to declare any variable with TF_VAR and have it passed to the container.

This file should be in your .gitignore if it contains sensitive values, consider instead using the secure AWS credentials to bootstrap access to secrets.

apply.sh

Due the function being named terraform you, can simply remove the source line for the functions and this is essentially just a generic wrapper around the terraform commands. While init is handled differently e.g. its a build vs a run all other terraform commands act identically to the command line. If a team member installed the latest version of terraform, they will not “accidentally” upgrade your state by doing an apply. The terraform version is pinned to the TERRAFORM_VERSION var we declared. This means this is likely a drop in replacement for any existing script you might have.

Given the caching aspects of docker this can also make things slightly more consistent for plugins. However until build kit supports docker networks, we won’t be able to use the --mount flag to prevent plugins from being downloaded on multiple containers.

Terraform cloud

If you find this configuration useful then you likely will enjoy migrating to Terraform Cloud as it supports pinned versions of terraform and shared states right out of the box. I normally try run most things in TFC when possible , however some builds need to run locally. E.g. Docker in Docker style builds.

Compile Go In Docker

Sun, 15 Mar 2020 00:00:00 +0000

I have recently been moving almost all of my workflows to docker. This makes allot of sense over time as things like your development environment can change from laptop to laptop or os to os. While I think something like NixOs is a great step forward in the direction of a common development and delivery platform. I still tend to fallback to docker when building simple shell scripts that I need to share with members of my team. Running things in docker can mean that you can also pin the version of the given software or libraries which has some huge stability and repeatability implications over time. Something many people miss when sharing assets among each other.

Versions tested
Software	Version	OS
docker	20.10.2	Ubuntu

In an upcoming article I will cover how I use this same trick to selectively upgrade terraform code across a suite of lambdas. In the case of Golang, using docker has the major benefits e.g. its likely a container is my final output as well as my compile time environment. Most of the code I write ends up in one way or another on a serverless platform like AWS Lambda. Lambdas recently started support docker containers in liue of the standard zip file we had all grown accustom to. This means I can use the environment to compile my code and then the resultant container to deploy it.

Dockerfile

 FROM golang:latest
 
 ARG target
 
 # Go build configuration
 ENV GO111MODULE="on"
 ENV GOPRIVATE="github.com/homeops-tech/slackbot"
 ENV GOOS="linux"
 ENV CGO_ENABLED="0"
 # Setup our container
 RUN mkdir /src
 COPY . /src/
 WORKDIR /src
 
 # Create the module (refresh with rm)
 RUN test -f go.mod || go mod init "github.com/homeops-tech/slackbot"
 
 # Setup SSH
 RUN git config --global url."git@github.com:".insteadOf "https://github.com/"
 RUN mkdir -p /root/.ssh/
 RUN ssh-keyscan github.com >> /root/.ssh/known_hosts
 
 # Download go modules
 RUN  --mount=type=ssh go mod download -x
 COPY . .
 
 # Build the module
 RUN --mount=type=cache,target=/root/.cache/go-build go build -ldflags="-s -w" ${target}

Lets break this down, the GOPRIVATE directive here is due to the fact that the repo/module we are using is private on Github.

RUN git config --global url."git@github.com:".insteadOf "https://github.com/"
RUN mkdir -p /root/.ssh/
RUN ssh-keyscan github.com >> /root/.ssh/known_hosts

This allows use to rewrite the http url to ssh and download the known host keys during build.

This can get quite abstract but you need to authenticate inside the container to these repos. This done normally by ssh agents. see the makefile example below.

Using Buildkit

RUN  --mount=type=ssh go mod download -x
COPY . .

Docker’s layer system is great for build processes as it will cache things in each layer and reuse them across builds. In the case of common libraries, especially external we don’t want that. Its just downloading the same cache over and over. Given a language like go has a compilation , this can slow down your development process. The --mount flag uses the new experimental BuildKit function of docker to cache just that layer across invocations.

This requires a special environmental variable you can see in this Makefile example

On linux you may also need to enable experimental (this is in the UI on mac)

echo $'{\n    "experimental": true\n}' | sudo tee /etc/docker/daemon.json;
sudo service docker restart
docker version

You can reset this cache with the Make target below. Be aware that it will also get reset/wonky if you run something like docker system prune -a , however restarting docker will fix it.

.PHONY: clean
 clean:
     @docker builder prune --filter type=exec.cachemount

Makefile

PROJECT="slackbot"

build:
    # Build go docker image with dependencies 
    @SSH_AUTH_SOCK=`launchctl getenv SSH_AUTH_SOCK` DOCKER_BUILDKIT=1 docker build \
        --rm \
        --force-rm \
        --ssh=default \
        --build-arg target='./...' \
        --tag \
        $(PROJECT):latest \
        .
    # Copy updated go.mod and go.sum out of container
    @docker cp $(shell docker create $(PROJECT):latest):/src/go.mod go.mod
    @docker cp $(shell docker create $(PROJECT):latest):/src/go.sum go.sum

.DEFAULT_GOAL := build

.PHONY: build

Dependencies

When using golang with other imported modules, you may create a go.mod file to pin versions of those libs. When using docker I find it useful to have that process happen inside the docker container.

docker cp $(shell docker create $(PROJECT):latest):/src/go.mod go.mod
docker cp $(shell docker create $(PROJECT):latest):/src/go.sum go.sum

This trick means you are getting the “internal” versions that were latest when you did the download. Paired with RUN test -f go.mod || go mod init "github.com/homeops-tech/slackbot" in your Dockerfile allows you pin version between builds. The simple example of this is on the first build, your container runs and downloads the modules and generates the file. The docker cp commands then copies the files out of the container. On the next run, your docker container is pinned to these libraries. You can manually update them by editing the file or if you want to get the latest of all libs, simply delete the files and they will be regenerated.

I find this incredibily helpful when testing newer go versions via the Docker tag FROM golang:latest by editing these two files you can check compatibilty without changing your local go installation.

SSH Agents

You may notice the SSH_AUTH_SOCK var above is pointed towards this command launchctl getenv SSH_AUTH_SOCK. This is essentially a supported hack on docker for mac. Unlike other platforms docker for mac is running virtually and thus it has to actually mount the ssh socket for the system inside that VM. This special path allows you to use this “hack” they built in to pass your ssh credentials via your agent to the docker container. This is very important when using golang with private repos as otherwise you would not be able to download them

Configure Code Manager via Ruby

Sun, 01 Mar 2020 00:00:00 +0000

In my previous post about automating Gitlab with python I showed you how you can add a webhook to Gitlabs repos dynamically. The other side of this configuration in Puppet Enterprise is configuring r10k to point towards that repo. While Gitlab has a nice python library the Puppet Enterprise classifier libraries are written in Ruby. In this quick how to I will show you the code required to configure Puppet Enterprise’s code manager. I actually was the one who wrote this code in Puppet enterprise back in the day and reason was all about automating your PE installations end to end.

Versions tested
Software	Version	OS
puppet	6.19.1	Centos
Puppet Enterprise	2019	Centos
Centos	7.9.2009	Centos

Requirements

This script requires the puppetclassify gem which can be installed via

gem install puppetclassify

This gem is an API wrapper around the rest calls used with the Puppet Enterprise classifier.

Once installed you can modify this script to add your own Github/Gitlab/etc control repo URL.

code_manager_config.rb

#!/usr/bin/env ruby
require 'puppetclassify'

OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE


@classifier_url =  "https://#{ARGV[0]}:4433/classifier-api"

def load_classifier()
  auth_info = {
    'ca_certificate_path' => '/dev/null',
    'token'               => ARGV[1],
  }
  unless @classifier
    @classifier = PuppetClassify.new(@classifier_url, auth_info)
  end
end

def update_pe_master_r10k_remote()
  load_classifier
  groups    = @classifier.groups
  pe_master = groups.get_groups.select { |group| group['name'] == 'PE Master'}.first
  classes   = pe_master['classes']

  puppet_enterprise_profile_master = classes['puppet_enterprise::profile::master']

  puppet_enterprise_profile_master.update(
    puppet_enterprise_profile_master.merge(
      'code_manager_auto_configure' => true,
      'r10k_remote'      => 'git@gitlab.homeops.tech:homeops-tech/control-repo.git',
      'r10k_private_key' => '/etc/puppetlabs/puppetserver/ssh/id-control_repo.rsa',
      'replication_mode' => 'none'
    )
  )
  # I feel like this composition is overkill if this is truly a delta
  pe_master['classes']['puppet_enterprise::profile::master'] = puppet_enterprise_profile_master
  groups.update_group(pe_master)
end

update_pe_master_r10k_remote

You can run this script with the first argument being the node classifier and the second being an RBAC Token. See automating Gitlab with python for an example of using the RBAC API via Python. (it does not currently have a ruby gem)

Note: that you need to ensure the Puppet server has the private key deployed at the location specified. Also: Sorry of the use of master here, its what the Puppet classes are named unfortunately in PE