Puppet Basics

This article is based on puppet 4.x . 3.x versions should work in a similiar way. Earlier versions, not so much. I will be using the master/agent model.

For an overview of how puppet works, see our blog article http://justsomestuff.co.uk/theblog

Installing Puppet

Prerequsites

1. Set up NTP

Accurate time is required because the pupet master acts as a certificate authority for the agent nodes. So make sure the time right and install and configure NTP

ntpdate pool.ntp.org
12 Apr 13:18:44 ntpdate[23034]: step time server 80.82.244.120 offset -27.955391 sec
apt-get update && apt-get -y install ntp

Configure ntp.conf. Select a pool of servers geographically near you (see http://www.pool.ntp.org/en/ ) e.g.

# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board

# on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for

# more information.

server 0.europe.pool.ntp.org
server 1.europe.pool.ntp.org
server 2.europe.pool.ntp.org
server 3.europe.pool.ntp.org
service ntp restart
* Stopping NTP server ntpd                                                                                                                    [ OK ]
* Starting NTP server ntpd        

(In case you need it, NTP troubleshooting here: http://www.justsomestuff.co.uk/wiki/doku.php/general/ntp_probs)

2) Update /etc/hosts and/or DNS and add puppet

127.0.0.1	localhost MicroServer puppet
Install puppet on the master server

I'm going to install the latest stable version from Puppetlabs rather than use the distro version. This example is based on a Linux Mint server BTW

1. Enable puppet repository

MicroServer # wget https://apt.puppetlabs.com/puppetlabs-release-pc1-trusty.deb
--2016-04-12 13:37:10--  https://apt.puppetlabs.com/puppetlabs-release-pc1-trusty.deb
Resolving apt.puppetlabs.com (apt.puppetlabs.com)... 192.155.89.90, 2600:3c03::f03c:91ff:fedb:6b1d
Connecting to apt.puppetlabs.com (apt.puppetlabs.com)|192.155.89.90|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2612 (2.6K) [application/x-debian-package]
Saving to: puppetlabs-release-pc1-trusty.deb
100%[============================================================================================================>] 2,612       --.-K/s   in 0s
2016-04-12 13:37:10 (24.4 MB/s) - puppetlabs-release-pc1-trusty.deb saved [2612/2612]

MicroServer  # dpkg -i puppetlabs-release-pc1-trusty.deb
Selecting previously unselected package puppetlabs-release-pc1.
(Reading database ... 182148 files and directories currently installed.)
Preparing to unpack puppetlabs-release-pc1-trusty.deb ...
Unpacking puppetlabs-release-pc1 (0.9.2-1trusty) ...
Setting up puppetlabs-release-pc1 (0.9.2-1trusty) ...

MicroServer  # sudo apt-get update
Ign http://www.mirrorservice.org rosa InRelease
Ign http://dl.google.com stable InRelease
....
....
Fetched 79.4 kB in 6s (12.2 kB/s)

2. Now that the puppet repository is added, puppet server can be installed

MicroServer  # apt-get install puppetserver

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  puppet-agent
The following NEW packages will be installed
  puppet-agent puppetserver
0 to upgrade, 2 to newly install, 0 to remove and 39 not to upgrade.
Need to get 46.9 MB of archives.
After this operation, 113 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://apt.puppetlabs.com/ trusty/PC1 puppet-agent amd64 1.4.1-1trusty [14.1 MB]
Get:2 http://apt.puppetlabs.com/ trusty/PC1 puppetserver all 2.3.1-1puppetlabs1 [32.8 MB]
Fetched 46.9 MB in 3s (14.6 MB/s)
Selecting previously unselected package puppet-agent.
(Reading database ... 182152 files and directories currently installed.)
Preparing to unpack .../puppet-agent_1.4.1-1trusty_amd64.deb ...
Unpacking puppet-agent (1.4.1-1trusty) ...
Selecting previously unselected package puppetserver.
Preparing to unpack .../puppetserver_2.3.1-1puppetlabs1_all.deb ...
Unpacking puppetserver (2.3.1-1puppetlabs1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up puppet-agent (1.4.1-1trusty) ...
update-rc.d: warning:  start runlevel arguments (none) do not match pxp-agent Default-Start values (2 3 4 5)
update-rc.d: warning:  stop runlevel arguments (none) do not match pxp-agent Default-Stop values (0 1 6)
Processing triggers for ureadahead (0.100.0-16) ...
Setting up puppetserver (2.3.1-1puppetlabs1) ...
usermod: no changes
Processing triggers for ureadahead (0.100.0-16) ...

3. You may want to adjust the amount of memory that is allocated to puppet. By default it's 2GB but I'm just messing around at home so I'm going to change it to 512MB

vi /etc/default/puppetserver
  
change the JAVA_ARGS as follows
  
JAVA_ARGS="-Xms512m -Xmx512m"`
 
service puppetserver restart
MicroServer default # ps -ef | grep puppet
puppet   27205     1 99 14:31 ?        00:02:08 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/dev/urandom -Xms512m -Xmx512m -XX:MaxPermSize=256m -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg

4. Ensure puppetserver starts on a reboot - we can use puppet to do this :)

MicroServer etc # /opt/puppetlabs/bin/puppet resource service puppetserver ensure=running enable=true
Notice: /Service[puppetserver]/enable: enable changed 'false' to 'true'
service { 'puppetserver':
  ensure => 'running',
  enable => 'true',
}

5. and the agent (puppet)

MicroServer etc # /opt/puppetlabs/bin/puppet resource service puppet ensure=running enable=true
service { 'puppet':
  ensure => 'running',
  enable => 'true',
}

(So just for clarity, with 4.x the server process is called puppetserver and the agent process, puppet)

To ensure secure communications between the master and agents, Puppet uses SSL certificates to authenticate hosts. Each puppet client creates a certificate request and submits it to the server. The server signs each request with the certificate (certname) created during install. This creates a closed system containing an authorized public/private key pair for each client.

The certname is the name of the Puppet Certificate Authority, which signs every client certificate. If certname ever changes, you have to delete and recreate every client certificate. It is therefore a good idea to set it to something abstract so you can move your server around without any authentication problems in the future.

Install puppet on a client server

I'm going to install the puppet agent on one of my laptops

1. Update repository

root@X220-laptop:~# cd ~ && wget https://apt.puppetlabs.com/puppetlabs-release-pc1-trusty.deb
--2016-04-15 15:35:24--  https://apt.puppetlabs.com/puppetlabs-release-pc1-trusty.deb
Resolving apt.puppetlabs.com (apt.puppetlabs.com)... 192.155.89.90, 2600:3c03::f03c:91ff:fedb:6b1d
Connecting to apt.puppetlabs.com (apt.puppetlabs.com)|192.155.89.90|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2612 (2.6K) [application/x-debian-package]
Saving to: ‘puppetlabs-release-pc1-trusty.deb’

100%[=======================================================>] 2,612       --.-K/s   in 0s      

2016-04-15 15:35:24 (32.1 MB/s) - ‘puppetlabs-release-pc1-trusty.deb’ saved [2612/2612]

root@X220-laptop:~# dpkg -i puppetlabs-release-pc1-trusty.deb
Selecting previously unselected package puppetlabs-release-pc1.
(Reading database ... 196019 files and directories currently installed.)
Preparing to unpack puppetlabs-release-pc1-trusty.deb ...
Unpacking puppetlabs-release-pc1 (0.9.2-1trusty) ...
Setting up puppetlabs-release-pc1 (0.9.2-1trusty) ...
root@X220-laptop:~# apt-get update
Ign http://archive.canonical.com trusty InRelease
Get:1 http://archive.canonical.com trusty Release.gpg [933 B]                                   
Get:2 http://archive.canonical.com trusty Release [9,359 B]                                    
....
...
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US                              
Fetched 6,359 kB in 9s (689 kB/s)                                                               
Reading package lists... Done

2. Install agent

root@X220-laptop:~# apt-get install puppet-agent
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  puppet-agent
0 upgraded, 1 newly installed, 0 to remove and 15 not upgraded.
Need to get 14.1 MB of archives.
After this operation, 74.8 MB of additional disk space will be used.
Get:1 http://apt.puppetlabs.com/ trusty/PC1 puppet-agent amd64 1.4.1-1trusty [14.1 MB]
Fetched 14.1 MB in 1s (7,038 kB/s)       
Selecting previously unselected package puppet-agent.
(Reading database ... 196023 files and directories currently installed.)
Preparing to unpack .../puppet-agent_1.4.1-1trusty_amd64.deb ...
Unpacking puppet-agent (1.4.1-1trusty) ...
Processing triggers for ureadahead (0.100.0-16) ...
ureadahead will be reprofiled on next reboot
Setting up puppet-agent (1.4.1-1trusty) ...
update-rc.d: warning:  start runlevel arguments (none) do not match pxp-agent Default-Start values (2 3 4 5)
update-rc.d: warning:  stop runlevel arguments (none) do not match pxp-agent Default-Stop values (0 1 6)
Processing triggers for ureadahead (0.100.0-16) ... 

3. Add address of puppet master to /etc/hosts (or DNS)

 192.168.11.9	microserver puppet

4. Make sure time is accurate. As this is a laptop, I'm going to use chrony

 root@X220-laptop:/opt/puppetlabs# apt-get install chrony
 Reading package lists... Done
 Building dependency tree       
 ....
 ....
 Setting up chrony (1.29-1) ...
 
 Creating config file /etc/chrony/chrony.conf with new version
 Starting /usr/sbin/chronyd...
 chronyd is running and online.
 Processing triggers for libc-bin (2.19-0ubuntu6.7) ...
 Processing triggers for ureadahead (0.100.0-16) ...
 #
 root@X220-laptop:/opt/puppetlabs# chronyc sources
 210 Number of sources = 4
 MS Name/IP address         Stratum Poll Reach LastRx Last sample
 ===============================================================================
 ^+ 85.93.7.12                    3   8    37   183  -2721us[-5314us] +/-   85ms
 ^* alpha.rueckgr.at              2   8    37   183  -4194us[-6787us] +/-   74ms
 ^+ time.ooonet.ru                2   8    37   182  -1546us[-1546us] +/-  109ms
 ^+ fr1.tomhek.net                3   8    37   183  +4760us[+4760us] +/-   74ms
 root@X220-laptop:/opt/puppetlabs# 

5. Make sure agent is running

 root@X220-laptop:~# /opt/puppetlabs/bin/puppet resource service puppet ensure=running enable=trueNotice: /Service[puppet]/ensure: ensure changed 'stopped' to 'running'
 service { 'puppet':
   ensure => 'running',
   enable => 'true',
 }
 root@X220-laptop:~# ps -ef | grep puppet
 root      4000     1 10 15:42 ?        00:00:01 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent

The first time Puppet runs on an agent node, it will send a certificate signing request to the Puppet master. Before Puppet Server will be able to communicate with and control the agent node, it must sign that particular agent node's certificate.

List the current certificates to be signed

 MicroServer ssl # /opt/puppetlabs/bin/puppet cert list
   "x220-laptop" (SHA256) 59:64:7E:D0:80:06:76:71:61:6C:B5:5D:FF:E0:6E:7C:E5:79:4A:C9:92:97:CC:C2:D6:2B:85:0E:56:FF:47:02

Sign the laptop certificate

 MicroServer ssl # /opt/puppetlabs/bin/puppet cert sign x220-laptop
 Notice: Signed certificate request for x220-laptop
 Notice: Removing file Puppet::SSL::CertificateRequest x220-laptop at '/etc/puppetlabs/puppet/ssl/ca/requests/x220-laptop.pem'

Check it's OK

 MicroServer ssl # /opt/puppetlabs/bin/puppet cert list --all
 + "microserver" (SHA256) 46:0B:76:51:14:6B:21:11:4F:88:F8:5B:68:F7:5A:E3:A9:26:82:05:51:33:67:1A:53:38:02:D0:F9:01:C8:B2 (alt names: "DNS:puppet", "DNS:microserver")
 + "x220-laptop" (SHA256) CE:8F:8D:16:24:25:F8:2F:D6:31:6E:7B:BE:AE:0F:54:7F:84:92:3D:11:90:C0:E2:E2:1B:E1:E1:78:3C:2C:2E

At this point the agent is ready to connect to the puppet master and download the catalog. The catalog is a distilled, compiled version of the Puppet policy which contains only those directives necessary for the client node. Currently this won't contain much as we haven't created anything yet. This is the next step.

Environments

However, before we start creating content, there's one more thing to consider, environments. The default puppet environment is called production. With the 4.x version from Puppetlabs, this is located at:

 /etc/puppetlabs/code/environments/production

It is definitely worth setting up a test environment. Why? One reason is that by default puppet agents poll at regular intervals looking for updates. Say you created some new content that had an error. Before you knew it, this could be propogated to hundreds of production servers, causing some major production issue and a rather cross manager. Best to try in test first before rolling out to production.

I decided to put my laptop into the test environmemt. On the master I created the following directory:

 /etc/puppetlabs/code/environments/test

Then on the laptop, I updated /etc/puppetlabs/puppet/puppet.conf so it looks as follows:

 [main]
 certname = x220-laptop
 server = puppet
 environment = test
 runinterval = 1h

Restart the puppet agent and the laptop is in the test puppet environment.

Manifests

The basis of puppet are files called manifests. They contain the instructions telling puppet what to do. To start with I want to roll out bleachbit to all my PCs and laptops. First in the test environment. My test directory contains the following:

 MicroServer test # pwd
/etc/puppetlabs/code/environments/test
MicroServer test # ls -l
total 12
-rw-r--r-- 1 root root  935 Apr 19 10:48 environment.conf
drwxr-xr-x 2 root root 4096 Apr 25 12:46 manifests
drwxr-xr-x 3 root root 4096 Apr 19 10:57 modules

Now I need to create a module for bleachbit

 mkdir -p modules/bleachbit/manifests

The base module file is called init.pp. I create one for bleachbit in modules/bleachbit/manifests . This is what it contains:

# Manage bleachbit utility
class bleachbit {
  package { 'bleachbit':
    ensure => 'installed',
  }
}

So what does this all mean? You will notice that the first (non comment) line starts with class. This is a type of file used in modules that is refered to later, i.e. a manifest higher up in the chain calls it. Next tell puppet that this class refers to the package bleachbit. The next line tells puppet to make sure it's installed and if it isn't, install it.

One question you may have is how does puppet know where to get the package from and how to install it. This is where facter comes in. When you install puppet it runs a program called facter. It will perform multiple checks about the hardware and OS it's running on. Using this information, puppet will know my OS is Ubuntu based and therefore use apt-get. The information facter gathers can be used for all types of things and it's what maked puppet so versatile.

I now create a mainfest to call this bleachbit module.

cd ../../../modules
MicroServer manifests # pwd
/etc/puppetlabs/code/environments/test/manifests
MicroServer manifests # cat nodes.pp
node 'x220-laptop' {
    include bleachbit
}

So, for node x220-laptop in the test environment, the bleachbit module will be called which will in turn install bleachbit if it's not already there. To get puppet to install it straight away (instead of waiting to the next time the agent checks), on node x220-laptop run:

puppet agent --test

The agent will then check the master for any updates and install bleachbit.

Now in production, but there after bleachbit is installed, I want to run it on a regular basis via cron. As before, I create a module, this time in /etc/puppetlabs/code/environments/prodiction. First a slightly modified bleachbit class (this time called install.pp)

MicroServer manifests # pwd
/etc/puppetlabs/code/environments/production/modules/bleachbit/manifests
MicroServer manifests # cat install.pp
class bleachbit::install {
  package { 'bleachbit':
    ensure => installed,
    notify => Class['bleachbit::cronadd'],
  }
}

And then my cron configuration

class bleachbit::cronadd {
  cron { 'bleachbit':
    ensure  => present,
    command => '/usr/bin/bleachbit',
    user    => 'root',
    weekday => '*',
    minute  => '10',
    hour    => '11',
    subscribe => Package['bleachbit'],
  }
}

And then my init.pp

class bleachbit {
  include bleachbit::install
  include bleachbit::cronadd
}

install.pp and cronadd.pp are effectively subclasses of the class bleachbit. So when the bleachbit class is called, it will in turn call install.pp and cronadd.pp. The notify and subscribe statements just ensure there's a dependency between the two (as there's not much point adding to bleachbit to cron if it ain't installed!)

Virtual resources

For my final task of this exercise, I'm going to manage users via puppet. For that we're going to use some advanced puppet techniques. A defined resource is a bunch of code that can be called various times with different parameters, like a bash function. A virtual resource is a resource that is ignored by puppet unless it is specifically called, i.e. it's not added to the catalog. The reason you may want that is because resources can only be configured once. However, a virtual resource, although only stated once, can be realized (called) multiple times. This can be useful for things like managing users. For example, you may divide users into two groups like developers and testers. However, some users may be members of both so without virtual resources you'd need to use both classes for nodes that applied to. Virtual resources allows you to get around this.

We'll use a virtual resource in this example. First create a module.

MicroServer manifests # pwd
/etc/puppetlabs/code/environments/production/modules/users/manifests
MicroServer manifests # cat virtual.pp 
define users::virtual ($uid,$realname,$pass,$userfiles = false) {
  
  user { $title:
    ensure            =>  'present',
    uid               =>  $uid,
    gid               =>  $title,
    shell             =>  '/bin/bash',
    home              =>  "/home/${title}",
    comment           =>  $realname,
    password          =>  $pass,
    managehome        =>  true,
    require           =>  Group[$title],
  }

  group { $title:
    gid               =>  $uid,
  }

  # userfiles
  if $userfiles == false {
    # just create the directory
    file { "/home/${title}":
      ensure  => 'directory',
      mode    => '0755',
      owner   => $title,
      group   => $title,
      require => User["$title"]
    }
  } else {
    # copy in all the files in the subdirectory
    file { "/home/${title}":
      recurse => true,
      mode    => '0755',
      owner   => $title,
      group   => $title,
      source  => "puppet:///modules/users/${title}",
      require => User["$title"]
    }
  }

}

A few things to note. The define statement signifies this file is a resource. As can be seen, it accepts variables. We will see later how they are passed. The couple of sections define the user and group attributes. The next section contains some if else logic. If $userfiles is false (which is set to the default on the first line), just the home directory is created, else the home directory is created and some files are copied into it. This is what will be copied:

MicroServer users # pwd
/etc/puppetlabs/code/environments/production/modules/users
MicroServer users # find files
files
files/tony
files/tony/.bashrc

So a custom .bashrc file.

The init.pp for the users module contains the following:

MicroServer manifests # cat init.pp
class users {

  @users::virtual { 'tony':
    userfiles       =>  true,
    uid             =>  1000,
    realname        =>  'Tony',
    pass            =>  '$6$QzIYITBc$RCs75ujfU/9XgffYYBn45VEyk4Zr4jJadzd9URVa1PFAMbOOCRvfUPulLtBlYHAR/OYcasrAQrgpAN33/RFkf/',
  }
  @users::virtual { 'melissa':
    uid             =>  1003,
    realname        =>  'Melissa',
    pass            =>  '$6$7ajWS2Fh$g0Apa3rwWzq0urAgvbYilw4XB09.cHibX0JGI6sE8XkMGvgrJIsrKcKWscnrBxHXNfITLtT2lL.eXQDua/xe40',
  }
  @users::virtual { 'daniel':
    uid             =>  1004,
    realname        =>  'Daniel',
    pass            =>  '$6$DuIkulz3$ZD/y5vV.FjrRCGj97V5l6Hv/IWt7x8RxL66Y5H.Uw9KdYaF/IE8D3kAYgmhvAdCQBrEmWm13miD4zmSFGcrUG0',
  }
  @users::virtual { 'jamie':
    uid             =>  1005,
    realname        =>  'Jamie',
    pass            =>  '$6$DuIkulz3$ZD/y5vV.FjrRCGj97V5l6Hv/IWt7x8RxL66Y5H.Uw9KdYaF/IE8D3kAYgmhvAdCQBrEmWm13miD4zmSFGcrUG0',
  }

As can be seen, the init.pp contains the variables that will be passed to the users::virtual resource. For tony, userfiles is set to true so the custom .bashrc file is copied for this user.The @ symbol signnifies that they're virtual resources. site.pp contains

node default {
}

node 'Microserver' {
    include bleachbit
    include users
    realize (Users::Virtual['tony'])
    realize (Users::Virtual['melissa'])
    realize (Users::Virtual['daniel'])
    realize (Users::Virtual['jamie'])
}

For virtual resources the realize statement is used. This is the result when puppet agent –test is run:

MicroServer manifests # puppet agent --test
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for microserver
Info: Applying configuration version '1462443980'
Notice: /Stage[main]/Users/Users::Virtual[tony]/User[tony]/comment: comment changed 'tony,,,' to 'Tony'
Notice: /Stage[main]/Users/Users::Virtual[melissa]/Group[melissa]/ensure: created
Notice: /Stage[main]/Users/Users::Virtual[melissa]/User[melissa]/ensure: created
Notice: /Stage[main]/Users/Users::Virtual[melissa]/File[/home/melissa]/owner: owner changed '1002' to 'melissa'
Notice: /Stage[main]/Users/Users::Virtual[daniel]/Group[daniel]/ensure: created
Notice: /Stage[main]/Users/Users::Virtual[daniel]/User[daniel]/ensure: created
Notice: /Stage[main]/Users/Users::Virtual[jamie]/Group[jamie]/ensure: created
Notice: /Stage[main]/Users/Users::Virtual[jamie]/User[jamie]/ensure: created
Notice: Applied catalog in 12.98 seconds

So that's our whistle stop tour of puppet. It barely scratches the surface but it does demonstrate the power of puppet .

Recent Changes