~~socialite:icon reddit googleplus stumbleupon twitter~~ ====Puppet Basics==== This article is based on puppet **4.x** . 3.x versions should work in a similiar way. Earlier versions, not so much. I will be using the master/agent model. For an overview of how puppet works, see our blog article [[http://justsomestuff.co.uk/theblog]] ===Installing Puppet=== ==Prerequsites== 1. **Set up NTP** Accurate time is required because the pupet master acts as a certificate authority for the agent nodes. So make sure the time right and install and configure NTP ntpdate pool.ntp.org 12 Apr 13:18:44 ntpdate[23034]: step time server 80.82.244.120 offset -27.955391 sec apt-get update && apt-get -y install ntp Configure ntp.conf. Select a pool of servers geographically near you (see http://www.pool.ntp.org/en/ ) e.g. # Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board # on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for # more information. server 0.europe.pool.ntp.org server 1.europe.pool.ntp.org server 2.europe.pool.ntp.org server 3.europe.pool.ntp.org service ntp restart * Stopping NTP server ntpd [ OK ] * Starting NTP server ntpd (In case you need it, NTP troubleshooting here: http://www.justsomestuff.co.uk/wiki/doku.php/general/ntp_probs) 2) **Update /etc/hosts and/or DNS and add puppet** 127.0.0.1 localhost MicroServer puppet ==Install puppet on the master server== I'm going to install the latest stable version from Puppetlabs rather than use the distro version. This example is based on a Linux Mint server BTW... 1. **Enable puppet repository** MicroServer # wget https://apt.puppetlabs.com/puppetlabs-release-pc1-trusty.deb --2016-04-12 13:37:10-- https://apt.puppetlabs.com/puppetlabs-release-pc1-trusty.deb Resolving apt.puppetlabs.com (apt.puppetlabs.com)... 192.155.89.90, 2600:3c03::f03c:91ff:fedb:6b1d Connecting to apt.puppetlabs.com (apt.puppetlabs.com)|192.155.89.90|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 2612 (2.6K) [application/x-debian-package] Saving to: puppetlabs-release-pc1-trusty.deb 100%[============================================================================================================>] 2,612 --.-K/s in 0s 2016-04-12 13:37:10 (24.4 MB/s) - puppetlabs-release-pc1-trusty.deb saved [2612/2612] MicroServer # dpkg -i puppetlabs-release-pc1-trusty.deb Selecting previously unselected package puppetlabs-release-pc1. (Reading database ... 182148 files and directories currently installed.) Preparing to unpack puppetlabs-release-pc1-trusty.deb ... Unpacking puppetlabs-release-pc1 (0.9.2-1trusty) ... Setting up puppetlabs-release-pc1 (0.9.2-1trusty) ... MicroServer # sudo apt-get update Ign http://www.mirrorservice.org rosa InRelease Ign http://dl.google.com stable InRelease .... .... Fetched 79.4 kB in 6s (12.2 kB/s) 2. Now that the puppet repository is added, **puppet server can be installed** MicroServer # apt-get install puppetserver Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: puppet-agent The following NEW packages will be installed puppet-agent puppetserver 0 to upgrade, 2 to newly install, 0 to remove and 39 not to upgrade. Need to get 46.9 MB of archives. After this operation, 113 MB of additional disk space will be used. Do you want to continue? [Y/n] y Get:1 http://apt.puppetlabs.com/ trusty/PC1 puppet-agent amd64 1.4.1-1trusty [14.1 MB] Get:2 http://apt.puppetlabs.com/ trusty/PC1 puppetserver all 2.3.1-1puppetlabs1 [32.8 MB] Fetched 46.9 MB in 3s (14.6 MB/s) Selecting previously unselected package puppet-agent. (Reading database ... 182152 files and directories currently installed.) Preparing to unpack .../puppet-agent_1.4.1-1trusty_amd64.deb ... Unpacking puppet-agent (1.4.1-1trusty) ... Selecting previously unselected package puppetserver. Preparing to unpack .../puppetserver_2.3.1-1puppetlabs1_all.deb ... Unpacking puppetserver (2.3.1-1puppetlabs1) ... Processing triggers for ureadahead (0.100.0-16) ... Setting up puppet-agent (1.4.1-1trusty) ... update-rc.d: warning: start runlevel arguments (none) do not match pxp-agent Default-Start values (2 3 4 5) update-rc.d: warning: stop runlevel arguments (none) do not match pxp-agent Default-Stop values (0 1 6) Processing triggers for ureadahead (0.100.0-16) ... Setting up puppetserver (2.3.1-1puppetlabs1) ... usermod: no changes Processing triggers for ureadahead (0.100.0-16) ... 3. **You may want to adjust the amount of memory that is allocated to puppet**. By default it's 2GB but I'm just messing around at home so I'm going to change it to 512MB vi /etc/default/puppetserver change the JAVA_ARGS as follows JAVA_ARGS="-Xms512m -Xmx512m"` service puppetserver restart MicroServer default # ps -ef | grep puppet puppet 27205 1 99 14:31 ? 00:02:08 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/dev/urandom -Xms512m -Xmx512m -XX:MaxPermSize=256m -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg 4. **Ensure puppetserver starts on a reboot** - we can use puppet to do this :) MicroServer etc # /opt/puppetlabs/bin/puppet resource service puppetserver ensure=running enable=true Notice: /Service[puppetserver]/enable: enable changed 'false' to 'true' service { 'puppetserver': ensure => 'running', enable => 'true', } 5. and the **agent** (puppet) MicroServer etc # /opt/puppetlabs/bin/puppet resource service puppet ensure=running enable=true service { 'puppet': ensure => 'running', enable => 'true', } (So just for clarity, with 4.x the server process is called puppetserver and the agent process, puppet) To ensure secure communications between the master and agents, Puppet uses SSL certificates to authenticate hosts. Each puppet client creates a certificate request and submits it to the server. The server signs each request with the certificate (certname) created during install. This creates a closed system containing an authorized public/private key pair for each client. The certname is the name of the Puppet Certificate Authority, which signs every client certificate. If certname ever changes, you have to delete and recreate every client certificate. It is therefore a good idea to set it to something abstract so you can move your server around without any authentication problems in the future. ==Install puppet on a client server== I'm going to install the puppet agent on one of my laptops 1. **Update repository** root@X220-laptop:~# cd ~ && wget https://apt.puppetlabs.com/puppetlabs-release-pc1-trusty.deb --2016-04-15 15:35:24-- https://apt.puppetlabs.com/puppetlabs-release-pc1-trusty.deb Resolving apt.puppetlabs.com (apt.puppetlabs.com)... 192.155.89.90, 2600:3c03::f03c:91ff:fedb:6b1d Connecting to apt.puppetlabs.com (apt.puppetlabs.com)|192.155.89.90|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 2612 (2.6K) [application/x-debian-package] Saving to: ‘puppetlabs-release-pc1-trusty.deb’ 100%[=======================================================>] 2,612 --.-K/s in 0s 2016-04-15 15:35:24 (32.1 MB/s) - ‘puppetlabs-release-pc1-trusty.deb’ saved [2612/2612] root@X220-laptop:~# dpkg -i puppetlabs-release-pc1-trusty.deb Selecting previously unselected package puppetlabs-release-pc1. (Reading database ... 196019 files and directories currently installed.) Preparing to unpack puppetlabs-release-pc1-trusty.deb ... Unpacking puppetlabs-release-pc1 (0.9.2-1trusty) ... Setting up puppetlabs-release-pc1 (0.9.2-1trusty) ... root@X220-laptop:~# apt-get update Ign http://archive.canonical.com trusty InRelease Get:1 http://archive.canonical.com trusty Release.gpg [933 B] Get:2 http://archive.canonical.com trusty Release [9,359 B] .... ... Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US Fetched 6,359 kB in 9s (689 kB/s) Reading package lists... Done 2. **Install agent** root@X220-laptop:~# apt-get install puppet-agent Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: puppet-agent 0 upgraded, 1 newly installed, 0 to remove and 15 not upgraded. Need to get 14.1 MB of archives. After this operation, 74.8 MB of additional disk space will be used. Get:1 http://apt.puppetlabs.com/ trusty/PC1 puppet-agent amd64 1.4.1-1trusty [14.1 MB] Fetched 14.1 MB in 1s (7,038 kB/s) Selecting previously unselected package puppet-agent. (Reading database ... 196023 files and directories currently installed.) Preparing to unpack .../puppet-agent_1.4.1-1trusty_amd64.deb ... Unpacking puppet-agent (1.4.1-1trusty) ... Processing triggers for ureadahead (0.100.0-16) ... ureadahead will be reprofiled on next reboot Setting up puppet-agent (1.4.1-1trusty) ... update-rc.d: warning: start runlevel arguments (none) do not match pxp-agent Default-Start values (2 3 4 5) update-rc.d: warning: stop runlevel arguments (none) do not match pxp-agent Default-Stop values (0 1 6) Processing triggers for ureadahead (0.100.0-16) ... 3. **Add address of puppet master **to /etc/hosts (or DNS) 192.168.11.9 microserver puppet 4. **Make sure time is accurate.** As this is a laptop, I'm going to use chrony root@X220-laptop:/opt/puppetlabs# apt-get install chrony Reading package lists... Done Building dependency tree .... .... Setting up chrony (1.29-1) ... Creating config file /etc/chrony/chrony.conf with new version Starting /usr/sbin/chronyd... chronyd is running and online. Processing triggers for libc-bin (2.19-0ubuntu6.7) ... Processing triggers for ureadahead (0.100.0-16) ... # root@X220-laptop:/opt/puppetlabs# chronyc sources 210 Number of sources = 4 MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^+ 85.93.7.12 3 8 37 183 -2721us[-5314us] +/- 85ms ^* alpha.rueckgr.at 2 8 37 183 -4194us[-6787us] +/- 74ms ^+ time.ooonet.ru 2 8 37 182 -1546us[-1546us] +/- 109ms ^+ fr1.tomhek.net 3 8 37 183 +4760us[+4760us] +/- 74ms root@X220-laptop:/opt/puppetlabs# 5. **Make sure agent is running ** root@X220-laptop:~# /opt/puppetlabs/bin/puppet resource service puppet ensure=running enable=trueNotice: /Service[puppet]/ensure: ensure changed 'stopped' to 'running' service { 'puppet': ensure => 'running', enable => 'true', } root@X220-laptop:~# ps -ef | grep puppet root 4000 1 10 15:42 ? 00:00:01 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent The first time Puppet runs on an agent node, it will send a certificate signing request to the Puppet master. Before Puppet Server will be able to communicate with and control the agent node, it must sign that particular agent node's certificate. **List** the current certificates to be signed MicroServer ssl # /opt/puppetlabs/bin/puppet cert list "x220-laptop" (SHA256) 59:64:7E:D0:80:06:76:71:61:6C:B5:5D:FF:E0:6E:7C:E5:79:4A:C9:92:97:CC:C2:D6:2B:85:0E:56:FF:47:02 **Sign** the laptop certificate MicroServer ssl # /opt/puppetlabs/bin/puppet cert sign x220-laptop Notice: Signed certificate request for x220-laptop Notice: Removing file Puppet::SSL::CertificateRequest x220-laptop at '/etc/puppetlabs/puppet/ssl/ca/requests/x220-laptop.pem' **Check** it's OK MicroServer ssl # /opt/puppetlabs/bin/puppet cert list --all + "microserver" (SHA256) 46:0B:76:51:14:6B:21:11:4F:88:F8:5B:68:F7:5A:E3:A9:26:82:05:51:33:67:1A:53:38:02:D0:F9:01:C8:B2 (alt names: "DNS:puppet", "DNS:microserver") + "x220-laptop" (SHA256) CE:8F:8D:16:24:25:F8:2F:D6:31:6E:7B:BE:AE:0F:54:7F:84:92:3D:11:90:C0:E2:E2:1B:E1:E1:78:3C:2C:2E At this point the agent is ready to connect to the puppet master and download the catalog. The catalog is a distilled, compiled version of the Puppet policy which contains only those directives necessary for the client node. Currently this won't contain much as we haven't created anything yet. This is the next step. ===Environments=== However, before we start creating content, there's one more thing to consider, environments. The default puppet environment is called **production**. With the 4.x version from Puppetlabs, this is located at: /etc/puppetlabs/code/environments/production It is definitely worth setting up a test environment. Why? One reason is that by default puppet agents poll at regular intervals looking for updates. Say you created some new content that had an error. Before you knew it, this could be propogated to hundreds of production servers, causing some major production issue and a rather cross manager. Best to try in test first before rolling out to production. I decided to put my laptop into the test environmemt. On the master I created the following directory: /etc/puppetlabs/code/environments/test Then on the laptop, I updated **/etc/puppetlabs/puppet/puppet.conf** so it looks as follows: [main] certname = x220-laptop server = puppet environment = test runinterval = 1h **Restart** the puppet agent and the laptop is in the test puppet environment. ===Manifests=== The basis of puppet are files called manifests. They contain the instructions telling puppet what to do. To start with I want to roll out bleachbit to all my PCs and laptops. First in the test environment. My test directory contains the following: MicroServer test # pwd /etc/puppetlabs/code/environments/test MicroServer test # ls -l total 12 -rw-r--r-- 1 root root 935 Apr 19 10:48 environment.conf drwxr-xr-x 2 root root 4096 Apr 25 12:46 manifests drwxr-xr-x 3 root root 4096 Apr 19 10:57 modules Now I need to create a module for bleachbit mkdir -p modules/bleachbit/manifests The base module file is called init.pp. I create one for bleachbit in modules/bleachbit/manifests . This is what it contains: # Manage bleachbit utility class bleachbit { package { 'bleachbit': ensure => 'installed', } } So what does this all mean? You will notice that the first (non comment) line starts with class. This is a type of file used in modules that is refered to later, i.e. a manifest higher up in the chain calls it. Next tell puppet that this class refers to the package bleachbit. The next line tells puppet to make sure it's installed and if it isn't, install it. One question you may have is how does puppet know where to get the package from and how to install it. This is where **facter** comes in. When you install puppet it runs a program called facter. It will perform multiple checks about the hardware and OS it's running on. Using this information, puppet will know my OS is Ubuntu based and therefore use apt-get. The information facter gathers can be used for all types of things and it's what maked puppet so versatile. I now create a mainfest to call this bleachbit module. cd ../../../modules MicroServer manifests # pwd /etc/puppetlabs/code/environments/test/manifests MicroServer manifests # cat nodes.pp node 'x220-laptop' { include bleachbit } So, for node x220-laptop in the test environment, the bleachbit module will be called which will in turn install bleachbit if it's not already there. To get puppet to install it straight away (instead of waiting to the next time the agent checks), on node x220-laptop run: puppet agent --test The agent will then check the master for any updates and install bleachbit. Now in production, but there after bleachbit is installed, I want to run it on a regular basis via cron. As before, I create a module, this time in /etc/puppetlabs/code/environments/prodiction. First a slightly modified bleachbit class (this time called install.pp) MicroServer manifests # pwd /etc/puppetlabs/code/environments/production/modules/bleachbit/manifests MicroServer manifests # cat install.pp class bleachbit::install { package { 'bleachbit': ensure => installed, notify => Class['bleachbit::cronadd'], } } And then my cron configuration class bleachbit::cronadd { cron { 'bleachbit': ensure => present, command => '/usr/bin/bleachbit', user => 'root', weekday => '*', minute => '10', hour => '11', subscribe => Package['bleachbit'], } } And then my init.pp class bleachbit { include bleachbit::install include bleachbit::cronadd } install.pp and cronadd.pp are effectively subclasses of the class bleachbit. So when the bleachbit class is called, it will in turn call install.pp and cronadd.pp. The notify and subscribe statements just ensure there's a dependency between the two (as there's not much point adding to bleachbit to cron if it ain't installed!) ===Virtual resources=== For my final task of this exercise, I'm going to manage users via puppet. For that we're going to use some advanced puppet techniques. A defined resource is a bunch of code that can be called various times with different parameters, like a bash function. A virtual resource is a resource that is ignored by puppet unless it is specifically called, i.e. it's not added to the catalog. The reason you may want that is because resources can only be configured once. However, a virtual resource, although only stated once, can be realized (called) multiple times. This can be useful for things like managing users. For example, you may divide users into two groups like developers and testers. However, some users may be members of both so without virtual resources you'd need to use both classes for nodes that applied to. Virtual resources allows you to get around this. We'll use a virtual resource in this example. First create a module. MicroServer manifests # pwd /etc/puppetlabs/code/environments/production/modules/users/manifests MicroServer manifests # cat virtual.pp define users::virtual ($uid,$realname,$pass,$userfiles = false) { user { $title: ensure => 'present', uid => $uid, gid => $title, shell => '/bin/bash', home => "/home/${title}", comment => $realname, password => $pass, managehome => true, require => Group[$title], } group { $title: gid => $uid, } # userfiles if $userfiles == false { # just create the directory file { "/home/${title}": ensure => 'directory', mode => '0755', owner => $title, group => $title, require => User["$title"] } } else { # copy in all the files in the subdirectory file { "/home/${title}": recurse => true, mode => '0755', owner => $title, group => $title, source => "puppet:///modules/users/${title}", require => User["$title"] } } } A few things to note. The define statement signifies this file is a resource. As can be seen, it accepts variables. We will see later how they are passed. The couple of sections define the user and group attributes. The next section contains some if else logic. If $userfiles is false (which is set to the default on the first line), just the home directory is created, else the home directory is created and some files are copied into it. This is what will be copied: MicroServer users # pwd /etc/puppetlabs/code/environments/production/modules/users MicroServer users # find files files files/tony files/tony/.bashrc So a custom .bashrc file. The init.pp for the users module contains the following: MicroServer manifests # cat init.pp class users { @users::virtual { 'tony': userfiles => true, uid => 1000, realname => 'Tony', pass => '$6$QzIYITBc$RCs75ujfU/9XgffYYBn45VEyk4Zr4jJadzd9URVa1PFAMbOOCRvfUPulLtBlYHAR/OYcasrAQrgpAN33/RFkf/', } @users::virtual { 'melissa': uid => 1003, realname => 'Melissa', pass => '$6$7ajWS2Fh$g0Apa3rwWzq0urAgvbYilw4XB09.cHibX0JGI6sE8XkMGvgrJIsrKcKWscnrBxHXNfITLtT2lL.eXQDua/xe40', } @users::virtual { 'daniel': uid => 1004, realname => 'Daniel', pass => '$6$DuIkulz3$ZD/y5vV.FjrRCGj97V5l6Hv/IWt7x8RxL66Y5H.Uw9KdYaF/IE8D3kAYgmhvAdCQBrEmWm13miD4zmSFGcrUG0', } @users::virtual { 'jamie': uid => 1005, realname => 'Jamie', pass => '$6$DuIkulz3$ZD/y5vV.FjrRCGj97V5l6Hv/IWt7x8RxL66Y5H.Uw9KdYaF/IE8D3kAYgmhvAdCQBrEmWm13miD4zmSFGcrUG0', } As can be seen, the init.pp contains the variables that will be passed to the users::virtual resource. For tony, userfiles is set to true so the custom .bashrc file is copied for this user.The @ symbol signnifies that they're virtual resources. site.pp contains node default { } node 'Microserver' { include bleachbit include users realize (Users::Virtual['tony']) realize (Users::Virtual['melissa']) realize (Users::Virtual['daniel']) realize (Users::Virtual['jamie']) } For virtual resources the realize statement is used. This is the result when puppet agent --test is run: MicroServer manifests # puppet agent --test Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Caching catalog for microserver Info: Applying configuration version '1462443980' Notice: /Stage[main]/Users/Users::Virtual[tony]/User[tony]/comment: comment changed 'tony,,,' to 'Tony' Notice: /Stage[main]/Users/Users::Virtual[melissa]/Group[melissa]/ensure: created Notice: /Stage[main]/Users/Users::Virtual[melissa]/User[melissa]/ensure: created Notice: /Stage[main]/Users/Users::Virtual[melissa]/File[/home/melissa]/owner: owner changed '1002' to 'melissa' Notice: /Stage[main]/Users/Users::Virtual[daniel]/Group[daniel]/ensure: created Notice: /Stage[main]/Users/Users::Virtual[daniel]/User[daniel]/ensure: created Notice: /Stage[main]/Users/Users::Virtual[jamie]/Group[jamie]/ensure: created Notice: /Stage[main]/Users/Users::Virtual[jamie]/User[jamie]/ensure: created Notice: Applied catalog in 12.98 seconds So that's our whistle stop tour of puppet. It barely scratches the surface but it does demonstrate the power of puppet . **Still stuck? Maybe we can help.** Contact us [[https://www.upwork.com/o/profiles/users/_~01207a4bc438b48c6e|at Upwork]] Never miss a thing [[http://www.justsomestuff.co.uk/subscribe.php|subscribe to our newsletter]] {{glyphicon>envelope}} or follow us [[http://twitter.com/itsjustsomestuf|on twitter]] {{glyphicon>hand-left}} For more super cool techie stuff [[http://www.justsomestuff.co.uk/theblog|check out our blog!!]] ===Recent Changes=== {{changes>}} ====Contribute to this wiki==== Why not help others by sharing your knowledge? Contribute something to this wiki and [[http://www.justsomestuff.co.uk/hall_of_fame.html|join out hall of fame!]]\\ [[http://www.justsomestuff.co.uk/theblog/contact-page|Contact us]] for a user name and password