Installing puppetdb

It just makes sense to use the puppet database in your puppet environment. It is used to store the inventory facts about every node in your puppet environment and as this can be big, you need to have a good back end database which can handle this. Here is how I installed puppet DB

There are no special requirements for SELINUX when installing puppet DB. Keep SELINUX set to enforcing mode.

1.    Install the puppet repo

rpm -ivh http://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-6.noarch.rpm

2.    Install the puppet agent and request a CSR

$ yum install puppet -y
$ puppet agent --test

3.    Sign the puppet agents CSR on the puppet master

$ puppet cert --list
$ puppet cert sign <puppet client>

4.    Install puppet DB on the puppet DB server

puppet resource package puppetdb ensure=latest

5.    Install postgres

Install instructions taken from http://wiki.postgresql.org/wiki/YUM_Installation

vi /etc/yum.repos.d/CentOS-Base.repo

To the [base] and [updates] sections in  /etc/yum.repos.d/CentOS-Base.repo, you need to append a line

exclude=postgresql*

Download and install the rpm repos for postgres

rpm -ivh  http://yum.postgresql.org/9.1/redhat/rhel-6-i386/pgdg-centos91-9.1-4.noarch.rpm
yum list postgres*
yum install postgresql91-server -y

6.    Start the postgres service and run at boot

service postgresql-9.1 initdb
service postgresql-9.1 start
chkconfig postgresql-9.1 on

7.    Configure postgres

vi /var/lib/pgsql/9.1/data/postgresql.conf
listen_addresses = '*'
log_line_prefix = '%t %u %d'

8.    Create the puppet DB

sudo -u postgres sh

cd /var/lib/puppetdb
createuser -DRSP puppetdb
createdb -O puppetdb puppetdb

9.    Allow host access to the postgresql pupetdb

vi /var/lib/pgsql/9.1/data/pg_hba.conf
local all all trust
host puppetdb puppetdb 10.10.10.0/24 trust

•    Comment out this line (near the end of pg_hba.conf)

#local    all    all    peer

•    Restart the postgres service

service postgresql-9.1 restart

•    Test login

psql -h 10.33.22.162 puppetdb puppetdb
and
psql -d puppetdb -U puppetdb -W

10.    Configure puppdb conf.d/config.ini file

vi /etc/puppetdb/conf.d/config.ini

Below details an example config.ini file

# See README.md for more thorough explanations of each section and
# option.

[global]

# Store mq/db data in a custom directory
vardir = /var/lib/puppetdb

# Use an external log4j config file

logging-config = /etc/puppetdb/conf.d/../log4j.properties

# Maximum number of results that a resource query may return

resource-query-limit = 20000

[database]

classname = org.postgresql.Driver
subprotocol = postgresql
subname = //localhost:5432/puppetdb
username = puppetdb
password = <password>

[jetty]

port = 8080

[command-processing]
 
# How many command-processing threads to use, defaults to (CPUs / 2)
# threads = 4

11.    Start the puppetdb services

sudo puppet resource service puppetdb ensure=running enable=true

12.    Open puppetdb port 8081  in iptables

Modify where necessary

iptables -I INPUT 5 -s 10.10.10.0/24 -m tcp -p tcp --dport 8081 -j ACCEPT
iptables -I INPUT 6 -s 10.10.10.0/24 -m tcp -p tcp --dport 8080 -j ACCEPT

service iptables save
service iptables restart

The 2nd firewall rule is used to access the puppetdb-dashboard

13.    Set up a puppet master to connect to puppetdb

•    Run the following on each of your puppet masters:

sudo puppet resource package puppetdb-terminus ensure=latest

•    Add this to /etc/puppet/puppetdb.conf. Note: you may have to create this file.

[main]

server = <puppetdb>
port = 8081

•    Add this to /etc/puppet/puppet.conf

[master]

  storeconfigs = true
  storeconfigs_backend = puppetdb

•    Add this to /etc/puppet/routes.yaml. Note: you may have to create this file.

master:
  facts:
    terminus: puppetdb
    cache: yaml

14.    Restart the puppet service on each puppet master

service httpd restart

15.    Check in an agent and monitor the puppetdb logs

On the puppetdb server, monitor the puppetdb log

tail -f /var/log/puppetdb/puppetdb.log

On an agent run the following:

puppet agent --test

16.    (Optional) Open up access to the puppet DB dashboard

Reference:

http://docs.puppetlabs.com/puppetdb/1/maintain_and_tune.html#monitor-the-performance-dashboard

To access the puppetdb, you need to make some configuration changes. Edit the following file 

/etc/puppetdb/conf.d/config.ini

[jetty]

...

host = 0.0.0.0

Access the puppet DB using the following URL, changing the host name accordingly

http://puppetdb.example.com:8080/dashboard/index.html

Troubleshooting issue:

If the puppet DB, port 8080 and 8081 are not listening when the puppetdb service is set to start and you see in /var/log/puppetdb/puppdb.log the following error:

2013-02-26 19:40:43,327 ERROR [main] [puppetlabs.utils] Uncaught exception
java.io.FileNotFoundException: /etc/puppetdb/ssl/keystore.jks (No such file or directory)

Run the following command and reboot the server

sudo /usr/sbin/puppetdb-ssl-setup

This will create a keystore and truststore in /etc/puppetdb/ssl and will print the password to both files in /etc/puppetdb/ssl/puppetdb_keystore_pw.txt.

Installing puppet dashboard

Here are my notes in installing puppet dashboard

1.    Install EPEL and puppet repos

rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm -ivh http://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-6.noarch.rpm
 
2.    Install required packages

yum install vim puppet-server puppet puppet-dashboard sudo mod_ssl rubygem-passenger mod_passenger policycoreutils-python mysql mysql-server -y

3.    Generate a puppet client cert and sign on the puppet master
 
•    Run the following:
 
puppet agent --test 

•    Sign the certificate on the puppet master
 
puppet cert --list
puppet cert sign <puppet-client>
 
for example

puppet cert sign puppet-dashboard.myorg.net4.    Configure MYSQL and create dashboard database

4.1     Start the mysqld server and configure chkconfig

service mysqld start
chkconfig mysqld on 

4.2     Run and secure mysql
 
/usr/bin/mysql_secure_installation

4.3     Log in to mysql and create a database and configure permissions / password
create database puppetdash;

GRANT ALL PRIVILEGES ON puppetdash.* TO puppet@'%' IDENTIFIED BY '<password>';
FLUSH PRIVILEGES;
set global max_allowed_packet = 33554432;

4.4     Update database entries for the production DB settings as follows:
 
vi /usr/share/puppet-dashboard/config/database.yml

production:
  database: puppetdash
  username: puppet
  password: <password>
  encoding: utf8
  adapter:  mysql
 
4.5     Create dashboard schema

cd /usr/share/puppet-dashboard

rake RAILS_ENV=production db:migrate

5.    Copy the example puppet virtual host config to /etc/httpd/conf.d/

cp /usr/share/puppet-dashboard/ext/passenger/dashboard-vhost.conf /etc/httpd/conf.d/puppet-dashboard.conf

6.    Remove the ssl.conf file /etc/httpd/conf.d/ directory. Note, the mod_ssl.so is loaded in the custom puppet-dashboard.conf virtual host file.

rm -f /etc/httpd/conf.d/ssl.conf

7.    Configure the puppet-dashboard virtual host file.

vim /etc/httpd/conf.d/puppet-dashboard.conf

# you may want to tune these settings
PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
# PassengerMaxRequests 1000
PassengerStatThrottleRate 120
RailsAutoDetect On


<VirtualHost *:80>
        RewriteEngine on
        ReWriteCond %{SERVER_PORT} !^443$
        RewriteRule ^/(.*) https://%{HTTP_HOST}/$1 [NC,R,L]
</VirtualHost>

LoadModule ssl_module modules/mod_ssl.so

Listen 443
<VirtualHost *:443>
        SSLEngine on
        SSLProtocol -ALL +SSLv3 +TLSv1
        SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

        SSLCertificateFile        /usr/share/puppet-dashboard/certs/dashboard.cert.pem
        SSLCertificateKeyFile     /usr/share/puppet-dashboard/certs/dashboard.private_key.pem
        SSLCACertificateFile      /usr/share/puppet-dashboard/certs/dashboard.ca_cert.pem

        # If Apache complains about invalid signatures on the CRL, you can try disabling
        # CRL checking by commenting the next line, but this is not recommended.
        SSLCARevocationFile       /usr/share/puppet-dashboard/certs/dashboard.ca_crl.pem

        SSLVerifyClient optional
        SSLVerifyDepth  1
        SSLOptions +StdEnvVars

        ServerName ukcrlcscin0010.tcl-oob.net
        DocumentRoot /usr/share/puppet-dashboard/public/

        <Directory /usr/share/puppet-dashboard/public/>
                Options None
                Order allow,deny
                allow from all
        </Directory>

      ErrorLog /var/log/httpd/puppet-dashboard-net_error.log
      LogLevel warn
      CustomLog /var/log/httpd/puppet-dashboard-net_access.log combined
      ServerSignature On


        <Location / >
                Order deny,allow
                Allow from ALL
                # Enable this to require client-side certificates for Dashboard connections
                #SSLVerifyClient require

</Location>
</VirtualHost>8.    Ensure the following log files exist

touch /var/log/httpd/ukcrlcscin0010.tcl-oob.net_access.log
touch /var/log/httpd/ukcrlcscin0010.tcl-oob.net_error.log
touch /usr/share/puppet-dashboard/log/production.log
chmod 0666 /usr/share/puppet-dashboard/log/production.log
 
9.    Create the puppet-dashboard SSL certs for https access

•    On the puppet-dashboard server run:

$ sudo -u puppet-dashboard rake cert:create_key_pair
$ sudo -u puppet-dashboard rake cert:request

•    On the puppet master, sign the certs

puppet cert sign dashboard

•    On the puppet-dashboard, retrieve the SSL certs

$ sudo -u puppet-dashboard rake cert:retrieve
 
10.    Copy the following script to the puppet dashboard server and all the puppet masters
 
Create /usr/lib/ruby/site_ruby/1.8/puppet/reports/https.rb with the following code

require 'puppet'
require 'net/http'
require 'net/https'
require 'uri'

Puppet::Reports.register_report(:https) do

  desc <<-DESC
  Send report information via HTTPS to the `reporturl`. Each host sends
  its report as a YAML dump and this sends this YAML to a client via HTTPS POST.
  The YAML is the `report` parameter of the request."
  DESC

  def process
    url = URI.parse(Puppet[:reporturl].to_s)
    http = Net::HTTP.new(url.host, url.port)
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_NONE

    req = Net::HTTP::Post.new(url.path)
    req.body = self.to_yaml
    req.content_type = "application/x-yaml"

    http.start do |http|
      response = http.request(req)
      unless response.code == "200"
        Puppet.err "Unable to submit report to #{Puppet[:reporturl].to_s} [#{response.code}] #{response.msg}"
      end
    end

  end
end
 
•    Remove the following file from the puppet dashboard and puppe masters

/usr/lib/ruby/site_ruby/1.8/puppet/reports/http.rb
rm -f /usr/lib/ruby/site_ruby/1.8/puppet/reports/http.rb

11.    Enable puppet-dashboard reports for  each puppet master in the environment 

Add to each puppet master in the puppet environment in /etc/puppet/puppet.conf:
 
 [master]

    reports = store, https, puppet_dashboard
    reporturl = https://<puppet-dashboard-server>/reports/upload
12.     Create the reports directories on each puppet master

mkdir -p $(puppet master --configprint libdir)/puppet/reports

13.  Copy the puppet_dashboard.rb files to each of the puppet masters
 
Note: To find out where to scp the files to on your puppet master, run the following command on you puppet master:
 
$ puppet master --configprint libdir

> /var/lib/puppet/lib
 
The scp command needs to be run on your puppet-dashboard server
 
scp /usr/share/puppet-dashboard/ext/puppet/puppet_dashboard.rb root@puppet:/var/lib/puppet/lib/puppet_dashboard.rb
Change the <puppet> tag accordingly

14.  Update the puppet_dashboard.rb file on each puppet master

Run on your puppet server and update the parameters HOST and PORT
 
vim /var/lib/puppet/lib/puppet_dashboard.rb

HOST = <puppet-dashboard-server>
PORT = 443 

15.    Edit the auth.conf file on each puppet master so inventory can pick up facts.
 
Make sure the following is set on each puppet master.
 
path /facts
method find
auth any
allow *

path /inventory
auth any
method search, find
allow dashboard

# deny everything else; this one is not stricly necessary, but it has the merit of showing the default policy, which is deny everything else

path /
auth any
 
Note: The config for /facts and /inventory must go above the config for `path /` - otherwise you may get an access forbidden 404 error message when running the inventory service on puppet-dashboard.
 
16.    Restart apache on puppet masters
 
service httpd restart 

17.    On the puppet dashboard server, turn on and configure the inventory service and cutoff time
 
vi /usr/share/puppet-dashboard/config/settings.yml


# The "inventory service" allows you to connect to a puppet master to retrieve and node facts

enable_inventory_service: true

# Hostname of the inventory server.

inventory_server: 'puppet'

# Port for the inventory server.

inventory_port: 8140

# Amount of time in seconds since last report before a node is considered no longer reporting

no_longer_reporting_cutoff: 43200

18.     Add the puppet-dashboard delayed_job script to rc.local

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local
cd /usr/share/puppet-dashboard
sudo -u puppet-dashboard env RAILS_ENV=production script/delayed_job -p dashboard -n 4 -m start

19.    Configure IP tables

iptables -I INPUT 5 -s 10.33.22.0/24 -m tcp -p tcp --dport 8140 -j ACCEPT
iptables -I INPUT 6 -s 10.33.22.0/24 -m tcp -p tcp --dport 80 -j ACCEPT
iptables -I INPUT 7 -s 10.33.22.0/24 -m tcp -p tcp --dport 443 -j ACCEPT
iptables -I INPUT 8 -s 10.33.22.0/24 -m tcp -p tcp --dport 3306 -j ACCEPT

service iptables save
service iptables restart
iptables -L

20.    Start apache and connect to the dashboard URL

service httpd start

Check the apache service has started and the webserver is listening on port 80 and 443.

21.     Start the delayed_job script

cd /usr/share/puppet-dashboard
sudo -u puppet-dashboard env RAILS_ENV=production script/delayed_job -p dashboard -n 4 -m start

Connect to the puppet-dashboard web UI. Run a puppet agent --test on the puppet-dashboard and you should see the puppet nodes report in to the puppet dashboard.

Installing a puppet master and ca server

Here are my install notes on how I set up my puppet server. Note, with EPEL 6.8, passenger is available via yum. This is a much better way to install passenger, rather than using ruby gems.

1 - Install EPEL and puppet repos

rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm -ivh http://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-6.noarch.rpm

2- Install required packages

yum install puppet-server puppet sudo mod_ssl rubygem-passenger mod_passenger policycoreutils-python vim rsync -y

3.    Create a puppet CA cert

puppet master --no-daemonize --verbose

ctrl+c to break out of puppet master deamon once the puppet daemon is running when you see the following message

Notice: Starting Puppet master version 3.x.x

4. Copy the example puppet virtual host config to /etc/httpd/conf.d/

cp /usr/share/puppet/ext/rack/files/apache2.conf /etc/httpd/conf.d/puppet-master.conf

5.    Edit the puppet-master.conf file and update accordingly

# you probably want to tune these settings

PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
PassengerMaxRequests 1000
PassengerStatThrottleRate 120
RackAutoDetect Off
RailsAutoDetect Off

Listen 8140

<VirtualHost *:8140>

        SSLEngine on
        SSLProtocol -ALL +SSLv3 +TLSv1
        SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
        SSLCertificateFile      /var/lib/puppet/ssl/certs/puppet-server.com.pem
        SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/puppet-server.com.pem
        SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
        SSLCACertificateFile    /var/lib/puppet/ssl/ca/ca_crt.pem

        # If Apache complains about invalid signatures on the CRL, you can try disabling
        # CRL checking by commenting the next line, but this is not recommended.

        SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
        SSLVerifyClient optional
        SSLVerifyDepth  1

        # The `ExportCertData` option is needed for agent certificate expiration warnings

        SSLOptions +StdEnvVars +ExportCertData

        # This header needs to be set if using a loadbalancer or proxy
        RequestHeader unset X-Forwarded-For
        RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
        RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
        RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

        DocumentRoot /usr/share/puppet/rack/puppetmaster/public/
        RackBaseURI /
        <Directory /usr/share/puppet/rack/puppetmaster/>
                Options None
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>
</VirtualHost>
6.    Create rack directories

mkdir -p /usr/share/puppet/rack/puppetmaster/{public,tmp}

7.    Copy config.ru rack file to rack web directory

cp /usr/share/puppet/ext/rack/files/config.ru /usr/share/puppet/rack/puppetmaster/

8.    Change ownership of config.ru rack file to puppet

chown puppet:puppet /usr/share/puppet/rack/puppetmaster/config.ru

9.    Set httpd to start on boot and puppet master to not start

chkconfig httpd on
chkconfig puppetmaster off

10.  IPTABLES configuration

This is environment specific depending on the bridges you need to create to let the puppet master communicate to its various networks. Port 8140 needs to be open on the puppet master for each interface you have added, specifying the source networks and interface adapter as applicable. Below is an example taken from the UK Cressex LAB puppet master deployment. All IPTABLES information is covered later in the document under the configured puppet environments section.

Here is the following content of the /etc/sysconfig/iptables

# Generated by iptables-save v1.4.

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [24:2304]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -s 10.10.10.0/24 -i eth0 -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -s 10.10.10.0/24 -i eth0 -p tcp -m tcp --dport 8140 -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -s 10.10.11.0/24 -i eth1 -p tcp -m tcp --dport 8140 -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -s 10.10.12.0/24 -i eth2 -p tcp -m tcp --dport 8140 -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -s 10.10.13.0/24 -i eth3 -p tcp -m tcp --dport 8140 -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -s 10.10.14.0/24 -i eth4 -p tcp -m tcp --dport 8140 -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

COMMIT

You can add these rules following the example command below

iptables -I INPUT 6 -i eth1 -p tcp -s 10.10.10.0/24 --dport 8140 -m state --state NEW,ESTABLISHED -j ACCEPT

service iptables save

or by editing the /etc/sysconfig/iptables and running iptables-save and service iptables restart

11.  Start apache

service httpd start

12. check that httpd is running and test connectivity

https://puppet-server1.tcl-oob.net:8140/

If everything is working correctly, you should see

The environment must be purely alphanumeric, not ''


Hope this helps some people. There are some more blogs I have written concerning how to install puppet DB, puppet dashboard and adding additional puppet masters.

Gitlab, Jenkins and puppet playing together

 Jenkins in one of my favourite open source tools. It must also be one of the most flexible
 
You can set up Jenkins to monitor your Gitlab project for any changes. If Jenkins sees any commits to the master repo, Jenkins will execute a build script and run any customised scripts. Here is how to set this up.

• Set up a project in Gitlab. This will be your git repo.
  
• Create a your git repo using a git client on a linux box of your choice.
• Configure Jenkins job to poll the git repo and to run a  build script if there are any changes.

Here is how to configure the Jenkins server to poll the git repo.

Install the gitlab plugin to Jenkins

https://wiki.jenkins-ci.org/display/JENKINS/Gitlab+Hook+Plugin

Configuring source code management

 

Set up a Jenkins job and specify the following under source code management



note: The repository URL can be obtained from gitlab as shown below

Setting up build triggers

Under build triggers, select the Poll SCM and set a Schedule. The schedule is in cron format and be set as desired. Here we have set Jenkins to poll the git repo every 5 minutes for changes.



Note: Once the job is setup, you can click on the

Here you will see the git polling working as shown below:

SSH keys and sudo access

We will need to set up ssh keys and sudo access for the jenkins_admin user which is used to log on to the puppet masters to sycnronise the manifests and modules.

To set up the keys, run the following commands on the jenkins box.

1  - Log in as jenkins_admin@jenkins on the jenkins server
2 - Run the following command to drop into the jenkins user

$ sudo - s -H -u jenkins

3 - Run the following to set up the ssh key pair

$ cd ~/.ssh/
$ ssh-sopy-id -i id_rsa.pub jenkins_admin@puppet-server # where puppet-server 
is the puppet master you need to sync the manifests on

4- Configure sudo on the puppet-master so Jenkins_admin is not prompted for the password. Note you can use a manifest to configure sudo via puppet.
on the puppet-server (as root)

sudo visudo

...

jenkins_admin ALL= NOPASSWD:/usr/bin/rsync

Make sure you do this process for each puppet-server you wish to configure sudo and ssh keys for

Configure the Jenkins build job

Configure the Jenkins build job as follows:

# Specifies a AD user who has sudo rights on the puppet server to run the rsync 
commands without requiring a password

USER=jenkins_admin

# List of puppet masters that will receive manifests and modules

for HOST in 'puppet-server1t' 'puppet-server2' 'puppet-server3'

do

# rsync command to sync changes from gitlab repo

rsync -e "ssh -t -l jenkins_admin" --rsync-path='sudo rsync' -avz --stats 
--progress --human-readable --exclude .git --exclude README $WORKSPACE/ 
$USER@$HOST:/etc/puppet/

done

This completes the setup.