Puppet

Puppet is an open source (with enterprise version and support available)  client/server tool from Puppet Labs to facilitate the configuration and management of computer systems.

Written in ruby and available on many platforms, it offers a DSL that allows the “programming” of operational tasks across many machines.

The DSL covers abstracting the computer resources in a extensible way, and providing structure like class, modules and graphs that a configuration language can manipulate.

The server is called puppet master. It controls  a puppet agent installed on client machines that are to be managed. In addition of master and agent, there are puppet apply for stand alone use and puppet resource for accessing the Puppet resource abstraction layer.

Its extensibility makes it future proof  and there are providers (implements a resource abstraction in Puppet) for many platforms like Virtual box VMs and Amazon EC2.

The communication between clients and server is secured using SSL certificates.

This post is mainly for me so I can remember how I did install Puppet on Mac OS X and to allow me to repeat it on many mac systems.

The install  basically boils down to running a script I put on Gist (assuming you want to install Puppet 2.7.11 with Facter 1.6.6 on a startup disk called Macintosh HD):

bash -s 1.6.6 2.7.11 /Volumes/Macintosh\ HD < <(curl -s https://raw.github.com/gist/1895594/install_puppet_mac.sh)

The remainder of the post describes the gory details. I’ll keep this post updated as I learn more about the idiosyncrasies of Puppet on Mac.

Deployment environment

I’ve tested these instructions on Mac OS X Lion (10.7.3)  and Mac OS X Snow Leopard (10.6.8).

hdiutil attach facter-1.6.6.dmg
hdiutil attach puppet-2.7.11.dmg

sudo installer -package /Volumes/facter-1.6.6/facter-1.6.6.pkg -target /Volumes/Macintosh\ HD
sudo installer -package /Volumes/puppet-2.7.11/puppet-2.7.11.pkg -target /Volumes/Macintosh\ HD

 max_gid=$(dscl . -list /Groups gid | awk '{print $2}' | sort -ug | tail -1)
 new_gid=$((max_gid+1))
 dscl . create /Groups/puppet
 dscl . create /Groups/puppet gid $new_gid

 max_uid=$(dscl . -list /Users UniqueID | awk '{print $2}' | sort -ug | tail -1)
 new_uid=$((max_uid+1))
 dscl . create /Users/puppet
 dscl . create /Users/puppet UniqueID $new_uid
 dscl . -create /Users/puppet PrimaryGroupID $new_gid

mkdir -p /var/lib/puppet
mkdir -p /etc/puppet/manifests
mkdir -p /etc/puppet/ssl

chown -R puppet:puppet  /var/lib/puppet
chown -R puppet:puppet  /etc/puppet

echo "[main]
pluginsync = false
server = `hostname`

[master]
vardir = /var/lib/puppet
libdir = $vardir/lib
ssldir = /etc/puppet/ssl
certname = `hostname`

[agent]vardir = /var/lib/puppetlibdir = $vardir/libssldir = /etc/puppet/sslcertname = `hostname`

" > /etc/puppet/puppet.conf

bash -s 1.6.6 2.7.11 /Volumes/Macintosh\ HD < <(curl -s https://raw.github.com/gist/1895594/install_puppet_mac.sh)

import "nodes"

node default {

  notify {"Hello World":;}

}

puppet apply /etc/puppet/manifests/site.pp

notice: Hello world

notice: /Stage[main]//Node[default]/Notify[Hello world]/message: defined 'message' as 'Hello world'

notice: Finished catalog run in 0.02 seconds

sudo puppet master --verbose --no-daemonize

notice: Starting Puppet master version 2.7.11

sudo puppet agent --test --server=`hostname`

info: Caching catalog for wall-e.home

info: Applying configuration version '1330800282'

notice: Hello world

notice: /Stage[main]//Node[default]/Notify[Hello world]/message: defined 'message' as 'Hello world'

info: Creating state file /var/lib/puppet/state/state.yaml

notice: Finished catalog run in 0.02 seconds

Exiting; no certificate found and waitforcert is disabled

sudo puppetca --sign <hostname of the client machine>

notice: Did not receive certificate

notice: Signed certificate request for <client host name>

notice: Removing file Puppet::SSL::CertificateRequest mc-s056627.home at '/etc/puppet/ssl/ca/requests/<client hostname>.pem'

info: Caching certificate for <client hostname>

sudo puppetca --revoke <client host name>

sudo rm -rf /etc/puppet/ssl

sudo puppet cert clean <client hostname>

/usr/bin/puppet:3:in `require': no such file to load -- puppet/util/command_line (LoadError)        from /usr/bin/puppet:3/usr/sbin/puppetd:3:in `require': no such file to load -- puppet/application/agent (LoadError)        from /usr/sbin/puppetd:3

rvm use system

info: Retrieving plugin

err: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve information from environment production source(s) puppet://wall-e.home/plugins

info: Could not find filesystem info for file 'plugins' in environment production

info: Could not find file_metadata for 'plugins'

Presentation on Scribd

I will post more on the subject in the future:

One of my personal project involves PubSubHubbub and EventMachine and I’m becoming curious|excited about Javascript’s Node.JS.

I’ve tried to compare the configuration difference between a successully running Debian Linux 5.0 (Lenny) and these Etch image without success.

I found the solution by chance on a vmware image vendor’s web site:

rm /etc/udev/rules.d/z25_persistent-net.rules && reboot

I’m going to remake them in two tutorials instead of one. In the meantime here’s the content of my bash profile with the elements I use to smooth my daily experience of git:

#### start Git #####
alias g='git'
alias gco='git checkout'
alias gma='git checkout master'
alias gst='git status'
alias glo='git log'
alias gca='git commit -a'
alias gsd='git svn dcommit'
alias gcav='git commit -v -a'
alias squash='git merge --squash'
alias gpatch='git format-patch'
alias saw='git branch -D'
alias rollback='git reset --hard git-svn'
alias uncommit='git reset --mixed HEAD'
alias fixlastcommit='git commit --amend'
alias branches='git branch -a'
alias grow='git checkout -b'
alias plant='git svn init'
alias gclone='git clone'
alias hide='git stash'
alias unhide='git stash apply'

export PS1='\w $(git branch &>/dev/null; if [ $? -eq 0 ]; then \
echo "(\[\033[00m\]$(git branch | grep ^*|sed s/\*\ //)) "; fi)\$\[\033[00m\] '

#### end Git #####

In addition, he noticed that on some Xen VMs, you may encounter the following error:


ERROR: Need to be root to increase max connections


in which case, you will need to update /etc/security/limits.conf with the values:


mogile soft nofile 65535
mogile hard nofile 65535


I came an across an elegant solution  to this problem as decribed on this blog.

I used 4 Xen virtual images running ubuntu 8.04.
Two will run a tracker and the database, the other two will be the storage nodes.

Lets say the IP addresses will be:


192.168.0.195
192.168.0.196
192.168.0.197
192.168.0.198


2. Initial Setup

Install iptables:


apt-get install iptables


then apply initial setup


iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport ssh -j ACCEPT
iptables -A INPUT -j DROP
iptables-save -c > /etc/iptables.rules

you can save the rule for beyond machine reboot by adding the two lines below to /etc/network/interfaces

pre-up iptables-restore < /etc/iptables.rules
post-down iptables-save -c > /etc/iptables.rules


install mysql and wget and perldoc


apt-get install mysql-server
apt-get install wget
apt-get install perl-doc
apt-get install libio-aio-perl
apt-get install subversion
apt-get install build-essential


3. Common steps for installing MogileFS

retrieve the code (ripped of the how to wiki):


cd /usr/local/src
mkdir mogilefs-src
cd mogilefs-src
svn checkout http://code.sixapart.com/svn/mogilefs/trunk


install perl dependencies


cpan Danga::Socket
cpan Gearman::Client
cpan Gearman::Server
cpan Gearman::Client::Async
cpan Net::Netmask
cpan Perlbal
cpan IO::WrapTie


install the servers:


cd mogilefs-src/trunk/server
perl Makefile.PL
make
make test
make install


At the moment the test seems to need mysql to be installed and user root without password, so some tests are skipped

you will need to install MogileFS::Client
(the tests expect a tracker to run locally on port 7001)


cd mogilefs-src/trunk/api/perl/MogileFS-Client
perl Makefile.PL
make
make test
make install


and some admin tools:


cd mogilefs-src/trunk/utils
perl Makefile.PL
make
make test
make install


4. Tracker install

create the database:


mysql -uroot -p
mysql> CREATE DATABASE mogilefs;
mysql> GRANT ALL ON mogilefs.* TO 'mogile'@'%';
mysql> SET PASSWORD FOR 'mogile'@'%' = OLD_PASSWORD( 'sekrit' );
mysql> FLUSH PRIVILEGES;
mysql> quit


Create the schema


./mogdbsetup --dbname=mogilefs --dbuser=mogile --dbpassword=sekrit


(admin privilege is required for the initial setup, so if you’re admin user is not root with no password, you will need to specify –dbroopassword and –dbrootuser)

create /etc/mogilefs/mogilefsd.conf:


db_dsn DBI:mysql:mogilefs
db_user mogile
db_pass ******
conf_port 7001
listener_jobs 5


create a mogile user:


adduser mogile


and starts the tracker under that user:


su - mogile
mogilefsd


open a port for the tracker


iptables -A INPUT -p tcp --dport 7001 -j ACCEPT

5. Storage node

On the storage server, create a configuration file at /etc/mogilefs/mogstored.conf with the following:


httplisten=0.0.0.0:7500
mgmtlisten=0.0.0.0:7501
docroot=/var/mogdata


open a port:


iptables -A INPUT -p tcp --dport 7500 -j ACCEPT

iptables -A INPUT -p tcp --dport 7501 -j ACCEPT

register a new storage node:


mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 host add mogilestorage --ip=192.168.0.197 --port=7500 --status=alive


it should now appears in the list:


mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 host list


Add a device to the storage:


mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 device add mogilestorage 1


and create the directory:


mkdir -p /var/mogdata/dev1

make sure /var/mogdata/* is owned by mogile:mogile


chown -R mogile:mogile /var/mogdata/*


6. Starting the storage server

as root:

mogstored --daemon


7. Starting the tracker


su - mogile
mogilefsd -c /etc/mogilefs/mogilefsd.conf --daemon
exit


8. Testing

check that mogilefs components are online:


mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 check


Quick sanity check of the storage daemon:


~/Projects/mogilefs $ telnet 192.168.0.197 7500
Trying 192.168.0.197...
Connected to 192.168.0.197.
Escape character is '^]'.
PUT /dev1/test HTTP/1.0
Content-length: 4
\n
test
HTTP/1.0 200 OK
Content-Type: text/html
Content-Length: 18
Server: Perlbal
Connection: close
200 - OK
Connection closed by foreign host.


create a domain


mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 domain add mydomain


and a class


mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 class add mydomain images


Quick sanity check of the tracker:


root@bbc-01:~# mogtool --trackers=127.0.0.1:7001 --domain=mydomain --class=images inject osname osname


on the store node, check /var/mogdata/dev1/0/000/000 for a file named xxxxxxxxxx.fid .
If the file exists it’s all good.

9. setting up the second pair

Replay instructions 1 to 8, then:

when you’ve got the second storage set up, you will need to register the second storage and its device to all trackers:


mkdir -p /var/mogdata/dev2
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 host add mogilestorage2 --ip=192.168.0.198 --port=7500 --status=alive
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 device add mogilestorage2 2
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.196:7001 host add mogilestorage2 --ip=192.168.0.198 --port=7500 --status=alive
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.196:7001 device add mogilestorage2 2


you will also need to register the first storage and its device to the second tracker:


mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.196:7001 host add mogilestorage --ip=192.168.0.197 --port=7500 --status=alive
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.196:7001 device add mogilestorage 1


sanity check the installation:


mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001,192.168.0.196:7001 check

Checking trackers…
192.168.0.195:7001 … OK
192.168.0.196:7001 … OK

Checking hosts…
[ 1] mogilestorage2 … OK
[ 2] mogilestorage … OK

Checking devices…
host device size(G) used(G) free(G) use% ob state I/O%
—- ———— ———- ———- ———- —— ———- —–
[ 1] dev2 9.921 0.757 9.164 7.63% writeable 0.0
[ 2] dev1 9.921 0.600 9.321 6.05% writeable 0.0
—- ———— ———- ———- ———- ——
total: 19.842 1.357 18.485 6.84%

And voila.

with_scope and returning.

Found a blog where they are both nicely explained:

• with_scope
• returning

Flickr has today announced support for video media.
I’m impressed on how the integration is done, it’s the same experience as for photo plus the added value of what they call “long photo”.
Very elegant indeed.

Now coming alongside the release of Safari 3.1, there’s a new Safari plugin that allow the parsing of microformats for integration in various Mac OS X applications (Adress book, iCal).

For Mac OSX users, the integration with Address Book and iCal could make Safari a better microformat reader than Firefox. I cannot test the plugin myself as I don’t have Leopard which is a requirement.

Alternatively the latest version of my feed reader NetNewsWire can also parse microformats but it requires you (obviously by the very definition of microformats) to render the web page in Netnewswire not just the summary.

Safari 3.1 is available for Tiger and Windows though.