[olug] Question: package / system for administrating multiple very similar systems ?

[Sean Kelly]

On Fri, Apr 20, 2007 at 07:59:11PM -0500, Will Langford wrote:
...
> Is there a pre-built 'update' system that will fetch new updates from a
> central source and apply them as they're discovered (at some set interval
> from cron or something) ?  The updates will all be custom built by us.

Oh boy. You've stepped into something you can never escape from now. Your
question will now hurl you into the complicated world of systems
infrastructure.

First up, I'd recommend checking out http://www.infrastructures.org
That site will give a good overview of the things you need to be thinking
of when you start to manage many servers and want to have a standard way of
managing them. This is a project I began to undertake a little over two
years ago, but set aside in order to bring up a VMware Infrastructure
environment. Shortly I'll be returning to this sort of thing.

> Our remote systems are running RH 9.0 with many modifications to suit our
> needs... so an rpm based solution would be fine.  Yes, RH9 - they've been in
> service for that long... and I believe one of our systems has a 400+ day
> uptime currently :).

Uptimes like that aren't always a good thing. It means you've got an
outdated kernel that could use some patching.

> The ideal solution would be something that's fairly simple and not overly
> complex.  Packages themselves can be configured to self-determine if they're
> required to be run on the server they find themselves on.

There really aren't that many simple tools. There are a few tools I can
point you at to start looking at:

 * cfengine - http://www.cfengine.org/
 This is a policy-based configuration management tool. The guy behind it is
 very smart, but it takes a while to wrap your mind around how it works. I
 sat in on a few sessions run by him at LISA '05.

 * Puppet - http://reductivelabs.com/projects/puppet/
 This is a project underway by Luke Kanies, another smart guy, to design
 something to replace cfengine. Cfengine is not dead, but Luke has some
 other ideas on how to manage system configurations.

 * radmind - http://rsug.itd.umich.edu/software/radmind/
 This tool is more for managing files and file sets. However, depending on
 how similar your systems are, that could potentially work for you. While
 the site feels very Apple-centric, it will work on Linux, FreeBSD, and
 others as well.

 * ISConf - http://trac.t7a.org/isconf/
 A fairly popular solution.

 * LCFG - http://www.lcfg.org/
 Another one that pops up frequently.

I've also seen other tools that I can't really comment much on. I've seen
some of how companies like FedEx and Yahoo! do it internally. Their models
are much simpler and more similar to what you describe above. Yahoo!'s
tool, called yinst, is a very cool tool that sits on top of FreeBSD's
package management and allows them to deploy a system in minutes. I wish
they'd release it, or even talk about it. All I have is its manpage...

> I've been a command line dweller for about 10 years, so I should have
> adequate experience to set anything up, but if it requires more than 2 hours
> of my time to get a skeleton up, it's out of the question.

Any good solution will require far more than two hours of time. You have to
look at it in the long term. You'll be designing a new way to manage your
machines that allows you to scale from your few servers up to hundreds. Any
solution that can be deployed in less than two hours will eventually break
and leave you sitting there trying to figure out how to glue the two pieces
back together.

> A few of the automatic systems I only gave a passing throught to would be
> up2date and yast... both of which seem like they might be a bear to
> configure... or well beyond our needs.  I could be wrong though ?

So, you mean like design your own RPMs and deploy them with up2date or
yast? That is one way to do it, and is very similar to what FedEx did last
I saw it. I can see some problems with it, but it seemed to work for them.
They didn't even use RPMs, though.


I would recommend you try hitting up the config-mgmt and infrastructures
mailing lists. There are some infrastructure veterans there that could help
point you in a good direction based on your environment:

config-mgmt: http://lopsa.org/cgi-bin/mailman/listinfo/config-mgmt
infrastructures: http://mailman.terraluna.org/mailman/listinfo/infrastructures

-- 
Sean Kelly          | PGP KeyID: D2E5E296
smkelly at smkelly.org | http://www.smkelly.org