Hosting Infrastructure Resources

This section has links and reviews of useful resources for managing hosting infrastructures. This includes sites, tools, books, and vendors.

Hosting Providers

Here are some of the hosting providers I've worked with.

DreamHost

DreamHost offers shared hosting accounts with an amazing amount of functionality for a very low price. Lots of virtual domains, mysql instances, and tons of extras. They also have dedicated servers, but they seem to be in flux. When I looked at them a while ago they had dedicated Debian servers for under $100 per month, but these are now "coming soon", and the pricing for dedicated RedHat servers (as opposed to VPS servers) aren't promising. So I'm not sure if their dedicated servers are as great a value as their shared hosting.

Aside from the feature heaviness of their shared hosting, DreamHost are pretty good on their communications, their newsletters and blogs are high quality. I haven't needed to contact them for support, so I can't vouch.

Network Monitoring Tools

This section has links and information about network monitoring tools. I've used Nagios a lot in the past 4 or 5 years, which is open source and pretty mature. It is mainly for detecting and reporting problems however, so it's useful to add something like Munin for tracking and graphing system resources and performance.

Another tool I'd like to try out for this is OpenNMS, which is written in Java, and includes the graphing as well as detection and reporting, and also auto-detects devices and services on a network.

Hyperic is another open source tool in this category I'd like to learn more about.

System management tools

Here are some of the standard tools I use for managing multi-server hosting infrastructures.

cfengine

I would call cfengine a configuration management tool. I just can't get into graphical and web-based tools for managing servers, I much prefer having a set of configuration files that I can check into version control. Once I've got a decent configuration set for an infrastructure, setting up, updating, or changing the role of a machine is a simple matter of tweaking the configuration files and running a command.

I find cfengine to be a bit awkward, it's configuration system suffers from being an academic research project. But so far I haven't found anything better.

A challenge I have is that less experienced sysadmins often find it difficult, they would rather just directly change a machine's configuration instead of changing central files and distributing them. But this inevitably leads to a bunch of inconsistent, out of date machines, so discipline is worth it.

Cfengine's syntax is too rigid, it doesn't make it easy to template configuration patterns and reuse them. For example, let's say I want to build Apache configurations for multiple virtual hosts. In Apache's main httpd.conf file I use an include directive to read in all files ending in .conf in the directory /etc/httpd/vhosts. So I just need to configure cfengine to put a .conf file in that directory for each virtual host to be hosted on a given machine.

Ideally I would just have a single set of rules in cfengine which create the appropriate vhost file, using the cfengine editfiles: tasks to build the file with the appropriate apache configuration directives, the copy: task to put it into place, and the files: task to ensure it has the right permissions. But cfengine doesn't let me do this, instead I need to copy the appropriate directives for each vhost.

This kind of inflexibility is one of the reasons cfengine configurations are difficult to understand, which discourages sysadmins from using it.

Configuration management with Puppet

I've started tinkering with puppet for configuration management. It's a far more flexible and extensible tool than cfengine, so it looks like the best way to go.

It's main drawback is lack of maturity. The documentation is fair, there's a decent reference, but there are only two examples of configuration files that I've seen so far, and neither one is very complex. It's also fairly buggy, although the author is quick to respond when told about specific problems.

I'll most likely be using Puppet to build a J2EE infrastructure based on Red Hat. I'd like to be able to contribute bug fixes, but I'm not sure how many spare cycles I'll have, given that I don't know Ruby. But hopefully I can at least contribute some example files, and some manifests related to Tomcat and general J2EE web application deployments.

Assuming I do use Puppet for this project, I'll try to post information here as I go along, in addition to the project itself.

cfengine alternatives

I've been working up a cfengine-based setup to manage a new server infrastructure. This will be my third cfengine-based infrastructure, so I should have learned enough to make a cleaner, tighter configuration. Unfortunately I'm still finding cfengine to be too damned awkward.

So, I'd like to put together a list of alternatives to cfengine. I'll add them to this page, and hopefully add on notes and reviews as I learn more. If you have experience with these or others, please add a comment.

  • Puppet seems to be an up and comer. It looks to be designed to be much more extensible than cfengine is. It also lets you make sure each host only sees its own configuration, which is one of my peeves about cfengine. It's my leading candidate at the moment.
  • bcfg2 was developed at the Argonne National Lab, according to this post they've been using it for 18 months. My main concern, without having even looked at the documentation, is that if it's only been used in one environment it may not have the flexibility to cope with different situations and approaches than its original infrastructure.

DNS Providers

There are scads of DNS providers out there. Hosting providers almost always provide DNS service as well, but I prefer to have a separate provider, so it's easier to switch hosting.

My favorite DNS service is easyDNS. They have a pretty comprehensive service, and they now do .co.uk domains, although you can't use their web-based interface to do it. There are cheaper services, but Easy DNS are by no means a rip off. The UI is clean and easy to use, their service is rock-solid and fast. I also find them to be a straight-shooting type of company, they don't use dodgy practices to squeeze extra money out of you.

I've had experiences with another, quite large DNS provider which managed to get me to renew a key domain for a client for 7 years by repeatedly sending me confusing emails, and not providing a clear way to see whether I had successfully renewed in my previous attempts. Losing the domain would have been a business disaster for my client, so even though I figured it was a scam I opted to renew anyway, just to be sure. I don't like dealing with companies like this, so I moved the domains to Easy DNS.