In my team, we need a Lead/Manager for the Support Team, and a Support Engineer. This has proved a tricky position to specify, since these people need to be all-rounders. They need to be able to manage our infrastructure, from OS up, so they need to be strong on Linux, scripting, and "infrastructure" tools and concepts, like monitoring, configuration management, etc. But they also need to be able to support the applications themselves, which requires knowing about Java app servers (Tomcat in particular), databases, and web servers. On top of that, they need to be customer-support focused - they don't deal with the end users directly, but do need to deal with internal customers and key people, particularly technical contacts, at our customers and partners.
On top of that we need a Senior Java Architect and an Architect/Senior Developer type. These are again tricky, I firmly believe in Architects being hands-on, and we are an agile team, so they can't just be UML monkeys. But the core development team is offshore in Poland, and our software is complex, so it will be a challenge to get up to speed with it, and be able to interface between the non-technical people in London and the developers there, take ownership of planning a major revision. Cool roles really, but a fairly new thing for us, so we can't be 100% certain how the role will work in practice.
We're also going to hire a QA manager and test engineer.
Fun times here!
]]>The responsibilities for the support team involve:
Our company combines the agility and rapid pace of a four year old software start-up with the stability and resources of an international, publically listed publishing company.
We develop and provide a software service that serves as a healthcare knowledge source across multiple healthcare organisations. We deliver this as a hosted service, and also as software that partner organisations can host.
The Map of Medicine application suite is a Java-server based web application that runs on Linux or Windows on a Tomcat application server using a MySQL, SQL Server, or Oracle database.
Our hosting infrastructure currently spans a production farm in redundant data centres, and a development/test "lab", and will expand to another data centre internationally. Our production systems are primarily Linux with Apache and Tomcat, with some Windows servers running SQL Server. Our test lab uses virtual servers on VMWare ESX, running Oracle, MySQL, and various other bits of technology.
Ensuring that our applications are highly available, reliable, and responsive is a key driver, so we implement monitoring, security, and centralized configuration management using mainly open source tools such as Nagios, Puppet, Subversion, and OpenLDAP, as well as custom scripts. Load balancing, database clustering, and data-centre failover are other cornerstones of our availability strategy.
We must be able to rapidly configure and deploy our applications for testing, pilot projects, and expansion. Centralized configuration management is essential to this, as is our use of virtualization technologies.
We also provide knowledge management services internally and externally, and provide leadership to the rest of the company on how to effectively use these. These are based around our wiki and bug tracker using Confluence and Jira.
The Support and Infrastructure Team Lead needs to be an experienced system administrator who can lead a team of 4+ in planning and implementing new technologies and expanding and improving use of existing technologies. This is a hands-on technical role. The Team Lead needs to be highly customer-focused, able to discuss daily operational problems and issues with non-business people and customers. They will also need to be able to think longer term and be involved in planning new applications and services and consult with client-organisations to plan their technical implementation of our software.
The responsibilities for the support team involve:
Our company combines the agility and rapid pace of a four year old software start-up with the stability and resources of an international, publically listed publishing company.
We develop and provide a software service that serves as a healthcare knowledge source across multiple healthcare organisations. We deliver this as a hosted service, and also as software that partner organisations can host.
The Map of Medicine application suite is a Java-server based web application that runs on Linux or Windows on a Tomcat application server using a MySQL, SQL Server, or Oracle database.
Our hosting infrastructure currently spans a production farm in redundant data centres, and a development/test "lab", and will expand to another data centre internationally. Our production systems are primarily Linux with Apache and Tomcat, with some Windows servers running SQL Server. Our test lab uses virtual servers on VMWare ESX, running Oracle, MySQL, and various other bits of technology.
Ensuring that our applications are highly available, reliable, and responsive is a key driver, so we implement monitoring, security, and centralized configuration management using mainly open source tools such as Nagios, Puppet, Subversion, and OpenLDAP, as well as custom scripts. Load balancing, database clustering, and data-centre failover are other cornerstones of our availability strategy.
We must be able to rapidly configure and deploy our applications for testing, pilot projects, and expansion. Centralized configuration management is essential to this, as is our use of virtualization technologies.
We also provide knowledge management services internally and externally, and provide leadership to the rest of the company on how to effectively use these. These are based around our wiki and bug tracker using Confluence and Jira.
The Support and Infrastructure Team Lead needs to be an experienced system administrator who can lead a team of 4+ in planning and implementing new technologies and expanding and improving use of existing technologies. This is a hands-on technical role. The Team Lead needs to be highly customer-focused, able to discuss daily operational problems and issues with non-business people and customers. They will also need to be able to think longer term and be involved in planning new applications and services and consult with client-organisations to plan their technical implementation of our software.
So the first pattern I came up with in that post was around keeping people involved throughout the process of planning, rolling out, and running a service, rather than having each of these things done by a different set of people, relying on the mythical "knowledge transfer" process. I'm sure there's a lot more to say on that one, but for the moment I'd like to get another idea out.
This pattern can be called DeployOften. I've found that a good rule in most areas of life is: if you find something difficult to get right, do it more often. This is obvious in some contexts - it's called practice - but in day to day business it goes against the grain to seek out painful tasks and repeat them more than is necessary to get through the job. The benefit is that the more you do something, the better you get at it, and it becomes less painful.
The principle of doing difficult things more often is found throughout agile development methodologies, an obvious example being test-driven development. Where testing is usually a painful and dull process, typically skipped or skimped, agile makes it a central behaviour, in fact the very first thing you should do when coding is to write up automated tests.
A difficult and painful part of service management is deploying new or updated software, part of what ITIL calls the transition phase. One organisation I know of tries to release updates to their software every 3 months, and each time it's a trial. Deploying the software to the server inevitably turns up surprises, and acceptance testing by users drags on with multiple rounds as updated releases are built to fix problems discovered.
This is in spite of automated nightly and iteration builds, which somehow never bring out the same issues that come out even on staging servers, which use snapshots of current live data sets, and more rigidly mimic the live deployment environment.
The DeployOften pattern suggests that the operations team should deploy each iteration build onto a production-like environment rather than waiting for the nearly-complete release. This will raise cries from the ops people, who don't exactly have light workloads already. But by deploying every two weeks they will get it down to a very quick process, and also turn up deployment problems much sooner in the development cycle, which should raise the developers' awareness of the kinds of things they need to keep in mind to make deployment easier.
]]>My focus has been much more on the technical than on the people processes for running and supporting the infrastructures. In my current job, the team I've brought in and the infrastructure we've built has reached a decent level, although there's certainly plenty more to do technically. But looking at what we've done and what I want to do next, I've realized thatrather than moving on and doing the same thing at another company, the more interesting challenge will be to take things to the next level.
The next level for me is going beyond the technical to focus on the people and processes. The technology infrastructure is going to grow in size and sophistication, including spreading to multiple data centres globally, but the technical challenges seem like more of the same to me. The challenges that seem newer and more intriguing to me personally are more along the lines of how the hell we're going to organize and coordinate people doing development, infrastructure, and support in three, four, or more countries.
So I went on a course in ITIL version 3. Yikes. ITIL is basically a blueprint for organizing a huge IT operation with lots of bureaucratic processes, forms, and signoffs that will make it nearly impossible to get anything done, and ensure that responsibilities are divided so that nobody who is doing anything productive sees the big picture.
I don't think it has to be this way. I actually did find the course useful, although not as useful as it could have been given that most of the people on it were more interested in ticking off the certification than getting ideas on how to improve the organisations they work at. There were some pretty interesting people there, some of whom were obviously interested in fixing real problems "back home". If the course had been more of a workshop where we shared war stories and ideas, it would have rocked.
A lot of the concepts in ITIL are useful, I think it's more a matter of using your head when applying them, making sure to adapt the ideas in ways that fit your needs and objectives. It's very easy to see how an organisations, especially large ones, take the ITIL material and use it to build horribly inefficient IT structures. I've worked with companies that use ITIL this way, and the course shed light on how they got this way.
The biggest problem with ITIL is that it's presented with clearly defined "phases" of strategizing, desiging, deploying ("transitioning"), operating, and improving IT services. This is an invitation to a waterfall model, where (as in at least one organization I know of) each phase can even be run by a completely different team of people.
So one group designs the service, hands it off to another than rolls it out (tests and installs it), and then hands off to a completely separate team that supports it. In the organisation I've encountered, the operations team hasn't got the vaguest clue about the service.
Of course the transition process involves "knowledge transfer" where the people who set up the service train the support team, but anybody who's done this stuff in the real world should know better.
Knowledge that is transferred in a handover process is never, ever, ever going to be learned as well as knowledge that comes from actually being involved throughout the whole process. Having some hands-off manager (ahem) overseeing things all the way through doesn't cut it, the people who will actually be diving into runtime problems with an application need to have gotten their hands dirty trying to install the application, and even have pitched into meetings where the details of how the application should be integrated into the infrastructure.
Otherwise, you're going to end up in the situation of my nameless organisation, one which is actually often held up as an exemplar of ITIL. They host an application on their servers, installed by the transition team, and their support team had training on how to log into the server and investigate problems with it. But when users call up with problems, the support people, who probably support dozens of applications, have forgotten all of this. They call up the software vendor - who have no access to the servers.
Can you imagine how incompetent your organisation looks when it's clear that your support people have no idea that the application they support is run by their own company?
But I do think it's possible to take many of the ideas of ITIL and apply them in a more agile manner. A bit of Googling shows I'm not the only one who thinks so, but that there doesn't seem to have been much work done on the idea, at least publicly. It's certainly something that would take a bit of thought and work.
My first thought, clearly, is that an agile IT services process would have to embrace the lean management principle of empowerment by having the "workers" (for lack of a better word) involved throughout the process.
I've also thought that the kanban approach to agile is paricularly suited to a sysadmin team, since it does away with the iterations/release cycle in favor of a queue of tasks that people pull from when they find they've got spare capacity.
Anyway, I'm looking forward to thinking this stuff through and trying out ideas over the next year. Although I'm going to be far less hands-on technically, my focus does need to involve a thorough understanding of the technical aspects of what we're doing, so I don't think I'm going to become a total suit.
]]>Found via virtualization.info.
]]>Virtualization.info is by Alessandro Perilli, a consultant who specialises in this stuff.
VMblog seems to be driven by industry press-releases, but is nevertheless a worthwhile resource.
]]>So maybe I can replace replace 4 or more servers with 1, but none of the boxes I currently have can handle that, so I either need to beef some up, or buy newer, bigger boxes. So there is probably some cash outlay, which means that even if I'm using fewer boxes in the end, if I've already paid for the boxes I'm cutting out I may not actually save money.
There's no guarantee I really can fit 4 server images onto a single box and get the same performance I had on 4 different boxes.
Let's look at my java application servers, which are the hungriest animals in my farm. They absolutely must have 2 gigs of RAM each, or they're not gonna work. They actually can't make use of more than this, due to the nature of the 32 bit JVM, so in theory they are an excellent candidate for virtualization.
So I could pimp a few of my current single CPU 2 gig servers with 8 gigs of RAM and 2 dual core CPU's each. Or I could satisfy my techo-lust and get servers with 2 quad core processors each, and 16 gigs of RAM.
In theory my pimped server will have the same power as 4 of my old servers, and the new quad-cored beast could run 8 servers. Now that would be consolidation!
But there's more to hardware than CPU and RAM, so when I load up one of these beasts I could find that the network or disk I/O are a bottleneck, keeping all of my virtual images crawling.
The point is, I don't know how many of my VM's a server can handle until I try it. It's even more difficult in the wonderful world of virtualization, where a box may be running a mixture of web, app, and database servers along with infrastructure services like DNS and email. I might actually get better peformance with a mix of servers than a monoculture where all of the VM's are trying to do the same type of thing and competing for the same hardware resources.
So rule number 1, before I spec out and price up my new virtualized server farm and get it signed off by the bosses, is to do plenty of testing to get an idea of what hardware I'll really need. I ought to test the pimped version of my existing servers, as well as higher specced boxes, maybe with different characteristics. What happens if I use hopped up caching RAID controllers, or high end network cards, or fibre channel rather than iSCSI for my shared storage?
One interesting tidbit I ran across when googling for virtualization limitations is that some of the groovier features can be picky about the hardware. In particular, the capability of shifting virtual machines between physical servers on the fly may require that the physical hardware be very similar, down to the CPU family and chipset. I've read this about VMware's vmotion technology (although I've lost the original reference, sorry).
This means I can't mix and match hardware, and could get trapped by legacy hardware that goes out of production. I can easily see ending up with several pools of hardware, and having to resort to old fashioned manual methods to migrate servers between them. This would mean having to choose between giving up the productivity gains of easy-auto migration or chucking out slightly older, but still perfectly good hardware to buy a raft of newer kit.
What's interesting about virtualization is that it is actually the opposite of the concept - epitomized by Google - of building a utility-style computing infrastructure on lots of cheap, commodity boxes. Instead, I will end up buying a few highly and carefully specified servers.
So let's say I work out that I can trim my current farm to a quarter of its former size, 4 servers per box. My old farm had failover servers, or load balanced servers scoped to be able to cope with one of them going down. I may have virtual servers in my new pool which do the same thing, so if one virtual image crashes others will pick up the slack.
But what happens if one of my physical boxes goes down?
I now have to find homes for all of the virtual servers on that box, but if I've been over-efficient in sizing my hardware capacity I won't have any place to put them. In practice, I will probably have virtual machines I can take offline in a crisis, like my staging servers, and of course the failover images.
But the point is, I need to think about this up front. I'll work out what my tolerance for failure should be - is it enough to be able to cope with 1 server croaking, or should I have 2, or some percentage of my total?
So if I work out that I will run 20 virtual servers on 4 boxes in a new server farm, then I only need to spend 20% of what I would for 20 separate boxes, right? Oops, no, if I'm running a commercial OS like Windows, Red Hat, or Suse, I've still got to buy 20 licenses. Common sense, but easy to overlook when costing initially, and it would be embarrassing to have to go back and ask for the extra budget.
OK, maybe I still have to pay for all those licenses (or I can just use Debian), but at least I only have 4 boxes to manage going forward, rather than 20. Phew!
Nope. That's 20 servers that need to be monitored, backed up, patched, disk space managed, user accounts kept up to date, configurations to be changed, etc.
Even worse, once I go to virtualization, I expect to expand my usage of virtual images to where I'll have some that are kept offline until needed. So I can do certain staging, testing, and other exercises by bringing up images I need for a short while, then putting them back into cold storage. So even with a finely honed, well-automated infrastructure management system, I need to work out how these get updated. Do I cycle them into memory periodically to run updates on, or have the update process (potentially a long one) run when they are brought online as needed?
There is obviously plenty I need to think about when planning to go to virtualization. I'm sure there's more I'm missing, and will learn the hard way. I do still think the payoff can outweigh the difficulties, but we'll see as I go along!
]]>Consolidation is the main selling point of virtualization. You may be able to use a single box to do the same work currently being done by some number (wave hands here) of boxes. Although licensing costs of virtualization software may range up to $5,000 or more, and the boxes will probably need to have more muscle and so cost more, if you can get the ratio of virtual machines to physical machines high enough, you probably can cut your costs.
There are a few keys to this, and some spinoff benefits.
Virtualization isn't likely to help you consolidate servers that are groaning under their workload. But in most data centers, the majority of servers aren't working all that hard. You have them because they're doing some essential service, and probably it's a service that for some reason or other shouldn't be crowded onto the same system with others. Different versions of software, libraries, OS, or whatever mean it needs its own machine.
You try to minimize the hardware you devote to it, put it on a single CPU 1U server, but that's still a fair amount of overhead. Then there are failover servers, staging servers, and various other machines that are usually idle, but need to be right on hand.
So you can have these as virtual servers, sharing a physical box but not using much of its resources unless called to duty.
How much savings you can get from server consolidation will largely depend on how many servers you have which are underutilized.
In my farm, I've got plenty of these. I've got staging servers configured with different sets of OS and application software, to replicate client environments. I've got a server running LDAP, email, and DNS, which almost never returns a number higher than 0.02 when I run uptime. My Apache servers also don't work very hard, but I need two machines devoted to these.
So I'm sure I could compress my running services onto a smaller set of hardware.
Once I've done that, I ought to be able to balance those images between physical boxes to make the most efficient use of my resources. This might be aided by splitting out some things currently on the same system. For instance, my email, LDAP, and DNS servers might each get their own virtual image, so they can be more finely balanced, and also to reduce their interdependence. I can upgrade or replace my DNS server without worrying about breaking a library needed by my LDAP server.
This could be a lot of work, but the better (and more expensive) virtualization tools can make it easier, and even automate it.
This said, there are a number of hidden traps which might make it harder to get these benefits. I'll follow up with my thoughts on what these are.
]]>I already use virtualization on my laptop. I set up virtual images to try out configurations and new versions of software that I run on my servers. Being able to run different operating systems, and also to have systems that I can mangle in various ways without trashing my laptop, is handy. So why run it on my servers?
The concept of virtualizing a server farm is that each server that is fulfilling some role, for example a web server, application server, or database server, can be converted into a virtual image, and more than one can be run on a single server. Things don't look any different to the software now running on a virtual server, nor do they look different to the users of the software. But a single physical server can run multiple virtual images, which can be doing different things, and can even be running different operating systems.
So what are the benefits of doing this?
As I plan our own potential adoption of server virtualization, I'm weighing the shiny wonders of this fabulous technology against a number of pitfalls. The vendors of this software tout the benefits loudly, giving the impression that you can't help but save money, even faced with their often exhorbitant licensing. But reading between the lines, and perusing the word on the Web, turns up some things to think about.
I'll go a little more into the possible benefits, and more into the pitfalls, in separate articles.
]]>Anyway, I've upgraded to a newer version of Drupal, and turned off comments for the time being. I had to switch off my old theme because it has issues. The current theme will probably stick around for a while. I do plan on tackling the comments though.
]]>One of the core components of this infrastructure is an OpenLDAP server, which I've been working on over the past week. Up until now it's been enough to have a couple of accounts which are created locally on all of the servers by puppet. I've got a chunk of disk space on a SAN which is shared across the machines, which is handy for having a common home area for key accounts I use to login and administer the machines, as well as the puppet templates and manifests.
I've added the LDAP server not so much for the server login accounts as for some of the services that we're putting in place for use by the company, in particular a wiki and bug tracking. Rather than having user accounts scattered across various applications and services, LDAP means everyone can have just one username, and one password, and most things that require user accounts are able to integrate with it.
LDAP does have some limitations. There are no really polished admin tools, most of what's out there is pretty rough and ready, with serious gaps. There's a reasonable collection of links to ldap tools on the bind9 site.
The biggest gap for me is letting users manage their own accounts, especially resetting passwords. Most of the users in my directory won't have login accounts on the servers, so I can't just whip up a script that they can use like "passwd".
I've settled on Gosa, which is actually a pretty nice, if some sketchily documented, tool. Once you've got it configured, it lets you create, edit, and delete users and groups, as well as other directory thingies I'm not using yet like machines and applications. Users can also be permitted to log in and edit fields which you enable, including the password. This is great, because it means users can change their own passwords from the Web UI.
Gosa is missing one feature I need, a "lost password" capability. You can only change your password if you know yoru current one. I intend to write a little script that takes a username, generates and sets a random password using ldappasswd, and emails it to the address in their directory entry with a little templated email message.
When a user forgets their password I'll have to run this script by hand, but it'll be quick and simple to run it. If I felt brave I could wrap it in a CGI script to let users run it themselves, but that would open it up to abuse. Even though it wouldn't let a baddy steal accounts, it would let them annoy users by resetting passwords.
Alexander Prohorenko wrote a decent article for ONLamp about setting up Gosa, which I found very helpful.
]]>One of the big ideas I have about Web 2.0 [is] that once we move to software as a service, everything we thought we knew about competitive advantage has to be rethought. Operations becomes the elephant in the room.
O'Reilly laments that most of the tools for deploying systems and applications on open source platforms (i.e. Linux) are not themselves open source. Luke Kaines and others have commented on the article with examples of open source deployment and operations management tools, including Puppet, and others I've mentioned for system configuration and network monitoring.
I think Luke's blog post makes a fair point, that there is more activity in this area than O'Reilly gives credit to, and in fact, the OSCon sponsored by O'Reilly Publishing had turned down Luke's proposal for a talk on Puppet. Luke reports that Tim O'Reilly emailed him directly in response to this, and gave him a slot on the OSCon speaker's schedule.
So this post by O'Reilly should mark a turning point. By expresing his interest in the subject of operations, O'Reilly has invited the commentary from people who are passionate about it, giving himself an opportunity to perhaps learn more about what's out there. I hope we'll see some new books in this area come out from O'Reilly.
I wouldn't expect that Puppet has a wide enough user base to carry a title of its own (yet), but a guide to managing systems and networks with open source tools would be cool, and would fit in well with O'Reilly's existing catalog of systems administration titles.
I also very much like Dominic Mitchell's suggestion that there is a need to work out application deployment patterns. I'd add that patterns for infrastructure operations in general would be really useful, something I'd like to think about a lot more.
]]>Another tool I'd like to try out for this is OpenNMS, which is written in Java, and includes the graphing as well as detection and reporting, and also auto-detects devices and services on a network.
Hyperic is another open source tool in this category I'd like to learn more about.
]]>I think the real division here is content providers vs. pipe owners, and the attempt to do away with network neutrality is essentially a coup attempt by the people who own the pipes.
People use the Net for the content, so that's where the value is. The pipes are just a commodity, which are expected to simply deliver the content. Selling a commodity service means competing on price, which means low margins.
So now the owners of the pipes are trying to restructure the market so they can charge content owners for preferential delivery of their content. This would overturn the current situation in their favor.
I'm doubtful this will work. Restructuring a market to foil the natural flow of the supply and demand of value is like trying to turn back the tide. There are only two ways it will work, barring government intervention (i.e. paying the politicians to restructure the market for them).
The first way is if the Unequal Net really does offer value. So if the content providers really feel a need to pay extra to get faster delivery to customers, they'll pay for it, and the Unequal Net will happen.
The second way is if the telecomms companies set up the Unequal Net, and degrade normal network service to the point where content providers will see the value of paying for the higher tier.
The first way is not going to happen. I've been providing technical infastructure to content providers for 10 years, and I just don't see them paying extra to get faster network to their users. It just isn't a huge pain point, and when it is the focus is how to reduce bandwidth costs, not increase them.
The second way could happen. The current political debate is around exactly this, with the telecomms companies vigorously denying they would slow down normal, non-premium network traffic. Instead they say they will be adding new capacity, and leave the existing Net as it is.
This is disingenuous, of course. As Internet usage continues to grow, with bandwidth hogging applications like video, the existing pipes will become more and more overloaded. Where are the telecomms companies going to invest in expanding capacity, the commodity, low-margin regular Internet, or in the new, high capacity, high cost infrastructure?
One would hope the market would overcome this strategy of blackmail. But there is a class of content providers who will need faster, more reliable bandwidth as they move more seriously into the Net as a distribution channel. They are the same content providers who are frightened of the Net as an open, level playing field. And they are content providers with deep pockets.
So it's easy to imagine a future market with two levels. The premium infrastructure will be for big-corp commercial content, TV, movies, music, and gaming, probably heavy with (ineffectual as always) DRM. The other level will be for the rest of us, the grass-roots, the non-commercial groups, individuals, and micro-businesses.
I don't think this would necessarily have an unhappy ending. That grass-roots market is huge, and as users we will will demand reasonable quality network access, so somebody will be able to make money providing it.
But I think the irony of the situation is that the telecomms companies who will eagerly build the infrastructure for the premium network in hopes of getting a bigger slice of the pie will in the end find themselves again competing for business strictly on price.
At the end of the day, people are interested in the content, and pipes are just pipes. If one telecomms company tries to hold Disney to ransom, they'll just switch to another.
Links:
Tim Berners Lee on Net Neutrality. Is it possible to have a better opening line?
]]>I'm setting up a new hosting infrastructure for a client which, among other things, involves moving from the free Fedora to commercial Redhat Linux. Although I've managed Redhat machines before, this is my first time using the Redhat Network (RHN) for installing and updating software.
In the past I've used apt-get on Debian, and yum on Fedora, and found them a godsend. Set up properly, it takes minimal effort to keep multiple systems up to date and consistent, whereas when I've had to go the "by-hand" route, machines invariably ended up with older versions of software. It's just too hard to keep up with all the various packages installed on various servers, not to mention the headache of chasing down various dependencies and resolving conflicts when you do upgrade or install a new package.
So faced with using RHN, I made the typical assumption of a user who is "upgrading" from a free system to a commercial one, i.e. I assumed it would be better. After all, if you're paying a company for it, it'll be better quality than something maintained by a bunch of volunteers, won't it? I should know better, having seen horrors that go on inside the closed-door sausage factories of commercial software development groups.
So what's my bitch this time? Well, there's not as much software available. The first thing I wanted to do with my new Redhat boxes was checkout my puppet manifests and related configuration code from my subversion repository. To do this, I needed the subversion client but sadly, RHN (at least for Redhat Enterprise 3) doesn't have subversion.
It might be available from channels that I don't have access to, I don't know. If so, it's an example of the quicksand that I always seem to find myself mired in with commercial software. Whenever I get saddled with Microsoft Windows servers, I end up wanting to do things that I could do for free on Linux, but can't do without paying thousands of pounds extra with Microsoft, buying add-ons, third-party software, not to mention "Enterprise" editions of everything when I want to run it on more than one server, all multiplied by CPU's of course. And, oh yeah, we need licenses for our development and staging servers as well. Bleh.
So for now I'm going back to the old fashioned days of searching and downloading packages. So far I'm up to 10 packages I've had to add manually, and will have to keep up to date across 15 or more machines.
I'm considering installing yum and just ditching RHN entirely. I guess this will effectively turn my systems into Fedora boxes, and might have weird side effects. So far I haven't found much commentary out on the web related to this situation.
]]>