I recently started working with RightScale to manage servers in various cloud environments. The servers that I am working with are in the Amazon EC2 cloud, but RightScale provides an interface to manages servers in multiple cloud spaces.

The underlying operating system for the servers that I am managing are running CentOS. While I prefer RedHat, CentOS is fine. I understand the reason for going with both, and when cost is an issue CentOS commonly wins out. The version is CentOS version 5.4.

The problem is that the servers are not getting updates. Based on the RightScale blog, you should be able to choose a RightScale supported frozen image of the CentOS mirrors from any given day, and they will have the most up to date patches to that day. As the servers do not maintain state between reboots, this means you can set the repo for a specific date, and the patches would be consistent for multiple machines no matter when they were booted.

This is the concept. The problem is that there was a glitch in the way that RightScale was updating the CentOS mirrors. As such, there is a length of time, approximately March of 2011 to Oct 13, 2011, where none of the CentOS mirrors were being updated. I am glad to say that working with RightScale support we were able to get them to correct this. The reporting of the glitch was actually documented in a forum discussion, but we also received it during a phone session we had with them.

One would think this would be all well and good, and that you could just choose the release date you wanted to use and be on your merry way. Wrong. Due to the way that the CentOS mirrors work, once a new release is available, they quit releasing updates into the older streams. This means that you have to change the repos to point to the 5/ mirror path instead of 5.4/ . To do this I have created a RightScript that takes to inputs and uses them to update the repo files.

You should run it early on in the process. I put it as my second or third script.

#!/usr/bin/env python

import re
import os
import sys
import subprocess

repoDir = "/etc/yum.repos.d/"

BASE_REPO_VERSION = os.getenv("BASE_REPO_VERSION", "5.4")
NEW_REPO_VERSION = os.getenv("NEW_REPO_VERSION", "5")

# Go through the list of repos, and change the Version from 5.x to base of 5 to get updates.
try:
	for filename in os.listdir(repoDir):
		if re.search("\.repo$", filename):
			try:
				os.rename (repoDir + "/" + filename, repoDir + "/" + filename + ".base")
			except Exception, e:
				sys.stderr.write("Error renaming file: %s\n" % (e))
			try:
				o = open( repoDir + filename, "w")
				data = open( repoDir + filename + ".base").read()
				o.write( re.sub(BASE_REPO_VERSION, NEW_REPO_VERSION, data))
				o.close()
			except Exception, e:
				sys.stderr.write("Error writing modified repo: %s\n" % (e) )
				sys.exit(1)

except Exception, e:
	sys.stderr.write("Error Listing Directory Contents: %s\n"  %  (e))
	sys.exit(1)

# Now we need to update the server with running patches.
# Done.
try:
	sysUpdate = subprocess.Popen(['yum', '--exclude', "kernel'*'", '-y', 'update'], shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
	sysUpdateOutput = sysUpdate.communicate()
	retcode = sysUpdate.returncode
	if retcode != 0:
		sts.stderr.write("Yum update returned an error\n%s\n%s" % (stdout, stderr))
	else:
		print ("Output from yum command:\n%s\n" % (stdout))

except Exception, e:
	print ("Output from yum command:\n%s\n" % (stdout))
	sys.stderr.write("Error Running yum update: %s\n" % (e))
	sys.exit(1)

I have recently added a wiki/faq section to the site. This is mainly for my own notes and thoughts, but I find that if I am working on something, then someone else has as well. You may or may not find the information useful.

I could have waited until I had populated it more fully, but better partial information than none. Most of the items will be computer related, but you never know what I might start looking into.  This being said, I have also not determined how I am going to handle comments. Currently they are disabled. This is mainly due to the wonderful fact that spam bots would fill up the entire thing in about 20 days if I was not careful.

To this effect, I may have to enable some sort of feedback loop to help minimize those types of responses.

As I stated earlier, the first 3 days of the Red Hat rapid track training went by in a blur. The amount of information covered was more than I believe a lot of people could absorb in the short time frame. There were a number of people at the training claiming that they were going to fail the certification exam at the end of the week.

Day 4.

The first half of the day was spent finishing the course material. In a strange twist, the stuff that we covered on the last day was some of the easiest items that we had covered all week. During the training I was able to pull out my laptop and answer a few emails that needed seeing to. While the material covered in the last day did not seem that important it is where I spent the second half of the day that made all the difference.

The course material contains a number of hands on labs. These labs are actually very well written, and actually require you to learn something if you want to be able to complete them. The catch is this. If you cannot complete the labs quickly without looking for how to do them in the notes of your book, then you are probably not ready for the exam, or will run out of time. This was the feel that I was getting, so during part of the morning session, I spent my time going back over the material that we had covered which was new to me. Could I have skipped this? Maybe. Did it help? Definitely.

The second half of the day I spent going over a review lab. This lab has you configure a system utilizing a number of the skills that you learned throughout the week. For the most part I was able to go through the lab of working with firewalls, LVMs, apache, and such without having to refer to the training material. However, I was not fast at it, and it took me 3 to 3.5 hours to complete. Not a good sign when the time that you have during the RHCE (Red Hat Certified Engineer) exam is shorter than this. The review was good though, and I felt that I had a good grasp of the material.

In addition to this, as I was checking my work against the guide in the back, I saw that they had additional work you could do if you wanted more practice. This seemed like a good idea, and I stayed longer to ensure that I could accomplish all of these goals as well.

Day 5.

Exam Day.

The RHCE exam consists of two parts. The first part is the RHCSA (Red Hat Certified System Administrator) and the second half is the RHCE. In order to obtain the RHCE you must first pass the RHCSA exam. Once you have passed both, then you have your RHCE.

In order to take the exam you must sign a non-disclosure agreement. Because of this, I am not able to go into any detail of the exam. I believe it is OK if I say that there is no multiple choice section. The entire exam is a practical exam. You must be able to do the work if you want to get either of the certifications.

You do not receive your results instantaneously. It is a shame, but it is also understandable. Red Hat states you should have them within 72 business hours. I received my results the following Saturday.

Overall thoughts.

Was the class worth it? Yes. There is enough stuff that you may not do on a daily basis to warrant taking the class.

Could you pass the RHCE exam without it? Yes, if you can do everything listed on the objectives provided by Red Hat then you should be fine. (If you can do them fast)

Did I pass? Yes, I passed both the RHCSA and RHCE exams.

The first 3 days of the Red Hat Rapid Track course have been fairly intense. The main reason for this is the shear amount of data that is covered in three days. I have taken other training courses that take 2 days before you even get into the training material. That would not be the case when it comes to the Red Hat rapid track course.

For starters, I am taking the course at a local Red Hat approved training facility. The location is in a part of Atlanta that is probably convenient to the largest number of people around town. If you know the Atlanta area, then it is just north of I-285 off on GA-400. For me it is a bit of a trek, but all in all it has not been a bad commute. If I was doing my normal work hours and not 9 to 5 it would be even better. The facility is clean, and the coffee is much better than what we have at my regular office.

You might be asking what this has to do with the training? Well, the facility and amenities have much to do with  how well training goes. If you are uncomfortable in the office, or feel like you might get mugged walking to the car, you are probably much less likely to be in a good learning mood. Needless to say, the facilities are pretty nice, and the low lighting in the room makes my eyes happy.

Thoughts on the training.

The way that the training is setup is that each student has a Red Hat 6 workstation. This workstation is what is used for all of the labs. The labs utilize the instructors server and a local virtual machine instance using built in kvm. At times, multiple instances are created to simulate a multi-server environment. Overall, the labs provide a good way to become familiar with the various topics that are covered.

What does the class cover? That is the 6 million dollar question. The answer would be a little bit of everything. The easiest way to describe it would be to look at the requirements for the RHCSA and RHCE exams. The class touches on almost all of the topics listed there. The topics range from adding iscsi devices to setting up firewalls with natting to configuring mail services. The course covers more than any but a few engineers would ever encounter at any given job.

So far I would say that the training is definitely worth it. If you are not a high level admin/engineer, I would recommend the courses that split the material up over a two week period. I say this, because the Rapid Track course is definitely not for beginners.

At this point, there is one day of training left, and then the test on Friday. More information coming after this point.

This week I am going to the Red Hat RHCE Rapid Track Course with RHCSA and RHCE Exams included. In the past I have been an RHCE (Red Hat Certified Engineer), but it expired a few years back. As the exam costs several hundred dollars, and it was not a priority, I never went and took the exam again. Now the opportunity has arisen for me not only to take the RHCE exam, but the week long Rapid Track course as well.

It should be interesting to see how the training stacks up to some of the other training that I have had. If it compares to the training that I had for JBoss, then I will be pleasantly surprised. Though, if I remember properly, that training did have areas where it could have been improved.

This is the course description as pulled off of the Red Hat website, at the time of me taking the class.

The RHCE Fast Track Course with RHCSA and RHCE Exams (RH300) is designed for senior Linux system administrators who want to validate their competencies by earning the RHCSA and RHCE credentials. This is a fast-paced preparation course that combines the RHCSA Fast Track Course (RH199) and System Administration III (RH254) courses, normally eight days of training, into a single four-day course. Building on the students’ extensive knowledge of command line based Linux administration, the course moves very quickly through the intermediate and advanced tasks covered by lab-based knowledge checks and facilitative discussions. By the end of this course, the senior Linux administrator students will have been exposed to all the intermediate and advanced competencies tested by the RHCSA and RHCE exams. The RHCSA and RHCE exams are included with this course.

Here is to a good week of training, and updates on how it goes.