VMWare Server Console 2.0

I just upgraded VMWare server to version 2.0.2 on a Linux machine. THEN I realized that the friendly folks at VMWare had taken most of what I knew about connecting to a virtual machine via the VMWare console and tossed it out in the waste bin.

The first clue was when the old VMWare console application, the one you used to be able to install in the normal way you install other windows apps, wouldn't let me connect any more. What exactly is "501 Global command GLOBAL server to non-host agent targets not supported" supposed to mean to me anyway? Well? VMWare? Are you trying to be as obtuse with your error messages as your buddies in Redmond? If so, then mission accomplished!!

So I relented, and read enough of the manual to realize that the whole remote console in VMWare server is now rewritten as some kind of web-browser based monster. I knew that I had set values for the http and https ports, 8222 and 8333 respectively, when I ran the vmware-configure script. I also knew that I would need to allow inbound access to those ports in the Linux firewall (iptables) configuration, so I had already added:
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8222 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8333 -j ACCEPT
What I didn't know is that using the web-browser based VMWare console is the only way in v2.0, at least initially, to access the virtual machine guests.

Getting the web-browser based VMWare console application to show me the console was a little frustrating. VMWare still has something to fix on this part of it, but until they do, if you get a blank page on the https:/myvmwareserverbox:8333/ui page:
1. Accept the SSL certificate
2. Reload/refresh the page (ctrl + F5) over and over until the login prompt finally shows up.

But wait, there's more!!!
Not only do you have to throw the Ginsu knives up in the air several times before they stick in your meatloaf, you have to pull out the nonstick pan instead of cooking on a diamond ring. Ok, it has nothing to do with mid-80s "as seen on TV" cookware, but it's about that goofy.

You need to install a browser add-on to see the desktop of any virtual machine guests. Before you're given the opportunity to install the browser add-on, you have to start one of the virtual machine guests by selecting it from the inventory list and clicking the "play" button, and then switch to its console tab. THEN you're offered the chance to install the browser add-on. At this point I was getting worried that all access to my virtual machine guests would be trapped inside a web browser window/tab. However, as it turns out, the browser add-on is actually the core client binary executable application portion of the old vmware console app.

The documentation detailing how to use this client binary outside of a browser has a few holes in it. This blog post is my feeble attempt to fill a few of those holes.

1. Finding the path to the executable.
Since the VMWare console client is installed as a browser add-on, it will most likely be buried somewhere deep in the OS user's browser profile directory. This varies depending on which browser and which OS, but with a VM guest (named MyVmGuest for instance) selected in the Inventory panel of the web-browser based VMWare console app, in the "Command" panel of the "Summary" tab, there should be a link labeled "Generate Virtual Machine Shortcut." Clicking that link pops up a box that has a link labeled "Install Desktop Shortcut to MyVmGuest" Click that, answer the prompts, and then examine the target of the shortcut it creates. On a windows machine using Firefox, it's probably something ugly like:
C:\Users\whirly\AppData\Roaming\Mozilla\Firefox\Profiles\jkbr2e9f.default\extensions\VMwareVMRC@vmware.com\plugins\vmware-vmrc.exe
That vmware-vmrc.exe IS the new console client binary!!!

2. Understanding the parameters.
I didn't look all that thoroughly in the documentation but most of what I found on VMWare's site, or elsewhere, about the parameters to vmware-vmrc.exe was in discussion thread form and required a bit of noise filtering to isolate the speculative conclusions that actually worked.

-X - Start in the client console for the VM guest in full screen mode.

-h {hostname} - this is just what it sounds like, the host name or ip address where VMWare server is running. However, unlike the old vmware console, which had a -P switch for the port, this one just takes a hostname:port combined, like myvmwareserverbox:8333

-u {username} - the username on the vmware server host that is permitted to access the vm guest. This is only useful in combination with the password switch. Note: Does not work as of v2.0.2 in combination with the -M switch.

-p {password} - supplies the password to be used with the username In the old version of vmware console, this was the -w switch. -p makes more sense I guess. Note: Does not work as of v2.0.2 in combination with the -M switch.

-M ## - this identifies the VM guest from the inventory in a simplified way (example -M 33). Unfortunately, this does not work when combined with the -u username -p password parameters.

{vm guest datastore path} - This is the last parameter and replaces the -M ## to specify a VM guest. The format of this may be documented somewhere, but I didn't end up finding that either. The forum posts I found on the topic were difficult to decipher because the brackets were translated into markup characters by the forum app. The format is:
"[standard] vmguest subdir/vmguest name.vmx"
Note: [standard] is a verbatim part of the pattern and is followed by a space. It tells VMWare Server to find the subdir and vmx file in the configured location for vmware guests.
Note: The -u and -p switches only work with this form of specifying the VM guest.

So, here's a full example of the target to a "click it and connect" shortcut (ignoring security precautions related to embedding a password):

"C:\Users\whirly\AppData\Roaming\Mozilla\Firefox\Profiles\jkbr2e9f.default\extensions\VMwareVMRC@vmware.com\plugins\vmware-vmrc.exe" -h myvmwareserverbox:8333 -u "vmwareadmin" -p "myvmwarepw" "[standard] Old Windows 2000 Pro VM/Old Windows 2000 Pro VM.vmx"

• Note: C:\Users is the standard home directory path for Windows Vista. On XP, this part of the path would probably be C:\Documents and Settings
  
• Note: whirly, in the example, is just the username on the client machine
  
• Note: The quotes around the executable path, username, password, and vm guest datastore path are necessary if there are spaces in the corresponding parameter values. Quotes may not be necessary when there are no spaces in a parameter value, but using quotes anyway removes some guessing if something still doesn't work.
• Note: Old Windows 2000 Pro VM, in the example, is the subdirectory that appears within /var/lib/vmware/virtual_machines on the Linux machine. /var/lib/vmware/virtual_machines is the location that is set in the VMWare Server configuration as the default place to store virtual machine guests.
• Note: Old Windows 2000 Pro VM.vmx is the VM guest configuration file that appears within the Old Windows 2000 Pro VM subdirectory.
• Note: In the datastore path, [standard] is interpreted on the VMWare Server end as equivalent to the configured default path of /var/lib/vmware/virtual_machines
  

I'm sure the reasons for these changes are related to VMWare's consolidation of components between the legacy GSX product and the ESX / infrastructure product line, which ultimately allows the development team to focus on features instead of dividing attention and support between products that have bigger differences in their behavior. But, I just can't resist being a little sarcastic, so I'd like to say thanks again VMWare for changing everything. I'm looking forward to learning it all from scratch again when 3.0 is released ;)

There are other posts, similar to this one, that refer to the ESXi product line. I found a few (links below) that may help if this didn't answer your questions. Searching for the executable name vmware-vmrc might find other information on the topic too.

• This one is about Generating a Virtual Machine Shortcut for ESXi 4
• This one is also about VM Shortcuts in ESXi
  

Another Fight between SELinux and PHP

After I installed the web portion of ZoneMinder, and navigated to the main console page, the system message log started filling up with a message saying:

SELinux is preventing /usr/bin/uptime from using potentially mislabeled files utmp (initrc_var_run_t)

Great, I thought... another annoying SELinux nag message to eliminate and another convoluted SELinux policy setting to figure out. Well, this time it was just PHP code, so then I thought, maybe there's an easier way to fix this one. There was.

ZoneMinder's PHP code was parsing the output of the uptime command just to get the system load information, and as long as the browser was sitting on the ZM console page, it would call the function again every few minutes. However, in addition to the system load info, the uptime command also collects the OS level user-session info by reading the /var/run/utmp file. SELinux prevents that file from being read by anything running in the web server (Apache) process. After a little searching around, I found that the load information is also available by just examining /proc/loadavg, so I changed the PHP code from
shell_exec('uptime')
to
shell_exec('cat /proc/loadavg')
which SELinux doesn't seem to care anything about.

When the uptime command is used in a PHP script, the output could be parsed in various ways, but in the ZoneMinder PHP code, the output from shell_exec( 'uptime' ), which looks like this:
16:22:09 up 284 days, 1:04, 1 user, load average: 0.06, 0.06, 0.00
was being parsed by the preg_match PHP function like this:
preg_match( '/load average: ([\d.]+)/', $uptime, $matches )
so that $matches[1] got the value of the first system load number.

Instead, since /proc/loadavg only has the system load numbers in it like this:
0.06 0.06 0.00 1/239 5581
the preg_match PHP function needed to be a little different, like this:
preg_match( '/([\d.]+)/', $loadRaw, $matches )
and the rest of the code worked just like it did when it used the output from the uptime command.

Now the logs are quiet again, SELinux is still standing guard and preventing access to utmp from anything running in Apache, and I can get back to whatever I was doing before this annoyance from SELinux popped up... well, if I can remember what that was. I think SELinux must be preventing access to my own short term memory now. Grrr.


Apparently, this will not be changed in SELinux's default policies per: https://bugzilla.redhat.com/show_bug.cgi?id=544145#c1

I reported this to the author(s) of ZoneMinder in case they care to fix it permanently in their product's PHP code. But, I'm sure this is a larger problem since there are many references floating around that describe executing the uptime command from PHP. ZoneMinder also has other SELinux conflicts, but they'll have to wait for another day... probably after I find a decent webcam for Linux that has a stable v4l compatible driver.

Joomla! Installation Gotchas

After installing a number of Unix/Linux applications (at least the ones for which there is no convenient RPM in a repository somewhere) you get accustomed to the routine of "tar-xzf some_software-1.2.3.tar.gz", then "cd some_software-1.2.3", followed by "./configure", then "make", sometimes "make test", and finally "make install". It's just a little inconvenient when something isn't packaged that way and you actually have to find and read the install-readme file. Likewise, most LAMP (Linux Apache MySQL PHP) applications follow an install sequence of "cd my_webserver_root", "tar -xzf /my_downloads/some_lamp_app.tar.gz", "cd some_lamp_app", maybe "cp config.php-dist config.php", then "chgrp apache config.php" (or similar), and "chmod 775 config.php", followed by pointing the browser to "http://mywebsite/some_lamp_app/installer" (or similar). It's just a little inconvenient when something doesn't quite follow that pattern, and more so if it chews up time trying to figure out why something is subtly different. Joomla! has a few of those little time munchers so until they fix it, here are a few things that I stumbled over and how I got them fixed.

Unpacking Joomla!
It is just irritating to unpack any distribution and find that it has created more than a single directory within which the rest of the contents are unpacked. Unfortunately, Joomla! is distributed in an archive with no internal top level directory to contain the rest. Unless you want a few dozen files unpacked directly into your working directory, Create a directory first, change into that directory, and THEN
'tar -xzf Joomla_1.5.15-Stable-Full_Package.tar.gz'

FTP Layer Setup
Issue 1: If you get an message like: Error: the XML response that was returned from the server is invalid, you may have made the same mistake I made by copying /var/www/http/joomla_1_5_15/configuration.php-dist to configuration.php. Apparently, the installer script depends on code somewhere in the product that uses the ftp settings in the existing configuration.php file instead of what you filled in on the form. So, instead of copying configuration.php-dist, just create an empty configuration.php, adjust its permissions so that the installer can write to it at the end, and start the install process over. Also, since you would have already gone past the database table creation step, the advanced options on the database connection information page allow you to drop the previous tables before creating new ones.
Note: Even if you got through the install process, this same issue can show up the first time the actual Joomla site is accessed but the message may be something like: JError Unable to load Database Driver Starting with an empty configuration.php seems to avoid this too.

Issue 2: If the "Verify FTP Settings" button reports that the ftp server may be incompatible, or it says that it the STOR ftp command failed, the Joomla! install directory is probably not writable and executable by the ftp user account (Note: Directories in a Unix/Linux file system must be 'executable' in order for their contents to be listed). Most likely this comes up because the ftp user account provided for Joomla's to use is a non-shell access account that is different from the owner of the files in the install directory. For example, if the ftp user account were 'joomlaftp' and the owner of the joomla files is set to a shell account called 'webmaster'). The group on the Joomla directory and files within it may be set to the group under which the webserver runs so that if the webserver process needs to write anything (e.g. an uploads directory) the write access can be granted using something like chmod g+w uploads. It probably wouldn't be a good idea to add the 'webmaster' group to the ftp user account's group list, because that could extend it's reach past the Joomla files, and defeat the purpose of even having the FTP Layer. It may not be practical to change the group on the Joomla files either. So, on most recent Unix/Linux distributions, ACL rules can provide more specific access on specific files, to a specific user or group. Add read/write/execute to the Joomla install directory for the special Joomla FTP account (-R is 'recursive' and -m is 'modify'):

setfacl -R -m u:joomlaftp:rwx /var/www/http/joomla_1_5_15

Then verify that the permissions now consist of the normal file system permissions plus the ACL.

getfacl /var/www/http/joomla_1_5_15
getfacl: Removing leading '/' from absolute path names
# file: var/www/http/joomla_1_5_15
# owner: webmaster
# group: apache
user::rwx
user:joomlaftp:rwx
group::r-x
mask::rwx
other::r-x


64 bit Linux Zip Library Bug
This has been reported to the Joomla! developers (see tracker 15044) but either they think it is a closed issue because they tested on a 32 bit system, or maybe it just isn't very high on the priority list because 64 bit Linux systems are a secondary concern. This issue may not affect more "bleeding edge" distributions but does affect CentOS 5 (i.e. RedHat Enterprise Linux 5) x86_64 distributions. If uploading a plugin or a template that is in pkzip (*.zip) format just silently fails on your 64bit system, you can try (at your own risk of course) applying this patch to the Joomla zip.php library code by saving the following patch file text into a file and running these commands...




cd /var/www/http/joomla_1_5_15/libraries/joomla/filesystem/archive
mv zip.php zip.php.orig
patch --ignore-whitespace < zip_php_fix.patch




------------- snip file: zip_php_fix.patch ----------------

--- zip.php.orig
+++ zip.php
@@ -223,8 +223,8 @@
   */
  function _extractNative($archive, $destination, $options)
  {
-               if ($zip = zip_open($archive)) {
-                       if (is_resource($zip)) {
+               $zip = zip_open($archive);
+               if (is_resource($zip)) {
                          // Make sure the destination folder exists
                          if (!JFolder::create($destination)) {
                                  $this->set('error.message', 'Unable to create destination');
@@ -248,9 +248,8 @@
                                  }
                          }
                          @zip_close($zip);
-                       }
          } else {
-                       $this->set('error.message', 'Unable to open archive');
+                       $this->set('error.message', 'Unable to open archive. Error code '.$zip);
                  return false;
          }
          return true;

------------- snip file: zip_php_fix.patch ----------------




SELINUX Issues
It is a struggle to resist the urge to just shut off Security Enhanced Linux. SELinux is always locking up something to send you on a quest to figure out how to unlock it. Even if SELinux does some useful things in terms of protecting against intrusions or unauthorized modifications, it may just be a notch more difficult than sendmail to configure it correctly. That said, these rules for Joomla's files may not be exactly right, but they seem to give Joomla the access it needs while keeping SELinux set to enforcing. (Note: This assumes that everything in /var/www is already labeled by a default SELinux rule as httpd_sys_content_t but you may need to add a corresponding rule explictly if Joomla's install directory is elsewhere in the file system).

semanage fcontext -a -t httpd_sys_script_rw_t '/var/www/html/joomla_1_5_15/components(/.*)?' 
semanage fcontext -a -t httpd_sys_script_rw_t '/var/www/html/joomla_1_5_15/configuration\.php' 
semanage fcontext -a -t httpd_sys_script_rw_t '/var/www/html/joomla_1_5_15/images(/.*)?' 
semanage fcontext -a -t httpd_sys_script_rw_t '/var/www/html/joomla_1_5_15/media(/.*)?' 
semanage fcontext -a -t httpd_sys_script_rw_t '/var/www/html/joomla_1_5_15/modules(/.*)?' 
semanage fcontext -a -t httpd_sys_script_rw_t '/var/www/html/joomla_1_5_15/plugins(/.*)?' 
semanage fcontext -a -t httpd_sys_script_rw_t '/var/www/html/joomla_1_5_15/templates(/.*)?' 
semanage fcontext -a -t httpd_sys_script_rw_t '/var/www/html/joomla_1_5_15/tmp(/.*)?' 
restorecon -R -F -v /www/joomlainstalldir

Hardware FakeRAID + LVM == A Big Mess

Every time I install a new distribution of Linux on newer hardware, I wonder just what new "features" will start brewing together in a little digital tempest until the universe of malicious luck senses that ideal, worst possible moment and twists the software around the hardware in a big convoluted knot with glue poured on it.

This time it was CentOS 5.x, a VIA brand hardware raid controller, and the Linux Volume Manager. Not knowing much about how Linux interacts with hardware RAID, I just set everything up at install time using the installer's suggested defaults. What could possibly go wrong? Doh! Never ask that question, even silently in your thoughts. You're just asking to get smacked in the head with the answer.

Then it came, while I was out of town and was depending on this machine to grant me access to the rest of the computers in the house via ssh tunnels. RAID1 mirrored volumes got out of sync, the kernel exhibited it's most endearing neurosis and reported its panic, leaving me locked out until I could get home and alleviate the poor kernel's anxiety attack. It often isn't a very good idea to tell a kernel that everything's okay until you have fixed something, so at least you can believe that you're telling it the truth. So, on to the harder part of fixing something, figuring out what needs to be fixed.

In the days that followed, I managed to disable one of the drives in the mirrored pair and get things running again without the benefit of RAID. However, in the process, the Linux Volume Manager came to believe that since there were two physical volumes with the same UUID, it should mount and use one, and hide the other one. LVM hid the /dev/sdb2 partition in a secret house of mirrors in order to thwart every one of the my attempts to obtain access to it, no matter how clever. Looking back, I wonder if it could have anticipated that I'd open the case, and boot up with only one of the drives' SATA cable plugged in but I somehow managed to do that before LVM decided to delete all my data out of spite. The plan was to use pvchange --uuid on one of the drives so that LVM would know the difference, but that doesn't work while the drive is in use. Rebooting in rescue mode from a CD/DVD (the CentOS 5.1 install disc) allowed the drive to be left unmounted, and allowed the pvchange --uuid command to do its thing.

After many unfruitful searches on Google with every permutation of command names, error message text fragments, and configuration file names I could muster, I tripped over a description of something called fake-RAID. All the symptoms of what happened to my Linux machine seemed to be telling me that that's the disease it had contracted. LVM reported a duplicate volume. The BIOS RAID1 mirrored pair still showed up to the OS as two individual devices. The rear fan made a sneezing noise. Okay, that last one probably wasn't related to the fake-RAID thing. The same article that described fake-RAID, recommended against using hardware RAID on a disk controller that works that way because the Linux drivers are an afterthought from the vendor, at best, and most likely guesswork from the Linux kernel developers. Reboot, enter BIOS RAID utility, delete RAID1 array, verify that POST shows 2 standalone drives in normal non-RAID SATA mode, done.

I have to say it was hard to suppress the feeling of accomplishment that came from making the hardware RAID controller do nothing instead of doing half of something, but my euphoria was short lived. I still had a river crossing ahead of me on the slippery stones that are Linux configuration files, and I didn't want to cross without at least the illusion of a safety rope. The rope couldn't be a worthy illusion if it were tied to a log floating in the river where the hazards lay so I plugged in an external USB hard drive and used the dump command to make my "rope," which took about 8 hours at 5MB/sec to transfer the 144GB root filesystem that I would soon attempt to destroy, multiple times.

With the backup made, it was time to shift data, reconfigure, shift data again, and reconfigure again. It would take time and hassle if I slipped from one of these stones but my safety rope gave me the confidence to step and leap towards the far bank.

1. fdisk -l -- to show the partitions and types at the start. There were 2 drives, with 2 partitions each. /dev/sda1 and /dev/sdb1 were the previously mirrored /boot partitions, so for now, they'll be left alone. /dev/sda2 and /dev/sdb2 were the previously mirrored PV partitions mapped to /VolGroup00/LogVol00 (root) and LogVol01 (swap). Only /dev/sda2 was now active as an LVM physical volume (PV). /dev/sdb2 was still a "Linux LVM" partition type, but was no longer mounted or included in the LVM Volume Group VolGroup00.
2. fdisk /dev/sdb -- to change the type id for the larger partition (i.e. sdb2) to 'fd' - Linux raid autodetect" (fdisk inputs: t, 2, fd, w -- to change the type id, on partition 2, to 'fd', and then write).
3. mdadm --zero-superblock /dev/sdb2 -- Since this wasn't part of a Linux software RAID array before, this step wasn't really necessary, but it's handy to know that the superblock is how Linux software RAID recognizes that a physical partition is part of a defined array. Zeroing the superblock is essentially the permanent removal of the partition from any defined array, so it avoids a warning message that would appear if the partition is specified as one of the devices when creating a new array.
4. mdadm --create /dev/md0 --verbose --level=1 --raid-devices=2 missing /dev/sdb1 -- This creates a new array with a device name of /dev/md0 that is a mirrored pair and specifies that one of the devices will be added later ('missing') and the other device should be /dev/sdb2 (which just had its superblock zeroed so it could be added to this array without warnings.)
5. pvcreate /dev/md0 -- This tells LVM that it can use /dev/md0 as a physical volume (PV) which can be added to a volume group. Note: LVM manages the space provided in a PV as a set of "extents" so there is no need to create a file system like ext3 on this or the underlying /dev/md0.
6. vgdisplay -- Show information about the LVM volume groups on the system. This allows verification of the VolGroup00 name to which the new 1/2 mirror array will be added.
7. vgextend -v VolGroup00 /dev/md0 -- This adds the PV /dev/md0 (an LVM physical volume which is referenced by its physical device name) to the volume group named VolGroup00. Now any data that is on a logical volume in the VolGroup00 volume group can be moved to the extents that are available on the /dev/md0 PV.
   
8. pvdisplay - shows that there are now 2 physical volumes (PV's) /dev/sda2 and /dev/md0 that are both allocated to the VolGroup00 volume group.
   
9. pvmove -v /dev/sda2 /dev/md0 -- This gives LVM instructions to migrate the logical volumes (e.g. LogVol00 and LogVol01) from the PV where all their extents are currently stored (/dev/sda2) over to the new PV (/dev/md0) Note: This may run for a while but the -v switch makes it report progress until it's done.
   
10. pvdisplay - Shows the details of the two LVM PV's in order to verify that /dev/md0 has all of its PE's (extents) allocated now (Free PE = 0), and /dev/sda2 has all of its PE's free (Allocated PE = 0)
    
11. vgreduce VolGroup00 /dev/sda2 -- This removes the PV (/dev/sda2) from the LVM volume group VolGroup00. Until this is done, the /dev/sda2 OS device cannot be reclaimed from LVM control.
12. pvremove /dev/sda2 -- Since the /dev/sda2 PV is now no longer allocated to any volume group, this actually removes the corresponding /dev/sda2 partition from LVM control and frees it for use outside LVM
13. fdisk /dev/sda -- to change the type id for the now freed partition (sda2) to 'fd' - Linux raid autodetect" (fdisk inputs: t, 2, fd, w -- to change the type id, on partition 2, to 'fd', and then write).
14. mdadm --add /dev/md0 /dev/sda2 -- This completes the 2 volume RAID1 mirrored pair named /dev/md0 by adding back the partition (/dev/sda2) from which data was just migrated onto the the array via pvmove. Now mdadm will automatically sync the data that was in the array (physically stored on /dev/sdb2) back over to the device that was added to the array (/dev/sda2).
15. watch cat /proc/mdstat -- Monitors the sync up progress. This will probably take just about as long as pvmove took to migrate the data to the /dev/md0 PV.

A few VERY Important Final Steps
The question that kept bugging me as I did all this was, "Will the boot process work correctly after these changes next time I reboot?" The answer to that required a few trips back to rescue mode booted from the CentOS install DVD. I suspect I would not have had to make those trips if I had known the following things.

1. It appeared that the configuration file for mdadm (/etc/mdadm.conf in CentOS 5.x) needed to be correctly configured to reference the mount points and device uuid of the /dev/md0 array. I was still a little fuzzy on how this config file could affect things at boot time. After all, the file is stored in the filesystem that is managed by LVM in a Logical Volume that in turn stores its data as extents in a physical volume (PV) that IS the software RAID array. I couldn't find anything in the /boot/grub setup that defined anything about the raid array or LVM, so how could /etc/mdadm.conf affect the initialization of the software RAID array at boot time, BEFORE the array or LVM is even activated? It doesn't. The boot process happens in stages. The stage where the boot and root filesystems are re-mounted as read/write is where /etc/mdadm.conf matters. So it must be correct. These commands will update it to reflect the actual, active Software RAID arrays.
   mv /etc/mdadm.conf /etc/mdadm.conf.old
   echo "DEVICE partitions" > /etc/mdadm.conf
   echo "MAILADDR root" >> /etc/mdadm.conf
   mdadm --examine --scan >> /etc/mdadm.conf
   
   
2. The init ram drive (initrd) boot image MUST be updated to include support for the software RAID (mdadm/dm-*) modules, so one of the things that I needed to do was build another initrd.?????.img file in the /boot directory (saving the old one just in case). The reason nothing in the grub config files offers a clue to why booting fails is that the relevant stuff is actually embedded within the init ram drive file /boot/initrd-{version}.img. Building a new initrd can be done using mkinitrd to generate an new init script that starts up Software RAID (md) and LVM based on the current, active system configuration. The generated init script embedded inside the initrd-{version}.img file will then have a section like this that bootstraps Software RAID and LVM enough to continue the boot process using resources on a root filesystem that is accessible only with RAID and LVM active:
   ...
   echo Scanning and configuring dmraid supported devices
   raidautorun /dev/md0
   echo Scanning logical volumes
   lvm vgscan --ignorelockingfailure
   echo Activating logical volumes
   lvm vgchange -ay --ignorelockingfailure VolGroup00
   resume /dev/VolGroup00/LogVol01
   echo Creating root device.
   mkrootdev -t ext3 -o defaults,ro /dev/VolGroup00/LogVol00
   echo Mounting root filesystem.
   mount /sysroot
   ...
   So, with all the Software RAID and LVM stuff running, so that mkinitrd will detect the RAID devices (/dev/md*), and detect the LVM configuration (/dev/mapper/*), and include any required boot-time kernel modules, and generate a workable init script, here's what I should have done BEFORE trying to reboot...
   mv /boot/initrd-$(uname -r).img /boot/initrd-$(uname -r).img.bak
   mkinitrd -v /boot/initrd-$(uname -r).img $(uname -r)
   
   

This blog post is partially a way to capture some of the solution and share it with someone else who is perplexed by the same issue, and the remainder of it is me taking an opportunity to write something on this blog that isn't totally useless. I hope you have enjoyed it as much as I have.

Addendum on Fixing initrd from a Rescue Environment
As you may have gathered, I tried to reboot before I had initrd-{version}.img updated to reflect the correct boot-time system configuration. I'm sure someone knows a faster, easier way to get initrd fixed from a rescue CD/DVD boot environment, but as is often the case, trudging through it helped me understand better how things work. The process for manually repairing this part of the boot environment consisted of mounting the boot partition, unpacking the initrd file, adding missing files, editing the init script, and repacking the initrd file for the next reboot. If you're in a similar spot, for this or some other reason, the following steps and references may help.

Manually repairing the init script and/or contents in the initrd (init ram drive) image.

REFERENCE: http://www.ibm.com/developerworks/linux/library/l-initrd.html

REFERENCE: http://wiki.openvz.org/Modifying_initrd_image

1. boot in rescue mode from the install CD/DVD but skip mounting the root filesystem
2. mount the boot partition

   mkdir /boot

   mount /dev/sda1 /boot

3. Make a copy of the existing initrd-{version}.img file with a .gz extension and uncompress it

   cd /boot

   cp initrd-2.6.18-53.1.6.el5.img initrd_for_unpack.img.gz

   gunzip initrd_for_unpack.img.gz

4. Create a temp directory and unpack the initrd.img contents into it

   mkdir initrd_unpacked

   cd initrd_unpacked

   cpio -i --make-directories < ../initrd_for_unpack.img

5. Find and fix what's wrong with the init script or initrd contents (e.g. add missing modules, correct device name references in the init script, etc.) For example:
   

   mkdir /mnt/sysimage
   

   mount {actual-path-to-root-fs} /mnt/sysimage
   

   cp /mnt/sysimage/lib/modules/{vers}/kernel/drivers/dm-zero.ko lib/
   

   cp /mnt/sysimage/lib/modules/{vers}/kernel/drivers/dm-mod.ko lib/
   

   cp /mnt/sysimage/lib/modules/{vers}/kernel/drivers/dm-mirror.ko lib/
   

   cp /mnt/sysimage/lib/modules/{vers}/kernel/drivers/dm-snapshot.ko lib/
   

   ...etc...
   

   vi init

6. Re-pack a new initrd.img file and compress it

   find ./ | cpio -F newc -o > ../initrd_fixed.img

   cd ..

   gzip initrd_fixed.img

7. Replace the existing initrd file with the fixed one.

   mv initrd-2.6.18-53.1.6.el5.img initrd-2.6.18-53.1.6.el5.img.bak

   mv initrd_fixed.img.gz initrd-2.6.18-53.1.6.el5.img

8. Unmount whatever was manually mounted and reboot

   umount /boot

   unmount /mnt/sysimage
   

   reboot

Tech Note: Sendmail MASQUERADE_AS

Some email service providers, including the one I use (1and1.com), have recently clamped down on spam by refusing email with an invalid sender domain in the header or the from-address. While I'm glad to see less garbage in my inbox, I had to dive into the Linux / sendmail configuration rabbit hole to get simple notification emails to show up again. The issue is that only the actual registered domain name is likely to be considered valid by these clamped-down email recipients' hosts. In other words, an email from backupsystem@machineabc.myregistereddomain.com will be refused, but backupsystem@myregistereddomain.com would work just fine.

So, after digging through web search results and documentation for a while, I found that sendmail (the service on Linux/Unix/*nix that forwards email messages) can be configured to "masquerade" local email addresses so that they appear to be sent from the base registered domain. So, according to the documentation at http://www.sendmail.org/m4/masquerading.html I set the following configuration options in /etc/mail/sendmail.mc

MASQUERADE_AS(`myregistereddomain.com')
FEATURE(`masquerade_envelope')
FEATURE(`masquerade_entire_domain')
MASQUERADE_DOMAIN(`myregistereddomain.com')

Then, a few commands to activate the configuration...
m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf
service sendmail restart

Of course this all has to be done as the root account since the modifications are being made to system configuration files. So, the next step was to send a simple test email to see if it worked, but here's where I stepped on one of those semi-hidden (i.e. inconspicuously documented) Linux software configuration land mines. Still working as the root account, I used the mail command to send a message to a GMail account so I could examine the newly masqueraded headers, from-address, etc.
mail -s "Test Sendmail Masquerading" mytestaccount@gmail.com
type in some text for the message body
type ctrl-d to finish the body, close and send

Note: As of December 2009, GMail still accepts email even if it can't resolve the sender domain, and the GMail message view has a "Show Original" option that displays the original, raw email text including headers.

The Linux aggravation meter went up a notch when I noticed that the headers in the email had NOT been masqueraded as advertised. They were all passed through in the form of root@machineabc.myregistereddomain.com. The land mine I mentioned was found in the generated /etc/mail/sendmail.cf file as the following few lines:
# class E: names that should be exposed as from this host, even if we masquerade
...
C{E}root

That means the root account is excluded from the masquerading rules. This makes sense because most of the email originated using the root account would probably pertain to the specific machine and it would usually be helpful if the machine name were included in the sender header or from-address. However, the recently clamped-down email recipients's hosts simply refuse to accept such emails. The documentation does mention that the root account will always be "exposed" but I hadn't noticed that before. Grrrr.

If you're not a sendmail administration expert, it may not be that obvious how to include the root account in the masquerading. Notification emails delivered to a clamped-down email host that originate from a non-root user on the Linux machine work fine. Just to be sure this will work, enter the same mail command as another user on the machine and it does exactly what the configuration options would lead you to expect. If you want to keep the root account from being excluded from the masquerading, you need to GET RID OF a configuration option, not add another one. Elsewhere in the sendmail.mc file, there is a line that explicitly excludes root as an EXPOSED USER account:

EXPOSED_USER(`root')dnl

Comment it out by changing it to:

dnl # EXPOSED_USER(`root')dnl

And remember to run m4 to regenerate the sendmail.cf file and restart the sendmail service again as described above. Now emails sent from the processes that run as the root account should also get the masquerade treatment.

The other option...
...if you have the ability to manage subdomains, and you only have one or a few machines for which you need the emails to be delivered, then you can add a subdomain to DNS for each of the machines. For instance, if you had machineabc set up in domain myregistereddomain.com, you could add a subdomain to your DNS configuration for machineabc.myregistereddomain.com. Then the clamped-down email servers would look up the sender domain for an email from root@machineabc.myregistereddomain.com, find it, and accept the email as usual.

I'm posting this to save someone (maybe even myself) a little time later on. I searched for a while to see if I could find out why the masquerade options for sendmail were just not working but as usual, they were working, but I was experiencing the effects of the built in exception case and didn't even have a good starting point for a search that would yield an answer.

Shopophrenia

As you'd guess from the title, I'm talking about the 2-mindedness of purchasing decisions. This has become worse for me lately with the spend-happy democrats in control of our national finances, making all the same stupid mistakes made by FDR and the U.S.S.R., and bringing Katrina-like dark skies into our economic forecast. As if the specter of ever-leapfrogging cheaper technology weren't reluctance-inducing enough, now I have another reason to wait and see on so many things I'd like to buy.

I need a new car, but next year's model might improve upon this year's price vs. fuel economy vs. power features, and I could lose my job and need to keep that money for mortgage payments, food, and medical care. My wife wants a new dishwasher, but they're still figuring out how to make them quieter, and that competition may push the cost lower. Although, inflation could drive prices WAY up so maybe we need to go ahead and buy now. I get another marketing letter from the phone company every week saying I can switch to a different wireless plan and get all sorts of features I'm pretty sure I don't need, but all those features sound cool and I might use them if I had them, but then there's the "better technology tomorrow" and I might lose my job things again. ARRRRRGGGGGGGH!!!!!

At least there's one bright spot in this picture. Almost all TV content SUCKS. The cable company is raising the rates we're paying for the same old stream of amoral, liberal slanted garbage. Broadcasters have just about completed the switch to digital and we happen to live in an area where the signal strength quality is actually pretty good with a respectable antenna. So, we can get the garbage anyway without paying monthly for it. This decision to "un-purchase" our cable service is actually getting easier. I guess, technically, that doesn't count as a purchase decision since, once again, we would move back to the "should we buy" status, but I guess I should be thankful that the basis is clearer for the choice we need to make today.

Aside from when something is clearly not worth buying, like TV and most anything "As Seen on TV," even if the economic outlook and technology / price trends are ignored for a moment, I consistently find that no one manufacturer has chosen a product design that has _the_ feature set I'd choose. I feel like I am forced to select one subset or another of what I want in a product, no matter what the product is. I'm always thinking to myself, "Well, product A has 3 things I want, and product B has 3 other things I want, and they both have at least a dozen things I care nothing about. If only product A or B had its rival's 3 features I want, I'd be ready to lay down the money and take it home." I know that patents and copyrights prevent some of that from ever being possible or practical, and that ultimately it benefits consumers to provide a company a chance to recover the investment in R & D. That doesn't stop me from wishing I could have my cake and eat it too though. Now there's an idea someone should develop and patent, cake that can be eaten, and kept for another day too.

How Do You Decide Whether To Spend?

Hearing our new president go on about how the federal government now has the right to control how private industry operates, and how the federal government is the only entity that can forcibly spend the country back to economic health, makes me wonder how so many people could still be lapping up the reeking, sour kool-aid he's pouring in their bowls. The U.S. federal government, with their notoriously wise decision making capabilities, has used our tax dollars to buy control over those private companies. Today they're also on the brink of spending even more of our tax dollars, and our kids' tax dollars, and their kids' tax dollars to buy more stuff the citizens of this country _should_ have a vested interest in _not_ being publically subsidized, never mind publically owned and run.

So, the only theory about why all that government pork spending would prop up and revive the economy is that if money is flowing, jobs will be created, and the economy will recover. Having worked for a while as a contractor in a federally run organization, I have observed first hand how that ridiculously expensive bandaid will utterly fail to have any positive effect on the economy. The bottom line problem with government spending is that the government is incapable of choosing things on which to spend our tax money that will cause an overall increase in productive activity. And, it's the productive activity that improves the economy, and the standard of living, not the wasteful spending. Two people standing on a beach exchanging a stack of $100 bills all day but doing nothing would represent a _lot_ of spending but it would have _ZERO_ effect on the wealth of either person.

Spending to stimulate an economy depends on a multiplier effect. Entity A spends money with entity B, and they spend money with entity C, and so on. In exchange, B does something useful for A, C does something useful for B, and so on. When the federal government spends money with entity A, it pays entity A a great deal more than what anyone with any sense would pay (not efficient), requires entity A to comply with boatloads of regulatory nonsense (not productive), and ties up entity A's accounts receivable for and absurdly long time (not effective). By the time entity A has a chance to spend anything with entity B, entity B has lost hope that entity A will spend, entity C has abandoned pursuit of spending from entity B, and so on. With the government overhead superimposed on their spending, less and less useful stuff gets done. That means fewer and fewer people can be engaged to do useful stuff, so fewer and fewer people have income which would have allowed them to _be_ entity A, B, or C. The "government" spending multiplier effect is essentially the opposite of what occurs when money is spent efficiently to get useful things done.

If you have a different opinion about how things work, just think about how you decide when and how much to spend. Is it based on what you got last year? Maybe it would be if nothing else had changed, but we've got "change" now whether we like it or not. Is it based on what you expect to happen this year? I suspect that has quite a bit more weight in your decision process. Do you think your prospects of keeping your job and your income _improve_ as more people make the same kind of financial decisions you're making based on what you expect? I suspect not. Finally, do you think it is better to pay 150% of the price for something even if you don't want or need it just to be sure the person offering the good or service continues to have an income, or do you buy only what you really want or need, at the lowest possible price, so your money does as much for you as possible? Unless you're a little slow, I'd venture you'd choose the latter.

If so many people have such high expectations of the newly elected administration, and those people are supposed to have adopted the nirvana of new "hope" in what is to come, why then must the federal government spend us out of the retreating economy? Does nearly 1 trillion dollars of government spending on things the public does not need give you more confidence to go spend your money, or does it throw a giant, spooky, wet blanket on that "hope?" Do you think the prospect of government bureaucratic control over the largest and most troubled players in the economic picture will make things better, or are you willing to face reality and admit that that would surely be a soviet style trade from bad to horrible?

If you voted for BO, you must think all we need is "hope" and "change" so call your senator and explain to them why you want them to vote against this detrimental redistribution of funds towards inefficient and ineffective spending.

If you voted against BO, you most likely knew that what the country really needed was a pragmatic, experienced leader, but since we don't have anything of the sort, call your senator and ask them to please vote against this unwise expansion of the size and control of our federal government.