I’ve moaned about the state of the SGI cluster here at WeSC before. I’m now happy to report that it’s almost back to full fitness.

After playing about with the external CD-ROM a bit more it became apparent that it would only behave itself if you followed a careful procedure of powering down the Origin it was attached to; power-cycling the CD-ROM and then powering the Origin back on.

At this point the internal disk of the dead node wasn’t even showing up on the SCSI bus so it was definitely knackered. Fortunately the UK is home to a fine purveyor of second-hand SGI parts. We ordered a replacement drive (complete with SGI firmware) and it arrived the next day. Ian Mapleson (for it is he that runs the SGI depot) also has written some excellent articles on Irix administration, one of which details an easy way to clone a root disk. With this info I was able to clone one of the other nodes in the cluster. A quick edit of /etc/sys_id so that it won’t wake up thinking it’s the wrong machine and we are ready to go.

The drive goes into the dead machine, we power it up and hey presto! one working Irix box.

I am jubilant until I realise that all the nodes of the cluster share the same CXFS volume and that this node no longer has a valid CXFS license. And of course we have no backups (actually this isn’t quite true it later transpired that there were copies of the license file on another machine but a backup that isn’t documented isn’t a very useful backup).

I put in a support call to SGI (remembering that this machine has no maintenance contract) without much hope. The very next day SGI email me the license file! SGI, you may be at death’s door but you are lovely people.

At this point all that’s left to do is to debug the condor install which doesn’t seem to be working properly/ But that is somebody else’s problem.

The SGI cluster here at WeSC is beginning to get me down. One of it’s nodes has been down since I started work here and I’ve finally gotten around to looking at it.

Step 1 was to try and get acces to the serial console. After hunting around for a cable I then had to fight with minicom to get it running. A process that would have been significantly easier if the terminals weren’t all runnning at a non-standard baud. Anyway a quick re-boot of the machine showed that it was finding it’s internal disk but failing to find it’s OS. Given the number of times the power has failed recently I wouldn’t be at all surprised if the partition table is corrupted. So it seemed like a re-install was worth trying before getting a replacement disk.

After finding a set of Irix install instructions that I could actually understand I hooked up the ancient external SCSI CD drive and put in the disk that contains the install tools. It took a couple of attempts to convince the drive that it should close but after that it made all the right kinds of whirring noises and I was quietly hopeful.

So boot to the command monitor and:

boot -f cdrom(1,1,7)sash64

And the monitor helpfully responds with a ‘no media found’ message. After trying several other CDs I realized that I couldn’t even ls them never mind run them. The conclusion? Knackered CD drive. Arse.

For my next trick: installing over bootp using an SGI Fuel workstation as a server.

Now that lack-of-sleep madness has passed I’ve managed to actually get some work done. In particular I’ve been chipping away at some of the tedious manual labour that comes with administering multiple machines.

To start off with I finally knuckled down to working out how to use ssh-agent. This nice article from SecurityFocus helped me get started. The most difficult bit was getting ssh-agent to run from fluxbox on start up. To fix that I added the following lines into .fluxbox/apps

[startup] {eval `ssh-agent -s`}
[startup] {ssh-add < /dev/null}

which pops up a dialog box for my passphrase on login.

I also started to get down to sorting out configuration management using cfengine. One of things that I've never been able to work out was how to make rules depend on one another. So if you have a rule that adds a line into the iptables config how do you then tell cfengine that iptables needs to be restarted. After hunting around on the web I found an example that does almost exactly what I need. A hacked up a quick example that would sort my root alias and then run the sendmail newaliases command.

editfiles::
{ /etc/aliases
BeginGroupIfNoSuchLine "root:           wescroot@wesc.ac.uk"
DeleteLinesStarting "#root"
Append "root:           wescroot@wesc.ac.uk"
EndGroup
DefineClasses "aliaseschanged"
}

shellcommands:
aliaseschanged::
'/usr/bin/newaliases'
useshell=false

Basically aliaseschanged is only set if the editfiles rule needs to be executed. So newaliases is only run if we actually update the aliases file. I have a more complicated set of rules that does the same thing for iptables. Next week globus4.