The Mandelbear's Musings

Early last Sunday afternoon I noticed that the battery-charge indicator had vanished from (main laptop)Sable's Gnome panel. (That's sort of like the row of icons and such you see along the bottom of the screen on a Mac, except that I've configured it to go vertically down the left-hand edge, where it doesn't reduce the hight of my browser window too much.)

Hmm, says I to myself, maybe it will come back after a reboot. So I did that, and logging in presented me with an empty screen background. ??? A little more experimentation showed that only the Gnome-2 desktop was affected; the Ubuntu one (which I detest) worked fine. So did a console terminal, and SSH. The obvious next step was to run fsck, the file-system checker (and many hackers' favorite stand-in for a certain four-letter expletive).

Well, not quite the next step. Since I figured that fixing file-system corruption might possibly make things worse, I moved over to one of my spare laptops, Raven, sat Sable on the shelf next to my desk, and logged in on Sable with SSH. Then I went to the top of my working tree and ran make status to see what needed to be checked in. I think I've mentioned MakeStuff before -- it's basically a multi-function build tool based on GNU Make, and one of the things it can do is find every git repository under the top-level directory, and do things like check its status, or pull. (Commit takes a little more thought, so you don't want to do it indiscriminately.)

Then I ran MakeStuff/scripts/scripts/pull-all on Raven. Done.

Well, almost. There are a few things in my home directory that aren't under my working tree, mostly Desktop, Documents, Downloads, my Firefox bookmarks, and my Gnome Panel configuration. I hauled out a USB stick, fired up tar (like zip, except that it can save everything about a file, not just what DOS knows about). The command I actually used, because I probably forgot a few things (and should have excluded a few more, like Ruby and Perl), was

    rsync -a --exclude vv --exclude ?cache --exclude ?golang . \
          nova:/vv/backups/steve\@sable

And ran straight into the fact that USB sticks are usually formatted with a FAT filesystem, and limit files to 4Gb. Growf! Faced with the unappetizing prospect of shipping 17GB of backups over WiFi, I carried Sable over to my server and plugged in the ethernet cable that I leave hanging off the router for just such occasions. After that finished, I fired up Firefox bookmarked all my tabs, and exported tabs and bookmarks to an HTML file. Should have done that before I backed up everything, but I didn't think of it.

Finally, I was ready to run fsck and find out the bad news. I plugged in the USB stick with the Ubuntu live installer (one does not run fsck on a mounted filesystem!), brought up a terminal, and ran

e2fsck -cfp /dev/sda5 # check for bad blocks, force, preen

(Force means to do a full check even if the disk claims it's okay; "preen" means to make all repairs that can be done without human approval.) Naturally, after turning up a few dozen bad blocks, it told me that I had to run it manually. I could have replaced the -p option with -y, to say "yes" to all requests for approval; instead I left it off and hit Enter a hundred times or so. Almost all the problems were "doubly-claimed blocks", mostly shared between some other file and the swapfile. Of course. Fsck offered to clone those blocks, and I took it up on that offer. Then ran it again to make sure it hadn't missed anything. It hadn't. But it was still broken, no doubt because of all those corrupted files.

So this morning, after a couple of searches, I installed the debsums program, which finds all of the files you've installed, and compares their checksums against the ones in the packages they came from. The following command then takes that list, and re-installs any package containing a file with a bad checksum:

apt-get install --reinstall $(dpkg -S $(debsums -c) \
       | cut -d : -f 1 | sort -u)

Sable now "works" again. I know at one zip file was corrupted (it was a download, and I was able to find it again), and fsck doesn't appear to have kept a log, so broken files will keep turning up for a while. I know there aren't any bad zip files left because there's an option in unzip, -t, that compares checksums, just the way debsums does, so I could loop through all my downloads with:

for f in *.zip; do echo -n $f:\ ; unzip -tqq $f; echo; done

I have two remaining tasks, I think: one is to validate all of my Git working trees (worst case -- just blow them away and re-clone them), and then comes the really hard one: deciding whether I still trust Sable's SSD, or need to get a new one. And if I get a new one, how big? Sable and its 500GB drive were purchased together, used, from eBay, and brand-new 1TB SSDs are pretty cheap right now. So there's that.

Another fine post from The Computer Curmudgeon (also at computer-curmudgeon.com).
Donation buttons in profile.

My trackball is going Republican -- it keeps dashing off to the far right. Where did I go wrong?

Yet another reason, if any were needed, not to like Macs. I pulled out my shiny new Lenovo thinkpad keyboard, which works like a dream. Nice touch, no errors. Trackpoint far enough down so that I don't touch it accidently. Three buttons. I've typed this entire paragraph with no errors due to missed or doubled keys. Perfect.

...but there doesn't seem to be an Apple key. Blerg. At least, if I use it for text, I won't go hitting Apple-W when I want to delete a word and end up deleting the window I was typing into.

There's a Thinkpad in my future, I suspect.

The fileserver froze again last night -- this is getting old. I remain convinced that it's some kind of driver race condition: it's not happening with the near-identical drive on the recording box. It could also be due to lurking disk corruption causing it to hang on a read. In that case, a fsck might fix it. It might not.

In either case, I'm tired of fighting it. I'm dropping the fileserver back to a single IDE drive, and moving the (SATA) mirror to another machine. Hopefully by the time I need it, SATA support will be better.

Meanwhile, I have lots of other work to do, and I don't want to have to deal with random fileserver hangs on top of it all.

10:48 It's even having trouble staying up long enough to copy a partition to the new drive. So here's the plan:

( plans within plans; wheels within wheels. Nothing to see here )

Disconnecting the mirror drive -- the Maxtor, with errors -- had no effect on the main problem, which still appears to be the Seagate sticking its fingers in its ears and going "la, la, la" while the kernel sulks in a corner because nobody will talk to it.

I'm really unclear as to whether this is a problem with the Seagate drive itself, (there's another one, only slightly bigger, over in the recording box that still works fine), with file corruption of some sort (hard reset after a crash could easily have left some lingering problems on the disk), with SATA drives in general (highly unlikely -- they're working fine on my system at work), or with this particular BIOS (also unlikely).

I'm also wondering to myself whether I should drop back to PATA, or get a 500GB SATA drive. I'm leaning toward the latter, if only because I put my shiny new 500GB PATA drive in an external box and the magic smoke went out. Grump, indeed. Smelled like the magic smoke in the box, so the drive is probably still ok. One hopes.

Plus the fact that I have a pretty full schedule today, so I'm not going to have time to do much work on this, no matter how urgent.

 ...

My inclination is to start by reconnecting the mirror drive, fsck it, and see if I can get a clean backup. Then go to Fry's and get one of the Maxtor 500 SATA's they have on sale. Then we'll see.

10:17 fsck done -- errors on both mirror/ and mirror/home, no doubt due to crashes while writing. Backing up now.

Apparently an updated kernel is not enough to keep the fileserver from occasionally freezing. Memory or disk problems? Grumble. Wasn't planning to buy either just yet.

Spent most of the morning plugging my new recording rig together. This mainly involved clearing off the little tray table I'm using as a desk so as to have enough room for the Delta 1010's rackmounted DAC/ADC box, chasing down patch cables for the two preamps and the headphone amp, plugging all the wall warts into the squid, and routing cables back to the computer.

Work was interrupted by the need to go out to OSH (Orchard Supply Hardware) for a little plastic garden shed they had on sale. [That reminds me: now that there are people in the house I have to assemble an unloading detail. It's all assembled (a major factor in deciding which of the two possible sheds to buy), so the box is bulky but not too heavy.]

There was one nasty moment when I flipped the switch on the squid and the 1010's power light didn't come on. I searched all over for a power switch and finally decided to see if it was waiting for the computer to power up. It was. Dorsai booted up (there are still quite a few configuration tweaks that need doing -- I haven't done serious work in Ubuntu before -- but those can wait) and the meters on envy24control lit up when I snapped my fingers in front of the mics. So it'll all work.

And my Lenovo Thinkpad keyboard arrived this evening shortly before the pizza -- comes with a lovely black leather-looking carrying case, too.

And one of the Wednesday crowd (hellloooonurse) brought along his new radio-controlled dalek.

So I'm a happy geek.

Just transferred about 1.3GB of concert data (two Audacity projects) from my MacTel laptop to my fileserver. Took under 5 minutes over gigabit ethernet. Yay for shiny new network hardware. (Had to sit the lappy on top of the laser printer because all I have in Cat6 or Cat5e at the moment are a couple of 1m patch cables.) Split-up concert pieces Real Soon Now.

The Mac has been a distinctly mixed bag. Sure, everything "just works", provided you drink the Apple KoolAid and haven't had over 20 years' worth of Unix experience setting your expectations. And we'll ignore the fact that I had to discharge the battery and leave the thing turned off and charging overnight to convince the power applet that it had anything other than an 83%-charged battery. I expect it'll make a nice Linux machine.

Meanwhile, the departure two weeks ago of our group's summer intern has freed up a lovely Dell widescreen (1920x1200) monitor, which is now sitting on my desk. I wasn't expecting to be able to get one until next fiscal year, having burned up this year's hardware budget on the Mac. It remains to be seen whether my desktop machine will drive it, but I'm hopeful. (Update: just added 1920x1200 to all the mode lists in xorg.conf, and it worked. You wouldn't think 320 pixels would make that much difference, but it does.)

My new little fanless X system appears to be working -- it was up for over a day running a session on the recording box with a copy of audacity running (and with the input meters active, though it wasn't recording). Hopefully a new power supply will fix the mysterious shut-offs without my having to resort to a dummy load; I ordered a PicoPSU from the nice folks at LOGIC Supply.

No recording, though; the washing machine was running, somewhat noisily, so I decided to do dishes instead of taking another stab at "Silk and Steel". Tomorrow. (Yeah, I know. This time for sure...) Between five and seven songs left to redo, and only two that really have to be done before Consonance.