OSX Mavericks Possible Data Corruption Bug

Over the past two weeks there has been much upheaval in my life. Involved with this upheaval has been one of the most unwanted activities any IT professional has to do as part of their professional lives and that is bowing out gracefully. Sometimes IT professionals can actually achieve this state of grace, however most of the time fear overwhelms grace and trust. The morality I will leave to another blog post to come.

In rescuing data from a computing device a few days ago I discovered that the act of using a USB external hard drive with a Macintosh MacBook Pro with OSX Mavericks may have a nasty bug lying in tall grass. I had about 212GB of data that needed to be moved to another medium, and I elected to use a Western Digital external hard drive using USB 2. This drive had never before shown any signs of failure however after copying the data onto the drive using OSX Mavericks, the HFS filesystem on the drive suffered some mystery damage that I’ve never witnessed before. Thankfully the volume was mountable and I could rescue the data from the errant drive and copy it to another drive and effectively save my bacon. The error concerned a failure in the node structure when fsck was asked to diagnose the HFS Journaled filesystem present on the suspect drive. Now I can’t say for sure that OSX Mavericks caused this failure, but the proximity of it and an earlier email from Western Digital stating that there might be drive problems with OSX Mavericks also rang in my mind as a potential problem that points to this particular possible bug. Now the Western Digital warning was just for their drives that used the extended WD software to mount the drives to the Macintosh file system, I suspect that the bug is indeed deeper than even WD knows, or Apple perhaps.

If you are using WD, or perhaps any other external hard drive or memory-stick technology with OSX Mavericks the smart money is on frequent backup and sync to multiple locations. Really smart administrators will backup over the network to some other computing platform with it’s own independent drive technology. If you are using Macintosh OSX Mavericks, I would say it’s better to be safe than sorry and for the love of all that is cute and fuzzy, make your backups!

G-RAID, Time Machine, and Spotlight Headache

A few days ago, of course right before the “Money Back Guarantee” expired on our G-RAID 8TB Time Machine drive at work both my S3 and I were battling with a rather nasty pernicious bug that was plaguing this device on our new fancy Mac Pro Server running OSX Mountain Lion.

The problem was this, you plug the drive in, using Firewire 800 and Time Machine sees it and starts backing up files. That works just fine. After say 1TB of files get backed up Time Machine works gamely for about three or four hours and then the drive suddenly goes deaf. What I mean is that the drive is still connected, the icon is on the Desktop, but you can’t do anything with it. It gives you a fusillade of meaningless errors, vague ones like “Unspecified error with file system” and the like and Time Machine is stuck and can’t do anything at all with the drive. It’s not really a headache for us currently because the server is brand-spanking-new, but still, it’s a concern for us. You have to eject the drive, and not a plain eject either, but a Force Eject. When you move it to another computer and plug it in and do a fsck on the drive everything pans out fine. Everything is hunky-dory, journal is fine, structures are peachy, the works. So annoying.

So off to Google we go. Turns out there MIGHT be a bug in the “Turn off Hard Drives when possible” in the Energy Saver preference pane in System Preferences. This strikes me as a wee bit of bullshit, the drive should go to sleep and wake up elegantly like anything connected to a Mac should (and almost always does!) so, fine, turn that off. Testing. Ah, failed. So next stop was to try to irritate the drive with constant actions. To that end I created a script:

!/bin/bash

while true
do
touch /Volumes/G-RAID/keepalive
sleep 60
done

So what this script does is touch, which is a Unix command in the Mac that just runs out and accesses a file, it’s size is zero, it just runs the most basic of file operation on a drive. If you touch a file on a sleeping drive, it should wake it up. If the drive is counting down until it goes to sleep, this operation will reset that counter. Then the entire thing takes a nap for a minute and does it again, and it does it over and over forever.

We tried that, and still ended up with a failed Time Machine backup and a drive that’s gone deaf. The exact error you get in Time Machine is “com.apple.backupd: Error: (22) setxattr for key:com.apple.backupd.HostUUID … ” So, still no solution to our problems. We finally figured out what the silver bullet was, and it came from an unexpected source. We added the G-RAID drive to the Privacy pane of Spotlight in the System Preferences on the server and voilà! Magical solution!

Since I did that, the drive has been working happily since I made the change, it’s been about a week. My working theory is that mds (which runs the Spotlight service) either locks a file or does something sneaky with this extended attribute on the HostUUID object and that, somehow, ruins access for the entire file system on that drive. It’s not that the file system is damaged, it’s just not working.

So, where’s the bug? Is it in mdsworker, mds itself, backupd (Time Machine), Firewire 800, the Firewire 800 cable, or the G-RAID drive? The answer is a definitive YES. Somewhere. Something is causing it and the only solution seems to keep mds’s muddy hands to itself and pester the drive every minute with a meaningless file operation via touch.

The upside is the damn thing works, so we’ll keep going with it until it stops working. I wish there was something clearer than this Error 22 from backupd to go on, but alas, this seems to be a valid workaround and frankly I don’t really need Spotlight to go futzing about on the drive anyways. There won’t be any searching done on it anyhow, just the indexing that Time Machine needs and that’s it.

I guess in the end, all’s well that ends well.

Installing a HP LaserJet 1505 printer on Apple OSX Mountain Lion

What a problem this was! We had a user with a MacBook Pro that had a new copy of Macintosh OSX Mountain Lion 10.8.2 running on it. Plugged in a rinky-dink HP LaserJet 1505 and nothing. Even though there was the exact same printer installed before, from the user’s home, the system refused to reuse the connection for the printer at work. Obviously that has to be because the system notices it’s a different device and refuses to play along, which I find stupid.

Plug in the printer, try to add it, and the Add Printer function goes out to Apple Software Update to look for the driver and then comes back and tells us that nothing is available. Then commence zombie debugging via muzzle flare, wandering around in the dark trying to fix what shouldn’t be happening but apparently is beyond all logic and reason.

So how you do diagnose a Mac? Here’s a handy-dandy guide which anyone can use to fix their Macs. I seriously doubt any issues ever survive this particular procedure:

  1. Clear PRAM – Turn off computer, turn on computer while holding down  Command-Option-P-R. The computer will restart and you’ll hear the startup chime twice. Let go of the keys. ~ For this, just do it. It doesn’t matter if you don’t think doing this will fix your problem, it will. Just shut your pie hole and do this. If you don’t do it, I don’t want to hear about your problems. It’s magical. I don’t care if Apple says it won’t do anything. This thing DOES EVERYTHING IN CREATION – apparently. That and it cannot hurt. Lots of fluids and plenty of bed-rest. 
  2. Repair Disk Permissions – Start Disk Utility, find your “Macintosh HD” and click “Repair Disk Permissions” and wait. Do this. Often. Regularly. Lots. Weekly. Now.
  3. Download Onyx. Pick which version of OSX you are using, download it, install it and use it. I recommend skipping everything it wants to do and going right for the Automation button. Uncheck “Repair Permissions” and “Display of folders content” and check the rest. Click Execute and wait. When the system asks for a reboot. Reboot. Everyone should do this weekly. Think of it like vitamins for your Mac. Plus, it can’t hurt.

At this point your system should be all spic and span and whatever niggling bit was bothering you should be dealt with. Of course, for the problem I had to deal with at work, there is one little thing extra, one thing more. Open Finder, click Go on the Menubar, then Go to Folder… and type in /Library/Printers and click Ok. You’ll see a list of folders. In this list find the folder named “hp” and KILL IT WITH EXTREME PREJUDICE. Y’arr! This !@#$ folder is at the very center of my hatred for all that is Hewlett-Packard. I’ve started to unceremoniously refer to them as Fudge Packard. Bastards. Anyways, killing the folder does the trick, it clears everything up and Mountain Lion can download software from Apple again for the HP Drivers – blah blah blah. I’d rather just get a sledgehammer and pound the HP LaserJet 1505 into foil, but hey, you have to cope or have some sort of attack. I regret buying HP. I regret the LaserJet 1505. What a piece of crap. Steaming.

First Look at Mountain Lion OSX for Macintosh

I purchased and downloaded the newest version of Macintosh OSX codenamed Mountain Lion. The download took a brief amount of time and once established I didn’t have a problem handling it. The first step was creating an independent system installer using a USB memory stick. I found some instructions that I remembered from when I did this with OSX Lion and the instructions worked well, up to a point. I was able to find the InstallESD.dmg file and I set up my 16GB memory stick with the proper format settings, specifically Mac HFS File System with Journaling and GUID partition map. The first issue I ran into was a strange memory error, that while restoring the dmg file to the USB memory stick, after the Mac was done really, in the verification step it failed with this odd arcane “cannot allocate memory” error. I went immediately to Google to look and found that if I mount the InstallESD.dmg file first, that *that* is the magic bullet. Turns out, it was.

Now that I have Mountain Lion on a USB memory stick I got a stock 24” iMac out of storage and set it up. Plugged the USB memory stick in, then the mouse and keyboard, main power, and while holding down the option key, turned it on. Everything worked as I expected it to! So far so good.

Once the system was up and running and in setup it prompted me to connect to a Wifi system, which was not a problem since I share Wifi from my primary work iMac (long story for another day) and it seemed satisfied. Then I ran into my first problem with Mountain Lion. During initial system setup I could not successfully log into any Apple ID. My personal one, or the one for work, either one didn’t work. The system allows you to continue without it and so that’s exactly what I did. Once I moved on to setting the time zone, this also failed, but I suspect it has everything to do with my shared Wifi coming from my Snow Leopard iMac and not something endemic to Mountain Lion. Instead of Mountain Lion successfully setting the time zone by it’s location I set it by hand. Not really a problem.

Once I got the system up and running, idle at the desktop everything was as it should be. My next step was to try to connect my test iMac up to my Apple ID. So logically I went first for System Preferences, then to Accounts, and there set my Apple ID. I was half hoping that setting it there would have had a chain reaction and set it everywhere else, but that didn’t happen. I noticed that iCloud wasn’t set up properly, so I found it in System Preferences, it wasn’t a problem, just a very weak annoyance. Then I tried the Mac App Store, had to do it again, same for iTunes. The only real irk that upset me was fiddling around with “Back To My Mac” feature which asked me to turn on sharing with a button that lead to the sharing panel. I was lost in there (no, not really, but I was in the headspace of an end-user) and it took me a while to notice that Apple did tell you where to go to set things up, so my one tweet about this being a problem is wrong, I was just hasty. I must say that much of this I will pin on me being in the “end user headspace” and not as an Admin, which I would have been much more careful and slow with in my approach to Mountain Lion. If you read and aren’t hasty, this isn’t a problem.

Every app that I’ve used worked well, some needed Java to be installed but the OS prompted to fetch it and install it for me without a problem so that was fine. Of the apps that work that I’ve tested, at least in that they open up are:

* Aqua Data Studio 11.0
* Dropbox
* iSquint
* KompoZer
* MarsEdit
* Miro Video Converter
* MPlayerX
* Music Manager (Google Cloud)
* OpenOffice.org
* Photo Wrangler 2.1
* Picasa (needed update)
* Postbox
* Seashore
* Spotify (needed update)
* The Unarchiver
* Transmission
* VLC
* What’s Keeping Me?
* XTabulator
* Zipeg

Of course, all the apps from the Mac App Store I assume work well. Dropbox was a non-issue, 1Password was smooth-as-glass, as I expected. But what really surprised me was Postbox. I recently fled Sparrow as an email client when they announced that Google was acquiring them. Postbox was my alternative. When I copied over Postbox and started it for the first time it offered to collect the settings form Mail.app which I didn’t think anything of and let it go ahead. Postbox seamlessly captured my iCloud email account and after I typed in my Apple ID password, I was up and running! For some strange reason, that really pleased me.

So, what is next? So far everything seems to test fine in Mountain Lion. There are some goobers from Lion that I still need to work out – such as secondary monitors in full screen mode being stupid, that sort of thing, and also to see if VirtualBox will work, but for the most part I’m satisfied that this new OS is exactly as Apple bills it, and they have done a very good job. There are some small irky bits and on my Twitter I’m sure it came across as being ranting-and-raving, but actually it’s quite good.

Next steps at work are tallying up all the people interested in Mountain Lion and figuring out how we’re to pay Apple for the licenses, then helping everyone set up Apple ID’s on their own. There is going to be a headache with all these new very independent and unmanaged Apple ID’s floating around in space, but if you want the Bright and Shiny you have to swallow a seed or two.