The lateness of this post is likely a little misleading. We finally got the server problems sorted out last week, and I took a 5 day mini-vacation in Montreal (actually 4 days, and I kind of screwed around on Tuesday). Apparently my MySQL-foo is pretty spot-on, and I was able to restore 10 jigabytes of data to to a backup server from our backup server. We're now rebuilding the failed DB server, which looks more and more as if its LSI Logic RAID card simply reset at some point, causing a massive RAID failure. Luckily, we do nightly backups.
However, the version of mysqldump had a bug in it, whereby it exported the system database, information_schema, along with our own db. This system database can't be imported, so the whole restore process kept failing. The only solution was to edit the db backup (which was, as I said, 10 jigabytes, making it impossible to open in a conventional editor), so we used a line editor to remove the first x lines of the backup. This worked, but it then took about 8-10 hours to re-import the data and only after upgrading the MySQL server and other required packages.
So, the server was offline for a total of 3 business days, and two of those days were spent trying to recover the server without destroying the data onboard. Turns out our time was better spent finding a new (if temporary) home for the data.
Montreal was fun.
; "The Blogger")
;)
;)
;)
;)
; "A honeycomb")
Leave a comment