good practices: protect/manage your linux server for/over 366 days

After 366 days of uptime, my dear Debian server sent me an email via SMARTd and you can imagine my joy when I read the title ("back up data NOW", haha).

Luckily, I learned something 366 days ago (when I just reinstalled on another drive, no real data to worry about): 

• use hddtemp to monitor the HDDs temperature (i.e. via syslog); if this goes up, your drives will start singing different tunes and none is pleasing to your ears/data :-)
• install SMART monitoring tools
• employ mirroring / redundancy via RAID 1, 5, or better
  

Luckily (again), I got a spare drive some days ago and already started the magnificent mirroring process on a running server (many thanks Falko!). Unfortunately (finally! :-), I didn't complete it then, since it required downtime and I wasn't that eager to reboot. Now, with the main drive failing I had to reconsider...

So, I just went on shutting down the services, bring the system to its knees (almost) and issue the last rsync (yeps, Falko is teaching you on how to do it on a working/ideal environment). Yes, there's a chance to ruin your good copy trying to rsync from a zombie device, but hey, I just let the blonde moments taking over... and they didn't dissapoint (at this point you should stop thinking about "good practices" and just harbor a smiley :-)

What Falko doesn't tell you, is that you should stop your services before taking a hot copy (via cp, see close to bottom on the second page), or else, you'll start fixing your services (like DBs, etc) when the RAID is ready.

Rsyncing went fine, but devil's tail came into play: I got this funny idea that I should start the services again (keeping uptime, heh?). Do NOT DO IT on a failing drive! In fact you should concentrate ONLY on the important data, back that up and once that's safe try to recover more. Sometimes you're lucky, some other times you're not and I was stuck with the latter; I couldn't do *it than reboot. To my surprise, the system came up and I had a new chance to fix it by following Falko's steps.

Remember, once the new/RAID drive is ready make sure you install the bootloader on it, or else, the reboot will teach you that. What else is the reboot teaching you? Make sure you add both (hd0,0) and (hd1,0) as root for your RAID stanzas! If the failing drive (i.e. hd0) dies for good, your BIOS will see your second drive as (hd0) and the grub config is now a bit off...

I still have one reboot underway (lets call it "maintenance break", shall we? :-), but with these new lessons learned I hope I can safely close this chapter... Happy rsyncing/mirroring to everyone!

the VIA iceman reviewed

A few months ago I won (via CrunchGear) a nice piece of hardware that now replaces the veteran hdd divx player I bought from Hong Kong about the same time last year. The enclosure was much more reliable than the other "made in china" (sic) stuff I bought back then, but somehow its remote died this summer and it's not that fun to pause/play/stop using the built-in buttons. It works, but one has to leave the couch in order to do it :-)

Introducing the iceman: VIA's Weather Resistant PC System, fanless and with MPEG-2 hardware acceleration. Yes, sounds like your ideal outdoor computer (or in-car?), but it fares pretty well in the home also. After unpacking, you'll see:

- main unit (includes 2.5" HDD PATA), all black

- power cord + interesting/round power socket

- CD with drivers for Windows (no Linux, Mac :-(

- 4 page booklet (outdated/useless)

- wireless module (yeps, I bought this one :-)

- screws, mounting frame, wireless antenna

Now, your engineering skill kicks in: mounting the wireless module. You need to open the (black) box, find a place for the module, then, screw it... yeah, sort of. VIA didn't provide me the small screws needed to attach the wireless module to the main chasis, therefore for aesthetic reasons :-) I decided to pack everything back and use it sans wireless (sits close to the router anyway, no biggie)

Still, in the process, I snapped a few photos:

(on the back of the unit)

- 10/100 NIC (argh, where is the Gigabit?)

- 4x USB 2.0 (quite close one to each other)

- TV-out (S-Video & RCA) + VGA

- serial (ha?) and 2x PS/2

- audio (Line-out, Line-in, Mic-in)

- on/green led

- red power button

Removing the back, the front screws, lifting the top, looking at the wireless module...

The next shock wave: how can I upgrade the RAM in this unit? Err, I took a look inside, applied some pressure, took another look inside, applied a bit more pressure then, an Angel whispered: be careful, you might break it! So I paused for a minute and then it hit me: the CPU's sink is glued to the chassis!

Well, that's quite ok if you want to better dissipate the heat... but it gives you more headache (or chances to FUBAR it) if you want to change some parts inside :-( For me it was now the U-turn point: wrap it up, power it up and start...

Searching for an USB floppy (what?) or an USB CD-ROM since iceman's HDD was virgin. I gave up quite quickly on installing Linux (no real Bluetooth support) and followed this nice guide on installing Windows XP SP2 using an USB memory stick. All went fine and I got my iceman with XP SP2 booting... from power up to complete log in... in something like... 5-10 seconds! WTF? :-)

Upgrading to XP SP3 solves this "problem", back to the 20-30s booting times :-|

Overall,

The Good:

• runs cool/just a bit warm after a few days on
• completely silent
• very good booting times
• can scale up DVD video to 720p on a wide screen
• plenty of options in the BIOS

The Bad:

• no mounting (mini) screws for the wireless module
• too slow for anything more than DVD quality (argh, where the hell is that MPEG-2 hardware acceleration?)
• no Gbit NIC
• the USB ports are too close one to each other

The Ugly:

• very hard to upgrade the RAM (if needed)

Missing from this review: the wireless keyboard I'm using with this system, the USB Bluetooth adapter that helps me send wirelessly the sound to my mini HIFI system (now you see why I had to ditch Linux? :-)