Replacing a failed hard drive

Posted on May 6, 2011

37


Shattered Drive Platter

Did you discover your Synology beeping at you?  Did you get a SMS message on your phone or an email saying “Volume 1 on NASname is in degraded mode [3/4]”?

Well then, you my friend are in need of this post…

Dealing with a failed hard drive…

Take a look at your NAS.  The Status light should be blinking yellow and the failed hard drive number will be indicated on the HDD lights.  Log into your NAS verify if Synology is reporting that a drive is degraded, defective or crashed.  You will see several notifications in the upper right corner, one of them will be labelled, “System beep reminder. Volume 1: Degraded.”

First things first.  Let’s s stop the incessant beeping of your Synology by going to Main Menu (drop down arrow in upper left corner) > Storage manager > Volume, and clicking on the “Beep off” button.  You should also see more details about the status of your hard drives.

Stop the beeping by clicking "Beep Off" DSM 3.1

If for some reason you don’t have any red text like in the screenshot shown above you should check your log file by going to: Main Menu > System Information and selecting the last tab labelled “Log” to see what issue is being reported.

System Log - Synology DSM 3.1

Look for red X’s that will have event messages that look something like: “Volume [1] was degraded, please repair it.”, “Write error at md2 disk2 sector 12345678”, “Internal disk [1,2,3 or 4] was defective.” or “System volume [x] was degraded, current disk status [3/4], please repair it.”

If the data is important and you haven’t backed it up, please do so.  Your NAS is currently running with no safety net if another drive fails (unless you are using SHR with 2 drive redundancy).  Please backup your data before proceeding.  If you don’t have enough storage space to backup your NAS you may want to look at buying more hard drives ($60-$90 per TB for life of drive) or using Amazon S3 backup ($143 per TB per month adds up).  You can find details on how to setup Synology to backup to Amazon S3 here.

Choosing the hard drives…

You can see my previous post on shopping for a new hard drive here.  If you planned ahead you’ll have a spare drive of the same make and model as the other drives in the NAS.  This will help ensure that you do not run into any problems rebuilding the new drive.  If you are buying a new drive try to stick with the same manufacturer (yes, I know you want to swear off that manufacturer, but for the sake of the other 3 drives you are better off sticking with the same manufacturer).

In order of preference:

  • same exact model drive (model – same TB size, rpm speed, etc)
  • same manufacturer with the same or larger capacity and RPM speed (you can go same size or larger, but never smaller) that is on the approved hard drive list (now is not the time to test an uncertified drive)
If you chose Synology Hybrid with data protection you can swap out the bad drive for a good one, and if the new drive is larger your NAS will dynamically resize to take advantage of the space.  If you are using a classic RAID style format you will not be able to take advantage of the extra space on your current volume setup.

Swapping out drives:

Before you shut down the NAS to swap out drives.  Verify which Drive is bad.  Better to double check now then to open it up and replace the wrong drive.  Make sure you note which HDD light was blinking on the front of the NAS and make sure it corresponds to the drive number that Storage Manager is reporting.  The lights on the DS411j are aligned with the drive order stacking inside (drive 1 on top, drive 4 on bottom).

Here are the steps:
  1. Make sure you noted which drive number was bad and make sure you match it to the drive numbers inside.  On stacked drives numbering starts at the top.  HDD lights on front should match drives inside.
  2. Make sure NAS is turned off, cooled down and unplugged from the wall.

Synology DS411j Drive Numbering and screws holding drive tray

Setting up the replacement drive:

Once you’ve finished the drive swap and reconnected and powered everything back on you’ll notice the NAS will start beeping again (default is on in beep control).  Go back to Storage manager and turn off the beep.  Main Menu >Storage Manager > Volume, and click beep off.
To rebuild the new drive to be a working member of your NAS choose Manage (Main Menu > Storage Manager > Volume > Manage).  You should be presented with the Volume Manager Wizard as shown below.  Choose Repair and click next.

Volume Manager Wizard

You will be prompted to choose the disk.  There should only be one choice.  Never try to upgrade your NAS by repairing more than one hard drive at a time (if you want to upgrade the other hard drives you need to add and repair/rebuild them one at a time, otherwise you’ll lose data).  You will be prompted with a warning that says all the data on the new drive will be erased.
You will then be prompted to verify your settings.  Clicking ok will initiate the process you see below.

3TB drive is only 12% "repaired" after 5 hours!

 The bad news is that you can expect the repair process to take a LONG time (20 hours in my case).  The drive is formatted EXT4, checked for any inconsistencies, and then one third of the data from the other 3 drives is rebuilt onto the 4th drive (1/2 the data if you are on a 2 drive Synology with data protection).
The good news is that Synology DSM can keep the NAS accessible for use while doing the hard disk rebuild.  You should get a notification once the process is completed.
You should then run the S.M.A.R.T. tests to verify the drives are fine.  Main Menu > Storage Manager > HDD Management Tab, S.M.A.R.T. Test sub tab.  Once completed with no errors, you my friend are back in business!

Additional Info:

You can take the failed hard drive and see if it is recoverable by using the drive diagnostic tools that can be found on the drive manufacturer’s website (often DOS like programs that require the drive to be connected to a PC).
You can use Apple’s Disk Utility to reformat the drive and see if it displays any error messages.  I would however be reluctant to use a drive that had reported any kind of errors as a reliable drive.  If the drive is under warranty you can send it back to the manufacturer for replacement.  Most manufacturers will honor the warranty period based on the manufacturer’s date on the sticker of the drive without need of a receipt.
If you had data on that drive that wasn’t backed up you can try to mount it on your Mac or PC running Ubuntu which can read EXT 4 formatted hard drives and allow you to copy the data off.  If your drive or folders were encrypted you’ll need to copy your encryption key as well in order to unlock the data.
Posted in: Hard Drives, Setup, Synology