Posted on 19 September 2008 @ 08:13

Hi Guys,

I have a huge problem as you probably gathered from the title. Last week I had a perfectly running system which included 6x
500GB WD sata drives in a geom::graid5 array with geli managing the encryption layer. 2x sata controllers that are the same as when system was working 100% so I assume there should be no conflict now.

In doing some troubleshooting I decided one of my drives was toast (freezing during re-syncs and UDMA xfer errors) and replaced it with a new one. After it was done resynchronizing I went to attach the encryption layer and here is where the trouble started. I get an error message about missing metadata.

NAS:~# geli attach /dev/raid5/storage
Cannot read metadata from /dev/raid5/storage: Invalid argument.

In troubleshooting this problem I have learned about the geli dump command, but it's probably too late now.

Here are some stats about the geom::graid5 array if it's helpful.

NAS:~# graid5 list
Geom name: storage
State: COMPLETE CALM
Status: Total=6, Online=6
Type: AUTOMATIC
Pending: (wqp 0 // 0)
Stripesize: 131072
MemUse: 0 (msl 0)
Newest: -1
ID: 68917578
Providers:
1. Name: raid5/storage
Mediasize: 2500539187200 (2.3T)
Sectorsize: 512
Mode: r0w0e0
Consumers:
1. Name: ad20
Mediasize: 500107862016 (466G)
Sectorsize: 512
Mode: r1w1e1
DiskNo: 4
Error: No
2. Name: ad18
Mediasize: 500107862016 (466G)
Sectorsize: 512
Mode: r1w1e1
DiskNo: 1
Error: No
3. Name: ad16
Mediasize: 500107862016 (466G)
Sectorsize: 512
Mode: r1w1e1
DiskNo: 0
Error: No
4. Name: ad14
Mediasize: 500107862016 (466G)
Sectorsize: 512
Mode: r1w1e1
DiskNo: 5
Error: No
5. Name: ad12
Mediasize: 500107862016 (466G)
Sectorsize: 512
Mode: r1w1e1
DiskNo: 3
Error: No
6. Name: ad10
Mediasize: 500107862016 (466G)
Sectorsize: 512
Mode: r1w1e1
DiskNo: 2
Error: No


freenas:~# graid5 dump ad10
Metadata on ad10:
Magic string: GEOM::RAID5
Metadata version: 2
Device name: storage
Device ID: 68917578
Disk number: 2
Total number of disks: 6
Provider Size: 500107862016
Verified: -1
State: 0
Stripe size: 131072
Newest: 4294967295
NoHot: No
Hardcoded provider:

## graid5 dump adXX output looks exactly the same besides the 'Disk
number:' output for all devices.

Posted on 4 October 2008 @ 03:23

edited 03:24

Hi Enlightenment!

Sorry for the late reply, but thanks for the assistance. I have not issued the command geli clear ...

NAS:~# ls -l /dev/raid5/*
crw-r----- 1 root operator 0, 87 Oct 4 01:12 /dev/raid5/storage

NAS:~# kldstat
Id Refs Address Size Name
1 4 0xc0400000 818d78 kernel
2 1 0xc5619000 5c798 acpi.ko
3 1 0xc994e000 13000 geom_raid5.ko

NAS:~# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/md0 73M 64M 9.1M 87% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/ad0a 35M 30M 4.7M 86% /cf
/dev/md1 31M 14K 28M 0% /var


NAS:~# uname -a
FreeBSD NAS.local 6.4-PRERELEASE FreeBSD 6.4-PRERELEASE #0: Sun Sep 14 02:55:29 UTC 2008

Posted on 6 October 2008 @ 15:55

I've also heard somewhere that there is a problem with Raid5 if you swap the data cables so that one drive (ex. ad2) now becomes another (ex ad4). I'm not sure I believe that as the array manager stores the drive number as shown above.