Diggin' technology every day


81,000 RAID arrays

I keep forgetting to post about this, I find this number interesting myself. It is the number of mini RAID arrays on my storage system, which has 200 spindles, which comes out to about 400 RAID arrays per disk! Why so many? Well it allows for maximum distribution of storage space and I/O across the system as well as massively parallel RAID rebuilds as every disk in the system will participate when a disk fails, which leads to faster rebuild times and much better service times during rebuild.

While 3PAR talks a lot about their mini RAID arrays(composed of virtual 256MB disks) it turns out there really isn't an easy way to query how many there are, I suppose because they expect it to be so abstracted that you should not care. But I like to poke around if you haven't noticed already!

The little script to determine this number is:


export ARRAY="mrt"
echo "showld -d" | ssh $ARRAY | grep cage | while read line;
        export RAWSIZE=`echo $line | awk '{print $7}'`;
        export LD_NAME=`echo $line | awk '{print $2}'`;
        export SETSIZE=`echo $line | awk '{print $10}'`;
        export ARRAYS=`echo "${RAWSIZE}/256/${SETSIZE}" | bc`;
        export ALL_ARRAYS=`echo "${ARRAYS_TOTAL}+${ARRAYS}" | bc `;
        export ARRAYS_TOTAL="$ALL_ARRAYS"; echo "NAME:${LD_NAME} Raw Size:${RAWSIZE}  Set Size:${SETSIZE} Micro arrays in LD:${ARRAYS}  Total Micro arrays so far:${ALL_ARRAYS}";

Hopefully my math is right..

Output looks like:

NAME:log2.0 Raw Size:40960  Set Size:2 Micro arrays in LD:80  Total Micro arrays so far:80
NAME:log3.0 Raw Size:40960  Set Size:2 Micro arrays in LD:80  Total Micro arrays so far:160
NAME:pdsld0.0 Raw Size:49152  Set Size:2 Micro arrays in LD:96  Total Micro arrays so far:256
NAME:tp-7-sd-0.242 Raw Size:19968  Set Size:6 Micro arrays in LD:13  Total Micro arrays so far:81351
NAME:tp-7-sd-0.243 Raw Size:19968  Set Size:6 Micro arrays in LD:13  Total Micro arrays so far:81364

Like the mini RAID arrays the logical disks (the command above is showld or show logical disks) are created/maintained/deleted automatically by the system, another layer of abstraction that you really never need to concern yourself with.

The exact number is 81,364 which is up from about 79,000 in June of this year. To me at least it's a pretty amazing number when I think about it, 80,000+ little arrays working in parallel, how does the system keep track of it all?

3PAR isn't unique in this technology though I think maybe they were first. I believe Compellent has something similar, and Xiotech constructs RAID arrays at the disk head level, which I find very interesting, I didn't know it was possible to "target" areas of the disk as specifically as the head. I think of these three implimentations though the 3PAR one is the most advanced because it's implimented in hardware(Compellent is software), and it's much more granular(400 per disk in this example, Xiotech would have up to 8 per disk I think).

The disks are not full yet either, they are running at about ~87% of capacity, so maybe room for another 10,000 RAID arrays on them or something..

I learned pretty quick there's a lot more to storage than just the number/type of disks..

(Filed under virtualization as well since this is a virtualized storage post)

TechOps Guy: Nate

Comments (3) Trackbacks (1)
  1. Hi just wanted to give you a quick heads up and let you know a few of the
    images aren’t loading properly. I’m not sure why but I think its a
    linking issue. I’ve tried it in two different internet browsers and both show the same results.

  2. I found my array has 3154 mini arrays. 144 disks but we are not even 50% on those. Any idea if this accounts for AO data that are created from moving “hot” chunklets between tiers?

  3. yes any chunklets that are provisioned for any purpose will be included. AO happens at a (much) higher level than these mini RAID arrays.

    thanks for reading, and thanks for the comment!!