List Exadata Storage Cell disk summary with cellpd.sql and cellpdx.sql scripts
Posted May 24, 2013In the previous post I explained how to list Exadata disk layout and topology details with the exadisktopo scripts, in this post I’ll introduce one celldisk overview script, which I use to quickly see the celldisk configuration, specs and error statuses. The cellpd.sql script (Cell Physical Disk) will show the following output:
SQL> @exadata/cellpd.sql Show Exadata cell versions from V$CELL_CONFIG.... DISKTYPE CELLNAME STATUS TOTAL_GB AVG_GB NUM_DISKS PREDFAIL POORPERF WTCACHEPROB PEERFAIL CRITICAL -------------------- -------------------- -------------------- ---------- ---------- ---------- ---------- ---------- ----------- ---------- ---------- FlashDisk 192.168.12.3 normal 183 23 8 FlashDisk 192.168.12.3 not present 183 23 8 #ff0000;">3 FlashDisk 192.168.12.4 normal 366 23 16 FlashDisk 192.168.12.5 normal 366 23 16 HardDisk 192.168.12.3 normal 22352 1863 12 HardDisk 192.168.12.4 normal 22352 1863 12 HardDisk 192.168.12.5 normal 22352 1863 12
Above you see that some of my Flash Disks are missing and before they went completely missing (because the Oracle Ninja removed them :) these disks indicated the “poor performance” status.
The cellpdx.sql script will display extended info for each celldisk, including disk model, firmware version, serial number and various error metrics. Warning, lots of wide output coming again:
SQL> @exadata/cellpdx.sql Show Exadata cell versions from V$CELL_CONFIG.... CELLNAME DISKNAME DISKTYPE LUNS MAKEMODEL PHYSICALFIRMWARE PHYSICALINSERTTIME PHYSICALSERIAL PHYSICALSIZE SECTORREMAPCOUNT SLOTNUMBER STATUS ID KEY_500 PREDFAILSTATUS POORPERFSTATUS WTCACHINGSTATUS PEERFAILSTATUS CRITICALSTATUS ERRCMDTIMEOUTCOUNT ERRHARDREADCOUNT ERRHARDWRITECOUNT ERRMEDIACOUNT ERROTHERCOUNT ERRSEEKCOUNT -------------------- -------------------- -------------------- -------------------- ---------------------------------------- -------------------- ------------------------------ -------------------- -------------------- -------------------- ------------------------------ -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- 192.168.12.3 35:0 HardDisk 0_0 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2013-02-07T10:50:29-06:00 JK11D1YAJTXVMZ 2000011650924 0 normal JK11D1YAJTXVMZ DMG_SYSTEM_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.3 35:1 HardDisk 0_1 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:10:46-05:00 JK11D1YAJB4V0Z 2000011650924 1 normal JK11D1YAJB4V0Z DMG_SYSTEM_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.3 35:10 HardDisk 0_10 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2013-02-07T10:52:02-06:00 JK11D1YAJM5X9Z 2000011650924 10 normal JK11D1YAJM5X9Z DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.3 35:11 HardDisk 0_11 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:10:59-05:00 JK11D1YAJAZNKZ 2000011650924 11 normal JK11D1YAJAZNKZ DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.3 35:2 HardDisk 0_2 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:10:47-05:00 JK11D1YAJAZMMZ 2000011650924 2 normal JK11D1YAJAZMMZ DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.3 35:3 HardDisk 0_3 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:10:49-05:00 JK11D1YAJ7JX2Z 2000011650924 3 normal JK11D1YAJ7JX2Z DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.3 35:4 HardDisk 0_4 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2012-03-01T15:31:04-06:00 JK11D1YAJB3J1Z 2000011650924 4 normal JK11D1YAJB3J1Z DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.3 35:5 HardDisk 0_5 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:10:51-05:00 JK11D1YAJB4J8Z 2000011650924 5 normal JK11D1YAJB4J8Z DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.3 35:6 HardDisk 0_6 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:10:52-05:00 JK11D1YAJ7JXGZ 2000011650924 6 normal JK11D1YAJ7JXGZ DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.3 35:7 HardDisk 0_7 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:10:54-05:00 JK11D1YAJB4E5Z 2000011650924 7 normal JK11D1YAJB4E5Z DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.3 35:8 HardDisk 0_8 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:10:55-05:00 JK11D1YAJ8TY3Z 2000011650924 8 normal JK11D1YAJ8TY3Z DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.3 35:9 HardDisk 0_9 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:10:56-05:00 JK11D1YAJ8TXKZ 2000011650924 9 normal JK11D1YAJ8TXKZ DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.3 FLASH_1_0 FlashDisk Sun Flash Accelerator F20 PCIe Card D20Y 2011-05-06T12:00:49-05:00 1014M02JC3 24575868928 0 "PCI Slot: 1; FDOM: 0" not present 1014M02JC3 DMG_UNKNOWN FALSE TRUE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.3 FLASH_1_1 FlashDisk Sun Flash Accelerator F20 PCIe Card D20Y 2011-05-06T12:00:49-05:00 1014M02JYG 24575868928 0 "PCI Slot: 1; FDOM: 1" not present 1014M02JYG DMG_UNKNOWN FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.3 FLASH_1_2 FlashDisk Sun Flash Accelerator F20 PCIe Card D20Y 2011-05-06T12:00:49-05:00 1014M02JV9 24575868928 0 "PCI Slot: 1; FDOM: 2" not present 1014M02JV9 DMG_UNKNOWN FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.3 FLASH_1_3 FlashDisk Sun Flash Accelerator F20 PCIe Card D20Y 2011-05-06T12:00:49-05:00 1014M02J93 24575868928 0 "PCI Slot: 1; FDOM: 3" not present 1014M02J93 DMG_UNKNOWN FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.3 FLASH_2_0 FlashDisk Sun Flash Accelerator F20 PCIe Card D20Y 2011-05-06T12:00:49-05:00 1014M02JFK 24575868928 0 "PCI Slot: 2; FDOM: 0" not present 1014M02JFK DMG_UNKNOWN FALSE TRUE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.3 FLASH_2_1 FlashDisk Sun Flash Accelerator F20 PCIe Card D20Y 2011-05-06T12:00:49-05:00 1014M02JFL 24575868928 0 "PCI Slot: 2; FDOM: 1" not present 1014M02JFL DMG_UNKNOWN FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.3 FLASH_2_2 FlashDisk Sun Flash Accelerator F20 PCIe Card D20Y 2011-05-06T12:00:49-05:00 1014M02JF7 24575868928 0 "PCI Slot: 2; FDOM: 2" not present 1014M02JF7 DMG_UNKNOWN FALSE TRUE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.3 FLASH_2_3 FlashDisk Sun Flash Accelerator F20 PCIe Card D20Y 2011-05-06T12:00:49-05:00 1014M02JF8 24575868928 0 "PCI Slot: 2; FDOM: 3" not present 1014M02JF8 DMG_UNKNOWN FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.3 FLASH_4_0 FlashDisk 4_0 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T12:00:49-05:00 1014M02HP5 24575868928 0 "PCI Slot: 4; FDOM: 0" normal 1014M02HP5 DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.3 FLASH_4_1 FlashDisk 4_1 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T12:00:49-05:00 1014M02HNN 24575868928 0 "PCI Slot: 4; FDOM: 1" normal 1014M02HNN DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.3 FLASH_4_2 FlashDisk 4_2 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T12:00:49-05:00 1014M02HP2 24575868928 0 "PCI Slot: 4; FDOM: 2" normal 1014M02HP2 DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.3 FLASH_4_3 FlashDisk 4_3 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T12:00:49-05:00 1014M02HP4 24575868928 0 "PCI Slot: 4; FDOM: 3" normal 1014M02HP4 DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.3 FLASH_5_0 FlashDisk 5_0 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T12:00:49-05:00 1014M02JUD 24575868928 0 "PCI Slot: 5; FDOM: 0" normal 1014M02JUD DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.3 FLASH_5_1 FlashDisk 5_1 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T12:00:49-05:00 1014M02JVF 24575868928 0 "PCI Slot: 5; FDOM: 1" normal 1014M02JVF DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.3 FLASH_5_2 FlashDisk 5_2 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T12:00:49-05:00 1014M02JAP 24575868928 0 "PCI Slot: 5; FDOM: 2" normal 1014M02JAP DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.3 FLASH_5_3 FlashDisk 5_3 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T12:00:49-05:00 1014M02JVH 24575868928 0 "PCI Slot: 5; FDOM: 3" normal 1014M02JVH DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.4 16:0 HardDisk 0_0 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2013-02-05T10:21:30-06:00 JK1151YAJ27ANZ 2000011650924 0 normal JK1151YAJ27ANZ DMG_SYSTEM_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.4 16:1 HardDisk 0_1 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2013-02-07T10:52:03-06:00 JK11D1B8J9LG8Z 2000011650924 1 normal JK11D1B8J9LG8Z DMG_SYSTEM_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.4 16:10 HardDisk 0_10 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:09:38-05:00 JK1160YAHSJGBT 2000011650924 10 normal JK1160YAHSJGBT DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.4 16:11 HardDisk 0_11 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:09:39-05:00 JK1160YAHNTS9T 2000011650924 11 normal JK1160YAHNTS9T DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.4 16:2 HardDisk 0_2 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:09:28-05:00 JK1160YAHP8NHT 2000011650924 2 normal JK1160YAHP8NHT DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.4 16:3 HardDisk 0_3 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:09:29-05:00 JK1160YAHP0AST 2000011650924 3 normal JK1160YAHP0AST DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.4 16:4 HardDisk 0_4 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:09:30-05:00 JK1160YAHNPEGT 2000011650924 4 normal JK1160YAHNPEGT DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.4 16:5 HardDisk 0_5 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2013-02-07T10:53:26-06:00 JK11D1YAJJAN8Z 2000011650924 5 normal JK11D1YAJJAN8Z DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.4 16:6 HardDisk 0_6 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:09:33-05:00 JK1160YAHP0ULT 2000011650924 6 normal JK1160YAHP0ULT DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.4 16:7 HardDisk 0_7 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:09:34-05:00 JK1160YAHSH6WT 2000011650924 7 normal JK1160YAHSH6WT DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.4 16:8 HardDisk 0_8 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:09:35-05:00 JK1160YAHPB4BT 2000011650924 8 normal JK1160YAHPB4BT DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.4 16:9 HardDisk 0_9 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2010-05-15T21:09:37-05:00 JK1160YAHSJ7PT 2000011650924 9 normal JK1160YAHSJ7PT DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.4 FLASH_1_0 FlashDisk 1_0 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T11:59:44-05:00 1014M02JWP 24575868928 0 "PCI Slot: 1; FDOM: 0" normal 1014M02JWP DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.4 FLASH_1_1 FlashDisk 1_1 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T11:59:44-05:00 1014M02JWN 24575868928 0 "PCI Slot: 1; FDOM: 1" normal 1014M02JWN DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.4 FLASH_1_2 FlashDisk 1_2 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T11:59:44-05:00 1014M02JWQ 24575868928 0 "PCI Slot: 1; FDOM: 2" normal 1014M02JWQ DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.4 FLASH_1_3 FlashDisk 1_3 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T11:59:44-05:00 1014M02JWA 24575868928 0 "PCI Slot: 1; FDOM: 3" normal 1014M02JWA DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.4 FLASH_2_0 FlashDisk 2_0 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T11:59:44-05:00 1016M02K6T 24575868928 0 "PCI Slot: 2; FDOM: 0" normal 1016M02K6T DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.4 FLASH_2_1 FlashDisk 2_1 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T11:59:44-05:00 1016M02KEE 24575868928 0 "PCI Slot: 2; FDOM: 1" normal 1016M02KEE DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.4 FLASH_2_2 FlashDisk 2_2 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T11:59:44-05:00 1016M02K6U 24575868928 0 "PCI Slot: 2; FDOM: 2" normal 1016M02K6U DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.4 FLASH_2_3 FlashDisk 2_3 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T11:59:44-05:00 1016M02KF0 24575868928 0 "PCI Slot: 2; FDOM: 3" normal 1016M02KF0 DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.4 FLASH_4_0 FlashDisk 4_0 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T11:59:44-05:00 1014M02HK2 24575868928 0 "PCI Slot: 4; FDOM: 0" normal 1014M02HK2 DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.4 FLASH_4_1 FlashDisk 4_1 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T11:59:44-05:00 1014M02HK4 24575868928 0 "PCI Slot: 4; FDOM: 1" normal 1014M02HK4 DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.4 FLASH_4_2 FlashDisk 4_2 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T11:59:44-05:00 1014M02HEM 24575868928 0 "PCI Slot: 4; FDOM: 2" normal 1014M02HEM DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.4 FLASH_4_3 FlashDisk 4_3 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T11:59:44-05:00 1014M02HGU 24575868928 0 "PCI Slot: 4; FDOM: 3" normal 1014M02HGU DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.4 FLASH_5_0 FlashDisk 5_0 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T11:59:44-05:00 0940M018DG 24575868928 0 "PCI Slot: 5; FDOM: 0" normal 0940M018DG DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.4 FLASH_5_1 FlashDisk 5_1 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T11:59:44-05:00 1014M02JXM 24575868928 0 "PCI Slot: 5; FDOM: 1" normal 1014M02JXM DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.4 FLASH_5_2 FlashDisk 5_2 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T11:59:44-05:00 1014M02JXP 24575868928 0 "PCI Slot: 5; FDOM: 2" normal 1014M02JXP DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.4 FLASH_5_3 FlashDisk 5_3 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-05-06T11:59:44-05:00 1014M02J2E 24575868928 0 "PCI Slot: 5; FDOM: 3" normal 1014M02J2E DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.5 16:0 HardDisk 0_0 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2013-02-05T12:20:03-06:00 JK11D1YAK4SRWZ 2000011650924 0 normal JK11D1YAK4SRWZ DMG_SYSTEM_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.5 16:1 HardDisk 0_1 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2013-02-07T10:53:25-06:00 JK11D1YAJE122Z 2000011650924 1 normal JK11D1YAJE122Z DMG_SYSTEM_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.5 16:10 HardDisk 0_10 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2011-10-26T12:38:40-05:00 JK11D1YAJB3PRZ 2000011650924 10 normal JK11D1YAJB3PRZ DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.5 16:11 HardDisk 0_11 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2011-10-26T12:38:46-05:00 JK11D1YAJ8THDZ 2000011650924 11 normal JK11D1YAJ8THDZ DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.5 16:2 HardDisk 0_2 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2011-10-26T12:37:58-05:00 JK11D1YAJ8T7YZ 2000011650924 2 normal JK11D1YAJ8T7YZ DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.5 16:3 HardDisk 0_3 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2011-10-26T12:38:03-05:00 JK11D1YAJ8TH7Z 2000011650924 3 normal JK11D1YAJ8TH7Z DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.5 16:4 HardDisk 0_4 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2011-10-26T12:38:09-05:00 JK11D1YAJ7WREZ 2000011650924 4 normal JK11D1YAJ7WREZ DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.5 16:5 HardDisk 0_5 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2011-10-26T12:38:14-05:00 JK1161YAGD96BV 2000011650924 5 normal JK1161YAGD96BV DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.5 16:6 HardDisk 0_6 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2011-10-26T12:38:19-05:00 JK11D1YAJ7WL8Z 2000011650924 6 normal JK11D1YAJ7WL8Z DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.5 16:7 HardDisk 0_7 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2011-10-26T12:38:24-05:00 JK11D1YAJ7WR0Z 2000011650924 7 normal JK11D1YAJ7WR0Z DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.5 16:8 HardDisk 0_8 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2011-10-26T12:38:29-05:00 JK11D1YAJAZKZZ 2000011650924 8 normal JK11D1YAJAZKZZ DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.5 16:9 HardDisk 0_9 "HITACHI H7220AA30SUN2.0T" JKAOA28A 2011-10-26T12:38:35-05:00 JK11D1YAJ8T7TZ 2000011650924 9 normal JK11D1YAJ8T7TZ DMG_DATA_HDD2 FALSE FALSE FALSE FALSE FALSE 0 0 192.168.12.5 FLASH_1_0 FlashDisk 1_0 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-10-26T12:38:46-05:00 1016M02N8T 24575868928 0 "PCI Slot: 1; FDOM: 0" normal 1016M02N8T DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.5 FLASH_1_1 FlashDisk 1_1 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-10-26T12:38:46-05:00 1016M02N75 24575868928 0 "PCI Slot: 1; FDOM: 1" normal 1016M02N75 DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.5 FLASH_1_2 FlashDisk 1_2 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-10-26T12:38:46-05:00 1016M02N9P 24575868928 0 "PCI Slot: 1; FDOM: 2" normal 1016M02N9P DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.5 FLASH_1_3 FlashDisk 1_3 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-10-26T12:38:46-05:00 1016M02N9Q 24575868928 0 "PCI Slot: 1; FDOM: 3" normal 1016M02N9Q DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.5 FLASH_2_0 FlashDisk 2_0 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-10-26T12:38:46-05:00 1016M02MNK 24575868928 0 "PCI Slot: 2; FDOM: 0" normal 1016M02MNK DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.5 FLASH_2_1 FlashDisk 2_1 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-10-26T12:38:46-05:00 1016M02N0D 24575868928 0 "PCI Slot: 2; FDOM: 1" normal 1016M02N0D DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.5 FLASH_2_2 FlashDisk 2_2 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-10-26T12:38:46-05:00 1016M02MWK 24575868928 0 "PCI Slot: 2; FDOM: 2" normal 1016M02MWK DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.5 FLASH_2_3 FlashDisk 2_3 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-10-26T12:38:46-05:00 1016M02MYY 24575868928 0 "PCI Slot: 2; FDOM: 3" normal 1016M02MYY DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.5 FLASH_4_0 FlashDisk 4_0 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-10-26T12:38:46-05:00 1016M02NJE 24575868928 0 "PCI Slot: 4; FDOM: 0" normal 1016M02NJE DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.5 FLASH_4_1 FlashDisk 4_1 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-10-26T12:38:46-05:00 1016M02NJ7 24575868928 0 "PCI Slot: 4; FDOM: 1" normal 1016M02NJ7 DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.5 FLASH_4_2 FlashDisk 4_2 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-10-26T12:38:46-05:00 1016M02NJH 24575868928 0 "PCI Slot: 4; FDOM: 2" normal 1016M02NJH DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.5 FLASH_4_3 FlashDisk 4_3 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-10-26T12:38:46-05:00 1016M02NGK 24575868928 0 "PCI Slot: 4; FDOM: 3" normal 1016M02NGK DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.5 FLASH_5_0 FlashDisk 5_0 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-10-26T12:38:46-05:00 1016M02MWE 24575868928 0 "PCI Slot: 5; FDOM: 0" normal 1016M02MWE DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.5 FLASH_5_1 FlashDisk 5_1 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-10-26T12:38:46-05:00 1016M02MWC 24575868928 0 "PCI Slot: 5; FDOM: 1" normal 1016M02MWC DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.5 FLASH_5_2 FlashDisk 5_2 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-10-26T12:38:46-05:00 1016M02MTG 24575868928 0 "PCI Slot: 5; FDOM: 2" normal 1016M02MTG DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 192.168.12.5 FLASH_5_3 FlashDisk 5_3 "Sun Flash Accelerator F20 PCIe Card" D21Y 2011-10-26T12:38:46-05:00 1016M02NH2 24575868928 0 "PCI Slot: 5; FDOM: 3" normal 1016M02NH2 DMG_FDD1 FALSE FALSE FALSE FALSE FALSE 0 0 0 0 0 0 84 rows selected.
The output should be more or less self-explanatory for Exadata DBAs.
Notes:
- Bane Radulović (the ASM Support Guy) just recently wrote a blog entry (and MOS note) about identifying poorly performing disks in Exadata
- Vishal Desai has also written some scripts which allow list Exadata cell disks (and other cool stuff too!)
Enjoy!
List Exadata Disk Layout and Topology with the exadisktopo scripts
Posted May 24, 2013Here are two more Exadata scripts for listing the end-to-end ASM<->Exadata disk topology from V$ASM_ views and from V$CELL_CONFIG. These scripts see both the ASM level layout and the storage cell-level disk topology.
The exadisktopo.sql script shows all disks starting from the ASM diskgroup layer, going deeper and deeper all the way to the OS disk device level in the storage cells. It uses outer joins, so will show celldisks even if there are no corresponding grid disks allocated on them (or if there are no ASM disks using them). It also shows the Flash cards used as flash cache, thus there are no ASM disks on them usually.
The output should be self-explanatory, you just read the data from left to right (from ASM diskgroup to ASM disk to Grid Disk level etc), the more rightwards you scroll, the “deeper” you go in the Exadata IO stack layers. Warning, lots of wide output coming :)
SQL> @exadata/exadisktopo Showing Exadata disk topology from V$ASM_DISK and V$CELL_CONFIG.... ASM_DISKGROUP ASM_DISK GRIDDISK CELLDISK CELLNAME CD_DEVICEPART PHYSDISK PHYSDISK_STATUS LUNWRITECACHEMODE ------------------------------ ------------------------------ ------------------------------ ------------------------------ -------------------- -------------------- ------------------------------ -------------------- ---------------------------------------------------------------------------------------------------- DATA DATA_CD_00_ENKCEL01 DATA_CD_00_enkcel01 CD_00_enkcel01 192.168.12.3 /dev/sda3 35:0 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_00_ENKCEL02 DATA_CD_00_enkcel02 CD_00_enkcel02 192.168.12.4 /dev/sda3 16:0 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_00_ENKCEL03 DATA_CD_00_enkcel03 CD_00_enkcel03 192.168.12.5 /dev/sdad3 16:0 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_01_ENKCEL01 DATA_CD_01_enkcel01 CD_01_enkcel01 192.168.12.3 /dev/sdb3 35:1 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_01_ENKCEL02 DATA_CD_01_enkcel02 CD_01_enkcel02 192.168.12.4 /dev/sdb3 16:1 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_01_ENKCEL03 DATA_CD_01_enkcel03 CD_01_enkcel03 192.168.12.5 /dev/sda3 16:1 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_02_ENKCEL01 DATA_CD_02_enkcel01 CD_02_enkcel01 192.168.12.3 /dev/sdc 35:2 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_02_ENKCEL02 DATA_CD_02_enkcel02 CD_02_enkcel02 192.168.12.4 /dev/sdc 16:2 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_02_ENKCEL03 DATA_CD_02_enkcel03 CD_02_enkcel03 192.168.12.5 /dev/sdb 16:2 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_03_ENKCEL02 DATA_CD_03_enkcel02 CD_03_enkcel02 192.168.12.4 /dev/sdd 16:3 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_03_ENKCEL03 DATA_CD_03_enkcel03 CD_03_enkcel03 192.168.12.5 /dev/sdc 16:3 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_04_ENKCEL01 DATA_CD_04_enkcel01 CD_04_enkcel01 192.168.12.3 /dev/sdl 35:4 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_04_ENKCEL02 DATA_CD_04_enkcel02 CD_04_enkcel02 192.168.12.4 /dev/sde 16:4 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_04_ENKCEL03 DATA_CD_04_enkcel03 CD_04_enkcel03 192.168.12.5 /dev/sdd 16:4 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_05_ENKCEL03 DATA_CD_05_enkcel03 CD_05_enkcel03 192.168.12.5 /dev/sde 16:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_06_ENKCEL01 DATA_CD_06_enkcel01 CD_06_enkcel01 192.168.12.3 /dev/sdg 35:6 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_06_ENKCEL02 DATA_CD_06_enkcel02 CD_06_enkcel02 192.168.12.4 /dev/sdg 16:6 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_06_ENKCEL03 DATA_CD_06_enkcel03 CD_06_enkcel03 192.168.12.5 /dev/sdf 16:6 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_07_ENKCEL01 DATA_CD_07_enkcel01 CD_07_enkcel01 192.168.12.3 /dev/sdh 35:7 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_07_ENKCEL02 DATA_CD_07_enkcel02 CD_07_enkcel02 192.168.12.4 /dev/sdh 16:7 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_07_ENKCEL03 DATA_CD_07_enkcel03 CD_07_enkcel03 192.168.12.5 /dev/sdg 16:7 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_08_ENKCEL01 DATA_CD_08_enkcel01 CD_08_enkcel01 192.168.12.3 /dev/sdi 35:8 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_08_ENKCEL02 DATA_CD_08_enkcel02 CD_08_enkcel02 192.168.12.4 /dev/sdi 16:8 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_08_ENKCEL03 DATA_CD_08_enkcel03 CD_08_enkcel03 192.168.12.5 /dev/sdh 16:8 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_09_ENKCEL01 DATA_CD_09_enkcel01 CD_09_enkcel01 192.168.12.3 /dev/sdj 35:9 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_09_ENKCEL02 DATA_CD_09_enkcel02 CD_09_enkcel02 192.168.12.4 /dev/sdj 16:9 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_09_ENKCEL03 DATA_CD_09_enkcel03 CD_09_enkcel03 192.168.12.5 /dev/sdi 16:9 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_10_ENKCEL01 DATA_CD_10_enkcel01 CD_10_enkcel01 192.168.12.3 /dev/sde 35:10 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_10_ENKCEL02 DATA_CD_10_enkcel02 CD_10_enkcel02 192.168.12.4 /dev/sdk 16:10 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_10_ENKCEL03 DATA_CD_10_enkcel03 CD_10_enkcel03 192.168.12.5 /dev/sdj 16:10 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_11_ENKCEL02 DATA_CD_11_enkcel02 CD_11_enkcel02 192.168.12.4 /dev/sdl 16:11 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DATA_CD_11_ENKCEL03 DATA_CD_11_enkcel03 CD_11_enkcel03 192.168.12.5 /dev/sdk 16:11 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG DBFS_DG_CD_02_ENKCEL01 DBFS_DG_CD_02_enkcel01 CD_02_enkcel01 192.168.12.3 /dev/sdc 35:2 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_02_ENKCEL02 DBFS_DG_CD_02_enkcel02 CD_02_enkcel02 192.168.12.4 /dev/sdc 16:2 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_02_ENKCEL03 DBFS_DG_CD_02_enkcel03 CD_02_enkcel03 192.168.12.5 /dev/sdb 16:2 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_03_ENKCEL02 DBFS_DG_CD_03_enkcel02 CD_03_enkcel02 192.168.12.4 /dev/sdd 16:3 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_03_ENKCEL03 DBFS_DG_CD_03_enkcel03 CD_03_enkcel03 192.168.12.5 /dev/sdc 16:3 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_04_ENKCEL01 DBFS_DG_CD_04_enkcel01 CD_04_enkcel01 192.168.12.3 /dev/sdl 35:4 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_04_ENKCEL02 DBFS_DG_CD_04_enkcel02 CD_04_enkcel02 192.168.12.4 /dev/sde 16:4 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_04_ENKCEL03 DBFS_DG_CD_04_enkcel03 CD_04_enkcel03 192.168.12.5 /dev/sdd 16:4 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_05_ENKCEL01 DBFS_DG_CD_05_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_05_ENKCEL03 DBFS_DG_CD_05_enkcel03 CD_05_enkcel03 192.168.12.5 /dev/sde 16:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_06_ENKCEL01 DBFS_DG_CD_06_enkcel01 CD_06_enkcel01 192.168.12.3 /dev/sdg 35:6 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_06_ENKCEL02 DBFS_DG_CD_06_enkcel02 CD_06_enkcel02 192.168.12.4 /dev/sdg 16:6 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_06_ENKCEL03 DBFS_DG_CD_06_enkcel03 CD_06_enkcel03 192.168.12.5 /dev/sdf 16:6 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_07_ENKCEL01 DBFS_DG_CD_07_enkcel01 CD_07_enkcel01 192.168.12.3 /dev/sdh 35:7 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_07_ENKCEL02 DBFS_DG_CD_07_enkcel02 CD_07_enkcel02 192.168.12.4 /dev/sdh 16:7 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_07_ENKCEL03 DBFS_DG_CD_07_enkcel03 CD_07_enkcel03 192.168.12.5 /dev/sdg 16:7 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_08_ENKCEL01 DBFS_DG_CD_08_enkcel01 CD_08_enkcel01 192.168.12.3 /dev/sdi 35:8 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_08_ENKCEL02 DBFS_DG_CD_08_enkcel02 CD_08_enkcel02 192.168.12.4 /dev/sdi 16:8 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_08_ENKCEL03 DBFS_DG_CD_08_enkcel03 CD_08_enkcel03 192.168.12.5 /dev/sdh 16:8 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_09_ENKCEL01 DBFS_DG_CD_09_enkcel01 CD_09_enkcel01 192.168.12.3 /dev/sdj 35:9 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_09_ENKCEL02 DBFS_DG_CD_09_enkcel02 CD_09_enkcel02 192.168.12.4 /dev/sdj 16:9 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_09_ENKCEL03 DBFS_DG_CD_09_enkcel03 CD_09_enkcel03 192.168.12.5 /dev/sdi 16:9 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_10_ENKCEL01 DBFS_DG_CD_10_enkcel01 CD_10_enkcel01 192.168.12.3 /dev/sde 35:10 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_10_ENKCEL02 DBFS_DG_CD_10_enkcel02 CD_10_enkcel02 192.168.12.4 /dev/sdk 16:10 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_10_ENKCEL03 DBFS_DG_CD_10_enkcel03 CD_10_enkcel03 192.168.12.5 /dev/sdj 16:10 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_11_ENKCEL01 DBFS_DG_CD_11_enkcel01 CD_11_enkcel01 192.168.12.3 /dev/sdk 35:11 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_11_ENKCEL02 DBFS_DG_CD_11_enkcel02 CD_11_enkcel02 192.168.12.4 /dev/sdl 16:11 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" DBFS_DG_CD_11_ENKCEL03 DBFS_DG_CD_11_enkcel03 CD_11_enkcel03 192.168.12.5 /dev/sdk 16:11 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO RECO_CD_00_ENKCEL01 RECO_CD_00_enkcel01 CD_00_enkcel01 192.168.12.3 /dev/sda3 35:0 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_00_ENKCEL02 RECO_CD_00_enkcel02 CD_00_enkcel02 192.168.12.4 /dev/sda3 16:0 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_00_ENKCEL03 RECO_CD_00_enkcel03 CD_00_enkcel03 192.168.12.5 /dev/sdad3 16:0 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_01_ENKCEL01 RECO_CD_01_enkcel01 CD_01_enkcel01 192.168.12.3 /dev/sdb3 35:1 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_01_ENKCEL02 RECO_CD_01_enkcel02 CD_01_enkcel02 192.168.12.4 /dev/sdb3 16:1 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_01_ENKCEL03 RECO_CD_01_enkcel03 CD_01_enkcel03 192.168.12.5 /dev/sda3 16:1 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_02_ENKCEL01 RECO_CD_02_enkcel01 CD_02_enkcel01 192.168.12.3 /dev/sdc 35:2 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_02_ENKCEL02 RECO_CD_02_enkcel02 CD_02_enkcel02 192.168.12.4 /dev/sdc 16:2 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_02_ENKCEL03 RECO_CD_02_enkcel03 CD_02_enkcel03 192.168.12.5 /dev/sdb 16:2 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_03_ENKCEL02 RECO_CD_03_enkcel02 CD_03_enkcel02 192.168.12.4 /dev/sdd 16:3 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_03_ENKCEL03 RECO_CD_03_enkcel03 CD_03_enkcel03 192.168.12.5 /dev/sdc 16:3 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_04_ENKCEL01 RECO_CD_04_enkcel01 CD_04_enkcel01 192.168.12.3 /dev/sdl 35:4 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_04_ENKCEL02 RECO_CD_04_enkcel02 CD_04_enkcel02 192.168.12.4 /dev/sde 16:4 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_04_ENKCEL03 RECO_CD_04_enkcel03 CD_04_enkcel03 192.168.12.5 /dev/sdd 16:4 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_05_ENKCEL01 RECO_CD_05_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_05_ENKCEL03 RECO_CD_05_enkcel03 CD_05_enkcel03 192.168.12.5 /dev/sde 16:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_06_ENKCEL01 RECO_CD_06_enkcel01 CD_06_enkcel01 192.168.12.3 /dev/sdg 35:6 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_06_ENKCEL02 RECO_CD_06_enkcel02 CD_06_enkcel02 192.168.12.4 /dev/sdg 16:6 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_06_ENKCEL03 RECO_CD_06_enkcel03 CD_06_enkcel03 192.168.12.5 /dev/sdf 16:6 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_07_ENKCEL01 RECO_CD_07_enkcel01 CD_07_enkcel01 192.168.12.3 /dev/sdh 35:7 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_07_ENKCEL02 RECO_CD_07_enkcel02 CD_07_enkcel02 192.168.12.4 /dev/sdh 16:7 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_07_ENKCEL03 RECO_CD_07_enkcel03 CD_07_enkcel03 192.168.12.5 /dev/sdg 16:7 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_08_ENKCEL01 RECO_CD_08_enkcel01 CD_08_enkcel01 192.168.12.3 /dev/sdi 35:8 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_08_ENKCEL02 RECO_CD_08_enkcel02 CD_08_enkcel02 192.168.12.4 /dev/sdi 16:8 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_08_ENKCEL03 RECO_CD_08_enkcel03 CD_08_enkcel03 192.168.12.5 /dev/sdh 16:8 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_09_ENKCEL01 RECO_CD_09_enkcel01 CD_09_enkcel01 192.168.12.3 /dev/sdj 35:9 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_09_ENKCEL02 RECO_CD_09_enkcel02 CD_09_enkcel02 192.168.12.4 /dev/sdj 16:9 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_09_ENKCEL03 RECO_CD_09_enkcel03 CD_09_enkcel03 192.168.12.5 /dev/sdi 16:9 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_10_ENKCEL01 RECO_CD_10_enkcel01 CD_10_enkcel01 192.168.12.3 /dev/sde 35:10 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_10_ENKCEL03 RECO_CD_10_enkcel03 CD_10_enkcel03 192.168.12.5 /dev/sdj 16:10 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_11_ENKCEL01 RECO_CD_11_enkcel01 CD_11_enkcel01 192.168.12.3 /dev/sdk 35:11 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_11_ENKCEL02 RECO_CD_11_enkcel02 CD_11_enkcel02 192.168.12.4 /dev/sdl 16:11 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_11_ENKCEL03 RECO_CD_11_enkcel03 CD_11_enkcel03 192.168.12.5 /dev/sdk 16:11 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_00_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_01_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_02_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_03_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_04_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_05_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_06_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_07_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_08_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_09_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_100_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_101_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_102_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_103_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_104_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_105_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_106_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_107_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_108_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_109_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_10_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_110_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_111_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_112_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_113_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_114_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_115_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_116_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_117_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_118_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_119_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_11_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_120_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_121_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_122_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_123_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_124_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_125_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_126_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_127_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_128_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_129_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_12_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_130_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_131_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_132_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_133_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_134_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_135_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_136_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_137_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_138_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_139_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_13_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_140_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_141_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_142_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_143_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_144_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_145_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_146_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_147_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_148_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_149_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_14_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_150_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_151_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_152_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_153_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_154_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_155_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_156_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_157_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_158_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_159_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_15_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_160_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_161_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_162_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_163_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_164_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_165_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_166_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_167_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_16_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_17_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_18_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_19_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_20_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_21_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_22_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_23_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_24_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_25_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_26_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_27_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_28_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_29_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_30_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_31_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_32_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_33_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_34_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_35_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_36_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_37_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_38_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_39_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_40_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_41_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_42_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_43_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_44_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_45_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_46_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_47_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_48_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_49_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_50_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_51_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_52_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_53_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_54_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_55_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_56_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_57_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_58_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_59_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_60_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_61_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_62_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_63_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_64_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_65_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_66_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_67_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_68_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_69_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_70_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_71_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_72_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_73_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_74_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_75_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_76_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_77_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_78_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_79_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_80_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_81_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_82_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_83_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_84_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_85_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_86_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_87_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_88_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_89_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_90_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_91_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_92_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_93_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_94_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_95_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_96_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_97_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_98_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" ACTEST_CD_05_99_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" RECO_CD_10_enkcel02 CD_10_enkcel02 192.168.12.4 /dev/sdk 16:10 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" SMITHERS20_CD_11_enkcel01 CD_11_enkcel01 192.168.12.3 /dev/sdk 35:11 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" SMITHERS21_CD_11_enkcel01 CD_11_enkcel01 192.168.12.3 /dev/sdk 35:11 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" SMITHERS22_CD_11_enkcel01 CD_11_enkcel01 192.168.12.3 /dev/sdk 35:11 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" SMITHERS_enkcel01 CD_05_enkcel01 192.168.12.3 /dev/sdf 35:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" CD_03_enkcel01 192.168.12.3 /dev/sdd 35:3 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" CD_05_enkcel02 192.168.12.4 /dev/sdf 16:5 normal "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" FD_00_enkcel01 192.168.12.3 /dev/sdn FLASH_4_0 normal FD_00_enkcel02 192.168.12.4 /dev/sdr FLASH_1_0 normal FD_00_enkcel03 192.168.12.5 /dev/sdr FLASH_1_0 normal FD_01_enkcel01 192.168.12.3 /dev/sdo FLASH_4_1 normal FD_01_enkcel02 192.168.12.4 /dev/sds FLASH_1_1 normal FD_01_enkcel03 192.168.12.5 /dev/sds FLASH_1_1 normal FD_02_enkcel01 192.168.12.3 /dev/sdp FLASH_4_2 normal FD_02_enkcel02 192.168.12.4 /dev/sdt FLASH_1_2 normal FD_02_enkcel03 192.168.12.5 /dev/sdt FLASH_1_2 normal FD_03_enkcel01 192.168.12.3 /dev/sdq FLASH_4_3 normal FD_03_enkcel02 192.168.12.4 /dev/sdu FLASH_1_3 normal FD_03_enkcel03 192.168.12.5 /dev/sdu FLASH_1_3 normal FD_04_enkcel01 192.168.12.3 /dev/sdr FLASH_5_0 normal FD_04_enkcel02 192.168.12.4 /dev/sdz FLASH_2_0 normal FD_04_enkcel03 192.168.12.5 /dev/sdz FLASH_2_0 normal FD_05_enkcel01 192.168.12.3 /dev/sds FLASH_5_1 normal FD_05_enkcel02 192.168.12.4 /dev/sdaa FLASH_2_1 normal FD_05_enkcel03 192.168.12.5 /dev/sdaa FLASH_2_1 normal FD_06_enkcel01 192.168.12.3 /dev/sdt FLASH_5_2 normal FD_06_enkcel02 192.168.12.4 /dev/sdab FLASH_2_2 normal FD_06_enkcel03 192.168.12.5 /dev/sdab FLASH_2_2 normal FD_07_enkcel01 192.168.12.3 /dev/sdu FLASH_5_3 normal FD_07_enkcel02 192.168.12.4 /dev/sdac FLASH_2_3 normal FD_07_enkcel03 192.168.12.5 /dev/sdac FLASH_2_3 normal FD_08_enkcel02 192.168.12.4 /dev/sdn FLASH_4_0 normal FD_08_enkcel03 192.168.12.5 /dev/sdn FLASH_4_0 normal FD_09_enkcel02 192.168.12.4 /dev/sdo FLASH_4_1 normal FD_09_enkcel03 192.168.12.5 /dev/sdo FLASH_4_1 normal FD_10_enkcel02 192.168.12.4 /dev/sdp FLASH_4_2 normal FD_10_enkcel03 192.168.12.5 /dev/sdp FLASH_4_2 normal FD_11_enkcel02 192.168.12.4 /dev/sdq FLASH_4_3 normal FD_11_enkcel03 192.168.12.5 /dev/sdq FLASH_4_3 normal FD_12_enkcel02 192.168.12.4 /dev/sdv FLASH_5_0 normal FD_12_enkcel03 192.168.12.5 /dev/sdv FLASH_5_0 normal FD_13_enkcel02 192.168.12.4 /dev/sdw FLASH_5_1 normal FD_13_enkcel03 192.168.12.5 /dev/sdw FLASH_5_1 normal FD_14_enkcel02 192.168.12.4 /dev/sdx FLASH_5_2 normal FD_14_enkcel03 192.168.12.5 /dev/sdx FLASH_5_2 normal FD_15_enkcel02 192.168.12.4 /dev/sdy FLASH_5_3 normal FD_15_enkcel03 192.168.12.5 /dev/sdy FLASH_5_3 normal 192.168.12.3 FLASH_1_0 not present 192.168.12.3 FLASH_1_1 not present 192.168.12.3 FLASH_1_2 not present 192.168.12.3 FLASH_1_3 not present 192.168.12.3 FLASH_2_0 not present 192.168.12.3 FLASH_2_1 not present 192.168.12.3 FLASH_2_2 not present 192.168.12.3 FLASH_2_3 not present 316 rows selected.
See how even the disks which don’t have any corresponding ASM disks allocated from them, are still displayed. Also in our sandbox Exadata V2 machine that I like to use for testing (because it’s constantly a little broken, great for troubleshooting practice:) 2 of the 4 PCI flash cards of one storage cell are physically missing, but thanks to the cell-level Flash cache metadata, they’re still listed although there are no corresponding OS level disk devices available for them.
The exadisktopo2.sql shows the same data, but in an opposite order (from bottom of the IO stack upwards, starting from the cell IP and OS level disk device, going upwards all the way to corresponding ASM diskgroups:
SQL> @exadata/exadisktopo2 Showing Exadata disk topology from V$ASM_DISK and V$CELL_CONFIG.... CELLNAME LUN_DEVICENAME PHYSDISK PHYSDISK_STATUS CELLDISK CD_DEVICEPART GRIDDISK ASM_DISK ASM_DISKGROUP LUNWRITECACHEMODE -------------------- -------------------- ------------------------------ -------------------- ------------------------------ -------------------- ------------------------------ ------------------------------ ------------------------------ ---------------------------------------------------------------------------------------------------- 192.168.12.3 /dev/sda 35:0 normal CD_00_enkcel01 /dev/sda3 DATA_CD_00_enkcel01 DATA_CD_00_ENKCEL01 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sda 35:0 normal CD_00_enkcel01 /dev/sda3 RECO_CD_00_enkcel01 RECO_CD_00_ENKCEL01 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdb 35:1 normal CD_01_enkcel01 /dev/sdb3 DATA_CD_01_enkcel01 DATA_CD_01_ENKCEL01 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdb 35:1 normal CD_01_enkcel01 /dev/sdb3 RECO_CD_01_enkcel01 RECO_CD_01_ENKCEL01 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdc 35:2 normal CD_02_enkcel01 /dev/sdc DATA_CD_02_enkcel01 DATA_CD_02_ENKCEL01 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdc 35:2 normal CD_02_enkcel01 /dev/sdc DBFS_DG_CD_02_enkcel01 DBFS_DG_CD_02_ENKCEL01 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdc 35:2 normal CD_02_enkcel01 /dev/sdc RECO_CD_02_enkcel01 RECO_CD_02_ENKCEL01 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdd 35:3 normal CD_03_enkcel01 /dev/sdd "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdl 35:4 normal CD_04_enkcel01 /dev/sdl DATA_CD_04_enkcel01 DATA_CD_04_ENKCEL01 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdl 35:4 normal CD_04_enkcel01 /dev/sdl DBFS_DG_CD_04_enkcel01 DBFS_DG_CD_04_ENKCEL01 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdl 35:4 normal CD_04_enkcel01 /dev/sdl RECO_CD_04_enkcel01 RECO_CD_04_ENKCEL01 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_00_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_01_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_02_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_03_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_04_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_05_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_06_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_07_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_08_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_09_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_100_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_101_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_102_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_103_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_104_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_105_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_106_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_107_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_108_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_109_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_10_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_110_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_111_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_112_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_113_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_114_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_115_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_116_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_117_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_118_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_119_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_11_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_120_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_121_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_122_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_123_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_124_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_125_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_126_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_127_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_128_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_129_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_12_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_130_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_131_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_132_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_133_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_134_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_135_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_136_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_137_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_138_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_139_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_13_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_140_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_141_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_142_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_143_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_144_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_145_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_146_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_147_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_148_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_149_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_14_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_150_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_151_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_152_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_153_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_154_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_155_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_156_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_157_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_158_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_159_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_15_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_160_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_161_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_162_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_163_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_164_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_165_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_166_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_167_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_16_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_17_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_18_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_19_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_20_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_21_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_22_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_23_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_24_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_25_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_26_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_27_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_28_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_29_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_30_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_31_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_32_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_33_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_34_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_35_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_36_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_37_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_38_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_39_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_40_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_41_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_42_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_43_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_44_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_45_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_46_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_47_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_48_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_49_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_50_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_51_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_52_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_53_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_54_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_55_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_56_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_57_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_58_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_59_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_60_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_61_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_62_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_63_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_64_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_65_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_66_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_67_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_68_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_69_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_70_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_71_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_72_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_73_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_74_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_75_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_76_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_77_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_78_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_79_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_80_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_81_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_82_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_83_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_84_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_85_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_86_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_87_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_88_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_89_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_90_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_91_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_92_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_93_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_94_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_95_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_96_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_97_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_98_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf ACTEST_CD_05_99_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf DBFS_DG_CD_05_enkcel01 DBFS_DG_CD_05_ENKCEL01 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf RECO_CD_05_enkcel01 RECO_CD_05_ENKCEL01 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 35:5 normal CD_05_enkcel01 /dev/sdf SMITHERS_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdg 35:6 normal CD_06_enkcel01 /dev/sdg DATA_CD_06_enkcel01 DATA_CD_06_ENKCEL01 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdg 35:6 normal CD_06_enkcel01 /dev/sdg DBFS_DG_CD_06_enkcel01 DBFS_DG_CD_06_ENKCEL01 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdg 35:6 normal CD_06_enkcel01 /dev/sdg RECO_CD_06_enkcel01 RECO_CD_06_ENKCEL01 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdh 35:7 normal CD_07_enkcel01 /dev/sdh DATA_CD_07_enkcel01 DATA_CD_07_ENKCEL01 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdh 35:7 normal CD_07_enkcel01 /dev/sdh DBFS_DG_CD_07_enkcel01 DBFS_DG_CD_07_ENKCEL01 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdh 35:7 normal CD_07_enkcel01 /dev/sdh RECO_CD_07_enkcel01 RECO_CD_07_ENKCEL01 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdi 35:8 normal CD_08_enkcel01 /dev/sdi DATA_CD_08_enkcel01 DATA_CD_08_ENKCEL01 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdi 35:8 normal CD_08_enkcel01 /dev/sdi DBFS_DG_CD_08_enkcel01 DBFS_DG_CD_08_ENKCEL01 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdi 35:8 normal CD_08_enkcel01 /dev/sdi RECO_CD_08_enkcel01 RECO_CD_08_ENKCEL01 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdj 35:9 normal CD_09_enkcel01 /dev/sdj DATA_CD_09_enkcel01 DATA_CD_09_ENKCEL01 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdj 35:9 normal CD_09_enkcel01 /dev/sdj DBFS_DG_CD_09_enkcel01 DBFS_DG_CD_09_ENKCEL01 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdj 35:9 normal CD_09_enkcel01 /dev/sdj RECO_CD_09_enkcel01 RECO_CD_09_ENKCEL01 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sde 35:10 normal CD_10_enkcel01 /dev/sde DATA_CD_10_enkcel01 DATA_CD_10_ENKCEL01 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sde 35:10 normal CD_10_enkcel01 /dev/sde DBFS_DG_CD_10_enkcel01 DBFS_DG_CD_10_ENKCEL01 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sde 35:10 normal CD_10_enkcel01 /dev/sde RECO_CD_10_enkcel01 RECO_CD_10_ENKCEL01 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdk 35:11 normal CD_11_enkcel01 /dev/sdk DBFS_DG_CD_11_enkcel01 DBFS_DG_CD_11_ENKCEL01 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdk 35:11 normal CD_11_enkcel01 /dev/sdk RECO_CD_11_enkcel01 RECO_CD_11_ENKCEL01 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdk 35:11 normal CD_11_enkcel01 /dev/sdk SMITHERS20_CD_11_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdk 35:11 normal CD_11_enkcel01 /dev/sdk SMITHERS21_CD_11_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdk 35:11 normal CD_11_enkcel01 /dev/sdk SMITHERS22_CD_11_enkcel01 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdn FLASH_4_0 normal FD_00_enkcel01 /dev/sdn /dev/sdo FLASH_4_1 normal FD_01_enkcel01 /dev/sdo /dev/sdp FLASH_4_2 normal FD_02_enkcel01 /dev/sdp /dev/sdq FLASH_4_3 normal FD_03_enkcel01 /dev/sdq /dev/sdr FLASH_5_0 normal FD_04_enkcel01 /dev/sdr /dev/sds FLASH_5_1 normal FD_05_enkcel01 /dev/sds /dev/sdt FLASH_5_2 normal FD_06_enkcel01 /dev/sdt /dev/sdu FLASH_5_3 normal FD_07_enkcel01 /dev/sdu FLASH_1_0 not present FLASH_1_1 not present FLASH_2_3 not present FLASH_2_2 not present FLASH_2_1 not present FLASH_1_2 not present FLASH_1_3 not present FLASH_2_0 not present 192.168.12.4 /dev/sda 16:0 normal CD_00_enkcel02 /dev/sda3 DATA_CD_00_enkcel02 DATA_CD_00_ENKCEL02 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sda 16:0 normal CD_00_enkcel02 /dev/sda3 RECO_CD_00_enkcel02 RECO_CD_00_ENKCEL02 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdb 16:1 normal CD_01_enkcel02 /dev/sdb3 DATA_CD_01_enkcel02 DATA_CD_01_ENKCEL02 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdb 16:1 normal CD_01_enkcel02 /dev/sdb3 RECO_CD_01_enkcel02 RECO_CD_01_ENKCEL02 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdc 16:2 normal CD_02_enkcel02 /dev/sdc DATA_CD_02_enkcel02 DATA_CD_02_ENKCEL02 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdc 16:2 normal CD_02_enkcel02 /dev/sdc DBFS_DG_CD_02_enkcel02 DBFS_DG_CD_02_ENKCEL02 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdc 16:2 normal CD_02_enkcel02 /dev/sdc RECO_CD_02_enkcel02 RECO_CD_02_ENKCEL02 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdd 16:3 normal CD_03_enkcel02 /dev/sdd DATA_CD_03_enkcel02 DATA_CD_03_ENKCEL02 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdd 16:3 normal CD_03_enkcel02 /dev/sdd DBFS_DG_CD_03_enkcel02 DBFS_DG_CD_03_ENKCEL02 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdd 16:3 normal CD_03_enkcel02 /dev/sdd RECO_CD_03_enkcel02 RECO_CD_03_ENKCEL02 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sde 16:4 normal CD_04_enkcel02 /dev/sde DATA_CD_04_enkcel02 DATA_CD_04_ENKCEL02 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sde 16:4 normal CD_04_enkcel02 /dev/sde DBFS_DG_CD_04_enkcel02 DBFS_DG_CD_04_ENKCEL02 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sde 16:4 normal CD_04_enkcel02 /dev/sde RECO_CD_04_enkcel02 RECO_CD_04_ENKCEL02 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 16:5 normal CD_05_enkcel02 /dev/sdf "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdg 16:6 normal CD_06_enkcel02 /dev/sdg DATA_CD_06_enkcel02 DATA_CD_06_ENKCEL02 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdg 16:6 normal CD_06_enkcel02 /dev/sdg DBFS_DG_CD_06_enkcel02 DBFS_DG_CD_06_ENKCEL02 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdg 16:6 normal CD_06_enkcel02 /dev/sdg RECO_CD_06_enkcel02 RECO_CD_06_ENKCEL02 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdh 16:7 normal CD_07_enkcel02 /dev/sdh DATA_CD_07_enkcel02 DATA_CD_07_ENKCEL02 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdh 16:7 normal CD_07_enkcel02 /dev/sdh DBFS_DG_CD_07_enkcel02 DBFS_DG_CD_07_ENKCEL02 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdh 16:7 normal CD_07_enkcel02 /dev/sdh RECO_CD_07_enkcel02 RECO_CD_07_ENKCEL02 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdi 16:8 normal CD_08_enkcel02 /dev/sdi DATA_CD_08_enkcel02 DATA_CD_08_ENKCEL02 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdi 16:8 normal CD_08_enkcel02 /dev/sdi DBFS_DG_CD_08_enkcel02 DBFS_DG_CD_08_ENKCEL02 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdi 16:8 normal CD_08_enkcel02 /dev/sdi RECO_CD_08_enkcel02 RECO_CD_08_ENKCEL02 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdj 16:9 normal CD_09_enkcel02 /dev/sdj DATA_CD_09_enkcel02 DATA_CD_09_ENKCEL02 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdj 16:9 normal CD_09_enkcel02 /dev/sdj DBFS_DG_CD_09_enkcel02 DBFS_DG_CD_09_ENKCEL02 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdj 16:9 normal CD_09_enkcel02 /dev/sdj RECO_CD_09_enkcel02 RECO_CD_09_ENKCEL02 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdk 16:10 normal CD_10_enkcel02 /dev/sdk DATA_CD_10_enkcel02 DATA_CD_10_ENKCEL02 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdk 16:10 normal CD_10_enkcel02 /dev/sdk DBFS_DG_CD_10_enkcel02 DBFS_DG_CD_10_ENKCEL02 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdk 16:10 normal CD_10_enkcel02 /dev/sdk RECO_CD_10_enkcel02 "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdl 16:11 normal CD_11_enkcel02 /dev/sdl DATA_CD_11_enkcel02 DATA_CD_11_ENKCEL02 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdl 16:11 normal CD_11_enkcel02 /dev/sdl DBFS_DG_CD_11_enkcel02 DBFS_DG_CD_11_ENKCEL02 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdl 16:11 normal CD_11_enkcel02 /dev/sdl RECO_CD_11_enkcel02 RECO_CD_11_ENKCEL02 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdr FLASH_1_0 normal FD_00_enkcel02 /dev/sdr /dev/sds FLASH_1_1 normal FD_01_enkcel02 /dev/sds /dev/sdt FLASH_1_2 normal FD_02_enkcel02 /dev/sdt /dev/sdu FLASH_1_3 normal FD_03_enkcel02 /dev/sdu /dev/sdz FLASH_2_0 normal FD_04_enkcel02 /dev/sdz /dev/sdaa FLASH_2_1 normal FD_05_enkcel02 /dev/sdaa /dev/sdab FLASH_2_2 normal FD_06_enkcel02 /dev/sdab /dev/sdac FLASH_2_3 normal FD_07_enkcel02 /dev/sdac /dev/sdn FLASH_4_0 normal FD_08_enkcel02 /dev/sdn /dev/sdo FLASH_4_1 normal FD_09_enkcel02 /dev/sdo /dev/sdp FLASH_4_2 normal FD_10_enkcel02 /dev/sdp /dev/sdq FLASH_4_3 normal FD_11_enkcel02 /dev/sdq /dev/sdv FLASH_5_0 normal FD_12_enkcel02 /dev/sdv /dev/sdw FLASH_5_1 normal FD_13_enkcel02 /dev/sdw /dev/sdx FLASH_5_2 normal FD_14_enkcel02 /dev/sdx /dev/sdy FLASH_5_3 normal FD_15_enkcel02 /dev/sdy 192.168.12.5 /dev/sdad 16:0 normal CD_00_enkcel03 /dev/sdad3 DATA_CD_00_enkcel03 DATA_CD_00_ENKCEL03 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdad 16:0 normal CD_00_enkcel03 /dev/sdad3 RECO_CD_00_enkcel03 RECO_CD_00_ENKCEL03 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sda 16:1 normal CD_01_enkcel03 /dev/sda3 DATA_CD_01_enkcel03 DATA_CD_01_ENKCEL03 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sda 16:1 normal CD_01_enkcel03 /dev/sda3 RECO_CD_01_enkcel03 RECO_CD_01_ENKCEL03 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdb 16:2 normal CD_02_enkcel03 /dev/sdb DATA_CD_02_enkcel03 DATA_CD_02_ENKCEL03 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdb 16:2 normal CD_02_enkcel03 /dev/sdb DBFS_DG_CD_02_enkcel03 DBFS_DG_CD_02_ENKCEL03 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdb 16:2 normal CD_02_enkcel03 /dev/sdb RECO_CD_02_enkcel03 RECO_CD_02_ENKCEL03 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdc 16:3 normal CD_03_enkcel03 /dev/sdc DATA_CD_03_enkcel03 DATA_CD_03_ENKCEL03 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdc 16:3 normal CD_03_enkcel03 /dev/sdc DBFS_DG_CD_03_enkcel03 DBFS_DG_CD_03_ENKCEL03 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdc 16:3 normal CD_03_enkcel03 /dev/sdc RECO_CD_03_enkcel03 RECO_CD_03_ENKCEL03 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdd 16:4 normal CD_04_enkcel03 /dev/sdd DATA_CD_04_enkcel03 DATA_CD_04_ENKCEL03 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdd 16:4 normal CD_04_enkcel03 /dev/sdd DBFS_DG_CD_04_enkcel03 DBFS_DG_CD_04_ENKCEL03 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdd 16:4 normal CD_04_enkcel03 /dev/sdd RECO_CD_04_enkcel03 RECO_CD_04_ENKCEL03 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sde 16:5 normal CD_05_enkcel03 /dev/sde DATA_CD_05_enkcel03 DATA_CD_05_ENKCEL03 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sde 16:5 normal CD_05_enkcel03 /dev/sde DBFS_DG_CD_05_enkcel03 DBFS_DG_CD_05_ENKCEL03 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sde 16:5 normal CD_05_enkcel03 /dev/sde RECO_CD_05_enkcel03 RECO_CD_05_ENKCEL03 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 16:6 normal CD_06_enkcel03 /dev/sdf DATA_CD_06_enkcel03 DATA_CD_06_ENKCEL03 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 16:6 normal CD_06_enkcel03 /dev/sdf DBFS_DG_CD_06_enkcel03 DBFS_DG_CD_06_ENKCEL03 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdf 16:6 normal CD_06_enkcel03 /dev/sdf RECO_CD_06_enkcel03 RECO_CD_06_ENKCEL03 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdg 16:7 normal CD_07_enkcel03 /dev/sdg DATA_CD_07_enkcel03 DATA_CD_07_ENKCEL03 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdg 16:7 normal CD_07_enkcel03 /dev/sdg DBFS_DG_CD_07_enkcel03 DBFS_DG_CD_07_ENKCEL03 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdg 16:7 normal CD_07_enkcel03 /dev/sdg RECO_CD_07_enkcel03 RECO_CD_07_ENKCEL03 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdh 16:8 normal CD_08_enkcel03 /dev/sdh DATA_CD_08_enkcel03 DATA_CD_08_ENKCEL03 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdh 16:8 normal CD_08_enkcel03 /dev/sdh DBFS_DG_CD_08_enkcel03 DBFS_DG_CD_08_ENKCEL03 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdh 16:8 normal CD_08_enkcel03 /dev/sdh RECO_CD_08_enkcel03 RECO_CD_08_ENKCEL03 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdi 16:9 normal CD_09_enkcel03 /dev/sdi DATA_CD_09_enkcel03 DATA_CD_09_ENKCEL03 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdi 16:9 normal CD_09_enkcel03 /dev/sdi DBFS_DG_CD_09_enkcel03 DBFS_DG_CD_09_ENKCEL03 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdi 16:9 normal CD_09_enkcel03 /dev/sdi RECO_CD_09_enkcel03 RECO_CD_09_ENKCEL03 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdj 16:10 normal CD_10_enkcel03 /dev/sdj DATA_CD_10_enkcel03 DATA_CD_10_ENKCEL03 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdj 16:10 normal CD_10_enkcel03 /dev/sdj DBFS_DG_CD_10_enkcel03 DBFS_DG_CD_10_ENKCEL03 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdj 16:10 normal CD_10_enkcel03 /dev/sdj RECO_CD_10_enkcel03 RECO_CD_10_ENKCEL03 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdk 16:11 normal CD_11_enkcel03 /dev/sdk DATA_CD_11_enkcel03 DATA_CD_11_ENKCEL03 DATA "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdk 16:11 normal CD_11_enkcel03 /dev/sdk DBFS_DG_CD_11_enkcel03 DBFS_DG_CD_11_ENKCEL03 DBFS_DG "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdk 16:11 normal CD_11_enkcel03 /dev/sdk RECO_CD_11_enkcel03 RECO_CD_11_ENKCEL03 RECO "WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU" /dev/sdr FLASH_1_0 normal FD_00_enkcel03 /dev/sdr /dev/sds FLASH_1_1 normal FD_01_enkcel03 /dev/sds /dev/sdt FLASH_1_2 normal FD_02_enkcel03 /dev/sdt /dev/sdu FLASH_1_3 normal FD_03_enkcel03 /dev/sdu /dev/sdz FLASH_2_0 normal FD_04_enkcel03 /dev/sdz /dev/sdaa FLASH_2_1 normal FD_05_enkcel03 /dev/sdaa /dev/sdab FLASH_2_2 normal FD_06_enkcel03 /dev/sdab /dev/sdac FLASH_2_3 normal FD_07_enkcel03 /dev/sdac /dev/sdn FLASH_4_0 normal FD_08_enkcel03 /dev/sdn /dev/sdo FLASH_4_1 normal FD_09_enkcel03 /dev/sdo /dev/sdp FLASH_4_2 normal FD_10_enkcel03 /dev/sdp /dev/sdq FLASH_4_3 normal FD_11_enkcel03 /dev/sdq /dev/sdv FLASH_5_0 normal FD_12_enkcel03 /dev/sdv /dev/sdw FLASH_5_1 normal FD_13_enkcel03 /dev/sdw /dev/sdx FLASH_5_2 normal FD_14_enkcel03 /dev/sdx /dev/sdy FLASH_5_3 normal FD_15_enkcel03 /dev/sdy 316 rows selected.
The output should again be self-explanatory.
Make sure you look inside the scripts to see where and how this data gets taken from. I’m using the WITH subquery with MATERIALIZE hints to avoid a case where CBO ends up accessing V$CELL_CONFIG data using a Nested Loop join as this would end up re-fetching the same configuration data again and again from storage cells each loop. So I better “materialize” it locally first.
Enjoy!
V$CELL_THREAD_HISTORY – “ASH” for Exadata Storage Cells
Posted May 23, 2013Did you know that there’s something like Active Session History also in the Exadata storage cells? ;-)
The V$CELL_THREAD_HISTORY view is somewhat like V$ACTIVE_SESSION_HISTORY, but it’s measuring thread activity in the Exadata Storage Cells:
SQL> @desc v$cell_thread_history Name Null? Type ------------------------------- -------- ---------------------------- 1 CELL_NAME VARCHAR2(1024) 2 SNAPSHOT_ID NUMBER 3 SNAPSHOT_TIME DATE 4 THREAD_ID NUMBER 5 JOB_TYPE VARCHAR2(32) 6 WAIT_STATE VARCHAR2(32) 7 WAIT_OBJECT_NAME VARCHAR2(32) 8 SQL_ID VARCHAR2(13) 9 DATABASE_ID NUMBER 10 INSTANCE_ID NUMBER 11 SESSION_ID NUMBER 12 SESSION_SERIAL_NUM NUMBER
It keeps about 10 minutes worth of samples of Exadata Storage Cell thread activity:
SQL> @minmax snapshot_time v$cell_thread_history Show min/max (low/high) values in column "snapshot_time" of table v$cell_thread_history... MIN(SNAPSHOT_TIME MAX(SNAPSHOT_TIME ----------------- ----------------- 20130419 14:42:15 20130419 14:52:54
Note that it’s not the V$ view or the database instance which stores this array – it’s the storage cells themselves. If you query the v$cell_thread_history view, your Oracle database session is going to “gather” this instrumentation data from all the required cells and present it to you, that’s why the “cell statistics gather” wait event shows up:
PARSING IN CURSOR #140596385017248 len=42 dep=0 uid=0 oct=3 lid=0 tim=1366404896817011 hv=4063158547 ad='19e452578' sqlid='63awy1gg t2xs8m' select count(*) from v$cell_thread_history END OF STMT PARSE #140596385017248:c=0,e=115,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=4272803188,tim=1366404896817010 EXEC #140596385017248:c=0,e=27,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=4272803188,tim=1366404896817127 WAIT #140596385017248: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=40 tim=1366404896817152 WAIT #140596385017248: nam='cell statistics gather' ela= 283 cellhash#=0 p2=0 p3=0 obj#=40 tim=1366404896818846 WAIT #140596385017248: nam='cell statistics gather' ela= 352 cellhash#=0 p2=0 p3=0 obj#=40 tim=1366404896819317 WAIT #140596385017248: nam='cell statistics gather' ela= 376 cellhash#=0 p2=0 p3=0 obj#=40 tim=1366404896820929 WAIT #140596385017248: nam='cell statistics gather' ela= 326 cellhash#=0 p2=0 p3=0 obj#=40 tim=1366404896822198 WAIT #140596385017248: nam='cell statistics gather' ela= 580 cellhash#=0 p2=0 p3=0 obj#=40 tim=1366404896823620 ...
Now, how frequently do the cells sample their “ASH” data:
SQL> SELECT DISTINCT (snapshot_time - LAG(snapshot_time)
OVER (ORDER BY snapshot_time)) * 86400 lag_seconds
FROM (SELECT distinct snapshot_time
FROM v$cell_thread_history WHERE cell_name = '192.168.12.3');
LAG_SECONDS
-----------
1Looks like the sampling is done exactly once per second!
So, great, what can we do with this data?
Here’s an Exadata performance script (cth.sql) which uses V$CELL_THREAD_HISTORY to dig into cell activity from the database layer.
First, I must warn you, the CTH (Cell Thread History) is not as powerful as the Database-level ASH (ASH has almost 100 columns in it already, but the v$cell_thread_history has only 12) and there seem to be some shortcomings in the instrumentation data, as you will see.
But it still allows us to drill down a bit deeper into Exadata storage cell activity straight from your favorite database access tool. And it also has database session-level details in it – which means this information can be used for systematic Exadata performance troubleshooting too!
As the database sessions do communicate the SQL_ID of the currently active SQL statement to the storage cells when sending them requests, we can list the TOP activity by SQL_ID inside the storage cells!
The cth.sql script takes 4 arguments:
- The columns to group the CTH activity data by
- The filter settings (which sessions, instances, sql_ids to report?)
- The begin time range to report
- The end time range to report
Note that the CTH view will only have about 10 minutes of history in it (unless the cells are almost idle). For longer-term analysis you might want to gather the history to some table every couple of minutes.
Also, the snapshot_time column shows the CELL time, so if there’s significant clock drift between the storage cells and DB nodes, you might measure the wrong range of data.
Anyway, let’s see the “TOP SQL” in all storage cells:
SQL> @exadata/cth sql_id 1=1 sysdate-1 sysdate Querying V$CELL_THREAD_HISTORY ("ASH" for Storage Cells) ... SECONDS SQL_ID ---------- ------------- 1755 977 akbpwk3j38vw7 146 25y04n1twszyu 135 98pza6kpfwczj 14 5n83a4q202674 12 0w2qpuc6u2zsp 12 147a57cxq3w5y 7 axdkf5tjztmy7 6 apgb2g9q2zjh1 2 dcq9a12vtcnuw 2 a9gvfh5hx9u98 1 f318xdxdn0pdc 1 0ws7ahf1d78qa 13 rows selected.
Immediately we see that majority of the time consumed does not have any SQL ID associated with it, but we will get back to this later. For now, let’s pick the TOP SQL_ID and drill down into it further, by grouping the data by some more columns:
SQL> @exadata/cth sql_id,job_type,wait_state,wait_object_name sql_id='akbpwk3j38vw7' sysdate-1 sysdate Querying V$CELL_THREAD_HISTORY ("ASH" for Storage Cells) ... SECONDS SQL_ID JOB_TYPE WAIT_STATE WAIT_OBJECT_NAME ---------- ------------- -------------------------------- -------------------------------- -------------------------------- 868 akbpwk3j38vw7 PredicateFilter working 52 akbpwk3j38vw7 PredicateDiskRead working 45 akbpwk3j38vw7 PredicateCacheGet waiting_for_SKGXP_send 7 akbpwk3j38vw7 CacheGet working 6 akbpwk3j38vw7 PredicateCacheGet working 3 akbpwk3j38vw7 CacheGet waiting_for_SKGXP_send 2 akbpwk3j38vw7 PredicateDiskRead waiting_for_mutex gdisk ext 2 akbpwk3j38vw7 CacheGet waiting_for_mutex RemoteSendPort 1 akbpwk3j38vw7 PredicateFilter waiting_for_mutex PredicateDisk-1 1 akbpwk3j38vw7 PredicateCachePut working
Now we have a breakdown of the storage cell thread activity, when (knowingly) doing work for the SQL_ID akbpwk3j38vw7. The job type shows the “phase” of execution in the data flow inside a storage cell. I think I need to write a separate post about the cell execution phases, but for now, the PredicateDiskRead, as the name says, deals with setting up the IOs for disk reads, the PredicateCacheGet is what deals with the (smart) IO requests handed to storage cells by DB sessions (and sending the results back too) and the PredicateFilter is the phase which does the actual processing of just read datablocks in storage cells (logically “opening” the block and passing it through cache, transactional layers to the data layer functions, which extract the required columns of the matching rows of any predicates offloaded to the storage cells). If you wonder where are the I/O wait states – it looks like for (at least) smart scans, thanks to asynchronous I/O the actual I/O waiting/reaping is done by separate threads, which do not “belong” to a single SQL_ID, but track the completion of all I/Os submitted by all threads independently. Cellsrv is a single (multithreaded) process, so it’s easy to separate tasks within the process. This is also how you avoid keeping all 100+ cellsrv threads “occupied” in case of slow IO.
The wait state column is more complicated than the database layer’s session wait state “waiting or not waiting” logic. There are no separate wait events listed in the CTH, but the wait state itself contains the reasons for some waits. Working means ON CPU (or trying to be on CPU if the OS scheduler hasn’t put the thread there yet) and all the waiting_ states, like the name says, mean waiting for something, like a next request over the network (waiting_for_SKGXP_receive) or asynchronous IO completion (waiting_for_IO_reap). Note that Exadata cellsrv architecture is radically different from what you’re used to from the Database world, it’s a multithreaded job queuing and scheduling engine where one thread may submit IOs and some completely different thread will be the one who checks for the asynchronous IO completion (IO reaping), while a yet another thread does the actual filter processing (offloaded execution) tasks. So don’t expect the cellsrv threads serving your smart scans to actually wait for any I/O, because there are independent threads doing I/O reaping for all incoming requests. If you want more deep details – I do talk about the cellsrv internals and data flow in my Advanced Exadata Performance seminar and also Roger MacNicol from Exadata Development will deliver an Exadata SmartScan Deep Dive presentation at Enkitec E4 in August this year (2013) ;-)
Here’s the (probably) full list of wait states from Oracle 11.2.0.3 BP11 running on Cellsrv 11.2.3.2.1)
initial working waiting_in_scheduler waiting_for_mutex waiting_for_condition_variable waiting_for_IO_reap waiting_for_SKGXP_receive waiting_for_SKGXP_send looking_for_job waiting_for_msg_hdr waiting_for_system_work waiting_for_offline_done waiting_for_storidx_lock waiting_for_issued_ios waiting_for_buffer_allocation waiting_for_connect waiting_for_flashcache_init waiting_for_io_quiescence waiting_for_io_draining
Here’s another example, where I show all activity in all storage cells, by SQL_ID (if known), the database_id, instance_id and the session_id of the database session:
SQL> @exadata/cth sql_id,database_id,session_id,job_type,wait_state 1=1 sysdate-1/24/60 sysdate Querying V$CELL_THREAD_HISTORY ("ASH" for Storage Cells) ... SECONDS SQL_ID DATABASE_ID SESSION_ID JOB_TYPE WAIT_STATE MIN(SNAPSHOT_TIME MAX(SNAPSHOT_TIME ---------- ------------- ----------- ---------- -------------------------------- -------------------------------- ----------------- ----------------- 1301 0qmhpg330zqfu 1538629110 2916 PredicateFilter working 20130514 09:20:11 20130514 09:21:10 313 0qmhpg330zqfu 1538629110 5 PredicateFilter working 20130514 09:20:11 20130514 09:21:10 242 0qmhpg330zqfu 1538629110 203 PredicateFilter working 20130514 09:20:11 20130514 09:21:10 73 0 0 UnidentifiedJob waiting_for_IO_reap 20130514 09:20:11 20130514 09:21:10 9 0 0 UnidentifiedJob working 20130514 09:20:13 20130514 09:20:58 4 1538629110 972 CachePut working 20130514 09:20:17 20130514 09:21:04 3 0 0 NetworkRead working 20130514 09:20:17 20130514 09:20:34 2 1538629110 584 CachePut working 20130514 09:20:13 20130514 09:20:59 2 1538629110 492 CachePut working 20130514 09:20:26 20130514 09:20:52 2 0qmhpg330zqfu 1538629110 5 PredicateDiskRead working 20130514 09:20:58 20130514 09:20:58 1 0 0 IO poor performance detection th 20130514 09:20:23 20130514 09:20:23 1 0 0 ProcessIoctl working 20130514 09:20:26 20130514 09:20:26 1 147a57cxq3w5y 1538629110 2140 CacheGet working 20130514 09:20:12 20130514 09:20:12 1 0w2qpuc6u2zsp 1538629110 1479 CacheGet working 20130514 09:21:04 20130514 09:21:04 1 0w2qpuc6u2zsp 1538629110 986 CacheGet working 20130514 09:20:44 20130514 09:20:44 1 0 0 System stats collection thread working 20130514 09:20:41 20130514 09:20:41 1 147a57cxq3w5y 1538629110 790 CacheGet waiting_for_SKGXP_send 20130514 09:20:29 20130514 09:20:29 1 apgb2g9q2zjh1 1538629110 1280 CacheGet working 20130514 09:21:01 20130514 09:21:01 1 apgb2g9q2zjh1 1538629110 1085 CacheGet working 20130514 09:21:02 20130514 09:21:02 1 4075070345 1177 CacheGet working 20130514 09:20:23 20130514 09:20:23
Notes:
- Exadata storage cell software (cellsrv) samples active thread states every second (like ASH), so every record in V$CELL_THREAD_HISTORY stands for 1 second of “cell activity time” if counted. You can COUNT the records in V$CELL_THREAD_HISTORY just like you can count them in ASH to get approximate (cell) activity time.
- You do not have to query the GV$ view as the V$ view itself returns all data from all storage cells (unless restricted by a where clause)
- There is no 1-1 correspondence of a cell thread and your database session, so don’t try to simply add these numbers up. A smart scan can asynchronously and independently work for your database session, working in advance, doing prefetching and preprocessing regardless of what your database session happens to be doing. One smart scan “session” can end up spawning multiple threads in each storage cell if the cell can not produce data fast enough than the database session can consume.
- The database_id column corresponds to the “global database ID” ( SELECT ksqdngunid FROM X$KSQDN ), not the usual DBID in v$database.
A warning: There have been a few cases where my session crashes (well, actually my process, but not the database) when querying this V$ view:
SQL> @exadata/cth wait_state,sql_id 1=1 sysdate-1/24 sysdate Querying V$CELL_THREAD_HISTORY ("ASH" for Storage Cells) ... , MIN(snapshot_time), MAX(snapshot_time) * ERROR at line 5: ORA-03113: end-of-file on communication channel
But it only has affected my own session so far, so when I log in and rerun the script, it works ok again. I yet have to file a bug for this – this is a documented V$ view and it shouldn’t crash when queried.
This post ended up quite long, so I will continue a Part 2 some day in the future :)
You can download the latest version of cth.sql from here.
Everything or Nothing in SQL
Posted May 23, 2013May 23, 2013 The following question recently came through an ERP mailing list (significantly rephrased): I would like to use the Microsoft Query tool in Microsoft Excel to extract records from the ERP database. I would like the list of parent records to be retrieved into Excel when all of the specified child records belong to […]
Vague and/or misleading claims can undermine otherwise useful themes
Posted May 23, 2013I’m pretty much a green sneaker, tree hugging conservationist. (The Nature Conservancy, Audubon, and Arbor Day get annual renewals like clockwork, I helped write and implement Scenic Road and Wetlands Preservation legislation here in Lebanon, NH in the late 1980s.) So I’m really disappointed when loss of species and habitat headlines and statistics are so oriented to shock value that my reaction is “Is there a seed of truth in this obvious attempt to mislead?” instead of concern for the subject matter. Today’s entry for my #please_read_tufte hall of shame: “…facing 50 percent drops in their numbers within seven years if the current rate of decline continues…” I’ll save you the math: that’s about 9.43 % per year. Now that is bad enough, and it avoids stirring up all manner of thoughts about “you’re lying to me somehow.” Now especially if they added some information about whether last year’s loss rate was an outlier or whether we should expect that to be about the rate for the upcoming years unless we do something. (Plenty of populations in the wild have cycles much steeper than that.) But no, all they wanted to do was publish 50% and damn the context. For someone who spends a lot of time trying to be clear and concise about the meaning of data and statistics this is really annoying – even if the underlying truth supports the claim, they sound like a vaporware sales team.
This is related to my friend Cary’s blog entry
http://carymillsap.blogspot.com/2011/01/describing-performance-improvements.html
(which I consider a classic.)
So when you post numbers and commentary about numbers, tell me something useful and succinct: Give me meaning in context, not the mathematical analog to making an ethical point by proof texting a fragment of the bible.
… and if you find yourself writing you improved performance by more than 100% make sure you’re clear that you’re talking about throughput of some transaction and not response time or be prepared to show me your time machine, ’cause without a time machine the asymptotic ceiling on response time reduction is 100%
Dynamic Sampling – 2
Posted May 23, 2013I’ve written about dynamic sampling in the past, but here’s a little wrinkle that’s easy to miss. How do you get the optimizer to work out the correct cardinality for a query like (the table creation statement follows the query):
select count(*) from t1 where n1 = n2 ; create table t1 as with generator as ( select --+ materialize rownum id from dual connect by level <= 1e4 ) select mod(rownum, 1000) n1, mod(rownum, 1000) n2 from generator v1, generator v2 where rownum <= 1e6 ;
If you’re running 11g and can changed the code there are a couple of easy options – adding a virtual column, or applying extended stats and then modifying the SQL accordingly would be appropriate.
-- Virtual Column alter table t1 add ( n3 generated always as ( case n1 when n2 then 1 end) virtual ) ; execute dbms_stats.gather_table_stats(user,'t1',method_opt=>'for columns n3 size 1') -- Extended Stats begin dbms_output.put_line( dbms_stats.create_extended_stats( ownname => user, tabname => 'T1', extension => '(case n1 when n2 then 1 else null end)' ) ); dbms_stats.gather_table_stats( ownname => user, tabname =>'T1', block_sample => true, method_opt => 'for columns (case n1 when n2 then 1 else null end) size 1' ); end; / select count(*) from t1 where (case n1 when n2 then 1 else null end)= 1 ;
If you can’t change the SQL statement, there’s always the option for bypassing the problem by fixing a suitable execution plan with an SQL Baseline, of course. Alternatively, if you can think of the right hint you could create an “SQL Patch” for the statement – but what hint might be appropriate ? I’ll answer that question in a minute.
Here’s another option, though: get Oracle to use dynamic sampling. (You probably guessed that from the title of the post.) So which level would you use to make this work ? Left to its own devices, Oracle would calculate the selectivity of the predicate n1 = n2 as the smaller of the two separate predicates “n1 = unknown” and “n2 = unknown”. So you might hope that level 3 (Oracle is “guessing”) or level 4 (more than one predicate on a single table) might be appropriate. It’s the latter that works. If you execute “alter session set optimizer_dynamic_sampling=4;” before executing this query, Oracle will sample the table before optimising.
The method works, but can you apply it ? Possibly not, if you’re not allowed to inject any extra SQL anywhere – after all, you probably don’t want to set the parameter at the system level (spfile or init.ora) because it may affect lots of other queries – introducing more work because of the sample, and then risking unexpected changes in execution plans. Setting the parameter for a session is often no better. And this brings me back to the SQL Patch approach – if you don’t want to create a baseline for the query then perhaps a patch with the hint /*+ opt_param(‘optimizer_dynamic_sampling’ 4) */ will do the trick. Don’t forget all the doubling of single quotes that you’ll need, though (this is the code fragment I used):
begin
sys.dbms_sqldiag_internal.i_create_patch(
sql_text =>
'
select
count(*)
from t1
where n1 = n2
',
hint_text => 'opt_param(''optimizer_dynamic_sampling'' 4)'
);
end;
/
For more analysis and commentary on the SQL Patch mechanism, you might like to read Dominic Brooks’ mini-series:
Laptop and Desktop SSD Update…
Posted May 22, 2013I recently wrote about installing SSDs in my Laptop and Desktop. I thought I would write a quick follow up post to mention how things are going.
I’m really happy with the changes to the performance of the desktop. As mentioned previously, it is now much quieter and really fast. A lot of my VMs run from the 1TB internal data drive, but the things I use most frequently are now sitting on the SSD. I’m starting to forget what life was like before SSD, except when I go to work and use the slowest PC that was ever built. 
The laptop upgrade was a really good move. Just before my first BGOUG presentation the projector seemed to freak out my MacBook and I was forced to reboot. With the old hard drive I would have been filling while waiting for the thing to start up. As it was, it restarted in a similar time it used to take to come out of hibernation and I was moving.
Having done the disk swap in the laptop so close to a conference I was a little bit nervous, so in addition to the laptop I had my old 500G external drive, my new 1TB external drive and the oringial internal hard drive in my bag. Unpacking all that, along with my Nexus 7, Nexus 4 and Kindle was very time consuming and a little embarrassing.
If you were at all in doubt about making the move to SSD, I can definitely recommend it.
Cheers
Tim…
PS. I reserve the right to start moaning about it when it wears out after a few weeks.
Laptop and Desktop SSD Update… was first posted on May 22, 2013 at 8:26 pm.
©2012 "The ORACLE-BASE Blog". Use of this feed is for personal non-commercial use only. If you are not reading this article in your feed reader, then the site is guilty of copyright infringement.
gather_plan_statistics – 2
Posted May 22, 2013Some time ago – actually a few years ago – I wrote a note about the hint /*+ gather_plan_statistics */ making some informal comments about the implementation and relevant hidden parameters. I’ve recently discovered a couple of notes from Alexander Anokhin describing the feature in far more detail and describing some of the misleading side effects of the implementaiton. There are two parts (so far): part 1 and part 2.
Oracle Exadata Database Machine: Proving 160 Xeon E7 Cores Are As “Slow” As 128 Xeon E5 Cores?
Posted May 22, 2013Reading Data Sheets
If you are in a position of influence affecting technology adoption in your enterprise you likely spend a lot of time reading data sheets from vendors. This is just a quick blog entry about something I simply haven’t taken the time to cover even though the topic at hand has always be a “problem.” Well, at least since the release of the Oracle Exadata Database Machine X2-8.
In the following references and screenshots you’ll see that Oracle cites 1.5 million flash read IOPS as an expected limit for both the full-rack Oracle Exadata Database Machine X3-2 and the Oracle Exadata Database Machine X3-8. All machines have limits and Exadata is no exception. Notice how I draw attention to the footnote that accompanies the flash read IOPS claim. Footnote number 3 says that both of these Exadata models are limited in flash read IOPS by the database host CPU. Let me repeat that last bit for anyone scrutinizing my words for reasons other than education: The Oracle Exadata Database Machine data sheets explicitly state flash read IOPS are limited by host CPU.
Oracle’s numbers in this case are SQL-driven from Oracle instances. I have no doubt these systems are both capable of achieving 1.5 million read IOPS from flash because, truth be told, that isn’t really all that many IOPS–especially when the IOPS throughput numbers are not accompanied by service times. In the 1990s it was all about “how much” but in modern times it’s about “how fast.” Bandwidth is an old, tired topic. Modern platforms are all about latency. Intel QPI put the problem of bandwidth to rest.
So, again, I don’t doubt the 1.5 million flash read IOPS citation. Exadata has a lot of flash cards and a lot of host processors to drive concurrent I/O. Indeed, with the concurrent processing capabilities of both of these Exadata models, Oracle would be able to achieve 1.5 million IOPS even if the service times were more in line with what one would expect with mechanical storage. Again, we never see service time citations so in actuality the 1.5 million number is just a representation of how much in-flight I/O the platform can handle.
Here is the new truth: IOPS is a storage bandwidth metric.
Host CPU Limited! How Many CPUs?
Here’s the stinger: Oracle blames host CPU for the 1.5 million flash read IOPS number. The problem with that is the X3-2 has 128 Xeon E5-2690 processor cores and the X3-8 has 160 Xeon E7-8870 processor cores. So what is Oracle’s real message here? Is it that the cores in the X3-8 are 20% slower than those in the X3-2 model? I don’t know. I can’t put words in Oracle’s mouth. However, if the data sheet is telling the truth then one of two things is true, either a) the E5-2690 processors are indeed 20% faster on a per-core basis than the E7-8870 or b) there is a processing asymmetry problem.
Not All CPU Bottlenecks Are Created Equal
Oracle would likely not be willing to dive into technical detail to the same level I do. Life is a series of choices–including who you chose to buy storage and platforms from. However, Oracle’s literature is clear about the number of active 40Gb QDR Infiniband ports there are in each configuration and this is where the asymmetry comes in. There are 8 active ports in both of these models. That means there are 8 streams of interrupt handling in both cases–regardless of how many cores there are in total.
As is the case with any networked storage, I recommend you monitor mpstat -P ALL output on database hosts to see whether there are cores nailed to the wall with interrupt processing at levels below total CPU-saturation. Never settle for high-level aggregate CPU utilization monitoring. Instead, drill down to the per-core level to watch out for asymmetry. Doing so is just good platform scientist work.
Between now and the time you should find yourself in a proof of concept test situation with Exadata, don’t hesitate to ask Oracle why–by their own words–both 128 cores and 160 cores are equally saturated when delivering maximum read IOPS in the database grid. After all, they charge the same per core (list price) to license Oracle Database on either of those processors.
Nice and Concise?
By the way, is there anyone who actually believes that both of these platforms top out at precisely 1.5 million flash read IOPS?
Oracle Exadata Database Machine X3-2 Datasheet
Oracle Exadata Database Machine X3-8 Datasheet
DISCLAIMER: This post tackles citations straight from Oracle published data sheets and published literature.
Filed under: oracle 
I’m speaking at Enkitec Extreme Exadata Expo E4
Posted May 22, 2013Just a short notice to those interested that I’m very proud to be in the lineup for Enkitec’s Extreme Exadata Expo. The event takes place August 5-6, 2013 and is held in the Four Seasons Resort & Spa, Irving, Texas. There is plenty of time for you to register.
I was really sorry I missed out last year but this time I’m glad to participate and attend!
The list of great speakers includes too many to name here-you should see for yourself about who is coming to Dallas this August and why this event is unmissable.
I’m hoping to see you there!
.png){kind=small}
Tweets