FCP_ARRAY_ERR8, FCP_ARRAY_ERR9, FSCSI_ERR6…

Bunch of messages popped up on a test box w/ FCP_ARRAY_ERR8,FCP_ARRAY_ERR8, FCP_ARRAY_ERR9, FSCSI_ERR6…. fget_config -Av was kinda hung, file system in Vg was unable to mount – all kinda SAN related things and the ultimate solution was to update microcode on fiber channel hba.

Error –

LABEL:          FCP_ARRAY_ERR8
IDENTIFIER:     483C9D10

Date/Time:       Mon Apr 20 15:07:39 CDT 2009
Sequence Number: 26895
Class:           H
Type:            INFO
Resource Name:   dac0
Resource Class:  array
Resource Type:   ibm-dac-V4
Location:        U0.1-P1-I4/Q1-W200200A0B8122A16
VPD:
Manufacturer…………….IBM
Machine Type and Model……1742-900
Part Number……………..348-0049782
ROS Level and ID…………0914

Solution – update microcode on card –

Microcode downloads

http://www14.software.ibm.com/webapp/set2/firmware/gjsn

Store rpm in /etc/microcode if you cant put on your NIM. Unpack rpm and use diag to update microcode  (diag–>Task Selection (Diagnostics, Advanced Diagnostics, Service Aids, etc.–>Microcode Tasks–>Download Microcode–>seletc fcsN –> follow the steps to seletc /etc/microcode )

SAN AIX HBA imp commands

• See the current OS levels

       uname -a

oslevel -r

oslevel -q

oslevel -g

 

• See adapters, drivers, and microcode.See the bottom of this section for AIX part number mappings

lsdev -Cc adapter -spci | grep fcs   For each of the FCS instances

 

• Micorcode

lsmcode -cd fcs0

 

• Firmware

lscfg -vl fcs0 | grep Z9

 

• Driver

lsdev -C -Ftype,name | grep -i fcs

 

• Software

lslpp -l | grep -i hba

lslpp -l | grep “FC Software”

lslpp -L EMC.Symm*   Should get two outputs – aix and fcp

lslpp -L devices.pci.df1000f7.com

lslpp -L devices.pci.df1000f*

 

• Disk type/info

lsattr -El hdisk

lsattr -El vpath

 

 

• List Powerpath info

Software

lslpp -l | grep -i powerpath

 

Devices

lsdev -C -t power

 

WWN

Loop for each adapter

lscfg -vl fcs0 | grep Z8

 

• IBM Part numbers

To determine the part numbers for the HBAs

lscfg -vpl fcs0 | grep “Part Number”

To corrolate the Part Number from above to a “Feature Code”

that EMC uses in their documentation OR to a HBA vendor model.

 

Emulex LP10000  FC 5716

80P4543 (FRU 80P4544)

80P4544

Emulex LP9002   FC 6228

00P2995 (FRU 00P2996)

00P2996

00P4494 (FRU 00P4495)

00P4495

03N2452 (FRU 09P0102)

09P0102

09P5079 (FRU 09P5080)

09P5080

80P4381 ???

80P4384

Emulex LP9802   FC 6239

00P4295

00P4297

Emulex LP7000   FC 6227

00P1882

03N4167

09P1162

09P1173

09P4038

24L0023

 

INITIAL load software, drivers and firmware

NOTE: You may need to modify lines for the APPROPRIATE version

mount nimserver:/depot/Software/UNIX /mnt

Drivers

See if they are already loaded

lslpp -L devices.pci.df1000f*

 

If not, then

cd /mnt/aix/aix52_apar_IY56722

installp -a -Q -d . devices.pci.df1000fa.rte

smit install_software

use dot (.) for directory

select the following:

devices.pci.df1000fa

devices.pci.df1000f7

• Multipath software

AIX

lsvpcfg

lspv

 

• AIX SDD

datapath query version

datapath query adapter

datapath query device

datapath query essmap

datapath query wwpn

datapath query portmap

datapath query adaptstats <n>

datapath query devstats <n>

 

• EMC Symmetrix software

cd /mnt/EMC/Drivers/AIX

installp -qa -d ./EMC.AIX.5.2.0.0  EMC.Symmetrix.aix.rte EMC.Symmetrix.fcp.rte

• EMC ECC package

cd /mnt/EMC/ECC/Control_Center_5.2

./install_master.sh ‘pwd‘

/opt/ecc

ECC-servername

default

default

Y

/opt/ecc/exec/start_master.csh

 

echo “ecc:2:once:/opt/ecc/exec/start_master.csh > /dev/console 2>&1 Start ECCagent” >> /etc/inittab

 

EMC PowerPath

Must have a valid PowerPath license for this host

cd /mnt/EMC/PowerPath/PowerPath4.4/aix

installp -qa -d ./EMCpower_install EMCpower

/usr/sbin/emcpreg -install

/usr/sbin/powermt check_registration

View SDD

lsvpcfg

datapath query adapter

datapath query device

5.X and greater

iostat -m

 

• Initialize HBAs in ODM

diag

Advanced Diag Routines

System Verification

(select each fcsX => F7)

loopback plug? no

F10

• Configure disks

cfgmgr -vl fcs0

cfgmgr -vl fcs1

 

EMC Symmetrix software

cfgmgr -v

/usr/lpp/EMC/Symmetrix/bin/emc_cfgmgr

/usr/sbin/powermt config

/usr/sbin/powermt save

 

• SDD

config HBA into SDD

/usr/sbin/allpaths

 

config Paths via HBA

cfgmgr -l dpo

**or**

/usr/lib/methods/cfallvpath -2

 

lsdev -Cc disk | grep -i emc

lspv

 

if this is the DR node, it needs to see failover disk’s serial numbers

so in can import the volumegroup

for DISK in $(lspv | grep hdiskpower | awk ‘{print $1}’)

do

echo $DISK

chdev -l $DISK -a pv=yes

done

 

What RAID level is best for my application?

Redundant Array of Independent Disks (RAID) describes a storage solution in which part of the storage capacity stores redundant information about user data stored on the remainder of the storage capacity. The redundant information enables regeneration of user data if one of the disk drives in the array  failes.

RAID relies on a series of configurations, called levels, to determine how user and redundancy

data is written and retrieved from the drives. RAID levels 1, 3, and 5 write redundancy data to the drive media for fault tolerance. The redundancy data might be a copy of the data (mirrored) or an error-correcting code derived from the data. If a drive fails, the redundancy data can be used to quickly reconstruct information on a replacement drive.

RAID Level Performance and Data Protection Summary

RAID Level and Data Protection Available
5
RAID level 5 is best for multiuser environments, such as database or file system storage, where typical I/O size is small, and there is a high proportion of read activity. RAID 5 is the most cost-effective RAID level that provides redundancy protection.
If a single drive fails in a RAID 5 array, all associated      become degraded but the redundant information enables the data to be accessed. If two or more drives fail in a RAID 5 array, all associated logical drives fail, and all data will be lost.
3
RAID level 3 is best for large data transfers in applications such as multimedia or medical imaging that write and read large sequential chunks of data. This RAID level offers redundancy protection, but any two drive failure in the same array results in data loss.
If a single drive fails in a RAID 3 array, all associated logical drives become degraded, but the redundant information enables the data to be accessed. If two or more drives fail in a RAID 3 array, all associated logical drives fail, and all data is lost.
1
RAID level 1 offers high performance and maintains identical copies of data on drive pairs, also known as mirrored pairs. Half of the drives are available for storing user data. Drive pair failure causes data loss. The strength of this RAID level is high safety and data availability. Loss of a mirrored pair of drives results in data loss.
If a single drive fails in a      array, all associated logical drives become degraded, but the mirror drive allows the data to be accessed. RAID 1 can survive multiple drive failures as long as no more than one failure exists per mirrored pair.
If a drive-pair fails in a RAID 1 array, then all associated logical drives fail, and all data is lost.
0
RAID level 0 offers high performance, but it does not provide any data redundancy.
If a single drive fails in a RAID 0 array, then all associated logical drives fail, and all data is lost.

Powerpath and AIX

INSTALLATION:
Prereq Files:
EMCpower.base 3.0.0.0
EMCpower.multi_path_clariio 3.0.0.0
EMCpower.multi_path 3.0.0.0
EMCpower.consistency_grp 3.0.0.0

Register PowerPath on the host: emcpreg -install (Enter the 12-character license key)

Validate your registration number using :powermt check_registration

To initialize devices at the command line :powermt config

Commands:

# emc_cfgmgr
# powermt display dev=all
# powermt config
# powermt display
# powermt remove hba=hba#
# powermt restore
# powermt check

How to uninstall –

Stop all applications/processes that access powerpath hdiskpower device.
Varyoff volumegroups that contains hdiskpower device (or) remove hdiskpower devices from the volumegroups
Uninstall powerpath filesets using or use SMIT
# installp –u EMCpower

To remove powerpath from the system entirely, remove the file “/etc/emcp_registration”

How to Upgrade HBA Firmware: fastT AIX

1.     determine the FRU Number and Microcode level of the adapter installed in the target system lsdev -C | grep “fcs”

2.     check the current microcode level on the adapter – lscfg -vl fcsX OR lsmcocde -d fcsX

        Device Specific.(Z9)……..CS3.91X4  ß Firmware level

3.     Download RPM package for AIX systems.

•  
  • Make a directory on an AIX system to receive theRPM format file. Enter: mkdir /etc/microcode  

  • Transfer the RPM format file to a temporarydirectory on the target server. You’ll see the filename ispci.df1080f9-1-91×4.aix.rpm
  • Unpack the file by executing the instructionsbelow:Enter the commands: If the firmware unpacks successfully, the microcodefiles will be added to /etc/microcode/ 
  • rpm -ihv –ignoreos

    pci.df1080f9-1-91×4.aix.rpm

     

   

4.     Alternately following command can be used – diag –d fcs1 –T download

smitty-like screen within diag and prompts should be something like this

[Inside Diag Screen]

 NOTICE – lets you know your updating microcode.  Hit ENTER

Warning that the current microcode isn’t in /etc/microcode.   Hit ENTER

Screen telling the current microcode and what the new one will be.

Hit ENTER   — this will cause the microcode to be updated.  May cause a system hang for 30 seconds to a minute or 2.

 

5.      Backout: Listed in the online documents is this command for backing out the HBA microcode update.  The command is:

            diag -d fcsX -T “download -f -l previous”