"Hardware not supported"

When installing Cisco Unified Communication products (CUCM, CUPS, CER, UC, CUMA, etc.), you might get a message saying that the hardware is not supported. It's a pain in the butt. Espeically when you (or you client) spent quite a few $$$ to get a brand new server and yielded "not supported".

History

Cisco itself does not manufacture servers (not until lately, with introducing of 'Unified Computing'). Cisco OEM servers (x86) from IBM, HP and Dell and brand it as "Cisco MCS" (Media Convergence Server). Cisco also labels those servers with it's own model number. For example, MCS-7845-H2 is actually a HP DL380 G5 server.

Cisco recommends customers purchase "Cisco MCS" server for Unified Communication products. The major advantage of that is it guarantees the compatibility between hardware and software. For example, you may find the CUCM compatibility matrix here: http://www.cisco.com/en/US/prod/collateral/voicesw/ps6790/ps5748/ps378/prod_brochure0900aecd8062a4f9.html


The mess

Even though Cisco recommends customers purchase MCS servers, it does not prohibit people from buying "equivalence" from manufacturers directly (ie. from IBM/HP/Dell).

If you decided to buy "equivalence", be careful, Cisco has very strict requirements on that. If you didn't order the right parts, the server could yield "not supported" (ie. the software install will fail).

What does it look for?

When the software being installed, it usually looks for the following attributes on the system:
1) Machine type (model number in BIOS)
2) Hard drive and RAID card
3) CPU speed
4) Memory

Frequently seen issues:

#1 You have an MCS I-series ("I" stands for IBM) server from Cisco. One day, the motherboard burnt out. IBM replaced the motherboard (yes, it's IBM who services the server, though you bought it from Cisco).

You tried to reinstall CUCM 6.1.2, but it kept saying the server was not supported.

Cause of the problem:
In IBM BIOS, there's a field called "Machine Type". The generic IBM machine type is different with Cisco MCS machine type.

Solution:
1) Obtain a BIOS update disk for the server from here (2000.4.4 supported version 1.14, this link is for 1.17, please note that 1.17 has NOT been tested): http://www-304.ibm.com/jct01004c/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-57074&brandind=5000008

2) Boot to the disk and flash the BIOS. During the BIOS flash, you should receive a prompt as to whether or not you would like to change the MTM. Please select yes, and enter the correct machine type (e.g. 884xxxx)

Note: the above should be performed by a Cisco TAC engineer.

#2 You have an MCS H-series ("H" stands for HP) server from Cisco. One day, the motherboard burnt out. HP replaced the motherboard (yes, it's HP who services the server, though you bought it from Cisco).

You tried to reinstall CUPS 6.0.4, but it kept saying the server was not supported.

Cause of the problem:
Your old motherboard was with a CPU at speed of 2.13 Ghz. Since the 2.13 Ghz CPU was end of life, HP gave you a 2.8 Ghz CPU and thought you'd be happy with that.

Resolution:
Though you might be happy with the faster CPU, the software was not happy at all. Based on the machine type, the software expect a 2.13Ghz CPU.

#3 You ordered a server from HP, you made sure the CPU, memory, hard disks meet the Cisco requirements. But the software stills said "hardware not supported".

Root cause of the problem:
You forgot to order a "PCI-X/E Mixed Riser Option" from HP.

How do we find the "equivalence" of a MCS server (so you can order it from HP/IBM)?

Step 1: Go to product compatibility page and find a supported MCS server. e.g. for CUPS 6, go to http://www.cisco.com/en/US/docs/voice_ip_comm/cups/7_0/english/compatibility/cupcompatibility.html#wp77512. Let say, you picked MCS-7825-I3-IPC1, which is good for CUPS 6.0.4 new install.

Step 2: Go to this page and click on IBM or HP link (http://www.cisco.com/en/US/products/hw/voiceapp/ps378/product_solution_overview_list.html). MCS-7825-I3-IPC1 is an IBM server. So you would click on the IBM link.

Step 3: Search for the 2nd and 3rd part of the MCS model number. e.g. 7825-I3 in this case. You'll find something like this:

IBM x3250 with Intel 3050 Xeon 2.13-GHz Processor
"... It is the configuration equivalent of the Cisco MCS 7825-I3"

And you'll find the parts list for it:

Table 19. Non-Country Specific Hardware for IBM x3250 with Intel 3050 Xeon 2.13-GHz Processor*

Quantity	IBM Type-Model Feature	Description
1	4364-AC1** or 4365-AC1	IBM System x3250
1	0992	CPU Retention Module
1	1128	x3250 Revision 1 System Planar
1	1272	Dual Core Intel Xeon 3050 (2.13 GHz / 2M L2)
2	1903	1-GB DDR2 667 SDRAM DIMM Memory
1	2007	BIOS GBM
1	2046	Front Bezel
1	2088	3.5inch DASD Cage
2	2091	SATA Filler 3.5inch
1	2268	Base Hardware
1	4144	CDRW/DVD Combo V UltraBay Enhanced
1	4256	Rack Mount Kit
1	4367	Simple Swap SATA RAID Kit
2	5291	160GB 7200 RPM 3.5 inch Simple Swap SATA HDD
1	9011	Internal RAID - Cabled only - setup by Customer

You'll have to order each piece on the list.

If we got a "hardware not supported" message, how do we know which part is not meeting the requirements?

To see the reason of the failure, you need to review the /tmp/hw_validation_err file on the hard drive. You may press ALT+F2 while the installation is in progress (before it gets to halt state). Then you'll get to the Linux command prompt. Type the command below to display the content of the file:

cat /tmp/hw_validation_err

Other useful files include: /tmp/hw_info, /tmp/anaconda.log and /tmp/install.log.

Licensing

Licensing is another mysterious area in Unified Communication. Each product has its own licensing model.

CUCM (CallManager)

CUCM license resides on publisher. The 'host ID' in license file needs to match the MAC address of the publisher.

There are three different types of license for CUCM: node license, feature license and device license (DLU). Node license is for server node. Feature license is required for upgrade or CUCM 7 and above. Device license is required to configure devices (such as phones).

CER (Emergency Responder or e911)

CER license resides on publisher and subscriber (if you have CER subscriber). License file on pub needs to match the MAC address of pub. License file on sub needs to match the MAC address of sub.

If you received a "license expired" message on a two-node CER deployment, 99% of the chance you don't have a license for sub (wrong MAC address).

CUPS (Presence)

Like CUCM, CUPS also has node license and device license. Node license resides on CUPS publisher. Device license resides on CUCM (it's just a regular DLU license).

Node license could contain proxy or PE (Presence Engine) or both.

CCIE Voice Lab v3

Cisco is going to change the CCIE Voice Lab in mid-July 2009. Major changes are:

1) Remove analog devices (such as VG248, ATA)
2) Remove CatOS (Catalyst 65xx)
3) Replace CCM with CUCM 7 (Linux Appliance)
4) Replace Unity with Unity Connection 7 (Linux Appliance)
5) Add CUPS 7 (Linux Appliance)
6) Add SIP phones

IPExpert (a training company) provides some practice labs. The topology would be like below:



Here are some tips if you're going to build your own lab.

PSTN-WAN simulator

As shown in the diagram above, we have a PSTN cloud and a Frame-Relay cloud. We can use a single router to simulate that. And this router could also be the terminal server.

I would choose a Cisco 2811 router with the following modules:

PVDM2-16 (or other PDVM resource).
HWIC-4T : 4-port Serial module for Frame-Relay simulation.
VWIC2-1MFT-T1/E1 : 2-port T1/E1 module for PSTN simulation. You need at least three ports. So you may order two of these. Or to save some money, order one 2-port module and one 1-port module
HWIC-8A & CAB-OCTAL-ASYNC : 8-port async module for terminal service

Don't forget the female serial cable (CAB-SS-V35MT). You need at least three.

On HQ, BR1, and BR2 Gateways:

PVDM2-8 (or other PDVM2)
HWIC-1T & CAB-SS-V35MT
VWIC2-1MFT-T1/E1

On BR1 and BR2 Gateways
HWIC-4ESW-POE : Ether net module to power up the IP phones.

On BR2 Gateway
NM-CUE : Unity Express Module


Switch

Any Cisco IOS switch that supports PoE.

Servers

I recommend using virtual machine(VM) for the lab:

1. If you don't use VM, you're going to need the expensive Cisco/HP/IBM server to install CUCM, UC, CUPS.
2. If you don't use VM, you're going to need license files for CUCM, UC, CUPS.
3. If you don't use VM, you're going to need 5 servers and a couple of workstations (to simulate soft phone and run CUPC/CAD)
4. With VM, you can easily clone VMs, which saves you lots of time.

I myself use VMWare. I have no experience on Microsoft Virtual PC (HyperV).

Here are some caveats you need to know about VMWare:

1) You may either use "VMWare Server" (a.k.a. GSX) or "VMWare Infrastructure") (a.k.a. ESX)
2) GSX is free while ESX is commercial license
3) VMs on GSX has issues with NTP. This is a known issue.
4) ESX does not support audio device. Your VM might need an audio device to launch soft phone (such as CIPC or IP Blue). Try google "virtual audio cable" and you should find a solution.

Following is my lab set up with four 2811 routers, one 3560 switch, one PC (8G RAM, 750G HDD):


When creating VM, make sure you choose RHEL 4(RedHat), 32-bit, one CPU. For CUCM 7, you need at least 1G RAM and 60G hard drive. For CUPS 7, you need 1296M RAM and 75G hard drive.

Change Notification

On Cisco Unified Communication products, there's a terminology called "Change Notification".

What is "Change Notification"? And how does it affect products' functionality?

Typical symptom of "Change Notification Issue" is: the changes you made didn't seem to take effect, including:

#1 You add a new DN (Directory Number) to CUCM. But you got fast busy when calling that DN.
#2 You try to reset the phone from GUI. But the phone didn't get reset.
#3 You updated the information on CUCM (such as PIN#, line association, etc.). But CUPS didn't get those changes.
...

"Change Notification" is one component notify another component that changes have been made. The other component should react to this notification.

For example, when you add a new DN to CUCM. The changes were made to database. Database component should send notification to the call routing component. So the call routing function could work properly (calls can be routed to the new DN).

If change notification was failed (or delayed), the call routing component will use outdated information to route calls. Thus calls to new DN would failed.

How do you know if there's a change notification issue? If the components are on the same box, you may use the command "show tech notify"

An example output would be link below:

32 I 0 P 118 H 118 T 118 S 80 ccm
...
38 I 0 P 119 H 119 T 119 S 27 EPAS_SyncAgent[10.88.229.209]:32958

The line that begin with numbers (32, 38, ...etc.) represent clients (applications) subscribed to change notify

"32" means it is the 32 client slot on this server.
"I 0" means there are 0 messages in shared memory to be processed by this client.
"P 118" means 118 messages have been processed.
"H 118" means the "Head" message position is 118.
"T 118" means the "Tail" message was in position 118.
This is the optimal situation: 118-118=0 (nothing to be processed).

“S 80 ccm” means there are 80 tables subscribed by this client and the client name is ccm (callprocssing).

You may also use RTMT to see the CN (Change Notification) queued. If you saw non-zero value in queue, either the server was busy, or you got CN issues. Restarting corresponding process (such as ccm) normally would clean up the queue and solve the problem.


Change notification across different boxes is a little bit complex. CN was sent via IPSEC tunnel in this scenario. IPSEC was controlled by Cluster Manager. Trust relationship was established during installation (when you add a box into cluster).

In order to add a new server into a cluster, two requirements have to be met:
1) The hostname of the new box needs to be presented in the Publisher's server list or application list. (this can be done manually or automatically depends on the server you try to add).
2) You need to know the cluster "secret password". The password needs to be entered during installation (of the new box).

If either one was changed after installation, the trust would be broken. You can verify that by looking at "Cluster Manager" logs.

If trust was broken, change notification won't work. For example, changes made on CUCM didn't populate to CUPS. Restart Sync Agent would force a synchronization, which is not affected by change notification or IPSEC.