UC Corner

CET Tool for UCCX

2011-09-27T06:15:00.000-07:00

Back in the age of Windows-based CallManager (4.x and before), UCCX (a.k.a CRS) stores its configuration in CallManager DC Directory.

When CallManager moves to Linux based (CUCM 5.0 and above), it removes the concept of DC directory. Thus UCCX has to store its configuration on UCCX server in XML files.

Cisco provides a tool called CET (Configuration Editing Tool?) to edit those configurations. The tool is supposed to be used by Cisco TAC only.

For Windows-based UCCX (7.x and before), you may find the CET tool on UCCX server C:\program files\wfavvid\cet.bat. Just the the cet.bat file.

For Linux-based UCCX (8.0 and above), you may find the CET installer on the UCCX installation DVD (\Installer\CetTool\CetTool.exe). Or download from an already-installed UCCX server http://142.100.64.14/uccxinstalls/CetTool.exe (case-sensitive). You'll have to install it on a Windows workstation.

You may use CET tool to modify the configuration of UCCX, such as removing license file or reset the system back to before-initialization state (for password recovery purpose). Please note that on Linux-based UCCX, you'll need a root credential to use CET tool. Either get it from Cisco TAC or follow instructions here.

You may also modify the configuration without CET tool. But it requires some reserve engineering.

For example, you want to set the UCCX 8.5 back to FRESH INSTALL state. You may edit the XML file in /opt/cisco/uccx/ClusterData/default/com.cisco.crs.cluster.config.AppAdminSetupConfig. Look at the blobValue. It's encoded in ASCII.

Translate that with the ASCII table:

You'll get:

46 = F
52 = R
45 = E
53 = S
48 = H
5F = _
49 = I
4E = N
53 = S
54 = T
41 = A
4C = L
4C = L

If you want to set the value to a specific string, you'll translate the string into ASCII code and put it in blobValue.

Modify License MAC

2011-09-25T18:15:00.000-07:00

Cisco used to bind license file to physical MAC address. Now, it moves to "License MAC", which is a hash value of multiple system parameters such as NIC speed, NTP, DNS, hostname, etc.

To view the license MAC, you need to install the system first (CUCM, UCCX, CUPS, etc.). Then use the "show status" command.

This is somehow inconvenient:

1) You cannot get the license file before finishing the installation.

I personally prefer to get everything ready before starting the installation, such as IP address, hostname, password, license file, installation media, etc. You could run into lots of surprises when trying to get the license file.

2) When the system parameter was changed, the license file yield invalid.

For example, you add/change DNS server settings on the system, which is pretty common during system integration.

It would be better if you could dictate what license MAC the system use. You may also use some schema like: AABBCCDDEEFF, where AA is the product code, BB is the site code, CC is the node number, DD is the version number, etc.

In the example above, the "License MAC" was changed to "abcdef123456".

Since you can use whatever License MAC you like, you may:

Get the license file before the system was installed
Keep using the same license file after system parameters was changed (such as DNS)

To specify the license MAC, you need to have root access to the system. Then you'll modify the file /usr/local/bin/base_scripts/LicenseMac.sh. To the bottom of the file, replace the line

FinalString=`expr substr "$SHA1sum" 1 12`

FinalString="abcdef123456"

In newer versions, you might have to change the /etc/selinux/config file so that selinux runs in permissive mode.

Reminder: Don't be evil. ;)

Database Access

2011-08-31T10:54:00.000-07:00

Many UC appliances (like CUCM, CUPS, etc.) use database to store configuration. For security and supportability reason, the regular CLI provides limited access to database. However, if you could get root access to the box, you would have full access.

su - informix

Change user role to 'informix' which is a built in Linux user to access the Informix database.

dbaccess $(dblutil -c) -

dbaccess is a SQL client to access Informix database. 'dblutil' is a DB role.

select * from licenseinfo

A standard SQL command to view all records in 'licenseinfo' table.

Notes:

Press Ctrl-D to execute SQL commands
Press Ctrl-C to exit
On some terminals, you may have to press Delete to Backspace

If you want to know the relationship between different database tables, refer to "Data Dictionary" on CCO Docs.

To list all tables:

select tabname from systables where tabid > 99 and tabtype = "T"

Grab a some books like "SQL for Dummies" and test in your lab.

I wondered what does this do? ;)

Root Access on Linux-based UC appliances

2011-08-30T21:12:00.000-07:00

There are many posts on Internet teaching you how to get root access on CUCM. This is not a secret. Since CUCM is Linux-based, the method is pretty straight forward - use a Linux boot CD to boot into rescue mode and modify the relevant files. Here's a simple walk through.

Assuming CUCM was already installed. Boot the box with a Linux installation CD (e.g. RedHat). Type "linux rescue" in the boot prompt.

Chose language. Default is 'English':

Choose keyboard. Default is 'US':

We don't need to set up network. Thus choose 'No' here.

Choose "Continue" to mount the CUCM file system.

The following message is telling you that the CUCM file system has been mounted under /mnt/sysimage. If you want to map the root directory to the CUCM file system (which is recommended), you may use command "chroot /mnt/sysimage".

Below are the commands and explanations.

chroot /mnt/sysimage

This is to map the root directory to CUCM file system.

cd /etc

Change the working directory to /etc, where most of the system configuration files are stored.

chattr -i passwd

chattr -i shadow
rm securetty 

Remove file protection on files "passwd" and "shadow", which stores user info and passwords. Remove file "securetty" to allow remote connections with root.

passwd root

Reset (change) password for the root user. Type a password that is easy for you to remember. Retype it to confirm. If the password was changed successfully, you'll see the prompt "passwd: all authentication token updated successfully".

Notes:

If you typed a simple password, you might get a warning like "BAD PASSWORD: it is based on a dictionary word". Just ignore it and retype to confirm.
There's no screen display for the password you're typing. Type carefully.

The following steps require some basic knowledge of the vi editor. If you're not familiar with vi, please search Internet for vi commands help.

vi passwd

Change the passwd file so the root user has a shell (command line interpreter) to use. Use vi commands. Change the line

Save and exit file.

For those who are not familiar with vi, here are the command sequence (case-sensitive):

Type /s to search for character 's'
Type D to delete to the end of line
Type A to enter append mode
Type bin/bash to set the shell
Press ESC key (it's a key on the upper-left corner of your keyboard) to exit append mode
Type :wq to save and exit file.

vi ssh/sshd_config

Change the sshd_config file so you can SSH as root (it's disabled by default). Use vi commands. Change the line

Save and exit file.

For those who are not familiar with vi, here are the command sequence (case-sensitive):

Type /Per to search for the word begins with 'Per'
Type X to delete the letter on the left (which is '#' in this case)
Type :wq! to save and exit this read-only file

Now type exit command twice to reboot the system.

Use a SSH client (such as putty) to test. You should be able to SSH into CUCM with root account.

This method applies to all Linux-based appliances such as Unity Connection, CUPS, CER, UCCX (Linux version), etc.

Where's my Unity Connection option?

2011-08-30T12:45:00.000-07:00

I was trying to install the UC 8.6.1 suite on my VMware ESXi. I'm pretty savvy at VMware and UC, thus I didn't bother to download the OVA template from Cisco. I created a VM with 75G hard drive and 2G RAM.

I had no problem getting UCM installed. However, when it came to Unity Connection, I realized that the option was not presented on screen like it did in previous versions (see below).

Maybe there's some magic in the OVA template. So let's download it.

Open up the OVA file with Notepad. You'll see that the minimum hardware requirement is 4G RAM and 160G hard disk.

If you're just testing it in the lab and want to save some hard disk space, you may choose "Thin Provision" while deploying the OVA. VMware will dynamically allocate space as needed up to 160G. ie. if the initial install occupies 75G, it'll only takes 75G physical space (even though the hard disk "looks like" 160G). This is not recommended for production environment for performance consideration.

But if you're the kind of people that like to control everything, this is pretty annoying. Why can't it just install on a 75G disk? Ya, why not? Let's hack it.

On the installation disc, look for the folder "Cisco/Install/conf".

Open the file "callmanager_product.conf" and search for "Unity Connection VMware rule". A couple lines below, you'll see the line "NOT, VMware, *, *, *, *, *, *, *, *". Change it to "VAL, VMware, *, *, *, *, *, *, *, *". This will allow any virtual machine specification.

You may do the same if you want to use a physical server. For example, if you have an old 7825H server and want to install Unity Connection 8.6.1 on it, just change the line "NOT, 7825H, *, *, *, *, *, *, *, *" to "VAL, 7825H, *, *, *, *, *, *, *, *".

You may also change the sections for CUCM and CUCM BE so they have less restricted requirements.

Save the file to the disc (image). Boot from it. Now you see the Unity Connection is available as shown below.

By the way, UC 8.6 supports VMTools (finally). However the VMTools bundled with the install might not be up to date.

From CUCM CLI:

You may update the version from VM client.

The installation will take a while. During install, you'll see the VMTools status as "not running". This is normal.

When the install is completed, you'll see the version is updated.

From CUCM CLI:

Some tools for home lab

2011-07-21T12:44:00.000-07:00

1) Pegboard - A board with holes so you can hang things on it.
So you can hang IP phones like this:

2) Remote Power Switch - allows you to power on/off device remotely through network.

I got NPS 115 from eBay for about $40. I used it a lot when I'm away from home. So I can practice from office or hotel room.
Please note that this one has 8 outlets and allows you to power on/off devices (routers/switch) individually. Some cheaper ones (like $20) have only one outlet. You have to power everything on or off at once.

Voice IE: Mysterious phone registration issue

2011-07-02T07:37:00.000-07:00

It's a hub-and-spoke topology - CUCM was at the HQ site while two branches BR1 and BR2 are connected to HQ router via Frame-Relay.

One of the phones on BR2 was not able to register to CUCM. The phone's screen displays "Registering".

Since the other phone on BR2 was able to registered, I thought the problem was specific to the phone. Thus I checked the phone configuration, reset the phone, restore it to factory default, re-located it to HQ site, etc.

The phone was able to register while it's on HQ. But it doesn't work while it's on BR2. So it seems to be site specific. However, if it's site specific, why the other phone on the same site could register?

Since the network path was "phone -> BR2 router -> Frame Relay -> HQ router -> CUCM", I focused on every elements on the path. The phone was able to get the correct IP config (subnet mask, default gw, TFTP) from DHCP. I wiped out the router configuration and reconfigure it. I deleted the phone from CUCM and re-added it. Reloaded CUCM and routers. Still no avail.

I pinged the BR2 phones from CUCM CLI. I can ping phone1 but not phone2. Then I realized it HAS TO be the network. But how come? For two hosts in the same subnet, if I could ping one but not the other, it HAS TO be the host's issue. But I already proved phone2 was working fine when it's on HQ site.

With the help of "debug ip packets detail", I found out that HQ router chose different paths for phone1 and phone2. But I didn't configure host routes. How could this happen?

"show ip route" discovered that BR1 was advertising same route as BR2. They both claimed to have the BR2 phones' subnet. Since they are equal cost routes, HQ router will load-balance the two routes (uses BR2 to reach phone1, then uses BR1 to reach phone2). That's why phone1 is always reachable while phone2 is always unreachable.

By reviewing BR2 router's config, I found out that I fat-fingered the IP address for the data VLAN (with the address of BR2 voice subnet).

This kind of mistake could easily cost you couple hours in the lab unless you had experienced it before and have some routing knowledge.

What is XMPP and why it is important in Unified Communication

2010-07-19T20:48:00.000-07:00

To get a complete understand of XMPP, I recommend the book "XMPP: The Definitive Guide".

In short, XMPP is a protocol (like SIP). XMPP was majorly used for presence and Instant Messaging. But it is being expanded to other areas like call signaling control.

XMPP is getting more and more popular in Unified Communication products like CUPS (Cisco Unified Presence), Cisco Quad (SharePoint-like product), Contact Center, etc.

Since it's a direct competitor with SIP, frequently asked questions would be:

Shall I learn XMPP?
Will SIP be replaced by XMPP?

For the first question, Yes. You definitely should learn XMPP because

XMPP was built from ground to support presence and IM
XMPP is "eXtensible"
XMPP is being (or has been) adopted by many big vendors like Cisco, Microsoft, Google, etc.

For the 2nd question, it really depends. On call control and telco interfacing, SIP will still dominate for quite a while because the install base and impact of change.

Anyway, you should take an hour or two to understand XMPP so you can understand what it can do (or do better) on different systems.

Happy birthday, CUPC 8!

2010-07-01T07:11:00.000-07:00

Two months after CUPS 8 was released, Cisco finally releases the next generation of communication client - Cisco Unified Personal Communicator 8.

With XMPP, now CUPC can do:

Group Chat
Persistent Chat Room
Chat History
Message Archiving and Auditing
Federation with external IM systems (like GoogleTalk, Sametime, Webex, OCS)
Integrated HD Video with soft phone or hard phone

Happy Birthday! :)

See ya in Las Vegas

2010-06-26T11:20:00.001-07:00

I'll be leaving for Las Vegas Sunday afternoon.

During Cisco Live (Networkers), I'll be in the Technical Solution Clinic (Unified Communications). I'll attend the CCIE party at VooDoo Lounge Tuesday (June 29).

See you guys there.

By the way, I'll bring two copies of my "Deploying Cisco Unified Presence" book. For those who need it, just drop by and take it. :)

7/1/2010 13:30 PDT - Whoever brought me a dessert at Technical Solution Clinic, thank you very much! I really enjoyed it. (though I don't know who it was) :)

Michael

Unified Communications Manager (CallManager) Upgrade Tool

2010-06-16T18:11:00.000-07:00

It seems that when planning a Cisco Unified Communications Manager (CUCM) upgrade, the amount of releases are sometimes dizzying! Can I direct upgrade or not? Do I need to PUT (product upgrade tool) for a license and CD? How do I match the release ID with the CUCM version ID (7.1(3a) vs. 7.1.3.20000-2). I know that pain and Cisco has created a tool to help us with the process. Check out the link here.

With this tool, you can input your version (supports only 5.1(3)+) and which version you want to migrate to. It will let you know if you can upgrade directly and what the licensing implications are. As a customer looking at upgrading to 8.x, this tool is a terrific resource for upgrade planning. Check it out!

-MW

Test mobility feature without PSTN

2010-05-12T09:12:00.000-07:00

The core of the mobility feature is "Remote Destination". Per SRND http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/srnd/7x/mobilapp.html#wp1043943,

"Remote destinations must be Time Division Multiplex (TDM) devices or off-system IP phones on other clusters or systems. You cannot configure IP phones within the same Unified CM cluster as remote destinations. "

So if you'd like to test mobility feature without involving PSTN gateways, you may build another CUCM cluster (with VMWare, it should be just another VM). Then you may create ICT (Inter-Cluster Trunk) between two clusters. You may use the IP phones on another clusters as "Remote Destination". With this setup, you may test the following features:

1) Mobile Connect (SNR)
2) Enterprise Feature Access (EFA)

You won't be able to test the following feature:

MVA (Mobile Voice Access), because it requires IVR function of the voice gateway.

If you have a voice gateway but does not have T1/PRI, you may use CME to simulate PSTN phones.

Make a non-bootable ISO image bootable

2010-04-22T06:55:00.000-07:00

For whatever reason, Cisco only post "non-bootable" ISO images on CCO for download. In some urgent situations, you might need a bootable disc to recover the system (or your client/boss would shoot you in the head). Here's the procedure to make a non-bootable ISO image bootable.

Before you continue, be aware that this procedure is NOT approved by Cisco. Neither Cisco nor I will be responsible for any loss caused by this.

Any bootable disc has to follow "El Torito" specification. No exception for Cisco discs. The only difference between a bootable disc and non-bootable disc is the "boot sector". Thus the solution is very simple - extract the boot sector from a bootable disc and inject it into a non-bootable disc.

The boot sector is a very small file (usually less than 10k). And the boot sector is usually content independent (i.e. you may extract the boot sector from CUCM 7.1.3 and inject it into 7.1.5). You may save the boot sector on your USB thumb drive and keep it handy.

To extract/inject the boot sector, you need some disc image tools like UltraISO. (You may also use other ISO tools with similar features)

Step 1: Extract the boot sector

Put a bootable CUCM disc into the DVD drive and launch UltraISO. Go to menu "Bootable > Extract Boot File from CD/DVD..."

Save the file to your hard drive as a "boot info file" (bif). In our example, we call it "boot.bif"

Step 2: Inject the boot sector

Open the non-bootable image in UltraISO. Go to menu "Bootable". Make sure "Generate Bootinfotable" was checked. Then choose "Load Boot File...".

Choose the boot file we saved before (boot.bif).

Note that the image type changed to "Bootable".

Now, you may go to "File > Save As" to save the bootable image to an ISO file. Then you may burn the ISO to a disc with your favorite disc burner software.

New book and Cisco Live

2010-04-06T18:03:00.000-07:00

New edition of the book has been published. Use coupon code "SHOWERS" to get 10% off. Use coupon "FREEMAIL305" to save on shipping. (Offer ends April 30, 2010)

It covers new topics like: XMPP, Jabber, CSF (Client Service Framework), Message Archiver, Persistent Chat, 3rd-Party Compliance, external database, etc.

If things worked out, I'll go to Cisco Live (Networkers) at Las Vegas from Jun 28 to July 1. I'll be at the Technical Solution Clinics. See you there.

Place to ask questions

2010-03-26T17:20:00.000-07:00

It's that time again - "Ask the Expert" event on Cisco NetPro forum.

If you have any questions, please feel free to ask there.

This time we'll focus on the next version of CUPS and CUPC, with new technology XMPP and CSF.

You may post your question at https://supportforums.cisco.com/message/3043328.

I'll update my book "Deploying Cisco Unified Presence" to version 8.x in couple weeks.

Client Service Framework - CSF

2010-03-09T05:45:00.000-08:00

History

Once upon a time, Cisco built a client software called "Cisco Unified Personal Communicator" (a.k.a. CUPC). CUPC has many features like Rich Presence, Instant Message, Desk Phone control (CTI), Soft Phone (SIP), Voicemail (IMAP), Web Conference, etc.

Later on, when Cisco built other client software (such as plug-in for Sametime Connect, plug-in for MOC), developers realized that there were many commonalities between those software and CUPC. Instead of "reinventing the wheel", they could just reuse the existing codes in CUPC.

However, "reuse" is not as simple as it sounds to be in software development. You'll have to "shape" the codes to fit into your specific software. To make the work easier, they decided to "extract" the codes from CUPC and standardize it. So any other software teams can easier build their application on top of it. This "extracted", "standardized" component is called "Client Service Framework". (if you have ever heard about ".Net Framework" from Microsoft, the concept is pretty similar).

The idea is: CSF has all the common feature built in and provides unified interfaces to upper applications (such as CUCIMOC, CUPC 8, CUCI Connect, etc.).

The code of CSF is usually referred as "core" (e.g. "core log" of CUCIMOC).

Because of the history, we sometimes see the "marks" of CUPC. For example, CUCIMOC dial rules are put in the "CUPC" folder on TFTP.

Applications using CSF

Right now, applications using CSF are:

CUCIMOC (for Microsoft Office Communicator)
CUCI Connect (for Webex Connect)
CUPC 8.0

Confusions

To use the Cisco Unified Personal Communicator soft phone feature, we have to create a device in CUCM. The device type happened to be called "Cisco Unified Personal Communicator". This is really a BAD idea. If I were the decision maker, I would name it with some netrual name like "Cisco SIP soft phone".

Because the device type has the same name as the client software, many people (including Cisco TAC engineers) thought it was required for CUPC to work. More confusion when it comes to licensing and troubleshooting. Such as:

"I licensed UPC on CUCM > System > Licensing > Capability Assigments. That took 1 DLU. But when I ran the license calculator, the 'Cisco Unified Personal Communicator' is taking 3 DLUs"
"My CUPC is not registering" "Did you mean the CUPC on your desktop(Windows XP)? Or the CUPC on CUCM?"

They made the same mistake again with CSF. To use the soft phone feature of CSF, you have to create a device in CUCM. The device type is called "Client Service Framework". Again, if I were the decision maker, I would name it with a neutral name like "Cisco SIP soft phone (2nd Generation)".

The point is: don't confuse one of the features with the whole piece. You don't need to configure CSF device in CUCM to get the CSF worked on client applications. Without CSF device on CUCM, the only feature you're missing is soft phone.

Interface

As you can guess, CSF does not have a user interface. User interface was provided by the upper layer applications such as CUCIMOC, CUCI Connect, CUPC 8 etc.

The only way you can "see" CSF is to use Windows Task Manager. You'll see a process called "cucsf.exe". This process usually starts up and close down with the upper layer applications (such as CUPC 8). But not every application does the same thing. For example, CUCIMOC does not shutdown CSF when you close MOC. If you want to kill or restart CSF for whatever reasons, you'll have to kill it from Task Manager.

Configuration

In current version of CSF, there's no GUI to configure it. All configuration was done via Windows registry.

Depending on your deployment, configuration data could be in either one of the following locations:

HKEY="HKCU\Software\Cisco Systems, Inc.\Client Services Framework\AdminData"
HKEY="HKCU\Software\Policies\Cisco Systems, Inc.\Client Services Framework\AdminData"

If both keys exist, "HKCU\Software\Policies\Cisco Systems, Inc.\Client Services Framework\AdminData" takes priority.

For centralized management, you may use a logon script (BAT) or a group policy to update the client's registry.

Registry key requirement varies between applications.

For CUCIMOC, registry key info as below: http://www.cisco.com/en/US/partner/docs/voice_ip_comm/cucimoc/7_1/english/integrat/guide/config_clients.html. CUCIMOC also provides a sample BAT file and REG file for your convenience. Those sample files are zipped into cucimoc-Admin-ffr.x-y-z.zip (x-y-z is the version of CUCIMOC. e.g. cucimoc-Admin-ffr.7-1-3.zip)

Limitations

Application Dial Rules (ADR)

Current version of CSF does not have an interface to "dip" into CUCM database to retrieve ADRs. As a workaround, Cisco provides a COP file. Whenever you run(install) the COP file on CUCM, CUCM will generate XML files in TFTP folder to reflect the latest ADRs. Then CSF needs to be restart to download those XML files. For details, see: http://www.cisco.com/en/US/partner/docs/voice_ip_comm/cucimoc/7_1/english/integrat/guide/config_servers.html#wp1054388

This means, whenever you make changes to CUCM > Call Routing > Dial Rules, you'll have to run the COP file and restart CSF.

Soft Phone

CSF soft phone is a SIP phone. It has quite a few limitations comparing with SCCP (skinny) phones. For example, the following features are not supported on SIP:

CUVA support
FAC/CMC support
MLPP target device
Direct transfer
Hold tone

... to be continued

Client Convergence

2010-03-03T06:25:00.000-08:00

In the "Unified Communication" world, you might have many different client software installed on your computer -

Click-to-Call for making calls easy
IP Communicator to be a soft phone
Video Advantage to be a "video add-on"
Personal Communicator to be an IM and presence client

It's kind of "un-unified" with so many different things. Fortunately, your voice was heard. In the future, you might have a truly "unified" client, which does everything with one piece.

Calendar presence with CUPS and Exchange

2010-03-01T05:04:00.000-08:00

Cisco Unified Presence Server (CUPS) has the capability to retrieve calendar (Exchange) status and populate to clients (CUPC). When your calendar status is "busy" in calendar, CUPC will display your status as "in a meeting".

To enable this feature, there are server-side configuration (CUPS/Exchange) and client side configuration(CUPC).

CUPC configuration

On CUPC side, you go to "File > Preferences > Status" and put a check mark on "Show me as 'In a Meeting' whenever my Exchange calendar shows me as busy".

Please note that when multiple 'presence' arrIve, CUPC can only display one of them. The priority, from high to low is: Phone Presence > Calendar Presence > Availability Status.

For example:

You logged into CUPC ==> Availability Status = "Available"
You're on the phone ==> Phone Status = "On the Phone"
Your calendar is busy ==> Calendar Presence = "In a Meeting"

In this scenario, CUPC will display the status as "On the Phone". You won't see "In a Meeting". However, you may move the mouse over a contact (or your self-status) to see all the presence.

In the screenshot below, you'll see the CUPC display its self-status as "On the phone". When you move the mouse over the self-status, you'll see "Online, On the phone, In a meeting" in the tooltip. This is how to explore the "hidden" presence from CUPC.

CUPS configuration

The most confusing (instead of difficult) part on CUPS is the SSL/Certificate configuration. CUPS can only talk to Exchange via HTTPS. Why is that?

That is because CUPS use an Exchange "service account" to read everybody's mailbox (calendar). If the communication was not encrypted (SSL), it would pose a threat to information security. A hacker could easily intercept the packets on network and retrieve everybody's (including CEO's) email and calendar.

With that said, here are the requirements on Exchange and CUPS to set up a SSL connection:
1) HTTPS needs to be enabled for OWA (Outlook Web Access)
2) The CA certificates of Exchange need to be trusted by CUPS.

Regarding #1, there's a catch. We'll discuss that in the "Exchange" section below.

Regarding #2, there are two catches -
A) Don't confuse the CA cert with identity cert.
B) Request address has to match certificate.

A) What is a "CA cert" and what is an "identity cert"?

For example, if your Exchange OWA has a certificate to identify itself as "owa.acme.local". This certificate is called "identity cert". Let say, this identity cert was issued by a CA (Certificate Authority) called "my-ca-server.acme.local". On the CA server, it has a certificate to identify itself. We call that cert a "CA cert".

It is the "CA cert" CUPS server needs to trust. In our example above, it's the "my-ca-server.acme.local" certificate needs to be uploaded to CUPS as "Presence Engine-Trust".

B) When you configure the "Outlook presence gateway" on CUPS, the address you put in there is the address CUPS will use to sent request to Exchange.

For simplicity, quite a few people just put the IP address of Exchange there. This is not going to work with SSL. For security, the request address has to match with the common name in the identity certificate. Usually, the certificate will identity the server in FQDN (e.g. owa.apps.local). Thus, you'll have to configure the same name in presence gateway configuration page.

Please note: I used the word "same name" instead of "FQDN". Because technically, you could configure any name in the certificate. Let's take a look at a couple examples:

Your OWA server IP = 192.168.1.100
Your OWA server FQDN in DNS = owa.apps.local

Example #1: You configured the common name in the OWA cert as "email.apps.local".

This won't work. Because:

If you used "192.168.1.100" in presence gateway configuration, it doesn't match "email.apps.local" in the identify cert.

If you used "owa.apps.local" in presence gateway configuration, it doesn't match "email.apps.local" in the identify cert.

If you used "email.apps.local" in presence gateway configuration, CUPS won't be able to resolve the name, because it's not in DNS.

Or even worse, you don't have DNS configured on CUPS so you can't use any FQDN.

Solution:
1) Add "email.apps.local" to DNS
2) Configure CUPS used DNS (use "set network dns primary" command)
3) Use "email.apps.local" in presence gateway

Example #2

In a lab environment, you may configure the certificate so it identifies the OWA server with common name "192.168.1.100" (though it's not a common practice, it's technically legit).

In this scenario, you may use IP address 192.168.1.100 in presence gateway config.

Whenever you made changes to presence gateway, don't forget to restart Presence Engine service.

Exchange

You don't have to be an Exchange expert. But you should at least know the following:

OWA, WebDAV, Receive-As permission, IIS certificate, CAS/Mailbox roles, authentication methods, Power Shell. (did I say you don't have to be an expert? :) )

OWA

OWA - Outlook Web Access, a web interface to access Exchange emails, calendars, etc. HTTPS needs to be enabled on OWA for CUPS integration to work.

OWA is actually a very useful tool for troubleshooting. For example:

When you tried to access OWA via HTTPS, OWA server will present you a certificate. You may view that certificate from web browser. You should be able to tell what the CA cert is from the web browser.

In the screenshot below, you open a web page to OWA. If you double-click the "lock" icon at the bottom of IE, you'll see a certificate viewer window. From the "certification" tab, you'll see the CA cert is "sametime.apps.local". The identity cert is "owa.apps.local".

Now you know that:
1) You need to upload "sametime.apps.local" cert to CUPS as "Presence Engine - Trust".
2) You need to configure "owa.apps.local" as the "presence gateway" on CUPS > Presence > Presence Gateway.

WebDAV

WebDAV is a protocol (on top of HTTP) to retrieve email/calendar information. See http://en.wikipedia.org/wiki/WebDAV for more details.

CUPS uses WebDAV for calendar integration. Microsoft obsoletes WebDAV on Exchange 2010 and advocates its own protocol EWS (Exchange Web Service). By the time this blog was being written, CUPS doesn't support EWS yet. Which means, CUPS doesn't work with Exchange 2010. I'm not sure if you can have an E2007 CAS in E2010 as a "WebDAV" gateway. I'll test it later.

Though WebDAV sits on HTTP/HTTPS, it's different from OWA. Don't assume WebDAV is working just because OWA was working.

Up to Exchange 2007, WebDAV was enabled by default. You don't have to explicitly enable it. Some people reported that WebDAV was not installed/enabled on E2007 with Win2008. This is not true. The "WebDAV Publishing" in screenshot below has nothing to do with CUPS integration. It's OK to leave it uninstalled.

Since E2007, Microsoft change the OWA URL from /exchange to /owa, but WebDAV URL stays the same (/exchange). This caused more confusions in troubleshooting.

When you troubleshoot CUPS calendar integration, you'll see CUPS always request to URL /exchange, which is the WebDAV URL. This is right and expected.

If you tried to "test" it by typing "https://owa.acme.local/exchange", you'll be redirected to "https://owa.acme.local/owa". This is also right and expected. For more details, please see: http://msexchangeteam.com/archive/2007/02/07/434523.aspx

The point is: there's no easy way to tell if WebDAV is working. But it's easy to tell if WebDAV is NOT working.

If you typed "https://owa.acme.local/exchange" and got some error like "401 unauthorized" or "503 service unavailable", WebDAV is more than likely not working. Though regular OWA might be working fine at this point.

If you got "401 unauthorized", you may take a look at http://msexchangeteam.com/archive/2008/02/01/447989.aspx. Default settings should work. If it doesn't work, most likely your authentication was expecting username in a different format. e.g. "domain\user" instead of "user".

If you got "503 service unavailable" or "500 internal server error", please make sure "ISAPI Extensions" is installed on mailbox server. See screenshot below. This usually happens if mailbox server and CAS are on two different servers.

Certificate

OWA/WebDAV relies on IIS to server HTTP request. It's IIS that presents certificate to SSL client.

By default, when you installed IIS, it'll generate a self-signed certificate. Unfortunately, this self-signed certificate doesn't work with CUPS, because it doesn't contain CA bit in its extension. Below is an example of the certificate that HAS the CA bit:

If your CA cert does not have CA bit, CUPS won't trust it. This is per RFC 2459. Don't blame Cisco on that.

Solution? Get a certificate from a CA (either external or internal) or use "makecert.exe" to create a self-signed cert with CA bit (see my book for details).

Receive-As Permission

In order to see everybody's calendar, CUPS needs an account with this permission. So the FAQ is: what's the minimum permission required?

The minimum permission required is "Receive-As" on the end user's mailbox.

Cisco documents said you need "View Only Administrator". That's not true. All you need is "Receive-As" permission on the end user's mailbox. Not more, not less.

How do we assign "Receive-As" permission to the account? I would recommend use Exchange Management Shell (command line).

Let say, in Active Directory, the service account used by CUPS is called "cupsexch". The end user account is called "JDoe" (full name: John Doe). The following command give "cupsexch" Receive-As permission on John Doe's mailbox:

Add-ADPermission -Identity "John Doe" -User cupsexch -ExtendedRights Receive-As

Of course, you're not going to repeat this command for 1000 users. The better way to do it is to assign permission on a "container" that contains the end users' mailboxes.

In Exchange, the "container" for mailboxes is "Mailbox Database". The container for "Mailbox Databases" is "Storage Group". You may have multiple databases in a storage group. And you may have multiple storage groups in your Exchange environment.

You may assign permission at different levels:

End User Mailbox
Mailbox Database
Storage Group

Since "Storage Group" is the highest level of "container", we usually assign permissions on storage group level. Permissions would populate down to end user mailboxes. This "population" takes some time (from 30 minutes to hours).

The command to assign permission on "First Storage Group" is as below:

Add-ADPermission -Identity "First Storage Group" -User cupsexch -ExtendedRights Receive-As

What if you have multiple storage groups and want to automate the process? You may use the command below:

Get-StorageGroup | add-ADPermission –user cupsexch -ExtendedRights Receive-As

"Get-StorageGroup" command will get all storage group names from system and feed the names to "Add-ADPermission" command.

The command to verify permission is as below:

Get-MailboxPermission jdoe -user cupsexch | Format-Table -autosize

This command will display "cupsexch" permission on jdoe's mailbox. Anything after the pipe sign (|) is just for formatting purpose.

If you got nothing, that means "no permission". If you assigned the permission on higher containers, it might not have flowed down yet.

If the permission has been populated, you should see something like the screenshot below:

"AccessRights = {FullAccess}" means "cups7exch" account has permission to read "John Doe's" mailbox.
"Is Inhereited = True" means this permission was inherited from some higher level containers.

This is the only reliable way to test permissions.

Cisco documents described a test method which use OWA to test the URL "https://owa-address/exchange/some_user/calendar" with the service account. This method may or may not work depending on other permissions you configured for the service account. For example, if you are like me to give minimum permissions, Cisco test method would yield a "no permission" page as below. However, CUPS calendar works just fine with the "minimum permission".

Troubleshooting

If calendar integration doesn't work, "Presence Engine" (PE) logs are the only traces we need.

Ideally, we want the logs since PE started. Thus if you could, do the following:

1) Restart PE
2) Wait unti PE started and stabilized
3) Log into CUPC
4) Collect PE logs since the time of restart
5) Use WinGrep to search for keyword "owa". It'll give you all message regarding OWA transactions.

Following are some common problems:

"Certificate not trusted"

This could be caused by the following:

The name configured in "CUPS > Presence > Presence Gateway" does not match with the name in identity certificate. (e.g. IP address vs. FQDN)
CA certificate was not imported as "Presence Engine - Trust". If there are multiple CAs in the certificate chain, all of them need to be imported.
CA bit was not set (usually happens on IIS self-signed certificate)

"440 Login Timeout"

When FBA (Form-Based Authentication) was enabled, you'll see one "440 login timeout" in PE log for each transaction. This is normal. If you keep getting "440 login timeout", it usually indicates an authentication issue.

To reveal the nature of the problem, it's recommended to change "Form-Based Authentication" to "Standard Authentication" on "/Exchange" virtual directory on OWA. See screenshot below:

Changing this will not change the way or feel of regular OWA logon. It's just change the authentication method for WebDAV. (you'll need to reset IIS by using command "iisreset /noforce")

"401 Unauthorized"

This is usually caused by permission issue. See "Receive-As" permission configuration above for testing method.

Another testing method would be configuring an end user's credential on presence gateway. An end user's credential would definitely have permission on his own mailbox. If you log in to CUPC with that end user account, calendar integration should work. This is a good way to do "problem isolation".

To be continued...

Hard Drive Partition and version control of UC Appliance

2010-01-23T19:02:00.000-08:00

Active and Inactive Version

Many Cisco Unified Communication appliances (CUCM, CUPS, CER, CUMA, etc.) share the same OS (Cisco customized Linux).

For maintenance purpose, you may install two copies (versions) of systems on the hard drive. Cisco call it "active version" and "inactive version".

CLI commands:

show version active
show version inactive
utils system switch-version

Please note: "active" and "inactive" are relative. When you switch the version (utils system switch-version), the "active" version will becomes "inactive".

If you're a Windows guy, you should be familiar with C:\boot.ini.

If you're a Linux guy, you should be familiar with grub.conf.

It's the same way Cisco UC appliance controls which version to boot from.

Partitions

The two copies of software are installed into two partitions: \ (referred as Partition A) and \partB (referred as Partition B). Whenever you use "utils system switch-version", the active partition will become inactive. The inactive partition will becomes active.

Upgrade and Switch Version

The word "upgrade/patch" has a different meaning in Cisco UC world. Instead of "replacing" files, the "upgrade/patch" process is actually installing a full copy of system in the inactive partition. This has two benefits:

1) You may perform an upgrade/patch during production hours.
2) It's easy to fall back to the old version.

Scenario:
You have 6 CUCM servers in the cluster. Let say, upgrade each server takes about 2 hours. Your business does not allow any downtime during business hours.

Questions:
Q1: How long does it take to upgrade the whole cluster?
Q2: How much time you'll have to spend in after hours for the upgrade?

Answers:
A1: About 4 hours (2 + 2)
A2: A couple minutes.

Explanation:

1. You need to finish the upgrade on CUCM publisher before you can do the upgrade on subscriber. It takes 2 hours to upgrade the publisher. The new code be installed into inactive partition. You don't have to switch to new version right after install. Thus you can do it in business hours.

2. Once the the new version has been installed on publisher (even it's in inactive partition), you may start upgrade process on subscribers (simultaneously). This takes about 2 hours (because you're upgrading all subscribers simultaneously). You don't have to switch to new version right after install. Thus you can do it in business hours.

3. In after hours, you may use "utils system switch-version" command to switch all boxes to new version. This usually takes less than 10 minutes.

However, there's a catch: if you made any configuration changes after the point of upgrade, those changes wouldn't be reflected in the new version. For example, you performed the upgrade at 10AM but didn't switch to new version. Then you switched to new version at 6:30PM. Any configuration changes made between 10AM and 6:30PM will be lost.

Under the hood of "utils system switch-version"

What actually happens when you type the command "utils system switch-version"?

1) It modifies /grub/boot/grub/grub.conf file to make the other partition active
2) It synchronizes UFF (User Faced Feature) to the other version. UFF refers to Call Forwarding, MWI (Message Waiting Indicator), etc.

If the system failed to switch version, here are some options:

Option 1: Try "utils system switch-version nodatasync"
This turn off the UFF data sync action.

Option 2: Use "Recovery CD" (downloadable from CCO) to switch version.

Option 3: If you're a Linux guy, it shouldn't be too difficult for you to get access to /grub/boot/grub/grub.conf.

OCS 2007 R2 on Windows 2008 R2 with SQL 2008

2010-01-23T18:43:00.000-08:00

Neither Windows 2008 nor SQL 2008 is supported by OCS 2007 R2. But if you really want to do it, here are some tips:

Use SQL 2008 R2

SQL 2008 will fail to install on Windows 2008 R2. Instead of trying to 'fix' it, you may just use SQL 2008 R2.

Pool Creation Failure

a) manually create database rtcconfig (assuming you know how to use SQL Management Studio)

b) manually run the script on OCS installation CD.
D:\Setup\amd64\DbSetup>cscript.exe poolcfgdbsetup.wsf /clean /sqlserver:rcdn /serverrole:EP /verbose

c) continue GUI install

Error: "Not available: IIS 6 Management Compatibility and IIS Windows Authentication role services must be installed before you Deploy Server."

Solution: Install all IIS6 compatibility options. Install all authentication options

Error: "The Windows Media Format Runtime is required in order to install this component. Installing the Windows Media Format Runtime may require a system restart to complete the installation. Click OK to continue with the installation."

Solution: install the Desktop Experience Feature.

Error: "[0xC3EC78D8] Failed to read the Office Communications Server version information. This can happen if the computer clock is not set to correct date and time."

Solution: Uninstall MS Crypto API security update KB974571

The art of troubleshooting

2010-01-22T05:29:00.000-08:00

Since I joined TAC, I've been the top case solver for 16 consequent quarters no matter what technology group I worked in. I'd like to share some tips on troubleshooting.

Understand the user's expectation.

Still remember the old joke that a user called IT support and said the "cup holder" on his computer stopped working? It turned out to be the CD drive. He insisted he's been using it for years.

Understanding user's expectation can help you determine if you should do customer education or troubleshooting.

Keep it simple

Which one is easier? Troubleshoot a light switch or troubleshoot a space shuttle?

Multiple-system integration adds complexity to the problem. You should try to simplify it as much as possible.

For example: When PSTN call comes in, it hits Unity Auto Attendant. Press 2 to transfer to sales queue, which is a CTI route point handled by UCCX. If no one answers the call, it should goes into a voicemail box dedicated for sales department. Instead of going into voicemail, the caller just heard repeating "transferring..."

In this case, we have too many elements in the picture - Unity, UCCX, CUCM, voice gateway, service provider. Instead of troubleshooting from end to end, we should troubleshoot it segment by segment -

1) What if we called the sale agent directly? If he didn't answer, would the call goes into voicemail? (get Unity Auto Attendant and UCCX out of picture)
2) What if we bypass Unity Attendant Console and call UCCX route point directly? Would it work properly? (get Unity Auto Attendant out of picture)
3) What if we make a test call from internal phone? Would the problem be the same? (get PSTN and voice gateway out of picture)

Other tips to make things simple during troubleshooting:
1) Use default settings. For example, use a "vanilla windows" (fresh installed with Microsoft CD) instead of using a "corporate customized" image.
2) Test on LAN instead of over VPN (again, decrease number of elements)
3) Always assume the system is case sensitive (err on the safe side)

Find a reference point

If a software doesn't work for one user and works for another one, use the good one as reference point and find out the difference.

Of course there are many differences between two users, such as their wife and kids. :) But we should look at the most relevant ones.

Most software nowadays are "client-server" model. The most relevant ones are accounts and computer. Switch the computer (or switch the account) to see if the problem follows the computer or account. If it follows the computer, it might be network or computer settings (client side). If it follows the account, it might be configuration issue (most likely server side).

Understand positive and negative result of the test

e.g.

"Dad, I couldn't find any Easter eggs in the backyard!". Does that mean there's no eggs there?

"Dad, I found some Easter eggs in the backyard!". That means there are some eggs there.

Map a UC appliance as a network drive

2009-08-29T12:05:00.000-07:00

Remember the "good old days" of CCM 4.x? You can do almost everything on the box. Because it's in fact a Windows 2000 box. However, this brings security and supportability issues.

With the introduction of Linux-based Unified Communication appliance (CUCM 5.x), Cisco locked down the box. You can only access the box via admin web page or a tailored command line.

One of the inconveniences is to review log files. On the old-school CCM 4.x, you may just view the logs in C:\Program Files\Cisco\Trace. On the new UC appliance, you'll have to use RTMT (RealTime Monitoring Tool). This is especially annoying if you're testing your system. For each test, you'll have to download a new set of logs to your computer. (though you may use 'Remote Browse' in RTMT, its function is very limited)

What if we can go back to the "good old days" and view the file system just like a Windows drive?

Take a look at the screenshot below. It's a CUCM 6.1.4 mapped to my Windows XP laptop. You can read/write files on CUCM just like a local hard drive. For those people who are not a fan of VI, you may use your favorite editor (such as Notepad++/UltraEdit). And you may use any Windows tools, such as Windows search, WinGrep, WinZip, etc. How's that? :)

To achieve this, you need two things: a root account on CUCM and a software who can map a SFTP server to a network drive (such as sFTPdrive).

UC Appliance on VMWare

2009-07-15T10:41:00.000-07:00

In theory, any software runs on x86 platform should be able to run on VMWare, unless the software vendor explicitly block it.

Cisco has many software running on x86 platform. We'll discuss Unified Communication products here - CUCM, CER, UCCX, etc.

CUCM

CUCM is the flagship of Cisco UC products. You may install CUCM on VMWare just fine. No hacking is required, but you'll receive a warning that VMWare is not "officially" supported. ie. you shouldn't use it in production. Cisco planned to support VMWare in production in the future ( probably with UC 8).

Though you may install CUCM on VMWare, there are some limitations.

Limitation #1 Licensing

Cisco limits the number of nodes and DLUs on a VMWare MAC address. (3 nodes, 125 DLUs at the time I'm writing this blog). If you should need more than 3 nodes and 125 DLUs, you may change the MAC address of the CUCM (change in VM guest, not in VM host). Just Google keywords "change MAC address on Linux" and you'll find the answer.

Limitation #2 SNMP Agent

You'll notice that "SNMP Master Agent" service fails to start if CUCM was installed on VMWare. This will cause problem if you're testing CER (Emergency Responder). CER needs SNMP connection to CUCM to retrieve phone info. The workaround would be issue the command on CUCM root shell "/sbin/chkconfig snmpd off". Then reboot the server.

Limitation #3 VMWare Acknowledgement

Since from version 7, CUCM requires you acknowledge the "VMWare agreement" during startup. If you reboot the CUCM remotely (either via OS Admin web or via CLI), the server will not boot up until you press the "Agree" button on the console. The workaround is to edit /usr/local/bin/base_scripts/hardware_check.sh,

change the line

if [ "$hwmodel" = "vmware" ];

to

if [ "$hwmodel" = "foobar" ];

On newer versions,

Change the line

    if isHardwareUnsupported || isOriginalHardwareUnsupported ; then

To

   #if isHardwareUnsupported || isOriginalHardwareUnsupported ; then
    if 0 ; then

CER

Besides the limitations mentioned about, CER has another limitation with VMWare.

CER retrieve information from CUCM via SNMP. This includes the machine type of CUCM. If CUCM is running on VMWare, the machine type will be "unsupported" from CER point of view. The workaround is to edit /usr/local/CER/etc/devices.xml file on the CER box. Add the following tag under "CcmHost" family tag:

member OID="1.3.6.1.4.1.99.1.1.3.28" OIDNAME="vm-ware" CAPTION="VMWare"

Reboot CER or restart Phone Tracking Engine.

UCCX

So far, UCCX has been running on "Cisco OS" 2000/2003, which in fact is Windows 2000/2003. However, UCCX will refuse to install if Cisco specific registry key is missing. Follow instructions on http://www.tek-tips.com/viewthread.cfm?qid=930128&page=1 to add the registry key.

Good news is: VMWare is supported since Cisco OS 2003.1.4. If you're using OS 2003.1.4 or newer, you don't need the registry hack.

Another tips is: if you want to bypass the hard drive/memory requirement check, you may create an empty file named "crstest.ini" on C:\. Then CRS won't require 72G HDD/2G memory to install. Of course, this is for testing purpose.

Root shell on UC appliance

Many of the hacking above requiress the root access to the appliance (CUCM, CER, etc.). Just use Google to find the answer. For example: http://www.blindhog.net/how-to-get-root-access-on-call-manager-56-server/

Virtualize everything!

2009-06-20T17:48:00.000-07:00

People in network world should have heard about 'simulators'. A router simulator gives you the command line interface you can practice on.

With more and more network equipments move to open source OS (linux) and x86 platform, the word 'simulator' has another meaning - virtualization. Which means, you can run the software (such as IOS, JUNOS, etc.) on a x86 computer just like it runs on the original hardware.

I still remember the excitment when I discover that I can run JUNOS on a 486 PC back in 1999. I built my first JNICE lab with nine of those PCs ($50 each).

Now, working in Cisco Unified Communication team, one of the challenges I'm facing is the availability of equipments. Sure we have access to IP phones, routers and switches. But getting mobile phones (BlackBerry, Nokia, WinMobile, Android, iPhone) and ASA (Adaptive Security Appliance) for each engineer is not as easy as we thought.

Mobile phones are required to test CUMC (Cisco Unified Mobile Communicator). ASA is required to test CUMA (Cisco Unified Mobility Advantage), Phone Proxy and CUPS Inter-domain Federation.

Fortunately, with simulators, everything can be run on a PC (or a virtual machine).

Below is a BB simulator and ASA simulator running on a VM.

When running a network appliance image (such as JUNOS or ASA) on a PC or VM, one thing to notice is that you cannot use the monitor and keyboard as console. Why? Because a router does not have video card and keyboard. The 'console' port is the serial port.

If you are using a PC, connect the console cable to the COM port.

If you are using a VM, you may direct the serial output to a named piple.

For the VM that running the appliance image (such as JUNOS or ASA), set the 'near end' to 'Server'. Set the 'far end' to 'A Virtual Machine'. You may use any name for 'pipe name'.

For the VM that acting as 'terminal' (such as WinXP or Linux), set the 'near end' to 'Client'. Set the 'far end' to 'A Virtual Machine'. The 'pipe name' needs to match the one you configured above. After this, it's like there's a serial cable connects the terminal VM(WinXP) and the appliance VM (ASA).

It's live - "Ask Expert" on Netpro

2009-06-06T14:14:00.000-07:00

If you have questions regarding CUPS/CUPC, presence, OCS/MOC, etc., you may ask questions on Netpro forum. They have a "Ask the Expert" event this week for CUPS and presence.

Link as below:
http://forum.cisco.com/eforum/servlet/NetProf?page=netprof&forum=Unified%20Communications%20and%20Video&topic=Unified%20Communications%20Applications&topicID=.ee835d2&fromOutline=&CommCmd=MB%3Fcmd%3Ddisplay_location%26location%3D.2cd34986

Thanks!

UC Appliance Command Line - Part 2

2009-05-26T20:30:00.000-07:00

Part 2 - Start, Stop, Restart

utils service list

This command will list all services on an appliance. It's usually used with the parameter 'page', so it'll pause at each page.

utils service stop
utils service start
utils service restart

These commands are used to stop/start/restart services. For example, if you'd like to restart "Cisco Tomcat" service, you type "utils service restart Cisco Tomcat".

utils system shutdown
utils system restart

These commands are used to shutdown or restart the system.

utils system switch-version

This command is used to switch software version (if you have two versions installed). For your information, Cisco Unified Appliance will keep two versions of software on hard drive - one in the root partition, the other one in "PartB" partition. This provides you an option to fall back to an old version.

To see the versions installed, use the commands below:

show version active
show version inactive

Every time you run "utils system switch-version", it'll make the active partition inactive and make the inactive partition active.

Please note that each partition (version) has it's own database, which means they don't share the same database (configuration). If you switched version, you might lose any changes you made in the other version.

UC Appliance Command Line - Part 1

2009-05-13T19:57:00.000-07:00

Cisco built many Unified Communication "Appliance" based on Linux, such as CUCM (Communication Manager, a.k.a. CallManager), CUPS (Presence Server), CER (Emergency Responder, a.k.a. e911), etc.

Even though those appliances are built on Linux, Cisco does not give you shell access to the box (if you know about Linux, you know what a "shell" means). This is for security and maintenance purpose.

However, some of the maintenance work needs to be done via command line. Cisco built a customized command line interface (CLI) for UC appliances. Since most of the UC appliances share the same OS, they also share the same sets of CLI commands.

Mastering some of the CLI commands would make your life easier (or you may impress your colleagues or boss by showing off some of the rarely used commands).

Some basics:
0) To get access to the CLI, you need the "OS Administration" credential. "OS Administration" credential is stored in /etc/passwd file, while "Application Administration" credential is stored in database.

1) To access the CLI, you may either go to the sever console or SSH to it. (Telnet is not supported for security reasons).

2) Cisco keeps adding new commands to CLI. Some of the commands are available on new versions (such as CUCM 7.x) but not available on old versions (such as CUCM 6.x).

3) You may always use question mark (?) and tab key to get help.

4) Unlink IOS, UC Appliance CLI doesn't take abbreviations. You'll have to give the full command (either type it yourself or use the tab key).

5) "show" command is to display information

6) "set" and "unset" commands are to change configuration

7) "utils" command is run maintenance utilities (such as system reboot, backup/restore, etc.)

8) "run sql" command is to run SQL query against the database.

Part 1: Getting system info

show status

This command will give you the following information:

Hostname of the box
Current date/time on the box
Current time zone configured on the box
Current version
How long the system has been up and running
CPU/Memory/Hard Disk usage

For example, if you cannot access the web interface of your CUCM box, you open a case. TAC engineer asks you what version is the CUCM. If you can access to the CLI, you may find out the version. This could possibly speed up the resolution.

show hardware

This command would give you the hardware information (such as serial number of the box). If you need to find out the serial number remotely, you may SSH to the box and use this command. Serial number is critical for entitlement and tech support.

show network eth0 detail

This gives you the following information:

IP address of the box
MAC address
DNS
Gateway

This command is useful if you need to check the MAC address quickly (for licensing purpose).

To see all "show" commands, type "show ?"

"Ask The Expert" on Cisco NetPro Forum

2009-05-07T20:10:00.001-07:00

Sorry I didn't post any new article lately.

I'll host a "Ask The Expert" event on Cisco NetPro forum June 8 - June 12.

For those new to the forum, "Ask The Expert" is a periodic event that the subject matter expert (SME) would answer questions on a specific topic (such as licensing, contact center, video conferencing, etc.). Of course, I'll be answering questions on CUPS/CUPC and presence-related questions. Bring your toughest questions! :)

Though I cannot guarantee every question be answered immediately, I'll make sure they get to the right people.

We believe Unified Communication will make our life better (though the process of deploying it might make your life tougher... LOL)

Book, charity and life

2009-04-04T14:20:00.000-07:00

I'm a customer support engineer that supporting Cisco Unified Communication products.

I built this blog and wrote the book "Deploying Cisco Unified Prsence" with the intention of helping my customers (and my employer). The book was priced very cheap (39.99). After deducting the manufaturer and distribution cost, the revenue I received from retailers is $5.16 for each copy sold (it'll be higher if it was sold from the publisher's website).

For some reasons, my motive was questioned. (Sorry I can't disclose more details. But it makes me feel really bad.)

Thus I make an announcement here (and Lulu.com), that all (100%) revenue from this book will go to "American Red Cross International Relief Fund".

I'll try my best to answer any technical questions you have. I'll be hosting a "Ask the Expert" session on Cisco NetPro Forum in June 2009 (on CUPS/CUPC products).

God bless America. God bless you!

Donation receipts for Q1 2009:

"Hardware not supported"

2009-03-31T10:44:00.000-07:00

When installing Cisco Unified Communication products (CUCM, CUPS, CER, UC, CUMA, etc.), you might get a message saying that the hardware is not supported. It's a pain in the butt. Espeically when you (or you client) spent quite a few $$$ to get a brand new server and yielded "not supported".

History

Cisco itself does not manufacture servers (not until lately, with introducing of 'Unified Computing'). Cisco OEM servers (x86) from IBM, HP and Dell and brand it as "Cisco MCS" (Media Convergence Server). Cisco also labels those servers with it's own model number. For example, MCS-7845-H2 is actually a HP DL380 G5 server.

Cisco recommends customers purchase "Cisco MCS" server for Unified Communication products. The major advantage of that is it guarantees the compatibility between hardware and software. For example, you may find the CUCM compatibility matrix here: http://www.cisco.com/en/US/prod/collateral/voicesw/ps6790/ps5748/ps378/prod_brochure0900aecd8062a4f9.html

The mess

Even though Cisco recommends customers purchase MCS servers, it does not prohibit people from buying "equivalence" from manufacturers directly (ie. from IBM/HP/Dell).

If you decided to buy "equivalence", be careful, Cisco has very strict requirements on that. If you didn't order the right parts, the server could yield "not supported" (ie. the software install will fail).

What does it look for?

When the software being installed, it usually looks for the following attributes on the system:
1) Machine type (model number in BIOS)
2) Hard drive and RAID card
3) CPU speed
4) Memory

Frequently seen issues:

#1 You have an MCS I-series ("I" stands for IBM) server from Cisco. One day, the motherboard burnt out. IBM replaced the motherboard (yes, it's IBM who services the server, though you bought it from Cisco).

You tried to reinstall CUCM 6.1.2, but it kept saying the server was not supported.

Cause of the problem:
In IBM BIOS, there's a field called "Machine Type". The generic IBM machine type is different with Cisco MCS machine type.

Solution:
1) Obtain a BIOS update disk for the server from here (2000.4.4 supported version 1.14, this link is for 1.17, please note that 1.17 has NOT been tested): http://www-304.ibm.com/jct01004c/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-57074&brandind=5000008

2) Boot to the disk and flash the BIOS. During the BIOS flash, you should receive a prompt as to whether or not you would like to change the MTM. Please select yes, and enter the correct machine type (e.g. 884xxxx)

Note: the above should be performed by a Cisco TAC engineer.

#2 You have an MCS H-series ("H" stands for HP) server from Cisco. One day, the motherboard burnt out. HP replaced the motherboard (yes, it's HP who services the server, though you bought it from Cisco).

You tried to reinstall CUPS 6.0.4, but it kept saying the server was not supported.

Cause of the problem:
Your old motherboard was with a CPU at speed of 2.13 Ghz. Since the 2.13 Ghz CPU was end of life, HP gave you a 2.8 Ghz CPU and thought you'd be happy with that.

Resolution:
Though you might be happy with the faster CPU, the software was not happy at all. Based on the machine type, the software expect a 2.13Ghz CPU.

#3 You ordered a server from HP, you made sure the CPU, memory, hard disks meet the Cisco requirements. But the software stills said "hardware not supported".

Root cause of the problem:
You forgot to order a "PCI-X/E Mixed Riser Option" from HP.

How do we find the "equivalence" of a MCS server (so you can order it from HP/IBM)?

Step 1: Go to product compatibility page and find a supported MCS server. e.g. for CUPS 6, go to http://www.cisco.com/en/US/docs/voice_ip_comm/cups/7_0/english/compatibility/cupcompatibility.html#wp77512. Let say, you picked MCS-7825-I3-IPC1, which is good for CUPS 6.0.4 new install.

Step 2: Go to this page and click on IBM or HP link (http://www.cisco.com/en/US/products/hw/voiceapp/ps378/product_solution_overview_list.html). MCS-7825-I3-IPC1 is an IBM server. So you would click on the IBM link.

Step 3: Search for the 2nd and 3rd part of the MCS model number. e.g. 7825-I3 in this case. You'll find something like this:

IBM x3250 with Intel 3050 Xeon 2.13-GHz Processor
"... It is the configuration equivalent of the Cisco MCS 7825-I3"

And you'll find the parts list for it:

Table 19. Non-Country Specific Hardware for IBM x3250 with Intel 3050 Xeon 2.13-GHz Processor*

Quantity	IBM Type-Model Feature	Description
1	4364-AC1** or 4365-AC1	IBM System x3250
1	0992	CPU Retention Module
1	1128	x3250 Revision 1 System Planar
1	1272	Dual Core Intel Xeon 3050 (2.13 GHz / 2M L2)
2	1903	1-GB DDR2 667 SDRAM DIMM Memory
1	2007	BIOS GBM
1	2046	Front Bezel
1	2088	3.5inch DASD Cage
2	2091	SATA Filler 3.5inch
1	2268	Base Hardware
1	4144	CDRW/DVD Combo V UltraBay Enhanced
1	4256	Rack Mount Kit
1	4367	Simple Swap SATA RAID Kit
2	5291	160GB 7200 RPM 3.5 inch Simple Swap SATA HDD
1	9011	Internal RAID - Cabled only - setup by Customer

You'll have to order each piece on the list.

If we got a "hardware not supported" message, how do we know which part is not meeting the requirements?

To see the reason of the failure, you need to review the /tmp/hw_validation_err file on the hard drive. You may press ALT+F2 while the installation is in progress (before it gets to halt state). Then you'll get to the Linux command prompt. Type the command below to display the content of the file:

cat /tmp/hw_validation_err

Other useful files include: /tmp/hw_info, /tmp/anaconda.log and /tmp/install.log.

Licensing

2009-03-26T17:23:00.000-07:00

Licensing is another mysterious area in Unified Communication. Each product has its own licensing model.

CUCM (CallManager)

CUCM license resides on publisher. The 'host ID' in license file needs to match the MAC address of the publisher.

There are three different types of license for CUCM: node license, feature license and device license (DLU). Node license is for server node. Feature license is required for upgrade or CUCM 7 and above. Device license is required to configure devices (such as phones).

CER (Emergency Responder or e911)

CER license resides on publisher and subscriber (if you have CER subscriber). License file on pub needs to match the MAC address of pub. License file on sub needs to match the MAC address of sub.

If you received a "license expired" message on a two-node CER deployment, 99% of the chance you don't have a license for sub (wrong MAC address).

CUPS (Presence)

Like CUCM, CUPS also has node license and device license. Node license resides on CUPS publisher. Device license resides on CUCM (it's just a regular DLU license).

Node license could contain proxy or PE (Presence Engine) or both.

CCIE Voice Lab v3

2009-03-11T09:18:00.000-07:00

Cisco is going to change the CCIE Voice Lab in mid-July 2009. Major changes are:

1) Remove analog devices (such as VG248, ATA)
2) Remove CatOS (Catalyst 65xx)
3) Replace CCM with CUCM 7 (Linux Appliance)
4) Replace Unity with Unity Connection 7 (Linux Appliance)
5) Add CUPS 7 (Linux Appliance)
6) Add SIP phones

IPExpert (a training company) provides some practice labs. The topology would be like below:

Here are some tips if you're going to build your own lab.

PSTN-WAN simulator

As shown in the diagram above, we have a PSTN cloud and a Frame-Relay cloud. We can use a single router to simulate that. And this router could also be the terminal server.

I would choose a Cisco 2811 router with the following modules:

PVDM2-16 (or other PDVM resource).
HWIC-4T : 4-port Serial module for Frame-Relay simulation.
VWIC2-1MFT-T1/E1 : 2-port T1/E1 module for PSTN simulation. You need at least three ports. So you may order two of these. Or to save some money, order one 2-port module and one 1-port module
HWIC-8A & CAB-OCTAL-ASYNC : 8-port async module for terminal service

Don't forget the female serial cable (CAB-SS-V35MT). You need at least three.

On HQ, BR1, and BR2 Gateways:

PVDM2-8 (or other PDVM2)
HWIC-1T & CAB-SS-V35MT
VWIC2-1MFT-T1/E1

On BR1 and BR2 Gateways
HWIC-4ESW-POE : Ether net module to power up the IP phones.

On BR2 Gateway
NM-CUE : Unity Express Module

Switch

Any Cisco IOS switch that supports PoE.

Servers

I recommend using virtual machine(VM) for the lab:

1. If you don't use VM, you're going to need the expensive Cisco/HP/IBM server to install CUCM, UC, CUPS.
2. If you don't use VM, you're going to need license files for CUCM, UC, CUPS.
3. If you don't use VM, you're going to need 5 servers and a couple of workstations (to simulate soft phone and run CUPC/CAD)
4. With VM, you can easily clone VMs, which saves you lots of time.

I myself use VMWare. I have no experience on Microsoft Virtual PC (HyperV).

Here are some caveats you need to know about VMWare:

1) You may either use "VMWare Server" (a.k.a. GSX) or "VMWare Infrastructure") (a.k.a. ESX)
2) GSX is free while ESX is commercial license
3) VMs on GSX has issues with NTP. This is a known issue.
4) ESX does not support audio device. Your VM might need an audio device to launch soft phone (such as CIPC or IP Blue). Try google "virtual audio cable" and you should find a solution.

Following is my lab set up with four 2811 routers, one 3560 switch, one PC (8G RAM, 750G HDD):

When creating VM, make sure you choose RHEL 4(RedHat), 32-bit, one CPU. For CUCM 7, you need at least 1G RAM and 60G hard drive. For CUPS 7, you need 1296M RAM and 75G hard drive.

Change Notification

2009-03-03T19:24:00.000-08:00

On Cisco Unified Communication products, there's a terminology called "Change Notification".

What is "Change Notification"? And how does it affect products' functionality?

Typical symptom of "Change Notification Issue" is: the changes you made didn't seem to take effect, including:

#1 You add a new DN (Directory Number) to CUCM. But you got fast busy when calling that DN.
#2 You try to reset the phone from GUI. But the phone didn't get reset.
#3 You updated the information on CUCM (such as PIN#, line association, etc.). But CUPS didn't get those changes.
...

"Change Notification" is one component notify another component that changes have been made. The other component should react to this notification.

For example, when you add a new DN to CUCM. The changes were made to database. Database component should send notification to the call routing component. So the call routing function could work properly (calls can be routed to the new DN).

If change notification was failed (or delayed), the call routing component will use outdated information to route calls. Thus calls to new DN would failed.

How do you know if there's a change notification issue? If the components are on the same box, you may use the command "show tech notify"

An example output would be link below:

32 I 0 P 118 H 118 T 118 S 80 ccm
...
38 I 0 P 119 H 119 T 119 S 27 EPAS_SyncAgent[10.88.229.209]:32958

The line that begin with numbers (32, 38, ...etc.) represent clients (applications) subscribed to change notify

"32" means it is the 32 client slot on this server.
"I 0" means there are 0 messages in shared memory to be processed by this client.
"P 118" means 118 messages have been processed.
"H 118" means the "Head" message position is 118.
"T 118" means the "Tail" message was in position 118.
This is the optimal situation: 118-118=0 (nothing to be processed).

“S 80 ccm” means there are 80 tables subscribed by this client and the client name is ccm (callprocssing).

You may also use RTMT to see the CN (Change Notification) queued. If you saw non-zero value in queue, either the server was busy, or you got CN issues. Restarting corresponding process (such as ccm) normally would clean up the queue and solve the problem.

Change notification across different boxes is a little bit complex. CN was sent via IPSEC tunnel in this scenario. IPSEC was controlled by Cluster Manager. Trust relationship was established during installation (when you add a box into cluster).

In order to add a new server into a cluster, two requirements have to be met:
1) The hostname of the new box needs to be presented in the Publisher's server list or application list. (this can be done manually or automatically depends on the server you try to add).
2) You need to know the cluster "secret password". The password needs to be entered during installation (of the new box).

If either one was changed after installation, the trust would be broken. You can verify that by looking at "Cluster Manager" logs.

If trust was broken, change notification won't work. For example, changes made on CUCM didn't populate to CUPS. Restart Sync Agent would force a synchronization, which is not affected by change notification or IPSEC.

Phone reset issue

2009-02-27T10:51:00.000-08:00

Phone reset issue is the most common yet most nasty issue (especially when it's "sporadic" or "intermittent").

Please note the different between reset and re-register.

If the phone lost connection with CUCM, it'll try to re-register. "Lost connection" means, the phone lost three keepalives from CUCM in a row. By default, those keepalives are sent every 30 seconds. You may verified that from "Cisco CallManager" trace. If CUCM sent keepalives but phone didn't receive it, it's usually network issue.

Reset usually happens when IP address on the phone was lost. In that case, the phone need to go through a reset process to acquire a new IP address. This is usually a DHCP (server) problem.

When the DHCP client reaches half-life, it'll try to renew the lease with DHCP server. e.g. If the DHCP lease was 72 hours, the client will try to renew at 36 hours. In normal situation, DHCP server will agree to renew. So the client can keep its IP address.

If DHCP server explicitly refused the renew, DHCP client has to release the IP. This is unusual and probably would be a problem of DHCP server.

On phone console log, you would see something like below:

NOT 08:11:10.854439 DHCP: Restart - delay = 0
NOT 08:11:10.866112 DHCP: Sending Release...
NOT 08:11:10.894059 DHCP: dhcpSendReq: status 0x12300000
NOT 08:11:10.894946 DHCP: Sending Request...
NOT 08:11:10.899614 DHCP: NAK received
NOT 08:11:10.901451 DHCP: clear info - IP = 10.2.16.37, state = 2
NOT 08:11:10.902400 DHCP: Sending Release...

"NAK received" means the DHCP server refused to renew the lease.

Next time, if you got phone reset periodically (say, every 36 hours), check DHCP lease time. If the cycle matches the "half-life", it's most likely DHCH server issue.

KISS - Keep It Simple Stupid

2009-02-25T19:54:00.000-08:00

Most IT guys would know about KISS rule (Keep It Simple Stupid). However, not too many really understand and utilize it. Let's take a look at some Cisco Unified Communication products and see how we can utilize KISS rule.

The most frequently seen problem description is "it doesn't work".

"It" could mean lots of things. "It" could involved different products from different products/vendors (such as CUCM/IPCC/CUPS from Cisco, MOC from Microsoft, PBX from Avaya, T1 trunks from AT&T, etc.).

In order to simply the problem, we have to narrow down the problem quickly.

For example, if a customer said "My call center agents cannot make phone calls", I would ask "can you make calls from IP phone to IP phone in the same office?". This question could potentially eliminate call center software, voice gateway, PSTN and codec issues. If you didn't ask, you'd have to troubleshoot those items one by one (assuming you know how to troubleshoot those items)

Another example is network issue. All Unified Communication software rely on network connectivity. They wouldn't function properly if network didn't. Sometimes, network issue was not as obvious as you had thought. For example:

1) Windows Firewall service was stopped. But traffic wouldn't pass through until you explicitly open ports on it. (Hard to believe. But it happens)

2) You're not using VPN. But VPN client was running as a service and have firewall option turned on. (works as designed)

3) You claimed there was no firewall in the network, but there's a FWSM(Firewall Switching Module) on the switch.

4) You opened all ports on ASA (Firewall and VPN), CUPC still won't work. That is because one of the ASA bugs prevent large SIP message from passing through.

...

To troubleshoot network issue, you have to:
a) Have visibility on every components in the network
b) Be very good at all network layers (from physical layer to application layer)
c) Know how to use sniffer (such as Wireshark)

The difficult part is: sometimes you wouldn't think it's the network because it's not that obvious. Hence you wouldn't go down that path at all. You have to use KISS rule to find it out.

Example #1:
Customer: "My CUPC doesn't work."
You: "Doesn't work for all users? Or for some users?"
Customer: "For those users working from home."
You: "If those users were in office, would CUPC work for them?"
Customer: "Yes."

Now you know the problem is outside CUPC. Probably on the network (VPN?)

Example #2:
Customer: "My CUPC doesn't work."
You: "Doesn't work for all users? Or for some users?"
Customer: "It works for John but doesn't work for Mary. And they are both in the same office."
You: "On John's computer, can you log into CUPC with Mary's account? See if it works?"
Customer: "Yes, it works."

Now you know the problem is outside CUPC. Probably on Mary's computer (Firewall?)

Some KISS rules for Cisco UC (Unified Communication) products:

#1 If you don't know if it's case sensitive, assume it is.
This becomes a problem when Cisco moved from Windows platform to Linux.

#2 Because of #1, try to use lower case as much as you could.
Some people use capital case for cosmetic purpose. This could potentially cause some problems and it could take weeks to troubleshoot.

#3 Eliminate dependencies as much as possible.

Example A: When you installing CUCM, you have the option to use DNS, NTP, etc. Do NOT use them. If you use them, the installation might fail if those components weren't configured properly. You chance to configure them after install. I can't tell you how many problems are caused by DNS (even after install).

Example B: Don't use same "service account" for different applications. For example, you used the same active directory account for CUCM LDAP integration, CUPC LDAP search and CUPS Calendar. If CUPS admin change the account password (for whatever reason), it breaks CUCM and CUPC.

Example C: Get rid of CUCM subscribers during Windows-to-Linux migration. When you migrate from CCM 3.x/4.x (Windows) to CUCM (Linux), DB replication is always a headache. DB replication would fail if hostname, IP address was changed during migration (or some other changes between Pub and Sub). To avoid those headaches, remove subscribers from the cluster before migration. With a single server (Publisher) in the cluster, your chance of failure is far less than a 10-server cluster. After migration, you may add the subscriber to cluster one by one.

#4 Be a "minimalist"
Sales people tend to sell all the "bells and whistles" to customer. Sure that's the selling point. But as an engineer, if you want to get the job done smoothly, try to start with the minimum.

Example A: Use TCP instead of TLS.
Sure we want the security of TLS. But don't try to run before you can walk. Make sure the product is working before attempting TLS. If it didn't work with TLS, you know where the problem is.

Example B: Use simple passwords.
Sure we want the security of a long, complex passwords (how about 1024-character long?). But for installation and troubleshooting purpose, keep it short and simple (don't use special characters)

Example C: Build a simple test bed.
I've seen some integrators tried to deploy their first CUPS/CUPC installation over the VPN (because they are not onsite). This is a bad idea unless VPN is what you want to test. If something didn't work, you won't know if it's the VPN or CUPC.

Same for the computers. Instead of testing on a computer with bunch of custom-installed software, you'd better test on a clear/fresh-installed computer. Stick with Windows XP. Stay away from Vista, unless you understand what is UAC, Windows Defender (or offender?), and other security "features".

Decrypt CUCM version numbers

2009-02-12T18:18:00.000-08:00

In an ideal world, version 6.x is better than 5.x, version 7.x is better than 6.x, and so on so forth. However, we're not in an ideal world.

Cisco builds different "trains" in parallel. Currently, the active trains for CUCM are 5.x, 6.x and 7.x.

This "multiple trains" approach is a compromise between market demands and compatibility. In order to support new features, big changes need to be made to the infrastructure (e.g. database schema). Sometimes, the changes are so big that it's impossible to be compatible with previous versions. So they introduce a totally different "train" to lower the risk.

It's really hard to tell which train is the "best". Of course, newer train would have more features. But they also have more requirements. For example, CUCM 6.x is compatible with CUPS 6.x and 7.x. But CUCM 7.x is only compatible with CUPS 7.x.

On each train, there are many "sub-versions". For example, on 6.x train, you have 6.1.1, 6.1.2 and 6.1.3. Read the release note carefully. Some versions won't be able to upgrade to another train. For example, CUCM 6.1.3 won't be able to upgrade to 7.0.x (because of different database schema)

On each sub-version, there are also "build-numbers". e.g. 6.1.2.1000, 6.1.2.2000, etc. Build-number is the most confusing part.

Generally speaking, build numbers should increase in 1000, such as 6.1.2.1000, 6.1.2.2000, etc.

CUCM is built on Linux OS. Whenever Cisco release an OS security patch, they'll increase the build number by 1000. This is called PSIRT patch.

Remember CUCM is an application running on Linux. OS patch does not contain any CUCM bug fixes. Any bug fixes would be in ES (Engineering Special). ES versions would be indentified by the last three digits in build numbers (e.g. 6.1.2.1112)

OS team and CUCM (application) team are two different teams. When the OS team release OS patches, they don't include any application patches at all. But the version number was increased by 1000.

Quiz: 6.1.2.2000 and 6.1.2.1112, which one is "better"?

Answer: it depends on how you define "better". But most of the people would think "less buggy" is better. When they say "less buggy", they mean "less bugs in CUCM". If that's the case, 6.1.2.1112 is better. Because it has ES number of 112, which means it fixed quite a lot bugs. While 6.1.2.2000 has no CUCM bug fixes at all (it contains OS patches though).

Confusing enough? I don't know which genius invented this version schema. But that's the way it is. If you try to "upgrade" 6.1.2.1112 to 6.1.2.2000, it'll fail with some vague error messages. You have to open a TAC case to understand why it failed.

Interesting? Yeah, that's the way to keep TAC engineers' jobs. :)

NTP - Network Time Protocol

2009-02-08T06:23:00.001-08:00

NTP is critical in Cisco voice products. Time synchronization not only provides consistent time in trace files, but also a mandatory requirement for some components.

Architecture

On a CUCM publisher, you may choose to use internal clock (computer hardware clock) or external clock (NTP server, such as a router).

Regardless of your choice, all other servers in the cluster will use NTP protocol to synchronize time with publisher. In another word, NTP is only configurable on publisher.

Basic concepts

http://en.wikipedia.org/wiki/Network_Time_Protocol

Tips

1. Before you configure NTP on publisher, configure the local time as accurate as possible. This will shorten the time to synchronize after you configure NTP.

2. Be patient after you configured NTP. It might take hours to synchonize based on the time difference between publisher and NTP source. This works as designed. This is to comply with IETF RFC.

3. If NTP was configured on publisher, subscribers won't synchronize to publisher until publisher is in-sync with NTP source. If you're having problem sync the publisher to NTP source, but you want the whole cluster in-sync on time, disable NTP on publisher.

Frequently used commands

utils ntp status

ntpd (pid 3638) is running...

remote refid st t when poll reach delay offset jitter
==============================================================================
127.127.1.0 127.127.1.0 10 l 9 64 377 0.000 0.000 0.008
*171.68.10.80 64.103.34.14 2 u 921 1024 377 38.233 3.336 1.182
+171.68.10.150 10.81.254.202 2 u 988 1024 377 37.044 3.252 12.236

synchronised to NTP server (171.68.10.80) at stratum 3
time correct to within 60 ms
polling server every 1024 s

Current time in UTC is : Sun Feb 8 14:38:36 UTC 2009
Current time in America/Chicago is : Sun Feb 8 08:38:36 CST 2009

The output above tells you:
1. The box is synchronized to 171.68.10.80 at stratum 2.
2. Internal clock is at stratum 10 (the box won't synchonrize to any time source with stratum equal or greater than 10)

Other commands include:

utils ntp config
utils ntp restart
utils ntp start

Troubleshooting

utils network capture port 123

Executing command with options:
size=128 count=1000 interface=eth0
src= dest= port=123
ip=

08:56:01.125718 cm6-sub.ntp > cm6-pub.ntp: v4 client strat 4 poll 10 prec -18 (DF) [tos 0x10]
08:56:01.125965 cm6-pub.ntp > cm6-sub.ntp: v4 server strat 3 poll 10 prec -17 (DF) [tos 0x10]
08:56:18.270720 cm6-pub.ntp > ntp-sj1.ntp: v4 client strat 3 poll 10 prec -17 (DF) [tos 0x10]
08:56:18.308956 ntp-sj1.ntp > cm6-pub.ntp: v4 server strat 2 poll 10 prec -18
08:57:24.271526 cm6-pub.ntp > ntp-sj2.ntp: v4 client strat 3 poll 10 prec -17 (DF) [tos 0x10]
08:57:24.309282 ntp-sj2.ntp > cm6-pub.ntp: v4 server strat 2 poll 10 prec -16

Port 123 is NTP port. The output above shows the incoming/outgoing NTP packets on publisher:
1) cm6-sub is the NTP client on stratum 4
2) cm6-pub is the NTP server on stratum 3 (because the external NTP source is on stratum 2)
3) ntp-sj1 and ntp-sj2 are the external NTP source on stratum 2

NTP logs

Use RTMT to get "ntp logs".

Troubleshooting time offset on phones

If the time on CUCM server was correct, but the phones showed wrong time, it's most likely due to misconfiguration.

First of all, we need to understand the difference between UTC time and local time.

There are many different time zones in the world. In US, we have EST, CST, MST, PST, etc. 8AM EST means 7AM CST. Daylight saving also adds more complex to this. Different countries have different daylight saving cutoff dates.

To provide consistency around the world, NTP server feeds UTC (GMT) time to clients. How to manipulate it to get "local time" would be the client's responsibility.

On CUCM Admin > System > Date/Time Group, you may configure different groups to reflect different time zones. Then you may associate date/time group to different device pools. Hence, different phones in different device pools can have different local time.

One thing to notice is:
The "old" phones (7940/7960) get local time from CUCM server.
The "new" phones (7941/7961 and newer) get UTC time and time zone info from CUCM server. Then they do the math and display the local time.

Use Windows server as NTP source

Depending on your Windows version, there are some registry settings you need to set:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\NTPServer\Enabled
Changing the ‘Enabled’ flag to the value 1 enables the NTP Server.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Parameters\Type
Change the server type to NTP by specifying ‘NTP’ in the ‘Type’ registry entry.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config\AnnounceFlags
Set the ‘Announce Flags’ registry entry to 5, to indicate a reliable time source.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config\LocalClockDispersion
Set 'LocalClockDispersion' to 0

The last one is most important one.

After changing registry, you need to restart "Windows Time" service.

Integration between CUPS and MOC/OCS

2009-02-04T18:27:00.000-08:00

For now, to integrate MOC (Microsoft Office Communicator) with Cisco IP phone system (CUCM) you need CUPS for phone presence and phone control.

Phone presence

Phone presence info will flow like this: CUCM -> CUPS -> OCS -> MOC.

Phone control

Phone control info will flow like this: MOC -> OCS -> CUPS -> CUCM.

Best Practices

Use TCP instead of TLS on your first deployment. TLS/Cetificates are not something fun to play with. They are optional for the integration. Per the KISS (Keep It Simple, Stupid) principle, don't mess with TLS unless you have to.

Let's talk about phone control first. Currently, Cisco support RCC (Remote Call Control) for MOC.

RCC was configured in Active Directory Users and Computers (ADUC) > Communications.

As you can see from the picture above, we need to configure Server URI and Line URI.

6002 is the IP phone DN (Directory Number).

htluo-cups is the proxy domain that will process the request. We'll further discuss this part later.

When MOC starts up, it'll send "INVITE 6002@htluo-cups6" to OCS.

When OCS receives the INVITE message, it'll try to route it to the right destination (CUPS in this case).

How OCS routes the message is more complicated than it looks like. It could be static route, it could be DNS lookup. For more details see "SIP domain and DNS domain".

Again, per KISS principle, it's recommended to use static route on OCS to eliminate any misconfiguration of DNS.

As shown above, the static route means "for any SIP message with domain htluo-cups6, forward it to 10.88.229.209, port 5060, with TCP protocol".

In order for CUPS to accept this message, OCS' IP address must be added to CUPS Incoming ACL. (or you may configure an "ALL" incoming ACL)

When CUPS server receives the message from OCS, the first thing it does is to determine if message has reached the final destination. CUPS compares its own configuration with the domain portion of the SIP message. If the domain portion of the SIP message matches one of the following, CUPS would think the message arrives at its final destination and take care of that.

a) SIP domain name (configured under CUPS Admin > System > Service Parameters)
b) CUPS node name (configured under CUPS Admin > System > Server)
c) node name + SIP domain
d) other alias name configured on CUPS

To see a full list of alias names, set "SIP Proxy" trace to detail. Restart SIP Proxy service. SIP Proxy would write a list of alias names to trace files during startup.

If CUPS decided to take care of the INVITE message from OCS, it will do the following:

1) Determine if the MOC user has permission to control the phone
2) If step 1 was ok, open a CTI request to CUCM CTIManager
3) If step 2 was successful, return "200 OK" SIP message to OCS

Determine if the MOC user has permission to control the phone

In the Server URI (tel:6002;phone-context=dialstring;device=SEP001E7A24429A), if a device name was specified (which was in this case), CUPS will check if that device was in the "Controlled Devices" list on CUCM Admin > User Management > End User.

If no device name was specified in Server URI, CUPS will try to find the device by DN. For details, please see: http://www.cisco.com/en/US/docs/voice_ip_comm/cups/6_0_1/install_upgrade/deployment/guide/dgmsint.html#wp1049685

Again, per KISS principle, you'd better specify device name in Server URI on your first deployment.

Open a CTI request to CUCM CTIManager

CUPS will open a CTI request to CUCM CTIManager with the credential configured on CUPS Admin > Application > CTI Gateway > Settings (CUPS 6.x).

Of course, the credential needs to exsit on CUCM > User Management > Application User. It needs to be in "Standard CTI Enabled" and "Standard CTI Allow Control of All Devices" groups.

And of course, the phone device needs to be registered to CUCM.

If all above was successful, CUPS will send "200 OK" to OCS as an response to the INVITE.

At this point, CUPS has done its job. But it does not necessarily mean MOC gets phone control.

In order for OCS to accept the "200 OK" message from CUPS, CUPS' IP address must be added to OCS "Host Authorization" (please note, it's IP address, not hostname or FQDN).

Don't forget to restart OCS Front End services after making changes.

The best tool to debug OCS/CUPS integration is on OCS. Right-click on the pool > Logging Tool > New Debug Session. Choose SIP Stack. Optionally, you may filter by the MOC user ID in the filter settings.

Known caveats:

1. CUPS sent "200 OK" for the INVITE. But MOC still not getting phone control.

This is because OCS doesn't trust CUPS. OCS SIP stack log will show "SIPPROXY_E_INVALID_RECORD_ROUTE"

Resolution: Check "Host Authorization" on OCS.

2. Load balancer

If you have load balancer for OCS, more likely than not, you will run into "one-way phone control" issue. The symptom is: you can make phone calls from MOC. But the call status was not updated on MOC. For example, when the call was connected, MOC still showing "calling".

This problem was caused by misconfiguration of load-balancing.

When OCS sends message to CUPS, it doesn't go through load-balancer (based on your exsiting configuration).

When CUPS tries to reply to OCS, it looks up DNS and DNS resolve the pool name to the load-balancer virtual IP. So the traffic goes through load-balancer to get to OCS. When OCS received the message, the last hop was load-balancer. However, the load-balancer didn't add its IP to the SIP header. OCS will reject this message and send "400 Missing correct Via header" to CUPS.

Resolution:
Check your load-balancer, see if it's capable of modifying SIP header. Or contact Microsoft to see if they can turn off the "Via header" check on OCS.

How does CUPC determine presence address

2009-01-28T10:34:00.000-08:00

In order for CUPC to receive presence information, it has to connect to a presence node.

Just to remind you again, "presence node" is NOT necessarily the same node as the logon server. For example, you have two CUPS servers - CUPS-A and CUPS-B. The logon server could be CUPS-A, but the "presence node" could be CUPS-B.

Even in a single-node environment, CUPC still has to run through a set of rules to determine the "presence address".

If you ever looked at CUPC > Show Server Health (or CUPC logs), you'll notice two parameters: "Presence.Primary.Address" and "Presence.Domain".

"Presence.Primary.Address" is the node name of the server. You may find this under CUPS Admin > System > Servers.

"Presence.Domain" is the SIP proxy domain. You may find this under CUPS Admin > System > Service Parameters > Cisco UP SIP Proxy.

If the node name was configured as dotted IP address (e.g. 192.168.1.100), CUPC will use that as the presence address.

If the node name was non dotted IP, things are more complicated. Here are the rules CUPC uses to determine the presence address:

When you specify a non IP string for the node name, CUPC looks at the string to see if it already ends with the domain value -- if not, it appends it. Examples:

1)
Proxy.Domain = cisco.com
Presence.Primary.Address = cup1

Registration will be to user@cup1.cisco.com (hostname, domain is appended)

2)
Proxy.Domain = cisco.com
Presence.Primary.Address = cup1.cisco.com

Registration will be to user@cup1.cisco.com (FQDN, domain was already included)

3)
Proxy.Domain = cisco.com
Presence.Primary.Address = 10.11.12.13

Registration will be to user@10.11.12.13 (address was dotted-decimal)

4)
Proxy.Domain = 10.11.12.13
Presence.Primary.Address = 10.11.12.13

Registration will be to user@10.11.12.13 (address was dotted-decimal)

5)
Proxy.Domain = 10.11.12.13
Presence.Primary.Address = cup1

Registration will be to user@cup1.10.11.12.13 (hostname, domain is appended) WRONG

6)
Proxy.Domain = 10.11.12.13
Presence.Primary.Address = cup1.cisco.com

Registration will be to user@cup1.cisco.com.10.11.12.13 (FQDN, domain is appended) WRONG

7)
Proxy.Domain = PROXY_DOMAIN_NOT_SET
Presence.Primary.Address = cup1.cisco.com

Registration will be to user@cup1.cisco.com.PROXY_DOMAIN_NOT_SET (FQDN, domain is appended) WRONG

The last 3 of these are wrong (there are other permutations) because the domain portion of the URI is neither an IP address nor DNS-resolvable FQDN. While the proxy might be content to match the strings, CUPC's SIP stack needs to resolve to a routable address.

Please note everything is case sensitive. If node name was cup1.cisco.com, proxy domain was CISCO.COM, presence address will be "cup1.cisco.com.CISCO.COM", which is also WRONG.

Mysterious "Invalid Crdentials" on CUPC

2009-01-26T10:48:00.000-08:00

On CUPC > Help > Show Server Health, sometimes you would see failed items with the message "invalid credential", such as "Presence", "Desk Phone", or "Voicemail".

This is very confusing. Since you already logged into CUPC, why it's giving you "invalid credential"? What kind of credential it was failing on?

Before we can move further, please take a look at "CUPS and CUPC, father and son? or not".

CUPS and CUPC's relationship is not as tight as you thought. CUPC has many features, but CUPS is only relevant in two of them (configuration repository and presence).

When you type username and password on CUPC login window, that is majorly for "Configuration Repository". If you typed in the wrong password, CUPC won't be able to download configuration from CUPS. No other functions CUPC can perform without configuration.

However, sucessfully downloading configuration does not guarantee other functionalities. To use other fucntions, a 2nd authentication might be required (either explicitly or implicitly).

Presence - Invalid Credential

For presence feature, 2nd authentication is required on SIP layer. This authentication is implicit. For more details on "Digest Authentication", please see http://www.ietf.org/rfc/rfc3261.txt.

Why is it implicit? Why does it fail?

To make it implicit is Cisco development's decision. If they made it explicit, you'd have to provide digest credential (2nd password) after login. This could be annoying since SSO (Single Sign On) was what we preferred.

So Cisco development made CUPS/CUPC worked this way:
1) You (system admin) configure digest credential on CUCM Admin > User Management > End User page.
2) CUPS synchronizes digest credentials from CUCM to CUPS.
3) CUPS transmits digest credential to CUPC during logon.
4) CUPC uses that degest credential to authenticate with SIP proxy.

Step 3 and 4 look funny because it's like a door keeper gives the key to you and asks you open the door with the key. But keep in mind:
a) The "door keeper" acutally verified your identify (username/password), before giving you the key.
b) The key was encrypted during transmission.
c) The key door keeper gave you might be for a different door (SIP proxy could be on a different server other than the logon server)
d) This is a compromise (or balance) between inconvenience of SSO and SIP protocol requirements.

If there's no digest credential configured on CUCM (ie. it's blank), you'll get "Invalid Credential" for presence. To fix it, take one of the following options:

Option 1: Go to CUCM Admin > User Management > End User, configure a dummy value for "digest credential". It could be any value. Why? See workflow explained above.

Option 2: Go to CUPS Admin > Cisco Unified Presence > Proxy Server > Incoming ACL. (on CUPS 7.x, it's "System > Security > Incoming ACL". Configure an address pattern that covers your CUPC machines. For example, a "all" pattern matches all machines.

This option is considered less secure, because any machine in that address pattern (subnet) would be able to connect to SIP proxy without digest authentication challenge.

Option 3: Go to System > Service Parameters > Cisco UP SIP Proxy. Set "Authentication Module" to "off". This is the least secure option, which turns off SIP authentication at all.

Desk Phone - Invalid Credential

This usually happens when CUCM was configured to use "LDAP Authentication".

To control desk phone from CUPC, CTI protocol was used. Before a CTI client (CUPC) can control the phone, it needs to authenticate with CTI server (CTIManager). This authentication is implicit. CUPC would use the same logon username/password to authenticate with CTIManager. CTIManager, in turn, would authenticate that with LDAP.

Question: Why the authentication would fail?
Answer: In short, this is a bug on CUCM.

Question: Any workaround for that before we can upgrade CUCM?
Answer: On CUCM, change LDAP authentication port to 3268 and restart CTIManager.

Question: Why it would fix the problem?
Answer: When LDAP referral happens, CTIManager would fail on authentication. Using 3268 (Global Catalog) port eliminate LDAP referals.

Question: Why it only affects CUPC?
Answer: CUPC is the only application (so far) that uses end user credential to authenticate with CTIManager.

Voicemail - Invalid username/password or account locked

Depending on what Unity edition you're running (Unity or Unity Connection), the cause could be different.

Before moving on, please take a look at "How to test IMAP connection".

On Exchange 2007, it's because IMAP login was disabled on TCP (port 143) by default.

On Unity Connection, make sure you reset "Web Application Password" instead of VoiceMail password.

Decrypt HTTPS traffic with Wireshark

2009-01-24T12:08:00.000-08:00

Wireshark is a useful tool in troubleshooting. However, if the traffic was encrypted (such as https between CUPS and Exchange), it's unless you can decrypt it.

Look at packet 11 in sniffer capture above. Application data was encrypted. There's not too much useful data in it.

To decrypt this data, we need the "private key" of the server certificate. You cannot get the private key from client side (such as web browsers). To get the private key, you need access to the server.

Step 1. Export the server certificate with private key

1-1: Go to IIS Admin > Right-click "Defautl Web Site" > Properties > "Directory Security" > "View Certificate".

1-2: Go to "Details" tab > "Copy to File" > Choose "Yes, export the private key"

1-3: You'll save the file in PKCS #12 (.PFX) with all three options UNCHECKED

1-4: You'll have to provide a password to protect the file. Because private key is a very sensitive information.

1-5: Save the file (system will add ".pfx" extension to the file name)

Now we have a PKCS #12 file (.pfx file).

Step 2: Extract the private key from .pfx file

openssl pkcs12 -in test.pfx -nocerts -out privateKey.pem -nodes

The command above take "test.pfx" as the input file, extract the private key, save it unencrypted in "privateKey.pem" file. You'll be asked for the password (where you entered on step 1-4).

Where to find openssl? Google!

Step 3: Go to Wireshark > Edit > Preferences > Protocols > SSL. In "RSA keys list", type the following:

10.88.229.196,443,http,C:\privateKey.pem

Where "10.88.229.196" is the server IP. "443" is the port number (HTTPS). "http" is the protocol you want Wireshark decode to. "C:\privateKey.pem" is the file name of the private key. "SSL debug file" is optional.

Step 4: Once you click OK, you'll notice the changes on Wireshark screen. Now the data was decrypted!

CUPS Calendar integration

2009-01-23T08:25:00.001-08:00

Calendar integration is probably the most mysterious part of CUPS.

There are many catchas in Calendar integration due to the following:

1) CUPS uses WebDAV protocol to query calendar, which is a pretty old-school protocol and has many limitations. Microsoft recommend developers use EWS (Exchange Web Service) for better compatibility and features. Cisco will eventually change to EWS. But no ETA yet.

2) CUPS requires HTTPS (TLS/SSL) connection between CUPS and Exchange. This adds more complexity to the picture because you have to deal with CA, certificates, etc.

3) Exchange authentication and permissions
Exchange (OWA/IIS) has many authentication methods (FBA, classic, NTLM, etc.). Exchange also has two different sets of permissions: AD permission and Mailbox permission.

It's impossible to elaborate all scenarios in this blog. But here are some recommendations:

1) Avoid "2003/2007 mixed mode"
More likely than not, it's not going to work with CUPS due to limitation of WebDAV.

2) Avoid any firewall (especially MSFT ISA server) between CUPS and Exchange.

3) Avoid any load-balancer or traffice directory between CUPS and Exchange.

4) Try to disable FBA (Form Based Authentication) for troubleshooting purpose.

5) If you don't care about certificates, use "makecert.exe" utility to create self-signed certificate for Exchange. See http://www.lulu.com/content/5552336 for details.

6) Make sure you set your meeting status to "Busy" for testing. Make sure you set it to "whole day event" to avoid time zone glitches. (be aware, when you set it to "whole day event", the status in outlook will be "FREE" by default)

SIP domain and DNS domain

2009-01-21T10:40:00.000-08:00

If you deal with SIP products (such as CUPS/CUPC), you'll have to deal with SIP domain sooner or later.

Here are some of the most asked questions:
1) What is a SIP domain?
2) Does the SIP domain has to match the DNS domain?
3) What if I'm not using DNS with the application?

SIP domain is more like on application layer versus DNS domain on network layer.

Let's take a look at a real life example. Let say, you have a SIP application that can send instant message and make phone calls (such as CUPC and MOC).

When the application initiate a call, the SIP message would look like this "INVITE 6002@acme.com". This message means - "I want to call extension number 6002 in ACME company".

Usually, the first stop of this SIP message would be your local proxy server (SIP proxy). The local proxy server will determine how to route this message to its destination.

Whenever a SIP proxy server receive a SIP message, it will always look at the domain part of the SIP request. Based on the domain, SIP proxy will determine how to route the message.

Here's the detailed workflow:

1) If the SIP message's domain matches with the SIP domain configured on SIP proxy, SIP proxy will handle that within the same domain. Otherwise, SIP proxy will forward it to a different domain (or just ignore/discard it depends on the design).

This is very important. Let say, CUPC sent a message "INVITE johndoe@acme.com" to SIP proxy. However, the SIP domain configured on SIP proxy was "abc.com". The SIP proxy will treat it as "foreign message" and try to forward it to its destination domain or discard/ignore it.

2) SIP proxy will look at the SIP static routes to determine the messages' destination.

SIP static routes are configured on the SIP proxy server on application layer. Don't confuse it with TCP/IP static routes.

On CUPS 6.x, you may configure static routes on CUPS Admin > Cisco Unified Presence > Proxy Server > Static Routes. On OCS 2007, you may configure static routes by right-click on the "Front End" folder > properties > routing.

You may configure IP address or FQHN for static routes.

For example:

Static route: acme.com ---> apple.acme.com
This means for all SIP messages with SIP domain acme.com will be routed to a host with FQHN apple.acme.com.

Static route: acme.com ---> 192.168.1.100
This means for all SIP messages with SIP domain acme.com will be routed to a host with IP address 192.168.1.100.

Keep in mind, the above is on application layer.

3) If there's no static route configured, SIP Proxy would try to determine the next hop by name resolution.

If SRV records were configured, SIP proxy would try to resolve the domain by SRV records.

Then SIP proxy would try to resolve by A records.

Now lets answer the questions.

Q1: What is a SIP domain?
Answer: A SIP domain is an application layer configuration that define the management domain of a SIP proxy.

2) Does the SIP domain has to match the DNS domain?
Answer: Yes, it has to match the DNS domain in most of the scenarios. And it's strongly recommended to match the SIP domain with DNS domain.

3) What if I'm not using DNS with the application?
Answer: You may or may not be able to use application features if you don't have appropriate name resolution configured. For example, in CUPC 1.2.x, you may use presence feature without name resolution to the presence FQHN. On CUPC 7.0.x, it doesn't work (because they changed the design on 7.0.x). If you don't have a DNS, you may use local host files to do name resolution.

To be continued...

LDAP account keeps locking out on CUPC

2009-01-19T20:57:00.000-08:00

CUPC has the capability to search LDAP. So you can easily add contacts to your CUPC contact list.

In order to search LDAP, the application (CUPC) has to authenticate with LDAP first. A service account was used. This service account was configured in CUPS Admin > Application > Cisco Unified Personal Communicator > LDAP Profile.

Once this service account was locked out, none of the CUPCs would be able to search LDAP.

The strange thing is: as soon as you unlocked the account, it got locked up pretty soon. By looking at Windows Event Viewer (Security Log), you'd see the source was the CUPC computer. You changed the password in LDAP, and changed it on CUPC. But the account still got locked up.

Now you got confused. Since you already "refreshed" the password, why the account still got locked up?

The answer is: the "refresh" didn't get populated to CUPC. Some of the CUPCs were still trying the LDAP with old (wrong) password.

When you change the LDAP profile on CUPS, CUPC didn't get the updated profile (password) until next logon. It'll keep trying LDAP with the old password and keep locking out the account.

To solve this problem, you'll have to logout all CUPC before you unlock the LDAP account.

Sometimes, this is "mission impossible" in a large network where you have hundreds of users.

The workaround is:

1) Create a 2nd LDAP account.

2) On CUPS, update LDAP profile to use the new account.

On step2, you have to make sure you put in the correct information in one shot. If you misconfigured something and tried to correct that, chances are some CUPC might get the wrong info before you corrected it. The loop starts again.

LDAP Integration with CUPS

2009-01-15T20:17:00.000-08:00

There's no GUI on CUPS to configure "LDAP Integration".

CUPS synchronizes users from CUCM. CUPS does not sychronize with LDAP.

How about "CUPS Admin > Application > Cisco Unified Personal Communicator > LDAP server"? Well, as the menu inidcated, it's a configuration for CUPC. It's not a configuration for CUPS. CUPC download this configuration upon login and use that to query LDAP (for contact search).

Even though LDAP integration is not configurable on CUPS, it might affect CUPS in an unexpected way.

Scenario 1: CUCM is using LDAP authentication. CUPC user not able to login.

When CUPC users try to login, the username/password was authenticated aginst CUPS (via SOAP). In turn, CUPS will authenticated the username/password against LDAP (because CUCM is using LDAP authentication). If for whatever reason, CUPS having problem with LDAP, authentication would failed.

Reason A: Access-list blocked traffic between CUPS and LDAP. Access-list allow traffic between CUCM and LDAP. Access-list allow traffic between CUPS and CUCM.

In this case, CUPS would synchronize CUCM users over. But those users won't be able to log into CUPC. Because CUPS cannot talk to LDAP.

Reason B: CUCM was configured to use LDAP over SSL (LDAPS). Certificates were uploaded to CUCM but not to CUPS.

In this case, CUPS won't be able to set up the LDAPS connection because the certificates were missing.

Frequently asked questions:

Do I need CUCM/LDAP integration to use CUPS/CUPC?

Answer: No. But there are some catchas. CUCM/LDAP integration is highly recommended.

Explanation:

1) Without LDAP, adding contacts to CUPC would be a pain. Because CUPC cannot search CUCM for contacts. (there's no such a thing called "CUCM native directory")

2) If you use LDAP with CUPC but not CUCM, potential problem is on the way. If you search LDAP from CUPC and add that contact to CUPC contact list, user ID in LDAP would be used as "presence user ID". If the "presence user ID" is different with the user ID in CUCM, you won't be able to see your contact's presence (your contact would always be offline).

Workarounds:

1) Make sure the user ID in LDAP matches the user ID in CUCM.

2) If the CUCM ID happens to be the phone extension number. And you happen to have the phone extension number configured in LDAP. You may go to "CUPS Admin > Application > Cisco Unified Personal Communicator > Settings" to map the LDAP phone attribute to the CUPC "user ID" attribute.

LDAP integration with CUCM 5.0 or above

2009-01-14T19:51:00.000-08:00

On CUCM 5.0 or above, LDAP integration actually consists of two parts:

Part 1: LDAP Sychronization.

Part 2: LDAP Authentication.

You may enable part 1 without part 2. But you cannot enable part 2 without part 1.

In either scenario, CUCM never, ever synchronizes passwords from LDAP.

Part 1: LDAP Sychronization

When LDAP synchronization was configured, CUCM will import user accounts from LDAP and create users in CUCM database. As mentioned before, CUCM won't import passwords. CUCM will import user ID, first name, last name, etc.

Any user that pre-exists in CUCM user database but not exists in LDAP will be deleted. CUCM will mark them as "delete pending" and give them 72 hours grace period. After 72 hours, those "delete pending" accounts will be deleted permanently and no way to recover.

Part 2: LDAP authentication

Whenever a CUCM user need to authenticate, CUCM will authenticate the password against LDAP.

Commonly seen problems:

1) Some user accounts in LDAP didn't synchronize over to CUCM

Usually, it's because some 'critical' attributes were missing. For CUCM, "last name" is a critical attribute. If the LDAP account does not have last name configured, it won't synchronize over to CUCM.

2) Slow synchronization

If you have a large active directory, use Global Catalog port (default is 3268) is recommended.

3) LDAP over SSL

After uploading the certificate to CUCM for LDAP over SSL, you need to restart Cisco Tomcat service to take effect.

CTI vs. Presence

2009-01-10T07:37:00.000-08:00

When talking about "phone presence", people are always confused with "line status".

CTI (Computer Telephony Interface) has been around for a while. It allows computer software control phone devices. So you can make calls, answer calls, tranfer calls, etc.

The software that controls the phone is usually referred as "CTI client". More often than not, there's a "CTI server". In Cisco products, CTI clients include Attendant Console client, Cisco Agent Desktop, etc. CTImanager on CUCM (CallManager) is the CTI server.

When a phone was controlled by CTI, the phone's staus is usually reported to CTI client. For example, when you pick up the handset on the phone, CTI client would display "off-hook" or "dial tone" on the screen. We usually refer this as "CTI status" instead of "Presence".

You can only see CTI status for the phones you controlled. You won't be able to see CTI status on other phones. CTI is more like a protocol to control the phone.

"Presence" is a rather new term comparing with CTI. "Presence" is usually referred to the SIP extension SIMPLE (Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions).

When using "Presence" with phones, we can see the status of the phone (off-hook, on-hook, do-not-disturb, etc.). You don't necessarily have control of the phone. In face, "presence" won't provide any function to control phone devices.

Keep that in mind when you troubleshoot clients (such as CUPC, MOC, etc.)

For example, when you make a call from your phone, MOC would display the phone status (CTI status). However, you might not be able to see the phone status of your contacts on MOC contact list. That part was "presence".

How to test IMAP connection

2009-01-06T20:23:00.000-08:00

IMAP is a protocol to retrieve email from mail server. With appropriate settings, Cisco Unity allows application retrieve voicemail via IMAP. CUPC is one of the clients that uses IMAP to retrieve voicemail.

There are two Unity families - regular Unity (which works on Windows and Exchange) and Unity Connection (a Linux appliance).

For regular Unity, MS Exchange is required as mail store. CUPC should always point to Exchange. But the GUI on CUPS refers this as "Unity Server". This is pretty confusing, especially in "Unfied Messaging" mode, where Unity server and Exchange are actually two different servers.

By default, IMAP service is disabled on Exchange. You'll have to go to Windows services to set the startup type to "Automatic" and start it. Please note the following difference between Exchange 2003 and 2007:

Exchange 2003 use IMAP over TCP (port 143) by default.

Exchange 2007 use IMAP over SSL (port 993) by default.

The bad thing about SSL is you cannot test it with telnet command. We'll talk about the workaround later.

For Exchange 2003, here's how to test the IMAP connection and login.

Each command begins with a number and a space. Though you may use any numbers and you may reuse the number, it's recommended you use unique, sequential numbers.

The first command we want to use is "capability". Pay attention to "LOGINDISABLED" in screen output. You usually saw that on Exchange 2007, which means LOGIN is disabled on TCP (available on SSL only). On Exchange you normally won't see that.

If you don't see "LOGINDISABLED", you may continue the test.

The next commend is "login JohnDoe pass123", where JohnDoe is the username you want to test and pass123 is the actually password. If you got a "OK", that means the username and password was correct.

As mentioned about, on Exchange 2007, LOGIN on TCP was disabled by default. To enable it, you run the following commands on in Exchange Management Shell:

Set-IMAPSettings –LoginType 1

Resart IMAP service after that.

Currently, CUPC does not support IMAP referral. Which means, if you have multiple mailbox servers, you have to enable IMAP service on each one and create multiple profiles on CUPS.

Name resolution in Cisco voice products

2009-01-05T20:07:00.000-08:00

In Cisco documentations, there's no specific requirements for name resultion. You may install Cisco voice products (such as CUCM, CUPS, CER) without worrying about name resolution. The reason was - there's a process on those products called "ClusterManager'.

ClusterManager is the process takes care of IPSec communication between nodes in a cluster. ClusterManager also takes care of host table (/etc/hosts). Whenever a node was added to the cluster, ClusterManager will add the hostname into host table. Hence every node in the cluster should be able to resolve each other's hostname without DNS involved.

Before we moving on, let's get familiar with some terminologies.

Hostname - The name for a computer. e.g. ccm-1. This name was entered during installation and used as the default "node name". To show the host name, use CLI command "show status".

Node name (or process node name) - The name for a voice appliance(application). By default, the node name was set to the hostname entered during installation. You may change the node name to any value. But the hostname stays the same. To show the node name, use CLI command "run sql select * from processnode".

Domain name - A name for a domain (a group of computers). e.g. sales domain, support domain, etc. A domain name could be ambiguous. For example, a sales domain could mean the sales domain in Asia Pacific or the sales domain in Europe. To eliminate this ambiguity, we want to use Fully Qualified Domain Name (or FQDN).

Fully Qualified Domain Name (or FQDN) - For example, sales.apac.acme.com. is a FQDN. (Technically, every FQDN should have a dot (.) at the end. The dot means the root. Without the dot, it's not a FQDN, it's a partial domain.)

Fully Qualified Host Name (or FQHN) - a FQHN consists of a hostname and a FQDN. For example, a FQHN ccm-1.acme.com. consists of a hostname ccm-1 and a FQDN acme.com.

For the reason mentioned above (to avoid ambiguity), all DNS query should use FQHN or FQDN. If a hostname was used, it's the client's responsibility to translate that into a FQHN. For example, on client PC, we do a "nslookup ccm-1". Windows actually translates "ccm-1" into "ccm-1.acme.com" before sending out the query to DNS server. A sniffer capture could prove this.

The key point is - whether the client can translate the hostname into a correct FQHN.

This translation is controlled by the "DNS suffix" configuration on client side. Client computers usually got this from DHCP. To see the DNS suffix, use "ipconfig/all" command in Windows command prompt.

Though we don't have to worry about name resolution on voice servers (CUCM, CUPC, etc.), we do have to take end points (clients) into consideration. End points might request hostname even though you thought you were using IP address.

IP Phones.

Let say, you just finish installing a CUCM 6.1.2 server (named ccm-1). You registered Cisco IP phones to the CUCM. You can make make/receive phone calls. Everything looks good, right? Right, until you press the "Directory" button on IP phone.

When you press the "Directory" button on IP phone, the phone actually acts like a web browser and initate a http request to http://ccm-1:8080/ccmcip/xmldirectory.jsp. If the phone cannot resolve ccm-1 into an IP address, it won't be able to make that connection.

"ccm-1" is the node name. By default, it's the hostname you entered during installation. You may change the node name on CCMAdmin page > System > Servers. Changing the node name does not change the hostname.

Application Clients (such as Attendant Console client)

On Attendant Console client logon, you specified IP address of the CUCM server. You successfully logged in. But you were not able to control the phones. Looking at log files, you realized that Attendant Console client was trying to initiate the JTAPI connection to CUCM server by node name (ccm-1). Adding ccm-1 to C:\Windows\system32\drivers\etc\hosts on client computer solved the problem.

CUPC (Cisco Unified Personal Client)

Things become more insteresting (complicated) when SIP comes into play.

On CUPC 7.x, some customer noticed an annoying behavior. On the first logon, user specified HQHN of CUPS server (e.g. cups1.acme.com) as the login server. CUPC logged in successfully. Exit and re-launch CUPC. User noticed that the login server field was automatically poplated with the CUPS node name (e.g. cups1). Since "cups1" was not resolvable from the client computer, logon failed. User changed the login server to FQHN cups1.acme.com. Logon succeeded. But on next logon, it changed to hostname again.

This annoying behavior is actually a new feature in CUPC. We might have multiple CUPS servers in a cluster. To better load-balancing between those servers, we specify a "home node" for each user. A user's request should designated to his "home node" (unless the "home node" is down). On the initial logon, CUPC gets "home node" information from CUPS. CUPC will try to use "home node" as logon server afterwards. This is why the login server field was populated with the node name (by default, it's the hostname).

You might want to ask "why does the client computer can resolve FQHN but not hostname?". Please refer to the earlier section of this article. It was because DNS suffix was "incorrect". I put a quote around "incorrect" because sometimes, it can't be fixed.

Scenario #1. The client computer is on a offsite location (mobile user, VPN, etc.), where the DNS suffix from DHCP is not the same as the corporate domain. (e.g. "isp.com" instead of "acme.com")

Scenario #2. Customer has multiple subdomains (e.g. "apac.acme.com" and "emea.acme.com"). The client computers are in different subdomains.

Workarounds (pick one below):

#1 On CUPC side, manually change the login server to FQHN on each login.

#2 On CUPC side, add the hostname to local hosts table (C:\Windows\System32\drivers\etc\hosts)

#3 On CUPS side, change the node name to FQHN.

#4 On CUPS side, change the node name to IP address.

CUPS and CUPC - Father and Son? Or not?

2009-01-01T12:39:00.000-08:00

Like many client/server models, people would think CUPS and CUPC are like father and son, like Exchange and Outlook.

However, CUPS and CUPC are more like peers. They were designed by two different groups of developers. These two groups treat each other as an "important customer", but not the only customer.

For example, CUPS can be used without CUPC.

Example #1: MOC RCC (Microsoft Office Communicator Remote Call Control).

MOC can control an IP Phone via CUPS and CUCM without CUPC in the picture.

Example #2: SIP Proxy

CUPS acts as a SIP proxy without CUPC in the picture.

Example #3: IPPM (IP Phone Messenger)

An IP phone can see buddy's status, send IMs, retrieve meeting schedules without CUPC in the picture.

On the other hand, CUPC was built as a multi-function software other than just a "CUPS client".

Right now, CUPC can't function without CUPS. Because it uses CUPS as a configuration repository. CUPC has to retrieve the configuration from CUPS for all its features. But except for the presence feature, all other features do NOT rely on CUPS.

For example, CUPC's soft phone feature is most demanded from Mac OS users. It's the only Cisco soft phone works on Mac OS. (CIPC - Cisco IP Communicator does not run on Mac). However, to use CUPC, you need a CUPS server. In this scenario, CUPS server provides no value other than supplying the TFTP address to CUPC. If CUPC can save the TFTP address locally (like CIPC), there's no need for CUPS server. It makes perfect sense from both technical and economic perspective.

Same for other CUPC features like Voicemail, Web Conference, LDAP search, click to dial, etc.

For this reason, CUPC developers are considering the options of "separating CUPC from CUPS" and make them more independent to each other.

With that said, CUPC and CUPS are more like peers other than father and son (or couple). Keep that in mind would help you understanding the function modules of CUPC.

Cisco Presence Products

2009-01-01T11:55:00.000-08:00

When talking about Cisco presence products, we usually refer to BLF (Busy Lamp Field) and CUPS (Cisco Unified Presence Server).

BLF is a built-in feature of CUCM (CallManager). It allows you to see a person's phone status (off hook / on hook) on a speed dial button or in phone directory.

BLF: Pone 6001 is on-hook (idle)

BLF: Pone 6001 is off-hook (active)

For more information regarding CUCM presence, please see: http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/admin/6_1_1/ccmfeat/fsprsnc.html

CUPS (Cisco Unified Presence Server) is a standalone products. It requires a separate server, it cannot coresident with CUCM. For supported server models, see: http://www.cisco.com/en/US/docs/voice_ip_comm/cups/7_0/english/compatibility/cupcompatibility.html

Usually, CUPS is used with a presence client - CUPC. The official name of CUPC is "Cisco Unified Personal Communicator". Some people also refer it as "Cisco Unified Presence Client".

CUPC can run on Windows and Mac OS. CUPC is a multi-function client. It can send IMs (like MSN Messenger). It can display your buddies' presence (on the phone, in a meeting, etc.). It can retrieve voicemail. It can control your desk phone. It can act as a soft phone as well. It support video calls. It can also be a outlook plugin. So you can "click-to-dial" from Outlook.

For more information regarding CUPC, please see: http://www.cisco.com/en/US/products/ps6844/index.html

Introduction of 'Presence'

2009-01-01T11:13:00.000-08:00

Many people are still confused about 'presence' and what it can do for us.

The official explanation of presence was "Presence is a real-time indicator of a person’s willingness and availability to communicate".

Ya, that explanation is as clear as mud. Let's take a look at some real life examples.

Instant Messaging.

If you're reading this blog, you're unlikely to be unaware of "Instant Messaging" (or IM). Just to name some of the IM software: MSN Messenger, Yahoo Messenger, AOL Messenger, ICQ, IBM Sametime.

Obviously, the major purpose of IM software is to send instant message. Yet another important information they provide is buddy status - 'Available', 'Away', 'Out to lunch', 'In a meeting', 'Do not disturb', etc. Those status are 'presence' info, and play a more and more important role in our life.

Phone presence.

With the adoption of IP phones, more and more features were built on it. In the old world, you call somebody and got a busy signal. You know he's on the phone. In the new world, you could see if he's on the phone or not, before you actually call him. This is called 'phone presence'.

Let say, you'd like to ask your buddy out for lunch. With presence tool, you know he's on the phone. You would use other method to communicate, such as instant message, email, etc. With appropriate tools, you can even send the instant message to his IP phone.

Calendar presence.

If you'd like, you may also have the presence software publish your calendar status. So your buddy would see your status as 'In a meeting' when you have a meeting going on in the calendar.