Monday, July 19, 2010

What is XMPP and why it is important in Unified Communication

To get a complete understand of XMPP, I recommend the book "XMPP: The Definitive Guide".

In short, XMPP is a protocol (like SIP). XMPP was majorly used for presence and Instant Messaging. But it is being expanded to other areas like call signaling control.

XMPP is getting more and more popular in Unified Communication products like CUPS (Cisco Unified Presence), Cisco Quad (SharePoint-like product), Contact Center, etc.

Since it's a direct competitor with SIP, frequently asked questions would be:
  • Shall I learn XMPP?
  • Will SIP be replaced by XMPP?
For the first question, Yes. You definitely should learn XMPP because
  • XMPP was built from ground to support presence and IM
  • XMPP is "eXtensible"
  • XMPP is being (or has been) adopted by many big vendors like Cisco, Microsoft, Google, etc.
For the 2nd question, it really depends. On call control and telco interfacing, SIP will still dominate for quite a while because the install base and impact of change.

Anyway, you should take an hour or two to understand XMPP so you can understand what it can do (or do better) on different systems.

Thursday, July 1, 2010

Happy birthday, CUPC 8!

Two months after CUPS 8 was released, Cisco finally releases the next generation of communication client - Cisco Unified Personal Communicator 8.

With XMPP, now CUPC can do:

  • Group Chat
  • Persistent Chat Room
  • Chat History
  • Message Archiving and Auditing
  • Federation with external IM systems (like GoogleTalk, Sametime, Webex, OCS)
  • Integrated HD Video with soft phone or hard phone

Happy Birthday! :)

Saturday, June 26, 2010

See ya in Las Vegas

I'll be leaving for Las Vegas Sunday afternoon.

During Cisco Live (Networkers), I'll be in the Technical Solution Clinic (Unified Communications). I'll attend the CCIE party at VooDoo Lounge Tuesday (June 29).

See you guys there.

By the way, I'll bring two copies of my "Deploying Cisco Unified Presence" book. For those who need it, just drop by and take it. :)

7/1/2010 13:30 PDT - Whoever brought me a dessert at Technical Solution Clinic, thank you very much! I really enjoyed it. (though I don't know who it was) :)


Wednesday, June 16, 2010

Unified Communications Manager (CallManager) Upgrade Tool

It seems that when planning a Cisco Unified Communications Manager (CUCM) upgrade, the amount of releases are sometimes dizzying! Can I direct upgrade or not? Do I need to PUT (product upgrade tool) for a license and CD? How do I match the release ID with the CUCM version ID (7.1(3a) vs. I know that pain and Cisco has created a tool to help us with the process. Check out the link here.

With this tool, you can input your version (supports only 5.1(3)+) and which version you want to migrate to. It will let you know if you can upgrade directly and what the licensing implications are. As a customer looking at upgrading to 8.x, this tool is a terrific resource for upgrade planning. Check it out!


Wednesday, May 12, 2010

Test mobility feature without PSTN

The core of the mobility feature is "Remote Destination". Per SRND,

"Remote destinations must be Time Division Multiplex (TDM) devices or off-system IP phones on other clusters or systems. You cannot configure IP phones within the same Unified CM cluster as remote destinations. "
So if you'd like to test mobility feature without involving PSTN gateways, you may build another CUCM cluster (with VMWare, it should be just another VM). Then you may create ICT (Inter-Cluster Trunk) between two clusters. You may use the IP phones on another clusters as "Remote Destination". With this setup, you may test the following features:

1) Mobile Connect (SNR)
2) Enterprise Feature Access (EFA)

You won't be able to test the following feature:

MVA (Mobile Voice Access), because it requires IVR function of the voice gateway.

If you have a voice gateway but does not have T1/PRI, you may use CME to simulate PSTN phones.

Thursday, April 22, 2010

Make a non-bootable ISO image bootable

For whatever reason, Cisco only post "non-bootable" ISO images on CCO for download. In some urgent situations, you might need a bootable disc to recover the system (or your client/boss would shoot you in the head). Here's the procedure to make a non-bootable ISO image bootable.

Before you continue, be aware that this procedure is NOT approved by Cisco. Neither Cisco nor I will be responsible for any loss caused by this.

Any bootable disc has to follow "El Torito" specification. No exception for Cisco discs. The only difference between a bootable disc and non-bootable disc is the "boot sector". Thus the solution is very simple - extract the boot sector from a bootable disc and inject it into a non-bootable disc.

The boot sector is a very small file (usually less than 10k). And the boot sector is usually content independent (i.e. you may extract the boot sector from CUCM 7.1.3 and inject it into 7.1.5). You may save the boot sector on your USB thumb drive and keep it handy.

To extract/inject the boot sector, you need some disc image tools like UltraISO. (You may also use other ISO tools with similar features)

Step 1: Get the boot file

There are two ways to get a boot file - extract from the DVD's file system (regardless bootable or not) or extract from a bootable DVD's boot sector.

Option 1: Extract from DVD's file system (regardless bootable or not)

This option is preferred as you don't have to find another bootable disc or ISO file.

The boot file should be available on any CUCM DVD, regardless bootable or not.  It is located in the "isolinux" folder.  File name is isolinux.bin.

Extract and save the isolinux.bin file to your hard drive.  We'll need to use that later.

Option 2: Extract the boot sector from a bootable DVD

If for some reason, you were not able to find/extract the isolinux.bin file, you may extract the boot file from a bootable disc (or ISO image).

Put a bootable CUCM disc into the DVD drive and launch UltraISO. Go to menu "Bootable > Extract Boot File from CD/DVD..."

Save the file to your hard drive as a "boot info file" (bif). In our example, we call it "boot.bif"

Step 2: Inject the boot file

Open the non-bootable image in UltraISO. Go to menu "Bootable". Make sure "Generate Bootinfotable" was checked (it will NOT work without this option). Then choose "Load Boot File...".

Choose the boot file we saved before (isolinux.bin or boot.bif).

Note that the image type changed to "Bootable".

Now, you may go to "File > Save As" to save the bootable image to an ISO file. Then you may burn the ISO to a disc with your favorite disc burner software.

Tuesday, April 6, 2010

New book and Cisco Live

New edition of the book has been published. Use coupon code "SHOWERS" to get 10% off. Use coupon "FREEMAIL305" to save on shipping. (Offer ends April 30, 2010)

It covers new topics like: XMPP, Jabber, CSF (Client Service Framework), Message Archiver, Persistent Chat, 3rd-Party Compliance, external database, etc.

If things worked out, I'll go to Cisco Live (Networkers) at Las Vegas from Jun 28 to July 1. I'll be at the Technical Solution Clinics. See you there.

Friday, March 26, 2010

Place to ask questions

It's that time again - "Ask the Expert" event on Cisco NetPro forum.

If you have any questions, please feel free to ask there.

This time we'll focus on the next version of CUPS and CUPC, with new technology XMPP and CSF.

You may post your question at

I'll update my book "Deploying Cisco Unified Presence" to version 8.x in couple weeks.

Tuesday, March 9, 2010

Client Service Framework - CSF


Once upon a time, Cisco built a client software called "Cisco Unified Personal Communicator" (a.k.a. CUPC). CUPC has many features like Rich Presence, Instant Message, Desk Phone control (CTI), Soft Phone (SIP), Voicemail (IMAP), Web Conference, etc.

Later on, when Cisco built other client software (such as plug-in for Sametime Connect, plug-in for MOC), developers realized that there were many commonalities between those software and CUPC. Instead of "reinventing the wheel", they could just reuse the existing codes in CUPC.

However, "reuse" is not as simple as it sounds to be in software development. You'll have to "shape" the codes to fit into your specific software. To make the work easier, they decided to "extract" the codes from CUPC and standardize it. So any other software teams can easier build their application on top of it. This "extracted", "standardized" component is called "Client Service Framework". (if you have ever heard about ".Net Framework" from Microsoft, the concept is pretty similar).

The idea is: CSF has all the common feature built in and provides unified interfaces to upper applications (such as CUCIMOC, CUPC 8, CUCI Connect, etc.).

The code of CSF is usually referred as "core" (e.g. "core log" of CUCIMOC).

Because of the history, we sometimes see the "marks" of CUPC. For example, CUCIMOC dial rules are put in the "CUPC" folder on TFTP.

Applications using CSF

Right now, applications using CSF are:

  • CUCIMOC (for Microsoft Office Communicator)
  • CUCI Connect (for Webex Connect)
  • CUPC 8.0


To use the Cisco Unified Personal Communicator soft phone feature, we have to create a device in CUCM. The device type happened to be called "Cisco Unified Personal Communicator". This is really a BAD idea. If I were the decision maker, I would name it with some netrual name like "Cisco SIP soft phone".

Because the device type has the same name as the client software, many people (including Cisco TAC engineers) thought it was required for CUPC to work. More confusion when it comes to licensing and troubleshooting. Such as:

  • "I licensed UPC on CUCM > System > Licensing > Capability Assigments. That took 1 DLU. But when I ran the license calculator, the 'Cisco Unified Personal Communicator' is taking 3 DLUs"
  • "My CUPC is not registering" "Did you mean the CUPC on your desktop(Windows XP)? Or the CUPC on CUCM?"

They made the same mistake again with CSF. To use the soft phone feature of CSF, you have to create a device in CUCM. The device type is called "Client Service Framework". Again, if I were the decision maker, I would name it with a neutral name like "Cisco SIP soft phone (2nd Generation)".

The point is: don't confuse one of the features with the whole piece. You don't need to configure CSF device in CUCM to get the CSF worked on client applications. Without CSF device on CUCM, the only feature you're missing is soft phone.


As you can guess, CSF does not have a user interface. User interface was provided by the upper layer applications such as CUCIMOC, CUCI Connect, CUPC 8 etc.

The only way you can "see" CSF is to use Windows Task Manager. You'll see a process called "cucsf.exe". This process usually starts up and close down with the upper layer applications (such as CUPC 8). But not every application does the same thing. For example, CUCIMOC does not shutdown CSF when you close MOC. If you want to kill or restart CSF for whatever reasons, you'll have to kill it from Task Manager.


In current version of CSF, there's no GUI to configure it. All configuration was done via Windows registry.

Depending on your deployment, configuration data could be in either one of the following locations:

  • HKEY="HKCU\Software\Cisco Systems, Inc.\Client Services Framework\AdminData"
  • HKEY="HKCU\Software\Policies\Cisco Systems, Inc.\Client Services Framework\AdminData"

If both keys exist, "HKCU\Software\Policies\Cisco Systems, Inc.\Client Services Framework\AdminData" takes priority.

For centralized management, you may use a logon script (BAT) or a group policy to update the client's registry.

Registry key requirement varies between applications.

For CUCIMOC, registry key info as below: CUCIMOC also provides a sample BAT file and REG file for your convenience. Those sample files are zipped into (x-y-z is the version of CUCIMOC. e.g.


  • Application Dial Rules (ADR)

Current version of CSF does not have an interface to "dip" into CUCM database to retrieve ADRs. As a workaround, Cisco provides a COP file. Whenever you run(install) the COP file on CUCM, CUCM will generate XML files in TFTP folder to reflect the latest ADRs. Then CSF needs to be restart to download those XML files. For details, see:

This means, whenever you make changes to CUCM > Call Routing > Dial Rules, you'll have to run the COP file and restart CSF.

  • Soft Phone

CSF soft phone is a SIP phone. It has quite a few limitations comparing with SCCP (skinny) phones. For example, the following features are not supported on SIP:

  • CUVA support
  • FAC/CMC support
  • MLPP target device
  • Direct transfer
  • Hold tone

... to be continued

Wednesday, March 3, 2010

Client Convergence

In the "Unified Communication" world, you might have many different client software installed on your computer -

  • Click-to-Call for making calls easy
  • IP Communicator to be a soft phone
  • Video Advantage to be a "video add-on"
  • Personal Communicator to be an IM and presence client
It's kind of "un-unified" with so many different things. Fortunately, your voice was heard. In the future, you might have a truly "unified" client, which does everything with one piece.

Monday, March 1, 2010

Calendar presence with CUPS and Exchange

Cisco Unified Presence Server (CUPS) has the capability to retrieve calendar (Exchange) status and populate to clients (CUPC). When your calendar status is "busy" in calendar, CUPC will display your status as "in a meeting".

To enable this feature, there are server-side configuration (CUPS/Exchange) and client side configuration(CUPC).

CUPC configuration

On CUPC side, you go to "File > Preferences > Status" and put a check mark on "Show me as 'In a Meeting' whenever my Exchange calendar shows me as busy".

Please note that when multiple 'presence' arrIve, CUPC can only display one of them. The priority, from high to low is: Phone Presence > Calendar Presence > Availability Status.

For example:

You logged into CUPC ==> Availability Status = "Available"
You're on the phone ==> Phone Status = "On the Phone"
Your calendar is busy ==> Calendar Presence = "In a Meeting"

In this scenario, CUPC will display the status as "On the Phone". You won't see "In a Meeting". However, you may move the mouse over a contact (or your self-status) to see all the presence.

In the screenshot below, you'll see the CUPC display its self-status as "On the phone". When you move the mouse over the self-status, you'll see "Online, On the phone, In a meeting" in the tooltip. This is how to explore the "hidden" presence from CUPC.

CUPS configuration

The most confusing (instead of difficult) part on CUPS is the SSL/Certificate configuration. CUPS can only talk to Exchange via HTTPS. Why is that?

That is because CUPS use an Exchange "service account" to read everybody's mailbox (calendar). If the communication was not encrypted (SSL), it would pose a threat to information security. A hacker could easily intercept the packets on network and retrieve everybody's (including CEO's) email and calendar.

With that said, here are the requirements on Exchange and CUPS to set up a SSL connection:
1) HTTPS needs to be enabled for OWA (Outlook Web Access)
2) The CA certificates of Exchange need to be trusted by CUPS.

Regarding #1, there's a catch. We'll discuss that in the "Exchange" section below.

Regarding #2, there are two catches -
A) Don't confuse the CA cert with identity cert.
B) Request address has to match certificate.

A) What is a "CA cert" and what is an "identity cert"?

For example, if your Exchange OWA has a certificate to identify itself as "owa.acme.local". This certificate is called "identity cert". Let say, this identity cert was issued by a CA (Certificate Authority) called "my-ca-server.acme.local". On the CA server, it has a certificate to identify itself. We call that cert a "CA cert".

It is the "CA cert" CUPS server needs to trust. In our example above, it's the "my-ca-server.acme.local" certificate needs to be uploaded to CUPS as "Presence Engine-Trust".

B) When you configure the "Outlook presence gateway" on CUPS, the address you put in there is the address CUPS will use to sent request to Exchange.

For simplicity, quite a few people just put the IP address of Exchange there. This is not going to work with SSL. For security, the request address has to match with the common name in the identity certificate. Usually, the certificate will identity the server in FQDN (e.g. owa.apps.local). Thus, you'll have to configure the same name in presence gateway configuration page.

Please note: I used the word "same name" instead of "FQDN". Because technically, you could configure any name in the certificate. Let's take a look at a couple examples:

Your OWA server IP =
Your OWA server FQDN in DNS = owa.apps.local

Example #1: You configured the common name in the OWA cert as "email.apps.local".

This won't work. Because:

If you used "" in presence gateway configuration, it doesn't match "email.apps.local" in the identify cert.

If you used "owa.apps.local" in presence gateway configuration, it doesn't match "email.apps.local" in the identify cert.

If you used "email.apps.local" in presence gateway configuration, CUPS won't be able to resolve the name, because it's not in DNS.

Or even worse, you don't have DNS configured on CUPS so you can't use any FQDN.

1) Add "email.apps.local" to DNS
2) Configure CUPS used DNS (use "set network dns primary" command)
3) Use "email.apps.local" in presence gateway

Example #2

In a lab environment, you may configure the certificate so it identifies the OWA server with common name "" (though it's not a common practice, it's technically legit).

In this scenario, you may use IP address in presence gateway config.

Whenever you made changes to presence gateway, don't forget to restart Presence Engine service.


You don't have to be an Exchange expert. But you should at least know the following:

OWA, WebDAV, Receive-As permission, IIS certificate, CAS/Mailbox roles, authentication methods, Power Shell. (did I say you don't have to be an expert? :) )


OWA - Outlook Web Access, a web interface to access Exchange emails, calendars, etc. HTTPS needs to be enabled on OWA for CUPS integration to work.

OWA is actually a very useful tool for troubleshooting. For example:

When you tried to access OWA via HTTPS, OWA server will present you a certificate. You may view that certificate from web browser. You should be able to tell what the CA cert is from the web browser.

In the screenshot below, you open a web page to OWA. If you double-click the "lock" icon at the bottom of IE, you'll see a certificate viewer window. From the "certification" tab, you'll see the CA cert is "sametime.apps.local". The identity cert is "owa.apps.local".

Now you know that:
1) You need to upload "sametime.apps.local" cert to CUPS as "Presence Engine - Trust".
2) You need to configure "owa.apps.local" as the "presence gateway" on CUPS > Presence > Presence Gateway.


WebDAV is a protocol (on top of HTTP) to retrieve email/calendar information. See for more details.

CUPS uses WebDAV for calendar integration. Microsoft obsoletes WebDAV on Exchange 2010 and advocates its own protocol EWS (Exchange Web Service). By the time this blog was being written, CUPS doesn't support EWS yet. Which means, CUPS doesn't work with Exchange 2010. I'm not sure if you can have an E2007 CAS in E2010 as a "WebDAV" gateway. I'll test it later.

Though WebDAV sits on HTTP/HTTPS, it's different from OWA. Don't assume WebDAV is working just because OWA was working.

Up to Exchange 2007, WebDAV was enabled by default. You don't have to explicitly enable it. Some people reported that WebDAV was not installed/enabled on E2007 with Win2008. This is not true. The "WebDAV Publishing" in screenshot below has nothing to do with CUPS integration. It's OK to leave it uninstalled.

Since E2007, Microsoft change the OWA URL from /exchange to /owa, but WebDAV URL stays the same (/exchange). This caused more confusions in troubleshooting.

When you troubleshoot CUPS calendar integration, you'll see CUPS always request to URL /exchange, which is the WebDAV URL. This is right and expected.

If you tried to "test" it by typing "https://owa.acme.local/exchange", you'll be redirected to "https://owa.acme.local/owa". This is also right and expected. For more details, please see:

The point is: there's no easy way to tell if WebDAV is working. But it's easy to tell if WebDAV is NOT working.

If you typed "https://owa.acme.local/exchange" and got some error like "401 unauthorized" or "503 service unavailable", WebDAV is more than likely not working. Though regular OWA might be working fine at this point.

If you got "401 unauthorized", you may take a look at Default settings should work. If it doesn't work, most likely your authentication was expecting username in a different format. e.g. "domain\user" instead of "user".

If you got "503 service unavailable" or "500 internal server error", please make sure "ISAPI Extensions" is installed on mailbox server. See screenshot below. This usually happens if mailbox server and CAS are on two different servers.


OWA/WebDAV relies on IIS to server HTTP request. It's IIS that presents certificate to SSL client.

By default, when you installed IIS, it'll generate a self-signed certificate. Unfortunately, this self-signed certificate doesn't work with CUPS, because it doesn't contain CA bit in its extension. Below is an example of the certificate that HAS the CA bit:

If your CA cert does not have CA bit, CUPS won't trust it. This is per RFC 2459. Don't blame Cisco on that.

Solution? Get a certificate from a CA (either external or internal) or use "makecert.exe" to create a self-signed cert with CA bit (see my book for details).

Receive-As Permission

In order to see everybody's calendar, CUPS needs an account with this permission. So the FAQ is: what's the minimum permission required?

The minimum permission required is "Receive-As" on the end user's mailbox.

Cisco documents said you need "View Only Administrator". That's not true. All you need is "Receive-As" permission on the end user's mailbox. Not more, not less.

How do we assign "Receive-As" permission to the account? I would recommend use Exchange Management Shell (command line).

Let say, in Active Directory, the service account used by CUPS is called "cupsexch". The end user account is called "JDoe" (full name: John Doe). The following command give "cupsexch" Receive-As permission on John Doe's mailbox:

Add-ADPermission -Identity "John Doe" -User cupsexch -ExtendedRights Receive-As
Of course, you're not going to repeat this command for 1000 users. The better way to do it is to assign permission on a "container" that contains the end users' mailboxes.

In Exchange, the "container" for mailboxes is "Mailbox Database". The container for "Mailbox Databases" is "Storage Group". You may have multiple databases in a storage group. And you may have multiple storage groups in your Exchange environment.

You may assign permission at different levels:

End User Mailbox
Mailbox Database
Storage Group

Since "Storage Group" is the highest level of "container", we usually assign permissions on storage group level. Permissions would populate down to end user mailboxes. This "population" takes some time (from 30 minutes to hours).

The command to assign permission on "First Storage Group" is as below:

Add-ADPermission -Identity "First Storage Group" -User cupsexch -ExtendedRights Receive-As
What if you have multiple storage groups and want to automate the process? You may use the command below:

Get-StorageGroup | add-ADPermission –user cupsexch -ExtendedRights Receive-As
"Get-StorageGroup" command will get all storage group names from system and feed the names to "Add-ADPermission" command.

The command to verify permission is as below:

Get-MailboxPermission jdoe -user cupsexch | Format-Table -autosize
This command will display "cupsexch" permission on jdoe's mailbox. Anything after the pipe sign (|) is just for formatting purpose.

If you got nothing, that means "no permission". If you assigned the permission on higher containers, it might not have flowed down yet.

If the permission has been populated, you should see something like the screenshot below:

"AccessRights = {FullAccess}" means "cups7exch" account has permission to read "John Doe's" mailbox.
"Is Inhereited = True" means this permission was inherited from some higher level containers.

This is the only reliable way to test permissions.

Cisco documents described a test method which use OWA to test the URL "https://owa-address/exchange/some_user/calendar" with the service account. This method may or may not work depending on other permissions you configured for the service account. For example, if you are like me to give minimum permissions, Cisco test method would yield a "no permission" page as below. However, CUPS calendar works just fine with the "minimum permission".


If calendar integration doesn't work, "Presence Engine" (PE) logs are the only traces we need.

Ideally, we want the logs since PE started. Thus if you could, do the following:

1) Restart PE
2) Wait unti PE started and stabilized
3) Log into CUPC
4) Collect PE logs since the time of restart
5) Use WinGrep to search for keyword "owa". It'll give you all message regarding OWA transactions.

Following are some common problems:

"Certificate not trusted"

This could be caused by the following:

  • The name configured in "CUPS > Presence > Presence Gateway" does not match with the name in identity certificate. (e.g. IP address vs. FQDN)
  • CA certificate was not imported as "Presence Engine - Trust". If there are multiple CAs in the certificate chain, all of them need to be imported.
  • CA bit was not set (usually happens on IIS self-signed certificate)

"440 Login Timeout"

When FBA (Form-Based Authentication) was enabled, you'll see one "440 login timeout" in PE log for each transaction. This is normal. If you keep getting "440 login timeout", it usually indicates an authentication issue.

To reveal the nature of the problem, it's recommended to change "Form-Based Authentication" to "Standard Authentication" on "/Exchange" virtual directory on OWA. See screenshot below:

Changing this will not change the way or feel of regular OWA logon. It's just change the authentication method for WebDAV. (you'll need to reset IIS by using command "iisreset /noforce")

"401 Unauthorized"

This is usually caused by permission issue. See "Receive-As" permission configuration above for testing method.

Another testing method would be configuring an end user's credential on presence gateway. An end user's credential would definitely have permission on his own mailbox. If you log in to CUPC with that end user account, calendar integration should work. This is a good way to do "problem isolation".

To be continued...

Saturday, January 23, 2010

Hard Drive Partition and version control of UC Appliance

Active and Inactive Version

Many Cisco Unified Communication appliances (CUCM, CUPS, CER, CUMA, etc.) share the same OS (Cisco customized Linux).

For maintenance purpose, you may install two copies (versions) of systems on the hard drive. Cisco call it "active version" and "inactive version".

CLI commands:

show version active
show version inactive
utils system switch-version

Please note: "active" and "inactive" are relative. When you switch the version (utils system switch-version), the "active" version will becomes "inactive".

If you're a Windows guy, you should be familiar with C:\boot.ini.

If you're a Linux guy, you should be familiar with grub.conf.

It's the same way Cisco UC appliance controls which version to boot from.


The two copies of software are installed into two partitions: \ (referred as Partition A) and \partB (referred as Partition B). Whenever you use "utils system switch-version", the active partition will become inactive. The inactive partition will becomes active.

Upgrade and Switch Version

The word "upgrade/patch" has a different meaning in Cisco UC world. Instead of "replacing" files, the "upgrade/patch" process is actually installing a full copy of system in the inactive partition. This has two benefits:

1) You may perform an upgrade/patch during production hours.
2) It's easy to fall back to the old version.

You have 6 CUCM servers in the cluster. Let say, upgrade each server takes about 2 hours. Your business does not allow any downtime during business hours.

Q1: How long does it take to upgrade the whole cluster?
Q2: How much time you'll have to spend in after hours for the upgrade?

A1: About 4 hours (2 + 2)
A2: A couple minutes.


1. You need to finish the upgrade on CUCM publisher before you can do the upgrade on subscriber. It takes 2 hours to upgrade the publisher. The new code be installed into inactive partition. You don't have to switch to new version right after install. Thus you can do it in business hours.

2. Once the the new version has been installed on publisher (even it's in inactive partition), you may start upgrade process on subscribers (simultaneously). This takes about 2 hours (because you're upgrading all subscribers simultaneously). You don't have to switch to new version right after install. Thus you can do it in business hours.

3. In after hours, you may use "utils system switch-version" command to switch all boxes to new version. This usually takes less than 10 minutes.

However, there's a catch: if you made any configuration changes after the point of upgrade, those changes wouldn't be reflected in the new version. For example, you performed the upgrade at 10AM but didn't switch to new version. Then you switched to new version at 6:30PM. Any configuration changes made between 10AM and 6:30PM will be lost.

Under the hood of "utils system switch-version"

What actually happens when you type the command "utils system switch-version"?

1) It modifies /grub/boot/grub/grub.conf file to make the other partition active
2) It synchronizes UFF (User Faced Feature) to the other version. UFF refers to Call Forwarding, MWI (Message Waiting Indicator), etc.

If the system failed to switch version, here are some options:

Option 1: Try "utils system switch-version nodatasync"
This turn off the UFF data sync action.

Option 2: Use "Recovery CD" (downloadable from CCO) to switch version.

Option 3: If you're a Linux guy, it shouldn't be too difficult for you to get access to /grub/boot/grub/grub.conf.

OCS 2007 R2 on Windows 2008 R2 with SQL 2008

Neither Windows 2008 nor SQL 2008 is supported by OCS 2007 R2. But if you really want to do it, here are some tips:

Use SQL 2008 R2

SQL 2008 will fail to install on Windows 2008 R2. Instead of trying to 'fix' it, you may just use SQL 2008 R2.

Pool Creation Failure

a) manually create database rtcconfig (assuming you know how to use SQL Management Studio)

b) manually run the script on OCS installation CD.
D:\Setup\amd64\DbSetup>cscript.exe poolcfgdbsetup.wsf /clean /sqlserver:rcdn /serverrole:EP /verbose

c) continue GUI install

Error: "Not available: IIS 6 Management Compatibility and IIS Windows Authentication role services must be installed before you Deploy Server."

Solution: Install all IIS6 compatibility options. Install all authentication options

Error: "The Windows Media Format Runtime is required in order to install this component. Installing the Windows Media Format Runtime may require a system restart to complete the installation. Click OK to continue with the installation."

Solution: install the Desktop Experience Feature.

Error: "[0xC3EC78D8] Failed to read the Office Communications Server version information. This can happen if the computer clock is not set to correct date and time."

Solution: Uninstall MS Crypto API security update KB974571

Friday, January 22, 2010

The art of troubleshooting

Since I joined TAC, I've been the top case solver for 16 consequent quarters no matter what technology group I worked in. I'd like to share some tips on troubleshooting.

Understand the user's expectation.

Still remember the old joke that a user called IT support and said the "cup holder" on his computer stopped working? It turned out to be the CD drive. He insisted he's been using it for years.

Understanding user's expectation can help you determine if you should do customer education or troubleshooting.

Keep it simple

Which one is easier? Troubleshoot a light switch or troubleshoot a space shuttle?

Multiple-system integration adds complexity to the problem. You should try to simplify it as much as possible.

For example: When PSTN call comes in, it hits Unity Auto Attendant. Press 2 to transfer to sales queue, which is a CTI route point handled by UCCX. If no one answers the call, it should goes into a voicemail box dedicated for sales department. Instead of going into voicemail, the caller just heard repeating "transferring..."

In this case, we have too many elements in the picture - Unity, UCCX, CUCM, voice gateway, service provider. Instead of troubleshooting from end to end, we should troubleshoot it segment by segment -

1) What if we called the sale agent directly? If he didn't answer, would the call goes into voicemail? (get Unity Auto Attendant and UCCX out of picture)
2) What if we bypass Unity Attendant Console and call UCCX route point directly? Would it work properly? (get Unity Auto Attendant out of picture)
3) What if we make a test call from internal phone? Would the problem be the same? (get PSTN and voice gateway out of picture)

Other tips to make things simple during troubleshooting:
1) Use default settings. For example, use a "vanilla windows" (fresh installed with Microsoft CD) instead of using a "corporate customized" image.
2) Test on LAN instead of over VPN (again, decrease number of elements)
3) Always assume the system is case sensitive (err on the safe side)

Find a reference point

If a software doesn't work for one user and works for another one, use the good one as reference point and find out the difference.

Of course there are many differences between two users, such as their wife and kids. :) But we should look at the most relevant ones.

Most software nowadays are "client-server" model. The most relevant ones are accounts and computer. Switch the computer (or switch the account) to see if the problem follows the computer or account. If it follows the computer, it might be network or computer settings (client side). If it follows the account, it might be configuration issue (most likely server side).

Understand positive and negative result of the test


"Dad, I couldn't find any Easter eggs in the backyard!". Does that mean there's no eggs there?

"Dad, I found some Easter eggs in the backyard!". That means there are some eggs there.