- Click-to-Call for making calls easy
- IP Communicator to be a soft phone
- Video Advantage to be a "video add-on"
- Personal Communicator to be an IM and presence client
Wednesday, March 3, 2010
Client Convergence
In the "Unified Communication" world, you might have many different client software installed on your computer -
Monday, March 1, 2010
Calendar presence with CUPS and Exchange
Cisco Unified Presence Server (CUPS) has the capability to retrieve calendar (Exchange) status and populate to clients (CUPC). When your calendar status is "busy" in calendar, CUPC will display your status as "in a meeting".
To enable this feature, there are server-side configuration (CUPS/Exchange) and client side configuration(CUPC).
CUPC configuration
On CUPC side, you go to "File > Preferences > Status" and put a check mark on "Show me as 'In a Meeting' whenever my Exchange calendar shows me as busy".
Please note that when multiple 'presence' arrIve, CUPC can only display one of them. The priority, from high to low is: Phone Presence > Calendar Presence > Availability Status.
For example:
You logged into CUPC ==> Availability Status = "Available"
You're on the phone ==> Phone Status = "On the Phone"
Your calendar is busy ==> Calendar Presence = "In a Meeting"
In this scenario, CUPC will display the status as "On the Phone". You won't see "In a Meeting". However, you may move the mouse over a contact (or your self-status) to see all the presence.
In the screenshot below, you'll see the CUPC display its self-status as "On the phone". When you move the mouse over the self-status, you'll see "Online, On the phone, In a meeting" in the tooltip. This is how to explore the "hidden" presence from CUPC.
CUPS configuration
The most confusing (instead of difficult) part on CUPS is the SSL/Certificate configuration. CUPS can only talk to Exchange via HTTPS. Why is that?
That is because CUPS use an Exchange "service account" to read everybody's mailbox (calendar). If the communication was not encrypted (SSL), it would pose a threat to information security. A hacker could easily intercept the packets on network and retrieve everybody's (including CEO's) email and calendar.
With that said, here are the requirements on Exchange and CUPS to set up a SSL connection:
1) HTTPS needs to be enabled for OWA (Outlook Web Access)
2) The CA certificates of Exchange need to be trusted by CUPS.
Regarding #1, there's a catch. We'll discuss that in the "Exchange" section below.
Regarding #2, there are two catches -
A) Don't confuse the CA cert with identity cert.
B) Request address has to match certificate.
A) What is a "CA cert" and what is an "identity cert"?
For example, if your Exchange OWA has a certificate to identify itself as "owa.acme.local". This certificate is called "identity cert". Let say, this identity cert was issued by a CA (Certificate Authority) called "my-ca-server.acme.local". On the CA server, it has a certificate to identify itself. We call that cert a "CA cert".
It is the "CA cert" CUPS server needs to trust. In our example above, it's the "my-ca-server.acme.local" certificate needs to be uploaded to CUPS as "Presence Engine-Trust".
B) When you configure the "Outlook presence gateway" on CUPS, the address you put in there is the address CUPS will use to sent request to Exchange.
For simplicity, quite a few people just put the IP address of Exchange there. This is not going to work with SSL. For security, the request address has to match with the common name in the identity certificate. Usually, the certificate will identity the server in FQDN (e.g. owa.apps.local). Thus, you'll have to configure the same name in presence gateway configuration page.
Please note: I used the word "same name" instead of "FQDN". Because technically, you could configure any name in the certificate. Let's take a look at a couple examples:
Your OWA server IP = 192.168.1.100
Your OWA server FQDN in DNS = owa.apps.local
Example #1: You configured the common name in the OWA cert as "email.apps.local".
This won't work. Because:
If you used "192.168.1.100" in presence gateway configuration, it doesn't match "email.apps.local" in the identify cert.
If you used "owa.apps.local" in presence gateway configuration, it doesn't match "email.apps.local" in the identify cert.
If you used "email.apps.local" in presence gateway configuration, CUPS won't be able to resolve the name, because it's not in DNS.
Or even worse, you don't have DNS configured on CUPS so you can't use any FQDN.
Solution:
1) Add "email.apps.local" to DNS
2) Configure CUPS used DNS (use "set network dns primary" command)
3) Use "email.apps.local" in presence gateway
Example #2
In a lab environment, you may configure the certificate so it identifies the OWA server with common name "192.168.1.100" (though it's not a common practice, it's technically legit).
In this scenario, you may use IP address 192.168.1.100 in presence gateway config.
Whenever you made changes to presence gateway, don't forget to restart Presence Engine service.
Exchange
You don't have to be an Exchange expert. But you should at least know the following:
OWA, WebDAV, Receive-As permission, IIS certificate, CAS/Mailbox roles, authentication methods, Power Shell. (did I say you don't have to be an expert? :) )
OWA
OWA - Outlook Web Access, a web interface to access Exchange emails, calendars, etc. HTTPS needs to be enabled on OWA for CUPS integration to work.
OWA is actually a very useful tool for troubleshooting. For example:
When you tried to access OWA via HTTPS, OWA server will present you a certificate. You may view that certificate from web browser. You should be able to tell what the CA cert is from the web browser.
In the screenshot below, you open a web page to OWA. If you double-click the "lock" icon at the bottom of IE, you'll see a certificate viewer window. From the "certification" tab, you'll see the CA cert is "sametime.apps.local". The identity cert is "owa.apps.local".
Now you know that:
1) You need to upload "sametime.apps.local" cert to CUPS as "Presence Engine - Trust".
2) You need to configure "owa.apps.local" as the "presence gateway" on CUPS > Presence > Presence Gateway.
WebDAV
WebDAV is a protocol (on top of HTTP) to retrieve email/calendar information. See http://en.wikipedia.org/wiki/WebDAV for more details.
CUPS uses WebDAV for calendar integration. Microsoft obsoletes WebDAV on Exchange 2010 and advocates its own protocol EWS (Exchange Web Service). By the time this blog was being written, CUPS doesn't support EWS yet. Which means, CUPS doesn't work with Exchange 2010. I'm not sure if you can have an E2007 CAS in E2010 as a "WebDAV" gateway. I'll test it later.
Though WebDAV sits on HTTP/HTTPS, it's different from OWA. Don't assume WebDAV is working just because OWA was working.
Up to Exchange 2007, WebDAV was enabled by default. You don't have to explicitly enable it. Some people reported that WebDAV was not installed/enabled on E2007 with Win2008. This is not true. The "WebDAV Publishing" in screenshot below has nothing to do with CUPS integration. It's OK to leave it uninstalled.
Since E2007, Microsoft change the OWA URL from /exchange to /owa, but WebDAV URL stays the same (/exchange). This caused more confusions in troubleshooting.
When you troubleshoot CUPS calendar integration, you'll see CUPS always request to URL /exchange, which is the WebDAV URL. This is right and expected.
If you tried to "test" it by typing "https://owa.acme.local/exchange", you'll be redirected to "https://owa.acme.local/owa". This is also right and expected. For more details, please see: http://msexchangeteam.com/archive/2007/02/07/434523.aspx
The point is: there's no easy way to tell if WebDAV is working. But it's easy to tell if WebDAV is NOT working.
If you typed "https://owa.acme.local/exchange" and got some error like "401 unauthorized" or "503 service unavailable", WebDAV is more than likely not working. Though regular OWA might be working fine at this point.
If you got "401 unauthorized", you may take a look at http://msexchangeteam.com/archive/2008/02/01/447989.aspx. Default settings should work. If it doesn't work, most likely your authentication was expecting username in a different format. e.g. "domain\user" instead of "user".
If you got "503 service unavailable" or "500 internal server error", please make sure "ISAPI Extensions" is installed on mailbox server. See screenshot below. This usually happens if mailbox server and CAS are on two different servers.
Certificate
OWA/WebDAV relies on IIS to server HTTP request. It's IIS that presents certificate to SSL client.
By default, when you installed IIS, it'll generate a self-signed certificate. Unfortunately, this self-signed certificate doesn't work with CUPS, because it doesn't contain CA bit in its extension. Below is an example of the certificate that HAS the CA bit:
If your CA cert does not have CA bit, CUPS won't trust it. This is per RFC 2459. Don't blame Cisco on that.
Solution? Get a certificate from a CA (either external or internal) or use "makecert.exe" to create a self-signed cert with CA bit (see my book for details).
Receive-As Permission
In order to see everybody's calendar, CUPS needs an account with this permission. So the FAQ is: what's the minimum permission required?
The minimum permission required is "Receive-As" on the end user's mailbox.
Cisco documents said you need "View Only Administrator". That's not true. All you need is "Receive-As" permission on the end user's mailbox. Not more, not less.
How do we assign "Receive-As" permission to the account? I would recommend use Exchange Management Shell (command line).
Let say, in Active Directory, the service account used by CUPS is called "cupsexch". The end user account is called "JDoe" (full name: John Doe). The following command give "cupsexch" Receive-As permission on John Doe's mailbox:
In Exchange, the "container" for mailboxes is "Mailbox Database". The container for "Mailbox Databases" is "Storage Group". You may have multiple databases in a storage group. And you may have multiple storage groups in your Exchange environment.
You may assign permission at different levels:
End User Mailbox
Mailbox Database
Storage Group
Since "Storage Group" is the highest level of "container", we usually assign permissions on storage group level. Permissions would populate down to end user mailboxes. This "population" takes some time (from 30 minutes to hours).
The command to assign permission on "First Storage Group" is as below:
The command to verify permission is as below:
If you got nothing, that means "no permission". If you assigned the permission on higher containers, it might not have flowed down yet.
If the permission has been populated, you should see something like the screenshot below:
"AccessRights = {FullAccess}" means "cups7exch" account has permission to read "John Doe's" mailbox.
"Is Inhereited = True" means this permission was inherited from some higher level containers.
This is the only reliable way to test permissions.
Cisco documents described a test method which use OWA to test the URL "https://owa-address/exchange/some_user/calendar" with the service account. This method may or may not work depending on other permissions you configured for the service account. For example, if you are like me to give minimum permissions, Cisco test method would yield a "no permission" page as below. However, CUPS calendar works just fine with the "minimum permission".
Troubleshooting
If calendar integration doesn't work, "Presence Engine" (PE) logs are the only traces we need.
Ideally, we want the logs since PE started. Thus if you could, do the following:
1) Restart PE
2) Wait unti PE started and stabilized
3) Log into CUPC
4) Collect PE logs since the time of restart
5) Use WinGrep to search for keyword "owa". It'll give you all message regarding OWA transactions.
Following are some common problems:
"Certificate not trusted"
This could be caused by the following:
"440 Login Timeout"
When FBA (Form-Based Authentication) was enabled, you'll see one "440 login timeout" in PE log for each transaction. This is normal. If you keep getting "440 login timeout", it usually indicates an authentication issue.
To reveal the nature of the problem, it's recommended to change "Form-Based Authentication" to "Standard Authentication" on "/Exchange" virtual directory on OWA. See screenshot below:
Changing this will not change the way or feel of regular OWA logon. It's just change the authentication method for WebDAV. (you'll need to reset IIS by using command "iisreset /noforce")
"401 Unauthorized"
This is usually caused by permission issue. See "Receive-As" permission configuration above for testing method.
Another testing method would be configuring an end user's credential on presence gateway. An end user's credential would definitely have permission on his own mailbox. If you log in to CUPC with that end user account, calendar integration should work. This is a good way to do "problem isolation".
To be continued...
To enable this feature, there are server-side configuration (CUPS/Exchange) and client side configuration(CUPC).
CUPC configuration
On CUPC side, you go to "File > Preferences > Status" and put a check mark on "Show me as 'In a Meeting' whenever my Exchange calendar shows me as busy".
Please note that when multiple 'presence' arrIve, CUPC can only display one of them. The priority, from high to low is: Phone Presence > Calendar Presence > Availability Status.
For example:
You logged into CUPC ==> Availability Status = "Available"
You're on the phone ==> Phone Status = "On the Phone"
Your calendar is busy ==> Calendar Presence = "In a Meeting"
In this scenario, CUPC will display the status as "On the Phone". You won't see "In a Meeting". However, you may move the mouse over a contact (or your self-status) to see all the presence.
In the screenshot below, you'll see the CUPC display its self-status as "On the phone". When you move the mouse over the self-status, you'll see "Online, On the phone, In a meeting" in the tooltip. This is how to explore the "hidden" presence from CUPC.
CUPS configuration
The most confusing (instead of difficult) part on CUPS is the SSL/Certificate configuration. CUPS can only talk to Exchange via HTTPS. Why is that?
That is because CUPS use an Exchange "service account" to read everybody's mailbox (calendar). If the communication was not encrypted (SSL), it would pose a threat to information security. A hacker could easily intercept the packets on network and retrieve everybody's (including CEO's) email and calendar.
With that said, here are the requirements on Exchange and CUPS to set up a SSL connection:
1) HTTPS needs to be enabled for OWA (Outlook Web Access)
2) The CA certificates of Exchange need to be trusted by CUPS.
Regarding #1, there's a catch. We'll discuss that in the "Exchange" section below.
Regarding #2, there are two catches -
A) Don't confuse the CA cert with identity cert.
B) Request address has to match certificate.
A) What is a "CA cert" and what is an "identity cert"?
For example, if your Exchange OWA has a certificate to identify itself as "owa.acme.local". This certificate is called "identity cert". Let say, this identity cert was issued by a CA (Certificate Authority) called "my-ca-server.acme.local". On the CA server, it has a certificate to identify itself. We call that cert a "CA cert".
It is the "CA cert" CUPS server needs to trust. In our example above, it's the "my-ca-server.acme.local" certificate needs to be uploaded to CUPS as "Presence Engine-Trust".
B) When you configure the "Outlook presence gateway" on CUPS, the address you put in there is the address CUPS will use to sent request to Exchange.
For simplicity, quite a few people just put the IP address of Exchange there. This is not going to work with SSL. For security, the request address has to match with the common name in the identity certificate. Usually, the certificate will identity the server in FQDN (e.g. owa.apps.local). Thus, you'll have to configure the same name in presence gateway configuration page.
Please note: I used the word "same name" instead of "FQDN". Because technically, you could configure any name in the certificate. Let's take a look at a couple examples:
Your OWA server IP = 192.168.1.100
Your OWA server FQDN in DNS = owa.apps.local
Example #1: You configured the common name in the OWA cert as "email.apps.local".
This won't work. Because:
If you used "192.168.1.100" in presence gateway configuration, it doesn't match "email.apps.local" in the identify cert.
If you used "owa.apps.local" in presence gateway configuration, it doesn't match "email.apps.local" in the identify cert.
If you used "email.apps.local" in presence gateway configuration, CUPS won't be able to resolve the name, because it's not in DNS.
Or even worse, you don't have DNS configured on CUPS so you can't use any FQDN.
Solution:
1) Add "email.apps.local" to DNS
2) Configure CUPS used DNS (use "set network dns primary" command)
3) Use "email.apps.local" in presence gateway
Example #2
In a lab environment, you may configure the certificate so it identifies the OWA server with common name "192.168.1.100" (though it's not a common practice, it's technically legit).
In this scenario, you may use IP address 192.168.1.100 in presence gateway config.
Whenever you made changes to presence gateway, don't forget to restart Presence Engine service.
Exchange
You don't have to be an Exchange expert. But you should at least know the following:
OWA, WebDAV, Receive-As permission, IIS certificate, CAS/Mailbox roles, authentication methods, Power Shell. (did I say you don't have to be an expert? :) )
OWA
OWA - Outlook Web Access, a web interface to access Exchange emails, calendars, etc. HTTPS needs to be enabled on OWA for CUPS integration to work.
OWA is actually a very useful tool for troubleshooting. For example:
When you tried to access OWA via HTTPS, OWA server will present you a certificate. You may view that certificate from web browser. You should be able to tell what the CA cert is from the web browser.
In the screenshot below, you open a web page to OWA. If you double-click the "lock" icon at the bottom of IE, you'll see a certificate viewer window. From the "certification" tab, you'll see the CA cert is "sametime.apps.local". The identity cert is "owa.apps.local".
Now you know that:
1) You need to upload "sametime.apps.local" cert to CUPS as "Presence Engine - Trust".
2) You need to configure "owa.apps.local" as the "presence gateway" on CUPS > Presence > Presence Gateway.
WebDAV
WebDAV is a protocol (on top of HTTP) to retrieve email/calendar information. See http://en.wikipedia.org/wiki/WebDAV for more details.
CUPS uses WebDAV for calendar integration. Microsoft obsoletes WebDAV on Exchange 2010 and advocates its own protocol EWS (Exchange Web Service). By the time this blog was being written, CUPS doesn't support EWS yet. Which means, CUPS doesn't work with Exchange 2010. I'm not sure if you can have an E2007 CAS in E2010 as a "WebDAV" gateway. I'll test it later.
Though WebDAV sits on HTTP/HTTPS, it's different from OWA. Don't assume WebDAV is working just because OWA was working.
Up to Exchange 2007, WebDAV was enabled by default. You don't have to explicitly enable it. Some people reported that WebDAV was not installed/enabled on E2007 with Win2008. This is not true. The "WebDAV Publishing" in screenshot below has nothing to do with CUPS integration. It's OK to leave it uninstalled.
Since E2007, Microsoft change the OWA URL from /exchange to /owa, but WebDAV URL stays the same (/exchange). This caused more confusions in troubleshooting.
When you troubleshoot CUPS calendar integration, you'll see CUPS always request to URL /exchange, which is the WebDAV URL. This is right and expected.
If you tried to "test" it by typing "https://owa.acme.local/exchange", you'll be redirected to "https://owa.acme.local/owa". This is also right and expected. For more details, please see: http://msexchangeteam.com/archive/2007/02/07/434523.aspx
The point is: there's no easy way to tell if WebDAV is working. But it's easy to tell if WebDAV is NOT working.
If you typed "https://owa.acme.local/exchange" and got some error like "401 unauthorized" or "503 service unavailable", WebDAV is more than likely not working. Though regular OWA might be working fine at this point.
If you got "401 unauthorized", you may take a look at http://msexchangeteam.com/archive/2008/02/01/447989.aspx. Default settings should work. If it doesn't work, most likely your authentication was expecting username in a different format. e.g. "domain\user" instead of "user".
If you got "503 service unavailable" or "500 internal server error", please make sure "ISAPI Extensions" is installed on mailbox server. See screenshot below. This usually happens if mailbox server and CAS are on two different servers.
CertificateOWA/WebDAV relies on IIS to server HTTP request. It's IIS that presents certificate to SSL client.
By default, when you installed IIS, it'll generate a self-signed certificate. Unfortunately, this self-signed certificate doesn't work with CUPS, because it doesn't contain CA bit in its extension. Below is an example of the certificate that HAS the CA bit:
If your CA cert does not have CA bit, CUPS won't trust it. This is per RFC 2459. Don't blame Cisco on that.
Solution? Get a certificate from a CA (either external or internal) or use "makecert.exe" to create a self-signed cert with CA bit (see my book for details).
Receive-As Permission
In order to see everybody's calendar, CUPS needs an account with this permission. So the FAQ is: what's the minimum permission required?
The minimum permission required is "Receive-As" on the end user's mailbox.
Cisco documents said you need "View Only Administrator". That's not true. All you need is "Receive-As" permission on the end user's mailbox. Not more, not less.
How do we assign "Receive-As" permission to the account? I would recommend use Exchange Management Shell (command line).
Let say, in Active Directory, the service account used by CUPS is called "cupsexch". The end user account is called "JDoe" (full name: John Doe). The following command give "cupsexch" Receive-As permission on John Doe's mailbox:
Add-ADPermission -Identity "John Doe" -User cupsexch -ExtendedRights Receive-AsOf course, you're not going to repeat this command for 1000 users. The better way to do it is to assign permission on a "container" that contains the end users' mailboxes.
In Exchange, the "container" for mailboxes is "Mailbox Database". The container for "Mailbox Databases" is "Storage Group". You may have multiple databases in a storage group. And you may have multiple storage groups in your Exchange environment.
You may assign permission at different levels:
End User Mailbox
Mailbox Database
Storage Group
Since "Storage Group" is the highest level of "container", we usually assign permissions on storage group level. Permissions would populate down to end user mailboxes. This "population" takes some time (from 30 minutes to hours).
The command to assign permission on "First Storage Group" is as below:
Add-ADPermission -Identity "First Storage Group" -User cupsexch -ExtendedRights Receive-AsWhat if you have multiple storage groups and want to automate the process? You may use the command below:
Get-StorageGroup | add-ADPermission –user cupsexch -ExtendedRights Receive-As"Get-StorageGroup" command will get all storage group names from system and feed the names to "Add-ADPermission" command.
The command to verify permission is as below:
Get-MailboxPermission jdoe -user cupsexch | Format-Table -autosizeThis command will display "cupsexch" permission on jdoe's mailbox. Anything after the pipe sign (|) is just for formatting purpose.
If you got nothing, that means "no permission". If you assigned the permission on higher containers, it might not have flowed down yet.
If the permission has been populated, you should see something like the screenshot below:
"AccessRights = {FullAccess}" means "cups7exch" account has permission to read "John Doe's" mailbox.
"Is Inhereited = True" means this permission was inherited from some higher level containers.
This is the only reliable way to test permissions.
Cisco documents described a test method which use OWA to test the URL "https://owa-address/exchange/some_user/calendar" with the service account. This method may or may not work depending on other permissions you configured for the service account. For example, if you are like me to give minimum permissions, Cisco test method would yield a "no permission" page as below. However, CUPS calendar works just fine with the "minimum permission".
Troubleshooting
If calendar integration doesn't work, "Presence Engine" (PE) logs are the only traces we need.
Ideally, we want the logs since PE started. Thus if you could, do the following:
1) Restart PE
2) Wait unti PE started and stabilized
3) Log into CUPC
4) Collect PE logs since the time of restart
5) Use WinGrep to search for keyword "owa". It'll give you all message regarding OWA transactions.
Following are some common problems:
"Certificate not trusted"
This could be caused by the following:
- The name configured in "CUPS > Presence > Presence Gateway" does not match with the name in identity certificate. (e.g. IP address vs. FQDN)
- CA certificate was not imported as "Presence Engine - Trust". If there are multiple CAs in the certificate chain, all of them need to be imported.
- CA bit was not set (usually happens on IIS self-signed certificate)
"440 Login Timeout"
When FBA (Form-Based Authentication) was enabled, you'll see one "440 login timeout" in PE log for each transaction. This is normal. If you keep getting "440 login timeout", it usually indicates an authentication issue.
To reveal the nature of the problem, it's recommended to change "Form-Based Authentication" to "Standard Authentication" on "/Exchange" virtual directory on OWA. See screenshot below:
Changing this will not change the way or feel of regular OWA logon. It's just change the authentication method for WebDAV. (you'll need to reset IIS by using command "iisreset /noforce")
"401 Unauthorized"
This is usually caused by permission issue. See "Receive-As" permission configuration above for testing method.
Another testing method would be configuring an end user's credential on presence gateway. An end user's credential would definitely have permission on his own mailbox. If you log in to CUPC with that end user account, calendar integration should work. This is a good way to do "problem isolation".
To be continued...
Saturday, January 23, 2010
Hard Drive Partition and version control of UC Appliance
Active and Inactive Version
Many Cisco Unified Communication appliances (CUCM, CUPS, CER, CUMA, etc.) share the same OS (Cisco customized Linux).
For maintenance purpose, you may install two copies (versions) of systems on the hard drive. Cisco call it "active version" and "inactive version".
CLI commands:
show version active
show version inactive
utils system switch-version
Please note: "active" and "inactive" are relative. When you switch the version (utils system switch-version), the "active" version will becomes "inactive".
If you're a Windows guy, you should be familiar with C:\boot.ini.
If you're a Linux guy, you should be familiar with grub.conf.
It's the same way Cisco UC appliance controls which version to boot from.
Partitions
The two copies of software are installed into two partitions: \ (referred as Partition A) and \partB (referred as Partition B). Whenever you use "utils system switch-version", the active partition will become inactive. The inactive partition will becomes active.
Upgrade and Switch Version
The word "upgrade/patch" has a different meaning in Cisco UC world. Instead of "replacing" files, the "upgrade/patch" process is actually installing a full copy of system in the inactive partition. This has two benefits:
1) You may perform an upgrade/patch during production hours.
2) It's easy to fall back to the old version.
Scenario:
You have 6 CUCM servers in the cluster. Let say, upgrade each server takes about 2 hours. Your business does not allow any downtime during business hours.
Questions:
Q1: How long does it take to upgrade the whole cluster?
Q2: How much time you'll have to spend in after hours for the upgrade?
Answers:
A1: About 4 hours (2 + 2)
A2: A couple minutes.
Explanation:
1. You need to finish the upgrade on CUCM publisher before you can do the upgrade on subscriber. It takes 2 hours to upgrade the publisher. The new code be installed into inactive partition. You don't have to switch to new version right after install. Thus you can do it in business hours.
2. Once the the new version has been installed on publisher (even it's in inactive partition), you may start upgrade process on subscribers (simultaneously). This takes about 2 hours (because you're upgrading all subscribers simultaneously). You don't have to switch to new version right after install. Thus you can do it in business hours.
3. In after hours, you may use "utils system switch-version" command to switch all boxes to new version. This usually takes less than 10 minutes.
However, there's a catch: if you made any configuration changes after the point of upgrade, those changes wouldn't be reflected in the new version. For example, you performed the upgrade at 10AM but didn't switch to new version. Then you switched to new version at 6:30PM. Any configuration changes made between 10AM and 6:30PM will be lost.
Under the hood of "utils system switch-version"
What actually happens when you type the command "utils system switch-version"?
1) It modifies /grub/boot/grub/grub.conf file to make the other partition active
2) It synchronizes UFF (User Faced Feature) to the other version. UFF refers to Call Forwarding, MWI (Message Waiting Indicator), etc.
If the system failed to switch version, here are some options:
Option 1: Try "utils system switch-version nodatasync"
This turn off the UFF data sync action.
Option 2: Use "Recovery CD" (downloadable from CCO) to switch version.
Option 3: If you're a Linux guy, it shouldn't be too difficult for you to get access to /grub/boot/grub/grub.conf.
Many Cisco Unified Communication appliances (CUCM, CUPS, CER, CUMA, etc.) share the same OS (Cisco customized Linux).
For maintenance purpose, you may install two copies (versions) of systems on the hard drive. Cisco call it "active version" and "inactive version".
CLI commands:
show version active
show version inactive
utils system switch-version
Please note: "active" and "inactive" are relative. When you switch the version (utils system switch-version), the "active" version will becomes "inactive".
If you're a Windows guy, you should be familiar with C:\boot.ini.
If you're a Linux guy, you should be familiar with grub.conf.
It's the same way Cisco UC appliance controls which version to boot from.
Partitions
The two copies of software are installed into two partitions: \ (referred as Partition A) and \partB (referred as Partition B). Whenever you use "utils system switch-version", the active partition will become inactive. The inactive partition will becomes active.
Upgrade and Switch Version
The word "upgrade/patch" has a different meaning in Cisco UC world. Instead of "replacing" files, the "upgrade/patch" process is actually installing a full copy of system in the inactive partition. This has two benefits:
1) You may perform an upgrade/patch during production hours.
2) It's easy to fall back to the old version.
Scenario:
You have 6 CUCM servers in the cluster. Let say, upgrade each server takes about 2 hours. Your business does not allow any downtime during business hours.
Questions:
Q1: How long does it take to upgrade the whole cluster?
Q2: How much time you'll have to spend in after hours for the upgrade?
Answers:
A1: About 4 hours (2 + 2)
A2: A couple minutes.
Explanation:
1. You need to finish the upgrade on CUCM publisher before you can do the upgrade on subscriber. It takes 2 hours to upgrade the publisher. The new code be installed into inactive partition. You don't have to switch to new version right after install. Thus you can do it in business hours.
2. Once the the new version has been installed on publisher (even it's in inactive partition), you may start upgrade process on subscribers (simultaneously). This takes about 2 hours (because you're upgrading all subscribers simultaneously). You don't have to switch to new version right after install. Thus you can do it in business hours.
3. In after hours, you may use "utils system switch-version" command to switch all boxes to new version. This usually takes less than 10 minutes.
However, there's a catch: if you made any configuration changes after the point of upgrade, those changes wouldn't be reflected in the new version. For example, you performed the upgrade at 10AM but didn't switch to new version. Then you switched to new version at 6:30PM. Any configuration changes made between 10AM and 6:30PM will be lost.
Under the hood of "utils system switch-version"
What actually happens when you type the command "utils system switch-version"?
1) It modifies /grub/boot/grub/grub.conf file to make the other partition active
2) It synchronizes UFF (User Faced Feature) to the other version. UFF refers to Call Forwarding, MWI (Message Waiting Indicator), etc.
If the system failed to switch version, here are some options:
Option 1: Try "utils system switch-version nodatasync"
This turn off the UFF data sync action.
Option 2: Use "Recovery CD" (downloadable from CCO) to switch version.
Option 3: If you're a Linux guy, it shouldn't be too difficult for you to get access to /grub/boot/grub/grub.conf.
OCS 2007 R2 on Windows 2008 R2 with SQL 2008
Neither Windows 2008 nor SQL 2008 is supported by OCS 2007 R2. But if you really want to do it, here are some tips:
Use SQL 2008 R2
SQL 2008 will fail to install on Windows 2008 R2. Instead of trying to 'fix' it, you may just use SQL 2008 R2.
Pool Creation Failure
a) manually create database rtcconfig (assuming you know how to use SQL Management Studio)
b) manually run the script on OCS installation CD.
D:\Setup\amd64\DbSetup>cscript.exe poolcfgdbsetup.wsf /clean /sqlserver:rcdn /serverrole:EP /verbose
c) continue GUI install
Error: "Not available: IIS 6 Management Compatibility and IIS Windows Authentication role services must be installed before you Deploy Server."
Solution: Install all IIS6 compatibility options. Install all authentication options
Error: "The Windows Media Format Runtime is required in order to install this component. Installing the Windows Media Format Runtime may require a system restart to complete the installation. Click OK to continue with the installation."
Solution: install the Desktop Experience Feature.
Error: "[0xC3EC78D8] Failed to read the Office Communications Server version information. This can happen if the computer clock is not set to correct date and time."
Solution: Uninstall MS Crypto API security update KB974571
Use SQL 2008 R2
SQL 2008 will fail to install on Windows 2008 R2. Instead of trying to 'fix' it, you may just use SQL 2008 R2.
Pool Creation Failure
a) manually create database rtcconfig (assuming you know how to use SQL Management Studio)
b) manually run the script on OCS installation CD.
D:\Setup\amd64\DbSetup>cscript.exe poolcfgdbsetup.wsf /clean /sqlserver:rcdn /serverrole:EP /verbose
c) continue GUI install
Error: "Not available: IIS 6 Management Compatibility and IIS Windows Authentication role services must be installed before you Deploy Server."
Solution: Install all IIS6 compatibility options. Install all authentication options
Error: "The Windows Media Format Runtime is required in order to install this component. Installing the Windows Media Format Runtime may require a system restart to complete the installation. Click OK to continue with the installation."
Solution: install the Desktop Experience Feature.
Error: "[0xC3EC78D8] Failed to read the Office Communications Server version information. This can happen if the computer clock is not set to correct date and time."
Solution: Uninstall MS Crypto API security update KB974571
Friday, January 22, 2010
The art of troubleshooting
Since I joined TAC, I've been the top case solver for 16 consequent quarters no matter what technology group I worked in. I'd like to share some tips on troubleshooting.
Understand the user's expectation.
Still remember the old joke that a user called IT support and said the "cup holder" on his computer stopped working? It turned out to be the CD drive. He insisted he's been using it for years.
Understanding user's expectation can help you determine if you should do customer education or troubleshooting.
Keep it simple
Which one is easier? Troubleshoot a light switch or troubleshoot a space shuttle?
Multiple-system integration adds complexity to the problem. You should try to simplify it as much as possible.
For example: When PSTN call comes in, it hits Unity Auto Attendant. Press 2 to transfer to sales queue, which is a CTI route point handled by UCCX. If no one answers the call, it should goes into a voicemail box dedicated for sales department. Instead of going into voicemail, the caller just heard repeating "transferring..."
In this case, we have too many elements in the picture - Unity, UCCX, CUCM, voice gateway, service provider. Instead of troubleshooting from end to end, we should troubleshoot it segment by segment -
1) What if we called the sale agent directly? If he didn't answer, would the call goes into voicemail? (get Unity Auto Attendant and UCCX out of picture)
2) What if we bypass Unity Attendant Console and call UCCX route point directly? Would it work properly? (get Unity Auto Attendant out of picture)
3) What if we make a test call from internal phone? Would the problem be the same? (get PSTN and voice gateway out of picture)
Other tips to make things simple during troubleshooting:
1) Use default settings. For example, use a "vanilla windows" (fresh installed with Microsoft CD) instead of using a "corporate customized" image.
2) Test on LAN instead of over VPN (again, decrease number of elements)
3) Always assume the system is case sensitive (err on the safe side)
Find a reference point
If a software doesn't work for one user and works for another one, use the good one as reference point and find out the difference.
Of course there are many differences between two users, such as their wife and kids. :) But we should look at the most relevant ones.
Most software nowadays are "client-server" model. The most relevant ones are accounts and computer. Switch the computer (or switch the account) to see if the problem follows the computer or account. If it follows the computer, it might be network or computer settings (client side). If it follows the account, it might be configuration issue (most likely server side).
Understand positive and negative result of the test
e.g.
"Dad, I couldn't find any Easter eggs in the backyard!". Does that mean there's no eggs there?
"Dad, I found some Easter eggs in the backyard!". That means there are some eggs there.
Understand the user's expectation.
Still remember the old joke that a user called IT support and said the "cup holder" on his computer stopped working? It turned out to be the CD drive. He insisted he's been using it for years.
Understanding user's expectation can help you determine if you should do customer education or troubleshooting.
Keep it simple
Which one is easier? Troubleshoot a light switch or troubleshoot a space shuttle?
Multiple-system integration adds complexity to the problem. You should try to simplify it as much as possible.
For example: When PSTN call comes in, it hits Unity Auto Attendant. Press 2 to transfer to sales queue, which is a CTI route point handled by UCCX. If no one answers the call, it should goes into a voicemail box dedicated for sales department. Instead of going into voicemail, the caller just heard repeating "transferring..."
In this case, we have too many elements in the picture - Unity, UCCX, CUCM, voice gateway, service provider. Instead of troubleshooting from end to end, we should troubleshoot it segment by segment -
1) What if we called the sale agent directly? If he didn't answer, would the call goes into voicemail? (get Unity Auto Attendant and UCCX out of picture)
2) What if we bypass Unity Attendant Console and call UCCX route point directly? Would it work properly? (get Unity Auto Attendant out of picture)
3) What if we make a test call from internal phone? Would the problem be the same? (get PSTN and voice gateway out of picture)
Other tips to make things simple during troubleshooting:
1) Use default settings. For example, use a "vanilla windows" (fresh installed with Microsoft CD) instead of using a "corporate customized" image.
2) Test on LAN instead of over VPN (again, decrease number of elements)
3) Always assume the system is case sensitive (err on the safe side)
Find a reference point
If a software doesn't work for one user and works for another one, use the good one as reference point and find out the difference.
Of course there are many differences between two users, such as their wife and kids. :) But we should look at the most relevant ones.
Most software nowadays are "client-server" model. The most relevant ones are accounts and computer. Switch the computer (or switch the account) to see if the problem follows the computer or account. If it follows the computer, it might be network or computer settings (client side). If it follows the account, it might be configuration issue (most likely server side).
Understand positive and negative result of the test
e.g.
"Dad, I couldn't find any Easter eggs in the backyard!". Does that mean there's no eggs there?
"Dad, I found some Easter eggs in the backyard!". That means there are some eggs there.
Saturday, August 29, 2009
Map a UC appliance as a network drive
Remember the "good old days" of CCM 4.x? You can do almost everything on the box. Because it's in fact a Windows 2000 box. However, this brings security and supportability issues.
With the introduction of Linux-based Unified Communication appliance (CUCM 5.x), Cisco locked down the box. You can only access the box via admin web page or a tailored command line.
One of the inconveniences is to review log files. On the old-school CCM 4.x, you may just view the logs in C:\Program Files\Cisco\Trace. On the new UC appliance, you'll have to use RTMT (RealTime Monitoring Tool). This is especially annoying if you're testing your system. For each test, you'll have to download a new set of logs to your computer. (though you may use 'Remote Browse' in RTMT, its function is very limited)
What if we can go back to the "good old days" and view the file system just like a Windows drive?
Take a look at the screenshot below. It's a CUCM 6.1.4 mapped to my Windows XP laptop. You can read/write files on CUCM just like a local hard drive. For those people who are not a fan of VI, you may use your favorite editor (such as Notepad++/UltraEdit). And you may use any Windows tools, such as Windows search, WinGrep, WinZip, etc. How's that? :)
To achieve this, you need two things: a root account on CUCM and a software who can map a SFTP server to a network drive (such as sFTPdrive).
With the introduction of Linux-based Unified Communication appliance (CUCM 5.x), Cisco locked down the box. You can only access the box via admin web page or a tailored command line.
One of the inconveniences is to review log files. On the old-school CCM 4.x, you may just view the logs in C:\Program Files\Cisco\Trace. On the new UC appliance, you'll have to use RTMT (RealTime Monitoring Tool). This is especially annoying if you're testing your system. For each test, you'll have to download a new set of logs to your computer. (though you may use 'Remote Browse' in RTMT, its function is very limited)
What if we can go back to the "good old days" and view the file system just like a Windows drive?
Take a look at the screenshot below. It's a CUCM 6.1.4 mapped to my Windows XP laptop. You can read/write files on CUCM just like a local hard drive. For those people who are not a fan of VI, you may use your favorite editor (such as Notepad++/UltraEdit). And you may use any Windows tools, such as Windows search, WinGrep, WinZip, etc. How's that? :)
To achieve this, you need two things: a root account on CUCM and a software who can map a SFTP server to a network drive (such as sFTPdrive).
Wednesday, July 15, 2009
UC Appliance on VMWare
In theory, any software runs on x86 platform should be able to run on VMWare, unless the software vendor explicitly block it.
Cisco has many software running on x86 platform. We'll discuss Unified Communication products here - CUCM, CER, UCCX, etc.
CUCM
CUCM is the flagship of Cisco UC products. You may install CUCM on VMWare just fine. No hacking is required, but you'll receive a warning that VMWare is not "officially" supported. ie. you shouldn't use it in production. Cisco planned to support VMWare in production in the future ( probably with UC 8).
Though you may install CUCM on VMWare, there are some limitations.
Limitation #1 Licensing
Cisco limits the number of nodes and DLUs on a VMWare MAC address. (3 nodes, 125 DLUs at the time I'm writing this blog). If you should need more than 3 nodes and 125 DLUs, you may change the MAC address of the CUCM (change in VM guest, not in VM host). Just Google keywords "change MAC address on Linux" and you'll find the answer.
Limitation #2 SNMP Agent
You'll notice that "SNMP Master Agent" service fails to start if CUCM was installed on VMWare. This will cause problem if you're testing CER (Emergency Responder). CER needs SNMP connection to CUCM to retrieve phone info. The workaround would be issue the command on CUCM root shell "/sbin/chkconfig snmpd off". Then reboot the server.
Limitation #3 VMWare Acknowledgement
Since from version 7, CUCM requires you acknowledge the "VMWare agreement" during startup. If you reboot the CUCM remotely (either via OS Admin web or via CLI), the server will not boot up until you press the "Agree" button on the console. The workaround is to edit /usr/local/bin/base_scripts/hardware_check.sh,
change the line
if [ "$hwmodel" = "vmware" ];
to
if [ "$hwmodel" = "foobar" ];
On newer versions,
Change the line
if isHardwareUnsupported || isOriginalHardwareUnsupported ; then
To
#if isHardwareUnsupported || isOriginalHardwareUnsupported ; then
if 0 ; then
CER
Besides the limitations mentioned about, CER has another limitation with VMWare.
CER retrieve information from CUCM via SNMP. This includes the machine type of CUCM. If CUCM is running on VMWare, the machine type will be "unsupported" from CER point of view. The workaround is to edit /usr/local/CER/etc/devices.xml file on the CER box. Add the following tag under "CcmHost" family tag:
UCCX
So far, UCCX has been running on "Cisco OS" 2000/2003, which in fact is Windows 2000/2003. However, UCCX will refuse to install if Cisco specific registry key is missing. Follow instructions on http://www.tek-tips.com/viewthread.cfm?qid=930128&page=1 to add the registry key.
Good news is: VMWare is supported since Cisco OS 2003.1.4. If you're using OS 2003.1.4 or newer, you don't need the registry hack.
Another tips is: if you want to bypass the hard drive/memory requirement check, you may create an empty file named "crstest.ini" on C:\. Then CRS won't require 72G HDD/2G memory to install. Of course, this is for testing purpose.
Root shell on UC appliance
Many of the hacking above requiress the root access to the appliance (CUCM, CER, etc.). Just use Google to find the answer. For example: http://www.blindhog.net/how-to-get-root-access-on-call-manager-56-server/
Cisco has many software running on x86 platform. We'll discuss Unified Communication products here - CUCM, CER, UCCX, etc.
CUCM
CUCM is the flagship of Cisco UC products. You may install CUCM on VMWare just fine. No hacking is required, but you'll receive a warning that VMWare is not "officially" supported. ie. you shouldn't use it in production. Cisco planned to support VMWare in production in the future ( probably with UC 8).
Though you may install CUCM on VMWare, there are some limitations.
Limitation #1 Licensing
Cisco limits the number of nodes and DLUs on a VMWare MAC address. (3 nodes, 125 DLUs at the time I'm writing this blog). If you should need more than 3 nodes and 125 DLUs, you may change the MAC address of the CUCM (change in VM guest, not in VM host). Just Google keywords "change MAC address on Linux" and you'll find the answer.
Limitation #2 SNMP Agent
You'll notice that "SNMP Master Agent" service fails to start if CUCM was installed on VMWare. This will cause problem if you're testing CER (Emergency Responder). CER needs SNMP connection to CUCM to retrieve phone info. The workaround would be issue the command on CUCM root shell "/sbin/chkconfig snmpd off". Then reboot the server.
Limitation #3 VMWare Acknowledgement
Since from version 7, CUCM requires you acknowledge the "VMWare agreement" during startup. If you reboot the CUCM remotely (either via OS Admin web or via CLI), the server will not boot up until you press the "Agree" button on the console. The workaround is to edit /usr/local/bin/base_scripts/hardware_check.sh,
change the line
if [ "$hwmodel" = "vmware" ];
to
if [ "$hwmodel" = "foobar" ];
On newer versions,
Change the line
if isHardwareUnsupported || isOriginalHardwareUnsupported ; then
To
#if isHardwareUnsupported || isOriginalHardwareUnsupported ; then
if 0 ; then
CER
Besides the limitations mentioned about, CER has another limitation with VMWare.
CER retrieve information from CUCM via SNMP. This includes the machine type of CUCM. If CUCM is running on VMWare, the machine type will be "unsupported" from CER point of view. The workaround is to edit /usr/local/CER/etc/devices.xml file on the CER box. Add the following tag under "CcmHost" family tag:
member OID="1.3.6.1.4.1.99.1.1.3.28" OIDNAME="vm-ware" CAPTION="VMWare"
UCCX
So far, UCCX has been running on "Cisco OS" 2000/2003, which in fact is Windows 2000/2003. However, UCCX will refuse to install if Cisco specific registry key is missing. Follow instructions on http://www.tek-tips.com/viewthread.cfm?qid=930128&page=1 to add the registry key.
Good news is: VMWare is supported since Cisco OS 2003.1.4. If you're using OS 2003.1.4 or newer, you don't need the registry hack.
Another tips is: if you want to bypass the hard drive/memory requirement check, you may create an empty file named "crstest.ini" on C:\. Then CRS won't require 72G HDD/2G memory to install. Of course, this is for testing purpose.
Root shell on UC appliance
Many of the hacking above requiress the root access to the appliance (CUCM, CER, etc.). Just use Google to find the answer. For example: http://www.blindhog.net/how-to-get-root-access-on-call-manager-56-server/
Subscribe to:
Posts (Atom)
