Sunday, October 1, 2023

SFTP Server

Network and UC engineers often need a SFTP server.  I've been using FreeFTPd for years due to it's small size and free.  However FreeFTPd's encryption algorithm was outdated and not supported by some version of Cisco UC appliances.  I have been looking for an alternative for quite some time with no luck.  Solarwinds' SFTP server is "Free" but with 4GB limit.  Other products are either too bukly or not free at all.

It turned out that Windows (Server or Desktop) has an "OpenSSH" option which works pretty well.

Go to Apps > Add optional features > Search for OpenSSH Server.  Once installed, there will be two services.  Start these two services.

By default, user has access to C:\, which is also the root directory of the SFTP server.  However, Cisco UC appliance cannot handle that (with the SFTP root as "/C:/").  To change the SFTP root, you may edit the %programdata%\ssh\sshd_config file.  Say, you want to make "C:\SFTP" as the SFTP root directory, add the following line to the file:

ChrootDirectory "C:\SFTP"

Restart the OpenSSH services to take effect.

Friday, August 25, 2023

Minimalist's SIP Lab (Part 2)

With may vendors going virtualization, it is possible to run virtual labs where it requires physical equipment before(such as Cisco ISR routers).  CML, VIRL, GNS3, EVE-NG, etc. we all heard about them.  But how can we run a meaningful lab with minimal resource?  That way, we don't have to fire up the lousy UCS servers?

Take the following diagram as an example.  I was working with a group of developers on a voice product, which require a lot of custom tagging in the SIP messages.  My job is to configure the routers to copy/modify/manipulate the SIP message.  (Yes, a lot of SIP profiles and regex).

It'd be great if I can run the lab on my laptop.  Did I mention that I have 2-3 production VMs (Virtual Machines) running on my laptop already?  Adding GNS3 and some virtual routers will definitely have impact on CPU and memory usage.

I achieved the goal with IOL (I86BI_LINUX-ADVENTERPRISEK9-M, Version 15.7(3)M2), which requires 384MB memory per each router.  The IOL image boots much faster than the CSR1000V or the C8000V virtual routers (which requires 4GB memory each).

I created 3 router instances.  Two of them are acting as CME, with MicroSIP softphone registered.  One of them is acting as a CUBE.  You don't actually need the CUBE license, nor the "mode border-element" command.  To test SIP profiles and SIP header manipulation, you just configure regular voip dial-peers.

As mentioned in the previous blog, Sandboxie comes handy when you need to run multiple softphones on the same computer.  The footprint is much smaller than launching multiple VMs.

With this lab, I can make test calls from CME1 to CME2 via the CUBE.  I can do all kinds of manipulations.

Saturday, August 12, 2023

Minimalist's SIP Lab

SIP has become dominant in UC(Unified Communications) now.  If you're interested in learning/practicing SIP, CUBE, SBC, etc., here are some ideas.

Software lab is always better than hardware lab, because of cost, power consumption, noise, portability, etc.   Unless you want to test transcoding (which requires hardware DPS, like PVDM), software is sufficient for most cases.

If you are reading this, you are probably already familiar with lab software like GNS3, EVE-NG, Virl(CML), etc.  I'll use GNS3 as an example.

There are different virtual images on GNS3.  Each one has its pros and cons.  I personally like IOL (a.k.a. IOU) because it's lightweight (as little as 256MB per instance) and fast boot time.  Most of the L3 IOL image comes with CME(CallManager Express) function.  You may use CME as a IP PBX and register SIP phones to it.  Codec is limited to G.711 only.

! Global configuration for VoIP, with SIP sub-section
voice service voip
 allow-connections sip to sip
  bind control source-interface eth0/0
  bind media source-interface eth0/0
  registrar server expires max 1200 min 300
voice class codec 1
 codec preference 1 g711ulaw
! Global config for SIP registration(CME)
voice register global
 mode  cme
 source-address port 5060
 max-dn 20
 max-pool 10
 authenticate register
! Create DNs to be used in later config
voice register dn  1
 number 2001
! Each register pool is a phone.  MAC address doesn’t matter
voice register pool  1
 id mac 0000.0000.0001
 number 1 dn 1
 dtmf-relay rtp-nte sip-notify
 voice-class codec 1
 username user1 password pass1

Then you may download SIP softphone apps, such as X-Lite(now known as "Bria", MicroSIP, etc.  Register SIP softphone to CME should be straightforward.  Make calls between two or three SIP phones and use debug commands (such as "debug ccsip message") to view the SIP messages is a good starting point.

Most of softphone allows single instance only.  You may work around that by using Sandboxie or virtual machines (such as VMware, VirtualBox, etc.)

If you want to explore the features of CUBE virtually, you may use CSR100v virtual router, which can also be run on GNS3.  There is no license needed.

If you want to play with codecs (such as g.729) or media resources (such as transcoding), you'll need a hardware router with DSPs.  You may get a Cisco ISR4K from eBay for about $200. And get a PVDM4-32 for about $100.  I'd get the routers comes with license (UC/K9 or VSEC/K9), though Cisco didn't seem to enforce CUBE license (yet).

Thursday, August 10, 2023

Cisco Expressway (MRA) and AT&T Wireless Interoperability Issue

When deploying a Cisco Expressway MRA(Mobile Remote Access) solution, I ran into a weird interoperability issue with AT&T wireless.  The symptom was: MRA calls to AT&T Wireless numbers went straight to voicemail without ringing the cell phone at all.  The same MRA call doesn't seem to have problem with other carriers like Verizon, T-Mobile or even AT&T wired phones.

At the first glance, this seems to be a carrier issue and there is not much we can do unless the carrier tells us what's wrong.  The demarcation point is at the CUBE.  I don't have any visibility beyond the CUBE.

I decided to do some troubleshooting within my scope.  I noticed that non-MRA calls didn't seem to have this problem.

If it's only the MRA calls having the problem, it is unfair to point the finger to the carrier.  But on the other hand, this only happens with one carrier.  It must be an interoperability between MRA and that particular carrier.

Both MRA calls and non-MRA calls go through the same CUBE.  I looked at the INVITEs sent from CUBE to carrier.  They are very similar except that the MRA calls have "Max-Forwards: 12" in the SIP messages while non-MRA calls have "Max-Forwards: 69".

I'm not sure if that's the root cause of the problem but that is the only thing sticks out.  By looking at Cisco documentations, Expressway has default Max-Forwards of 15 and CUCM has default of 70.  These values are very close to 12 and 69 from the CUBE logs.

Max-Forwards tag was designed to prevent infinite loops in call routing, similar to the TTL in IP packets or hop-count in routing protocols.  The value will be decreased by 1 on each hop along the path.  If one of the hops has a different value on Max-Forwards, the lower value takes precedence.  The diagram below explains why the MRA calls have a value of 12 while the non-MRA calls have a value of 69.

Without seeing the AT&T Wireless logs, I cannot tell what happened within the cellular network.  But imagine there are 12 or more hops in the cellular network before the call reaches the wireless endpoints (cell phones).  What would happen?

When the Max-Forwards value decreased to 0 on the way, the call will be dropped.  If that happens, the call controller within the cellular network will think the cell phone is unreachable (like when the cell phone is powered off or out of signal).  The call controller will send a REFER (redirect) SIP message back to the originator.  The call will be redirected to the cell phone's voicemail.  This is exactly what happens when the cell phone is "unreachable".

  • If it take less hops for the CUBE to reach the voicemail server (less than 12 hops), the call will be established.  The caller will hear voicemail greetings.
  • If it takes 12 hops or more for the CUBE to reach the voicemail server, the caller will hear reorder tone (fast busy) or the carrier's error announcement.  Because the call will fail for the same reason (Max-Forwards decreased to 0).

In my case, it is the prior.  Again, all these are just my guess, but an educated guess.  Is there a way we can fix this problem without carrier involved?  Of course.

The solution is to change Expressway default value from 15 to 70.  It doesn't necessarily have to be 70.  It just needs to be a value large enough so that the SIP message can survive the number of hops before the Max-Forwards decreased to 0.  Since CUCM has a default value of 70 and it seems to work, I decided to set Expressway to 70 as well.  If you are one of those OCD (Obsessive-compulsive disorder) persons, you may set Expressway to 72.  Then both MRA and non-MRA calls will leave the CUBE with the same value of 69, making it "consistent" from carrier point of view.

After the change, MRA calls to AT&T wireless numbers work as expected.

Friday, December 25, 2020

Guest Shell on CSR1000v 17.3.2

Software used:

  • GNS3 2.2.17
  • VMware Workstation for Windows 16.1.0
  • Windows 10 x64 Version 20H2 (Build 19042.685)
  • Cisco CSR 1000v (csr1000v-universalk9.17.03.02-serial.qcow2)

In VMware Virtual Network Editor, a NAT network was created with subnet address (Your subnet might be different.  But the NAT network should have been created when you install VMware).

When I created GNS3 VM, I told it to use NAT network.  As shown in the picture below, it got a DHCP IP

In GNS3, create a CSR instance and connect to GNS3-VM cloud, so that the CSR can have Internet access (to download software).  If your GNS3-VM has only one NIC, it'd be eth0.  My GNS3-VM has two NICs and the eth1 is connected to the NAT network.  When connecting CSR's Gi1 to the GNS3-VM eth1, we actually put the CSR Gi1 into the NAT network.

When CSR boots up, it shall get a DHCP IP from NAT network.  It also gets the DNS IP from DHCP.

To verify Internet is working, try to ping

Guest Shell is like a service module in the router (like the RSM in Catalyst 5500 switch, or the CUE module in Cisco 2800 router).  A Virtual PortGroup (VPG) is needed to be the gateway between Guest Shell and the physical interfaces.  In Cisco's document, VPG and Guest Shell are configured with private IP and  NAT.  See diagram below.

In home lab, you may do it in a different way.  You may configure VPG with "ip unnumber Gi1".  VGP will use the IP of Gi1.  Then configure the Guest Shell interface in the same subnet as Gi1 (but a different IP).  The advantage is - one less subnet in the network.  No NAT is needed.  The disadvantage is - you need to allocate an IP in the same subnet as Gi1. (this shouldn't be a problem in home lab though).  This option is illustrated in the diagram below:

Enable IOX:

Configure VirtualPortGroup0:
interface VirtualPortGroup0
 ip unnumbered GigabitEthernet1

Check the VPG IP:

Configure Guest Shell parameters.  In the example below, is the VPG IP. is an arbitrary (available) IP in the same subnet. is a DNS server.
app-hosting appid guestshell
 app-vnic gateway0 virtualportgroup 0 guest-interface 0
  guest-ipaddress netmask
 app-default-gateway guest-interface 0
 app-resource profile custom
  cpu 1500
  memory 512

VERY IMPORTANT: the router needs to know how to send the traffic to guest shell:
ip route VirtualPortGroup 0

Enable Guest Shell:

Enter Guest Shell.  Optionally, sudo:
[guestshell@guestshell ~]$ 
[guestshell@guestshell ~]$ sudo su -
Last login: Fri Dec 25 20:45:49 UTC 2020 on pts/4
[root@guestshell ~]# 

DNS resolution within Guest Shell is independent of host platform itself. The name-server configured in "Guest Shell parameters" will automatically get injected into the /etc/resolv.conf file on the CSR1000v. For NX-OS you must explicitly configure the /etc/resolv.conf entry.

[root@guestshell ~]# cat /etc/resolv.conf

Verify Guest Shell can ping Internet host by DNS name.

Check versions:
[root@guestshell ~]# cat /etc/*-release
CentOS Linux release 8.1.1911 (Core) 
NAME="CentOS Linux"
VERSION="8 (Core)"
ID_LIKE="rhel fedora"
PRETTY_NAME="CentOS Linux 8 (Core)"


CentOS Linux release 8.1.1911 (Core) 
CentOS Linux release 8.1.1911 (Core) 
[root@guestshell ~]#             
[root@guestshell ~]# hostnamectl
   Static hostname: guestshell
         Icon name: computer-container
           Chassis: container
        Machine ID: d1eabe2de31449ccbbc0bae3567b0b83
           Boot ID: 222a6b054eda4e3f8bb93705a9bb7a44
    Virtualization: lxc-libvirt
  Operating System: CentOS Linux 8 (Core)
       CPE OS Name: cpe:/o:centos:centos:8
            Kernel: Linux 4.19.106
      Architecture: x86-64
[root@guestshell ~]# 
[root@guestshell ~]# uname -a
Linux guestshell 4.19.106 #1 SMP Fri Oct 2 17:55:01 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@guestshell ~]# 
[root@guestshell ~]# uname -mrs
Linux 4.19.106 x86_64
[root@guestshell ~]# 
[root@guestshell ~]# cat /proc/version
Linux version 4.19.106 (oe-user@oe-host) (gcc version 8.2.0 (GCC)) #1 SMP Fri Oct 2 17:55:01 UTC 2020
[root@guestshell ~]# 

[root@guestshell ~]# python3
Python 3.6.8 (default, Nov 21 2019, 19:31:34) 
[GCC 8.3.1 20190507 (Red Hat 8.3.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.


Sunday, October 18, 2020

CUCM Security Tips

CUCM Security Tips

  • Secure Conference Bridge has special requirement.  Though configuration for secure conference bridge (CFB) is pretty much the same as secure transcoder(TRA) and secure media termination point(MTP), secure CFB requires the register name to match the hostname portion of the subject name (in the certificate/trust-point).  For instance,
    • You router's FQDN is
    • Most likely you would enroll a certificate for the router with a subject name "".
    • On the router, you created a trust point R1-Cert to hold the above certificate.
    • You would think you could use R1-Cert for secure SIP-trunk, secure media resource, etc.  Until it comes to the point that you try to register a secure conference bridge.
      • On CUCM, you saw the conference bridge status is "rejected", yet secure transcoder and MTP are registered.  So it's unlikely certificate problem.
      • On CUCM, you named them R1-CFB, R1-TRA, and R1-MTP

The problem is with the name "R1-CFB".  It has to match with the hostname portion in the certificate (which is "R1" in this case).  OK, no big deal, I'll just change the name from "R1-CFB" to "R1".  Well... you cannot do that.  Because IOS won't accept a conference bridge name shorter than 6 characters or longer than 15 characters.  So you either change the router name (make it between 6 and 15 characters), re-enroll the certificate; or keep the router name/keep the router certificate, and enroll another certificate for conference bridge (yes, for conference bridge only).  For instance,

    • Your router certificate is
    • Your conference bridge certificate is

No, you don't need DNS entry created for those names.  You may use IP addresses and TLS handshake will still succeed.

  • On IOS-XE (all ISR G3 / ISR 4000 series), Cisco decided not to support secure transcoding, which means, when you create transcoding profile on the router, there's no "security" option available.  Available options are:
    • Register non-secure transacoder to CUCM (application sccp)
    • Register LTI transcoder to the router (application cube)
    • No transcoding at all.  on IOS-XE, SRTP to RTP internetworking does not require transcoding.
  • Crypto Mismatch in ad-hoc conference (call dropped)
31948894.000 |11:19:34.039 |SdlSig   |MXErrorReport                          |interfacesEstablished          |MediaExchange(3,100,114,571)     |AgenaInterface(3,100,11,249)     |3,100,247,81913.201^^*      |[R:N-H:0,N:2,L:0,V:0,Z:0,D:0] error=0 CallMediaFailureCause=block unencrypted  media Reason=prepareAndSendOLC - Serv Param 'Block Unencrypted Calls' is true and the call is unencrypted. Hence blocking the call.
31948895.000 |11:19:34.039 |SdlSig   |MXErrorReport                          |waitStopped                    |MediaExchange(3,100,114,571)     |AgenaInterface(3,100,11,249)     |3,100,247,81913.201^^*      |[R:N-H:0,N:4,L:0,V:0,Z:0,D:0] error=0 CallMediaFailureCause=block unencrypted  media Reason=openOutgoingAudioChannel - Serv Param 'Block Unencrypted Calls' is true and sRTP keys not generated successfully. Hence blocking the call.

    The above messages indicate there's a crypto mismatch between the conference resource and the intended call leg.  Use "show sccp" command to see support crypto on conference resource.  e.g.:

    Router#show sccp
    Conferencing Oper State: ACTIVE - Cause Code: NONE
    Active Call Manager:, Port Number: 2443
    TCP Link Status: CONNECTED, Profile Identifier: 1
    Signaling Security: ENCRYPTED TLS
    Media Security: SRTP
    Supported crypto suites: AES_CM_128_HMAC_SHA1_32, AES_CM_128_HMAC_SHA1_80

    On CUBE, create a voice class for crypto and apply it to the dial-peer:

    voice class srtp-crypto 1
     crypto 1 AES_CM_128_HMAC_SHA1_32
     crypto 2 AES_CM_128_HMAC_SHA1_80
    dial-peer voice 50 voip
     voice-class sip srtp-crypto 1

    Reference #1: 

    Reference #2: (For SCCP-based signalling, only TLS_RSA_WITH_AES_128_CBC_SHA cipher suite is supported)

    Use IOS router as CA (Certificate Authority) Server

    If you have a router handy (especially with GNS3), IOS CA is probably the most convenient way to sign certificates.  Though you may set up IOS CA for online enrollment, I strong recommend you practice terminal method.  In most cases, terminal is the quickest (and probably the only) way to get your job done.

    To set up CA server on IOS, you just need a couple commands:

    conf t
    crypto key generate rsa general-keys exportable label myCA modulus 2048
    crypto key export rsa myCA pem url nvram: des myCAkey
    ip http server
    crypto pki server myDB
     database level minimum
     database url nvram:
     issuer-name cn=myCA, l=Dallas, c=US
     lifetime certificate 7305
     grant auto

     no shut

    From a router (say, R1) you want to enroll certificate, do the following:

    crypto key generate rsa label MyRSAkey exportable modulus 2048
    crypto pki trustpoint R1-Cert
     serial-number none
     fqdn none
     ip-address none
     revocation-check none
     rsakeypair MyRSAkey
     enrollment terminal
    crypto pki enroll R1-Cert

    R1 will generate CSR and print it on the terminal.  You're going to copy/paste the CSR to the CA server we created above.  An example output is like below:

    R1(config)#crypto pki enroll R1-cert
    % Start certificate enrollment .. 
    % The subject name in the certificate will include:
    % The fully-qualified domain name will not be included in the certificate
    Display Certificate Request to terminal? [yes/no]: yes
    Certificate Request follows: 
    ---End - This line not part of the certificate request--- 
    Redisplay enrollment request? [yes/no]: no

    On CA server, use global command "crypto pki server myDB request PKCS10 terminal" to sign a CSR.  An example output is like below:

    CA#crypto pki server myDB request PKCS10 terminal
    PKCS10 request in base64 or pem 
    % Enter Base64 encoded or PEM formatted PKCS10 enrollment request.
    % End with a blank line or "quit" on a line by itself.
    % Granted certificate:

    The yellow portion above is our input.  The green portion the the signed certificate (for R1).  We're going to copy/paste it to R1.  This is called "import" a signed certificate into R1.  But before importing a signed certificate, we need to import the signer (CA) certificate first.  On CA, use config command "crypto pki export myDB pem terminal" to export CA certificate.  An example is as below:

    CA(config)#crypto pki export myDB pem terminal
    % The specified trustpoint is not enrolled (myDB).
    % Only export the CA certificate in PEM format.
    % CA certificate:
    -----END CERTIFICATE----- 

    Now on router R1, do the following:

    1. Import CA certificate with config command "crypto pki authenticate R1-Cert".  R1-Cert is the trust point name, which is just a placeholder for a cert and its signer.  Paste the CA cert (the 2nd green block above).
    2. Import R1 cert with config command "crypto pki import R1-Cert cert".  Paste the signed R1 cert (the 1st green block above).

    Now you have successfully installed certificate on R1.

    If you got "Authentication failed - could not validate certificate" when trying to install the R1 certificate, it is most likely you have Revocation-Check enabled at the R1-cert trust-point.

    Instead of:

    crypto pki trustpoint R1-Cert
      revocation-check none

    You might have:

    crypto pki trustpoint R1-Cert
      revocation-check crl

    To fix the problem, set the revocation-check to none.  Then you'll have to re-generate the CSR and have it signed again by CA.

    Wednesday, October 14, 2020

    Revisit "Urgent Priority"

    Cisco CUCM (CallManager) has "urgent priority" option for translation patterns and route patterns.  At the first glance, it is pretty straight forward.  It is usually used with emergency patterns like "911".  The purpose is to eliminate potential inter-digit timeout.  For instance, when user dialed 911, CUCM will route the call immediately, even if there are potential matches (like 911XXX).

    But what happens if we use urgent priority on variable length pattern (like "!")?  Since wildcard ! means "one or more digits", shouldn't the system wait for more digit anyway?  Would interdigit timeout happen or not?

    The short answer is "No".  If you have urgent priority on pattern "!", and you are dialing digit by digit, the system will start routing the call after the first digit is pressed.  Because that matches the definition "one or more digits".  That seems pretty useless.  Why would people do that?

    It is not totally useless.  You may still dial multiple digits with the following options:

    Option 1: Bloc-Dial

    Keep the phone on hook (do not get a dial tone), enter all the digits you want, then hit the "Dial" button.  This is called bloc-dial.  You may pass all digits to ! pattern with urgent priority with bloc-dial.

    Option 2: From previous hop

    In large-scale dial plan design, we usually expose translation patterns(TPs) to phones, but not route patterns(RPs).  The intend is to use TPs to do all kinds of digit manipulation and class of control.  Then pass the manipulated digits to RPs.  In this case, TP has no problem passing all digits to RP (even if the RP has urgent priority).

    Let say, you have a two-tier dial plan design (TP/RP).  You have emergency TPs with urgent priority.  When those TPs pass digits to RPs, it may or may not experience interdigit timeout depending on your RP setup.  Interdigit timeout is evaluated at each hop.  If you use RP ! to catch all digits passed by TPs, you might want to enable urgent priority on that ! pattern.  Or as an alternative, you may enable "Do Not Wait for Interdigit Timeout On Subsequent Hops" on the TP.

    Another interesting topic is the interaction of urgent priority and "longest match".  Take a look at the following patterns:

    • 7XXX
    • 700XX

    When you dial 7, 0, 0, 1, 2 digit-by-digit, which one will be matched?  At the first glance, 700XX seems to be the best candidate because it matches more digits.  Enable urgent priority on 7XXX seems harmless for this dialing string, right?  Actually not.  When urgent priority is enabled on 7XXX, you won't be able to enter the fifth digit.  Once you entered the 4th digit, system immediately routes the calls.  The only way to work around that is to use bloc-dialing.

    In summary:

    1. Interdigit timeout applies to digit-by-digit dialing.  It does not apply to bloc-dialing.
    2. Digits passed by previous hop does NOT equal to bloc-dialing.  For instance, urgent priority on TP level does not necessarily immune to interdigit timeout at RP level.
    3. Be careful of overlapped patterns.  Using urgent priority might have side effects.