ESXi 6.0 on HP DL380 G5

Just upgraded ESXi from 5.5 to 6.0 on my HP DL380 G5 with the "HP Custom Image".  Thought it'd work better than the "vanilla" (generic) image.

The install went well.  But from time to time, it kept giving me the purple screen (system crash), usually 10 - 20 minutes after the VMs boot up.

A search led me to the URL http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2085921

But it didn't fix my problem.

Finally, I decided to "upgrade" with a generic image - overwrite the HP custom version with a generic version.

The installer gave me warning on the RAID controller.  P400 controller was "not supported" by 6.0 any more.  I ignored and continue.  The install completed successfully.

Now I'm living happily ever after.  :)

IOU, GNS3, Cisco VIRL/CML

GNS3 is a well known network emulator software.  Many people compare it with IOU, which is not quite an apple-to-apple compare.

GNS3 is actually a front end add-on to DynamIP - which is the real emulator. I'm not going to compare DynamIP and IOU here as this has been done many times.  You may search Internet for those articles.

DynamIP and IOU requires quite some command line effort to get them running.  Thus some "front end" add-ons were built to make them easier to use.  GNS3 is the most popular front end of DynamIP.  On IOU side, it's probably IOU Web.  These add-ons allow you crate network topologies, save/restore configurations, etc.

From easy-of-use perspective, GNS3 is no doubt a winner.  You may drag-and-drop routers/switches in GNS3.  However, I prefer IOU because it uses much less resource (CPU/memory) and I'm not afraid of writing up the NETMAP file.

IOU Web doesn't have drag-and-drop.  But it has quite a few neat features.  For example, you may use any existing network diagram and map the elements on it to any virtual devices.  It also has a very powerful configuration/lab management interface.

Things become interesting now when GNS3 can act as IOU front-end.

I tested GNS3 version 1.3 with IOU back end and it works pretty well.  I think here's how I would choose between GNS3 and IOU Web:

If I need to create a lab in a very short time (say 20 routers in 15 minutes), I'd use GNS3.  Otherwise I'd probably use IOU Web as it gives me more flexibility on documentation and network drawings.  Also, as a "minimalist", IOU Web doesn't require any front end software (as it has a http server built-in and uses web interface).

How about Cisco VIRL/CML?

Sigh...  when I first heard about the idea two years ago, I was so excited.  Finally, an official / vendor-supported network emulator!  With the progress lagging on, we're getting more and more frustrated and disappointed.

Yes, Cisco VIRL/CML does have some unique features such as auto-generated configuration, TAC support, etc.  But aren't those features more "novice facing"?  The most disappointing part was - Cisco didn't (and has no plan) to invest more on the L2 features.

I had a conversation with the CML product manager lately.  He admitted that the NX-OS feature on CML was "70% done" because quite a few developers in the Nexus team were laid off.  Thus the most interested features such as VDC, VPC are not available on CML.  I asked if there's ANY possibility that Cisco continues the development and complete the rest 30% in the future, his answer was a very definite "No".  Then I asked him what's the selling point of CML?  He just kept repeating "more than 15 nodes" and "TAC supported".

VIRL (personal edition) is selling for $200.  CML (enterprise edition)'s price structure is unknown.  But price is not the point here.  Even if they are free, what's the advantage they have over GNS3 and IOU?

ESXi 5.5 support for CSR 10.5

I've been testing CSR 10.5 (UCM 10.5, UCCX 10.5) on ESXi 5.5 U2.

I kept running into problems I've never seen before, such as the lovely VMware "pink screen" (Pink Screen Of Dead).  Consistent high CPU usage on UCM and UCCX, etc.

I couldn't find the pattern of "pink screen".  But it happens quite a couple time, which I've never seen in the past 8 years with Cisco UC on VMware.

The other issue is high CPU on UCM and UCCX.  The CPU usage was consistently at 99% that I couldn't even log into CLI.

Just FYI that that server is a HP DL380 G5 with 32G of RAM and 8x146G SAS drives (a.k.a. Cisco MCS-7845-H2).

Did some research online.  The culprit seems to be the Cisco OVA file.  Somehow the OVA works fine on ESXi 5.0 but not ESXi 5.5.  The solution is NOT to use the OVA.  Instead of importing the OVA, I manually created the virtual machine with the specification in OVA.

CUCM:
1 CPU 1 Core
4G RAM
80G HDD
LSI Logic Parallel
VMXNET 3 NIC

UCCX:
1 CPU 2 Core
8G RAM
146G HDD
LSI Logic Parallel
VMXNET 3 NIC

Things seem to be much better now.  CPU is around 10%.  Maybe it's just me.  But if you're running into the same issue, it's worth trying.  You don't have to reinstall the VM.  Just create a VM from scratch (without using OVA).  Then point the hard drive to the existing VM.

This is CPU utilization with fresh created OVA.  After the system "warmed up" (about 30 minutes), utilization drops from 99% to 10%.

Some reference links:
http://ciscocollab.wordpress.com/2014/01/28/esxi-5-5-support-latest-information/

http://docwiki.cisco.com/wiki/Unified_Communications_in_a_Virtualized_Environment

https://communities.vmware.com/thread/459962

"Cloud" device in IOU Web

I've been using IOU Web for network emulation.

"Cloud" device is the bridge between internal devices (such as routers within IOU) and external devices (such as PCs, a real/virtual router outside of IOU, etc.).

I'm not going to get into the details of how to set up VMware network or IOU.  There are plenty of documents online about that.

What I'm going to share is the solution to a weird problem.

I wanted to build a simple lab as shown below.  Two LAN segments are connected via two routers back-to-back.

NETMAP file and device config as below.

Pretty straight forward, right?  But the problem is - I cannot turn on device 1 (LAN1).  Notice that device "LAN1" stays in red below which means it's off.

I scratched my head for quite a while.  Tried to tweak the parameters, device ID, naming, IOU host, VMware Network Editor.  No avail.

Then I looked at the logs and noticed the following:

Why it asked me check the NETMAP file?  I don't see any error there.  What is "instance"?  Why is it not found?

After a little bit research, I realized "instance" is the same as "device".  As shown in the diagram above, we have four instances - 1, 2, 3 and 4.

We have problem with instance 1 (LAN1), which is connecting (referencing) instance 2 (R1).  If the system was complaining about "instance not found", it can only be 1 or 2.

I also noticed that instance 4 (LAN2) always works.  What's the difference between 1 and 4?

It turns out that in NETMAP file (connection definition), the "cloud" device cannot be the preceding one.  The "correct" NETMAP should be written like this:

 Notice that instead of "1:0/0 2:0/0", I swap them and make it "2:0/0 1:0/0".  Then try to start the LAN1 device.  There we go:

This seems to be a software bug.  But the point is - a good engineer should be able to recognize the pattern from the symptom, perform deductive reasoning, and propose possible solution.  :)

UC 10.5, ESXi 5.5U2, DL380 G5

My home lab has been collecting dust for a while.  During the weekend, I wanted to refresh it with the latest and greatest, which means:

1) Upgrade the server (MCS7845-H2 a.k.a. HP DL380 G5) BIOS and firmware.
2) Upgrade VMware ESXi 5.0 to 5.5U2.
3) Upgrade UC 7.0 to UC 10.5.

It turned out that upgrading a system that's been collecting dust is VERY different from upgrading a system that's been up and running.

First of all, the system won't boot.  Just gives me long beeps and the "Internal Health" and "External Health" LEDs are both red.  Pull all memory chips out and resit them solves the problem.

Then iLO configuration seems to be lost due low power level of the system battery.  I can't log into iLO at all (the 'default password' is system specific with unique numbers).  Set the "System Maintenance Switch" S1 to "On" bypasses the iLO password.

When trying to upgrade to ESXi 5.5 U2, I got the following error:

I know what it is.  But how could this be not enabled while I have ESXi 5.0 on it before?  Maybe it's also due to the motherboard battery?  Anyway, go into BIOS and enable the "No-Execute Memory Protection".

After ESXi upgrade, I noticed that VMware persuade move from native VM client (based on C#) to "Web Client" (based on Adobe Flash).  The initiative is to move from "fat client" to "thin client" so all new features can be hosted on the vCenter server.   You may still use the "native client" but some of the features will be missing.  Features as basic as editing a version 10 VM settings.

In order to use the "Web Client", you'll have to set up a vCenter server.  Also, to view VM console from web browser, you'll need to install a plug-in, which doesn't work with Internet Explorer (as of today).

When installing UCM 10.5, it took extremely long (> 10 hours).  Further investigation revealed that the array controller battery died.  Without battery, the array controller will disable cache, which makes it very, very slow on a RAID5 (slower than my laptop).

I have multiple options:

Option 1: Order one from eBay.  It's not expensive (~ $12 a piece).  The problem is - this kind of batteries are obsolete.  Thus the ones on eBay are all used ones, which were manufactured a couple years ago.  Who knows how long they'll last.

Option 2: Make my own battery like this: http://opensource.wrenhill.com/?p=63.  Then I can use cheap AA or AAA batteries instead of buying proprietary ones.

Neither of the above options is quick enough for me.  Thus I choose...

Option 3: "Enable Cache Without Battery".

To do this, you'll need ACU (Array Configuration Utility).  You can do it with the ROM-based interface (BIOS).

With VMware ESXi, the easiest way is to download the "offline ACU", which is a CD you boot from.  Then configure the array controller from there.

For a RAID, it's the write operation that takes more time.  Thus you want to make sure the write cache is not zero.

Last but not the least, download HP SPP DVD to update all firmwares and BIOS.

P.S. DHCP doesn't work on UCM 10.5 in case you want to use UCM as a DHCP server.  https://supportforums.cisco.com/discussion/12224526/cucm-105-dhcp-not-working

Network Engineer Should Know A Little Bit Scripting and Excel

I was working on a network migration project for a large enterprise.  They are migrating their Catalyst 6509 network to Nexus (7ks, 5ks, 2ks).

Part of the migration is to move hundreds (if not thousands) of servers from 6509 switches to Nexus 2Ks.

In an ideal world, it would be as easy as copy the interface configuration from 6509 and paste it into N5K (where N2K homed to).  But we don't live in an ideal world.

The challenge we are facing are:

1) There are many local significant VLANs due to poor network design, which means, VLAN 100 on legacy switch may or may not be the same VLAN 100 on new switch.  Thus you cannot just blindly copy the "switchport access vlan 100" command from legacy switch and paste it into new switch.  We might have to create a L2 trunk from legacy switch to new switch.  We might have to create new VLANs and SVIs.

2) Even if the VLANs are perfectly fine, copy/paste the configuration for hundreds of ports are still a tedious work and prone to human errors.  Some Catalyst commands need to be translated into NX-OS commands.

3) Port-mapping is another process prone to human error.  Cabling team might tell you the cable from Catalyst-Switch-23 port G3/27 is going to be moved to FEX-Switch-19 port 11.  If the cabling team fat-fingered the FEX port number, network team could overwrite a FEX port that is currently being used and cause an outage.  Sure you may review the FEX port before applying the changes.  But again, reviewing hundreds of ports is a tedious work.

4) Due to project schedule, cabling team has to build the port-mapping even before the FEX was online at N5K.  Thus they reference the FEX by their grid location (e.g. "AB23") versus the "FEX number" in N5K (e.g. "Ethernet101").  How do we build the configuration script with mapping table referencing grid numbers?

Solution:

Spreadsheet is a very useful tool because:

• (Almost) everyone has a spreadsheet application on their computer (Microsoft Excel)
• Spreadsheet is easy to use and format data, even the user is not very computer savvy (such as the cable guys)
• Formulas can be used to validate data and generate desired results

I asked server team provide us a spreadsheet with servers they want to migrate in the first phase.  Each row of the spreadsheet contains server IP address, subnet mask, default gateway, current switch name and switch port the server is connecting to.

I wrote a VB script to format the "show run" output from switches into Excel spreadsheet with switch name, switch port, and interface configuration.

By cross-referencing server team's spreadsheet and the "show run" spreadsheet (done by computer of course), I have a new spreadsheet that tells me what VLANs and what default GWs are required by the servers.  I review the configuration on new switches.  If VLANs or default GWs are not ready, I submit change request to create them.

This is just the preparation stage.  We haven't got to the FEX script stage yet.

Next is to build a script that translate the Catalyst commands into NX-OS commands in the "show run" spreadsheet.  (You may also do "find/replace".  But IMO, scripting is more flexible).

Next is to use a formula to translate the FEX grid number into N5K FEX numbers (i.e. from "AB23" to "Ethernet101").  Since we have more than one pair of N5K, this can't be done by simply "find/replace).  E.g. "AB23" is corresponding to "Ethernet101" on first pair of N5Ks.  However, "CD45" is corresponding to "Ethernet101" on the 2nd pair of N5Ks.  Excel VLOOKUP function can achieve this.

Next is to use a formula to build the FEX interface configuration.  As we need to look up both switch name and port number, Excel INDEX function is used.

Last but not the least, we also need to factor human errors.

1) For each server on the spreadsheet, we should have old switch name, old port number, new switch name and new switch number.  We cannot migrate the server if one of those was missing.  I build a column to validate this.  If something is missing, the value on corresponding row will be 'ERR'.  Then I can filter all 'ERR' rows by this column.

2) For each port we're migrating, there should be no existing config on the new switch (FEX).  If there's existing config, we might have a conflict.  I build another column to validate this.  Again, it'll generate 'ERR' if a port was already configured.  Then I can filter all 'ERR' rows by this column.

In summary, with VB script and spreadsheet formulas, I save 95% of the time and lower the risk of human errors.

Put a text file on router flash without file transfer

Say, you want to put a text-based file on a router's flash memory.  It could be a license file, a config file, or some scripts.

The 'regular' way is to use TFTP/FTP to transfer the file.  But it could be a problem in some circumstances.  For example:

1) You're accessing the router through a terminal server (console port).  There's no network connectivity between your PC and the router.
2) Firewall/security policy prevents TFTP/FTP from happening.

It would be great if Cisco IOS has a 'notepad' (or 'vi') so we can create/edit the file from IOS CLI.  But it has not.

Fortunately, Cisco IOS has tclsh.  You may use tclsh create a file in flash memory and write some text to it.

Router#tclsh
Router(tcl)#puts [open "flash:script.txt" w+] "Some sample text"
Router(tcl)#tclquit

Router#dir flash:
Directory of flash:/
2 -rwx 2072 Jan 9 2014 10:24:23 -06:00 multiple-fs
3 -rwx 676 Feb 28 1993 18:01:35 -06:00 vlan.dat
4 -rwx 3570 Jan 9 2014 10:24:23 -06:00 private-config.text
5 -rwx 16 Jun 9 2014 09:34:35 -05:00 script.txt
6 drwx 192 Feb 28 1993 18:06:36 -06:00 c2960-lanbasek9-mz.122-55.SE7
562 -rwx 7340 Jan 9 2014 10:24:23 -06:00 config.text

32514048 bytes total (18987520 bytes free)

Router#more flash:script.txt
Some sample text

Router#

 What if you want to create a file with multiple lines?  Just escape the 'enter' with '\n'.  For example:

Router(tcl)#puts [open "flash:script.txt" w+] "Line 1 \n Line 2 \n Line 3"

Hope this helps!

Ref: https://www.cisco.com/c/en/us/support/docs/ip/telnet/116214-technote-technology-00.html