APIC Failure Testing

I had an interesting conversation the other day about failure of the APICs within an ACI fabric. This particular customer had been burned by another vendor’s fabric solution with regards to failure or upgrade scenarios. I mentioned that the APICs could all blow up and the data plane of the fabric would keep chugging along. This customer said, “I want to see it. I want to pull the power to the controllers and see it.”

I realized that I had been saying this scenario was possible for a while now, but I had never explicitly tested it. I figured I had better ensure this behavior before I have my customer come out and pull the plug on my APICs.

Here is the setup. I have two EPGs on either side of an F5 LTM. This LTM is doing a simple round robin load balancing of some apache web servers. In my test, I am pinging the virtual server on the LTM. I also tested that I could get to the web pages served by the pool defined in the LTM.



I am advertising the bridge domain subnet ( associated with F5 Outside EPG via an L3 Out.

Starting a ping to all three controllers and the VS on the LTM appliance ( I can ping all four addresses. I can also access web pages from servers in the Web EPG that are load balanced by the LTM.

I logged into the CIMC of all three controllers and did a reboot. At this point I lost pings to all three controllers as expected. I did, however, keep my pings to the VS and I was still able to access web pages.


Now I have proof that the APIC failures do not affect data plane traffic. I can rest easy the next time a customer questions that statement.


Update yer APICs (To Live and Die in ACI)

A scenario I ran across last week that I anticipate coming up again: new ACI fabric install and the version of code on the switches is newer than the code on the APICs. In this case the APICs won’t recognize the leaf switches, so I can’t add them to the fabric.

All of the equipment was fresh out of the box, everything was cabled correctly. I did the initial setup on the APICs, but when I went to start adding members to the fabric, there was nothing. I consoled into the leaf switches, a pair of 93180YC-EX, to see what version of code they had. They are running 12.1, which would equal something in the 2.1 code train.

The APICs were running 1.0 code, well before the 93180s were introduced as a product. So it made sense that it wouldn’t recognize them. So how do we get them to talk?

I thought maybe I could downgrade the switches to a lower version. I copied code and moved it into the bootflash directory and tried to run the setup-clean.sh script to get them to boot into that older version of code. That did not work and I didn’t want to start changing too much with the boot order for fear of bricking a switch.

The second option is to upgrade the APICs, but I can’t upgrade them via the normal means because my fabric isn’t built meaning that my APICs haven’t formed a cluster as of yet.

The process to upgrade the APICs in a standalone fashion is actually pretty straightforward.

  • Assuming you’ve downloaded the APIC code (2.0(2g) in my case), access the server via CIMC.
  • Login to the CIMC and launch KVM console.
  • Once in there, click on activate virtual media and map the CD/DVD to the ISO file you’ve downloaded.
  • Next reboot the server and press F6 during the bootup process to manually select boot location. Choose your mapped virtual media.
  • Follow the on-screen prompts to install the new OS.

In my case it took about 30-45 minutes to load the new OS. Once the new image is installed it will bring you to the initial fabric startup screen again.

One warning is that you will need Java to launch the KVM console. That always brings it own joys.

I imagine this scenario will present itself more often as new switches are introduced into the nexus 9K line.

There is a Cisco reference document here.

UCS and ACI VLAN Creation (Secret San Jose)

To follow-up on the previous post about the UCS Python tool. Here is the script I wrote to create the VLAN pool within UCS and ACI.


The script takes an argument for a VLAN range (for example 100-199).

It takes that VLAN range and creates the VLAN pool in ACI. It also creates those VLANs in UCSM.

It them takes those VLANs and applies them to the vnic templates for the A fabric and B fabric of UCS. The template names would need to be modified to match your environment.

Updating ACI Python SDK

Leaving this here for future reference.

After the ACI fabric is upgraded, you must update the egg files in your acicobra SDK.

The egg files are available on your upgraded controller.



Download the egg files to wherever you store them on your system. Mine are in /home/bob/Cobra.

Once the files are downloaded, uninstall the current acicobra and reinstall the new version.

bob@UbuntuBob:~/Cobra$ pip uninstall acicobra

bob@UbuntuBob:~/Cobra$ sudo easy_install -Z ./acimodel-2.1_1h-py2.7.egg

bob@UbuntuBob:~/Cobra$ sudo easy_install -Z ./acicobra-2.1_1h-py2.7.egg



Cisco UCS Python Conversion

I am working on a project to automate VLAN pools used in ACI and UCS. When you are using a standard DVS (not AVS) with ACI, you define a pool of VLANs that the DVS will use when creating port groups. These port groups are created when you associate an EPG with a VMM domain.

In UCS, it is easy to create a VLAN pool with many members. On the LAN tab, under VLANs, you can specify a range of VLAN IDs. The sucky part comes when you want to add those VLAN IDs to a VNIC. This is the purpose of the script I want to build–to automate the addition of those VLAN IDs to the VNIC and add them to the ACI VLAN pool. The first step for me is figuring out how to add those VLANs to UCS via python.

There is an SDK named  UCSMSDK that you can install through pip. There is a function within this SDK that will take actions performed within UCSM and convert it to python code.

First I install the package.

sudo pip install ucsmsdk

I then run a simple script to log the UCSM actions. This script defines our login credentials, opens the GUI, then runs the convert to ucs python function.

from ucsmsdk.ucshandle import UcsHandle
from ucsmsdk.utils.ucsguilaunch import ucs_gui_launch
from ucsmsdk.utils.converttopython import convert_to_ucs_python
# login to UCS
handle = UcsHandle("", 'admin', 'ucspassword')
# launch the GUI
#launch the convert to ucs python function

The terminal screen will log any actions and spit out the equivalent python code.

In this case I added vlan 2001 and 2002 to a vnic named Env7-gst-A. It converted those GUI actions to a python script. I can then take this code and convert it into a useful script to create VLAN IDs in bulk.

from ucsmsdk.mometa.vnic.VnicLanConnTempl import VnicLanConnTempl
from ucsmsdk.mometa.vnic.VnicEtherIf import VnicEtherIf

mo = VnicLanConnTempl(parent_mo_or_dn="org-root", templ_type="updating-template", nw_ctrl_policy_name="CDP-Enable", descr="", stats_policy_name="default", admin_cdn_name="", switch_id="A", pin_to_group_name="", mtu="9000", policy_owner="local", qos_policy_name="", ident_pool_name="Mac-Pool-Fabric-A", cdn_source="vnic-name", name="Env7-gst-A")
mo_1 = VnicEtherIf(parent_mo_or_dn=mo, default_net="no", name="PYTHON2001")
mo_2 = VnicEtherIf(parent_mo_or_dn=mo, default_net="no", name="PYTHON2002")
handle.add_mo(mo, True)


Note: I use Ubuntu for most of my python work. I had to install Java to get this to work. This method will not work with the HTML5 GUI, only the Java version.

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

These resources were used to research this process:

Installing Java in Ubuntu.

The UCSMSDK documentation.



Scenario: My Customer is an ISP who has deployed an ACI fabric as a backbone for their different customer interconnects. They want to re-use the same VLAN IDs across leaf switches. They want these different connections to be different EPGs even though they are using the same VLAN ID.

There is an L2 setting within ACI that changes the VLAN scope from global to local. Utilizing this setting, we can have overlapping VLAN IDs associated with static bindings on different EPGs.

However, it is not as simple as just changing this one setting. There are some order-of-operations or design caveats. Here is how I have configured this per-port VLAN functionality.

  1. VLAN Pools – You must have different VLAN pools for each instance. Even though the same VLAN IDs will be in the pools. The pools must be unique.
  2. Physical Domains – You must have different physical domains for each instance where you will deploy the overlapping VLANs in EPGs.
  3. Create the AEP associated with the previously created physical domain.
  4. Create the interface policy for the per-port VLAN characteristic. Interface Policies > Policies > L2 Interface. Create a policy user the port-local option.
  5. Create the policy group for include the previously created attributes.
  6. Create the leaf profiles and include the interface selectors.
  7. Create the switch policy profile.
  8. You can then create the ANP and EPGs using the static binding to associate to the same VLAN and interface on the leaf switch.

In summary, the two factors that I ran into during testing this is that you have to configure unique VLAN pools and physical domains for the per-port VLAN to work.


There are a few different ways to interact with ACI. The GUI, the API, or through some sort of orchestration/automation system such as UCS Director. The GUI is confusing and can be cumbersome. Speaking only for myself, I learned through the GUI the concepts of ACI. Understanding the necessary policies and groups helped me know what I needed to configure via the API. I have played around with the python SDK. I don’t have enough background in programming or interacting with APIs to tell you how it compares to other vendors API. Especially when it comes to documentation.

What I wanted to document here is using the post functionality from the GUI. This is an easy way to deploy pieces of an ACI configuration.

In this example we are going to configure a VLAN Pool using the XML functionality.

If we right-click on any existing VLAN pool we can pull the XML or JSON configuration for that object.


I am using XML for this example. Choose configuration only and subtree.


Save that XML file and open to edit. We then edit the XML file with our new information. In this case I am creating a new pool with the name of “BL_New_Pool” and a VLAN range of 2000-20001.

<?xml version="1.0" encoding="UTF-8"?>
<imdata totalCount="1">
	<fvnsVlanInstP allocMode="dynamic" descr="" dn="uni/infra/vlanns-[BL_New_Pool]-dynamic" name="BL_New_Pool" ownerKey="" ownerTag="">
		<fvnsEncapBlk allocMode="dynamic" descr="" from="vlan-2000" name="" to="vlan-2001"/>

Save this edited file. Right click on any object within ACI and choose “Post.” In the Parent DN field, remove all except the base uni/ and then choose your file to import.


This will create your new object within ACI.


This is a quick and easy way to create new object within ACI without all of the clicking.