Networking in AWS – VPC Edition

I’ve been spending some time attempting get a more in-depth knowledge around Amazon Web Services. It is dead simple to start spinning up compute instances and S3 buckets. As I dive deeper in, however, I have started to uncover some of the more complicated topics that a person or an organization would run into while beginning a “journey to the cloud.”

One of those areas is networking. Maybe I gravitate towards that area first since my background involves a heavy dose of traditional on-premise networking. I would imagine anyone in my shoes would have no problem grasping the concepts of VPCs, NAT gateways, and Internet Gateways. Still, there are a certain amount of steps involved in customizing a VPC.

A Virtual Private Cloud (VPC) is a networking space in an AWS footprint. They are unique to regions, meaning you can’t span VPCs between Virginia and Ireland for example. The use cases for building your own customized VPCs include some or all of the following:

  • The ability to pick your IP addressing scheme per data center (could be important in building VPNs from a local data center to a VPC, or VPC-to-VPC peering–larger topics for another time)
  • The ability to separate networking space for different business units such as HR, Finance, IT (And the ability to apply separate, granular firewall controls through security groups and ACLs)
  • The ability to separate dev/test/prod environments from one another

There is a default VPC, which contains default subnets, you can start deploying instances into this networking space right away. But you may want to take a more prescriptive approach to networking in AWS and build your own custom VPCs.

To create a new VPC

  • Go to AWS Services, under “Networking and Content Delivery,” choose VPC
  • Go to “Your VPCs,” click the big blue button that says “Create VPC”
    • Give the VPC a name (I called mine Bob VPC) and a CIDR block (A /16 is the largest block you can create, you will create subnets from this block)


Next, we need to create subnets within our VPC

  • Go to “Subnets,” click “Create Subnet”
  • Give the subnet a friendly name (I chose the format of my VPC name and the network address)
  • choose which VPC it goes in (my BobVPC)
  • you can specify the availability zone in which you want it to reside
  • specify the CIDR block for the subnet (I chose /24s for my example)


In the details pane, you can see how many IP addresses are left, VPC membership, network, and subnet-ID. AWS reserves the first three addresses in any network. I have created two new subnets, one for public instances and one for private instances.


By default, any subnet created in a custom VPC doesn’t assign a public IP to instances created within it. If you want instances in the subnet to automatically get a public IP, you can enable auto-assignment of those public IPs.

  • choose which subnet ( I chose my Bob subnet to be my public-facing subnet)
  • from the dropdown on “Subnet Actions,” choose modify auto-assign IP settings
  • Check the box for auto-assign


For the subnets with public IPs, we need to create a way for those instances to get to the internet. This is an Internet Gateway.

  • Go to Internet Gateway
  • click “Create Internet Gateway”
  • Give the IGW a friendly name ( I chose BobVPC IGW in this case)


  • We then need to associate the IGW with the VPC
  • choose your new IGW and click “Attach to VPC,” select your VPC (Bob VPC)


OK, we have a VPC created, we have a subnet that auto-assigns public IPs to instances within it, we have an internet gateway. We now need to create a route for that subnet to get to the Internet Gateway.

  • Go to “Route Tables”
  • Click “Create Route Table,” lets name it “BobVPC Public Route,” make sure it is associated with the correct VPC


  • Our route table is created, let’s create a route, choose our route table
  • Click on the routes tab and add a new route, by clicking edit
  • Click add another route, since this is default, we are going to do an all 0s destination (
  • Click in the target box, and it should show you the options, choose the IGW you created, Click Save


  • Lastly, we need to associate the subnets with this route table, click on the Subnet tab
  • Click Edit, then choose the subnets you wish to route to the internet, Click Save


We can now create an EC2 instance. During the launch process, choose our new VPC and the subnet you want the instance to live in. Once that instance is launched into my “Bob VPC” subnet, it will get a public IP address and will be able to access the internet via our Internet Gateway.


What if we don’t want our EC2 instance to get a public IP? We have our old friend NAT.

Much like we created an Internet Gateway for our public instance, we create a NAT Gateway for our private instance. The key here is that when creating this NAT Gateway, it needs to be associated with a public subnet.

  • Go to NAT Gateways, click “Create NAT Gateway”
  • Choose a subnet in which to place the NAT GW (This is where we need to choose our public subnet, “Bob VPC”


  • We can have AWS automatically assign our external IP, which is called an Elastic IP


Next we’ll need to add a route to the NAT GW for your private subnets. You can either edit the default route table within your VPC, or create a new route table. I am going to create a new route table.

  • Go to “Route Tables”
  • Click “Create Route Table,” lets name it “Bob VPC NAT,” make sure it is associated with the correct VPC
  • Let’s add a default route, click on the Routes tab, choose Edit
  • Add another route, and choose all 0s again (
  • In the target, you should see the NAT gateway you created, choose that and click save


  • Then we need to associate this route table with our private subnet, click on the Subnet tab
  • Click Edit, then choose your private subnet (in my example it is “Bob VPC NAT”)


Any new EC2 instances created with this subnet will have NAT access to the internet, but will not have a public IP.

Like many of the services within AWS, there is a low barrier to entry for getting started, but once you get past the surface, there is a world of dragons. Beware.

Here is the link to the VPC user guide:



DevOps is People (Or it’s not)

Google what is devops and you’ll be deluged by a lot of definitions that loosely land on some sort of collaboration between development groups and IT operations groups. These definitions rarely delve into what exactly those groups definitions are. Because it is all nebulous. I have taken an interest in this little slice of the IT industry and I had the chance to attend the recent devopsdays MSP here in Minneapolis.

As an old infrastructure guy, I definitely felt out of my element around this crew. I don’t have a dev background, I have done sysadmin work (ops), and only recently have I gotten back into some “dev” work around infrastructure automation. I put that dev in quotes because what I am doing is nothing like real application development, but I am familiarizing myself with the languages and tools used in their world, I just don’t live in that world every day. My main focus over the past 15 years has been around network engineering.

It really clicked for me at this conference that this devops scene is not really about tools, but processes. It’s about people in the sense that people make up a community and that community defines a culture. This community is in IT shops within organizations. A broader community of support and education is evolving around this culture. This conference was part of it.

There were many excellent talks, and as I said, not necessarily about tools, but about the culture of devops. The two that stood out to me were from Brian Liles and Pete Cheslock.

Brian gave a good overview of the state of “devops” for whatever that term means to anybody. One of the themes that he touched on was human element of working in IT. The concept of empathy as a tool is a realization that the longer you work in IT, the more closely you associate with success. Especially as a consultant like me, you need to attempt to understand the circumstances that the person sitting across the desk from you is coming from. Many of the speakers and conversations in the open group sessions kept coming back to empathy. I think this underlines what was most impressed upon me by this conference–people. The devops movement is about people. The community. Teams. The tools are almost secondary, it really is about culture.

Brian stressed “diversity of thought” in the enterprise as a valuable tool. He did give a list of high-level areas that any one working anywhere near the devops space should be familiar with.

  1. Linux – “scripting isn’t necessarily automation.” Automation is “throw this thing over there and it just works, it complains when it doesn’t”
  2. Know networking
  3. Know how sys services works – Systemd, docker
  4. Know tools of space
  5. Most important piece is empathy. Live through how your customer is seeing it…
  6. Continuous integration
  7. Monitoring (point in time and long-term trends) – alerts only go out when they are actionable (reduce noise)
  8. Logging

The second speaker that stood out to me was Pete Cheslock. Mostly because it was a very compelling story that he told. Even as a non-dev, I could relate with projects gone sideways due to changing requirements and unclear direction. Definitely worth your time.


I look forward to my continued immersion into this world. This event was a good way to jumpstart that journey.

ACI Config Backup via Python

The configuration backup/rollback tool in ACI is a great feature that has saved my ass many times.

Any time I start making changes, I go do a one-time backup. If I get too far down a wrong path, I can easily rollback to that snapshot to set the fabric right again.

One thing that always bugged me though, was the inability to give that backup a customizable, user-friendly name.

I discovered this week that if we trigger that backup via the cobra SDK, we can assign any name we want. Here is the code, which is also available on my Github page.

#!/usr/bin/env python

# list of packages that should be imported for this code to work
import cobra.model.fabric
import cobra.model.pol
import cobra.model.config
import credentials
from cobra.internal.codec.xmlcodec import toXMLStr
import requests.packages.urllib3

def take_backup ():
#Login to APIC
ls ='https://'+credentials.ACI_login["ipaddr"], credentials.ACI_login["username"], credentials.ACI_login["password"])
md =

polUni = cobra.model.pol.Uni('')
fabricInst = cobra.model.fabric.Inst(polUni)

backup = cobra.model.config.ExportP(fabricInst, name="backup_created_with_python", snapshot="true", adminSt="triggered")

c =


Palo Alto and NSX Configuration

VMware NSX offers a distributed firewall that applies security policy across your virtual environment and allows for what VMware has coined “micro-segmentation.” Additionally, you can add Palo Alto virtual firewalls to gain further visibility and Next-Gen Firewall capabilities. I recently deployed this in a lab environment and at a customer site. Here is an overview of my experience and understanding.

The Palo Alto documentation for inserting the PAN virtual appliance into the NSX data plane is pretty good. It gives you the step-by-step on “how” to configure it. The thing I find lacking in Palo Alto documentation is the “why.”

For all the qualms I have with Cisco, much of their documentation does a good job explaining the theory behind a certain feature or technology. I find that understanding the why, makes the actual configuration step-by-step instructions more sensible.

That being said, I can expound on a few things in configuring PAN and NSX that is not explicitly mentioned, but came in handy for me understanding what we are doing.

First, the basic overview of service insertion for this scenario.

We are going to establish a connection between Panorama and NSX manager. We are going to tell NSX what traffic to send to the PAN device for inspection. We are going to set up policies in PAN to do that inspection.

To deploy PAN with NSX, you’ll need a license for every ESXi host that is going to have a virtual firewall. You will also need the Palo Alto centralized management console, which is called Panorama. Lastly, you will need a web server that can host the .ovf and .vmdk files that make up the PAN virtual appliance.

An overview of the setup:

We are going to tell NSX manager where it can download the .OVF file for the virtual appliance and where to pull the license file. We are also going to define an IP pool for the management IP of the virtual appliances.

Once all that information is populated, in vCenter networking and security we pick on which hosts we want the virtual appliance installed. Once those are installed correctly, those will show up in Panorama.

We then create security groups in NSX using either dynamic or static membership. We create a distributed firewall rule in NSX to define which traffic is redirected to the Palo Alto appliances.

In Panorama, we define security policies based on the security groups we created in NSX.

This is an important point, the Palo Alto only inspects the traffic you redirect. You can also create exceptions in NSX to specifically define traffic you don’t want redirected.  So you could have an any-any redirect rule, but then define a smaller subset of traffic to not redirect.

My thinking is that if there is traffic you know you would block anyway, it doesn’t make sense to redirect that to the PAN to block, just block it in the distributed firewall in NSX.

One other thing that I ran into during installation is the pre- and post-rules in Panorama. When creating your security policies you have the option for pre- and post-rules. What does that mean?

Panorama only manages the policies that it creates. When the PAN NSX virtual appliance is installed, there is a local policy installed that is a deny all. If you create a post-rule policy in Panorama, that policy will be placed after the local deny all rule, so all traffic will be denied. When creating policies for the NSX virtual PAN, you must create pre-rule policies that will be placed above the local rules.


It might be hard to see in the image, but there is a yellow line between the PAN-NSX policy and the default-deny policy. The PAN-NSX policy is the one created in Panorama as a pre-rule. The default-deny is the local policy.



APIC Failure Testing

I had an interesting conversation the other day about failure of the APICs within an ACI fabric. This particular customer had been burned by another vendor’s fabric solution with regards to failure or upgrade scenarios. I mentioned that the APICs could all blow up and the data plane of the fabric would keep chugging along. This customer said, “I want to see it. I want to pull the power to the controllers and see it.”

I realized that I had been saying this scenario was possible for a while now, but I had never explicitly tested it. I figured I had better ensure this behavior before I have my customer come out and pull the plug on my APICs.

Here is the setup. I have two EPGs on either side of an F5 LTM. This LTM is doing a simple round robin load balancing of some apache web servers. In my test, I am pinging the virtual server on the LTM. I also tested that I could get to the web pages served by the pool defined in the LTM.



I am advertising the bridge domain subnet ( associated with F5 Outside EPG via an L3 Out.

Starting a ping to all three controllers and the VS on the LTM appliance ( I can ping all four addresses. I can also access web pages from servers in the Web EPG that are load balanced by the LTM.

I logged into the CIMC of all three controllers and did a reboot. At this point I lost pings to all three controllers as expected. I did, however, keep my pings to the VS and I was still able to access web pages.


Now I have proof that the APIC failures do not affect data plane traffic. I can rest easy the next time a customer questions that statement.

Update yer APICs (To Live and Die in ACI)

A scenario I ran across last week that I anticipate coming up again: new ACI fabric install and the version of code on the switches is newer than the code on the APICs. In this case the APICs won’t recognize the leaf switches, so I can’t add them to the fabric.

All of the equipment was fresh out of the box, everything was cabled correctly. I did the initial setup on the APICs, but when I went to start adding members to the fabric, there was nothing. I consoled into the leaf switches, a pair of 93180YC-EX, to see what version of code they had. They are running 12.1, which would equal something in the 2.1 code train.

The APICs were running 1.0 code, well before the 93180s were introduced as a product. So it made sense that it wouldn’t recognize them. So how do we get them to talk?

I thought maybe I could downgrade the switches to a lower version. I copied code and moved it into the bootflash directory and tried to run the script to get them to boot into that older version of code. That did not work and I didn’t want to start changing too much with the boot order for fear of bricking a switch.

The second option is to upgrade the APICs, but I can’t upgrade them via the normal means because my fabric isn’t built meaning that my APICs haven’t formed a cluster as of yet.

The process to upgrade the APICs in a standalone fashion is actually pretty straightforward.

  • Assuming you’ve downloaded the APIC code (2.0(2g) in my case), access the server via CIMC.
  • Login to the CIMC and launch KVM console.
  • Once in there, click on activate virtual media and map the CD/DVD to the ISO file you’ve downloaded.
  • Next reboot the server and press F6 during the bootup process to manually select boot location. Choose your mapped virtual media.
  • Follow the on-screen prompts to install the new OS.

In my case it took about 30-45 minutes to load the new OS. Once the new image is installed it will bring you to the initial fabric startup screen again.

One warning is that you will need Java to launch the KVM console. That always brings it own joys.

I imagine this scenario will present itself more often as new switches are introduced into the nexus 9K line.

There is a Cisco reference document here.

UCS and ACI VLAN Creation (Secret San Jose)

To follow-up on the previous post about the UCS Python tool. Here is the script I wrote to create the VLAN pool within UCS and ACI.

The script takes an argument for a VLAN range (for example 100-199).

It takes that VLAN range and creates the VLAN pool in ACI. It also creates those VLANs in UCSM.

It them takes those VLANs and applies them to the vnic templates for the A fabric and B fabric of UCS. The template names would need to be modified to match your environment.