Securing Nutanix Flow Networking VPC workloads using Palo Alto Networks VM-Series Firewall with PBR (Policy Based Routing)

10 min readAug 18, 2021

A VPC or Virtual Private Cloud is an isolated logical unit of the provider network that a tenant has complete control from defining IP address ranges to creation of subnets, gateways and configuration of routing tables. Nutanix AHV now provides multi-tenant VPC capability on a Nutanix AHV cluster. When you’re building your VPC, you need to consider how to secure the resources that you will be hosting in it. A best practice is to secure your infrastructure, utilizing multiple layers of defense and segmentation.

In this article, we will provide an example of network segmentation or isolating one subnet from another by routing the traffice between these sunbets or VPC subnet and external networks, to traverse a security system such as a firewall. In this case we will use Palo Alto Networks VM-Series firewall deployed inside a Nutanix AHV VPC which can also be managed by a Panorama appliance deployed virtually outside the VPC network.

Getting the Image

The Images of Panorama and VM-Series firewall compatible with Nutanix AHV are available for download at https://support.paloaltonetworks.com/.

Login to https://support.paloaltonetworks.com/
Customer Support Portal → Updates → Software Updates
Select the Base Image option for Panorama and PAN-OS and download the Qcow2

Creating Panorama and VM-Series PAN-OS Virtual Appliances on Nutanix AHV

The minimum requirements for Panorama and VM-Series PAN-OS virtual appliances are available on Paloalto Networks documentation site:

The same requirements hold good on Nutanix AHV as well.

The qcow2 images for Panorama and VM-Series can be uploaded to AHV’s image service and be used to create a Virtual Appliance with the above requirements.

Steps to create a Virtual Machine on Nutanix AHV — Prism 5.20 — Creating a VM (AHV)

The Panorama and VM-Series virtual appliance expects serial port connection. Not configuring a serial port will cause longer boot times. So it’s recommended to create a serial port via Nutanix acli command line for these virtual machines running on AHV.

To create a serial port and attach it to the virtual appliance, follow the below steps:

Power off the VAs
ssh/login to any of the CVMs in the AHV cluster and execute the following:
acli vm.serial_port_create <vm-name> type=kServer index=0
Replace <vm-name> with the name of the VM.
Power on the VAs

NOTE:

The qcow2 image of Panorama and VM-Series PAN-OS does not contain virtio balloon driver. For this reason, the memory utilisation statistics for these virtual appliances are not visible in Prism.
In earlier releases of PAN-OS, in-guest dpdk had to be disabled (set system setting dpdk‐pkt‐io off). But from 9.1.0 and above of PAN-OS releases dpdk need not be disabled for AHV environments (enabled by default).
Nutanix AHV also supports bootstrap of VM-Series firewall with an ISO with the same steps mentioned in Paloalto Networks documentation — Bootstrap the VM-Series Firewall

Deployment Modes on Nutanix AHV

Panorama and VM-Series can be configured in the following deployment modes on Nutanix AHV.

Panorama Deployment modes

Supported deployment modes for Panorama on Nutanix AHV

Panorama Mode — Set up a Panorama Virtual Appliance in Panorama Mode
Management-Only Mode — Set up a Panorama Virtual Appliance in Management Only Mode
Panorama Virtual Appliance as a Log Collector — Set Up The Panorama Virtual Appliance as a Log Collector

If a new Panorama virtual appliance is created on AHV and an additional vdisk is attached to the virtual appliance, it would be enabled without any additional configuration. In the below Prism screenshot, 500 GiB of vdisk was attached to the VM before the 1st boot.

The same can be verified from Panorama CLI as shown below:

If additional vDisks are required post deployment to expand the log storage capacity, the same steps will apply as mentioned in Paloalto documentation (Expand Log Storage Capacity on the Panorama Virtual Appliance) assuming the additional vDisk is added via Prism.

In short, Nutanix AHV can support Panorama similar to any other industry standard hypervisor.

VM-Series Deployment Modes

Paloalto VM-Series virtual firewall can be configured in

Standalone Mode
HA mode
vWire mode
Nutanix Flow Networking — Virtual Private Cloud

Nutanix Flow Networking — Virtual Private Cloud

Pre-Reads — https://www.nutanix.com/products/flow/networking

This section of the document describes the concepts and steps to be followed for 3rd party service insertion (ex firewall appliance) in Nutanix VPC on a per tenant basis. In this article we will consider Paloalto Networks VM-series firewall managed by Panorama virtual appliance.

The VM-Series firewall will be deployed in one-arm mode within the tenant’s VPC running on a Nutanix AHV cluster. Below is the network topology diagram as a reference for the rest of the article.

The use case here is for all traffic within subnets of the VPC and traffic from VMs of the VPC subnets to the outside network need to traverse through the firewall, so layer 7 security policy configured on the firewall can be enforced. This capability is achieved via policy based routing capability available in the VPC (Policies) section which we will demonstrate in this example.

For this example we create a Tenant (Tenant2_VPC) with 3 subnets — T2_SN1_10, T2_SN2_20, T2_SNFW_30. The subnets T2_SN1_10 and T2_SN2_20 are for client VMs inside the VPC and T2_SNFW_30 is the subnet for Firewall VMs with the VPC.

Note: Firewalls deployed inside a VPC will need to have their separate subnet than the clients/server VMs inside a VPC if policy based routing needs to be implemented.

Note: The VMs that are assigned vNICs from the above subnets will be assigned the default gateway as mentioned in the above screenshot.

These networks will be visible and available to be attached to VMs under PC → Network & Security → Subnets with the ‘type’ overlay as shown below:

An AHV subnet/VLAN is created to provide ‘External connectivity and NAT’ for the VPC Tenant — Tenant2_VPC, which is ANC_EXT_3116. The purpose of creating this AHV subnet is for the VPC to use it for assigning Floating IPs and SNAT IPs for incoming and outgoing traffic to the VPC. The Floating and SNAT IPs will be assigned from the range of IPs defined below (10.16.16.31–10.16.16.50) as shown below.

The IP assignment of Floating IPs to specific VMs inside the VPC can be visible from PC UI (Network & Security → Floating IPs) as shown below:

Referring to the above topology diagram, the floating IP is mapped to the VM-Series management IP to manage/access it from outside the VPC. This IP will be used by Panorama (which is deployed outside of the VPC) to register and manage the VM-Series firewall (T2-FWSN-vFW1). This is required for the Panorama VM to push and commit security policies from Panorama to the VM-Series firewall within the VPC.

In this example. We create 2 VMs (T2_SN1_VM1 and T2_SN2_VM1), one for each subnet within the VPC and deploy VM-Series firewall (T2_FWSN_vFW1) in one arm mode.

VM creation is similar to any VM creation workflow on AHV. The only difference for VMs in a VPC is in network configuration as shown below:

For VM-Series firewall since it is in one arm mode (T2_FWSN_vFW1), there are 2 interfaces one for management interface and the other for data interface.

Note: the management interface is assigned to a Floating IP from the AHV subnet ANC_EXT_3116 that was configured above. The data interface is not assigned to any Floating IP as it will be used for policy based routing.

The 2 vNICs attached to VM-Series firewall within the VPC will show up as below:

Verify the interfaces are also recognised from VM-Series console:

VM-Series Configuration for PBR (inside VPC)

This section describes the VM-Series configuration for security policies and route traffic (In/Out) of the data interface ethernet1/1.

Create a Zone with Ethernet1/1 (name: T2-Zone)

Create a virtual router (T2-VR) and add the interface Ethernet1/1

Define static routes for traffic from VPC subnets to external and vice versa.

Verify the routing and forwarding table to ensure you have the appropriate routes in place

Create a URL category with the purpose to block traffic from VPC VMs to www.facebook.com

Create a URL filtering profile and map the above URL category to the profile and block access.

Create a security policy with ‘Rule Type’ as ‘intrazone’ and the source zone we defined above

In the Actions section of the security policy, attache the URL Profile we defined above

NOTE: the same workflow can be defined from Panorama or on individual VM-Series.

Save and Commit the configuration

Policy Configuration in VPC to direct traffic to VM-Series

Create a VPC policy to direct traffic to the data interface of VM-Series (Ethernet1/1) before being routed to external destination.

NOTE: 2 policies will need to be created, one each for outgoing traffic and the other for return traffic as shown below.

The below policy is for traffic originating from 172.16.10.0/24 subnet to an external network via VM-Series firewall and rerouted to ethernet1/1 IP.

Similarly configure a policy based routing for the return traffic to 172.16.10.0/24 via the ethernet1/1 of VM-Series

The same configuration applies to other subnets within the VPC that needs policy based routing via the firewall

NOTE: The priority of the ACL determines which ACL will be processed first. The higher the priority number the higher the priority.

Verify layer 7 security policy is taking effect

To verify security policy is taking effect when VMs from subnets in VPC are trying to reach the blocked site.

In this example, we did a curl from VM (T2-SN1-VM1 with IP 172.16.10.46) and it was blocked by the firewall as shown in VM-Series monitoring logs:

The same can also be seen in security policy hit count in VM-series.

Reroute traffic between subnets in a VPC via VM-Series firewall

In this example, we also demonstrate how traffic from 2 VMs, one in each subnet of a VPC can be routed via the firewall using VPC policies.

We have setup VM (T2-SN1-VM1–172.16.10.46) in subnet T2_SN1_10 (172.16.10.0/24) to be an iperf client and VM (T2-SN2-VM1–172.16.20.47) in subnet T2_SN2_20 (172.16.20.0/24) to be an iperf server. The use case is to restrict iperf traffic from T2-SN1-VM1 to T2-SN2-VM1 while allowing other traffic (ex. ping)

This can be achieved by creating VPC policy to redirect traffic to ethernet1/1 of the VM-Series firewall and creating a security policy to deny any iperf between the source and destination.

VPC policy configuration is shown below