Let’s talk about FlexVPN, a prime contender as a DMVPN replacement and sometimes referred to as DMVPN phase 4. In this post, I’m going to explore the nuts and bolts of getting FlexVPN up and running between (3) routers and, for added flare, I’m going to also configure dynamic spoke-2-spoke tunnels. The dynamic spoke-2-spoke portion of the config will look very similar to those of you familiar with DMVPN, with some key differences. Before diving in, let’s take a look at our diagram. I’d like to also point out in this lab, the Hub is also our certificate authority (because pre-shared keys are for suckers).
Ingredients
- (3) Cisco Routers. I’ll be using CSR1000v running 16.03.01.
- Patience.
Estimate lab time: ~30min
Configuring IOS CA & Provision Certificates
Since we’ll be using RSA for authentication, as opposed to pre-shared keys, we’re going to need a trusted certificate authority and for each router to have it’s own certificate issued from said CA. Now, in real life this can be either an online or offline CA running on IOS, Linux, or Windows. However, to keep things simple, in this lab I’m just going to configure the Hub router as the CA, then have all routers request a certificate from the CA.
IOS CA Configuration:
ip domain name hop16.com
!
! Replace 12:00:00 10 JUL 2019 with current time and date
do clock set 12:00:00 10 JUL 2019
!
!
ntp master 3
!
crypto pki server flexvpnca
database level complete
no database archive
issuer-name CN=Hub.hop16.com,OU=NOC,L=FOO,C=US
hash sha256
database url flash:
grant auto
no shutdown
%Some server settings cannot be changed after CA certificate generation.
% Please enter a passphrase to protect the private key
% or type Return to exit
Password:
Re-enter password:
% Generating 1024 bit RSA keys, keys will be non-exportable…
!
All Routers (yes, even the Hub/IOS CA from above)
ip domain name hop16.com
!
crypto key generate rsa modulus 2048 label FLEXKEYS
!
crypto pki trustpoint flexvpn-tp
enrollment url http://169.254.0.5:80
hash sha256
subject-name CN={ChangeForEachSite}.hop16.com,OU=NOC,L=FOO,C=US
fqdn {ChangeForEachSite}.hop16.com
# e.g. subject-name CN=S2.hop16.com,OU=NOC,L=FOO,C=US for S2
# e.g. subject-name CN=S3.hop16.com,OU=NOC,L=FOO,C=US for S3
revocation-check crl
rsakeypair FLEXKEYS
!
crypto pki authenticate flexvpn-tp
% Do you accept this certificate? [yes/no]: yes
!
crypto pki enroll flexvpn-tp
# This command will prompt you to create a password for revocation,
# and some additional options about including router serial number,
# and IP address in the certification subject name. Both are
# optional, I opted to include the router’s IP address but
#exclude the serial number.
After a couple seconds, you’ll see a prompt on each router confirming they’ve received their cert from the CA “%PKI-6-CERTRET: Certificate received from Certificate Authority”. At this point, assuming you haven’t hit any errors, we should be able to verify we have a valid certificate from our CA with ‘show crypto pki certificates flexvpn-tp’.
S2#show crypto pki certificates
Certificate
Status: Available
Certificate Serial Number (hex): 03
Certificate Usage: General Purpose
Issuer:
cn=Hub.hop16.com
ou=NOC
l=FOO
c=US
Subject:
Name: S3.hop16.com
IP Address: 169.254.3.5
ipaddress=169.254.3.5+hostname=S3.hop16.com
cn=S3.hop16.com
ou=NOC
l=FOO
c=US
Validity Date:
start date: 14:26:30 UTC Jul 10 2019
end date: 14:26:30 UTC Jul 9 2020
Associated Trustpoints: flexvpn-tp
Storage: nvram:Hubhop16com#3.cer
CA Certificate
Status: Available
Certificate Serial Number (hex): 01
Certificate Usage: Signature
Issuer:
cn=Hub.hop16.com
ou=NOC
l=FOO
c=US
Subject:
cn=Hub.hop16.com
ou=NOC
l=FOO
c=US
Validity Date:
start date: 14:20:00 UTC Jul 10 2019
end date: 14:20:00 UTC Jul 9 2022
Associated Trustpoints: flexvpn-tp
Configuring FlexVPN – Hub and Spoke
The biggest downside to FlexVPN in my humble opinion is that it’s ssoooo config intensive. The benefit is that once you get the configs ironed out, it’s incredibly easy to template. Most everything (including the vast majority of the Hub configs) can be copy/paste to other routers. That said, there are a couple bits of config unique to the Hub, and I’ll call those out as we go.
Hub Router:
aaa new-model
aaa authorization network flexauthz local
aaa session-id common
!
! Defining a local pool is only needed on the Hub router, and only
! required if you wish to have the spokes get their tunnel IP
! dynamically which is preferred.
!
ip local pool flexpool 10.254.0.2 10.254.0.254
!
ip access-list standard flexroutes
permit 10.0.0.0 0.255.255.255
#
# Note here, normally your route ACL only has local routes, but we
# need to be more general on the Hub so S2 knows how to reach S3.
#
!
crypto ikev2 authorization policy flexauthzpol
# Using an address pool here so our spokes can negotiate a tunnel
# IP from the Hub.
pool flexpool
route set access-list flexroutes
!
crypto ikev2 proposal flexprop
encryption aes-gcm-256
prf sha256
group 20
!
crypto ikev2 policy flexpol
proposal flexprop
!
crypto ikev2 profile flexprofile
match identity remote fqdn domain hop16.com
identity local fqdn Hub.hop16.com
authentication remote rsa-sig
authentication local rsa-sig
pki trustpoint flexvpn-tp
aaa authorization group cert list flexauthz flexauthzpol
virtual-template 1
#
# Calling out a virtual-template within our IKEv2 profile is only
# needed if the router has a DVTI interface, which in our initial
# configurations will only be the Hub. Also, if you forget to
# specify this here, your Hub won’t work and you’ll start
# pulling your hair out troubleshooting IKEv2/IPsec messages
# (which will all report successful authentications). Maddening.
#
crypto ipsec transform-set flextset esp-gcm 256
mode tunnel
!
crypto ipsec profile flexprotect
set transform-set flextset
set ikev2-profile flexprofile
!
interface Loopback1
ip address 10.254.0.1 255.255.255.0
!
interface Virtual-Template1 type tunnel
# Again, DVTI interface is only required on the Hub
# while we’re doing a Hub and spoke topology.
ip unnumbered Loopback1
tunnel source GigabitEthernet1
tunnel protection ipsec profile flexprotect
!
Easy peasy right? It’s a lot of config… like all of the config lol. We could trim this down a little if we were doing PSK with an IKEv2 keyring (follow up blog post for that). However, I like doing certificate based authentication so here we are.
Moving on to the spoke router configs, let’s look to S2 as an example. I’ll call out the bits in red that will need to change before copy/pasting this config to S3’s router.
Spoke Router (S2):
ip access-list standard flexroutes
permit 10.0.2.0 0.0.0.255
!
aaa new-model
aaa authorization network flexauthz local
aaa session-id common
!
crypto ikev2 proposal flexprop
encryption aes-gcm-256
prf sha256
group 20
!
crypto ikev2 policy flexpol
proposal flexprop
!
crypto ikev2 profile flexprofile
match identity remote fqdn domain hop16.com
identity local fqdn S2.hop16.com
authentication remote rsa-sig
authentication local rsa-sig
pki trustpoint flexvpn-tp
aaa authorization group cert list flexauthz flexauthzpol
!
crypto ipsec transform-set flextset esp-gcm 256
mode tunnel
crypto ipsec profile flexprotect
set transform-set flextset
set ikev2-profile flexprofile
!
interface Tunnel0
ip address negotiated
tunnel source GigabitEthernet1
tunnel destination 169.254.0.5
tunnel protection ipsec profile flexprotect
See how easy that is to template out though? Absolutely nuts, and we could potentially make this even more generic with only injecting routes to our local interfaces (route set interface instead of route set access-list) and running EIGRP or BGP over the tunnels. Moving on with this config, router S3 has an identical config, except the IKEv2 local identity is S3.hop16.com and my access-list flexroutes is permitting 10.0.3.0 0.0.0.255. Now that our Hub and Spoke FlexVPN is up and running, let’s see what it takes to convert that to a mesh style VPN topology like DMVPN.
Configuring FlexVPN – Mesh (DMVPN Phase 4)
Finally we’re getting at the subject of this post, spoke to spoke dynamic tunnels. The mechanism for getting spoke to spoke communication is the very same used in DMVPN deployments, NHRP. With some key changes.
- No NHRP Next-Hop Server (NHS)
- Instead mapping NBMA to Tunnel IP is derived from IKEv2
- Spokes use SVTI (Tunnel0 interface) for communication to the Hub
- Spokes use DVTI (Virtual-Template) for dynamic tunnels between one another.
Now, in order to make the required changes to enable NHRP we have to shutdown our SVTI (Tunnel0) interfaces on the spokes. Otherwise the Hub’s virtual-template will be locked and IOS will prevent us from making any changes. After shutting down Tunnel0 on both routers, we can add the following to our Hub.
Hub Router:
interface Virtual-Template1 type tunnel
ip nhrp network-id 1337
ip nhrp redirect
Believe it or not, that’s actually we need to add on the Hub side. We’re enabling NHRP redirects, just like DMVPN Phase 3, so the Hub can signal the Spokes to build dynamic tunnels. Also just like DMVPN, the first packet or two will transit the Hub while the Spokes are receiving NHRP information via the redirect message and building their dynamic tunnels. Alright, let’s look at the required config on our Spokes.
Spoke Routers:
crypto ikev2 profile flexprofile
virtual-template 1
!
interface Virtual-Template1 type tunnel
no ip address
ip unnumbered Tunnel0
ip nhrp network-id 1337
ip nhrp shortcut virtual-template 1
ip nhrp redirect
tunnel source GigabitEthernet1
tunnel protection ipsec profile flexprotect
!
interface Tunnel0
ip nhrp network-id 1337
ip nhrp shortcut virtual-template 1
ip nhrp redirect
Much like most things FlexVPN, while easily templated and generic, it’s configuration intensive. That said, this is a super interesting configuration. Where in DMVPN all the traffic happens on our multipoint Tunnel interface, in FlexVPN the tunnel interface remains a static P2P tunnel destined to the Hub. Then we’re pushing our NHRP shortcut to a seperate virtual-template (DVTI) that is will build P2P virtual-access interfaces to other spokes. If we had multiple dynamic tunnels to other spokes each would get their own Virtual-Access interface. Alright, let’s bring the tunnels on the spokes back up, and do some verification.
S2#show ip route static | begin ^Gate
Gateway of last resort is 169.254.2.6 to network 0.0.0.0
S* 0.0.0.0/0 [1/0] via 169.254.2.6
10.0.0.0/8 is variably subnetted, 5 subnets, 3 masks
S 10.0.0.0/8 is directly connected, Tunnel0
S 10.254.0.1/32 is directly connected, Tunnel0
S2#show crypto ikev2 sa
IPv4 Crypto IKEv2 SA
Tunnel-id Local Remote fvrf/ivrf Status
1 169.254.2.5/500 169.254.0.5/500 none/none READY
Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: RSA, Auth verify: RSA
Life/Active Time: 86400/26 sec
Now I’m going to enable NHRP debugs, and try to ping 10.0.3.1 which is connected to S3.
S2#debug nhrp
NHRP protocol debugging is on
S2#ping 10.0.3.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.3.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/8/11 ms
S2#
[…]
*Jul 11 11:53:55.363: NHRP: Sending NHRP Resolution Request for dest: 10.0.3.1 to nexthop: 10.0.3.1 using our src: 10.254.0.9 vrf:global(0x0)
*Jul 11 11:53:55.363: NHRP: Attempting to send packet through interface Tunnel0 via DEST dst 10.0.3.1
*Jul 11 11:53:55.364: NHRP: Send Resolution Request via Tunnel0 vrf global(0x0), packet size: 72
*Jul 11 11:53:55.364: src: 10.254.0.9, dst: 10.0.3.1
[…]
*Jul 11 11:53:55.364: NHRP: 96 bytes out Tunnel0
*Jul 11 11:53:55.372: NHRP: Receive Resolution Request via Tunnel0 vrf global(0x0), packet size: 92
*Jul 11 11:53:55.372: NHRP: Route lookup for destination 10.254.0.9 in vrf global(0x0) yielded interface Tunnel0, prefixlen 32
*Jul 11 11:53:55.373: NHRP: Request was to us. Process the NHRP Resolution Request.
*Jul 11 11:53:55.373: NHRP: nhrp_rtlookup for 10.254.0.9 in vrf global(0x0) yielded interface Tunnel0, prefixlen 32
*Jul 11 11:53:55.373: NHRP: Request was to us, responding with ouraddress
*Jul 11 11:53:55.373: NHRP: Checking for delayed event 10.254.0.8/10.254.0.9 on list (Tunnel0 vrf: global(0x0))
*Jul 11 11:53:55.373: NHRP: No delayed event node found.
*Jul 11 11:53:55.374: NHRP: Enqueued Delaying resolution request nbma src:169.254.2.5 nbma dst:169.254.3.5 reason:IPSEC-IFC: need to wait for IPsec SAs.
*Jul 11 11:53:55.374: NHRP: Interface: Tunnel0 configured with FlexVPN. Deferringcache creation for nhop 10.254.0.8
*Jul 11 11:53:55.374: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to down
[…]
*Jul 11 11:53:55.387: Instructing NHRP to create Virtual-Access from Virtual template 1 for interface Virtual-Access1
*Jul 11 11:53:55.387: NHRP: NHRP Redirect Feature PI-code Initialized
*Jul 11 11:53:55.387: NHRP: Redirect Feature Initialized – Attempting Platform Init
*Jul 11 11:53:55.393: NHRP: Updating delayed event with destination 169.254.3.5 on interfaceTunnel0 with the new interface Virtual-Access1
*Jul 11 11:53:55.595: NHRP:
*Jul 11 11:53:55.595: Fetched address from underlying IKEv2 for interfaceVirtual-Access1. Pre-NATed = 169.254.2.5, Post-NATed = UNKNOWN
*Jul 11 11:53:55.595: NHRP: Processing delayed event on interface Tunnel0 with NBMA 169.254.3.5
[…]
*Jul 11 11:53:55.598: NHRP: No need to delay processing of resolution event nbma src:169.254.2.5 nbma dst:169.254.3.5
*Jul 11 11:53:55.598: NHRP: Attempting to send packet through interface Virtual-Access1 via DEST dst 10.254.0.8
*Jul 11 11:53:55.598: NHRP: Send Resolution Reply via Virtual-Access1 vrf global(0x0), packet size: 120
*Jul 11 11:53:55.598: src: 10.254.0.9, dst: 10.254.0.8
*Jul 11 11:53:55.598: NHRP: 144 bytes out Virtual-Access1
*Jul 11 11:53:55.604: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to up
[…]
!
!
S2#show ip route static | begin ^Gate
Gateway of last resort is 169.254.2.6 to network 0.0.0.0
S* 0.0.0.0/0 [1/0] via 169.254.2.6
10.0.0.0/8 is variably subnetted, 7 subnets, 3 masks
S 10.0.0.0/8 is directly connected, Tunnel0
S % 10.0.3.0/24 is directly connected, Virtual-Access1
S 10.254.0.1/32 is directly connected, Tunnel0
S % 10.254.0.8/32 is directly connected, Virtual-Access1
S2#show crypto ikev2 sa
IPv4 Crypto IKEv2 SA
Tunnel-id Local Remote fvrf/ivrf Status
2 169.254.2.5/500 169.254.3.5/500 none/none READY
Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: RSA, Auth verify: RSA
Life/Active Time: 86400/399 sec
Tunnel-id Local Remote fvrf/ivrf Status
1 169.254.2.5/500 169.254.0.5/500 none/none READY
Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: RSA, Auth verify: RSA
Life/Active Time: 86400/443 sec
IPv6 Crypto IKEv2 SA
I did my best to trim down the NHRP debug messages, but there’s a lot of interesting things happening here. Now looking at our static routes, we have the original summary to 10.0.0.0/8 via Tunnel0 to the Hub, but we also have a specific route to 10.0.3.0/24 and a route to 10.254.0.8/32 via Virtual-Access1 directly to router S3 with the % indicating next hop override (due to NHRP). Well that’s about it for now, I’ll kick out a quick follow up post for those interested in using pre-shared-keys for authentication.
