First off, I know it’s been a while since I’ve posted anything. Between failing the CCIE (again), moving and changing jobs… I’ve been busy. Back off. That being said, I’m getting the itch again, and it’s time to start studying, I don’t care if it takes 20 tries over 3 different blueprints I’m going to get my number in R&S. So to get things rolling, I thought I’d start with my favorite topic – MPLS.
Let’s get some lingo straight first, that way you impress all your friends with new acronyms.
LSR – Label Switch Router. This guy’s life is label switching, LSRs make up your MPLS backbone.
LER – Label Edge Router. Your provider edge (PE), and as the name implies the edge of your MPLS network.
LSP – Label Switch Path. The MPLS path for a particular source and destination.
Now that we know who all the players are, lets take a look at our topology
You’ll notice we’re dealing with more “P” routers than usual, but having a few extra LSRs should help visual the MPLS process. First off, before setting up any VRFs, or doing any l2 vpns let’s take a look at probably the single most important command for troubleshooting MPLS. That’s right kids, I’m talking about show mpls forwarding-table. As a sample we’ll look at the LFIB (Label Forwarding Information Base) on P1.
P1#show mpls forwarding-table
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or VC or Tunnel Id Switched interface
16 Pop Label 11.11.11.11/32 0 Et1/0 172.16.111.11
17 Pop Label 172.16.112.0/24 0 Et1/1 172.16.12.2
Pop Label 172.16.112.0/24 0 Et1/0 172.16.111.11
18 Pop Label 3.3.3.3/32 0 Et1/2 172.16.13.3
19 Pop Label 2.2.2.2/32 0 Et1/1 172.16.12.2
20 Pop Label 172.16.223.0/24 0 Et1/2 172.16.13.3
21 Pop Label 172.16.34.0/24 0 Et1/2 172.16.13.3
Pop Label 172.16.34.0/24 0 Et1/3 172.16.14.4
22 Pop Label 172.16.24.0/24 0 Et1/1 172.16.12.2
Pop Label 172.16.24.0/24 0 Et1/3 172.16.14.4
23 Pop Label 172.16.23.0/24 0 Et1/1 172.16.12.2
Pop Label 172.16.23.0/24 0 Et1/2 172.16.13.3
24 26 22.22.22.22/32 0 Et1/2 172.16.13.3
26 22.22.22.22/32 0 Et1/3 172.16.14.4
25 Pop Label 4.4.4.4/32 0 Et1/3 172.16.14.4
26 Pop Label 172.16.224.0/24 0 Et1/3 172.16.14.4
Alright… what are looking at right?? I’ll get to that, first let’s examine why this command is so darn important, simple. By issuing it on our MPLS enabled routers, we can establish our LSP between two given endpoints. After all, MPLS is multiprotocol label switching, so when we’re troubleshooting it we should always give the labels the majority of our attention. Now, what are we looking at? The first column is our local labels, generated by the router we’re on. The second column, when all is working well, will have either an out going label received (in this case) via LDP or it will show “Pop Label”. Without digging too deep, Pop Label comes from penultimate hop popping, really fun to say by the way. This is the process whereby the second to the LSR in an LSP will pop the label off an mpls packet, this is an efficiency mechanism.
Now back to show mpls forwarding-table. Let’s say there is a VPN built between PE1 and PE2, routing tables on the CEs look correct, import export statements look good while you’re troubleshooting. However, CE1 and not ping CE2. So spoiler alert, I played proctor and jacked up P3 with some control plane policing, more on that later. Further more, since the LFIB is built off the FIB I adjusted the ospf cost on P4’s link to PE2 so that P3 was the preferred path. VPNs are built between the PEs, and for a VPN to work there has to be a fully functional LSP between their peering address (loopbacks in our example). So starting on PE1, we’ll verify we have labels to get to 22.22.22.22 (PE2’s loopback).
PE1#show mpls forwarding-table 22.22.22.22
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or VC or Tunnel Id Switched interface
26 24 22.22.22.22/32 0 Et1/0 172.16.111.1
26 22.22.22.22/32 0 Et1/1 172.16.112.2
So far so good. We have egress labels of 24 and 26 going to P1 and P2 respectfully. Next hop, I’m going to look at P1, but you could go to either.
P1#show mpls forwarding-table 22.22.22.22
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or VC or Tunnel Id Switched interface
24 No Label 22.22.22.22/32 1523 Et1/2 172.16.13.3
Trouble in paradise, P1 shows that it’s forwarding the packet out with no label towards P3, and has no other path. Let’s see what P2 shows.
P2#show mpls forwarding-table 22.22.22.22
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or VC or Tunnel Id Switched interface
26 No Label 22.22.22.22/32 126 Et1/3 172.16.23.3
Same thing, at this point we can jump onto P3 and start diagnosing. First up, look at the labels.
P3#show mpls forwarding-table 22.22.22.22
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or VC or Tunnel Id Switched interface
26 No Label 22.22.22.22/32 1218 Et1/0 172.16.223.22
Verify it has any labels
P3#show mpls forwarding-table
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or VC or Tunnel Id Switched interface
16 No Label 11.11.11.11/32 0 Et1/2 172.16.13.1
No Label 11.11.11.11/32 0 Et1/3 172.16.23.2
17 No Label 2.2.2.2/32 0 Et1/3 172.16.23.2
18 No Label 1.1.1.1/32 0 Et1/2 172.16.13.1
19 No Label 172.16.24.0/24 0 Et1/3 172.16.23.2
No Label 172.16.24.0/24 0 Et1/1 172.16.34.4
20 No Label 172.16.14.0/24 0 Et1/2 172.16.13.1
No Label 172.16.14.0/24 0 Et1/1 172.16.34.4
21 No Label 172.16.112.0/24 0 Et1/3 172.16.23.2
22 No Label 172.16.111.0/24 0 Et1/2 172.16.13.1
23 No Label 172.16.12.0/24 0 Et1/2 172.16.13.1
No Label 172.16.12.0/24 0 Et1/3 172.16.23.2
24 No Label 4.4.4.4/32 0 Et1/1 172.16.34.4
25 No Label 172.16.224.0/24 0 Et1/0 172.16.223.22
26 No Label 22.22.22.22/32 0 Et1/0 172.16.223.22
Well that’s not good! Why doesn’t P3 have any labels?? I know I’ve told you the problem already, but humor me. So what next? We verify the router has any mpls enabled interfaces, then if we have any neighbors.
P3#show mpls interfaces
Interface IP Tunnel BGP Static Operational
Ethernet1/0 Yes (ldp) No No No Yes
Ethernet1/1 Yes (ldp) No No No Yes
Ethernet1/2 Yes (ldp) No No No Yes
Ethernet1/3 Yes (ldp) No No No Yes
P3#show mpls ldp neighbor
P3#
So from this we see that all the expected interfaces are in fact MPLS enabled and using LDP. However we have no LDP. Since LDP is our current environment’s mechanism for exchanging labels, this would explain why we’re having issues. This point requires you to know a little more about LDP, LDP sends hello messages out 224.0.0.2 on udp port 646. However, the LDP session is actually built between the LDP router-ids (loopback interfaces for us, this is also the LDP/TDP default behavior) on TCP port 646. That being said, you need to check a couple things on R3
Can we ping our neighbors? Can we ping their LDP RIDs (loopbacks here) from our RID? We’ll pick on R1 for these tests
P3#ping 172.16.13.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.13.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/20/28 ms
P3#ping 1.1.1.1 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 3.3.3.3
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/21/60 ms
Alright that looks good, so now back our TCP and UDP ports… do we have any access-lists configured?
P3#sh ip access
Extended IP access list 100
10 permit ospf any any (949 matches)
20 permit tcp any any eq bgp
30 permit tcp any eq bgp any
40 permit udp any any eq 646 (314 matches)
Extended IP access list 101
10 permit icmp any any (10 matches)
Extended IP access list 103
10 permit ip any any (2229 matches)
We do! Great… now where are they applied??
P3#show run | inc 101
match access-group 101
access-list 101 permit icmp any any
“match access-group” sounds like a class-map, let’s take a look.
!
class-map match-all ICMP
match access-group 101
class-map match-all ALL
match access-group 103
class-map match-any ROUTING
match access-group 100
!
end
P3#show run policy-map
Building configuration…
Current configuration : 141 bytes
!
policy-map CoPP
class ROUTING
class ICMP
police rate 100 pps
exceed-action drop
class ALL
drop
class class-default
!
end
And finally, where is it applied?
P3#show policy-map interface <– Nothing here
P3#show policy-map control-plane <– Nailed it!
Control Plane
Service-policy input: CoPP
Class-map: ROUTING (match-any)
… out ommitted
So now we can modify CoPP to allow for tcp communication on 646. After doing that LDP will come back up, and CE1 will be able to ping CE2. Mystery solved.
