Thanks for tuning in again! We’re back for more of the excitement known as Troubleshooting! Today we’re going to look at little more at some of the more nefarious (my word for the day) things that may come your way. How simple little commands can certainly change the way your lab is going!
In case you haven’t noticed by now, the CCIE lab is a largely psychological event. Technical knowledge is a very good thing, but if you can’t handle the pressure then it doesn’t help much! I still remember my first lab exam. Or more importantly the weeks leading up to my first lab exam, and I couldn’t make simple things work correctly! Stuff I’d been doing for years. And it was all in my head.
So what kinds of things can be on your lab which may have an impact on this stuff? Some are very simple, some are not.
So in the last post, I mentioned a little about process. The process by which we troubleshoot things (or how we even start our lab) may make a tremendous amount of difference in what our outcome, or at least our psychological state may end up being. Remember, you are there for the proctor’s entertainment. And sometimes they are very entertained!
Let’s take an obvious one. What if “no ip routing” was in one of your routers? “What?” you say… “Something THAT obvious, any CCNA could figure out, that’s just plain (insert appropriate word of shock or exasperation here).”
Well, yes and no. A lot depends on WHEN you discover it. So let’s assume the obvious. You check the routing table:
R1
no ip routing
In my case, I still have “debug ip routing” turned on, so there’s LOTS of stuff going on at the moment. In the beginning of your lab, you wouldn’t see a thing.
So if you did your IP address checks, and immediately dove into configuring your lab, you’d get to frame-relay and insist that you were going insane!
If you look really closely, you’ll see something more interesting on that. The packets are going out your FastEthernet0/0 interface. Any idea why? Not treating the local interface as valid there, your router is trying to ARP for the address. In this functional, pre-configured network, SW1 (the other end of Fa0/0) has a route to the IP and therefore will reply with Proxy ARP. If other routers did not have a route, you would see “encapsulation failed” messages in the debug.
What other things can we do? How about this…. Can you assign 140.100.0.1/24 to Fa0/0 for me? Sounds simple…
Fun, huh?
Ever see an OSPF interface not work? Sure… but here’s one. What can cause this?
Before:
So there’s a couple more things to be paranoid about for your lab exam. But how do we check for them? Honestly, as part of your initial discovery, I’d do a very simple command!
There may be many things shown (as above) that we really don’t care about. But some things like the “subnet-zero” and the “routing dynamic” should definitely leap out at us! A quick scan on each device like this can save HOURS of troubleshooting later.
The fixes are simple!
Never fear…. There’s more coming…. Will our router be boiled in hot lava? Will our switch get hijacked by the proctor? Have we been punk’d by the lab? Stay tuned next time for the exciting conclusion to CCIE Troubleshooting!
In case you haven’t noticed by now, the CCIE lab is a largely psychological event. Technical knowledge is a very good thing, but if you can’t handle the pressure then it doesn’t help much! I still remember my first lab exam. Or more importantly the weeks leading up to my first lab exam, and I couldn’t make simple things work correctly! Stuff I’d been doing for years. And it was all in my head.
So what kinds of things can be on your lab which may have an impact on this stuff? Some are very simple, some are not.
So in the last post, I mentioned a little about process. The process by which we troubleshoot things (or how we even start our lab) may make a tremendous amount of difference in what our outcome, or at least our psychological state may end up being. Remember, you are there for the proctor’s entertainment. And sometimes they are very entertained!
Let’s take an obvious one. What if “no ip routing” was in one of your routers? “What?” you say… “Something THAT obvious, any CCNA could figure out, that’s just plain (insert appropriate word of shock or exasperation here).”
Well, yes and no. A lot depends on WHEN you discover it. So let’s assume the obvious. You check the routing table:
That’s an obvious one to figure out. Ok…. What if you don’t check that, but go to implement a routing protocol???TestRouter(config)#do sh ip ro Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route Gateway of last resort is not set TestRouter(config)#no ip routing TestRouter(config)# TestRouter(config)# TestRouter(config)#do sh ip ro Default gateway is not set Host Gateway Last Use Total Uses Interface ICMP redirect cache is empty TestRouter(config)#
Ok, another obvious one. Mostly because the router TELLS you what’s wrong! Let’s hop to another router and look at some not-so-obvious problems with this command! Again, some of my routers are already fully configured from Mock Lab 4, so there’s a functional network going on that we’re going to mess with!TestRouter(config)#router rip IP routing not enabled TestRouter(config)#
Looks like a great setup, and things are working!Rack1R1(config-if)#do sh frame map Serial0/0/0 (up): ip 145.1.125.2 dlci 105(0x69,0x1890), static, CISCO, status defined, active Serial0/0/0 (up): ip 145.1.125.5 dlci 105(0x69,0x1890), dynamic, broadcast, CISCO, status defined, active Serial0/1/0.12 (up): point-to-point dlci, dlci 112(0x70,0x1C00), broadcast status defined, active Serial0/1/0.13 (up): point-to-point dlci, dlci 131(0x83,0x2030), broadcast status defined, active Rack1R1(config-if)#do sh ip int br | ex un Interface IP-Address OK? Method Status Protocol FastEthernet0/0 145.1.17.1 YES manual up up Serial0/0/0 145.1.125.1 YES manual up up Serial0/1/0.12 145.1.12.1 YES manual up up Serial0/1/0.13 145.1.13.1 YES manual up up Loopback0 150.1.1.1 YES manual up up Loopback1 145.1.111.111 YES manual up up Rack1R1(config-if)# Rack1R1(config-if)#do ping 145.1.125.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 145.1.125.2, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 56/59/64 ms Rack1R1(config-if)#do ping 145.1.125.5 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 145.1.125.5, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 28/29/32 ms Rack1R1(config-if)#do ping 145.1.12.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 145.1.12.2, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 56/56/60 ms Rack1R1(config-if)#do ping 145.1.13.3 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 145.1.13.3, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 28/29/32 ms Rack1R1(config-if)#
R1
no ip routing
In my case, I still have “debug ip routing” turned on, so there’s LOTS of stuff going on at the moment. In the beginning of your lab, you wouldn’t see a thing.
Now how’s that for an interesting turn of events???Rack1R1(config)#do ping 145.1.125.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 145.1.125.2, timeout is 2 seconds: ..... Success rate is 0 percent (0/5) Rack1R1(config)#do ping 145.1.125.5 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 145.1.125.5, timeout is 2 seconds: ..... Success rate is 0 percent (0/5) Rack1R1(config)#do ping 145.1.12.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 145.1.12.2, timeout is 2 seconds: ..... Success rate is 0 percent (0/5) Rack1R1(config)#do ping 145.1.13.3 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 145.1.13.3, timeout is 2 seconds: ..... Success rate is 0 percent (0/5) Rack1R1(config)#
Nothing has changed in my mapping or other configuration! In other words, my frame-relay configuration is perfectly fine!Rack1R1(config)#do sh frame map Serial0/0/0 (up): ip 145.1.125.2 dlci 105(0x69,0x1890), static, CISCO, status defined, active Serial0/0/0 (up): ip 145.1.125.5 dlci 105(0x69,0x1890), dynamic, broadcast, CISCO, status defined, active Serial0/1/0.12 (up): point-to-point dlci, dlci 112(0x70,0x1C00), broadcast status defined, active Serial0/1/0.13 (up): point-to-point dlci, dlci 131(0x83,0x2030), broadcast status defined, active Rack1R1(config)#
I can ping other interfaces perfectly fine. Just not frame-relay.Rack1R1(config)#do ping 145.1.17.7 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 145.1.17.7, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/202/1000 ms Rack1R1(config)#
So if you did your IP address checks, and immediately dove into configuring your lab, you’d get to frame-relay and insist that you were going insane!
You’ll notice a couple of incoming BGP messages there as well trying to re-establish that connection. Bottom line is that things LOOK like they are being sent, and yet nothing actually goes out. How long would you spend pulling your hair out? Would you ever think to look at “no ip routing” as your culprit? Perhaps if you have been through this before then “yes” but otherwise, it’s not part of what you would expect for a local interface connection! Process, process, process. Otherwise, definitely a time killer!Rack1R1(config)#do debug ip packet detail IP packet debugging is on (detailed) Rack1R1(config)#do ping 145.1.125.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 145.1.125.2, timeout is 2 seconds: *Feb 16 16:24:57.185: IP: tableid=0, s=145.1.17.1 (local), d=145.1.125.2 (FastEthernet0/0), routed via RIB *Feb 16 16:24:57.185: IP: s=145.1.17.1 (local), d=145.1.125.2 (FastEthernet0/0), len 100, sending *Feb 16 16:24:57.185: ICMP type=8, code=0. *Feb 16 16:24:59.185: IP: tableid=0, s=145.1.17.1 (local), d=145.1.125.2 (FastEthernet0/0), routed via RIB *Feb 16 16:24:59.185: IP: s=145.1.17.1 (local), d=145.1.125.2 (FastEthernet0/0), len 100, sending *Feb 16 16:24:59.185: ICMP type=8, code=0 *Feb 16 16:24:59.745: IP: tableid=0, s=150.1.1.1 (local), d=150.1.6.6 (FastEthernet0/0), routed via RIB *Feb 16 16:24:59.745: IP: s=150.1.1.1 (local), d=150.1.6.6 (FastEthernet0/0), len 145, sending *Feb 16 16:24:59.745: TCP src=65067, dst=179, seq=2727314646, ack=3832900916, win=16289 ACK PSH FIN. *Feb 16 16:25:00.613: IP: s=150.1.4.4 (Serial0/0/0), d=145.1.125.1, len 64, rcvd 1 *Feb 16 16:25:00.613: TCP src=45959, dst=179, seq=2516853527, ack=0, win=16384 SYN *Feb 16 16:25:00.613: IP: tableid=0, s=145.1.125.1 (local), d=150.1.4.4 (FastEthernet0/0), routed via RIB *Feb 16 16:25:00.613: IP: s=145.1.125.1 (local), d=150.1.4.4 (FastEthernet0/0), len 40, sending *Feb 16 16:25:00.613: TCP src=179, dst=45959, seq=0, ack=2516853528, win=0 ACK RST *Feb 16 16:25:01.185: IP: tableid=0, s=145.1.17.1 (local), d=145.1.125.2 (FastEthernet0/0), routed via RIB *Feb 16 16:25:01.185: IP: s=145.1.17.1 (local), d=145.1.125.2 (FastEthernet0/0), len 100, sending *Feb 16 16:25:01.185: ICMP type=8, code=0. *Feb 16 16:25:02.613: IP: s=150.1.4.4 (Serial0/0/0), d=145.1.125.1, len 64, rcvd 1 *Feb 16 16:25:02.613: TCP src=45959, dst=179, seq=2516853527, ack=0, win=16384 SYN *Feb 16 16:25:02.613: IP: tableid=0, s=145.1.125.1 (local), d=150.1.4.4 (FastEthernet0/0), routed via RIB *Feb 16 16:25:02.613: IP: s=145.1.125.1 (local), d=150.1.4.4 (FastEthernet0/0), len 40, sending *Feb 16 16:25:02.613: TCP src=179, dst=45959, seq=0, ack=2516853528, win=0 ACK RST *Feb 16 16:25:03.185: IP: tableid=0, s=145.1.17.1 (local), d=145.1.125.2 (FastEthernet0/0), routed via RIB *Feb 16 16:25:03.185: IP: s=145.1.17.1 (local), d=145.1.125.2 (FastEthernet0/0), len 100, sending *Feb 16 16:25:03.185: ICMP type=8, code=0. *Feb 16 16:25:05.185: IP: tableid=0, s=145.1.17.1 (local), d=145.1.125.2 (FastEthernet0/0), routed via RIB *Feb 16 16:25:05.185: IP: s=145.1.17.1 (local), d=145.1.125.2 (FastEthernet0/0), len 100, sending *Feb 16 16:25:05.185: ICMP type=8, code=0. Success rate is 0 percent (0/5) Rack1R1(config)# Rack1R1(config)#do un all All possible debugging has been turned off Rack1R1(config)#
If you look really closely, you’ll see something more interesting on that. The packets are going out your FastEthernet0/0 interface. Any idea why? Not treating the local interface as valid there, your router is trying to ARP for the address. In this functional, pre-configured network, SW1 (the other end of Fa0/0) has a route to the IP and therefore will reply with Proxy ARP. If other routers did not have a route, you would see “encapsulation failed” messages in the debug.
As soon as we re-enable it, things start working again! And OSPF, and BGP, and PIM.Rack1R1(config)#ip routing Rack1R1(config)# Rack1R1(config)#do ping 145.1.125.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 145.1.125.2, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 56/58/60 ms Rack1R1(config)# *Feb 16 16:29:24.525: %PIM-5-NBRCHG: neighbor 145.1.125.5 UP on interface Serial0/0/0 *Feb 16 16:29:24.529: %PIM-5-DRCHG: DR change from neighbor 145.1.125.1 to 145.1.125.5 on interface Serial0/0/0 Rack1R1(config)#
What other things can we do? How about this…. Can you assign 140.100.0.1/24 to Fa0/0 for me? Sounds simple…
What happened there? I swear I’ve typed IP addresses like that for years. Really, my children can probably type that IP address correctly. What’s wrong? How about “no ip subnet-zero” in your configuration? Would you discover it? Perhaps, perhaps not. But it can be very frustrating if you haven’t seen it before! On a side note, if you configure addresses and THEN use “no ip subnet-zero” your existing interfaces will work just fine, but any new ones within the first available subnet (subnet-zero) cannot be used!TestRouter(config)#do sh run int f0/0 Building configuration... Current configuration : 92 bytes ! interface FastEthernet0/0 no ip address no ip route-cache duplex auto speed auto end TestRouter(config)#int f0/0 TestRouter(config-if)#ip addr 140.100.0.1 255.255.255.0 Bad mask /24 for address 140.100.0.1 TestRouter(config-if)#
Fun, huh?
Ever see an OSPF interface not work? Sure… but here’s one. What can cause this?
Before:
and After:Rack1R1(config-if)#do sh ip o i b Interface PID Area IP Address/Mask Cost State Nbrs F/C Lo0 1 0 150.1.1.1/24 1 LOOP 0/0 VL7 1 0 145.1.125.1/24 64 P2P 1/1 VL6 1 0 145.1.13.1/24 64 P2P 1/1 VL5 1 0 145.1.12.1/24 64 P2P 1/1 VL4 1 0 145.1.17.1/24 1 P2P 1/1 Fa0/0 1 17 145.1.17.1/24 1 BDR 1/1 Se0/1/0.13 1 123 145.1.13.1/24 64 P2P 1/1 Se0/1/0.12 1 123 145.1.12.1/24 64 P2P 1/1 Se0/0/0 1 125 145.1.125.1/24 64 P2MP 1/1 Rack1R1(config-if)#
The virtual-link VL4 is down (could be many things), but I’ve lost a neighbor on Fa0/0. You probably wouldn’t see a change like this in the live running (because that would usually imply something you did!) but let’s pretend I’m setting OSPF up for the first time, and my neighbor on a fast ethernet is not coming up (see above, it does work, and there’s nothing on the other side causing a problem, I promise). So you have local configuration….Rack1R1(config-if)#do sh ip o i b Interface PID Area IP Address/Mask Cost State Nbrs F/C Lo0 1 0 150.1.1.1/24 1 LOOP 0/0 VL7 1 0 145.1.125.1/24 64 P2P 1/1 VL6 1 0 145.1.13.1/24 64 P2P 1/1 VL5 1 0 145.1.12.1/24 64 P2P 1/1 VL4 1 0 0.0.0.0/0 65535 DOWN 0/0 Fa0/0 1 17 145.1.17.1/24 1 DR 0/0 Se0/1/0.13 1 123 145.1.13.1/24 64 P2P 1/1 Se0/1/0.12 1 123 145.1.12.1/24 64 P2P 1/1 Se0/0/0 1 125 145.1.125.1/24 64 P2MP 1/1 Rack1R1(config-if)#
Ok, there’s a hint…. Passive interface. But look in the OSPF section. There is nothing about passive-interface up there!Rack1R1(config-if)#do sh run | s ospf ip ospf network point-to-multipoint ip ospf authentication-key CISCO12 ip ospf authentication null router ospf 1 log-adjacency-changes area 0 authentication message-digest area 17 virtual-link 150.1.7.7 message-digest-key 1 md5 CISCO area 123 authentication area 123 virtual-link 150.1.3.3 message-digest-key 1 md5 CISCO area 123 virtual-link 150.1.2.2 message-digest-key 1 md5 CISCO area 125 virtual-link 150.1.5.5 message-digest-key 1 md5 CISCO network 145.1.12.1 0.0.0.0 area 123 network 145.1.13.1 0.0.0.0 area 123 network 145.1.17.1 0.0.0.0 area 17 network 145.1.125.1 0.0.0.0 area 125 network 150.1.1.1 0.0.0.0 area 0 Rack1R1(config-if)# Rack1R1(config-if)#do sh ip o i f0/0 FastEthernet0/0 is up, line protocol is up Internet Address 145.1.17.1/24, Area 17 Process ID 1, Router ID 150.1.1.1, Network Type BROADCAST, Cost: 1 Transmit Delay is 1 sec, State DR, Priority 1 Designated Router (ID) 150.1.1.1, Interface address 145.1.17.1 No backup designated router on this network Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5 oob-resync timeout 40 No Hellos (Passive interface) Supports Link-local Signaling (LLS) Index 1/7, flood queue length 0 Next 0x0(0)/0x0(0) Last flood scan length is 2, maximum is 15 Last flood scan time is 0 msec, maximum is 0 msec Neighbor Count is 0, Adjacent neighbor count is 0 Suppress hello for 0 neighbor(s) Rack1R1(config-if)#
In fact, there’s nothing on the entire router about passive-interface! There’s a new one! We know that loopbacks are automatically treated as stub hosts unless otherwise specified, but there’s nothing to automatically treat a FastEthernet as passive is there? Particularly not in the “router ospf” section!?!?Rack1R1(config-if)#do sh run | in passive Rack1R1(config-if)#
While it may seem like an innocuous command, and one you may have never seen before… the “no routing dynamic” will cause this passive behavior without typing “passive-interface fa0/0″ anyplace! And if you aren’t looking for it, we can see all sorts of issues with it.Rack1R1(config-if)#do sh run int f0/0 Building configuration... Current configuration : 135 bytes ! interface FastEthernet0/0 ip address 145.1.17.1 255.255.255.0 ip pim sparse-mode duplex auto speed auto no routing dynamic end Rack1R1(config-if)#
So there’s a couple more things to be paranoid about for your lab exam. But how do we check for them? Honestly, as part of your initial discovery, I’d do a very simple command!
There’s a space after the “” character up there (to tell GREP about the special character to follow). That way you avoid anything with just “no” in there and only get “no ” as a match.Rack1R1(config)#do sh run | in no no service password-encryption no ip subnet-zero no ip domain lookup no dspfarm no routing dynamic no ip address no frame-relay inverse-arp IP 102 no frame-relay inverse-arp IP 103 no frame-relay inverse-arp IP 104 no frame-relay inverse-arp IP 113 no ip address no synchronization no auto-summary no ip http secure-server Rack1R1(config)#
There may be many things shown (as above) that we really don’t care about. But some things like the “subnet-zero” and the “routing dynamic” should definitely leap out at us! A quick scan on each device like this can save HOURS of troubleshooting later.
The fixes are simple!
Within about 10 seconds, things are back to working order again.Rack1R1(config)#ip sub Rack1R1(config)#int f0/0 Rack1R1(config-if)#rou dyn Rack1R1(config-if)# *Feb 16 17:06:49.197: %OSPF-5-ADJCHG: Process 1, Nbr 150.1.7.7 on FastEthernet0/0 from LOADING to FULL, Loading Done Rack1R1(config-if)# *Feb 16 17:07:04.225: %OSPF-5-ADJCHG: Process 1, Nbr 150.1.7.7 on OSPF_VL4 from LOADING to FULL, Loading Done Rack1R1(config-if)#
Never fear…. There’s more coming…. Will our router be boiled in hot lava? Will our switch get hijacked by the proctor? Have we been punk’d by the lab? Stay tuned next time for the exciting conclusion to CCIE Troubleshooting!
0 comments:
Post a Comment