BRKARC-3470 - Cisco Nexus 7K

Cisco Nexus 7000 / 7700 Switch
Architecture
BRKARC-3470
Rohan Grover (rohang@cisco.com)

@mountainviewer
Technical Marketing Manager
Session Abstract
This session presents an in-depth study of the architecture of the latest generation
of Nexus 7000 and Nexus 7700 data center switches. Topics include supervisors,
fabrics, I/O modules, forwarding engines, and physical design elements, as well
as a discussion of key hardware-enabled features that combine to implement
high-performance data center network services.
BRKARC-3470 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Session Goal
• To provide a thorough understanding of the Nexus 7000 / Nexus 7700 switching
architecture, supervisor, fabric, and I/O module design, packet flows, and key
forwarding engine functions
• This session will examine the Nexus 7700 system, as well as the latest additions
to the Nexus 7000
• This session will not examine NX-OS software architecture or other Nexus
platform architectures
What Is Nexus 7000?
Data-center class Ethernet switch designed to deliver high performance, high availability,
system scale, and investment protection
Nexus 7000 designed for general-purpose Data Center deployments, focused on 10G
density plus 40G/100G Supervisor Engines
I/O Modules
Chassis
Fabrics
What Is Nexus 7700?
Data-center class Ethernet switch designed to deliver high performance, high availability,
system scale, and investment protection
Nexus 7700 designed for SP and MSDC Data Center deployments, focused on high-
density 40G/100G
Supervisor Engine
I/O Modules
Chassis
Fabrics
Nexus 7000 / Nexus 7700 – Common Foundation
Nexus 7000 Nexus 7700
General purpose DC switching w/10/40/100G Targeted at Dense 40G/100G deployments
Common Foundation
• Same release vehicles, versioning, feature-sets

• Common configuration model
• Common operational model
• Common fabric ASICs (Fab2) and architecture

• Same central arbitration model
• Same VOQ/QoS model
• Identical forwarding ASICs (F2E, F3)

• Consistent hardware feature sets
• Parallel evolution of hardware capability/scale
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Fabric Architecture
• Forwarding Engine Architecture
• I/O Module Queuing
• Layer 2 Packet Walk
• Conclusion
BRKARC-3470 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 7000 Chassis Family NX-OS 4.1(2) and later
Back
25RU
21RU Side Side
Front
Front N7K-C7010 Rear N7K-C7018

Front Rear
NX-OS 5.2(1) and later
Back
14RU 7RU
Side Side Side
Front N7K-C7004 Rear

Front N7K-C7009 Rear
Nexus 7700 Chassis Family
Nexus 7718

Back
Nexus 7710
Back
26RU Nexus 7706
Back
14RU
Front
9RU
Front
Front
Front Rear Front Rear Front Rear

N77-C7718 N77-C7710 N77-C7706
Cisco Nexus 7702
Launched here at Cisco Live! Milan
Deployment Flexibility
• Small to medium DCI solution
• Compact Service Chassis
• Comprehensive Layer 2 and Layer 3 feature set
Operation and Feature Consistency

• Supports all current and future Nexus 7700 Linecards, Sups and
Power supplies
• 3RU Form Factor based on N7700 architecture
• Same proven common architecture, ASICs and Cisco NX-OS
o One SUP and One N7700 I/O Module
o Two Power Supplies software
o No fabric Modules • Same software train across Nexus 7700 and 7000 Series,
o Front-to-Back Airflow ensuring consistency
• Up to 48 x 1/10GE or 24* x 40GE or 12 x
100GE non-blocking ports
* With breakout cables this linecard can offer up to 76p 10GE + 5p 40GE)
Key Chassis Components
• Common components: • Common components:
– Supervisor engines – Supervisor engines
– I/O modules – I/O modules
– Power supplies (except 7004) – Power supplies
• Chassis-specific components: • Chassis-specific components:
– Fabric modules – Fabric modules
– Fan trays – Fan trays
Common hardware components between Nexus 7000 and Nexus 7700:

NONE
No interchangeable hardware components between
Nexus 7000 and Nexus 7700
Agenda
• Conclusion
Supervisor Engine 2 / 2E
• Next generation supervisors providing control plane and management functions
Supervisor Engine 2 (Nexus 7000) Supervisor Engine 2E (Nexus 7000 / Nexus 7700)
Base performance High performance
One quad-core 2.1GHz CPU with 12GB DRAM Two quad-core 2.1GHz CPU with 32GB DRAM
• Connects to fabric via 1G inband interface

N77-SUP2E
• Interfaces with I/O modules via 1G switched EOBC
• Second-generation dedicated central arbiter ASIC
– Controls access to fabric bandwidth via dedicated arbitration path to I/O modules
N7K-SUP2/N7K-SUP2E
ID and Status USB Expansion

LEDs Flash
ID and Status Management USB Host USB Log USB Expansion Console Port Management
LEDs Console Port Ethernet Ports Flash Flash Ethernet
Nexus 7000 / 7700 I/O Module Families
M2 10G / 40G / 100G

M1 1G and 10G
F2E 10G F3 40G F3 100G

F1 10G F2 10G
F2E 10G
F3 10G / 40G / 100G
F3 closes the
F/M feature gap!
Nexus 7000 Module Evolution
FSA
VXLAN
F3(2013) Offload M2-XL(2012)
40G / 100G 240G/slot
1.2T/slot Segment-ID
MPLS
Large Tables
F2(2011) Sampled OTV Full
480G/slot
NetFlow NetFlow
Layer 2 Layer 3 EoMPLS/ Large
F1(2010) VPLS Buffers
230G/slot
FabricPath FEX
LISP
M1-XL(2008)
FCoE 80G/slot
Supported in NX-OS release 6.1(1) and later
Nexus 7000 M2 I/O Modules

N7K-M224XP-23L / N7K-M206FQ-23L / N7K-M202CF-22L
N7K-M224XP-23L
• 10G / 40G / 100G M2 I/O modules
• Share common hardware architecture
N7K-M206FQ-23L
• Two integrated forwarding engines (120Mpps)
– Support for “XL” forwarding tables (licensed)
• Distributed L3 multicast replication

• 802.1AE LinkSec on all ports
N7K-M202CF-22L
• Supports Nexus 2000 (FEX) connections
Module Port Density Optics Bandwidth
M2 10G 24 x 10G (plus Nexus 2000 FEX support) SFP+ 240G
M2 40G 6 x 40G (or up to 24 x 10G via breakout) QSFP+ 240G
M2 100G 2 x 100G CFP 200G
Nexus 7000 M2 I/O Module Architecture
N7K-M224XP-23L / N7K-M206FQ-23L / N7K-M202CF-22L
EOBC To Fabric Modules To Central Arbiters
LC Fabric 2 ASIC Arbitration

CPU Aggregator
…
VOQs VOQs Forwarding Forwarding VOQs VOQs

Engine Engine
Replication Replication
Engine Engine
Engine Engine
LinkSec + LinkSec +
12 X 10G MAC -or- 12 X 10G MAC -or-
3 X 40G MAC -or- 3 X 40G MAC -or-
1 X 100G MAC 1 X 100G MAC
Front Panel Ports
7000: Supported in NX-OS release 6.1(2) and later
7700: Supported in NX-OS release 6.2(2) and later
Nexus 7000 / 7700 F2E I/O Modules

N7K-F248XP-25E / N7K-F248XT-25E / N77-F248XP-23E
• 48-port 1G/10G with SFP/SFP+ transceivers N7K-F248XP-25E N7K-F248XT-25E

• 480G full-duplex fabric connectivity
• System-on-chip (SoC) forwarding engine design
– 12 independent SoC ASICs
• Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS)

• Interoperability with M1/M2, in Layer 2 mode on Nexus 7000 N77-F248XP-23E
– Proxy routing for inter-VLAN/L3 traffic
• LinkSec support
– Last 8 ports (SFP+)
– All 48 ports (Copper)
Nexus 7000 F2E Module Architecture
N7K-F248XP-25E / N7K-F248XT-25E
To Fabric Modules To Central Arbiters
EOBC
LC Arbitration
CPU Aggregator …
Fabric 2
4 X 10G 4 X 10G 4 X 10G 4 X 10G 4 X 10G 4 X 10G 4 X 10G 4 X 10G 4 X 10G 4 X 10G 4 X 10G 4 X 10G
SoC SoC SoC SoC SoC SoC SoC SoC SoC SoC SoC SoC
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48
Front Panel Ports LinkSec-capable (F2E fiber)
LinkSec-capable (F2E copper)
Nexus 7700 F2E Module Architecture
N77-F248XP-23E
To Fabric Modules To Fabric Modules To Central Arbiters
EOBC
LC Arbitration
CPU Aggregator …
Fabric 2 Fabric 2
SoC SoC SoC SoC SoC SoC SoC SoC SoC SoC SoC SoC
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48
Front Panel Ports LinkSec-capable
Nexus 7000 Series F3 Module Portfolio
Shipped
Oct ‘14
Nexus 7000 100G
Nexus 7700 100G
Shipped
Jan ‘14 F3 6-port 100G Module
N7K-F306CK-25
Nexus 7000 40G F3 12-port 100G Module
N77-F312CK-26
Nexus 7700 40G
New
Shipping
Jan ‘15
F3 12-port 40G Module
N7K-F312FQ-25
F3 24-port 40G Module
N77-F324FQ-25
Nexus 7000 10G Nexus 7700 10G
Shipped
Jan ‘14
F3 48-port 10G Module F3 48-port 10G Module
N77-F348XP-23
N7K-F348XP-25
F3 ASIC
The most comprehensive ASIC for the Data Center
• Classic Layer2 and Layer3
Innovation but also core functionalities
• FabricPath
Next-Gen Multi-Tenant Fabrics
• FCoE
Converged Network Infrastructure
F3 ASIC
• OTV, LISP and MPLS (L2 and L3 VPNs)

DCI and MPLS deployments
• VXLAN
Next Generation hardware Overlay
Note: Not all the features are enabled in the current NX-OS Software
Fabric Services Accelerator (FSA) on F3 modules
EOBC
• High-performance module CPU with on-board
acceleration engines
– 6Gbps inband connectivity from SOCs to FSA FSA CPU
– Multi-Mpps packet processing
– 2 X 2GB dedicated DRAM
• Performance/scale boost for distributed fabric Dual-Core Acceleration

services, including BFD and sampled LC CPU Engines
NetFlow*.
• Other potential applications include distributed
ARP/ping processing, data plane packet
2GB 2GB
analysis (wireshark), network probing, etc. DRAM DRAM
I/O
* In 7.2 release 6 x 1Gbps

Module Inband
Nexus 7000 F3 48-Port 1G/10G Module
New!
• 48-port 1G/10G with SFP/SFP+ transceivers Shipping
Jan’ 15
• SoC-based forwarding engine design
• Layer 2/Layer 3 forwarding with L3/L4 services

(ACL/QoS) and advanced features
• Fabric Services Accelerator (FSA) CPU
• LinkSec support (last 8 ports)
N7K-F348XP-23
Nexus 7000 F3 48-Port 1G/10G Module Architecture
FSA Arbitration
CPU Aggregator
x6
1G switch Fabric ASIC
x6
…
… x6
LC Inband
to FSA
to ARB CPU
8 X 10G 8 X 10G 8 X 10G 8 X 10G 8 X 10G 8 X 10G

SoC 1 SoC 2 SoC 3 SoC 4 SoC 5 SoC 6
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48
Front Panel Ports (SFP/SFP+) LinkSec-capable
Nexus 7000 F3 40G and 100G Modules
• 12-port 40G QSFP+ module/
6-port 100G CPAK module
• SoC forwarding engine design
• Layer 2/Layer 3 forwarding with L3/L4
services (ACL/QoS) and advanced N7K-F312FQ-25
features
• 40G breakout cable support
• Requires Supervisor Engine 2 / 2E
Nexus 7000 F3 12-Port 40G Module Architecture
FSA Arbitration
CPU Aggregator
x6
x6 …
1G switch
… x6
Fabric ASIC
LC Inband
to FSA
to ARB CPU
2 X 40G 2 X 40G 2 X 40G 2 X 40G 2 X 40G 2 X 40G

1 2 3 4 5 6 7 8 9 10 11 12
Front Panel Ports (QSFP+)
FSA Arbitration
CPU Aggregator
x6
x6 …
1G switch
… x6
Fabric ASIC
LC Inband
to FSA
to ARB CPU
1 X 100G 1 X 100G 1 X 100G 1 X 100G 1 X 100G 1 X 100G

1 2 3 4 5 6
Front Panel Ports (CPAK)
Nexus 7700 F3 48-Port 1G/10G Module
• 48-port 1G/10G with SFP/SFP+ transceivers
• SoC-based forwarding engine design

• LinkSec support (last 8 ports)
N77-F348XP-23
Nexus 7700 F3 48-Port 1G/10G Module Architecture
FSA Arbitration
CPU Aggregator
x6
1G switch
x6
…
… x6
Fabric ASIC Fabric ASIC
LC Inband
to FSA
to ARB CPU
8 X 10G 8 X 10G 8 X 10G 8 X 10G 8 X 10G 8 X 10G

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48
Front Panel Ports (SFP/SFP+) LinkSec-capable
Nexus 7700 F3 40G and 100G Modules
N77-F324FQ-25
• 24-port 40G QSFP+ module /
12-port 100G CPAK module
• 960G/1.2T full-duplex fabric connectivity
• SoC forwarding engine design
• 40G breakout cable support
N77-F312CK-26
FSA Arbitration
CPU Aggregator
x6
x 12 …
1G switch Fabric ASIC Fabric ASIC
… x 12
LC Inband
to FSA
to ARB CPU
SoC 1 SoC 2 SoC 3 SoC 4 SoC 5 SoC 6 SoC 7 SoC 8 SoC 9 SoC 10 SoC 11 SoC 12
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Front Panel Ports (QSFP+)
FSA Arbitration
CPU Aggregator
x6
x 12 …
1G switch Fabric ASIC Fabric ASIC
… x 12
LC Inband
to FSA
to ARB CPU
SoC 1 SoC 2 SoC 3 SoC 4 SoC 5 SoC 6 SoC 7 SoC 8 SoC 9 SoC 10 SoC 11 SoC 12
1 2 3 4 5 6 7 8 9 10 11 12
Front Panel Ports (CPAK)
Agenda
• Conclusion
Head of Line Blocking
What is the Problem?
Red Cars
blocked! No Congestion
BRKARC-3470 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 38 38
Head of Line Blocking
Solution – Create Separate Lanes but… Stadium
Full
Cars turned
back
Head of Line Blocking Stadium
Full
Solution – Have an arbiter
Supermarket
Not Full
Fabric, VOQ, and Arbitration
• Crossbar fabric = Highway
Provides dedicated, high-bandwidth interconnects between ingress and egress
I/O modules
• Virtual Output Queues (VOQs) = Separate Lanes for each Destination
Provide buffering and queuing for ingress-buffered switch architecture
• Central arbitration = Traffic police
Controls scheduling of traffic into fabric based on fairness, priority, and bandwidth
availability at egress ports
• Fabric, VOQ, and arbitration combine to provide all necessary infrastructure for
packet transport inside switch avoiding HOLB
Crossbar Switch Fabric Modules
N77-C7718-FAB-2
• Provide interconnection of I/O modules N77-C7710-FAB-2
N77-C7706-FAB-2
• Each installed fabric increases available per-payload slot bandwidth

• Nexus 7000 and Nexus 7700 fabrics based on Fabric 2 ASIC
Per-fabric module Max fabric Total bandwidth per
Fabric Module Supported Chassis
bandwidth modules slot
Nexus 7000 Fabric 2 7009 / 7010 / 7018 110Gbps per slot 5 550Gbps per slot
Nexus 7700 Fabric 2 7706 / 7710 / 7718 220Gbps per slot 6 1.32Tbps per slot
• Different I/O modules leverage different amount of available fabric bandwidth

• Access to fabric bandwidth controlled using QoS-aware central arbitration with
VOQ
N7K-C7018-FAB-2
N7K-C7010-FAB-2
N7K-C7009-FAB-2
Multistage Crossbar
Nexus 7000 / Nexus 7700 implement 3-stage crossbar switch fabric
• Stages 1 and 3 on I/O modules
• Stage 2 on fabric modules 2nd stage Fabric Modules
Fabric Modules
Fabric Fabric Fabric Fabric Fabric Fabric
1 Fabric 2 Fabric 3 Fabric 4 Fabric 5 Fabric ASIC ASIC ASIC ASIC ASIC ASIC
Fabric Fabric Fabric Fabric Fabric Fabric
ASIC ASIC ASIC ASIC ASIC ASIC ASIC ASIC ASIC ASIC ASIC
1 2 3 4 5 6
550G 1.32T
110G
110G (2 x 55G)
(2 x 55G)
Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC
3rd stage
1st stage Ingress Egress Ingress Module Egress Module
1st stage
Module Module
3rd stage
I/O Module Capacity – Nexus 7000
Fabric 2 Modules
550Gbps
110Gbps
440Gbps
220Gbps
330Gbps Fabric 2
1
per slot bandwidth ASIC
One fabric: Local Fabric 2
(240G)
• Any port can pass traffic to any
2
other port in VDC Fabric 2
ASIC
Three fabrics:
• 240G M2 module has maximum Local Fabric 2 Fabric 2
3
bandwidth (480G) ASIC
Five fabrics:
4
• 480G F2E/F3 module has maximum Fabric 2
ASIC
bandwidth Local Fabric 2
(550G)
• 600G F3 module is slightly 5
Fabric 2
oversubscribed for fabric switched ASIC
traffic
What About Nexus 7004?
• Nexus 7004 has no fabric modules
• I/O modules have local fabric with 10 available fabric channels
– I/O modules connect “back-to-back” via 8 fabric channels
– Two fabric channels “borrowed” to connect supervisor engines
Sup Slot 1 Fabric Fabric Sup Slot 2

ASIC ASIC
2 * 55G
fabric channels
M2/F2E/F3 M2/F2E/F3
Module 3 Fabric 2 Fabric 2 Module 4
ASIC ASIC
8 * 55G local fabric channels

interconnect I/O modules (440G)
I/O Module Capacity – Nexus 7700
Fabric 2 Modules
1320Gbps
1100Gbps
880Gbps
660Gbps
440Gbps
220Gbps Local Fab2
Fab2 Fabric 2
ASICs
1
per slot bandwidth #2

#1 (480G)
One fabric:
2
• Any port can pass traffic to any other port Fabric 2
ASICs
in VDC
Three fabrics: Fab2
Local Fab2 3
#2
#1 (960G) Fabric 2
• 480G F2E/F3 10G module has maximum ASICs
bandwidth
4
Five fabrics: Fabric 2
ASICs
Fab2
• 960G F3 40G module has maximum Local Fab2
#2
bandwidth #1 (1.2T) 5
Fabric 2
ASICs
Six fabrics:
• 1.2T F3 100G module has maximum 6
Fabric 2
bandwidth ASICs
Agenda
• Conclusion
M-Series Forwarding Engine Hardware
• Two hardware forwarding engines integrated on every • MPLS/VPLS/EoMPLS
M2 I/O module
• OTV
• 120Mpps (60Mpps per forwarding engine) Layer 2
bridging with hardware MAC learning • RACL/VACL/PACL
• 120 Mpps (60Mpps per forwarding engine) Layer 3 • QoS remarking and policing policies
IPv4
• Policy-based routing (PBR)
• 60Mpps (30Mpps per forwarding engine) Layer 3 IPv6
unicast • Unicast RPF check and IP source guard
• Layer 3 IPv4 and IPv6 multicast support (SM, SSM, • IGMP snooping
Bidir)
• Ingress and egress NetFlow (full and sampled)
Hardware Table M-Series Modules M-Series Modules with

without Scale License Scale License
MAC Address Table 128K 128K
FIB TCAM 128K IPv4 / 64K IPv6 900K IPv4 / 350K IPv6
Classification TCAM (ACL/QoS) 64K 128K
NetFlow Table 1M 1M
M-Series Forwarding Engine Architecture
FE Daughter Card
L3 Engine
FIB TCAM/
Layer 3 FIB
ADJ
NetFlow
Policing
Classification
CL TCAM
(ACL/QoS)
L2 Engine
MAC L2 Lookup (post-L3)
Table L2 Lookup (pre-L3)
Ingress Parser Final Results
From I/O Module To I/O Module

BRKARC-3470
Replication Engines
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Replication Engines 49
FE Daughter Card
L3 Engine
FIB TCAM/
Layer 3 FIB
ADJ
NetFlow
Policing
Classification
CL TCAM
(ACL/QoS)
L2 Engine
PKT
Table L2 LookupHDR
(pre-L3)
 Ingress MAC table lookups Ingress Parser Final Results

 Port-channel hash result
 Ingress IGMP snooping
lookups
BRKARC-3470
Replication Engines
Cisco Public
FE Daughter Card
L3 Engine
FIB TCAM/
Layer 3 FIB
ADJ
NetFlow
 Ingress ACL/QoS classification Policing
Classification
PKT
CL TCAM HDR
(ACL/QoS)
Ingress lookup pipeline
L2 Engine

lookups
BRKARC-3470
Replication Engines
Cisco Public
FE Daughter Card
L3 Engine
FIB TCAM/
Layer 3 FIB
 Ingress NetFlow collection ADJ
PKT
NetFlow
HDR
 Ingress ACL/QoS classification Policing
Classification
CL TCAM
(ACL/QoS)
L2 Engine

lookups
BRKARC-3470
Replication Engines
Cisco Public
FE Daughter Card
L3 Engine
FIB TCAM/
Layer PKT
3 FIB
 Ingress NetFlow collection HDR ADJ
NetFlow
 FIB TCAM and adjacency table
lookups for Layer 3 forwarding
 Ingress ACL/QoS classification Policing  ECMP hashing
 Multicast RPF check
Classification
CL TCAM
(ACL/QoS)
L2 Engine

lookups
BRKARC-3470
Replication Engines
Cisco Public
FE Daughter Card
L3 Engine
FIB TCAM/
Layer 3 FIB
NetFlow
 Ingress ACL/QoS classification PKT
Policing  ECMP hashing
HDR
Classification
CL TCAM
(ACL/QoS)  Ingress policing
L2 Engine

lookups
BRKARC-3470
Replication Engines
Cisco Public
FE Daughter Card
L3 Engine
FIB TCAM/
Layer 3 FIB
NetFlow
Classification PKT Egress lookup
CL TCAM
(ACL/QoS) HDR pipeline  Ingress policing
L2 Engine

lookups
BRKARC-3470
Replication Engines
Cisco Public
FE Daughter Card
L3 Engine
 Egress NetFlow collection
FIB TCAM/
Layer 3 FIB
 Egress ACL/QoS classification NetFlow

Classification Egress lookup
CL TCAM
(ACL/QoS) pipeline  Ingress policing
Ingress lookup pipeline  Egress policing
L2 Engine
PKT
MAC L2 LookupHDR
(post-L3)
Table  Egress MAC lookups
L2 Lookup (pre-L3)
 Egress IGMP snooping
lookups
lookups
BRKARC-3470
Replication Engines
Cisco Public
FE Daughter Card
L3 Engine
 Egress NetFlow collection
FIB TCAM/
Layer 3 FIB
 Egress ACL/QoS classification NetFlow

Classification Egress lookup
CL TCAM
(ACL/QoS) pipeline  Ingress policing
Ingress lookup pipeline  Egress policing
L2 Engine
Table  Egress MAC lookups
L2 Lookup (pre-L3)
 Egress IGMP snooping
lookups
lookups
From I/O Module To
PKT I/O Module
BRKARC-3470
Replication Engines
Cisco Public
Replication
HDR Engines 57
F2E Forwarding Engine Hardware
• Each SoC forwarding engine services 4 front-panel • QoS remarking and policing policies
10G ports (12 SoCs per module)
• Policy-based routing (PBR)
• 60Mpps per SoC Layer 2 bridging with hardware MAC
learning • Unicast RPF check and IP source guard
• 60Mpps per forwarding engine Layer 3 IPv4/ IPv6 • IGMP snooping

unicast
• Ingress sampled NetFlow
• Layer 3 IPv4 and IPv6 multicast support (SM, SSM,
Bidir*) • FabricPath forwarding
• RACL/VACL/PACL • FCoE (with Sup2 / Sup2E)

– Roadmap on Nexus 7700
Hardware Table Per F2E SoC Per F2E Module

MAC Address Table 16K 192K*
FIB TCAM 32K IPv4/16K IPv6 32K IPv4/16K IPv6
Classification TCAM (ACL/QoS) 16K 192K*
* Assumes specific configuration to scale SoC resources
* Roadmap item
F3 Forwarding Engine Hardware
• Each SoC forwarding engine services: • Policy-based routing (PBR)
– 8 front-panel 10G ports
• Unicast RPF check and IP source guard
– 2 front-panel 40G ports
– 1 front-panel 100G port • IGMP snooping
• 148Mpps per SoC Layer 2 bridging with hardware • Ingress/egress* sampled NetFlow
MAC learning
• FabricPath forwarding
• 148Mpps per forwarding engine Layer 3 IPv4/ IPv6
unicast • GRE Tunnels
• Layer 3 IPv4 and IPv6 multicast support (SM, SSM, • Overlay Transport Virtualization (OTV)
Bidir*)
• MPLS/VPLS/EoMPLS, LISP, VXLAN, FCoE*
• QOS remarking and policing
• RACL/PACL/VACL Hardware Table Per F3 SoC Per F3 Module

MAC Address Table 64K 384K/768K**
FIB TCAM 64K IPv4/32K IPv6 64K IPv4/32K IPv6
Classification TCAM (ACL/QoS) 16K 96K/192K**
** Assumes specific configuration to scale SoC resources
* Roadmap items
F3 Forwarding Engine
To/From Central
Arbiter
To Fabric From Fabric
F3 SoC
Decision Engine
Virtual
Queuing
L2 Lookup (post-L3)
Ingress MAC FIB/ADJ Layer 3 Lookups Egress

Buffer Table CL QoS / ACL Buffer
L2 Lookup (pre-L3)
Ingress Parser Egress Parser
1G / 10G / 40G / 100G
Front-panel
To/From Central
Arbiter
F3 SoC
Decision Engine
Virtual
Queuing
L2 Lookup (post-L3)

L2 Lookup (pre-L3)
PKT
Ingress HDR
Parser Egress Parser
1G / 10G / 40G / 100G

capable interface MAC
1G / 10G / 40G / 100G
Front-panel
8 x 1/10G OR
2 x 40G OR
1 x 100G per ASIC
To/From Central
Arbiter
F3 SoC
Decision Engine
Virtual
Queuing PKT
L2 Lookup (post-L3)

Egress fabric
receive buffer
Ingress buffer
memory L2 Lookup (pre-L3)
Ingress HDR
Parser Egress Parser
1G / 10G / 40G / 100G

1G / 10G / 40G / 100G
Front-panel
8 x 1/10G OR
2 x 40G OR
1 x 100G per ASIC
To/From Central Ingress and egress
Arbiter Forwarding forwarding decisions
To Fabric tables From Fabric
(L2/L3 lookups,
ACL/QoS, features etc.)
F3 SoC
Decision Engine
Virtual
Queuing PKT
L2 Lookup (post-L3)

Egress fabric
receive buffer
Ingress buffer
memory HDR
L2 Lookup (pre-L3)
1G / 10G / 40G / 100G

1G / 10G / 40G / 100G
Front-panel
8 x 1/10G OR
2 x 40G OR
1 x 100G per ASIC
(L2/L3 lookups,
F3 SoC
Decision Engine
Virtual
Queuing PKT
L2 Lookup (post-L3)

HDR Egress fabric
receive buffer
Ingress buffer
1G / 10G / 40G / 100G

1G / 10G / 40G / 100G
Front-panel
8 x 1/10G OR
2 x 40G OR
1 x 100G per ASIC
(L2/L3 lookups,
F3 SoC
Decision Engine
Virtual
Queuing PKT
HDR
L2 Lookup (post-L3)

Egress fabric
receive buffer
Ingress buffer
1G / 10G / 40G / 100G

1G / 10G / 40G / 100G
Front-panel
8 x 1/10G OR
2 x 40G OR
1 x 100G per ASIC
(L2/L3 lookups,
F3 SoC
Decision Engine
VOQ Virtual
Queuing PKT HDR
HDR
L2 Lookup (post-L3)

Egress fabric
receive buffer
Ingress buffer
1G / 10G / 40G / 100G

1G / 10G / 40G / 100G
Front-panel
8 x 1/10G OR
2 x 40G OR
1 x 100G per ASIC
To Fabric
PKT HDR tables From Fabric
(L2/L3 lookups,
F3 SoC
Decision Engine
VOQ Virtual
Queuing
L2 Lookup (post-L3)

Egress fabric
receive buffer
Ingress buffer
1G / 10G / 40G / 100G

1G / 10G / 40G / 100G
Front-panel
8 x 1/10G OR
2 x 40G OR
1 x 100G per ASIC
Agenda
• Conclusion
Buffering, Queuing, and Scheduling
• Buffering – storing packets in memory
– Needed to absorb bursts, manage congestion
• Queuing – buffering packets according to traffic class

– Provides dedicated buffer for packets of different priority
• Scheduling – controlling the order of transmission of buffered packets

– Ensures preferential treatment for packets of higher priority and fair treatment for packets of equal
priority
• Nexus 7000 / Nexus 7700 use queuing policies and network-QoS policies to define
buffering, queuing, and scheduling behavior
• Default queuing and network-QoS policies always in effect in absence of any user
configuration
BRKDCT-3346
End-to-End QoS Implementation and Operation with Cisco Nexus
Thursday 1/29 9:00am to 11:00am
I/O Module Buffering Models
• Buffering model varies by I/O module family
– M-series modules: hybrid model combining ingress VOQ-buffered architecture with

egress port-buffered architecture
– F-series modules: pure ingress VOQ-buffered architecture
M2 – Hybrid Ingress/Egress Buffered Supervisor Engine
10G M2 module used as example
Central Arbiter
Fabric Module 1 Fabric Module 2 Fabric Module 3

Fabric ASIC Fabric ASIC Fabric ASIC
SP SP Local Local Local Local Local Local

q1 q1 Ports Ports Ports VQIs VQIs VQIs
1/2 - 1/12 1/13 - 1/23 1/14 - 1/24 2/2 - 2/12 2/13 - 2/23 2/14 - 2/24
q2 q2 Even Odd Even SP DWRR SP DWRR Even Odd Even
e2/1… q3 q3
e2/1 – e2/11 Odd
VOQ 0 e1/1 - e1/11 Odd VOQ 1 VOQ 2 VOQ 3 VOQ 0 VOQ 1 VOQ 2 VOQ 3
Replication Engine 0 RE 1 RE 2 RE 3
DWRR DWRR DWRR

Local
k k k Local
Module 2
Module 1
Ports Ports
1/13 - 1/24 SP SP SP 2/13 - 2/24
DWRR DWRR DWRR
Port ASIC 0 Port ASIC 1 Port ASIC 0 Port ASIC 1
e1/1 … e1/12
e1/13-24
e2/1 … e2/12
BRKARC-3470 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public e2/13-24 71
Central Arbiter


q1 q1 Ports
Buffering / queuing / Ports
scheduling Ports VQIs VQIs VQIs
1/2 - 1/12 1/13 - 1/23 1/14 - 1/24 2/2 - 2/12 2/13 - 2/23 2/14 - 2/24
e2/1… port buffer – Manages congestion of
q3 Ingressq3
ingress forwarding/replication engines, and e2/1 – e2/11 Odd
VOQ 0 e1/1 - congestion
e1/11 Odd toward egress VOQdestinations
1 VOQ(VQIs)
2 VOQ 3 VOQ 0 VOQ 1 VOQ 2 VOQ 3
8 ingress
queues
INGRESS QUEUING POLICIES Replication Engine 0
per port RE 1 RE 2 RE 3
DWRR DWRR DWRR

Local
k k k Local
Module 2
Module 1
Ports Ports
1/13 - 1/24 SP SP SP 2/13 - 2/24
DWRR DWRR DWRR
e1/1 … e1/12
e1/13-24
e2/1 … e2/12
Central Arbiter

FabricVOQ
ASIC
buffer Fabric ASIC
carved by 4 priority
source and levels
priority
1/2 - 1/12 1/13 - 1/23 1/14 - 1/24 2/2 - 2/12 2/13 - 2/23 2/14 - 2/24
e2/1… q3 q3
e2/1 – e2/11 Odd
VOQ 0 e1/1 - e1/11 Odd VOQ 1 VOQ 2 VOQ 3 VOQ 0 VOQ 1 VOQ 2 VOQ 3
Buffering / queuing Replication Scheduling

Engine 0 RE 1 RE 2 RE 3
Ingress VOQ buffer – Buffers traffic Egress VOQ buffer – Schedules
and manages congestion toward traffic toward egress destinations
egress destinations (VQIs) (VQIs) and receives frames from fabric
DWRR DWRR DWRR
Local
kFABRIC-QOS
k POLICY k Local
Module 2
Module 1
Ports Ports
FABRIC-QOS POLICY 1/13 - 1/24 2/13 - 2/24
SP SP SP
DWRR DWRR DWRR
e1/1 … e1/12
e1/13-24
e2/1 … e2/12
Central Arbiter


1/2 - 1/12 1/13 - 1/23 1/14 - 1/24 2/2 - 2/12 2/13 - 2/23 2/14 - 2/24
q2 q2 Even Odd Even SP DWRR SP DWRR Buffering /Even
queuing / scheduling
Odd Even
e2/1… q3 q3
e2/1 – e2/11 Odd
Egress port buffer – Manages congestion at
e1/1 - e1/11 Odd VOQ 0 VOQ 1 VOQ 2 VOQ 3
VOQ 0 VOQ 1 VOQ 2 VOQ 3 egress physical interface
8 egress
queues
Replication Engine 0
EGRESS QUEUING POLICIES
per port RE 1 RE 2 RE 3
DWRR DWRR DWRR

Local
k k k Local
Module 2
Module 1
Ports Ports
1/13 - 1/24 SP SP SP 2/13 - 2/24
DWRR DWRR DWRR
e1/1 … e1/12
e1/13-24
e2/1 … e2/12
Nexus 7700 10G F3 module used as example
F2E/F3 – Ingress Buffered

Supervisor Engine
Central Arbiter
Module 1 Module 2
SP SP
Local Local q1 q1 Local Local Local Local
Ports Ports Ports Ports Ports VQIs
1/1 - 1/8 1/9 - 1/16
e2/1… q2 q2 1/25 - 1/32 1/33 - 1/40 1/41 - 1/48 SP DWRR SP DWRR 2/9 – 2/16
q3 q3 …
Virtual Queuing
e1/17 … e1/24 e2/1 e2/8
Egress Buffer
SOC 1
SOC 1
SOC 1
Ingress Buffer SOC 1
SOC 0 SOC 1 SOC 2 SOC 3 SOC 4 SOC 5 SOC 0 SOC 1
e1/17 … e1/24 e2/1 … e2/8

e1/1-8 e1/9-16 e1/25-32 e1/33-40 e1/41-48 e2/9-48
Nexus 7700 10G F3 module used as example
F2E/F3 – Ingress Buffered

Supervisor Engine
Central Arbiter

Scheduling
Buffering / queuing Fabric ASIC Fabric ASIC Fabric ASIC
Egress VOQ buffer – Schedules
Ingress VOQ buffer – Buffers traffic traffic toward egress destinations
and manages congestion toward (VQIs) and receives frames from fabric
Module 1 egress destinations (VQIs) Module 2
Fabric ASIC Fabric ASIC EGRESS QUEUING POLICY
INGRESS QUEUING POLICY
4 ingress queues 8 priority
per port levels
SP SP
Local Local q1 q1 Local Local Local Local
Ports Ports Ports Ports Ports VQIs
1/1 - 1/8 1/9 - 1/16
e2/1… q2 q2 1/25 - 1/32 1/33 - 1/40 1/41 - 1/48 SP DWRR SP DWRR 2/9 – 2/16
q3 q3 …
Virtual Queuing
e1/17 … e1/24 e2/1 e2/8
Egress Buffer
SOC 1
SOC 1
SOC 1
Ingress Buffer SOC 1
SOC 0 SOC 1 SOC 2 SOC 3 SOC 4 SOC 5 SOC 0 SOC 1
e1/17 … e1/24 e2/1 … e2/8

e1/1-8 e1/9-16 e1/25-32 e1/33-40 e1/41-48 e2/9-48
FAQ: What Is a VQI?
• VQI = Virtual Queuing Index
• “A Destination Across the Fabric”
• For M2 / F2E / F3 10G modules, VQI == 10G interface
• For F3 40/100G modules, VQI == 40/100G interface
• For M2 40/100G ports, uses multiple 10G VQIs
M2 Module 40G and 100G Flow Limits
Internal to Nexus 7000 System On the Wire (40G)
40G Port
Ingress Modules
Tx 1 5 1
Destination
1 packet
64/66B Encoding
Spines VQIs
Spines Tx 2 6 2
Spines
Spines
Fabrics n … 4 3 2 1
Tx 3 … 3
64 bits
Tx 4 4
10G 10G 40G 40G 100G
1 VQI 1 VQI 4 VQIs 4 VQIs 10 VQIs
66 bits
Egress Interfaces
• Each Virtual Queuing Index (VQI) sustains 10G • Packets split into 66-bit “code words”
traffic flow
• Four code words transmitted in parallel, one on each physical
• All packets in given 5-tuple flow hash to single VQI Tx fiber
• BRKARC-3470
Single-flow limit is 10G
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public
• No per-flow limit imposed – splitting occurs at physical layer 78
F3 Module 40G and 100G Flow Limits
Internal to Nexus 7000 / 7700 System
Ingress Modules
Destination
Spines VQIs
Spines
Spines
Spines
Fabrics
10G 10G 40G 40G 100G

1 VQI 1 VQI 1 VQI 1 VQI 1 VQI
Egress Interfaces
• Virtual Queuing Index (VQI) sustains 10G, 40G, or 100G traffic flow based on destination interface type
• No single-flow limit – full 40G/100G flow support
Agenda
• Conclusion
Hardware Layer 2 Forwarding Process
Layer 2 forwarding – traffic steering based on destination MAC address
• MAC table lookup drives Layer 2 forwarding
• Source MAC and destination MAC lookups performed for each frame, based on
{VLAN,MAC} pairs
• Source MAC lookup drives new learns and refreshes aging timers
• Destination MAC lookup dictates outgoing switchport
HDR = Packet Headers DATA = Packet Data CTRL = Internal Signaling
M2 L2 Packet Flow Supervisor Engine
Central Arbiter
Module 1 Module 2
Fabric 2 ASIC Fabric 2 ASIC
Forwarding Forwarding
Engine Engine
Layer 3 VOQs Layer 3 VOQs

Engine VOQs Engine VOQs
Layer 2 Engine Layer 2 Engine
Engine Replication Engine Replication
Engine Engine
10G/40G/100G MAC / LinkSec 10G/40G/100G MAC / LinkSec
e1/1 e2/2
Central Arbiter
Module 1 Module 2
Engine Engine

Engine Engine
10G/40G/100G MAC / LinkSec 10G/40G/100G MAC / LinkSec
Receive
packet from e1/1 e2/2
wire
Central Arbiter
Module 1 Module 2
Engine Engine

Engine Engine
10G/40G/100G MAC / LinkSec  LinkSec decryption 10G/40G/100G MAC / LinkSec

 Ingress port QoS
Receive
wire
Central Arbiter
Module 1 Module 2
Engine Engine

 Static or hash- Engine Engine
based RE uplink
selection
Receive
wire
Central Arbiter
Module 1 Module 2
Engine Engine

 Submit packet
Replication headers for Replication
Layer 2 Engine lookup Layer 2 Engine
based RE uplink
selection
Receive
wire
Central Arbiter
Module 1 Module 2
Engine Engine
 L2 SMAC/ DMAC
lookups Layer 3 VOQs Layer 3 VOQs
 Port-channel hash
result Engine VOQs Engine VOQs
 Submit packet
based RE uplink
selection
Receive
wire
Central Arbiter
Module 1 Module 2
 ACL/QoS/
NetFlow
lookups
Engine Engine
 L2 SMAC/ DMAC
 Submit packet
based RE uplink
selection
Receive
wire
Central Arbiter
Module 1 Module 2
 ACL/QoS/
NetFlow
lookups
Engine Engine
 L2 SMAC/ DMAC
 Submit packet
 Return result – Replication Replication
headers for
destination +
hash result Layer 2 Engine lookup Layer 2 Engine
based RE uplink
selection
Receive
wire
Central Arbiter
Module 1 Module 2
 ACL/QoS/
NetFlow
lookups
Engine Engine
 L2 SMAC/ DMAC
lookups Layer 3 VOQs  Hash-based uplink Layer 3 VOQs
 Port-channel hash and VQI selection
 Submit packet
headers for
destination +
based RE uplink
selection
Receive
wire
M2 L2 Packet Flow  Credit grant for

fabric access
Supervisor Engine
Central Arbiter

 VOQ arbitration
and queuing
Module 1 Module 2
 ACL/QoS/
NetFlow
lookups
Engine Engine
 L2 SMAC/ DMAC
 Submit packet
headers for
destination +
based RE uplink
selection

Receive
wire

fabric access
Supervisor Engine  Return
credit
to pool
Central Arbiter

 Receive from
Fabric ASIC Fabric ASIC Fabric ASIC fabric
 VOQ arbitration  Return buffer
and queuing credit
Module 1 Module 2
 ACL/QoS/  Round-robin
NetFlow
lookups
Fabric 2 ASIC transmit to VQI Fabric 2 ASIC
 Round-robin
transmit to fabric
Engine Engine
 L2 SMAC/ DMAC
 Submit packet
headers for
destination +
based RE uplink
selection

Receive
wire

fabric access
credit
to pool
Central Arbiter

 Receive from
and queuing credit
Module 1 Module 2
NetFlow
lookups
 Round-robin
transmit to fabric
Engine Engine
 L2 SMAC/ DMAC
 Submit packet
headers for
destination +
Layer 2 Engine lookup Layer 2 Engine  Static
hash result
downlink
Engine Replication Engine Replication selection
based RE uplink
selection  Egress
10G/40G/100G MAC / LinkSec  LinkSec decryption 10G/40G/100G MAC / LinkSec  LinkSec
port QoS
encryption

Receive
wire

fabric access
credit
to pool
Central Arbiter

 Receive from
and queuing credit
Module 1 Module 2
NetFlow
lookups
 Round-robin
transmit to fabric
Engine Engine
 L2 SMAC/ DMAC
 Submit packet
headers for
destination +
Layer 2 Engine lookup Layer 2 Engine  Static
hash result
downlink
based RE uplink
selection  Egress
port QoS
encryption

Receive  Transmit
packet from e1/1 packet on e2/2
wire wire
F2E / F3 L2 Packet Flow

Supervisor Engine
Central Arbiter
Fabric Module 1 Fabric Module 2 Fabric Module 3 Fabric Module 4 Fabric Module 5
Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC
VOQ DE VOQ
SoC Module 1 SoC Module 2
e1/1 e2/2

Supervisor Engine
Central Arbiter
VOQ DE VOQ

 Ingress
port QoS  Receive
(VOQ) e1/1 packet e2/2
from wire

Supervisor Engine
Central Arbiter
 Submit packet headers for lookup

 Ingress L2 SMAC/ DMAC

lookups, ACL/QoS lookups,
VOQ DE NetFlow sampling VOQ
 Return result –
destination
 Ingress
from wire

 Credit grant for
Supervisor Engine
fabric access
Central Arbiter

 VOQ arbitration Fabric ASIC Fabric ASIC

destination
 Ingress
from wire

Supervisor Engine
fabric access
Central Arbiter
 Transmit
to fabric
 Receive from fabric


destination
 Ingress
from wire

credit
fabric access to pool
Central Arbiter
 Transmit
to fabric
 Receive from fabric


destination
 Ingress  Egress port QoS
port QoS  Receive  Transmit (Scheduling)
(VOQ) e1/1 packet packet on e2/2  Return buffer credit
from wire wire
Agenda
• Conclusion
Layer 3 Forwarding
• Nexus 7000 decouples control plane and data plane
• Forwarding tables built on control plane using routing protocols or static
configuration
–OSPF, EIGRP, IS-IS, RIP, BGP for dynamic routing
• Tables downloaded to forwarding engine hardware for data plane forwarding
–FIB TCAM contains IP prefixes
–Adjacency table contains next-hop information
Hardware Layer 3 Forwarding Process
• FIB TCAM lookup based on longest-match destination prefix comparison
• FIB “hit” returns adjacency, adjacency contains rewrite information (next-hop)
• Pipelined forwarding engine architecture also performs ACL, QoS, and NetFlow
lookups, affecting final forwarding result
IP FIB TCAM Lookup
Generate TCAM lookup key
(destination IP address)
Ingress
Generate
unicast IP
Lookup Key
packet header
10.1.1.10 Compare
lookup key
Flow Forwarding Engine
Data
10.1.1.2 Index, # next-hops Next-hop 1 (IF, MAC)
Load-Sharing
10.1.1.3 Index, # next-hops Hash Next-hop 2 (IF, MAC)
10.1.1.4 Index, # next-hops
10.10.0.100 Index, # next-hops Next-hop 3 (IF, MAC)

Offset
mod
Return lookup
10.1.1.xx
10.1.2.xx Index, # next-hops result
10.1.3.xx Index, # next-hops Next-hop 4 (IF, MAC)

# next-
10.10.100.xx Index, # next-hops
hops
Next-hop 5 (IF, MAC)
HIT! 10.1.1.xx Index, # next-hops Next-hop 6 (IF, MAC)

Adj Index Result
10.100.1.xx Index, # next-hops Next-hop 7 (IF, MAC)
10.10.0.xx Index, # next-hops

Hit in FIB
Modulo function
10.100.1.xx Index,
returns result # next-hops Adjacency
selects exact
index identifies
in FIB DRAM
ADJ block to next hop entry
use to use
FIB TCAM FIB DRAM Adjacency Table
Central Arbiter
Module 1 Module 2
Engine Engine

 Static or Hash-based
Engine Engine
uplink selection


Receive
wire
Central Arbiter
 L3 FIB/ADJ lookup Module 1 Module 2

 Ingress and egress
ACL/QoS/NetFlow
lookups Fabric 2 ASIC Fabric 2 ASIC
Engine Engine
 L2 ingress and egress
SMAC/ DMAC lookups Layer 3 VOQs Layer 3 VOQs
 Submit packet
 Return result – Replication headers for Replication
destination + Layer 2 Engine lookup Layer 2 Engine
hash result
Engine Engine
uplink selection


Receive
wire

fabric access
credit
to pool
Central Arbiter

 Receive from
and queuing credit
 L3 FIB/ADJ lookup Module 1 Module 2

 Ingress and egress
 Round-robin
ACL/QoS/NetFlow
lookups Fabric 2 ASIC transmit to VOQ Fabric 2 ASIC
 Round-robin
transmit to fabric
Engine Engine
 L2 ingress and egress
SMAC/ DMAC lookups Layer 3 VOQs  Hash-based uplink Layer 3 VOQs
 Port-channel hash result (and VQI) selection
 Submit packet
 Return result – Replication headers for Replication
destination + Layer 2 Engine lookup Layer 2 Engine  Static RE
hash result downlink
Engine Engine
uplink selection
 Egress
port QoS
encryption

Receive  Transmit
packet from e1/1 packet on e2/2
wire wire
Layer 3 Forwarding – Module Interoperability Models
Two interoperability models for L3 forwarding:
• “Proxy Forwarding”
• “Ingress Forwarding” with Lowest Common Denominator
Proxy Forwarding Model – Conceptual
• From F2E perspective, Router MAC reachable through giant port-channel
• All packets destined to Router MAC forwarded through fabric toward one
“member port” in that channel
10.1.10.100 All F2E modules Up to 128 “links”

interface vlan 10
vlan 10 ip address 10.1.10.1/24
All M2 modules !
interface vlan 20
ip address 10.1.20.1/24
10.1.20.100
vlan 20
Proxy Forwarding Model – Actual
M2 Replication e3/20
Engine e3/19
VOQs
Replication e3/14
Engine e3/13
FE
Fabric
FE FE
Fabric Fabric Replication e3/8
SoC Engine e3/7
e1/1 VOQs
10.1.10.100 Replication e3/2
vlan 10 Mod 1
F2E … Mod 3
Engine e3/1
Fabric M2 Replication e4/20

FE
Fabric Engine e4/19
SoC VOQs
e2/1 Fabric Replication e4/14
Modules Engine e4/13 interface vlan 10
10.1.20.100 F2E FE
ip address 10.1.10.1/24
vlan 20 Mod 2 Fabric !
FE interface vlan 20
Replication e4/8 ip address 10.1.20.1/24
Engine e4/7
VOQs
Replication e4/2
Engine e4/1
Mod 4
Proxy Forwarding Model – Actual
VLAN DMAC Dest Port Programming of all
10 router_mac → internal_channel (e3/1-24,e4/1-24) F2E forwarding engines
EtherChannel Hash Function
hash_input (from packet) → select_member_port
Engine e3/19
VOQs
Replication e3/14
Engine e3/13
FE
Fabric
FE FE
SoC Engine e3/7
e1/1 VOQs
vlan 10 Mod 1
F2E … Mod 3
Engine e3/1

FE
Fabric Engine e4/19
SoC VOQs
10.1.20.100 F2E FE
ip address 10.1.10.1/24
Engine e4/7
VOQs
Replication e4/2
Engine e4/1
Mod 4
Proxy Forwarding Model – Actual Can be up to 128
M2 VQIs

Engine e3/19
VOQs
Replication e3/14
Engine e3/13
FE
Fabric
FE FE
SoC Engine e3/7
e1/1 VOQs
vlan 10 Mod 1
F2E … Mod 3
Engine e3/1

FE
Fabric Engine e4/19
SoC VOQs
10.1.20.100 F2E FE
ip address 10.1.10.1/24
Engine e4/7
VOQs
Replication e4/2
Engine e4/1
Mod 4
M2 VQIs

Engine e3/19
VOQs
Replication e3/14
Engine e3/13
FE
Fabric
FE FE
SoC Engine e3/7
e1/1 VOQs
vlan 10 Mod 1
F2E … Mod 3
Engine e3/1

FE
Fabric Engine e4/19
SoC VOQs
10.1.20.100 F2E FE
ip address 10.1.10.1/24
Engine e4/7
VOQs
Replication e4/2
Engine e4/1
Mod 4
M2 VQIs

EtherChannel Hash Function Ingress MAC:
M2 Replication e3/20 VLAN DMAC Dest Port
Engine
10 router_mac → L3_lookup
e3/19
VOQs Routing:
Replication e3/14 DIP Next Hop
FE
Engine e3/13 10.1.20.100 → server_2_mac (v20)
Fabric Egress MAC:

FE VLAN DMAC Dest Port
FE
Fabric Fabric Replication e3/8 20 server_2_mac → e2/1
e1/1 SoC VOQs

Engine e3/7
vlan 10 Mod 1
F2E … Mod 3
Engine e3/1
Programming of all
Fabric M2 Replication e4/20 M2 forwarding engines
FE
Fabric Engine e4/19
SoC VOQs
10.1.20.100 F2E FE
ip address 10.1.10.1/24
Engine e4/7
VOQs
Replication e4/2
Engine e4/1
Mod 4
M2 VQIs

Engine
e3/19
VOQs Routing:
FE
Fabric Egress MAC:

FE
e1/1 SoC VOQs

Engine e3/7
vlan 10 Mod 1
F2E … Mod 3
Engine e3/1
Programming of all
FE
Fabric Engine e4/19
SoC VOQs
10.1.20.100 F2E FE
ip address 10.1.10.1/24
Engine e4/7
VOQs
Replication e4/2
Engine e4/1
Mod 4
M2 VQIs

Engine
e3/19
VOQs Routing:
FE
Fabric Egress MAC:

FE
e1/1 SoC VOQs

Engine e3/7
vlan 10 Mod 1
F2E … Mod 3
Engine e3/1
Programming of all
FE
Fabric Engine e4/19
SoC VOQs
10.1.20.100 F2E FE
ip address 10.1.10.1/24
Engine e4/7
VOQs
Replication e4/2
Engine e4/1
Mod 4
M2 VQIs

Engine
e3/19
VOQs Routing:
FE
Fabric Egress MAC:

FE
e1/1 SoC VOQs

Engine e3/7
vlan 10 Mod 1
F2E … Mod 3
Engine e3/1
Programming of all
FE
Fabric Engine e4/19
SoC VOQs
10.1.20.100 F2E FE
ip address 10.1.10.1/24
Engine e4/7
VOQs
Replication e4/2
Engine e4/1
Mod 4
M2 VQIs

Engine
e3/19
VOQs Routing:
FE
Fabric Egress MAC:

FE
e1/1 SoC VOQs

Engine e3/7
vlan 10 Mod 1
F2E … Mod 3
Engine e3/1
Programming of all
FE
Fabric Engine e4/19
SoC VOQs
10.1.20.100 F2E FE
ip address 10.1.10.1/24
Engine e4/7
VOQs
Replication e4/2
Engine e4/1
Mod 4
Ingress Forwarding with Lowest Common
Denominator Model
• F3 module interoperability always Ingress Forwarding – NO proxy forwarding
with F3
– The ingress module makes all the forwarding decisions
• Supported feature set based on Lowest Common Denominator
– Feature available if all modules support the feature Not all features
supported by
software today
Fabric
VDC Type Layer 2 Layer 3 vPC VXLAN FEX MPLS OTV LISP FCoE Table Sizes
Path
F3 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ F3 size
M2 + F3 ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✗ ✗ F3 size
F2/F2E + F3 ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✓ F2E size
Agenda
• Conclusion
Nexus 7000 / Nexus 7700 Architecture Summary
Supervisor Engines
I/O Modules
Chassis
Fabrics
Conclusion
• You should now have a thorough understanding of the
Nexus 7000 / Nexus 7700 switching architecture, I/O module
design, packet flows, and key forwarding engine functions…
• Any questions?
Call to Action
• Visit the World of Solutions for
– Datacenter Networking – See the new 7702 chassis
• Meet the Engineer
• Lunch time Table Topics
• DevNet zone related labs and sessions
• Recommended Reading: for reading material and further resources for this
session, please visit www.pearson-books.com/CLMilan2015
Complete Your Online Session Evaluation
• Please complete your online session
evaluations after each session.
Complete 4 session evaluations
& the Overall Conference Evaluation
(available from Thursday)
to receive your Cisco Live T-shirt.
• All surveys can be completed via

the Cisco Live Mobile App or the
Communication Stations

BRKARC-3470 - Cisco Nexus 7K

Uploaded by

Copyright:

Available Formats

BRKARC-3470 - Cisco Nexus 7K

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKARC-3470 - Cisco Nexus 7K

Uploaded by

Copyright:

Available Formats

What are Nexus 7000 and Nexus 7700 switches designed for?

What are Nexus 7000 and Nexus 7700 switches designed for?

What are some common foundations between Nexus 7000 and Nexus 7700 switches?

What are some common foundations between Nexus 7000 and Nexus 7700 switches?

Cisco Nexus 7000 / 7700 Switch

Rohan Grover (rohang@cisco.com)

• Same release vehicles, versioning, feature-sets

• Common fabric ASICs (Fab2) and architecture

• Identical forwarding ASICs (F2E, F3)

Nexus 7010 Nexus 7018

Front N7K-C7010 Rear N7K-C7018

Front N7K-C7004 Rear

NX-OS 6.2(2) and later

Front Rear Front Rear Front Rear

Operation and Feature Consistency

Common hardware components between Nexus 7000 and Nexus 7700:

• Connects to fabric via 1G inband interface

ID and Status USB Expansion

M2 10G / 40G / 100G

F2E 10G F3 40G F3 100G

Nexus 7000 M2 I/O Modules

• Distributed L3 multicast replication

LC Fabric 2 ASIC Arbitration

VOQs VOQs Forwarding Forwarding VOQs VOQs

Nexus 7000 / 7700 F2E I/O Modules

• 48-port 1G/10G with SFP/SFP+ transceivers N7K-F248XP-25E N7K-F248XT-25E

• Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS)

• Supports Nexus 2000 (FEX) connections

Front Panel Ports LinkSec-capable (F2E fiber)

LinkSec-capable (F2E copper)

Front Panel Ports LinkSec-capable

• OTV, LISP and MPLS (L2 and L3 VPNs)

• Performance/scale boost for distributed fabric Dual-Core Acceleration

* In 7.2 release 6 x 1Gbps

• Layer 2/Layer 3 forwarding with L3/L4 services

EOBC To Fabric Modules To Central Arbiters

8 X 10G 8 X 10G 8 X 10G 8 X 10G 8 X 10G 8 X 10G

EOBC To Fabric Modules To Central Arbiters

2 X 40G 2 X 40G 2 X 40G 2 X 40G 2 X 40G 2 X 40G

EOBC To Fabric Modules To Central Arbiters

1 X 100G 1 X 100G 1 X 100G 1 X 100G 1 X 100G 1 X 100G

• Layer 2/Layer 3 forwarding with L3/L4 services

EOBC To Fabric Modules To Central Arbiters

8 X 10G 8 X 10G 8 X 10G 8 X 10G 8 X 10G 8 X 10G

EOBC To Fabric Modules To Central Arbiters

EOBC To Fabric Modules To Central Arbiters

Solution – Have an arbiter

• Each installed fabric increases available per-payload slot bandwidth

• Different I/O modules leverage different amount of available fabric bandwidth

Sup Slot 1 Fabric Fabric Sup Slot 2

8 * 55G local fabric channels

per slot bandwidth #2

Hardware Table M-Series Modules M-Series Modules with

Ingress Parser Final Results

From I/O Module To I/O Module

 Ingress MAC table lookups Ingress Parser Final Results

 Ingress ACL/QoS classification Policing

 Ingress MAC table lookups Ingress Parser Final Results

 Ingress ACL/QoS classification Policing

 Ingress MAC table lookups Ingress Parser Final Results