Data Center Network Architecture
Exploring the evolution, challenges, and innovations in modern data center networking, with a focus on effective data center interconnect design principles.
Requirements for Modern Data Centers
First, we examine some of the requirements that emerging large-scale data centers place on communication and networking infrastructure, including critical aspects of data center interconnect design. The first consideration is the target scale of the data center. While, from an economies of scale perspective, larger data centers are more advantageous, they are typically constrained by the power supply at their location. Furthermore, to ensure fault tolerance and minimize latency, data centers should be distributed across various global locations, each with optimized data center interconnect design.
The second consideration is the total computational power and communication capacity required by the target applications within the data center. Taking social networks as an example, their websites must store and replicate all user-generated content across a server cluster. The network supporting such applications is crucial because, for each external request, hundreds or even thousands of servers must be connected in parallel to satisfy the request—making data center interconnect design a critical factor in performance.
The final consideration is the degree of multiplexing of individual servers across multiple applications and properties. For instance, a portal website like Yahoo! can host hundreds of user-facing personalized services as well as a similar number of internal applications to support batch data processing, index generation, advertising delivery, and general business activities. Each of these functions requires careful data center interconnect design to ensure efficient operation.
While there are no definitive data to answer these questions, we believe that the trend toward increasing computational density within data centers will certainly reach the level of tens of thousands of servers. It is possible to separate various applications and have them run on machines with dedicated interconnection architectures, meaning each dedicated machine's interconnection network would be a small-scale network. However, ideally, we want the incremental cost of network expansion to be moderate, with sufficient flexibility to dynamically move computations and support increasingly large-scale applications—all of which depend on thoughtful data center interconnect design.
Traditional Scale-Up Architecture
Figure 2.1 shows a typical data center network architecture using the traditional scale-up approach. Each rack contains dozens of servers connected via copper cables or optical fibers to a Top-of-Rack (ToR) switch. The ToR switches then connect to access layer switches via optical transceivers. If each ToR switch uses "n" uplinks, the entire network can support "n" access switches within a single cluster, as ToR switches are typically connected in parallel to multiple switches.
The number of ports on each access switch determines the total number of ToR switches that can be supported. If each ToR switch uses "d" downlinks to connect to hosts, the scale of each cluster in the network can expand to c×d×u ports (with a convergence ratio of d:c at the ToR layer). If this two-tier architecture, which is usually limited by the switching chip capacity, is insufficient in scale, additional layers can be added to the hierarchy (creating an aggregation layer at the cost of increased latency and higher internal network connection overhead). For connecting multiple clusters, three-tier cluster routers (CR) are often used at the top of the data center structure, a critical component in traditional data center interconnect design.
Ideally, a fully meshed network structure that directly connects any two servers in the data center would provide full bisectional bandwidth while simplifying programming and improving server computational efficiency. However, such a design is prohibitively expensive due to the convergence that typically applies at each layer. When the system cannot support bandwidth requirements, new hardware with higher capacity can be purchased to build a larger core (the scale-up approach), which represents a fundamental principle in traditional data center interconnect design.
Figure 2.1: Traditional Hierarchical Data Center
Scale-up network model with hierarchical structure and various convergence ratios
Limitations of Scale-Up Architectures
While scale-up network architectures can save costs and are easier to set up, particularly for small to medium-sized data centers, they require significant upfront investment in more expensive and highly reliable high-capacity hardware. In particular, switches and routers at higher levels in the hierarchy need to handle more traffic, with costs increasing significantly as their availability requirements grow. Furthermore, their inability to scale beyond current deployment limits makes them less attractive for large-scale data centers, highlighting the need for more flexible data center interconnect design approaches.
Over the past decade, with the development of commercial silicon switching chips and software-defined networking (SDN) control planes (http://www.openflow.org/), the scale-out model has replaced the scale-up model as the foundation for providing large-scale computing and storage platforms. This shift has revolutionized data center interconnect design, enabling more flexible and cost-effective network expansions.
Scale-Out Architecture
Figure 2.2 shows a scale-out data center architecture, representing a modern approach to data center interconnect design. To build large-scale, non-blocking network structures, an array of small pods composed of identical switches (built on commercial switching chips) should be employed. The access layer can be traditional ToR switches performing layer 2 switching functions or transparently aggregated server links connected to aggregation switches. The network features full bisectional bandwidth with extensive path diversity both within and between pods, a key advancement in contemporary data center interconnect design.
The scale-out data center model offers numerous advantages for building large-scale data centers with optimized data center interconnect design: ① Flexibility—network bandwidth can be allocated to different applications in a modular fashion; ② Scalability—through its modular approach, computing and storage capacity can be added on demand, allowing the data center architecture to scale while maintaining constant cost per port and per bit/second of bisectional bandwidth; ③ Accessibility—without bandwidth fragmentation and convergence in large interchangeable server pools, each server's computing power can be widely accessed; ④ Reliability—with extensive path diversity, network performance degrades only gradually in the event of failures; ⑤ Manageability—through a software-defined control plane, thousands of servers can be managed as a single computer. Petabytes (1 PB = 10¹⁵ B) of data can be moved and managed under a single distributed system and a global namespace, all enabled by sophisticated data center interconnect design.
Figure 2.2: Modern Scale-Out Data Center
Scale-out model with non-blocking network structure and full bisection bandwidth
Challenges in Scale-Out Data Centers
Scale-out data centers also present numerous technical and deployment challenges at the petabyte scale and beyond, many of which directly impact data center interconnect design strategies. While the software and management technologies involved in building data centers are beyond the scope of this chapter, existing construction techniques still have many limitations, including the following aspects that affect effective data center interconnect design.
Management Complexity
The large number of electrical packet switches (EPS) significantly increases management complexity and overall operational costs. This challenge is magnified in data center interconnect design where coordination between numerous network elements becomes increasingly complex.
Cost Considerations
The cost of fiber optic cables and optical transceivers dominates the total cost of network architectures. In data center interconnect design, these components represent a significant portion of the budget, especially as data centers scale to larger sizes.
Power Consumption
As bandwidth increases, the power consumption of optical transceivers limits port density. This is a critical factor in data center interconnect design, as power constraints can restrict scaling capabilities and increase operational expenses.
Wiring Complexity
Millions of meters of fiber optic cable are required to interconnect large scale-out data centers, leading to daunting deployment and operational overhead. This complexity is a major consideration in data center interconnect design, affecting both initial deployment and ongoing maintenance.
Scale-Up vs Scale-Out Architectures
Comparison of key characteristics between traditional scale-up and modern scale-out approaches to data center interconnect design
Data Center Interconnect Design Focus Areas
The evolution from scale-up to scale-out architectures represents a fundamental shift in how data centers are designed and operated, with profound implications for data center interconnect design. As data demands continue to grow exponentially, the ability to efficiently scale network infrastructure while maintaining performance, reliability, and cost-effectiveness will remain a critical challenge. Advanced data center interconnect design principles will play an increasingly important role in addressing these challenges, enabling the next generation of large-scale data centers to meet the needs of emerging applications and services.
Learn more