Data Center Network Traffic Characteristics

A profound understanding of data center traffic characteristics is crucial for designing high-performance networks within data centers. This analysis presents the main features of internal data center network traffic and explains how these characteristics impact optical network design, including considerations for the dci network infrastructure. Several research reports, such as those from Microsoft Research, have analyzed data center network traffic, providing valuable insights for optimizing both local data center networks and the broader dci network ecosystem.

Data centers can be categorized into three types: campus data centers, enterprise private data centers, and cloud computing data centers. While these three types share some common traffic characteristics (such as average packet length), they exhibit significant differences in areas like applications and data flows, which in turn affect both their internal architecture and their connection to the dci network.

The traffic characteristics presented in these research reports are derived from measurements in real-world data centers and provide a foundation for understanding how to design more efficient systems, including the critical dci network components that connect geographically distributed data centers.

Traffic Applications

The applications running in a data center depend on its type, which directly influences traffic patterns that must be accommodated by both the internal network and the connected dci network. In campus data centers, HTTP traffic dominates as these facilities primarily support web-based educational and research applications.

In contrast, enterprise private data centers and cloud computing data centers handle a more diverse mix of traffic types, including HTTP, HTTPS, LDAP, and database traffic (such as MapReduce operations). This diversity requires more flexible network architectures that can efficiently manage different protocols and traffic priorities, both within the data center and across the dci network.

The specific application mix directly impacts network design decisions, including bandwidth allocation, quality of service implementations, and security measures. For example, cloud data centers with heavy database traffic require high-throughput, low-latency connections that may extend across the dci network to connect geographically distributed database clusters.

Traffic Locality

When data flows between two servers, a connection (typically a TCP connection) is established. Traffic locality distinguishes whether data flows occur between servers within the same rack (intra-rack traffic) or between servers in different racks (inter-rack traffic). This distinction is vital not only for internal data center design but also for optimizing how traffic is routed to and from the dci network.

Statistics show that inter-rack traffic can account for 10% to 80% of total traffic, depending on the specific applications. Notably, in campus and enterprise private data centers, intra-rack traffic typically represents 10% to 40% of total traffic, with the remainder being inter-rack or destined for the dci network.

In cloud computing data centers, the majority of traffic is intra-rack, sometimes reaching up to 80% of total traffic. Operators often place servers that exchange large volumes of data within the same rack to minimize latency and reduce load on the core network and dci network connections. Traffic locality significantly influences data center network topology design, including how optical networking components are integrated with traditional electrical switching infrastructure, both within the data center and in the dci network.

When inter-rack communication represents a large portion of traffic, high-speed networking between racks becomes necessary, while cost-effective commercial switches can be used within racks. In such cases, optical networks can be effectively utilized to provide the required inter-rack bandwidth, with lower-cost electrical switches handling intra-rack communication. This same principle applies to the dci network, where high-capacity optical links connect geographically separate data centers, while electrical switching handles local traffic within each facility.

Traffic Size and Duration

A data flow is defined as an active connection between two or more servers. Most data center traffic consists of lightweight flows (less than 10KB), and the majority of these flows have durations of a few hundred milliseconds or less. These characteristics influence both internal data center design and dci network optimization strategies.

Flow duration also significantly impacts the design of data center optical network topologies, including components of the dci network. If flow durations are measured in seconds, optical network devices with longer reconfiguration times can be used to provide higher bandwidth, as the reconfiguration overhead becomes relatively acceptable in such scenarios.

Understanding flow size and duration patterns is crucial for optimizing both packet switching and circuit switching technologies in modern data centers. Short-lived flows benefit from the low-latency characteristics of electrical switching, while longer-lived flows can take advantage of the higher bandwidth and lower energy consumption of optical switching, both within the data center and across the dci network. This hybrid approach allows for optimal resource utilization across the entire network infrastructure, from local server-to-server communication to wide-area dci network connections.

Concurrent Flows

The number of concurrent data flows on each server significantly influences data center network topology design, including how servers connect to both local switches and the broader dci network infrastructure. If the number of concurrent flows can be supported by the number of optical connections, optical switching networks offer significant advantages over electrical switching networks.

In most data centers, the average number of concurrent flows per server is approximately 10. This relatively modest number of concurrent connections per server allows for efficient implementation of optical switching technologies, as the required number of optical ports and connections remains manageable.

This characteristic has important implications for both intra-data center design and dci network architecture. By understanding typical concurrency patterns, network designers can optimize the balance between electrical and optical switching components, ensuring that the network can efficiently handle peak loads while maintaining cost-effectiveness. For the dci network, this means designing with appropriate connection density and switching capacity to handle the aggregate concurrent flows between interconnected data centers.

Packet Size

Data center packet sizes exhibit a bimodal distribution, with packets typically being around 200B or 1400B. This pattern exists because packets are either small control packets or fragments of larger files, which are split at the data link layer into maximum Ethernet frames (1550B). This characteristic influences many aspects of network design, from buffer sizing to transmission efficiency across both local networks and the dci network.

The bimodal nature of packet sizes presents unique challenges for network optimization. Small control packets require low latency and minimal processing overhead, while large data packets benefit from high-throughput, efficient transmission. This duality influences the design of both internal data center networks and the dci network, where balancing these requirements is essential for optimal performance. Network equipment must be optimized to handle both packet size categories efficiently, ensuring that neither small control messages nor large data transfers are disadvantaged in terms of latency or throughput.

Link Utilization

Research reports indicate that across all types of data centers, intra-rack and aggregation layer link utilization is relatively low, while core layer link utilization is significantly higher. This pattern holds true for both the internal data center network and the connecting dci network, where core interconnections typically operate at higher utilization rates.

Within racks, link speeds are generally 1Gb/s (in some cases, servers within a rack may be configured with two or more 1Gb/s links), while aggregation and core layer links typically operate at 10Gb/s. These different speed tiers create a hierarchical bandwidth structure that must be carefully managed, both within the data center and in the dci network that connects multiple facilities.

The findings on link utilization suggest that deploying high-bandwidth links in the core layer is essential, while the 1Gb/s links currently deployed within racks can meet future network demands. This principle extends to the dci network, where core interconnections between data centers require the highest bandwidth to accommodate the aggregated traffic from multiple facilities. As data volumes continue to grow, this hierarchical approach to bandwidth allocation will become even more important for maintaining efficient operation across the entire network infrastructure, from local server connections to global dci network links.

Data Center Traffic Growth

While the qualitative traffic characteristics of data center networks remain relatively consistent, the volume of network traffic within data centers is growing rapidly. Emerging network applications (such as cloud computing) and continuous improvements in access network performance are driving the need for larger-scale data centers to handle increasing traffic volumes, both internally and across the dci network.

The growth in data center network traffic stems not only from the expansion of data center规模 but also from improvements in server performance. With the proliferation of multi-core processors, communication demands between servers within data centers will continue to grow. According to Amdahl's Law, every 1MHz increase in processor clock speed requires a 1MB increase in memory capacity and a 1Mb/s increase in I/O speed.

Considering current data center servers, which are typically configured with 4 parallel 4-core processors each running at 2.5GHz, each server requires a total I/O bandwidth of 40Gb/s. For a data center with 100,000 servers, the total required I/O bandwidth reaches petabits per second, a staggering figure that underscores the importance of both high-performance internal networks and robust dci network connections.

To address the upcoming bandwidth challenges, service providers worldwide are competing to upgrade existing networks with higher bandwidth links. Statistics show that between 2011 and 2016, the compound annual growth rate (CAGR) of 100G Ethernet ports exceeded 170%, reflecting the rapid adoption of higher-speed technologies in both data center cores and the dci network infrastructure.

Figure 1.2 predicts future changes in server data rates within data centers. As shown, while only a small percentage of servers used 40G Ethernet NICs in 2012, the majority of servers had adopted 40G Ethernet by 2017. This trend continues, with 100G and higher speeds becoming increasingly common, driving the need for corresponding upgrades in both internal data center networks and the dci network.

High-performance switches will need to consume significant energy for transceiver electro-optical and opto-electrical conversions, as well as for electrical domain switching. Clearly, if data rates continue to grow exponentially, data center networks—including both internal infrastructure and the dci network—will face increasing demands for higher speeds, lower latency, and reduced energy consumption. Meeting these challenges will require continued innovation in both optical and electrical networking technologies, as well as new approaches to designing and managing the entire data center network ecosystem.