Introduction

Azure ExpressRoute provides private connections between your on-premises infrastructure and Microsoft Azure. By following these best practices, you can ensure optimal performance, reliability, and security for your ExpressRoute circuits.

This document outlines key recommendations for designing, implementing, and managing your ExpressRoute connections.

Network Design

High Availability and Redundancy

• Redundant Circuits: Deploy at least two ExpressRoute circuits to ensure service continuity. These circuits should terminate at different peering locations.
• Diverse Paths: Utilize diverse physical paths for your network connections to the ExpressRoute edge routers.
• Redundant Routers: Implement redundant routers on your on-premises network for BGP peering.
• Provider Redundancy: Work with your connectivity provider to ensure their infrastructure also offers high availability.

IP Addressing and Routing

• Public IP Address Space: Use a registered public IP address space that you own for your on-premises network when advertising to Microsoft. Do not use RFC 1918 private IP address space.
• ASN Configuration: Use a 4-byte ASN if your on-premises network uses 4-byte ASNs. For 2-byte ASNs, ensure they are not in the reserved range for Microsoft.
• BGP Session Management: Configure appropriate BGP timers and use route-maps or prefix lists for route filtering.
• Peerings: Understand the differences and use cases for Microsoft peering, Azure private peering, and Azure public peering. Typically, Azure private peering is used for accessing Azure IaaS resources.

Bandwidth Planning

• Accurate Forecasting: Accurately forecast your bandwidth needs based on application traffic patterns and expected growth.
• Over-provisioning: Consider a slight over-provisioning of bandwidth to accommodate traffic bursts and future needs.
• Circuit Scaling: Be prepared to scale up your circuit bandwidth if required, as this is a manageable process.

Performance Optimization

Latency Considerations

• Peering Location Selection: Choose an ExpressRoute peering location geographically close to your Azure regions to minimize latency.
• Traffic Flow Analysis: Analyze your traffic flow to identify critical applications that are sensitive to latency and ensure they are routed optimally.

Throughput Enhancement

• MTU Configuration: Ensure consistent MTU (Maximum Transmission Unit) settings across your network path, including your on-premises devices and virtual network gateways. Jumbo frames can be beneficial for specific workloads.
• TCP Window Scaling: Enable TCP window scaling on your network devices to improve throughput over high-latency links.

Security Considerations

Network Security

• Network Security Groups (NSGs) and Firewalls: Implement NSGs and Azure Firewall to control traffic flow to and from your virtual networks over ExpressRoute.
• Private Peering for IaaS: Use Azure private peering exclusively for accessing Azure IaaS resources to keep your traffic within the Microsoft network.
• Internet Breakout: For internet access from Azure, consider using Azure Firewall or Network Virtual Appliances (NVAs) with appropriate security policies. Avoid direct internet breakout over ExpressRoute unless strictly necessary and properly secured.
• Encryption: ExpressRoute itself does not provide end-to-end encryption. For sensitive data, consider implementing IPsec tunnels over the ExpressRoute circuit or using Azure services like Azure VPN Gateway for site-to-site VPN encryption.

Access Control

• Principle of Least Privilege: Apply the principle of least privilege to network access controls.
• Route Filters: Use ExpressRoute route filters to control the prefixes advertised to Microsoft and received from Microsoft.

Monitoring and Management

Monitoring Tools

• Azure Monitor: Leverage Azure Monitor for ExpressRoute to track circuit status, traffic utilization, and connection health.
• Network Performance Monitor (NPM): Use NPM to gain deeper insights into network latency and packet loss.
• On-Premises Monitoring: Integrate ExpressRoute monitoring with your existing on-premises network monitoring tools.

Alerting

• Proactive Alerts: Set up alerts for circuit status changes, high latency, or high utilization to proactively address issues.
• BGP State Changes: Monitor BGP session status for any unexpected changes.

Circuit Management

• Tagging: Use Azure resource tags to organize and manage your ExpressRoute circuits and related resources.
• Documentation: Maintain up-to-date documentation of your ExpressRoute configuration, including peering locations, IP addresses, and routing policies.

Cost Management

Bandwidth Costs

• Right-Sizing: Accurately size your ExpressRoute circuits to avoid paying for unused bandwidth.
• Data Transfer: Understand the data transfer costs associated with your ExpressRoute circuits, especially for outbound data from Azure.

Port Charges

• Circuit Ports: Be aware of the port charges from both Microsoft and your connectivity provider.
• Redundancy Costs: Factor in the cost of redundant circuits for high availability.