Troubleshooting Azure Application Gateway

Azure Application Gateway is a web traffic load balancer that enables you to manage traffic to your web applications. This document provides guidance on troubleshooting common issues you might encounter.

Common Issues and Solutions

1. Application Gateway Not Reachable

Symptoms:

• You receive timeouts or connection refused errors when trying to access your application via the Application Gateway's public or private IP address/FQDN.
• Health probes are failing for all backend targets.

Possible Causes & Solutions:

• Network Security Group (NSG) Restrictions: Ensure that NSGs associated with the Application Gateway's subnet and the backend subnet allow traffic on ports 80 and 443 (or custom ports if configured) from the Application Gateway's IP.
• Firewall Rules: Check any intervening firewalls (e.g., Azure Firewall, on-premises firewalls) that might be blocking traffic.
• Backend Pool Health: Verify that your backend servers are running and accessible from the Application Gateway. Check the health probe status.
• Routing Issues: Confirm that the route tables associated with the subnets are correctly configured to allow traffic flow.
• Application Gateway Configuration: Double-check the frontend IP configuration, listener settings (port, protocol, hostname), and routing rules.

2. Health Probes Failing

Symptoms:

• The Application Gateway marks backend servers as unhealthy, leading to traffic not being routed to them.
• You see errors like "No healthy backend servers."

Possible Causes & Solutions:

• Incorrect Probe Configuration: Verify the probe's protocol (HTTP/HTTPS), port, host, URI path, and interval settings. The URI path should point to a resource that returns a 2xx or 3xx status code.
• Backend Server Issues: Ensure the backend application is running, responding to requests on the configured probe path, and not overloaded.
• Network Connectivity: Confirm that the Application Gateway can reach the backend servers on the probe's configured port. Check NSGs and firewalls.
• SSL Certificate Issues (for HTTPS probes): If using HTTPS probes, ensure the backend server's SSL certificate is valid, trusted, and matches the configured hostname. If the certificate is self-signed, you need to configure the probe to trust it.
• HTTP Response Codes: The backend application must return an HTTP status code between 200 and 399 for the probe to be considered successful.

3. SSL/TLS Errors

Symptoms:

• Clients receive SSL/TLS certificate errors when connecting to the Application Gateway.
• HTTPS traffic is not being established.

Possible Causes & Solutions:

• Invalid or Expired Certificate: Ensure the SSL certificate uploaded to the Application Gateway is valid, not expired, and trusted by clients.
• Incorrect Certificate Format: The certificate must be in PEM or PFX format.
• Hostname Mismatch: The hostname in the client's request must match a hostname in the SSL certificate's Subject Alternative Name (SAN) or Common Name (CN).
• Listener Misconfiguration: Verify that the HTTPS listener is configured with the correct certificate.
• Backend SSL Settings: If using end-to-end SSL, ensure the backend server's certificate is valid and trusted by the Application Gateway. For HTTPS probes, the same applies.

4. Performance Degradation or High Latency

Symptoms:

• Web applications are slow to respond.
• High CPU or memory utilization on backend servers.

Possible Causes & Solutions:

• Under-provisioned Application Gateway: Consider scaling up your Application Gateway SKU or increasing the instance count if it's a bottleneck.
• Backend Server Capacity: Ensure your backend servers have sufficient resources (CPU, memory, network I/O) to handle the load.
• Inefficient Application Code: Optimize your web application code for better performance.
• Suboptimal Load Balancing Settings: Review the Application Gateway's load balancing algorithms and session affinity settings.
• Network Latency: Check network latency between the Application Gateway and backend servers, and between clients and the Application Gateway.

5. Rewrites or Redirection Not Working

Symptoms:

• HTTP headers are not being rewritten as expected.
• Redirection rules are not functioning correctly.

Possible Causes & Solutions:

• Incorrect Rule Configuration: Carefully review the source/target for rewrites and the condition/target for redirection rules.
• Order of Operations: Ensure rewrite rules are applied in the correct order if multiple rules are configured.
• Special Characters and Encoding: Be mindful of special characters and proper URL encoding.
• Backend Application Interference: Sometimes, backend applications might also perform rewrites or redirects, which can conflict with Application Gateway rules.

Diagnostic Tools and Logs

Leverage the following Azure tools to diagnose issues:

• Application Gateway Diagnostics: Enable diagnostics and send logs to a Log Analytics workspace for detailed analysis. Key logs include:
  • Access Logs: Details about requests processed by the gateway.
  • Application Gateway Performance Logs: Metrics on request latency, throughput, and backend health.
  • Application Gateway Firewall Logs (WAF): For WAF-related events and blocked requests.
  • Application Gateway Execution Logs: Detailed operational logs.
• Azure Monitor: Use Azure Monitor metrics to track Application Gateway performance, traffic, and error rates.
• Network Watcher: Tools like Connection Troubleshoot and IP Flow Verify can help diagnose network connectivity issues.
• Backend Server Logs: Examine the logs on your backend application servers for application-specific errors.

By systematically checking these common issues and utilizing the available diagnostic tools, you can effectively troubleshoot and resolve problems with your Azure Application Gateway deployments.