Troubleshooting Azure Application Gateway

1. Unhealthy Backend Instances

When Application Gateway reports backend instances as unhealthy, it means the health probes are failing to receive a successful response. Follow these steps:

• Verify that the backend servers are running and accessible from the Application Gateway's subnet.
• Check Network Security Groups (NSGs) and Azure Firewall rules to ensure traffic from Application Gateway's frontend IP address to backend IPs on the required port is allowed.
• Confirm that the health probe configuration (protocol, port, path, host) accurately matches the backend application's listening settings.
• Ensure the backend application responds to the health probe path within the configured timeout period.
• Check application logs on the backend servers for any errors that might prevent it from responding to probes.
• If using custom probes, ensure the response code is within the success range (typically 2xx or 3xx).

Example Health Probe Configuration:

    {
        "name": "myHealthProbe",
        "properties": {
            "protocol": "Http",
            "host": "10.0.0.1",
            "path": "/health",
            "interval": 30,
            "timeout": 15,
            "unhealthyThreshold": 3,
            "retries": 5,
            "provisioningState": "Succeeded"
        }
    }
                

2. Connection Timeouts

Connection timeouts indicate that Application Gateway cannot establish a connection to the backend instances within the configured timeout period. Consider the following:

• Network Path: Trace the network path from the Application Gateway subnet to your backend resources. Look for any network devices (e.g., firewalls, VPN gateways) that might be dropping packets or introducing latency.
• Backend Server Load: High CPU utilization or memory pressure on backend servers can lead to slow responses or connection failures. Monitor backend server performance metrics.
• Application Responsiveness: The application itself might be slow to respond to requests. Analyze application performance and optimize critical code paths.
• Socket Keep-Alive: Ensure that your backend applications are configured to handle keep-alive connections appropriately. Application Gateway might reuse connections, and stale connections can cause issues.
• Timeout Settings: Review the HTTP settings' request timeout in Application Gateway. If your application operations are long-running, you might need to increase this value, but be mindful of potential resource exhaustion.

3. SSL/TLS Errors

SSL/TLS errors can prevent clients from securely connecting to your application. Common causes and solutions include:

• Certificate Validity: Ensure the SSL certificate installed on Application Gateway (or the backend if using end-to-end SSL) is not expired, has the correct domain name, and is trusted.
• Certificate Chain: Verify that the entire certificate chain (intermediate certificates) is correctly installed.
• Cipher Suites and Protocols: Application Gateway's SSL policy determines which cipher suites and TLS versions are supported. Ensure your clients' configurations are compatible. You can customize the SSL policy to support specific requirements.
• End-to-End SSL: If you've configured end-to-end SSL, ensure that the backend servers have valid SSL certificates trusted by Application Gateway. You may need to upload the backend server's root certificate to Application Gateway.
• SNI (Server Name Indication): If your backend servers host multiple SSL-enabled sites, ensure SNI is correctly configured on both Application Gateway and the backend.

4. HTTP/HTTPS Redirection Issues

Problems with HTTP to HTTPS redirection or vice-versa can be frustrating. Here's how to troubleshoot:

• Redirection Rule Configuration: Double-check the details of your redirection rule in Application Gateway. Ensure the source listener, target listener, and redirect type (permanent/temporary) are correctly set.
• Listener Configuration: Verify that the listener corresponding to the redirect target is correctly configured with the appropriate protocol and port.
• Order of Rules: If you have multiple rules, ensure the redirection rule is evaluated before other rules that might conflict. Rules are processed in order of priority.
• Backend Health: While less common for redirection, ensure your backend pools are healthy if the redirection is part of a larger flow.

Example Redirection Rule:

    {
        "name": "redirect-http-to-https",
        "properties": {
            "priority": 100,
            "conditions": [
                {
                    "name": "url",
                    "parameters": [
                        {
                            "name": "path",
                            "pattern": "/*",
                            "ignoreCase": true
                        }
                    ]
                }
            ],
            "actions": [
                {
                    "type": "RedirectConfig",
                    "parameters": {
                        "redirectType": "Permanent",
                        "targetListener": {
                            "id": "/subscriptions/.../listeners/appgw-listener-https"
                        }
                    }
                }
            ]
        }
    }
                

5. WAF Blocking Legitimate Traffic

If your Web Application Firewall (WAF) is too aggressive and blocks legitimate requests, follow these steps to diagnose and resolve:

• Review WAF Logs: The most crucial step is to examine the WAF logs. These logs will indicate which rule was triggered and why a specific request was blocked. Enable WAF diagnostic logging to send logs to a storage account, Log Analytics workspace, or Event Hub.
• Identify the Triggering Rule: Note the `ruleId` and `ruleSetType` from the logs. This tells you which WAF rule fired.
• Tune Exclusion Rules: If a specific rule is consistently blocking valid traffic, you can create exclusion rules. You can exclude specific request headers, cookies, or parts of the request body based on variable names, operators, and values.
• Custom Rules: For more granular control, consider creating custom WAF rules that allow or block traffic based on specific IP addresses, geographical locations, or other request attributes.
• Mode: Ensure your WAF is in the correct mode. For troubleshooting, you might temporarily set it to "Detection" mode (which logs but doesn't block) to identify problematic rules before switching back to "Prevention" mode.
• Managed Rule Sets: Application Gateway uses managed rule sets (e.g., OWASP Core Rule Set). If a specific rule within a managed set is problematic, consider disabling that specific rule ID if possible, or use exclusions.

Example WAF Exclusion Rule Snippet:

    "exclusionManagedRuleSets": [
        {
            "ruleSetType": "OWASP",
            "ruleSetVersion": "3.2",
            "exclusions": [
                {
                    "matchVariable": "RequestBody",
                    "selector": "variable.UserAgent",
                    "operator": "Contains",
                    "negationCondition": false,
                    "values": ["MyCustomAgent/1.0"]
                }
            ]
        }
    ]