How to troubleshoot GCP Cloud Armor Errors

Sometimes you may be experiencing intermittent blocking and passing of traffic with a preconfigured rule in  GCP Cloud Armor. 

For example, evaluatePreconfiguredExpr('php-stable') rule in  where some requests get a 403 (Forbidden) while others pass. This is a common scenario when dealing with WAF rules, and there are several potential reasons for this behavior:

1. False Positives:

• Legitimate PHP Code/Parameters: The php-stable rule set contains numerous signatures designed to detect common PHP vulnerabilities. Sometimes, legitimate requests might contain patterns that coincidentally match these signatures, leading to a false positive block.

• Dynamic Content: If your application generates dynamic content or accepts user input, the exact content of a request can vary. A slight change in a parameter value or a different combination of inputs might trigger a signature in one instance but not another.

• Sensitivity Level: Cloud Armor preconfigured WAF rules can often be configured with different sensitivity levels (e.g., 1 to 4). A higher sensitivity level means more aggressive detection and a higher chance of false positives. If the rule is set to a high sensitivity, it might be too strict for your application's normal traffic.

Logging and Debugging (Crucial Step!):

• Enable Verbose Logging: This is the most important step to diagnose the issue. Cloud Armor logs (available in Cloud Logging under http_load_balancer resource type) can provide detailed information about why a request was blocked, including the specific rule ID and the part of the request that triggered it.

  • To enable verbose logging: gcloud compute security-policies update [CLOUD_ARMOR_POLICY_NAME] --log-level=VERBOSE

• Analyze Logs: Look for log entries with outcome: "DENY" and examine the enforcedSecurityPolicy object. This object will show:

  • configuredAction: The action taken (e.g., "DENY").

  • name: The name of your Cloud Armor security policy.

  • outcome: The final outcome.

  • preconfiguredExprIds or signature_id: This is critical. It will tell you the specific OWASP CRS rule ID that caused the block within the php-stable set.

• Preview Mode: Before applying any changes, use Cloud Armor's "preview mode" for your rules. This allows you to see what traffic would be blocked without actually enforcing the block, helping you identify false positives.

Troubleshooting Steps:

1. Enable Verbose Logging for your Cloud Armor policy.

2. Reproduce the issue: Make repeated requests that intermittently get blocked.

3. Go to Cloud Logging and filter for logs from your HTTP(S) Load Balancer (resource.type="http_load_balancer") and look for entries where statusDetails indicates denied_by_security_policy or where jsonPayload.enforcedSecurityPolicy.outcome is "DENY".

4. Examine the logs carefully to identify the preconfiguredExprIds (or signature ID) that are causing the blocks.

5. Identify the problematic rule(s): Once you have the specific rule ID(s) that are causing the false positives, you have a few options:

   • Exclude the specific rule ID(s): If a certain part of your legitimate traffic consistently triggers a specific signature, you can exclude that signature from the php-stable rule set for your policy. This is generally preferred over disabling the entire php-stable rule.

     
     

     gcloud compute security-policies rules update [PRIORITY] \
         --security-policy [POLICY_NAME] \
         --expression "evaluatePreconfiguredExpr('php-stable', ['owasp-crs-v030001-idXXXXX-php'])" \
         --action "deny-403"
     

     (Replace [PRIORITY], [POLICY_NAME], and idXXXXX-php with your actual values and the identified rule ID).

   • Adjust Sensitivity (if applicable): If the rule allows it, you might consider lowering the sensitivity for the php-stable rule. However, this should be done with caution as it might reduce overall protection.

   • Create Request Exclusions: If the false positive is tied to a specific URL path or a header, you can create an exclusion within the rule to prevent it from being evaluated for those specific conditions.

     
     

     gcloud compute security-policies rules update [PRIORITY] \
         --security-policy [POLICY_NAME] \
         --expression "evaluatePreconfiguredExpr('php-stable')" \
         --action "deny-403" \
         --preconfigured-waf-config-exclusions="requestHeadersToExclude=[{op:EQUALS, value: 'User-Agent', targetRuleIds: ['owasp-crs-v030001-id941180-xss']}]"
     

     This example shows excluding a header, but you can also exclude request URIs, query parameters, or request fields.

6. Test in Preview Mode: Always test your exclusions or sensitivity changes in preview mode first to ensure they resolve the issue without opening up new vulnerabilities.

7. Review Rule Priority: Cloud Armor rules are evaluated in order of priority (lower number = higher priority). Ensure that any allow rules you might have are at a higher priority than your deny rules to prevent unintended blocks.