Grilled Cheese and Service Control Policies (SCPs)

Marty Henderson -

Today I made, quite possibly, the worst grilled cheese of my life. The cheese was cold, I didn't wait for the pan to get warm before throwing the butter and bread in, and it all just became a too-toasted-bread-and-unmelted-cheese mess. It's quite disappointing that I did this because I am usually proficient at making grilled cheese. Perhaps I should've shown more patience but I was hungry.

Needless to say, I still ate half of it. I was hungry, after all.

But while munching on the sad substitute for a sandwich, I realized that it kind of reminds me of how service control policies often go in half baked, come out bad, and then are hated when, really, a bit more preparedness can help quite a bit. So, let's discuss how Service Control Policies work and some common tips and tricks.

Before Service Control Policies can be enabled

Service Control Policies, or SCPs, are done based on your organization, so AWS Organizations need to be set first. If you don't have an org, you can't even start on SCPs. Plus, if you don't have control of the main organization, whether through console or through automation - the better choice - you can't apply them.

I highly recommend using AWS Orgs, even on personal accounts, so you can divide your projects up and create a "blast radius" 0r places you can test and destroy things without impacting your long-term or more serious projects. My accounts are setup with an org main account, a couple of testing accounts, my "production" account, and my DeepRacer account (mostly for budgeting reasons on that one). It also helps that I control which of my friends can use certain accounts through SSO (using Identity Center and an IdP called Jumpcloud).

Alright, with all the prework done, let's jump into SCPs.

What are SCPs?

SCPs, at their core, are policies that allow or deny certain functions or entire services on a given account. For instance, you can deny any DeepRacer functions on an account with an appropriate policy, prevent removal of GuardDuty, or even enforce tagging.

They feel a lot like IAM policies, which I've written on before. They control down to granular functions if you desire, but they impact an entire account instead of a role or user. They can attach to an account or an entire Organizations Unit (OU) - something out of scope for this discussion, but still applies to this.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": [
        "cloudwatch:DeleteAlarms",
        "cloudwatch:DeleteDashboards",
        "cloudwatch:DisableAlarmActions",
        "cloudwatch:PutDashboard",
        "cloudwatch:PutMetricAlarm",
        "cloudwatch:SetAlarmState"
      ],
      "Resource": "*"
    }
  ]
}

An SCP contains an allow or deny statement - like IAM policies - but attaches to the account. One of AWS' examples is the "Prevent users from disabling CloudWatch or altering its configuration", which, as expected, prevents users from disabling CloudWatch or altering it's alarms and similar. It looks like this:

As you can see from this build, it denies, on all resources, the ability to delete or create alarms, delete or create dashboards, or snooze an alarm. Applied to an account, this impacts all non-root users - even AdministratorAccess. In fact, an SCP deny overrides IAM allow policies.

The fact it overrides an IAM policy (with denies) is important. A common issue I have seen is administrators and engineers believing they can grant the ability to delete alarms or expecting the AdministratorAccess policy to override it. This is an important consideration when applying a deny SCP as you cannot override it within the account.

You can also layer SCPs and, like everything in AWS, a deny will always override an allow. If you have another SCP like

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:PutDashboard",
        "cloudwatch:PutMetricAlarm",
      ],
      "Resource": "*"
    }
  ]
}

The deny policy above would still now allow anyone on the account to create new alarms or dashboards.

Let's discuss some tactics to create better SCPs to use.

Limit by User

Some policies can be applied directly to certain users from the SCP. For example, if you only wanted to let the "IAMAdmin" role to edit IAM policies, you could apply something like 

{    
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyAccessWithException",
      "Effect": "Deny",
      "Action": [
        "iam:DeleteRole",
        "iam:DeleteRolePermissionsBoundary",
        "iam:DeleteRolePolicy",
        "iam:PutRolePermissionsBoundary",
        "iam:PutRolePolicy",
        "iam:UpdateAssumeRolePolicy",
        "iam:UpdateRole",
        "iam:UpdateRoleDescription"
      ],
      "Resource": [
        "*"
      ],
      "Condition": {
        "StringNotLike": {
          "aws:PrincipalARN":"arn:aws:iam::*:role/IAMAdmin"
        }
      }
    }
  ]
}

This would prevent all changes to IAM roles (except attachments) unless you're current role is the IAMAdmin role. However, you can apply an even more narrow policy.

A good example of this is if you have a deployer role policy that you don't want people to grant more abilities to, you can tie it directly to that role, such as

{    
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyAccessWithException",
      "Effect": "Deny",
      "Action": [
        "iam:AttachRolePolicy",
        "iam:DeleteRole",
        "iam:DeleteRolePermissionsBoundary",
        "iam:DeleteRolePolicy",
        "iam:DetachRolePolicy",
        "iam:PutRolePermissionsBoundary",
        "iam:PutRolePolicy",
        "iam:UpdateAssumeRolePolicy",
        "iam:UpdateRole",
        "iam:UpdateRoleDescription"
      ],
      "Resource": [
        "arn:aws:iam::*:role/deployer-role"
      ],
      "Condition": {
        "StringNotLike": {
          "aws:PrincipalARN":"arn:aws:iam::*:role/IAMAdmin"
        }
      }
    }
  ]
}

This is an example of a more nuanced policy that likely won't need to be removed in the future. It allows only a specific role, IAMAdmin, to edit the deployer role. This means you can prevent other account access, even AdminAccess from editing the deployer role. These kind of policies prevent what are known as "supply chain attacks" that by gaining access to one thing, they can gain access to other things.

Limit insecure practices

A common use of SCP is to limit practices that can risk security or are otherwise best practices.

A common best practice is secure objects that go into your S3 buckets. The policy recommended by AWS is 

{
  "Effect": "Deny",
  "Action": "s3:PutObject",
  "Resource": "*",
  "Condition": {
    "Null": {
      "s3:x-amz-server-side-encryption": "true"
    }
  }
}

What this does is prevent anyone from putting an object in that is not encrypted by server side policies.

However, where this can be a problem is when you have existing accounts and functions that use unencrypted buckets from legacy events. I've come across applications using buckets that don't enforce server-side encryption from 4+ years ago. In this case, this SCP could break your application - yikes!

When applying SCPs, especially hard denials with no exceptions, it's best to test them in lower environments to see if they will stop your legacy applications from running.

Tagging

Another favorite SCP to have is to force tagging. However, it's important to realize what the default tagging policy requires.

The default tagging policy forces two tags - Project and CostCenter - to create EC2 instances or SecretsManager secrets.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyCreateSecretWithNoProjectTag",
      "Effect": "Deny",
      "Action": "secretsmanager:CreateSecret",
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:RequestTag/Project": "true"
        }
      }
    },
    {
      "Sid": "DenyRunInstanceWithNoProjectTag",
      "Effect": "Deny",
      "Action": "ec2:RunInstances",
      "Resource": [
        "arn:aws:ec2:*:*:instance/*",
        "arn:aws:ec2:*:*:volume/*"
      ],
      "Condition": {
        "Null": {
          "aws:RequestTag/Project": "true"
        }
      }
    },
    {
      "Sid": "DenyCreateSecretWithNoCostCenterTag",
      "Effect": "Deny",
      "Action": "secretsmanager:CreateSecret",
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:RequestTag/CostCenter": "true"
        }
      }
    },
    {
      "Sid": "DenyRunInstanceWithNoCostCenterTag",
      "Effect": "Deny",
      "Action": "ec2:RunInstances",
      "Resource": [
        "arn:aws:ec2:*:*:instance/*",
        "arn:aws:ec2:*:*:volume/*"
      ],
      "Condition": {
        "Null": {
          "aws:RequestTag/CostCenter": "true"
        }
      }
    }
  ]
}

You can see that it's also Case Sensitive - a tag of costcenter will fail when a tag of CostCenter will work. It can also apply to an entire deploy - if you are deploying with Terraform/OpenTofu without tags, it'll deny the creation of secrets and EC2s and fail! (though using default_tags and amap of tags will help with that)

Summation

  • Make sure your pan is hot and ready - test your policies in lower environments
  • Make sure your cheese is room temperature - be patient and craft good policies, not quick policies you'll regret/roll back

Overall, SCPs are a good, powerful tool, but take time to craft them well before throwing them in front of your accounts and make sure to test them before stopping your applications!