How to setup AWS development accounts with centralized cost control using AWS Budgets

What are development accounts and when to use them

In our daily work, sometimes we need to investigate the possibility of introducing a new AWS service or trying out some AWS features before applying them to production resources. In such scenarios using a separate AWS account for development purposes is beneficial before using a production AWS account. We do not want people to create resources for testing in the production account, which might affect existing production resources.

Benefits of using development accounts:

Learning AWS and hands-on training
Teams can try out new services and upgrade their infrastructure skills
Quick and ready to use
Performing tests without impacting the production environment

Purpose of using separate accounts per team

When considering an organization with multiple teams, it is important to have some budget threshold to keep the costs within organization-approved limits.

Using separate accounts helps in having a separate budget for every account rather than a single team utilizing all the budgets when using a single account.

Benefits of using separate accounts:

Handle and monitor the cost of usage per team
Add budgets per account to limit usage within a threshold
Isolating resources between teams
Apply policies

Considerations when using development accounts

We have considered providing admin access to all members for their respective team accounts in our approach. This will help speed up the development process, but we still leverage policies to limit the services users can use.

Hence using development accounts properly is each member's responsibility.

Considerations for our approach:

Members have complete freedom within their account
Self-responsibility of deleting resources after the use
Restrict regions and services in which the users can create resources

Constructing development accounts using Organizations and Budgets

Before discussing the architecture of development accounts, we will talk about the separate components and how they can benefit when using development accounts.

Introducing AWS Organizations for easy management

AWS Organizations is an account management service enabling you to consolidate multiple AWS accounts into an organization you create and centrally manage.

It has two main features:

Consolidated billing to check account information of member accounts and pay for them
Apply policies to centrally control each account using Service Control Policies (SCPs) and group accounts using Organizational Units (OUs)

Using SCPs to centrally control the member accounts:

We can use the policies shown below to control what resources can be created and in which regions they can be created:

Deny resource creation in regions
- The global services list can be found at this link: https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps_examples_general.html#example-scp-deny-region
- For example, deny all regions except the Tokyo region.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DenyAllOutsideTokyo",
            "Effect": "Deny",
            "NotAction": [
                "<all-global-services>"
            ],
            "Resource": "*",
            "Condition": {
                "StringNotEquals": {
                    "aws:RequestedRegion": [
                        "ap-northeast-1"
                    ]
                }
            }
        }
    ]
}

Allowed services to be used for the account
- As needed, owners/admins can add more services when needed
- For example, allowing only ec2 and s3 services to be created in the account

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DenyAllExceptList",
            "Effect": "Deny",
            "NotAction": [
                "ec2:*",
                "s3:*"
            ],
            "Resource": "*"
        }
    ]
}

AWS Budgets for adding usage thresholds per account

You can use AWS Budgets to track and take action on your AWS costs and usage. For example, when a cost threshold is reached for an account, we can take actions like sending mail and applying deny creation SCP policy.

The issue with AWS Budgets is that, as of writing this article, nested OUs are not supported by Budget Actions. Currently, you can apply Budget Action with SCPs to only top-level OUs and not directly to accounts.

Hence every team account will have its own team OU inside the Root OU as shown below:

Root OU

Team A OU
- Team A Account
Team B OU
- Team B Account
Management Account

Using a deny creation SCP when the threshold has reached:

A creation deny SCP policy will be added to the account OU who reaches the fixed limit for their team.

Owners/admins can add the creation policies to deny by referencing the following tools:

AWS Policy Generator tool
SCP Policy Creation tool

Example SCP policy to deny creation for ec2 and s3:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyCreation",
      "Effect": "Deny",
      "Action": [
        "ec2:Run*",
        "ec2:Start*",
        "ec2:Create*",
        "s3:Create*"
      ],
      "Resource": "*"
    }
  ]
}

Combining it all together

You can use Terraform to manage the creation of the below infrastructure.

As shown above, both the Team A account and Team B account have a budget of $1000 and two budget thresholds set:

Amount reached 80% of the budgeted amount (See Team A above)
- Send a WARNING message to slack; members should delete resources in order not to reach 100% budgeted amount
Amount reached 100% of the budgeted amount (See Team B above)
- Apply deny creation SCP policy so that members cannot create new resources
- Send Deny Creation has been applied message to slack; members will need to wait until next month to try out new services

Accessing team accounts:

New accounts will be created with IAM role OrganizationAccountAccessRole, which has admin permissions and a trust relationship with the root account. This is automated when you use AWS Organizations.
IAM group will be created for every account, which will allow to AssumeRole from the root account to OrganizationAccountAccessRole of your team's account.
Since this is a development account, all users are provided admin permissions to their respective team accounts. Owners can generate passwords for their team members.
Users cannot access resources in the root account; they can only perform assume roles in their respective accounts.

Conclusion

Using development accounts improves how we confirm service utilization before applying it to production.

Using Organizations and Budgets and managing them using terraform helps us to quickly and easily manage the infrastructure and control costs.

Future points to consider:

Automate stopping of resources when budget threshold reached
- We can create Lambda functions that will stop EC2 and RDS instances when a 100% budget threshold is reached for that account.
Automate reverse budget action process
- If no remediation action is taken, the restrictive SCPs will be automatically detached at the start of the next budget period.
- We can use CLI to reverse the budget action if any account owner wants to increase the budget amount.
- We can automate the process using Lambda functions.