Table of Contents

Overview
The Goal
The Architecture
Pre-Requisites
Resources
Import Resources into a CloudFormation Stack
Getting Started
Create the Change Set using the AWS CLI
Create Target Template State
Generate the Change Resources Object
Create the Import Change Set:
Execute the Change Set
Create the Remaining Resources
AWS::S3::BucketPolicy
AWS::Route53::RecordSetGroup
Project setup
Run the Deployment to Re-Create Deleted Resources
What's next?
Appendix

Overview

The Goal

Following on from part 2 and 3A in this series, this post aims to convert the existing architecture into template that will simplify the process by creating and configuring the resources. This was already achieved by part 3A for only new resources. This post covers creating CloudFormation change sets in order to import existing resources to avoid deleting and recreating existing resources.

The Architecture

A reminder of the architecture from Part 2: Static Site CloudFront Distribution

Pre-Requisites

Completed Part 2 of this series successfully
This post is not required if Part 3A of this series has been successfully completed and deployed for the target environment

Resources

Import Resources into a CloudFormation Stack

Getting Started

In Part 3A, an entirely new stack was deployed for the dev environment which utilize dev.slsdeploy.com. In Part 2, resources were created manually for the production slsdeploy.com and is currently live. If we deploy the template derived from Part 3A, the deployment will fail (see below) with an error message Error: CREATE_FAILED: WebAppS3Bucket (AWS::S3::Bucket) slsdeploy.com already exists. In addition, we may failures for other conflicting resources or duplicate and unnecessary resources being created.

$ sls deploy --region ap-southeast-2 --stage prod --verbose

Deploying sls-deploy to stage prod (ap-southeast-2)

Packaging
Retrieving CloudFormation stack
Creating CloudFormation stack
Creating new change set
Waiting for new change set to be created
Change Set did not reach desired state, retrying
Executing created change set
  CREATE_IN_PROGRESS - AWS::CloudFormation::Stack - sls-deploy-prod
  CREATE_IN_PROGRESS - AWS::S3::Bucket - ServerlessDeploymentBucket
  CREATE_IN_PROGRESS - AWS::S3::Bucket - ServerlessDeploymentBucket
  CREATE_COMPLETE - AWS::S3::Bucket - ServerlessDeploymentBucket
  CREATE_IN_PROGRESS - AWS::S3::BucketPolicy - ServerlessDeploymentBucketPolicy
  CREATE_IN_PROGRESS - AWS::S3::BucketPolicy - ServerlessDeploymentBucketPolicy
  CREATE_COMPLETE - AWS::S3::BucketPolicy - ServerlessDeploymentBucketPolicy
  CREATE_COMPLETE - AWS::CloudFormation::Stack - sls-deploy-prod
Uploading
Uploading CloudFormation file to S3
Uploading State file to S3
Updating CloudFormation stack
Creating new change set
Waiting for new change set to be created
Change Set did not reach desired state, retrying
Executing created change set
  UPDATE_IN_PROGRESS - AWS::CloudFormation::Stack - sls-deploy-prod
  CREATE_IN_PROGRESS - AWS::CloudFront::OriginAccessControl - CloudFrontOAC
  CREATE_IN_PROGRESS - AWS::S3::Bucket - WebAppS3Bucket
  CREATE_FAILED - AWS::S3::Bucket - WebAppS3Bucket
  CREATE_FAILED - AWS::CloudFront::OriginAccessControl - CloudFrontOAC
  UPDATE_ROLLBACK_IN_PROGRESS - AWS::CloudFormation::Stack - sls-deploy-prod
  UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - sls-deploy-prod
  DELETE_COMPLETE - AWS::S3::Bucket - WebAppS3Bucket
  DELETE_IN_PROGRESS - AWS::CloudFront::OriginAccessControl - CloudFrontOAC
  DELETE_COMPLETE - AWS::CloudFront::OriginAccessControl - CloudFrontOAC
  UPDATE_ROLLBACK_COMPLETE - AWS::CloudFormation::Stack - sls-deploy-prod

✖ Stack sls-deploy-prod failed to deploy (84s)
Environment: darwin, node 18.12.1, framework 3.23.0, plugin 6.2.2, SDK 4.3.2
Credentials: Local, "default" profile
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
CREATE_FAILED: WebAppS3Bucket (AWS::S3::Bucket)
slsdeploy.com already exists

View the full error: https://ap-southeast....

Looking at the CloudFormation portal in the management console, a Stack for the prod account was created but is in a failure state. Any resources, other than the deployment bucket and policy, are rolled back to their previous state as the deployment is baulked.

Create the Change Set using the AWS CLI

The resources created as part of the manual deployment are listed below. A full list of resources that can be imported can be found here.

Resource	Can be imported?
S3 Bucket	Yes
S3 Bucket Policy	No
CloudFront Distribution	Yes
CloudFront Origin Access Control (OAC)	Yes (?)
Route 53 Record Set Group	No

(?)Note: Although not specified at the time of writing in the supported resource list, this has worked in the below example

Note: When managing the OAC and the DNS records via CloudFormation, deleting the stack may not result in these resources being deleted with the stack. If recreating after deleting the stack, this will result in a conflict and these resources will need to be manually deleted/imported before redeploy.

Each time a resource is updated or added in CloudFormation deployment, the changes are tracked within in the CloudFormation templates in the .serverless/ directory. The goal is to bring all the resources into to these templates by creating a change set. Once the resources have been imported, the serverless.yml resource template created in Part 3A can be used to manage the stack.

Create Target Template State

If the target stack doesn't exist in CloudFormation, it needs to be created. This post follows the process of importing the existing stacks from a failed deployment. The target template needs to include the BOTH the existing resources and imported resources. This example will not include any resources that have not been created in this section.

Locate and download the template from the deployment S3 bucket. The easiest way to find this is in the resources tab of the CloudFormation deployment
Edit the template to remove the resources which cannot be imported: DnsRecordSet, WebAppS3BucketPolicy, and CloudFrontOAC.
This should leave the WebAppS3Bucket, CloudFrontOAC, and WebAppCloudFrontDistribution resources which will be imported and into the stack with the existing resources (ServerlessDeploymentBucket and ServerlessDeploymentBucketPolicy)
Add the deletion policy for each imported resource. In the example Retain was chose as these had existed outside of the stack and are production resources and should be retained incase the stack is deleted. Ideally, the production stack will be in a different account with additional restrictions and mechanisms in place to avoid accidental updates or deletes to the production resources.

An example target template can be viewed below.

// TargetTemplate.json
{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "TheAWSCloudFormationtemplateforthisServerlessapplication",
    "Resources": {
        "ServerlessDeploymentBucket": {
            "Type": "AWS::S3::Bucket",
            "Properties": {
                ...
            }
        },
        "ServerlessDeploymentBucketPolicy": {
            "Type": "AWS::S3::BucketPolicy",
            "Properties": {
                ...
            }
        },
        "WebAppS3Bucket": {
            "Type": "AWS::S3::Bucket",
            "DeletionPolicy": "Retain",
            "Properties": {
                ...
            }
        },
        "CloudFrontOAC": {
            "Type": "AWS::CloudFront::OriginAccessControl",
            "DeletionPolicy": "Retain",
            "Properties": {
                ...
            }
        },
        "WebAppCloudFrontDistribution": {
            "Type": "AWS::CloudFront::Distribution",
            "DeletionPolicy": "Retain",
            "Properties": {
                ...
            }
        }
    },
    "Outputs": {
        "ServerlessDeploymentBucketName": {
            "Value": {
                "Ref": "ServerlessDeploymentBucket"
            },
            "Export": {
                "Name": "sls-sls-deploy-prod-ServerlessDeploymentBucketName"
            }
        }
    }
}

Generate the Change Resources Object

Along with the target template, the import object needs to be provided to point the target template to the correct resources.

// ResourcesToImport.json
[
    {
        "ResourceType": "AWS::S3::Bucket",
        "LogicalResourceId": "WebAppS3Bucket",
        "ResourceIdentifier": {
            "BucketName": "slsdeploy.com"
        }
    },
    {
        "ResourceType": "AWS::CloudFront::Distribution",
        "LogicalResourceId": "WebAppCloudFrontDistribution",
        "ResourceIdentifier": {
            "Id": "UNIQUE_IDENTIFIER"
        }
    },
    {
        "ResourceType": "AWS::CloudFront::OriginAccessControl",
        "LogicalResourceId": "CloudFormationOAC",
        "ResourceIdentifier": {
            "Id": "UNIQUE_IDENTIFIER"
        }
    },
]

Create the Import Change Set:

Note:: The values for FORMATTED_IMPORT_OBJECT and TEMPLATE_BODY are surrounded by double and single quotes respectively

$ aws cloudformation create-change-set \
  --stack-name sls-deploy-prod --change-set-name ImportChangeSet \
  --change-set-type IMPORT \
  --resources-to-import file://path/to/ResourcesToImport.json \
  --template-body file://path/to/TargetTemplate.json

Once created, the change set can be viewed in the CloudFormation portal. Any errors in the creation will also be displayed here. CloudFormation Change Set

Execute the Change Set

aws cloudformation execute-change-set --change-set-name ImportChangeSet --stack-name sls-deploy-prod

The progress of the change set can be viewed in the CloudFormation Events tab:

Once the change set has successfully completed, the resources will now be located within the stack: CloudFormation Imported Resources

Create the Remaining Resources

The WebAppS3BucketPolicy of type AWS::S3::BucketPolicy and DnsRecordSet of type AWS::Route53::RecordSetGroup are not supported resources for the import and need to be handled separately.

Note: Performing any of the below steps will result in downtime until the serverless deployment is able to recreate these resources. It may be a better option to remove the bucket policy and dns routing from the serverless.yml template entirely but this would require updating/adding these manually for each new environment / stack. Especially since importing manually created resources will only occur once, it was chosen to delete and recreate these resources at a trade off of 1-2 minutes of downtime.

`AWS::S3::BucketPolicy`

The WebAppS3BucketPolicy can simple be deleted from the bucket. This can be done from the permissions tab in the S3 Management Console or via the CLI. Once deleted, running the sls deploy ... command will create the new policy which can be handled within the change set.

`AWS::Route53::RecordSetGroup`

Likewise, the DnsRecordSet cannot be imported and will need to be deleted to allow the serverless deployment to recreate these records.

Project setup

This example uses complete serverless.yml and .config/env.json files from Part 3A. These can be found in the [Appendix](# Appendix) section below. Edit the files as directed in the comments. The final file structure is:

.
├── .config
│   └── env.json
└── serverless.yml

Run the Deployment to Re-Create Deleted Resources

$ sls deploy --region ap-southeast-2 --stage prod --verbose

Deploying sls-deploy to stage prod (ap-southeast-2)

Packaging
Retrieving CloudFormation stack
Uploading
Uploading CloudFormation file to S3
Uploading State file to S3
Updating CloudFormation stack
Creating new change set
Waiting for new change set to be created
Change Set did not reach desired state, retrying
Executing created change set
  UPDATE_IN_PROGRESS - AWS::CloudFormation::Stack - sls-deploy-prod
  UPDATE_COMPLETE - AWS::S3::Bucket - WebAppS3Bucket
  UPDATE_COMPLETE - AWS::CloudFront::Distribution - WebAppCloudFrontDistribution
  CREATE_IN_PROGRESS - AWS::S3::BucketPolicy - WebAppS3BucketPolicy
  CREATE_IN_PROGRESS - AWS::Route53::RecordSetGroup - DnsRecordSet
  CREATE_IN_PROGRESS - AWS::S3::BucketPolicy - WebAppS3BucketPolicy
  CREATE_COMPLETE - AWS::S3::BucketPolicy - WebAppS3BucketPolicy
  CREATE_IN_PROGRESS - AWS::Route53::RecordSetGroup - DnsRecordSet
  CREATE_COMPLETE - AWS::Route53::RecordSetGroup - DnsRecordSet
  UPDATE_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - sls-deploy-prod
  UPDATE_COMPLETE - AWS::CloudFormation::Stack - sls-deploy-prod
Retrieving CloudFormation stack
Removing old service artifacts from S3

✔ Service deployed to stack sls-deploy-prod (65s)


Stack Outputs:
  ServerlessDeploymentBucketName: sls-deploy-prod-serverlessdeploymentbucket-UNIQUE_IDENTIFIER

What's next?

Our AWS infrastructure has now been converted to into a CloudFormation template and can be managed directly this way. In the next post we will turn out attention to the deployment pipelines to automate the deployment of our resources based on specific triggers (ie update in GitHub repo).

Appendix

// .config/env.json
{
    "dev": {
        "domainName": "dev.example.com",
        "hostedZoneId": "HOSTED_ZONE_ID", // Found in: Route 53 > Hosted Zones > example.com
        "hostedZone": "example.com",
        "s3Bucket": "dev.example.com",
        "certificateId": "CERTIFICATE_ID",
        "aliases": [
            "dev.example.com" // Note difference between dev an prod aliases. Aliases cannot be shared by multiple different distributions
        ]
    },
    "prod": {
        "domainName": "example.com",
        "hostedZoneId": "HOSTED_ZONE_ID", // Found in Route 53 > Hosted Zones > example.com
        "hostedZone": "example.com",
        "s3Bucket": "example.com",
        "certificateId": "CERTIFICATE_ID", // Found in: AWS Certificate Manager > Certificates
        "aliases": [
            "example.com",
            "www.example.com",
        ]
    }
}

# serverless.yml
service: sls-deploy # replace with service name

frameworkVersion: '3'

provider:
  name: aws
  runtime: python3.8

custom:
  env: ${file(./.config/env.json):${opt:stage, 'dev'}}
  domainName: ${self:custom.env.domainName}
  hostedZoneId: ${self:custom.env.hostedZoneId}
  hostedZone: ${self:custom.env.hostedZone}
  s3Bucket: ${self:custom.env.s3Bucket}
  certificateId: ${self:custom.env.certificateId}
  aliases: ${self:custom.env.aliases}

resources:
  Resources:
    WebAppS3Bucket:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: ${self:custom.s3Bucket}
        AccessControl: Private
        WebsiteConfiguration:
          IndexDocument: index.html
          ErrorDocument: 404.html

    CloudFrontOAC:
      Type: AWS::CloudFront::OriginAccessControl
      DeletionPolicy: Retain
      Properties:
        OriginAccessControlConfig:
          Description: 'access-identity-${self:custom.s3Bucket}.s3.${aws:region}.amazonaws.com'
          Name: ${self:custom.s3Bucket}
          OriginAccessControlOriginType: s3
          SigningBehavior: always
          SigningProtocol: sigv4

    WebAppS3BucketPolicy:
      Type: AWS::S3::BucketPolicy
      Properties:
        Bucket:
          Ref: WebAppS3Bucket
        PolicyDocument:
          Statement:
            - Sid: AllowCloudFrontServicePrincipal
              Effect: Allow
              Principal:
                Service: 'cloudfront.amazonaws.com'
              Resource: arn:aws:s3:::${self:custom.s3Bucket}/*
              Action: s3:GetObject
              Condition:
                StringEquals:
                  "AWS:SourceArn": !Sub
                      - 'arn:aws:cloudfront::${AWS::AccountId}:distribution/${DistributionId}'
                      - { DistributionId: !Ref WebAppCloudFrontDistribution }

    # DOCS https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-cloudfront-distribution-distributionconfig.html
    WebAppCloudFrontDistribution:
      Type: AWS::CloudFront::Distribution
      Properties:
        DistributionConfig:
          Origins:
            - DomainName: '${self:custom.s3Bucket}.s3.${aws:region}.amazonaws.com'
              Id: WebAppS3Origin
              S3OriginConfig:
                OriginAccessIdentity: ''
              OriginAccessControlId: !GetAtt CloudFrontOAC.Id
          Enabled: 'true'
          Aliases: ${self:custom.aliases}
          DefaultRootObject: index.html
          DefaultCacheBehavior:
            AllowedMethods:
              - GET
              - HEAD
            CachePolicyId: 658327ea-f89d-4fab-a63d-7e88639e58f6 # Id for the AWS managed CachingOptimized policy
            TargetOriginId: WebAppS3Origin
            ForwardedValues:
              QueryString: 'false'
              Cookies:
                Forward: none
            ViewerProtocolPolicy: redirect-to-https
          HttpVersion: http2
          IPV6Enabled: 'true'
          ViewerCertificate:
            AcmCertificateArn: !Sub 'arn:aws:acm:us-east-1:${AWS::AccountId}:certificate/${self:custom.certificateId}'
            MinimumProtocolVersion: TLSv1.2_2021
            SslSupportMethod: sni-only
          CustomErrorResponses:
            - ErrorCode: 403
              ResponseCode: 404
              ResponsePagePath: /404.html
    
    # DOC: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-route53-recordset.html#cfn-route53-recordset-ttl
    DnsRecordSet:
      Type: AWS::Route53::RecordSetGroup
      DeletionPolicy: Retain
      Properties:
        HostedZoneName: '${self:custom.hostedZone}.' # Note the trailing '.'
        RecordSets:
          - Name: '${self:custom.domainName}'
            Type: A
            AliasTarget:
              DNSName: !GetAtt WebAppCloudFrontDistribution.DomainName
              HostedZoneId: Z2FDTNDATAQYW2 # Do not change. Always the hosted zone ID for alias records that routes traffic to a CloudFront distribution
          - Name: '${self:custom.domainName}'
            Type: AAAA
            AliasTarget:
              DNSName: !GetAtt WebAppCloudFrontDistribution.DomainName
              HostedZoneId: Z2FDTNDATAQYW2 # Do not change. Always the hosted zone ID for alias records that routes traffic to a CloudFront distribution
    
        # Uncomment for production deployment or configure manually in the Route 53 portal
          - Name: 'www.${self:custom.domainName}'
            Type: CNAME
            TTL : "300"
            ResourceRecords: 
              - '${self:custom.domainName}'

Import Resources into Existing CloudFormation Stack

Part 3B of "Deploy Static Website To AWS"