Lab Deployment

Lab Overview

In this lab, you will use CloudFormation to deploy

  • An AWS CloudTrail Trail. CloudTrail collects logs from AWS services when you make calls to their APIs

  • An Amazon S3 bucket to serve as the destination for CloudTrail logs

  • An Amazon CloudWatch Logs log group for CloudTrail to deliver logs

  • An Amazon Elasticsearch Service domain. CloudWatch Logs (via Lambda) sends CloudTrail’s log lines to this Amazon Elasticsearch Service domain

  • An Amazon Cognito User Pool and Identity Pool. Amazon Elasticsearch Service’s Kibana integrates with Cognito to provide a login experience for Kibana. The template also includes a custom Lambda function that creates a Cognito user and Cognito domain for serving login requests

  • [OPTIONAL] A second CloudFormation template provides a second S3 bucket and a Lambda function to write objects to that bucket. You can use this Lambda to generate S3 API calls and enable logging of those calls to generate traffic for your domain

Send CloudTrail logs to Amazon Elasticsearch Service

Deploy the first CloudFormation Stack

  1. Sign in to the AWS Management Console

  2. Select US East (N. Virginia) in the region selector

  3. In the search box, type CloudFormation and click CloudFormation.

  4. Click Create Stack (If you don’t have any stacks, click the Create New Stack).

  5. Click Specify an Amazon S3 template URL radio button and paste the following URL in the text box.

https://s3.us-east-2.amazonaws.com/search-sa-log-solutions/cloudtrail/CT-CWL-AES.json

  1. Click Next.

  2. Give your stack a Stack name. Leave the ES Version as 6.3, and set a Stack Prefix. The stack prefix can be any short string and will be used as a prefix for all of the resource names in the stack.

  3. Click Next.

  4. Leave the Options and Advanced Options at their defaults, scroll down and click Next.

  5. On the Review page, scroll down and check the box next to I acknowledge that AWS CloudFormation might create IAM resources with custom names.

  6. Click Create Stack.

  7. The stack will take about 15 minutes to deploy. Wait until CloudFormation shows the stack Status as CREATE_COMPLETE (You will need to click the refresh button on the right-hand corner to see if the status has changed)

Enable Cognito sign on for Amazon Elasticsearch Service

  1. Click Services and type Elasticsearch. Click Elasticsearch Service.

  2. Looking at the Dashboard find and click on the domain you just created, named <stack prefix>-domain to view the domain’s dashboard.

  3. Click Edit Domain.

  4. Scroll down to the Amazon Cognito authentication section, and click Enable Amazon Cognito for authentication.

  5. Leave the Region as US East (N. Virginia)

  6. In the Cognito User Pool drop down, select the user pool named <stack prefix>UserPool.

  7. In the Cognito Identity Pool drop down, select the identity pool named <stack prefix>IdentityPool.

  8. Click Submit.

  9. Your domain will enter the Processing state. This will take approximately 10 minutes to complete. Wait until the domain changes to the Active state before going on to the next section.

Start streaming logs from CloudWatch Logs to Amazon ES

  1. From the Services Dropdown, select the Cloudwatch Service.

  2. From the CloudWatch console select Logs Groups in the left-hand navigation.

  3. Locate the log group associated with the lab <stack prefix>-CloudTrailLogs.

  4. Click the radio button next to your log group (don’t click the name of the group itself).

  5. From the Actions menu, select Stream to Amazon Elasticsearch Service.

  6. Select your domain from the Amazon ES cluster menu.

  7. From the Lambda IAM Execution Role menu, select <stack prefix>-CWLRoleForStreaming.

  8. Click Next.

  9. For Log Format, select AWS CloudTrail.

  10. Leave the other fields at their defaults and click Next.

  11. On the Review page, click Next.

  12. Click Start Streaming.