Visualize

Visualize your AWS usage with a pie chart

  1. Click the Visualize tab, then Create a visualization.

  2. You can see that Elasticsearch provides a number of different visualizations for underlying data. Click the Pie chart.

  3. On the following screen, you tell Kibana which index pattern to use as a data source for the visualization. We’ve only set up one. Click cwl-*.

  4. Congratulations! You’ve made your first visualization. Unfortunately, it’s not showing anything useful at present. It’s just a pie chart of everything. To set the slices, you need to select a field that you want visualized.

  5. Under Buckets, click Split Slices. This will reveal a new menu, Aggregations. From that menu, choose Terms (You might have to scroll the menu to see it, it’s at the bottom). Click to drop down the Fields menu and the search box. Type awsRegion.keyword in the search box, and click its item in the menu. Finally, click to apply the changes.

  6. This will give you a more informative pie chart that shows the regions for the CloudTrail logging data (The Lab Accounts only have entries corresponding to us-east-1).

  7. You can experiment with adding a search term in the search bar above the visualization. For instance, to see a regional breakdown of Amazon Elasticsearch Service API calls, type eventSource:\“es.amazonaws.com\“. Your pie chart will change to show only calls matching the search term – that is, calls to Amazon Elasticsearch Service. Clear your search by changing the search text to *.

  8. You can also see a service and call breakdown by region by clicking Add sub-buckets at the bottom of the left rail (you might have to scroll down to see it).

  9. Click Add sub-buckets, then choose Split Slices again. Select Terms as the Sub aggregation and eventSource.keyword as the Field. Apply the changes to see which services you are calling in the regions displayed.

  10. Add another Terms, Sub-bucket for the userIdentity.accountId field.

  11. You can work with textual data using Terms aggregations in a number of different forms: Pie Charts, Histograms, Line Graphs, and more. Experiment with the different Fields in the domain to see what you can generate.

  12. You can also save your visualization and build it into a dashboard. Click Save at the top of the screen. Type Pie Chart by Region as the Name.

Visualize call volume by region

  1. Select Visualize in the left rail.

  2. You will see your saved visualization. Click the to create a new visualization.

  3. Scroll down, and select Visual Builder.

  4. You will see a time-based graph of all of the calls logged by CloudTrail. Drop the Group By menu and select Terms.

  5. For By, use the search box or scroll to select awsRegion.keyword. You now have a graph of calls logged by CloudTrail, by region.

  6. Click Save at the top of the screen to save your visualization. Name it Calls by region.

Add real-time updating

  1. Click at the top of the screen.

  2. Choose a Refresh Interval. 30 seconds or more are good choices.

Build a dashboard

  1. Select the Dashboard pane, and choose Create a dashboard.

  2. Click the Add button

  3. Click Calls by Region and then Pie Chart By Region.

  4. Click the Add button at the top of the screen to collapse the Add Panels pane.

  5. You can adjust the size and position of the panes. You can also Save your dashboard so that you can open it next time you use Kibana.

[OPTIONAL] Generate traffic to S3 and monitor it

For further investigation, we have also created a CloudFormation template that creates a Lambda function to put random keys to S3. You can use CloudTrail to capture and stream read and write events to this bucket. You can use Kibana to create a visualization to monitor these events. If you don’t want to follow these steps, skip down below to the Cleanup section.

  1. Navigate to the CloudFormation console (Services menu, search for and select CloudFormation).

  2. Click Create Stack

  3. Choose Specify an Amazon S3 template URL and paste the following URL in the text box

    https://s3.us-east-2.amazonaws.com/search-sa-log-solutions/cloudtrail/S3-Traffic-Gen-Lambda.json

  4. Click Next.

  5. Enter a Stack Name and a Stack Prefix (I used s3gen for both)

  6. Click Next.

  7. Scroll to the bottom of the Options page and click Next.

  8. Scroll to the bottom of the Review page, Click the check box next to I acknowledge that AWS CloudFormation might create IAM resources with custom names.

  9. Click Create Stack.

  10. Wait until the stack status is CREATE_COMPLETE. Now you need to enable logging S3 events.

  11. Navigate to the CloudTrail Console (Services menu, search for and select CloudTrail).

  12. Select Trails and find the trail generated by the first stack, above. Click the trail’s name to open its configuration page.

  13. Scroll down to the Data Events section and click Configure.

  14. Click Add S3 Bucket

  15. In the Bucket name text box, start typing your stack prefix from the second stack until you find your <stack prefix>-s3bucket-<uuid>. Select it.

  16. You can choose to enable Read or Write events or both.

  17. Click Save.

  18. Note, you can also configure Lambda data events by selecting the Lambda tab and configuring it.

  19. Navigate to the Lambda console (Services menu, search for and select Lambda)

  20. Find the function named <stack prefix>-TestFunction. Click the function’s name.

  21. Click the Test button.

  22. Give the test data an Event name. You can leave the default values. Scroll down and click Create.

  23. Click the Test button again. The Lambda function will run for 5 minutes, creating objects in the S3 bucket.

[OPTIONAL] Visualize S3 data events

  1. In this example, you’ll create a time-based line graph, scoped to a particular search query.

  2. Navigate back to Kibana. You might have to wait for the objects to arrive in your Amazon ES domain. (You can use the search bar and search for PutObject to find these events)

  3. Click Visualize and then click the to create a new visualization.

  4. Create a Line graph.

  5. Click cwl-* as the Name.

  6. Set the X-Axis to bucket by time. For Bucket, click X-Axis.

  7. For the Aggregation, select Date Histogram. You now have time buckets along the X-Axis, and the event count on the Y-Axis. This is a very common graph to create. You can change the Y-Axis settings to graph sums, averages, etc. for numeric fields.

  8. In this case, we want to graph GetObject, PutObject, and DeleteObject separately on the same graph.

  9. In the X-Axis section, click Add sub-buckets.

  10. For the Buckets type, click Split Series.

  11. For the Sub Aggregation, select Filters from the menu.

  12. In the Filter 1 box, type eventName:\“GetObject\” (with the quotation marks).

  13. Click the Add Filter button.

  14. In the Filter 2 box, type eventName:\“PutObject\”.

  15. Click the Add Filter button.

  16. In the Filter 3 box, type eventName:\“DeleteObject\”.

  17. Click to make the changes.

  18. You will not have any GetObject or DeleteObject events at this time. You can go to the S3 console and click a couple of the keys, and delete a couple of the keys to see those events