Introducing CloudMapper: An AWS Visualization Tool
Today we are pleased to announce a new open-source tool from Duo Security for visualizing Amazon Web Services (AWS) cloud environments!
Duo has a number of AWS accounts run by different teams for different projects. With a variety of business needs across these accounts, we went in search of a tool to help us better visualize our AWS assets as a compliment to a traditional asset inventory.
Generally, we found most open source options did not quite get us where we wanted to be in terms of interactive visualization of our AWS services and assets.
Duo built CloudMapper to generate interactive network diagrams of AWS accounts and is releasing it as an open-source tool to the larger developer community. Check out CloudMapper on Github.
Screenshot of CloudMapper visualizing a demo account
Using CloudMapper, we can quickly answer a number of questions such as:
- Which resources are publicly exposed?
- What resources can communicate internally with which other resources?
- Do we have a robust architecture in the event of an availability zone failure?
- How many regions is this account using? How "big" is this account? How complex is it?
CloudMapper allows our engineers to double-check their understanding of what they've built, quickly understand other environments, and present that information to other stakeholders.
How it Works
There are three steps to getting up and running with CloudMapper:
- Collect information about an AWS account via a shell script that uses the AWS CLI.
- Convert that data into a format usable by the web browser.
- Run a simple web server to view the collected data in your browser.
The first step of collecting information only requires the privileges to describe and list information about an account. This can be done with the AWS SecurityAuditor policy. If you don't have direct access to the account, someone who does can run this script and send you the bundle of files it creates.
The second step of converting these cached files into something for the web browser display is where most of the logic is. This is where the Security Groups are analyzed to determine what network pathways exist, and parent/child relationships are created between nodes such as EC2 instances, and compound node structures such as subnets, availability zones, VPCs, regions, and accounts.
The final step of visualizing the data in the browser makes heavy use of cytoscape.js to perform the graph layout and allow interaction with the resources. Historically, this problem would have been solved with graphviz, but that solution is more suited toward generating static images as output. Cytoscape was originally created to visualize molecular interaction networks, but has been found to be well-suited for a variety of other network visualizations
By using cytoscape.js and presenting the diagram to the user in a web browser, you can click on nodes to get more information about them, move them around, delete them, and more. This is only for visualizing the data, so any actions you take will not impact your actual AWS environments.
You can zoom in and out, pan, save a high-definition image, or save and load the layout after you've moved nodes around. You can click on a compound node, such as a VPC, and compress it to a single node in order to simplify the visualization. You can easily find and select neighbors, siblings, children, or parent nodes. You can click on edges to get details about the Security Groups that are allowing that communication to happen.
Compressed node and highlighted neighbors
Improving the Layout
Visualizing large networks is a hard problem. CloudMapper uses the CoSE (Compound Spring Embedder) layout for Cytoscape.js that was developed by the i-Vis Lab in Bilkent University, which is regarded as one of the best algorithms for laying out graphs with compound nodes. However, any graph with a large number of nodes and edges is inherently complicated. To manage this problem, CloudMapper has a number of filtering options that can be used when preparing the data for visualization.
Options to reduce the amount of data displayed include:
- Show only specified regions.
- Ignore internal edges if you only want to see what resources are exposed publicly.
- Aggregate similar EC2 instances to a single node based on a tag name.
Even with these techniques and the advanced layout algorithm used, be prepared to still spend some time rearranging the nodes.
Stay in Touch!
At Duo, we are using CloudMapper internally for a number of purposes. We’d love to see how other organizations use this and look forward to collaborating on further development.
If you’re interested in the intersection between security and running a highly-available service on AWS, please contact Duo's Production Engineering team at prodeng-ext@duo.com.