In the current microservice landscape, there are more than enough automation tools and infrastructure as code (IaC) solutions to make deployments swift and effortless. This is great for managing and deploying a single service, but what happens when there's a need to deploy your entire product to a new environment? Specifically, how do you know which services need to be deployed and in what order?
In this post, I'll discuss how JupiterOne's DevOps team leverages JupiterOne to understand service coupling and deployment order, by mapping CodeRepo USES CodeRepo relationships. We also use an Insights dashboard to help answer questions about deployment order.
What is a Deploy Dependency?
A deploy dependency is a service whose outputs are used as inputs in another service (specifically Terraform outputs). Based on that, we know if service B must be deployed prior to service A. Ideally, we would like to see a relationship in JupiterOne between these two CodeRepo entities.
Where are Deploy Dependencies Defined?
We define these deploy dependencies in a dependencies.yaml file that exists in all of our deployable repositories.
Example dependencies.yaml for an account-service:
---
terraform:
- user-service
- invitation-service
- feature-toggle-service
Since the account-service USES the outputs of the services user-service, invitation-service, and the feature-toggle-service, we know that those must be deployed prior to the account-service itself.
By answering the questions above, we have what we need to automate the creation of CodeRepo USES CodeRepo relationships in JupiterOne.
Our team developed a script that parses the dependencies.yaml of our deployable repositories in GitHub, creates relationships based on the matching CodeRepo entities with the following properties:
const relationshipKey = `${mainRepo.entity._key}|uses|${dep.entity._key}`;
const relationship: RelationshipForSync = {
_key: relationshipKey,
_class: 'USES',
_type: 'repo_dependency',
_fromEntityId: mainRepo.entity._id,
_toEntityId: dep.entity._id,
deployRelationship: true,
};
payload.push(relationship);
Next, we do a bulk upload into JupiterOne using the Node.js Client and CLI, which is available on our GitHub.
const jobState = await J1Client.bulkUpload ({
scope: 'summary-relationships-code-repo-dependencies-workload',
relationships: payload,
});
Currently, our DevOps team runs this on a daily cadence to keep our deploy dependency relationships up to date.
The JupiterOner node.js API client wrapper and CLI utility supports uploading entities in either JSON or YAML format. For another example use case, we recommend: Using JupiterOne as a central repository for SecOps and compliance artifacts.
How to Visualize Deploy Dependencies
Now that the relationships exist in JupiterOne we can start to visualize our deploy dependencies. Here is an example JupiterOne query that shows us the deploy dependencies of the account-service example used above:
Find CodeRepo with name = 'account-service' that uses >> CodeRepo return TREE
We also created a useful dashboard to get an overview of our deployment coupling using JupiterOne Insights.
You can learn how to get started with prebuilt or customized Insights Dashboards here.
Repository Deployment Coupling
Here is the configuration we use for the first widget displayed on our DevOps team's Insights dashboard above, which creates the pie chart labeled "Repository Deployment Coupling":
Find CodeRepo with tag.AccountName="YourAccount" as f
that USES as rx CodeRepo
where rx.deployRelationship = true
return f.displayName as none, count(rx) as value
Repository Deploy Dependency Direction
And here are the configurations for the second widget, the lower bar chart labeled "Repository Deploy Dependency Direction":
Find CodeRepo with tag.AccountName='YourAccount' as f
that USES << as rxIn CodeRepo
return f.displayName as x, count(rxIn) as y
order by y DESC
Find CodeRepo with tag.AccountName='YourAccount' as f
that USES >> as rxOut CodeRepo
Return f.displayName as x, count(rxOut) as y
Order by y ASC
Future Goals: Identify Direct and Indirect Circular Dependencies
These relationships have proven to be extremely useful in determining deployment order, but there is quite a bit of extra insight we could get by iterating on these relationships. For example, we could start to identify circular dependencies (direct and indirect) and with additional logic determine a specific order for services to be deployed. A shared dashboard for common understanding of deploy dependencies has been a valuable tool for our DevOps team, so we look forward to visualizing these additional DevOps and SRE use cases.
For more use cases and insight into how JupiterOne's team uses our own solution, we recommend the JupiterOne SecOps Automation Examples repo on Github.
