Deploying Markdown To Confluence

00. Introduction

A couple of months ago, we open-sourced Journal, the system we use here in Duo Labs to share research notes within the team.

We made Journal because we recognized that writing and sharing research notes is a critical part of the research process. Since adopting it across our team, we've seen the quality and quantity of our research notes go up.

However, while Journal opens the door for us as a Lab to communicate our findings more regularly and effectively within the team, it also has the potential to introduce a serious problem with any research group: becoming a silo of information.

For background, most of Duo uses Confluence for sharing information. This works fine for many use cases, but it didn't fit the requirements we needed to take daily notes without breaking from our workflow. However, we feel it's important to make our findings available to the wider company. To that end, we're open-sourcing a project we built called markdown-to-confluence which is able to mirror posts we write in Journal (or any Markdown posts, for that matter) to an upstream Confluence instance.

This didn't come without challenges, so in this post we'd like to walk through how we were able to leverage the Confluence API to both let us keep the Journal experience that works for our team, while making our research more accessible to the wider company.

01. The Confluence REST API

While we're far from Confluence experts, before talking about how we handled our specific use-case, it helps to give a quick background on how the Confluence API works.

Confluence offers a REST API that exposes all the CRUD (Create, Replace, Update, Delete) operations you would expect for posts. Posts are organized into a "space", as well as a hierarchy where each post can have both parent and child posts.

There are currently two formats Confluence supports for uploading content: the XHTML "storage" format, or the traditional "wiki" format. The unique aspect of the storage format is the ability to use macros, which are Confluence-defined tags that render content in a special way, like code blocks, embedded third-party content, or user profile information.

In addition to the content itself, each post also contains labels (aka "tags"), and attachments such as images.

With this background in mind, let's talk about the details of how leverage this API to deploy Markdown posts into Confluence.

02. How We Manage Posts

The Confluence hierarchy, content structure, and associated metadata introduces some challenges, specifically with the way we generate the content and the order of operations we use to create a new post.

Generating the Post

The first hurdle is generating the post content. At first glance, it would make sense to use the wiki format, since it looks similar to Markdown. However, there are enough differences to make this translation really tricky.

Instead, we opted to use the storage format. In general, basic HTML rendered from Markdown is compatible with the storage format. To do this rendering, we leveraged the mistune Python library.

The biggest challenge we encountered was leveraging macros to support image links, code blocks, and more. For example, if we want to display a local image that we've uploaded as an attachment, we'd need to use the following markup:

<ac:image>
    <ri:attachment ri:filename="atlassian_logo.gif" />
</ac:image>

Fortunately for us, the mistune library supports creating custom renderers, allowing us to override what the output HTML looks like for a given element. We use this to return the correct macros for images and code blocks.

It's worth noting that right now markdown-to-confluence doesn't support rendering Hugo shortcodes, since this requires hooking into the lexer. We're keeping an eye out for Mistune v2 which supports plugins, addressing this problem.

Deploying the Post

The next hurdle to overcome is the actual process of uploading a post. To start, we want to update a post if it already exists, and create a new one if it doesn't. To make this work, we create a unique label for each post which is a slug of its filename plus the author name and add that label to the post. Knowing if a post exists is simple using a CQL query, but applying the label is a bit trickier.

Both the "Add Labels" and the "Create Attachments" endpoints require the caller to specify the post ID as part of the URL in the request. Normally this wouldn't be a problem—we could just upload the post content (including referring to local attachments such as images under the assumptions they'll be uploaded later), then upload the attachments and add the labels. However, our testing suggests that this isn't supported.

Instead, we create a placeholder post in order to get a post ID that we can then use to upload the attachments and apply the labels. Finally, we can upload the full post content, taking care of minor housekeeping such as updating the post version.

03. Moving Forward

While there were challenges we had to overcome to mirror our content to Confluence, it was well worth it. After all, research is only effective if it can be shared with others.

We think the goal of deploying Markdown to Confluence is a common problem, and we think that markdown-to-confluence helps solve it. Take it for a spin and, as always, don't hesitate to file an issue if you have any issues or feature requests!