Sleuth wants to use AI to measure software developer productivity

As knowledge workers including software engineers shifted to remote work during the pandemic, executives expressed a concern that productivity would suffer as a result. The evidence is mixed on this, but in the software industry particularly, remote work exacerbated many of the challenges that employees already faced. According to a 2021 Garden survey, the majority of developers found slow feedback loops during the software development process to be a source of frustration, second only to difficult communication between teams and functional groups. Seventy-five percent said the time they spend on specific tasks is time wasted, suggesting it could be put to more strategic use.

In search of a solution to bolster developer productivity, three former Atlassian employees — Dylan Etkin, Michael Knighten and Don Brown — cofounded Sleuth, a tool that integrates with existing software development toolchains to provide insights to measure efficiency. Sleuth today announced that it raised $22 in Series A funding led by Felicis with participation from Menlo Ventures and CRV, which CEO Etkin says will be put toward product development and expanding Sleuth’s workforce (specifically the engineering and sales teams).

“With the avalanche of remote work brought on by the pandemic the need for developers, managers and executives to understand and communicate about engineering efficiency has increased sharply,” Etkin told TechCrunch via email. “Developers, no longer in the same room, need a way to coordinate around deploys and a quick way to discover when a deploy has gone wrong. Managers need an unobtrusive way to proactively learn about bottlenecks affecting their teams. Executives need an unobtrusive way to understand the impact of their organization-wide initiatives and investments. Sleuth takes the burden of understanding and communicating engineering efficiency off-line and makes it digestible by all.”

Etkin, Knighten and Brown were colleagues Atlassian, where they claim that they helped the company’s engineering organizations move from releasing software every nine months to releasing daily. Etkin was an architect on the Jira team before becoming the development manager at Bitbucket and StatusPage, while Knighten and Brown were a VP of product and an architect/team lead, respectively.

While at Atlassian, which grew from 50 to over 5,000 employees in the time that Sleuth’s cofounders worked there, Etkin says it became “crystal clear” that many engineering teams lack a quantitative way of measuring efficiency — and that this gap could hold them back from growing and improving.

“Measuring engineering efficiency is a known, large and growing problem that’s now become solvable. Because every company is investing more heavily into software engineering, the need for visibility into engineering efficiency has intensified,” Etkin said. “However, measuring efficiency has historically been very challenging for a multitude of reasons, namely tooling complexity, lack of access to data and use of dubious proxy metrics that bred micromanagement and distrust.”

Sleuth’s solution is DevOps Research and Assessment (DORA) metrics, an emerging standard used by developer teams to measure how long it takes to deploy code, the average time for a service to bounce back from failures, and the how often a team’s fixes lead to issues post-deployment. DORA arose from an academic research team at Google, which between 2013 and 2017 surveyed over 31,000 engineers on DevOps practices to identify the key differentiators between “low performers” and “elite performers.”

Sleuth isn’t the only platform that uses DORA metrics to quantify productivity. LinearB, Jellyfish and Athenian are among the rival solutions that have adopted the DORA standard. But Etkin claims that its competitors don’t “fully or accurately” track these metrics.

“Sleuth is unique … because we employ deployment tracking to model how engineers are shipping their work from concept through to launch,” he explained. “Accurately modeling exactly how engineers ship across their pre-production and production environments and how they interact with issue trackers, CI/CD, error trackers and metrics allows Sleuth to build a fully automated … view of a team’s DORA metrics and their engineering efficiency.”

Sleuth uses AI to attempt to figure out a team’s baseline change failure rate (i.e., the percentage of changes that resulted in degraded services) and mean time to recovery — two of the four DORA metrics — from existing systems such as Datadog and Sentry. The platform can automatically determine when a metric is outside that baseline, Etkin says, and even automate steps in the development process to potentially improve on the metric.

From Sleuth’s project dashboard, individual teams can track their DORA metrics. An organization-wide dashboard reveals trends across different projects and teams.

“Customers just point Sleuth at at … error data and Sleuth lets engineers know when they’ve pushed these metrics into a failure range. Using AI to determine these values means engineers can focus on their work without needing to understand every metric in their system or what ‘normal’ looks like for each.”

Tracking DORA metrics with Sleuth.

DORA metrics aren’t the end-all be-all, of course. They can be a hindrance when an organization’s focus on them becomes all-consuming. As Sagar Bhujbal, VP of technology at Macmillan Learning, told InfoWorld in a recent piece: “Developer productivity should not be measured by the number of errors, delayed delivery or incidents. It causes unneeded angst with development teams that are always under pressure to deliver more capabilities faster and better.”

Etkin agrees, emphasizing that engineering managers need to avoid the temptation to micromanage.

“Engineering is a creative endeavor, and engineers are more similar to artists than assembly line workers,” Etkin said. Engineering managers need to … track the right metrics [and] track them accurately [but also] give engineers the tools they need to improve on the metrics.”

Sleuth customers vary from enterprises like Atlassian to startups including Launchdarkly, Puma, Matillion and Monte Carlo. Etkin says that the platform has tracked nearly a million deploys and undertaken over a million automated actions on behalf of developers. He declined to reveal revenue numbers when asked, but said that 12-employee Sleuth has grown 700% last year with a “very healthy” margin and cash flow.