Download presentation

The software engineering process have to be continuously improving to account for new people, company growth and the production excellence. It’s hard to measure and assess. There are a number of approaches and most of them are either subjective or have a little correspondence to reality or both. In my talk I will describe how we approach to measuring performance of the engineering as a part of the Lean Production process for a SAAS company (Bookmate).

The software engineering process have to be continuously improving to account for new people, company growth and the production excellence. It’s hard to measure and assess. There are a number of approaches and most of them are either subjective or have a little correspondence to reality or both.

In my talk I will describe how we approach to measuring performance of the engineering as a part of the Lean Production process for a SAAS company (Bookmate). It is based on a ratio analysis that allows for end-to-end decomposing of overall Productivity into accuracy of estimation, ability to ship, ability to control the process and utilisation of resources.

Currently there’re a number of metrics and KPI used for agile processes:

• Completed stories of the committed stories (actual stories completed / stories committed)
• Technical debt management (backlog growth)
• Team Velocity (effort on stories this sprint)
• Acceleration (effort on stories completed this sprint to previous sprint)
• Story Cycle Time (average days from committed to done)
• CTA / QA Cycle Time (everage days from done to accepted by customer or QA)
• estimation accuracy (ratio of actual and planned effort of the completed stories)
• defects per the release cycle
• defects per story point
• number of tests run per release cycle

Almost all of these metrics are bound to to the size of stories or actual value of the story point.

However, they may vary over time and across teams. Also, these metrics don’t form a KPI tree, they are constructed ad-hoc based on particular problems in particular teams.

We propose a system of KPI for lean development of SAAS, when the main objective is to conduct the maximum number of experiments per iteration or sprint:

They form a KPI tree with productivity in its root (estimated effort on the planned and completed stories / potential team resources). This KPI shows how the team is able to perform the planned experiments in time.

Then we the productivity is decomposed into the product of the the following ratios:

1. Estimation accuracy (estimated effort on planned and completed stories / actual effort on planned and completed stories)
2. Ability to ship (actual effort on planned and completed stories / amount of resources spent in the sprint on planned stories)
3. Ability to plan (the amount of resources spent in the sprint on the planned stories / total amount of resources spent during the sprint)
4. Utilization (the total amount of resources spent during the sprint /  potential team capability team)

The advantages of the proposed KPI:

• They are invariant to the type or actual value of the story point
• They are invariant to the average size of stories
• This, in turn, allows for comparing team dynamics over time and accross different teams
• The main KPI (productivity) is fully decomposed into second-order metrics, to enable convenient and unambiguous identification of the process bottlenecks.

Konstantin Savenkov

Chief Operating Officer, Bookmate

Graduated MSU in 2003, got PhD in 2008. In 2010 joined Zvooq as Head of R&D, then became a deputy CTO and finally COO of Dream Industries holding, which consists of Zvooq, Bookmate, Theory&Practice, Exchang.es, DI Telegraph and Unisound. Starting from 2015, I am fully focused on Bookmate, social reading service with subscription, which currently expands internationally.