Measuring Developer Productivity

If you do a Google search for "measuring developer productivity" you will hit a long list of articles and posts about the "right way" to do this.  You will get such gems as this from Quora...

Smell their code. Read the comments. Observe the respect levels between them. Look for who wrote the code that other devs like to work on and code they're scared of.

I don't know about you, but this does not sit well with my engineering slanted brain. I want some sort of chart or statistic that can help me read what is going on with the engineering team.  Some other metrics that are mentioned in other posts that I found interesting...

• Active Days: How many days of the week are developers committing code?
• And that's about it... no seriously.

So what is productivity anyways? At the highest level, a company's productivity is simply revenue divided by number of employees.  That's it.  How much output can the company produce with the number of employees it has.  The market is deciding how much value is created with the resources at the company's disposal.

Drilling down to an engineering team, the "value" is the hard part to measure.  If the team cranks out features and products that no one wants, then was it not productive?  I think you have to extract out the variable of the market dynamics since an engineering team will usually build in the direction being set by product managers and the company in general.  Once this is pulled out, you should be able to target if the team is performing or not against the given goals.

I am still experimenting and trying to figure out how to measure, but here are things that I've started tracking for our dev team:

• Pull Request Merges
• Story Points
• Bug Counts

Pull Request Merges

On our team a pull request must be a "complete" piece of work that can be merged into the main branch to be deployed into production.  It must be peer reviewed by at least one other engineer and it must not break unit tests or hourly automated tests.

For the last few months this is one of the metrics I've been tracking for the team.  I think it is a simple heuristic for how much work the team is completing.  Anecdotally it does correlate with perceived effort by the team.  Around vacations this number goes down, and during crunch time it goes up.

This is an easily game-able statistic though... and I like it!  I want developers to make smaller logic merges and I want them to do it more often.  Smaller changes are easier to review, unit test, and integration test to verify if it is working.

My other favorite thing to bring up about gaming developers to making smaller and more frequent code changes relates to the idea of Quantity over Quality.  It turns out, if you just get people to crank out work and focus on just churning it out, they will end up with better end results than people who take their time trying to be super precise.  The more iterations of creating something allows you to learn faster and incorporate that feedback.  

I like this metric overall to give me a trend of the velocity of the team.

Story Points

OK, so everyone hates story points, including me.  No one wants to do task breakdowns or estimate work effort against some nebulous amount of work (well please message me if you like this stuff... Do I have a great job for you!).

It turns out though, that there's always someone who wants to know "How long will X take?".  So far I haven't found any reliable way to give an estimate for length of time for work other than story points.  So this is another "reasonable" heuristic for determining how much work is being completed, but the only negative is that you have to be diligent in estimating and also tracking (it isn't a byproduct like "number of PRs merged" that you can just look up after the fact).

Story points are also easily game-able, but I don't think there are any positive side effects of having developers try to game them.  Instead, you can try to do group point estimation to mitigate individual developers from inflating numbers, but the whole team might join in on the fun.  

If you told my team we would all go to Disney World at the end of the quarter if we completed 1,000,000 story points, I would already be ordering my Mickey hat for the trip.

Bug Counts

Counting bugs is is the most negative thing that we are tracking.  I've been breaking it down to both number of bugs fixed and total number of bugs reported.  This allows us to not only see how many bugs are getting fixed, but to track the velocity of new bugs being reported.  If the two numbers don't track, it means the code is getting out of the dev teams control and we should expect a poor user experience.

I can think of both positive and negative aspects of tracking bug counts for the team as well.  If the team shies away from creating bugs since it makes us look bad, then we're just sweeping problems under the rug.  However, if the team starts closing a lot of small bugs to get the fixed numbers to go up, that would be a good thing.  Less small bugs means a higher quality product.

Overall, I really see bug count tracking as just a sanity check number to watch.  Is it slowly drifting up over time and the product getting worse?  Are specific people on the team introducing more bugs than others and a more robust review process should be used?  These are the types of questions it could possibly answer.

The Subjective

While there isn't a seemingly "one size fits all" approach for measuring a dev team, I think it does provide value to monitor and watch the statistics that the team feels is important.  Not only can it give you a sense of whether the team is firing on all cylinders, but it can also provide guidance to the individuals on what metrics are important (just watch out for gaming these).  However, there has to be some subjective aspect to performance for individuals, since not everything can be measured like our engineering brains want.