- Figure out what to measure. How will you know if your open source software project is wildly successful? What will be different with the world? What problem will it have solved? If you don’t know the answers to these questions, you won’t know what to measure.
Often when projects start out, they measure what ever is possible. For example, you start a blog and you measure web site visitors. But that in itself is not a measure of success. It might be a measure of popularity or reach, but not of success. Why’d you start the blog? Maybe you want people to know Linux is an awesome operating systems. In that case, a measure of success might be how many people use Linux. Or how many people try Linux after reading your blog. Or how many repeat visitors are now on Linux instead of an alternative operating system. Obviously, some of these things are harder to measure than others. But defaulting to number of visitors is a crutch you should only use temporarily while you figure out better metrics. - Change your metrics over time. What you measure will also change over time. As your project evolves, you’ll be able to measure more sophisticated things that more accurately reflect your goal. You might start out measuring number of visitors and evolve into number of people who convert within a month of visiting your blog. Both the way you can measure things and what your project can do might change. And your goals themselves might change. Maybe you started out to create a login widget and then realized what was really needed was a user management tool. You should never be afraid to change your metrics. If you are afraid of losing data over time, you can keep measuring the old metrics.
- Keep it simple. It’s much better to have one or two key metrics than to worry about all the possible metrics. You can capture all the possible metrics, but as a project, you should focus on moving one. It’s also better to have a simple metric that correlates directly to something in the real world than a metric that is a complicated formula or ration between multiple things. As project members make decisions, you want them to be able to intuitively feel whether or not it will affect the project’s key metric in the right direction.
And then, make sure your metrics are working. Double check that when you are closer to your goal when your metrics get better. If not, time to figure out what else you could measure. For example, maybe you were measuring number of contributors, but signing up more contributors does not seem to be getting you closer to your goal. You might want to look at measuring monthly contributions or active contributors.
If I was going to recommend one book for getting started in thinking about metrics, it would be Lean Analytics. Thanks to Luke Crouch for the recommendation and the many good conversations.