3 important things to consider when measuring your success

Photo by Thomas Leuthard.
  1. Figure out what to measure. How will you know if your open source software project is wildly successful? What will be different with the world? What problem will it have solved? If you don’t know the answers to these questions, you won’t know what to measure.
    Often when projects start out, they measure what ever is possible. For example, you start a blog and you measure web site visitors. But that in itself is not a measure of success. It might be a measure of popularity or reach, but not of success. Why’d you start the blog? Maybe you want people to know Linux is an awesome operating systems. In that case, a measure of success might be how many people use Linux. Or how many people try Linux after reading your blog. Or how many repeat visitors are now on Linux instead of an alternative operating system. Obviously, some of these things are harder to measure than others. But defaulting to number of visitors is a crutch you should only use temporarily while you figure out better metrics.
  2. Change your metrics over time. What you measure will also change over time. As your project evolves, you’ll be able to measure more sophisticated things that more accurately reflect your goal. You might start out measuring number of visitors and evolve into number of people who convert within a month of visiting your blog. Both the way you can measure things and what your project can do might change. And your goals themselves might change. Maybe you started out to create a login widget and then realized what was really needed was a user management tool. You should never be afraid to change your metrics. If you are afraid of losing data over time, you can keep measuring the old metrics.
  3. Keep it simple. It’s much better to have one or two key metrics than to worry about all the possible metrics. You can capture all the possible metrics, but as a project, you should focus on moving one. It’s also better to have a simple metric that correlates directly to something in the real world than a metric that is a complicated formula or ration between multiple things. As project members make decisions, you want them to be able to intuitively feel whether or not it will affect the project’s key metric in the right direction.

And then, make sure your metrics are working. Double check that when you are closer to your goal when your metrics get better. If not, time to figure out what else you could measure. For example, maybe you were measuring number of contributors, but signing up more contributors does not seem to be getting you closer to your goal. You might want to look at measuring monthly contributions or active contributors.

If I was going to recommend one book for getting started in thinking about metrics, it would be Lean Analytics. Thanks to Luke Crouch for the recommendation and the many good conversations.

Create better processes to avoid difficult conversations

The best way to handle difficult conversations is to prevent them from ever starting. The way to prevent them from happening (or at least to keep them less painful) is to have good governance. Governance is the solution to difficult conversations and trolls.

My motto is that you’ve done the best job you can do when you work yourself out of a job. When you have solved the problem, you eliminate the need for anyone to need to work on it again in the future. So the best solution to difficult conversations is one that means we no longer have them. We’ll still have differences of opinion and a need for conversation but hopefully it won’t happen in painful ways. In order for that to happen, you need to have clearly defined ways to bring up new ideas, discuss differences of opinions, to make decisions and to execute on them.

Why do difficult conversations happen?

You maybe tempted to say difficult conversations happen because some people are awful. But nobody really likes to go out and argue. Worst case they are lonely or feel misunderstood. However, in open source, difficult conversations usually happen because people feel passionate about something and are missing either the tools, the communication skills or the place to do something about it. Maybe a decision was made without giving people a chance to comment. Maybe they didn’t even know a decision was being made about something they thought was really important. Maybe they felt like their opinion wasn’t getting heard. Maybe they felt like everyone was rehashing the same conversation for the 100th time and nothing was getting done. Maybe they want to get something done and there’s no obvious way to sign up or just do it. Maybe they made a decision they feel applies to the whole community and nobody is listening. Maybe …

So how do you fix that?

In order to prevent difficult conversations, or to make conversations easier and more productive, you make sure your project has governance, processes and tools to let people express their ideas, debate the merits and work out the details. Sometimes you are able to create them before you have problems. Often they are created in reaction to problems. But no matter when you create them, creating good governance is an opportunity for your community to discuss and agree on culture, processes and mission.

Some of the components of governance, which may overreach into ethics and culture, are:

  • Leadership. Who leads? How are they selected? By whom? What if they do something wrong? Who succeeds them?
  • Decision making processes. When are decisions made by the leader? When are they made by a vote? What if someone doesn’t like the decision? How are things proposed? How long and where does discussion happen.
  • Codes of conduct. What behavior is expected? What behavior is inappropriate? What happens if someone misbehaves? Who decides that it was misbehavior?
  • Communication. How does it happen? Online vs in person? Who can attend? Who is expected to attend?
  • Mission & Vision. Who defines them and through what process? How can they change?
  • Money. Can your project make money? Accept money? Can individuals in your community make or accept it for work they do in relationship with your project?
  • … what else would you add?

Often governance grows like culture. Slowly, without anyone noticing. So it’s helpful to stop and take a look at what governance you have, what’s explicitly defined and what your project might need moving forward, especially if it’s growing.

For an example of how governance and cultures evolve in communities, you can follow along as Fedora redefines its mission, Gluster decides who is a maintainer and Drupal evolves its governance.

Can your community get too big?

Open source software communities, like companies and cities, can come in all sizes. They don’t get too big but they can grow faster than their infrastructure and processes.

Take for comparison, the size of companies and cities. Companies come in sizes from one person LLCs to 100,000 employee megaliths. Cities come in sizes from a single person farm to enormous cities with 10’s of millions. Are some big companies too big for some people? Absolutely. Are big companies good at getting certain types of tasks done? Yes. Some people prefer some companies and small towns (maybe not the same set of people) and some people prefer big cities and large companies (again, maybe not the same set of people.)

So, can open source software communities get too big? Not really. But they can grow too fast. They can grow faster than their infrastructure; they can outgrow their governance; they can leave their mission behind.

So how can a community grow productively? By considering things like:

  • Onboarding processes. I’ve actually heard of communities having so many new people show up that to answer all their questions would have meant nobody would have time to get work done!
  • Infrastructure that scales. Where are you going to put all those new people? Will your source code control system, your chat system, and your blogging system scale?
  • Governance. As you have more people involved, you may want different ways of deciding membership and making decisions. OSI moved to individual membership from a self elected board. Gluster is moving to a new maintainer model as it grows. The evolution of process takes time and thought.
  • Mission statements that evolve. Sometimes you accomplish your original mission and you’ve moved on to something else and you need to state that for new and existing contributors like Fedora is doing now.

What else needs to grow along with your community?

Part of the OSS Communities series — maintaining existing community. Contribute your stories and tag them on Twitter with #osscommunities to be included.