groovecoder

tl;dr – I want to reboot MDN as a Lean Startup – a platform for web developer knowledge and community. There's a bunch of new kinds of work to do. Go to the MDN Metrics project proposal on wiki.moz, join the thread on mdn-drivers, or comment here to give feedback.

After I read Kanban and The Phoenix Project I picked up Lean Startup and Lean Analytics. I love that the Lean Startup community is inclusive of intrapreneurhsip – people making new products inside larger host companies. The “Developer Program” we've discussed at Mozilla is a perfect example of intrapreneurship, and we should treat it as a User-Generated Content startup.

We're already doing this ...

We're redesigning MDN and already launched the redesign to beta testers. It looks great - it highlights important content, improves search and discover-ability, and it's easier to read article text. We're improving the redesign with quick continuous deployments to beta users; we even pushed to production 4 times during Mozilla Summit - 2 of which were changes made by new contributors from our impromptu MDN hack night!

But ...

We need to measure

It's no secret that the redesign project has been stressful across all teams - Product, UX, Creative, Development, Content, and our community. Pressure can be healthy - constraints foster creativity, innovation, and some pretty epic hacks. But pressure without purpose is just wasteful stress. And the driving philosophy of Lean is to eliminate inefficient and wasteful activity.

Dan McKinley from Etsy gave a very insightful talk about Designing for Continuous Experimentation at Warmgun design conference. He tells the story of Etsy implementing an infinite scroll project:

So when we decided to do this we just went for it. We designed and built the feature, and then we figured we'd release it and it'd be great.

...

Eventually we came to terms with the fact that infinite scroll had made the product worse, and we had changed too many things in the process to have any clue which was the culprit.

...

So if we go back to our "product plan," we see a couple of major things wrong with it. We did a lot of work and it was pointless. A better way to have done this would have been to validate those premises ahead of time and then make the call.

(Emphasis mine) So, I'm pushing hard for MDN to validate premises before we expend time, effort, and stress on projects.

What should we measure?

MDN is site for user-generated content like Wikipedia, reddit, Twitter, Facebook, etc. Lean Analytics has an entire chapter devoted to UGC metrics for:

• Visitor Engagement - how often do our readers come back, and how long do they stay?
• Engagement Funnel and Changes - how many users register; how deeply are they involved with MDN and how do we encourage them to deeper activities?
• Notification Effectiveness - how many newsletter subscribers and twitter followers do we have? How many of them open the email or click thru to the links?
• Content Creation and Interaction - how many people tag, edit, translate, or review articles; how many comment or create demos?
• Value of Created Content - how visitors read certain topics and articles; which traffic sources bring more engaged users?
• Content Sharing & Virality - how many users tweet, like, or otherwise share articles; how do the recipients behave as a result?

For example, I've been digging into our MDN Engagement Conversion Funnel:

For me, the engagement cliff between visiting and creating an account is too huge. So that measurement tells me we should add more activity features between visiting and creating an account. Maybe something like:

If we add features, we should implement minimum viable features to validate our assumptions about the impact we want - e.g., does a sharing or voting feature actually increase account registrations and editing behavior?  So we'll also want to define "the impact we want" - higher quality content, more visitors, etc.

How should we measure?

So far I've done all this by hand, but I'm proposing that we start a project for MDN Metrics on the site itself. Something similar to the interface of SUMO's KPI dashboards, though our KPI's will be different. That way everyone in the MDN community has the access to our metrics when discussing MDN product and priorities. If you've read this far and want to give feedback, leave comments here, or join the thread on mdn-drivers.

Question or comment about this post? Tell me on GitHub.