MDN Agile and status
tl;dr: We're settling into an Agile process for MDN with daily standups and weekly planning. In the last few months we've added and enhanced developer profiles, events, demos dev derby, and other MDN features. We've also restored, added, and enhanced drafts, review flags, search, article properties, section editing, and authentication in the new wiki. I'll try to write status posts more frequently - each time we make a release, if possible.
Agile
Since my last MDN & Kuma progress post, we've settled into a regular, yet customized, Agile methodology in the MDN team with our agile Bugzilla JS tool. We've pushed MDN 0.9.8, 0.9.9, 1.0, 1.1, 1.2, 1.3, 1.4, and 1.5 thru it relatively smoothly. We've learned and adapted a few techniques in our Agile process worth noting:
Daily standups on IRC
- What have you done since we last met?
- What do you plan to do until we meet next?
- Are you blocked by anything?
By far the 3rd question here is the most important. The first questions are mostly intended to lead into the 3rd. Our scrum-master (now me) is tasked with removing any and all blockers as much as possible - usually chasing down people in other groups from whom we need something (IT, DevEngage, Legal, Security, etc.)
(Bi-)Weekly Retro & Planning Poker
We push code on Tuesdays, so we alternate between post-/pre- and mid-sprint planning meetings. Mid-sprint planning meetings are shorter and faster, little more than a standup. We use an etherpad template for a running minutes of our weekly Wednesday meetings. The basic format is:
- Retro - what's go[(ing)|(ne)] good or bad in the sprint?
- Discuss - general items to discuss, usually flowing from Retro
- Planning (post/pre-sprint only) - re-assess bugs carried over from previous sprint, use planningpoker.com to estimate the current sprint backlog in bugzilla
Status
We're developing major features for both MDN and the upcoming wiki.
MDN
- Developer profiles
My MDN profile is missing demos because I haven't made any yet. :( We want to hook this up with Mozillians.org in the (near) future. - Events
We've tried to make the MDN events page a comprehensive resource to find the events Mozillians attend. Inspired originally by Christian's site for Mozilla events. - Demos & Dev Derby
We've done significant work on the Dev Derby feature - streamlining the process and clearing out bugs. This is one of my favorite MDN features. When I need to show anyone cool web technology I can open the MDN demo studio and inevitably find something way cool with canvas, video, audio, and webgl. - Miscellaneous
We're also finding and squashing bugs as often as we can. We finally restored production server error emails, so we should be much better about squashing bugs that real users are are hitting. And we're trying to add smaller optimizations and fun features as often as we can.
I enjoy enhancing MDN in addition to building the new wiki system. The wiki is important and should help us take more control over our docs, but it's still behind-the-scenes; I like shipping code to a production site that helps people, especially developers, immediately.
Wiki
We hope to soon (by the end of the year) run the new wiki side-by-side with MindTouch so we can work from a single line of code and allow testers to compare MindTouch with the new wiki on the same server. But so far we've developed the following features on the separate kuma wiki staging server:
- Drafts
We use localStorage to periodically save the editing session so a writer doesn't lose precious work if their client crashes. - Review Flags
We added review flags for both technical and editorial review. - Search
We restored the Sphinx-powered search backend we inherited from SUMO/kitsune (just in time for them to move on to ElasticSearch). We still need to restore all of the search frontend and ancillary features. - Page Properties
We restored and refactored document properties we inherited from SUMO/kitsune. - Section Editing
We added section editing to the wiki - our first major modification to the wiki functionality we inherited from SUMO/kitsune. - Authentication
We're in the middle of switching MDN user authentication from MindTouch to django. We will run both authentication backends together, first django then falling back to MindTouch and auto-registering and migrating users to django authentication as they log in.
The wiki work is going pretty well, but we still have a couple of major hurdles to clear: data migration & user scripting. We are already investigating data migration a bit, and once we start migrating data over it should help direct our work on user scripts. In my opinion, we can see light at the end of the tunnel but we're still watching out to avoid potential train-wrecks.
The rest
Outside of MDN, I helped pull together an HTML5 track for Tulsa Tech Fest, helped organize a Tulsa Hackathon with Tulsa Web Devs, and I participated in Startup Weekend Tulsa. At SWT we started a quick web app on top of the TRIF project from Hackathon called OttoZen to send an SMS alert to someone when there's a traffic incident on their commute. It's making us consider building a broader Data/App site for Oklahoma. We'll see.
From now on I'll try to make a status post every time we make an MDN release.
{{ your_town }} Hackathon
Last weekend we did our Tulsa Hackathon - the summary post has links to sites and code we made and a local media live stream where I talk too much like an idiot.
It was a great experience. Mozilla sponsored our after-party (thanks Stormy!) where I emphasized Mozilla's mission to open and improve the web for everyone - from 450 million Firefox users all over the world to 40 developers in Tulsa, Oklahoma. Here's a quick summary of the projects that more-or-less finished (more details at the website):
- http://cfbeo.qrimp.com - a public website to help our local food bank register and organize volunteers
- http://trif.tulsawebdevs.org - a site and platform aggregating Tulsa traffic data
- http://www.fablabtulsa.com/the-lab/equipment - added equipment calendars and requests to the site
At Mozilla we have lots of remote employees who live in towns outside of the "hugeness bubbles" like Silicon Valley. I want to encourage any and all remote employees to get involved with their local technology and developer communities. To promote the Hackathon, I went on a whirlwind tour of our local groups - I spoke about Solr @ Tulsa Java, mobile web development @ Mobile Tulsa, attended a Tulsa .Net developers meeting, arranged an HTML5 track at Tulsa Tech Fest (where Dave gave us the big auditorium stage!), and attended our community leadership town hall. It was tiring, humbling, and inspiring. I'm still processing it all.
Programs like Mozilla Reps help amazing and effective volunteers to promote the Mozilla mission in local communities. In fact, out of all this activity, one of our Tulsa Web Dev's is applying to the program. But that shouldn't offset employees evangelizing for a free, open, and better web right in our own back yards!
Virtual Machines and Real Humans
Encouraging contributors to open source website development
Mozilla, like any good open source project, is a community. 25% of contributions to Mozilla Core, Firefox, Thunderbird, and other products come from the community rather than employees. We, i.e. Mozilla Webdev, want more community web developers to contribute to Mozilla websites. Here are community contribution stats for some Mozilla websites:
- SUMO
github: kitsune
ADV*: 357k
community contributors: 0† - AMO
github: zamboni
ADV*: 736k
community contributors: 3-4† - Socorro
github: socorro
ADV*: ?
community contributors: 1-2† - MDN
github: kuma
ADV*: 56k
community contributors: 4
* Average Daily Visitors
† Guesstimated figures - looked at commit logs and our LDAP phonebook; open to corrections
tofumatt wrote a good post about how we're starting to use vagrant and puppet to give developers easier access to working on Mozilla Webdev sites. lmorchard has a passion for this idea, and he's been extremely patient with all my troubles and questions about Vagrant and Puppet as I've tried to jump on the bandwagon of putting clouds in boxes. I don't want to discredit their work and opinions, but let me offer my own.
I am slow and make mistakes
Instructions for humans are nowhere near as good as instructions for computers. Humans make mistakes, misunderstand, and are generally not as fast as computers when doing this kind of thing.
- tofumatt
Amen to that. If you read some choice excerpts from our #mdn chat log, you'll see how slow I am (and how patient lmorchard is) re: vagrant. Even though I'm slow, I presumably have more "sysadmin" experience than the oft-cited hypothetical design contributor just trying to make a CSS fix, and yet vagrant and puppet make me cringe. So far, vagrant and puppet have only introduced MORE setup pain - and not the regular MySQL setup pain that I (and yes, even designers) have done dozens of times already.
Making boxes put boxes into clouds for your boxes
I've also started playing with Puppet by itself, to build plain-vanilla VM appliances and to spin up cloud servers on Amazon EC2 and Rackspace Cloud Servers. It's going pretty well, and it's conceivable we could do without Vagrant to script the build of downloadable VM appliances and to spin up disposable cloud servers.
- lmorchard
lmorchard is spot on - maybe because he's tired of dragging my slow brain thru vagrant and puppet. Our CI machine should generate vm images with 1-button deploy to Rackspace or Amazon. I think this idea of using VM's to encourage community contribution to website projects is an all-or-nothing deal. If it doesn't work 100%, it's worse than failure. If a community contributor runs into problems with vagrant or puppet, it's much worse than if they run into problems with MySQL or pip. At least there's a bunch of established resources to debug MySQL - I couldn't find much debugging resources for vagrant, besides lmorchard.
At SourceForge we had this kind of "ideal" setup for years. The Service Operations Group at SourceForge maintains sandbox servers with an admin dashboard for creating, managing, and destroying per-developer sandbox vm's that run instances of SourceForge.net. (Though sadly it's behind corporate LDAP - only given to employees) I don't know if/how that model could scale from a single site to dozens of sites all with different moving parts. And there are big latency issues developing on a remote vm. Still, it's exactly what we would want for that elusive CSS designer who wants to fix Mozilla sites without setting one up. She clicks "Create MDN Sandbox" and gets an email containing the domain and access credentials to start editing code on her sandbox. Beautiful.
The virtual machine is us
The web is no longer just linking information ... the web is linking people. People sharing, trading, and collaborating.
In all this technology, we sometimes lose track of the people. I don't want to abandon the idea of using virtual machines to encourage community contribution; the SourceForge sandbox model is, for me, the ideal system for enabling easy contributions from casual community developers. I'll join work for something similar at Mozilla.
I do want to highlight an additional on-ramp for getting community developers involved with Mozilla websites - Mozilla webdev's should get involved with our developer communities!
A couple weeks ago at our Tulsa Web Devs coworking day, Buddy and I were talking about Mozilla and he asked how he can get a job at Mozilla. I said Mozilla really likes looking at your open-source code, and also likes hiring community contributors. He asked how he should become a contributor. The easy answer is go to the Get Involved with Mozilla page, but since I was working on MDN that moment, I showed him the kuma repository I was working on. He forked it and went thru the "menial and often fragile setup" in our installation.rst; I fielded his questions as he went.
It was incredibly valuable - he filed a kuma setup and installation bug and made the appropriate fixes. Then he went on to fix an easy 1-liner bug, and got comfortable enough to do a significant bug involving cron, urllib, and json to generate a humans.txt file for MDN - which now includes him as a community contributor!
Of course, we can't spend all our time recruiting and mentoring community developers to do our work for us. [Or can we? ;) ] And Buddy is an exception in his level of commitment to contribute. So, yes we should reach out to those developers looking to make a casual CSS fix without wanting to set up a whole website environment (though we've received those 1-line CSS commits into MDN even without vm's). That's a good reason to pursue turn-key VM's.
But if you're at a conference or an event and someone asks you how to get involved with or fix a bug on Mozilla websites, don't just send them to a page or a virtual machine - pull up a chair and work with the real human sitting next to you.
['Tulsa %s' % e for e in ('Tech Fest', 'Hackathon')]
Tulsa Tech Fest
In addition to our work on MDN, this is a busy week for me in Tulsa - the annual Tulsa Tech Fest is Friday Oct 7th. Our Tulsa Web Devs group is doing an HTML5 track; my talk is the intro - HTML5: Code for all the platforms! (Soon to be a link to the slides)
TTF is a Microsoft-heavy event, but we've come a long way since TTF 2007 when Steven and I did the entire PHP track ourselves, covering Eclipse & PDT, PHPUnit & Selenium, CI w/ Phing & CruiseControl, DB refactoring w/ dbdeploy - sophisticated topics for 2007!
As far as the Tech Fest goes, I'm really looking forward to the community town hall the night before - developer group leaders from all around Tulsa get together to talk about how to grow the Tulsa developer community. I'm eager to see how the other groups are doing and to share experiences.
I'll spend all of Friday in the HTML5 room, unless I'm pulled off to the
Tulsa Hackathon
Tulsa Hackathon is a big event for me, as I've done a bunch of work for it.
It all started when Tulsa Web Devs took it upon ourselves to put Metro Tulsa Transit Authority data onto Google Maps. After a few hack days we made it work - barely. We still don't have all of MTTA's data, and we didn't make a shape file to smooth out the way Google draws the bus routes to match the roads, but we did pass the GTFS validation test! All that's left is to get MTTA to use our tool to publish the feed for Google - something that's proving harder than any of the code, I think.
In any case, most of us enjoyed working on a cool project that improved Tulsa. So, I reached out to other organizations to see what other projects we might try. Over the last couple months, we've set up projects for INCOG (another transit org in the area), Tulsa City County Library, Community Food Bank of Eastern Oklahoma, and Fab Lab Tulsa. We're an official Random Hacks of Kindness community event!
Scott Phillips and I have arranged all the facilities and venues, logistics, swag, meals, etc. I'm anxious to see how much we actually get done. But really, there are just two goals - improve Tulsa, and our developer community.
Thank you Stormy (and Mozilla) for helping to sponsor our Hackathon! Everyone in Tulsa loves having Mozilla involved! I love that Mozilla promotes the web for public benefit - from hundreds of millions of Firefox users to a few dozen hackers in Tulsa Oklahoma.
The Long Local Tail of Web Developers
At my last work week in Mountain View, I sat in on a few #devengage meetings. I heard someone (maybe Christian), say we do a good job talking to the top 20% of web developers but not necessarily the bottom 80% - the long tail of web developers. At Mozilla, I know I'm not in the "top" 20% of Webdev - there are just too many awesome webdevs around me. And though it's intimidating, it's also exhilarating to work alongside so many top-notch developers who can teach me the "top" 20% of web development skills. (Les, I'm looking at YOU!)
Still, something many of us remote Mozilla employees appreciate is the "rock star" status that working for Mozilla grants us in our local places outside of "Hugeness bubbles" like Silicon Valley. The Mozilla reputation is relatively higher and commands more attention outside these bubbles.
For example, at the Tulsa School of Dev conference, with (I think) 150 attendees, a few Tulsa Web Devs arranged a "web" track that really amounted to a "WordPress" track, since all of us had at least some WordPress experience to share on an entry level. That afternoon, I gave a Web App Security talk using our own WordPress site as the victim. (Sorry guys). My session had about 40-50 attendees - maybe 1/3 of the entire conference!
I'm smart enough to know that I'm NOT smart enough to attract that many developers on my own merit. The simple fact is putting "Mozilla" next to a speaker's name in Tulsa, Oklahoma attracts relatively more developers than it does at High-Profile-Town Tech Conference 2.0. In Portland, at Open Source Bridge with 480 attendees, where more knowledgeable and more adept Mozilla Webdevs spoke; they attracted roughly the same 40-50 number of attendees at each of their talks - including those of us in Webdev!
What if Mozilla leveraged our remote employees' clout to reach out specifically to their local web developer communities - the "bottom" 80% of web developers like me? What would that do? I would hope a few very cool and important things:
- Push & pull more web developers towards the "top" 20% which ...
- Flattens the head of web developer knowledge and skill, which ...
- Raises the overall level of web developers
I've read the Long Tail. I agree power laws are natural and healthy distributions. We need the top 1% and the top 0.1% who are pushing development limits at the razor's edge. At the same time, Mozilla is about flattening power for the benefit of the individual and the betterment of the web. We could do really well if we purposely reach further outside the "top" 20% of web developers.
But what would it look like? Does Mozilla send staff to smaller local events where we already have remote employees? Do we sponsor these events? Do we bring some developers from small communities to Mozilla HQ for a work-week? I don't know - I'm asking.
