Improved Selection Process for DrupalCon Munich

Apr
20
2012
15

Improved Selection Process for DrupalCon Munich

Apr 20 2012

Dear Drupal Community

During the preparations of the content for the Munich DrupalCon the responsible track chairs found themselves discussing - at length - the selection process for proposed sessions. Within minutes the issue of voting came up and after hours we found ourselves still talking of the problems, advantages, risks, correctness etc... of voting. To make a long story short, it became clear that most regarded the voting process as a means for the community to let us know what they like, but, and here comes the problem, past experience show that the results of the voting do not provide a good indicator of session quality, relevance, interest in the topic, or attendance.

Voting

I believe it is clear to all that voting as a popularity contest is not good. In addition, voting may give the wrong impression: getting the most votes does not mean getting picked to do a session. Voting puts external speakers at a disadvantage in comparison to popular Drupalistas and strong votes on "flamewar" topics might take focus away from otherwise great sessions. The issue is not new, and many have voiced their opinions in favor of a change. Among others, Morten - a long-time speaker, track chair and DrupalCon Organizer - published a proposal to address the inefficiency on voting (see http://morten.dk/blog/drupalcon-voting-waste-time-resources).

Voting done right would take a tremendous amount of thought, preparation, planning, and last but not least, (voting) time. As in any Base-Democracy there is an inherent obligation for the voter to study all available information to be able to make a well-thought-out decision. Try to do that with 400+ proposed sessions! Since this is virtually impossible voting becomes emotional and gets skewed. The result: voting doesn’t provide any valuable information for selection.

So how do we select sessions for this DrupalCon?

We will not have voting on sessions, we are strongly encouraging comments instead. We feel this can lead to more relevant feedback, both for track chairs and speakers. An added effect we are looking forward to is, that the better prepared a session proposal/description is, the better its chances are to receive positive and constructive feedback and the better the chances for selection thus are. For any track, the track chair together with the global track chair and possible others evaluate the proposals, the respective feedback and the presenters reaction to that feedback and then make a justified decision for selection or not.

Feedback is extremely important

Needless to say, we need feedback. We need constructive feedback. We need reasons why, or reasons why not. We need a situation where the presenters can work with the feedback on their proposed sessions that will help them improve their presentations and thus raise the quality of the session and the overall DrupalCon experience of all participants.

In this spirit, I am looking forward to a lot of high-quality proposals, constructive feedback and good sessions in Munich.

Jos Doekbrijder - Conference Content Manager

sessions
community
voting
comments

Comments

> We will not have voting on sessions, we are strongly encouraging comments instead.

Good! But then you should also encourage comments in a proposed session, then today there is no reason written why I should write a comment to a proposed session.

Submitted by Schnitzel on April 20, 2012 - 11:11

It also limits the con's ability to attract non-dev disciplines b/c voting turns up a lot more dev content. A well-rounded con would serve us all better :)

Great job!

Submitted by svettes on April 20, 2012 - 11:16

This is a really good step in the right direction for improving both our processes for sessions and our quality of sessions selected. From this post, it reads that there will be a period of public comment - what's not clear is if session submitters will respond publicly or privately.
Can you clarify?

Submitted by amye on April 20, 2012 - 16:36

All comments will be public, if there's a need for private conversations, the commenters and the session submitters can contact each other using contact forms.

Submitted by floretan on April 25, 2012 - 11:00

It should be noted that track chairs will moderate comments on session proposals. "+1" and similarly unproductive comments will be deleted.

Submitted by sun on April 20, 2012 - 16:41

I don't really understand then...

If the point is not "+1" comments, and you instead need us to write a few sentences/paragraphs discussing pros and cons of each session, how does this actually help with the problems you outline in the post related to the impossibility of being well-informed on 400+ sessions? It seems like you'll get even less feedback on fewer sessions this way, because each session takes 5-10 mins to process by the viewer, rather than 5 - 10 seconds. I guess the assumption is we'll only make the time to do this on sessions we think are really good, and this will act as a curation process on its own. But at the end of the day, most still won't have time to even read the titles of 400+ sessions, so it's once again going to come down to who has the most Twitter followers and who's already in Drupal Planet. :\ Aka, a popularity contest. :\

Don't get me wrong. I'm all about switching away from voting as a means of assessing a proposed session's quality—I led a discussion about this last year in the Drupal Association group at http://groups.drupal.org/node/151174—because I totally agree with all of the drawbacks you've outlined. It just doesn't seem like this approach actually addresses them. :(

Submitted by webchick on May 11, 2012 - 15:43

This is spot on. Voting and statistical analysis is not something to treat lightly.

It is also important to gauge interest if room space is an issue. I don't think DrupalCon runs into this issue any more, but while organizing a DrupalCamp fellow organizers had to look at room sizes and try to best allocate "popular" sessions into the biggest rooms.

Submitted by mradcliffe on April 20, 2012 - 17:52

Well done! This is brave.

Submitted by moshe weitzman on April 21, 2012 - 04:49

Needless to say, we need feedback.

How do you propose to capture that feedback? I don't see a mechanism it can be captured.

Submitted by greggles on April 24, 2012 - 05:04

I am very happy to read that we finally have taken this step towards removing the on of the BS noise points in our community and can begin to talk + discuss this with open eyes & not hiding behind false promises of "but votes have bla bla bla"

what i truly hope for is that local community activist looks out for the real good stuff they see at camps, events local group meeting etc. and push the hot new ones forward backup their sessions if they have the content that really needs to be spread to the rest of the drupal world.

Not the hidden Marketing pitches, not the sessions that people wanna give because of the publicity, not the sessions that dont push anything forward, not the beginner sessions that dont teaches anyone any new stuff (that is a complete waste of time & space) - but the talks, subjects & discussions that are needed for the Drupal project to constantly be moving forward.

So lets stop the standstill and attack the future!

/mortendk

Submitted by mortendk on April 26, 2012 - 11:18

How about asking session proposers to add a 2/3/4/5 (?) minute video describing their talk / giving a preview of the talk and how they'd present?

This would force potential presenters to take the time to prepare well and would give the audience an idea of whether or not they would like to attend IRL.

It would also raise the barrier to entry a bit - only the really motivated speakers would take the time and do the effort to submit a proposal.

Submitted by jpoesen on April 26, 2012 - 11:26

I barely had time to provide the minimum info required to submit, let alone do all my preparation upfront and create a video preview. Though I think it would be great, I personally wouldn't have time to do all that, and I'm guessing I'm not alone.

If all that were required to submit a session, I can guarantee you that I wouldn't have submitted my 2 propositions for this con.

Submitted by svettes on May 16, 2012 - 23:21

It's one thing to require a link to a site where the speaker has presented at another event, e.g. a DrupalCamp, preferably where the event was put on video. It's another thing for them to have to own (and be proficient with using) a decent video camera and have video editing skills to even be able to manage this task. Anyway, that would take far longer to produce and to process than text proposals (for the "information part"). Anyone whose session proposal looks good should be scrutinized in the final round of selection and it should be determined whether the proposed speaker has the necessary public speaking experience to manage a DrupalCon session.

I would suggest, for those who have not spoken at DrupalCons, requiring two links to previous DrupalCamp sessions or, if they are not speaking about something directly about Drupal, but e.g. about Symfony, then similar session video links. A minimum of two, well-received DrupalCamp (or similar/equivalent) sessions should be a prerequisite to proposing a DrupalCon session. This would help keep the "distractions" to a minimum and give the community a chance to possibly select from a reasonable pool of candidate session proposals. It's not a "popularity contest" (or shouldn't be), but the speakers do need to be able to present well -- hundreds of proposals from people who are virtually unknown in the community is a bit on the ridiculous side. It makes it near impossible to read all the session proposals.

Submitted by LoMo on May 26, 2012 - 12:59

Let me try heuristically. Randomly and blindly take, lets say, seven proposals for each proposer to vote a grade. The ones that give comparatively low grades and receive high grades, probably have the best proposals. The ones that give high grades and get low grades, probably have the worst proposals. And so on...

Submitted by rodrigo panchin... on May 3, 2012 - 23:44

-1

I don't agree that this proposal is going to prevent a popularity contest at all. Adding comments is a more time intensive and intimidating process. So instead of being able to run through and add votes to 20-30 session I want to see I have to come up with something thoughtful and relevant to say about each session I like.

In lieu of allowing for a simple system that gauges feedback this is moving to a system where only a few people will guide the decision to accept a session. Personally I think the respective chairs are going to be more likely to vote for people they know, it's simply human nature.

Yes -1 on this proposal.

Submitted by bigjim on June 2, 2012 - 02:20