Open Stack

Generally speaking as a track chair my experience has been the same as 
described below, we've used spreadsheets to try to balance opinion and 
shape things appropriately.

When I was at HP we had a massive internal conference, each submission is 
reviewed by four people, who score it according to various requirements. 
Each reviewer is also required to write a paragraph of feedback for each 
paper, there's regularly thousands of submissions for this conference and 
it would basically take a week of effort from every reviewer to deal with 
their load. Reviewers were broken into groups and divisions, each with 
their own leaders who dedicated even more time to the project. Abstracts 
are also typically 2-3 pages of varying quality academic content.

It's an awesome conference that I always enjoyed being a part of, however 
I imagine there must have been at least 50 reviewers and 15 or so group 
chairs etc, each putting in at least a working week of effort which makes 
a conservative estimate of the effort involved around 2535 man hours 
(65*38) - I don't think that the OpenStack community could sustain similar 
levels of effort every 6 months. At the moment I'm guessing being a track 
chair on Security, a smaller track, takes up maybe 10-15 hours of my time 
- read through all the submissions a couple of times, score them, do 
background reading, stack-rank, discuss with other track chairs, reach out 
to authors etc.

The process of putting together a track is additive, we are not rejecting 
talks because they're bad necessarily, they just didn't fit into the 
overall delivery of a compelling, well balanced track. I think the 
OpenStack community, TC, Summit organizers and Track Chairs are actually 
doing a pretty good job. The tool is much enhanced over previous years and 
I'm sure it'll continue to be improved.

-Rob





From:   Nick Chase <nchase at mirantis.com>
To:     Matt Fischer <matt at mattfischer.com>
Cc:     "openstack-track-chairs at lists.openstack.org" 
<openstack-track-chairs at lists.openstack.org>
Date:   25/08/2016 21:45
Subject:        Re: [Openstack-track-chairs] Track Chair Feedback




On 8/25/2016 4:28 PM, Matt Fischer wrote:
I hope that the track chairs were not only selected on the basis of 
technical knowledge but on their integrity and of putting the summit & 
community ahead of company goals. I also think that the foundation has 
done a good job ensuring that different communities and companies are 
chosen, which hopefully avoids the ability of one company to push their 
talks over others. For me, I will never push a team talk unless the other 
track chairs have it under consideration, and if they do I'm happy to 
provide honest feedback on it. I hope fellow track chairs will concur with 
that.

Exactly!!

Opening up a requirement that track chairs have to justify why they chose 
one talk over another will only foster further argument, it won't resolve 
anything. It's also untenable when rejecting 200 talks to justify the 
ranking of each one. And regardless of what you do and whether you justify 
your reasons or not, many good talks were not picked, and some feelings 
were probably hurt.

I'm not suggesting that we explain why talks were rejected, just why they 
were SELECTED.

Feelings are always going to be hurt.  I've already gotten 3 phone calls 
from dejected colleagues who got their "sorry, no thanks" letter and I've 
had to tell them, "Look, you're in good company."  Hell, I didn't get in 
either. :)

But while we've definitely been improving in the "transparency" 
department, I think it wouldn't be a bad thing to go further.

----  Nick


On Thu, Aug 25, 2016 at 4:13 PM, Nick Chase <nchase at mirantis.com> wrote:
Well, when we were working on our track (both this time and for Austin) 
what we did was break it down into sections (for example, we might have 3 
talks on "DevOps") and then try to choose a talk from each section.  In 
some cases, we had to choose between similar talks, and discuss what our 
rationale was; it wouldn't be a bad thing to provide a place for track 
chairs to explain their choices.  It would even provide a way for 
submitters to improve their submissions for next time.
----  Nick

On 8/25/2016 3:50 PM, Shilla Saebi wrote:
I think we should come up with a way for track chairs who have to make 
selections on talks that were submitted by members of their own company or 
team to be fair. I noticed a few talks go through where the track chairs 
worked for the same company as the talks that were selected, and other 
talks with more complete abstracts (in my opinion) and a more diverse set 
of speakers, was not selected. 

On Thu, Aug 25, 2016 at 2:18 PM, Nathan C Ziemann <ziemann at us.ibm.com> 
wrote:
+1 on export. 

We wrapped up our final selections using a spreadsheet which makes it easy 
to evaluate multiple criteria and chair votes/comments in one view. 

Nate Ziemann 
Sent from my iPhone

On Aug 25, 2016, at 9:54 AM, Nick Chase <nchase at mirantis.com> wrote:

Actually the last two summits tracks I've worked with have finished up in 
spreadsheets, where we've organized submissions by topic and marked off 
who voted for each.  It really made the final processing easier.  

I'd love to see a simple "export" function so we don't wind up doing it by 
hand.

---- Nick

On Thursday, August 25, 2016, Amrith Kumar <amrith at tesora.com> wrote:
On the whole, I think the process and the tool were fine, there were some 
hiccups that you have already identified.
 
I found the user interface (specifically the two pane format and the 
sizing requirement) to be a little troublesome. Eventually I just grabbed 
the json off the interface and populated a spreadsheet and that worked out 
much easier.
 
Why was that?
 
I wanted to tag each of the 93 talks with a bunch of tags that I found to 
be useful, things like the names of the projects covered, sales-pitch, 
customer-story, performance, … and so on, and later filter talks based on 
those tags.
 
So, if the next iteration of the tool had a way in which each chair could 
tag talks using an arbitrary (private) set of tags, that would be nifty.
 
-amrith 
 
From: Kunzmann, Gerald [mailto:kunzmann at docomolab-euro.com] 
Sent: Thursday, August 25, 2016 3:22 AM
To: openstack-track-chairs at lists.openstack.org
Subject: Re: [Openstack-track-chairs] Track Chair Feedback
 
+1 nice tool.
 
Would be great to have a sort function e.g. by “community vote”.
 
Gerald
 
From: Travis McPeak [mailto:travis.mcpeak at gmail.com] 
Sent: Donnerstag, 25. August 2016 02:02
To: Christopher Aedo <doc at aedo.net>; 
openstack-track-chairs at lists.openstack.org
Subject: Re: [Openstack-track-chairs] Track Chair Feedback
 
+1 great tool and easy process overall.
 
Thanks!
On Wed, Aug 24, 2016, 4:47 PM Christopher Aedo <doc at aedo.net> wrote:
Thank you for all the hard work you've put into this tool (and all the
other OpenStack bits you're always improving.)  This was my second
time around and I thought the tool worked great - looking forward to
seeing even more improvements for Boston!

Thank you Jimmy!

-Christopher

On Wed, Aug 24, 2016 at 3:20 PM, Jimmy McArthur <jimmy at openstack.org> 
wrote:
> All -
>
> Thanks again for your hard (and quick!) work on the OpenStack Barcelona
> Summit Track selections. You all did an amazing job and we're really
> grateful to each of you.
>
> I took notes of feedback, but please feel free to add to the list if you
> feel I've missed something:
>
> Better ability for Track Chairs to merge
>
> Option to email all speakers from one or more presentations
> Ability for Speakers to submit/propose a new Title / Abstract as an
> attachment or within the thread
> Track Chairs ability to deactivate merged talks, and save one with new
> abstract (or merge into a new record so we don't taint the originals)
>
> Email issues
>
> Track conversations with users within the Track Chair tool i/o just via
> email. Could also consider using ZenDesk API for this?
> There was a problem with Sendgrid API this go round, so there were just
> generally problems. They were:
>
> No CC of other track chairs
> Reply address was noreply at openstack.org i/o the ZenDesk ticket
> Adding name of Presentation and other details automatically within the 
email
> "Track chair XYZ has a question about your presentation: ABCD"
> Email all speakers i/o just one
>
> Need the option back of being able to compare everyone's list side by 
side
>
> The new method of using the dropdown makes it difficult for the team to
> compare selected talks
> A way to auto-select common talks for placement in the team section 
would be
> cool, or at least a way to quickly view that list, along with the # of 
team
> members that selected each talk
>
> An option for a Track to "Close" their selections
>
> Send an email to track chairs list
> Mark Track as "Closed" for non-admins
>
> Another suggestion for the Boston CFP process might be to consider 
separate
> user and developer sub-tracks. For example. What might be considered a 
How
> To/Best Practice for developers might be completely irrelevant to users.
> We at Dev Tools track found out that there are a lot of drafts related 
to
> the CI/CD and Testing processes. Seems reasonable to add such a track
> (Deployment and Testing) as an option for the next summit. What do you
> think?
>
>
> Thank you again!
> Jimmy
>
> _______________________________________________
> Openstack-track-chairs mailing list
> Openstack-track-chairs at lists.openstack.org
> 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-track-chairs
>

_______________________________________________
Openstack-track-chairs mailing list
Openstack-track-chairs at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-track-chairs


_______________________________________________
Openstack-track-chairs mailing list
Openstack-track-chairs at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-track-chairs



-- 
Nick Chase, Head of Technical and Marketing Content, Mirantis
Editor in Chief, OpenStack:Unlocked

_______________________________________________
Openstack-track-chairs mailing list
Openstack-track-chairs at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-track-chairs



-- 
Nick Chase, Head of Technical and Marketing Content, Mirantis
Editor in Chief, OpenStack:Unlocked
_______________________________________________
Openstack-track-chairs mailing list
Openstack-track-chairs at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-track-chairs



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-track-chairs/attachments/20160826/ac34a34f/attachment-0001.html>