Open Stack

Taking Stefano’s advice and Ccing the operators mailing list.  Including
the conversation thus far to add context.

The UX team is preparing to run a number of studies to help improve the
user experience associated with Horizon.  The first study is a usability
study to evaluate the proposed Launch Instance workflow to understand
whether users feel whether its an improvement over the current design.
This will likely be an unmoderated study using a tool like
usertesting.com.  

The second study will be a card sort exercise to understand how users
would organize the panels in the left hand navigation in Horizon.  An
example of an online card sort can be found at
http://www.optimalworkshop.com/optimalsort-demo.

The motivation for these studies is to take an evidence-based approach to
UX design in Horizon. Also, I want to stress that the results will be
shared with the community.

The challenge is that recruiting for these studies is typically a huge
barrier.  Usability studies are a bit easier because the sample generally
ranges from 8 - 16 participants depending on what is actually being
tested.  Card sorts are more challenging because the sample should be over
fifty participants to see consistent results.

One potential approach would be to solicit participants through an online
survey (aka screener) that categorizes the respondents.  A link to the
screener could potentially be added to the OpenStack public website.  I
would imagine that Tom would have admin privileges to any data collected.
The survey could actually be created using an open source product like
limesurvey.org.

Researchers interested in conducting a study could reach-out to Tom with a
brief description of the study and a description of the kind of folks they
would need, i.e. “end Users.” If accepted, Tom would provide an email list
of folks that meet the criteria.

Let me know if the makes sense and your thoughts.

Piet










On 11/25/14, 4:39 PM, "Stefano Maffulli" <stefano at openstack.org> wrote:

>These seem to be a valuable discussions, both the one about recruiting
>testers and the one about the tool to select them. Why are we not having
>it in public on one of our mailing lists? OpenStack-dev or operators
>would be both appropriate venues.
>
>/stef
>
>On 11/25/2014 07:07 AM, Kruithof, Piet wrote:
>> There is an open source product called lime survey (limesurvey.org) that
>> could be used to create the surveys and host the data.
>>
>>
>>
>>
>>
>> On 11/25/14, 7:58 AM, "Jonathan Proulx" <jon at csail.mit.edu> wrote:
>>
>> >
>> > I think it's some thingworth considering.
>> >
>> > There's a host of technical and social issues around it.  Who would
>> > build & maintain the interface, and what criteria do we use to gate
>> > requests for data come to mind.  Those two main points likely need to
>> > be addressed before going into tthe finer details of what and how...
>> >
>> > -Jon
>> >
>> > On Tue, Nov 25, 2014 at 05:39:33AM +0000, Kruithof, Piet wrote:
>> > :I had a quick question for the entire group.
>> > :
>> > :
>> > :Should we create a database of users that are willing to participate
>>in
>> > :research studies?  Tom could distribute a survey (screener) that
>>includes
>> > :~10 questions to classify respondents. In fact, a link to the survey
>> > could
>> > :be posted to the OpenStack public site.
>> > :
>> > :In this case, only the user committee would have access to the
>>database.
>> > :Someone interested in running a study would need to approach the user
>> > :committee with a request for participants that meets specific
>>screener
>> > :criteria, such as end users, and a brief overview for the study.  One
>> > :requirement would be that any data generated from a study would be
>>shared
>> > :with the user committee.
>> > :
>> > :Upon acceptance of the study, the user committee would provide the
>> > :researcher with a list that only includes email addresses.  The
>> > researcher
>> > :would use the list to either distribute a link to an online study,
>>such
>> > as
>> > :a card sort, or ask to ask for volunteers to participate in a
>>moderated
>> > :study, such as a usability.
>> > :
>> > :
>> > :
>> > :Does that make sense?  My communication skills tend to go a bit
>>sideways
>> > :after 10PMŠ.
>> > :
>> > :
>> > :
>> > :Here are examples of some questions that could be asked:
>> > :
>> > :Does your IT department currently offer cloud computing services at
>>you
>> > :company?
>> > :
>> > :* Yes
>> > :* No
>> > :
>> > :Please select the cloud service providers and technologies that you
>>have
>> > :good working knowledge of.
>> > :
>> > :* Amazon Web Services (EC2)
>> > :* Google Compute Engine
>> > :* Microsoft Windows Azure
>> > :* OpenStack
>> > :* CloudStack
>> > :* VMware vCloud Hybrid Service
>> > :* Other (please specify)
>> > :
>> > :Note that I don¹t mean to imply that these questions are perfect, but
>> > they
>> > :should give a sense of what we would ask.
>> > :
>> > :
>> > :I would like to stress the importance of user research to OpenStack.
>> One
>> > :thing I consistently hear is that we need to improve the user
>>experience,
>> > :but that any changes need to be evidence-based.
>> > :
>> > :
>> > :
>> > :Thanks,
>> > :
>> > :
>> > :Piet
>> > :
>> > :
>> > :
>> > :
>> > :
>> > :
>> > :
>> > :
>> > :On 11/24/14, 1:39 AM, "Tom Fifield" <tom at openstack.org> wrote:
>> > :
>> > :>On 24/11/14 16:36, Tom Fifield wrote:
>> > :>> On 20/11/14 02:54, Kruithof, Piet wrote:
>> > :>>> Hi Tom,
>> > :>>>
>> > :>>> I wanted to make you aware of a couple of studies that we will be
>> > :>>> conducting on behalf of the community and get your advice.
>> > :>>>
>> > :>>>
>> > :>>>
>> > :>>> The first is a usability study to evaluate the proposed Launch
>> > Instance
>> > :>>> workflow in Horizon using either usertesting.com or trymyui.com
>>to
>> > run
>> > :>>>the
>> > :>>> studies.
>> > :>>>
>> > :>>> A good description of this approach can be found on user
>>testing.com.
>> > :>>>In
>> > :>>> our case, there is a strong possibility that we may try to
>>recruit
>> > :>>>through
>> > :>>> the community.
>> > :>>>
>> > :>>> http://www.usertesting.com/how-it-works
>> > :>>>
>> > :>>>
>> > :>>> The work on the Launch Instance wizard can be found here:
>> > :>>>
>> > :>>> http://invis.io/7E1LGP0SB
>> > :>>>
>> > :>>> Anyone on this email - please let me know if you have any
>>interest in
>> > :>>> receiving an invitation to use Invision.  We currently have about
>> > five
>> > :>>> projects that are being reviewed by the designers, devs, etc.
>> > :>>>
>> > :>>>
>> > :>>>
>> > :>>>
>> > :>>> The second study will be a card sort activity to help understand
>>how
>> > :>>>users
>> > :>>> conceptualize the panels in Horizon.  Wikipedia has a good
>> > description
>> > :>>>of
>> > :>>> card sorts:
>> > :>>>
>> > :>>> https://en.wikipedia.org/wiki/Card_sorting.
>> > :>>>
>> > :>>> I also have the study set-up in Optimal Sort if you want to take
>>a
>> > :>>>look at
>> > :>>> a preview:
>> > :>>>
>> > :>>> https://i2471lxv.optimalworkshop.com/optimalsort/002h4457-0
>> > :>>>
>> > :>>>
>> > :>>>
>> > :>>>
>> > :>>> We are looking for 16 participants for the usability study and
>>up to
>> > :>>>500
>> > :>>> folks for the card sort.
>> > :>>>
>> > :>>> Can you suggest some ways to recruit participants for the study?
>> We
>> > :>>>would
>> > :>>> like to continue to do these kinds of activities, but one of the
>> > :>>> significant barriers is finding individuals that are willing to
>> > :>>> participate in these studies.
>> > :>>>
>> > :>>
>> > :>> Hi Piet,
>> > :>>
>> > :>> Thanks for the awesome work.
>> > :>>
>> > :>> I see you've already got something on the ops mailing list, which
>>is
>> > one
>> > :>> of the best ways we have of finding users right now.
>> > :>>
>> > :>> If there's something on twitter for the larger test, I can retweet
>> > that.
>> > :>>
>> > :>> You could also do a write-up of what you're trying to achieve for
>> > :>> SuperUser, which has an ops-heavy audience.
>> > :>>
>> > :>> For the smaller/more detailed study, there might be some people at
>> > :>> non-commercial places that have more flexibility in their
>>schedule.
>> > :>> Consider asking places like CERN, NeCTAR, MIT, ANL, Cybera, INFN
>>to
>> > ask
>> > :>> their users ?
>> > :>>
>> > :>>
>> > :>> Regards,
>> > :>>
>> > :>> Tom
>> > :>
>> > :>
>> > :>(Assuming you also wanted to reach the user committee, I've swapped
>>out
>> > :>the old members with the new ones)
>> > :>
>> > :>
>> > :
>> >
>> > --
>>
>
>-- 
>OpenStack Evangelist - Community
>Ask and answer questions on https://ask.openstack.org
>