[openstack-dev] [ironic] [nova] traits discussion call - moved to Tue!!
Dmitry Tantsur
dtantsur at redhat.com
Thu Nov 2 09:17:47 UTC 2017
Hi all,
The recording of the call is https://bluejeans.com/s/K3wZZ
On 10/30/2017 03:32 PM, Dmitry Tantsur wrote:
> It seems that the new time works for the most of key people, so let's move it to
> tomorrow (Tue), the same time, the same bluejeans.
>
> Apologies to those who won't be able to attend, and sorry for the late notice.
>
> On 10/30/2017 03:13 PM, Jay Pipes wrote:
>> I'd prefer to have you on the call, Dima. How about we push it back to
>> tomorrow at the same time?
>>
>> Can everyone make it then?
>>
>> -jay
>>
>> On 10/30/2017 10:11 AM, Dmitry Tantsur wrote:
>>> Aaaand sorry again, but due to sudden errands I won't be able to attend.
>>> Please feel free to use my bluejeans room anyway. I think my position on
>>> traits is more or less clear from previous discussions with John, Sam and Eric.
>>>
>>> 2017-10-24 18:07 GMT+02:00 Dmitry Tantsur <dtantsur at redhat.com
>>> <mailto:dtantsur at redhat.com>>:
>>>
>>> Sigh, sorry. I forgot that we're moving back to winter time this
>>> weekend. I *think* the time is 3pm UTC then. It seems to be 11am
>>> eastern US:
>>> https://www.timeanddate.com/worldclock/converter.html?iso=20171030T150000&p1=37&p2=tz_et
>>>
>>> <https://www.timeanddate.com/worldclock/converter.html?iso=20171030T150000&p1=37&p2=tz_et>.
>>>
>>>
>>>
>>> On 10/24/2017 06:00 PM, Dmitry Tantsur wrote:
>>>
>>> And the winner is Mon, 30 Oct, 2pm UTC!
>>>
>>> The bluejeans ID is https://bluejeans.com/757528759
>>> <https://bluejeans.com/757528759>
>>> (works without plugins in recent FF and Chrome; if it asks to
>>> install an app, ignore it and look for a link saying "join with
>>> browser")
>>>
>>> On 10/23/2017 05:02 PM, Dmitry Tantsur wrote:
>>>
>>> Hi all!
>>>
>>> I'd like to invite you to the discussion of the way to
>>> implement traits in
>>> ironic and the ironic virt driver. Please vote for the time at
>>> https://doodle.com/poll/ts43k98kkvniv8uz
>>> <https://doodle.com/poll/ts43k98kkvniv8uz>. Please vote by
>>> EOD tomorrow.
>>>
>>> Note that it's going to be a technical discussion - please
>>> make sure you
>>> understand what traits are and why ironic cares about them.
>>> See below for more
>>> context.
>>>
>>> We'll probably use my bluejeans account, as it works without
>>> plugins in modern
>>> browsers. I'll post a meeting ID when we pick the date.
>>>
>>>
>>> On 10/23/2017 04:09 PM, Eric Fried wrote:
>>>
>>> We discussed this a little bit further in IRC [1].
>>> We're all in
>>> agreement, but it's worth being precise on a couple of
>>> points:
>>>
>>> * We're distinguishing between a "feature" and the
>>> "trait" that
>>> represents it in placement. For the sake of this
>>> discussion, a
>>> "feature" can (maybe) be switched on or off, but a
>>> "trait" can either be
>>> present or absent on a RP.
>>> * It matters *who* can turn a feature on/off.
>>> * If it can be done by virt at spawn time, then it
>>> makes sense to have
>>> the trait on the RP, and you can switch the feature
>>> on/off via a
>>> separate extra_spec.
>>> * But if it's e.g. an admin action, and spawn has
>>> no control, then the
>>> trait needs to be *added* whenever the feature is *on*,
>>> and *removed*
>>> whenever the feature is *off*.
>>>
>>> [1]
>>> http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2017-10-23.log.html#t2017-10-23T13:12:13
>>>
>>> <http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2017-10-23.log.html#t2017-10-23T13:12:13>
>>>
>>>
>>>
>>> On 10/23/2017 08:15 AM, Sylvain Bauza wrote:
>>>
>>>
>>>
>>> On Mon, Oct 23, 2017 at 2:54 PM, Eric Fried
>>> <openstack at fried.cc
>>> <mailto:openstack at fried.cc
>>> <mailto:openstack at fried.cc>>> wrote:
>>>
>>> I agree with Sean. In general terms:
>>>
>>> * A resource provider should be marked with a
>>> trait if that feature
>>> * Can be turned on or off (whether it's
>>> currently on or not); or
>>> * Is always on and can't ever be turned off.
>>>
>>>
>>> No, traits are not boolean. If a resource provider
>>> stops providing a
>>> capability, then the existing related trait should
>>> just be removed,
>>> that's it.
>>> If you see a trait, that's just means that the
>>> related capability for
>>> the Resource Provider is supported, that's it too.
>>>
>>> MHO.
>>>
>>> -Sylvain
>>>
>>>
>>>
>>> * A consumer wanting that feature present
>>> (doesn't matter whether it's
>>> on or off) should specify it as a required
>>> *trait*.
>>> * A consumer wanting that feature present and
>>> turned on should
>>> * Specify it as a required trait; AND
>>> * Indicate that it be turned on via some
>>> other mechanism (e.g. a
>>> separate extra_spec).
>>>
>>> I believe this satisfies Dmitry's (Ironic's)
>>> needs, but also Jay's drive
>>> for placement purity.
>>>
>>> Please invite me to the hangout or whatever.
>>>
>>> Thanks,
>>> Eric
>>>
>>> On 10/23/2017 07:22 AM, Mooney, Sean K wrote:
>>> >
>>> >
>>> >
>>> >
>>> > *From:*Jay Pipes [mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>
>>> <mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>>]
>>> > *Sent:* Monday, October 23, 2017 12:20 PM
>>> > *To:* OpenStack Development Mailing List
>>> <openstack-dev at lists.openstack.org
>>> <mailto:openstack-dev at lists.openstack.org>
>>> <mailto:openstack-dev at lists.openstack.org
>>> <mailto:openstack-dev at lists.openstack.org>>>
>>> > *Subject:* Re: [openstack-dev] [ironic]
>>> ironic and traits
>>> >
>>> >
>>> >
>>> > Writing from my phone... May I ask that
>>> before you proceed with any plan
>>> > that uses traits for state information that
>>> we have a hangout or
>>> > videoconference to discuss this?
>>> Unfortunately today and tomorrow I'm
>>> > not able to do a hangout but I can do one
>>> on Wednesday any time of the day.
>>> >
>>> >
>>> >
>>> > */[Mooney, Sean K] on the uefi boot topic I
>>> did bring up at the
>>> ptg that
>>> > we wanted to standardizes tratis for
>>> “verified boot” /*
>>> >
>>> > */that included a trait for uefi secure
>>> boot enabled and to
>>> indicated a
>>> > hardware root of trust, e.g. intel boot
>>> guard or similar/*
>>> >
>>> > */we distinctly wanted to be able to tag
>>> nova compute hosts with those
>>> > new traits so we could require that vms
>>> that request/*
>>> >
>>> > */a host with uefi secure boot enabled and
>>> a hardware root of
>>> trust are
>>> > scheduled only to those nodes. /*
>>> >
>>> > */ /*
>>> >
>>> > */There are many other examples that effect
>>> both vms and bare
>>> metal such
>>> > as, ecc/interleaved memory, cluster on die, /*
>>> >
>>> > */l3 cache code and data prioritization,
>>> vt-d/vt-c, HPET, Hyper
>>> > threading, power states … all of these
>>> feature may be present on the
>>> > platform/*
>>> >
>>> > */but I also need to know if they are
>>> turned on. Ruling out state in
>>> > traits means all of this logic will
>>> eventually get pushed to scheduler
>>> > filters/*
>>> >
>>> > */which will be suboptimal long term as
>>> more state is tracked.
>>> Software
>>> > defined infrastructure may be the future
>>> but hardware defined
>>> software/*
>>> >
>>> > */is sadly the present…/*
>>> >
>>> > */ /*
>>> >
>>> > */I do however think there should be a
>>> sperateion between asking for a
>>> > host that provides x with a trait and
>>> asking for x to be
>>> configure via/*
>>> >
>>> > */A trait. The trait secure_boot_enabled
>>> should never result in the
>>> > feature being enabled It should just find a
>>> host with it on. If
>>> you want/*
>>> >
>>> > */To request it to be turned on you would
>>> request a host with
>>> > secure_boot_capable as a trait and have a
>>> flavor extra spec or image
>>> > property to request/*
>>> >
>>> > */Ironic to enabled it. these are two very
>>> different request and
>>> should
>>> > not be treated the same. /*
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Lemme know!
>>> >
>>> > -jay
>>> >
>>> >
>>> >
>>> > On Oct 23, 2017 5:01 AM, "Dmitry Tantsur"
>>> <dtantsur at redhat.com <mailto:dtantsur at redhat.com>
>>> <mailto:dtantsur at redhat.com
>>> <mailto:dtantsur at redhat.com>>
>>> > <mailto:dtantsur at redhat.com
>>> <mailto:dtantsur at redhat.com>
>>> <mailto:dtantsur at redhat.com
>>> <mailto:dtantsur at redhat.com>>>> wrote:
>>> >
>>> > Hi Jay!
>>> >
>>> > I appreciate your comments, but I think
>>> you're approaching the
>>> > problem from purely VM point of view.
>>> Things simply don't work the
>>> > same way in bare metal, at least not if
>>> we want to provide the same
>>> > user experience.
>>> >
>>> >
>>> >
>>> > On Sun, Oct 22, 2017 at 2:25 PM, Jay
>>> Pipes <jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>
>>> <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>
>>> > <mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>
>>> <mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>>>> wrote:
>>> >
>>> > Sorry for delay, took a week off
>>> before starting a new job.
>>> > Comments inline.
>>> >
>>> > On 10/16/2017 12:24 PM, Dmitry
>>> Tantsur wrote:
>>> >
>>> > Hi all,
>>> >
>>> > I promised John to dump my
>>> thoughts on traits to the
>>> ML, so
>>> > here we go :)
>>> >
>>> > I see two roles of traits (or
>>> kinds of traits) for
>>> bare metal:
>>> > 1. traits that say what the
>>> node can do already (e.g. "the
>>> > node is
>>> > doing UEFI boot")
>>> > 2. traits that say what the
>>> node can be *configured* to do
>>> > (e.g. "the node can
>>> > boot in UEFI mode")
>>> >
>>> >
>>> > There's only one role for traits.
>>> #2 above. #1 is state
>>> > information. Traits are not for
>>> state information. Traits are
>>> > only for communicating capabilities
>>> of a resource provider
>>> > (baremetal node).
>>> >
>>> >
>>> >
>>> > These are not different, that's what
>>> I'm talking about here. No
>>> > users care about the difference between
>>> "this node was put in UEFI
>>> > mode by an operator in advance", "this
>>> node was put in UEFI
>>> mode by
>>> > an ironic driver on demand" and "this
>>> node is always in UEFI mode,
>>> > because it's AARCH64 and it does not
>>> have BIOS". These situation
>>> > produce the same result (the node is
>>> booted in UEFI mode), and
>>> thus
>>> > it's up to ironic to hide this difference.
>>> >
>>> >
>>> >
>>> > My suggestion with traits is one way to
>>> do it, I'm not sure
>>> what you
>>> > suggest though.
>>> >
>>> >
>>> >
>>> >
>>> > For example, let's say we add the
>>> following to the os-traits
>>> > library [1]
>>> >
>>> > * STORAGE_RAID_0
>>> > * STORAGE_RAID_1
>>> > * STORAGE_RAID_5
>>> > * STORAGE_RAID_6
>>> > * STORAGE_RAID_10
>>> >
>>> > The Ironic administrator would add
>>> all RAID-related traits to
>>> > the baremetal nodes that had the
>>> *capability* of
>>> supporting that
>>> > particular RAID setup [2]
>>> >
>>> > When provisioned, the baremetal
>>> node would either have RAID
>>> > configured in a certain level or
>>> not configured at all.
>>> >
>>> >
>>> > A very important note: the
>>> Placement API and Nova
>>> scheduler (or
>>> > future Ironic scheduler) doesn't
>>> care about this. At all.
>>> I know
>>> > it sounds like I'm being callous,
>>> but I'm not. Placement and
>>> > scheduling doesn't care about the
>>> state of things. It only
>>> cares
>>> > about the capabilities of target
>>> destinations. That's it.
>>> >
>>> >
>>> >
>>> > Yes, because VMs always start with a
>>> clean state, and
>>> hypervisor is
>>> > there to ensure that. We don't have
>>> this luxury in ironic :) E.g.
>>> > our SNMP driver is not even aware of
>>> boot modes (or RAID, or BIOS
>>> > configuration), which does not mean
>>> that a node using it cannot be
>>> > in UEFI mode (have a RAID or BIOS
>>> pre-configured, etc, etc).
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > This seems confusing, but it's
>>> actually very useful.
>>> Say, I
>>> > have a flavor that
>>> > requests UEFI boot via a trait.
>>> It will match both the
>>> nodes
>>> > that are already in
>>> > UEFI mode, as well as nodes
>>> that can be put in UEFI mode.
>>> >
>>> >
>>> > No :) It will only match nodes that
>>> have the UEFI capability.
>>> > The set of providers that have the
>>> ability to be booted
>>> via UEFI
>>> > is *always* a superset of the set
>>> of providers that *have been
>>> > booted via UEFI*. Placement and
>>> scheduling decisions only care
>>> > about that superset -- the
>>> providers with a particular
>>> capability.
>>> >
>>> >
>>> >
>>> > Well, no, it will. Again, you're purely
>>> basing on the VM idea,
>>> where
>>> > a VM is always *put* in UEFI mode, no
>>> matter how the hypervisor
>>> > looks like. It is simply not the case
>>> for us. You have to care
>>> what
>>> > state the node is, because many drivers
>>> cannot change this state.
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > This idea goes further with
>>> deploy templates (new concept
>>> > we've been thinking
>>> > about). A flavor can request
>>> something like CUSTOM_RAID_5,
>>> > and it will match the
>>> > nodes that already have RAID 5,
>>> or, more
>>> interestingly, the
>>> > nodes on which we
>>> > can build RAID 5 before
>>> deployment. The UEFI example above
>>> > can be treated in a
>>> > similar way.
>>> >
>>> > This ends up with two sources
>>> of knowledge about traits in
>>> > ironic:
>>> > 1. Operators setting something
>>> they know about hardware
>>> > ("this node is in UEFI
>>> > mode"),
>>> > 2. Ironic drivers reporting
>>> something they
>>> > 2.1. know about hardware
>>> ("this node is in UEFI mode" -
>>> > again)
>>> > 2.2. can do about hardware
>>> ("I can put this node in
>>> UEFI
>>> > mode")
>>> >
>>> >
>>> > You're correct that both pieces of
>>> information are important.
>>> > However, only the "can do about
>>> hardware" part is relevant to
>>> > Placement and Nova.
>>> >
>>> > For case #1 we are planning on
>>> a new CRUD API to set/unset
>>> > traits for a node.
>>> >
>>> >
>>> > I would *strongly* advise against
>>> this. Traits are not for
>>> state
>>> > information.
>>> >
>>> > Instead, consider having a DB (or
>>> JSON) schema that lists
>>> state
>>> > information in fields that are
>>> explicitly for that state
>>> > information.
>>> >
>>> > For example, a schema that looks
>>> like this:
>>> >
>>> > {
>>> > "boot": {
>>> > "mode": <one of 'bios' or 'uefi'>,
>>> > "params": <dict>
>>> > },
>>> > "disk": {
>>> > "raid": {
>>> > "level": <int>,
>>> > "controller": <one of 'sw' or
>>> 'hw'>,
>>> > "driver": <string>,
>>> > "params": <dict>
>>> > }, ...
>>> > },
>>> > "network": {
>>> > ...
>>> > }
>>> > }
>>> >
>>> > etc, etc.
>>> >
>>> > Don't use trait strings to
>>> represent state information.
>>> >
>>> >
>>> >
>>> > I don't see an alternative proposal
>>> that will satisfy what we have
>>> > to solve.
>>> >
>>> >
>>> >
>>> >
>>> > Best,
>>> > -jay
>>> >
>>> > Case #2 is more interesting. We
>>> have two options, I think:
>>> >
>>> > a) Operators still set traits
>>> on nodes, drivers are simply
>>> > validating them. E.g.
>>> > an operators sets
>>> CUSTOM_RAID_5, and the node's RAID
>>> > interface checks if it is
>>> > possible to do. The downside is
>>> obvious - with a lot of
>>> > deploy templates
>>> > available it can be a lot of
>>> manual work.
>>> >
>>> > b) Drivers report the traits,
>>> and they get somehow
>>> added to
>>> > the traits provided
>>> > by an operator. Technically,
>>> there are sub-cases again:
>>> > b.1) The new traits API
>>> returns a union of
>>> > operator-provided and
>>> > driver-provided traits
>>> > b.2) The new traits API
>>> returns only operator-provided
>>> > traits; driver-provided
>>> > traits are returned e.g. via a
>>> new field
>>> > (node.driver_traits). Then nova
>>> will
>>> > have to merge the lists itself.
>>> >
>>> > My personal favorite is the
>>> last option: I'd like a clear
>>> > distinction between
>>> > different "sources" of traits,
>>> but I'd also like to reduce
>>> > manual work for
>>> > operators.
>>> >
>>> > A valid counter-argument is:
>>> what if an operator wants to
>>> > override a
>>> > driver-provided trait? E.g. a
>>> node can do RAID 5, but I
>>> > don't want this
>>> > particular node to do it for
>>> any reason. I'm not sure if
>>> > it's a valid case, and
>>> > what to do about it.
>>> >
>>> > Let me know what you think.
>>> >
>>> > Dmitry
>>> >
>>> >
>>> > [1]
>>> http://git.openstack.org/cgit/openstack/os-traits/tree/
>>> <http://git.openstack.org/cgit/openstack/os-traits/tree/>
>>> <http://git.openstack.org/cgit/openstack/os-traits/tree/
>>> <http://git.openstack.org/cgit/openstack/os-traits/tree/>>
>>> > [2] Based on how many attached
>>> disks the node had, the
>>> presence
>>> > and abilities of a hardware RAID
>>> controller, etc
>>> >
>>> >
>>> >
>>> >
>>> __________________________________________________________________________
>>> > OpenStack Development Mailing List
>>> (not for usage questions)
>>> > Unsubscribe:
>>> >
>>>
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>>
>>> >
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>>>
>>> >
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>>
>>> >
>>> >
>>> >
>>> >
>>> >
>>> __________________________________________________________________________
>>> > OpenStack Development Mailing List (not
>>> for usage questions)
>>> > Unsubscribe:
>>> >
>>>
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>>
>>> >
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>>>
>>> >
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>>
>>> >
>>> >
>>> >
>>> >
>>> __________________________________________________________________________
>>> > OpenStack Development Mailing List (not for
>>> usage questions)
>>> > Unsubscribe:
>>>
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>>
>>> >
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>>
>>> >
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for
>>> usage questions)
>>> Unsubscribe:
>>>
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>>
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>>
>>>
>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage
>>> questions)
>>> Unsubscribe:
>>>
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>>
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>>
>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>>
>>>
>>>
>>>
>>> --
>>> --
>>> -- Dmitry Tantsur
>>> --
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
More information about the OpenStack-dev
mailing list