[openstack-dev] [magnum] About clean none use container image
449171342
449171342 at qq.com
Tue Apr 14 08:16:19 UTC 2015
@Jay Lau I agree with you that image pull be time-consuming.
Did is much better to supply a magnum api let end-user can clean image that not used?
Clean or not clean is decide by the end-user. Or we can supply a method let
end-user can decide whether or not clean image.
------------------ Original ------------------
From: "openstack-dev-request";<openstack-dev-request at lists.openstack.org>;
Date: Mon, Apr 13, 2015 04:59 PM
To: "openstack-dev"<openstack-dev at lists.openstack.org>;
Subject: OpenStack-dev Digest, Vol 36, Issue 32
Send OpenStack-dev mailing list submissions to
openstack-dev at lists.openstack.org
To subscribe or unsubscribe via the World Wide Web, visit
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
or, via email, send a message with subject or body 'help' to
openstack-dev-request at lists.openstack.org
You can reach the person managing the list at
openstack-dev-owner at lists.openstack.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of OpenStack-dev digest..."
Today's Topics:
1. Re: [Nova] VMware CI (Davanum Srinivas)
2. Re: [Nova] VMware CI (Gary Kotton)
3. Re: [TripleO] Consistent variable documentation for
diskimage-builder elements (Gregory Haynes)
4. Re: [neutron] Neutron scaling datapoints? (Joshua Harlow)
5. Re: [OpenStack-docs] What's Up Doc? Apr 10 2015 (Monty Taylor)
6. Re: [neutron] Neutron scaling datapoints? (Kevin Benton)
7. Re: [neutron] Neutron scaling datapoints? (Kevin Benton)
8. [all][pbr] splitting our deployment vs install dependencies
(Robert Collins)
9. Re: [all][pbr] splitting our deployment vs install
dependencies (Monty Taylor)
10. Re: [all][pbr] splitting our deployment vs install
dependencies (James Polley)
11. Re: [all][pbr] splitting our deployment vs install
dependencies (Monty Taylor)
12. Re: [all][pbr] splitting our deployment vs install
dependencies (Robert Collins)
13. Re: [all][pbr] splitting our deployment vs install
dependencies (Robert Collins)
14. [all] Problems with keystoneclient stable branch (and maybe
yours too) (Brant Knudson)
15. Re: In loving memory of Chris Yeoh (Michael Still)
16. Re: [all][pbr] splitting our deployment vs install
dependencies (Robert Collins)
17. Re: [neutron] Neutron scaling datapoints? (joehuang)
18. Re: [neutron] Neutron scaling datapoints? (Joshua Harlow)
19. Re: [neutron] Neutron scaling datapoints? (Joshua Harlow)
20. Re: [neutron] Neutron scaling datapoints? (Joshua Harlow)
21. Re: [puppet] Puppet PTL (Emilien Macchi)
22. Re: In loving memory of Chris Yeoh (Gary Kotton)
23. Re: [neutron] Neutron scaling datapoints? (Attila Fazekas)
24. Re: [Neutron] Regarding neutron bug # 1432582 (Kevin Benton)
25. ?magnum?About clean none use container imag
(=?gb18030?B?NDQ5MTcxMzQy?=)
26. Re: ?magnum?About clean none use container imag (Jay Lau)
27. Re: [neutron] Neutron scaling datapoints? (Attila Fazekas)
28. Re: In loving memory of Chris Yeoh (wu jiang)
29. Re: [oslo] eventlet 0.17.3 is now fully Python 3 compatible
(Victor Stinner)
30. [Fuel] Nominate Andrey Skedzinskiy for fuel-qa(devops) core
(Anastasia Urlapova)
31. Re: [Fuel] Nominate Andrey Skedzinskiy for fuel-qa(devops)
core (Alexander Kislitsky)
32. Re: [all][pbr] splitting our deployment vs install
dependencies (Miguel Angel Ajo Pelayo)
33. ??: [neutron] Neutron scaling datapoints? (Wangbibo)
----------------------------------------------------------------------
Message: 1
Date: Sun, 12 Apr 2015 08:04:43 -0400
From: Davanum Srinivas <davanum at gmail.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [Nova] VMware CI
Message-ID:
<CANw6fcHxxWT215_AtJeX5LjvwnMrTvGpqXXuGi=oFj9prw_SyA at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Gary, John,
Just to speed things up, i filed a backport:
https://review.openstack.org/#/c/172710/
thanks,
dims
On Sun, Apr 12, 2015 at 4:23 AM, John Garbutt <john at johngarbutt.com> wrote:
> I have updated the bug so it's high priority and tagged with
> kilo-rc-potential, and added your note from below as a comment on the bug.
>
> It looks like it might be worth a backport so it gets into RC2? Can anyone
> take that bit on please?
>
> Thanks,
> John
>
>
> On Sunday, April 12, 2015, Gary Kotton <gkotton at vmware.com> wrote:
>>
>> Hi,
>> Can a core please take a look at https://review.openstack.org/#/c/171037.
>> The CI is broken due to commit e7ae5bb7fbdd5b79bde8937958dd0a645554a5f0.
>> Thanks
>> Gary
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
--
Davanum Srinivas :: https://twitter.com/dims
------------------------------
Message: 2
Date: Sun, 12 Apr 2015 12:36:51 +0000
From: Gary Kotton <gkotton at vmware.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [Nova] VMware CI
Message-ID: <D150418B.7456D%gkotton at vmware.com>
Content-Type: text/plain; charset="us-ascii"
Thanks!
On 4/12/15, 3:04 PM, "Davanum Srinivas" <davanum at gmail.com> wrote:
>Gary, John,
>
>Just to speed things up, i filed a backport:
>https://review.openstack.org/#/c/172710/
>
>thanks,
>dims
>
>On Sun, Apr 12, 2015 at 4:23 AM, John Garbutt <john at johngarbutt.com>
>wrote:
>> I have updated the bug so it's high priority and tagged with
>> kilo-rc-potential, and added your note from below as a comment on the
>>bug.
>>
>> It looks like it might be worth a backport so it gets into RC2? Can
>>anyone
>> take that bit on please?
>>
>> Thanks,
>> John
>>
>>
>> On Sunday, April 12, 2015, Gary Kotton <gkotton at vmware.com> wrote:
>>>
>>> Hi,
>>> Can a core please take a look at
>>>https://review.openstack.org/#/c/171037.
>>> The CI is broken due to commit
>>>e7ae5bb7fbdd5b79bde8937958dd0a645554a5f0.
>>> Thanks
>>> Gary
>>
>>
>>
>>_________________________________________________________________________
>>_
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
>--
>Davanum Srinivas ::
>https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_dims&d=Aw
>ICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzkWT5jqz9JY
>Bk8YTeq9N3-diTlNj4GyNc&m=O_vdKYuE0xFSaX6xbIHw3qdu0asR94NVcbUKhC9t2vs&s=zv9
>uaaxIRcEZR4SDIuS8EJW7YjE4-2QEZKZtxKcl4ow&e=
>
>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
------------------------------
Message: 3
Date: Sun, 12 Apr 2015 16:36:32 +0000
From: Gregory Haynes <greg at greghaynes.net>
To: openstack-dev <openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [TripleO] Consistent variable
documentation for diskimage-builder elements
Message-ID: <1428855849-sup-8142 at greghaynes0.opus.gah>
Content-Type: text/plain; charset=UTF-8
Excerpts from Clint Byrum's message of 2015-04-08 23:11:29 +0000:
>
> I discussed a format for something similar here:
>
> https://review.openstack.org/#/c/162267/
>
> Perhaps we could merge the effort.
>
> The design and implementation in that might take some time, but if we
> can document the variables at the same time we prepare the inputs for
> isolation, that seems like a winning path forward.
>
The solution presented there would be awesome for not having to document
the variables manually at all - we can do some sphinx plugin magic to
autogen the doc sections and even get some annoying to write out
features like static links for each var (Im sure you knew this, just
spelling it out).
I agree that itd be better to not put a lot of effort into switching all
the README's over right now and instead work on the argument isolation.
My hope is that in the meanwhile new elements we create and possibly
README's we end up editing get moved over to this new format. Then, we
can try and autogen something that is pretty similar when the time
comes.
Now, lets get that arg isolation donw already. ;)
Cheers,
Greg
------------------------------
Message: 4
Date: Sun, 12 Apr 2015 09:38:20 -0700
From: Joshua Harlow <harlowja at outlook.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID: <BLU437-SMTP29438D4D9324695F8F13EFD8F80 at phx.gbl>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Kevin Benton wrote:
> So IIUC tooz would be handling the liveness detection for the agents.
> That would be nice to get ride of that logic in Neutron and just
> register callbacks for rescheduling the dead.
>
> Where does it store that state, does it persist timestamps to the DB
> like Neutron does? If so, how would that scale better? If not, who does
> a given node ask to know if an agent is online or offline when making a
> scheduling decision?
Timestamps are just one way (and likely the most primitive), using redis
(or memcache) key/value and expiry are another (and letting memcache or
redis expire using its own internal algorithms), using zookeeper
ephemeral nodes[1] are another... The point being that its backend
specific and tooz supports varying backends.
>
> However, before (what I assume is) the large code change to implement
> tooz, I would like to quantify that the heartbeats are actually a
> bottleneck. When I was doing some profiling of them on the master branch
> a few months ago, processing a heartbeat took an order of magnitude less
> time (<50ms) than the 'sync routers' task of the l3 agent (~300ms). A
> few query optimizations might buy us a lot more headroom before we have
> to fall back to large refactors.
Sure, always good to avoid prematurely optimizing things...
Although this is relevant for u I think anyway:
https://review.openstack.org/#/c/138607/ (same thing/nearly same in nova)...
https://review.openstack.org/#/c/172502/ (a WIP implementation of the
latter).
[1]
https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Ephemeral+Nodes
>
> Kevin Benton wrote:
>
>
> One of the most common is the heartbeat from each agent. However, I
> don't think we can't eliminate them because they are used to determine
> if the agents are still alive for scheduling purposes. Did you have
> something else in mind to determine if an agent is alive?
>
>
> Put each agent in a tooz[1] group; have each agent periodically
> heartbeat[2], have whoever needs to schedule read the active members of
> that group (or use [3] to get notified via a callback), profit...
>
> Pick from your favorite (supporting) driver at:
>
> http://docs.openstack.org/__developer/tooz/compatibility.__html
> <http://docs.openstack.org/developer/tooz/compatibility.html>
>
> [1]
> http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping
> <http://docs.openstack.org/developer/tooz/compatibility.html#grouping>
> [2]
> https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315
> <https://github.com/openstack/tooz/blob/0.13.1/tooz/coordination.py#L315>
> [3]
> http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes
> <http://docs.openstack.org/developer/tooz/tutorial/group_membership.html#watching-group-changes>
>
>
> ______________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.__openstack.org?subject:__unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
------------------------------
Message: 5
Date: Sun, 12 Apr 2015 13:43:04 -0400
From: Monty Taylor <mordred at inaugust.com>
To: Bernd Bausch <berndbausch at gmail.com>, "'OpenStack Development
Mailing List (not for usage questions)'"
<openstack-dev at lists.openstack.org>,
openstack-docs at lists.openstack.org, openstack-i18n at lists.openstack.org
Cc: 'Jesse Noller' <jesse.noller at rackspace.com>
Subject: Re: [openstack-dev] [OpenStack-docs] What's Up Doc? Apr 10
2015
Message-ID: <552AAEA8.3020409 at inaugust.com>
Content-Type: text/plain; charset=windows-1252
On 04/12/2015 04:16 AM, Bernd Bausch wrote:
> There is nothing like a good rage on a Sunday (yes Sunday) afternoon. Many
> thanks, Monty. You helped me make glance work for my particular case; I will
> limit any further messages to the docs mailing list.
Rage on a Sunday followed up by rage coding:
https://review.openstack.org/172728
I figured I should stop flapping my mouth and write some code.
> For now I will use API v1 (export OS_IMAGE_API_VERSION=1), pending further
> discussions in the install guide team. To me, install guides are more a way
> to enter the OpenStack world than an official installation guide; no need to
> expose newbies including myself to the complexity of v2.
>
> Bernd
>
> -----Original Message-----
> From: Monty Taylor [mailto:mordred at inaugust.com]
> Sent: Sunday, April 12, 2015 6:22 AM
> To: OpenStack Development Mailing List (not for usage questions);
> openstack-docs at lists.openstack.org; openstack-i18n at lists.openstack.org
> Cc: Jesse Noller
> Subject: Re: [OpenStack-docs] [openstack-dev] What's Up Doc? Apr 10 2015
>
> Sorry for top posting - I wasn't subscribed to the doc list before clarkb
> told me about this thread. Warning ... rage coming ... if you don't want to
> read rage on a Saturday, I recommend skipping this email.
>
> a) There may be a doc bug here, but I'm not 100% convinced it's a doc bug -
> I'll try to characterize it in this way:
>
> "As a user, I do not know what version of glance I am or should be
> interacting with"
>
> That part of this is about the default version that python-glanceclient may
> or may not use and what version you may or may not need to provide on the
> command line is a badness I'll get to in a second - but a clear "so you want
> to upload an image, here's what you need to know" is, I think, what Bernd
> was looking for
>
> b) Glance is categorically broken in all regards related to this topic.
> This thing is the most painful and most broken of everything that exists in
> OpenStack. It is the source of MONTHS of development to deal with it in
> Infra, and even the workarounds are terrible.
>
> Let me expand:
>
> glance image-upload MAY OR MAY NOT work on your cloud, and there is
> absolutely no way you as a user can tell. You just have to try and find out.
>
> IF glance image-upload does not work for you, it may be because of two
> things, neither of which are possible for you as a user to find out:
>
> Either:
>
> - Your cloud has decided to not enable image upload permissions in their
> policy.json file, which is a completely opaque choice that you as a user
> have no way of finding out. If this is the case you have no recourse, sorry.
> - Your cloud has deployed a recent glance and has configured it for glance
> v2 and has configured it in the policy.json file to ONLY allow v2 and to
> disallow image-upload
>
> If the second is true, which you have no way to discover except for trying,
> what you need to do is:
>
> - upload the image to swift
> - glance task-create --type=import --input='{"import_from":
> "$PATH_TO_IMAGE_IN_SWIFT", "image_properties" : {"name": "Human Readable
> Image Name"}}'
>
> Yes, you do have to pass JSON on the command line, because BONGHITS (/me
> glares at the now absent Brian Waldon with withering disdain for having
> inflicted such an absolutely craptastic API on the world.)
>
> Then, you need to poll glance task-status for the status of the import_from
> task until your image has imported.
>
> c) The python-glanceclient command line client should encapsulate that
> ridiculous logic for you, but it does not
>
> d) It should be possible to discover from the cloud which of the approaches
> you should take, but it isn't
>
> Now - I'm honestly not sure how far the docs team should take working around
> this - because fully describing how to successfully upload an image without
> resorting to calling people names is impossible - but is it really the Docs
> team job to make an impossible API seem user friendly? Or, should we not
> treat this as a docs bug and instead treat it as a Glance bug and demand a
> v3 API that rolls back the task interface?
>
> I vote for the latter.
>
> BTW - the shade library encodes as much of the logic above as it can.
> That it exists makes me sad.
>
> Monty
>
> On Sat, Apr 11, 2015 at 10:50 AM, Matt Kassawara <mkassawara at gmail.com>
> wrote:
>
>> Sounds like a problem with one or more packages (perhaps
>> python-glanceclient?) because that command using the source version
>> (not
>> packages) returns the normal list of help items. Maybe try the source
>> version using "pip install python-glanceclient"?
>>
>> On Sat, Apr 11, 2015 at 5:55 AM, Bernd Bausch <berndbausch at
>> gmail.com>
>> wrote:
>>
>>> glance help image-create. Sorry for being vague.
>>>
>>> When running glance with the parameters from the install guide (the
>>> trunk version), I am told that I am not doing it correctly; I don't
>>> have the precise message handy.
>>>
>>>
>>>
>>> My fear is that I will hit similar problems later. You solving the
>>> problem would be nice but not enough :)
>>>
>>>
>>>
>>> *From:* Matt Kassawara [mailto:mkassawara at gmail.com]
>>> *Sent:* Saturday, April 11, 2015 1:59 PM
>>> *To:* Bernd Bausch
>>> *Cc:* openstack-docs at lists.openstack.org
>>>
>>> *Subject:* Re: [OpenStack-docs] [install-guide] RE: What's Up Doc?
>>> Apr
>>> 10 2015
>>>
>>>
>>>
>>> When you run "glance help image-create" or just "glance image-create"
>>> with no arguments?
>>>
>>>
>>>
>>> On Fri, Apr 10, 2015 at 11:45 PM, Bernd Bausch <berndbausch at
>>> gmail.com>
>>> wrote:
>>>
>>> This is what I get when running glance image-create:
>>>
>>>
>>>
>>> usage: glance image-create [--property <key=value>]
>>> [--file <FILE>]
>>>
>>>
>>> [--progress]
>>>
>>>
>>> <unavailable>
>>>
>>>
>>>
>>> Create a new image.
>>>
>>>
>>>
>>> Positional arguments:
>>>
>>> <unavailable> Please run with connection
>>> parameters set to retrieve
>>>
>>> the schema for
>>> generating help for this command
>>>
>>>
>>>
>>> So I wonder how I can get to the bottom of this.
>>>
>>>
>>>
>>> *From:* Matt Kassawara [mailto:mkassawara at gmail.com]
>>> *Sent:* Saturday, April 11, 2015 1:39 PM
>>> *To:* Bernd Bausch; openstack-docs at lists.openstack.org
>>> *Subject:* Re: [OpenStack-docs] [install-guide] RE: What's Up Doc?
>>> Apr
>>> 10 2015
>>>
>>>
>>>
>>> I'd use the conventional python-*client for all services except
>>> keystone because the Openstack client doesn't seem very complete for
>>> them. If
> you're
>>> using the glance client, it defaults to the v1 API and the commands
>>> from the Juno installation guide should work. If you use the v2 API,
>>> one thing changes with how to set public/private visibility.
>>>
>>>
>>>
>>> On Fri, Apr 10, 2015 at 8:11 PM, Bernd Bausch <berndbausch at
>>> gmail.com>
>>> wrote:
>>>
>>> Regarding the installation guide, I need some advice. Perhaps the
>>> docs community can help?
>>>
>>> I am trying to install Kilo on yum-based systems using a repo from
>>> the RDO project. I have hit a few roadblocks that I have been able to
>>> deal with, but I am unsure what to do with the current one.
>>>
>>> My questions are: Is it appropriate to ask developers about the
>>> intended way of doing things, if the old ways don't work anymore? If
>>> yes, what are the best channels - chat, dev mailing list, personal
>>> email, .? If no,
> what
>>> else can I do? Do developers make such changes public somewhere?
>>>
>>> Below is the problem I am currently trying to solve. **Note** that I
>>> am including it as an illustration what I am struggling with (more
>>> problems will show up as I continue working on this); I am not asking
>>> you to solve this particular problem for me.
>>>
>>> So far, to upload an image to Glance, the "glance image-create"
>>> command is used. This command doesn't work anymore as in the past,
>>> and I don't understand what the "glance help image-create" is trying
>>> to say. On the other hand, I haven't found an equivalent command in the
> new "openstack"
>>> CLI client. So my question is - what is the correct way to upload an
> image
>>> these days.
>>>
>>> Have a great weekend,
>>>
>>> Bernd
>>>
>>> From: Anne Gentle [mailto:annegentle at justwriteclick.com]
>>> Sent: Saturday, April 11, 2015 12:24 AM
>>> To: openstack-docs at lists.openstack.org; OpenStack Development
>>> Mailing List; openstack-i18n at lists.openstack.org
>>> Cc: Jesse Noller
>>> Subject: [OpenStack-docs] What's Up Doc? Apr 10 2015
>>>
>>> Hi all,
>>>
>>> As you probably saw from PTL nominations last week, I'm happy to hand
>>> the docs PTL baton to Lana Brindley! I loved leading this group and
>>> thank you all for supporting me. Thank you Lana for your willingness
>>> to lead. I'm still here to bring us to the Kilo release, so this
>>> week's What's Up Doc brings sharp focus requests to everyone to work
>>> on docs. These are
> the top
>>> priorities that we all need to work on - devs, writers, testers,
>>> gaters, everyone.
>>>
>>> 1. Bug triaging and fixing, especially for openstack-manuals. There
>>> are nearly 300 DocImpact bugs logged that we need developers to
>>> circle
> back to.
>>> With nearly 600 bugs overall, we need lots of focus here. To that
>>> end, I propose we hold a bug triage day. I'll send details in a separate
> email.
>>>
>>> 2. Install Guide testing and reviewing. The Install Guide team has a
>>> published spec that will help reviewers see what's changing with the
>>> Kilo Install guide:
>>>
> http://specs.openstack.org/openstack/docs-specs/specs/kilo/installguide-kilo
> .html
>>> Join them for weekly meetings Tuesdays at at 13:00 UTC (8:00 AM US
> CDT) in
>>> Google Hangout:
>>>
> https://plus.google.com/hangouts/_/calendar/a2FyaW4ua2F0aG9kZUBnbWFpbC5jb20.
> jj2lu2nbj71a0dan11vatdav3k
>>>
>>> If you do nothing else but these two focus areas we'll be in good shape.
>>> There are other activities going on leading up to Vancouver but those
>>> two are top priorities.
>>>
>>> _RST Migration_
>>>
>>> We are working to resolve translation tool expectations with the i18N
>>> team. I want to publish the RST-based English End User Guide and
> Admin User
>>> Guide once we're all comfortable with the way forward. Daisy will
>>> discuss the implications at the next i18N team meeting Thursday at
>>> 0800 UTC, and we'll implement and communicate the plan.
>>>
>>> _Networking Guide_
>>>
>>> Next on the list is Networking Guide testing and reviewing. The
>>> Networking Guide team has a talk in Vancouver and needs to get their
> guide
>>> in shape for publishing. The neutron team is holding a doc day April 23.
>>> Please join in -- they'll post details in their notes.
>>>
>>> _First App Tutorial_
>>>
>>> There's also the First Application Tutorial that needs to finish the
>>> spec and needs an editing cleanup prior to publishing. Ideally that
>>> will
> happen
>>> before Vancouver, we need to get it to the finish line.
>>>
>>> _HA Guide_
>>>
>>> With everything else going on we need an updated spec for the HA
>>> Guide - the wiki page isn't enough. Based on this week's doc team
>>> meeting it
> sounds
>>> like that can go to RST as well but we need it in the spec so we can
>>> plan and cross-coordinate with the other priorities leading up to
> Vancouver.
>>>
>>> _Vancouver Summit _
>>> The docs team has four Fishbowl time slots, 2 Workroom slots, and 1
>>> Meetup allocated now. If you'd like to discuss a cross-project idea,
> please
>>> use the form to suggest new ideas:
>>> http://goo.gl/forms/S69HM6XEeb. You can see the current suggestions
>>> already posted here:
>>>
>>>
> https://docs.google.com/spreadsheets/d/1vCTZBJKCMZ2xBhglnuK3ciKo3E8UMFo5S5lm
> IAYMCSE/edit?usp=sharing
>>>
>>> Lana or I will send out an etherpad in the next week or so with topic
>>> suggestions for our allocation.
>>>
>>> Thanks,
>>> Anne
>>>
>>> --
>>> Anne Gentle
>>> annegentle at justwriteclick.com
>>>
>>> _______________________________________________
>>> OpenStack-docs mailing list
>>> OpenStack-docs at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs
>>>
>>>
>>>
>>>
>>>
>>
>>
>> _______________________________________________
>> OpenStack-docs mailing list
>> OpenStack-docs at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs
>>
>>
>
> _______________________________________________
> OpenStack-docs mailing list
> OpenStack-docs at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs
>
>
------------------------------
Message: 6
Date: Sun, 12 Apr 2015 12:40:07 -0700
From: Kevin Benton <blak111 at gmail.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID:
<CAO_F6JMFRGSGAQu_CyXGZZmkuHmy8vuO2gCUWHcL+DMeuyxd1Q at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
>I assumed that all agents are connected to same IP address of RabbitMQ,
then the connection will exceed the port ranges limitation.
Only if the clients are all using the same IP address. If connections
weren't scoped by source IP, busy servers would be completely unreliable
because clients would keep having source port collisions.
For example, the following is a netstat output from a server with two
connections to a service running on port 4000 with both clients using
source port 50000: http://paste.openstack.org/show/203211/
>the client should be aware of the cluster member failure, and reconnect to
other survive member. No such mechnism has been implemented yet.
If I understand what you are suggesting, it already has been implemented
that way. The neutron agents and servers can be configured with multiple
rabbitmq servers and they will cycle through the list whenever there is a
failure.
The only downside to that approach is that every neutron agent and server
has to be configured with every rabbitmq server address. This gets tedious
to manage if you want to add cluster members dynamically so using a load
balancer can help relieve that.
Hi, Kevin,
I assumed that all agents are connected to same IP address of RabbitMQ,
then the connection will exceed the port ranges limitation.
For a RabbitMQ cluster, for sure the client can connect to any one of
member in the cluster, but in this case, the client has to be designed in
fail-safe manner: the client should be aware of the cluster member failure,
and reconnect to other survive member. No such mechnism has
been implemented yet.
Other way is to use LVS or DNS based like load balancer, or something else.
If you put one load balancer ahead of a cluster, then we have to take care
of the port number limitation, there are so many agents will require
connection concurrently, 100k level, and the requests can not be rejected.
Best Regards
Chaoyi Huang ( joehuang )
------------------------------
*From:* Kevin Benton [blak111 at gmail.com]
*Sent:* 12 April 2015 9:59
*To:* OpenStack Development Mailing List (not for usage questions)
*Subject:* Re: [openstack-dev] [neutron] Neutron scaling datapoints?
The TCP/IP stack keeps track of connections as a combination of IP + TCP
port. The two byte port limit doesn't matter unless all of the agents are
connecting from the same IP address, which shouldn't be the case unless
compute nodes connect to the rabbitmq server via one IP address running
port address translation.
Either way, the agents don't connect directly to the Neutron server, they
connect to the rabbit MQ cluster. Since as many Neutron server processes
can be launched as necessary, the bottlenecks will likely show up at the
messaging or DB layer.
On Sat, Apr 11, 2015 at 6:46 PM, joehuang <joehuang at huawei.com> wrote:
> As Kevin talking about agents, I want to remind that in TCP/IP stack,
> port ( not Neutron Port ) is a two bytes field, i.e. port ranges from 0 ~
> 65535, supports maximum 64k port number.
>
>
>
> " above 100k managed node " means more than 100k L2 agents/L3 agents...
> will be alive under Neutron.
>
>
>
> Want to know the detail design how to support 99.9% possibility for
> scaling Neutron in this way, and PoC and test would be a good support for
> this idea.
>
>
>
> "I'm 99.9% sure, for scaling above 100k managed node,
> we do not really need to split the openstack to multiple smaller openstack,
> or use significant number of extra controller machine."
>
>
>
> Best Regards
>
>
>
> Chaoyi Huang ( joehuang )
>
>
> ------------------------------
> *From:* Kevin Benton [blak111 at gmail.com]
> *Sent:* 11 April 2015 12:34
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [neutron] Neutron scaling datapoints?
>
> Which periodic updates did you have in mind to eliminate? One of the
> few remaining ones I can think of is sync_routers but it would be great if
> you can enumerate the ones you observed because eliminating overhead in
> agents is something I've been working on as well.
>
> One of the most common is the heartbeat from each agent. However, I
> don't think we can't eliminate them because they are used to determine if
> the agents are still alive for scheduling purposes. Did you have something
> else in mind to determine if an agent is alive?
>
> On Fri, Apr 10, 2015 at 2:18 AM, Attila Fazekas <afazekas at redhat.com>
> wrote:
>
>> I'm 99.9% sure, for scaling above 100k managed node,
>> we do not really need to split the openstack to multiple smaller
>> openstack,
>> or use significant number of extra controller machine.
>>
>> The problem is openstack using the right tools SQL/AMQP/(zk),
>> but in a wrong way.
>>
>> For example.:
>> Periodic updates can be avoided almost in all cases
>>
>> The new data can be pushed to the agent just when it needed.
>> The agent can know when the AMQP connection become unreliable (queue or
>> connection loose),
>> and needs to do full sync.
>> https://bugs.launchpad.net/neutron/+bug/1438159
>>
>> Also the agents when gets some notification, they start asking for
>> details via the
>> AMQP -> SQL. Why they do not know it already or get it with the
>> notification ?
>>
>>
>> ----- Original Message -----
>> > From: "Neil Jerram" <Neil.Jerram at metaswitch.com>
>> > To: "OpenStack Development Mailing List (not for usage questions)" <
>> openstack-dev at lists.openstack.org>
>> > Sent: Thursday, April 9, 2015 5:01:45 PM
>> > Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
>> >
>> > Hi Joe,
>> >
>> > Many thanks for your reply!
>> >
>> > On 09/04/15 03:34, joehuang wrote:
>> > > Hi, Neil,
>> > >
>> > > From theoretic, Neutron is like a "broadcast" domain, for example,
>> > > enforcement of DVR and security group has to touch each regarding
>> host
>> > > where there is VM of this project resides. Even using SDN
>> controller, the
>> > > "touch" to regarding host is inevitable. If there are plenty of
>> physical
>> > > hosts, for example, 10k, inside one Neutron, it's very hard to
>> overcome
>> > > the "broadcast storm" issue under concurrent operation, that's the
>> > > bottleneck for scalability of Neutron.
>> >
>> > I think I understand that in general terms - but can you be more
>> > specific about the broadcast storm? Is there one particular message
>> > exchange that involves broadcasting? Is it only from the server to
>> > agents, or are there 'broadcasts' in other directions as well?
>> >
>> > (I presume you are talking about control plane messages here, i.e.
>> > between Neutron components. Is that right? Obviously there can also be
>> > broadcast storm problems in the data plane - but I don't think that's
>> > what you are talking about here.)
>> >
>> > > We need layered architecture in Neutron to solve the "broadcast
>> domain"
>> > > bottleneck of scalability. The test report from OpenStack cascading
>> shows
>> > > that through layered architecture "Neutron cascading", Neutron can
>> > > supports up to million level ports and 100k level physical hosts. You
>> can
>> > > find the report here:
>> > >
>> http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers
>> >
>> > Many thanks, I will take a look at this.
>> >
>> > > "Neutron cascading" also brings extra benefit: One cascading Neutron
>> can
>> > > have many cascaded Neutrons, and different cascaded Neutron can
>> leverage
>> > > different SDN controller, maybe one is ODL, the other one is
>> OpenContrail.
>> > >
>> > > ----------------Cascading Neutron-------------------
>> > > / \
>> > > --cascaded Neutron-- --cascaded Neutron-----
>> > > | |
>> > > ---------ODL------ ----OpenContrail--------
>> > >
>> > >
>> > > And furthermore, if using Neutron cascading in multiple data centers,
>> the
>> > > DCI controller (Data center inter-connection controller) can also be
>> used
>> > > under cascading Neutron, to provide NaaS ( network as a service )
>> across
>> > > data centers.
>> > >
>> > > ---------------------------Cascading Neutron--------------------------
>> > > / | \
>> > > --cascaded Neutron-- -DCI controller- --cascaded Neutron-----
>> > > | | |
>> > > ---------ODL------ | ----OpenContrail--------
>> > > |
>> > > --(Data center 1)-- --(DCI networking)-- --(Data center 2)--
>> > >
>> > > Is it possible for us to discuss this in OpenStack Vancouver summit?
>> >
>> > Most certainly, yes. I will be there from mid Monday afternoon through
>> > to end Friday. But it will be my first summit, so I have no idea yet as
>> > to how I might run into you - please can you suggest!
>> >
>> > > Best Regards
>> > > Chaoyi Huang ( Joe Huang )
>> >
>> > Regards,
>> > Neil
>> >
>> >
>> __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
> Kevin Benton
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
--
Kevin Benton
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150412/ea01d543/attachment-0001.html>
------------------------------
Message: 7
Date: Sun, 12 Apr 2015 12:51:30 -0700
From: Kevin Benton <blak111 at gmail.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID:
<CAO_F6JM61uqXNY_ebNNXKxFNV5-C7wr7A3LSZC_0RY+NMQKcdw at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
>Timestamps are just one way (and likely the most primitive), using redis
(or memcache) key/value and expiry are another (and letting memcache or
redis expire using its own internal algorithms), using zookeeper ephemeral
nodes[1] are another... The point being that its backend specific and tooz
supports varying backends.
Very cool. Is the backend completely transparent so a deployer could choose
a service they are comfortable maintaining, or will that change the
properties WRT to resiliency of state on node restarts, partitions, etc?
The Nova implementation of Tooz seemed pretty straight-forward, although it
looked like it had pluggable drivers for service management already. Before
I dig into it much further I'll file a spec on the Neutron side to see if I
can get some other cores onboard to do the review work if I push a change
to tooz.
On Sun, Apr 12, 2015 at 9:38 AM, Joshua Harlow <harlowja at outlook.com> wrote:
> Kevin Benton wrote:
>
>> So IIUC tooz would be handling the liveness detection for the agents.
>> That would be nice to get ride of that logic in Neutron and just
>> register callbacks for rescheduling the dead.
>>
>> Where does it store that state, does it persist timestamps to the DB
>> like Neutron does? If so, how would that scale better? If not, who does
>> a given node ask to know if an agent is online or offline when making a
>> scheduling decision?
>>
>
> Timestamps are just one way (and likely the most primitive), using redis
> (or memcache) key/value and expiry are another (and letting memcache or
> redis expire using its own internal algorithms), using zookeeper ephemeral
> nodes[1] are another... The point being that its backend specific and tooz
> supports varying backends.
>
>
>> However, before (what I assume is) the large code change to implement
>> tooz, I would like to quantify that the heartbeats are actually a
>> bottleneck. When I was doing some profiling of them on the master branch
>> a few months ago, processing a heartbeat took an order of magnitude less
>> time (<50ms) than the 'sync routers' task of the l3 agent (~300ms). A
>> few query optimizations might buy us a lot more headroom before we have
>> to fall back to large refactors.
>>
>
> Sure, always good to avoid prematurely optimizing things...
>
> Although this is relevant for u I think anyway:
>
> https://review.openstack.org/#/c/138607/ (same thing/nearly same in
> nova)...
>
> https://review.openstack.org/#/c/172502/ (a WIP implementation of the
> latter).
>
> [1] https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#
> Ephemeral+Nodes
>
>
>> Kevin Benton wrote:
>>
>>
>> One of the most common is the heartbeat from each agent. However, I
>> don't think we can't eliminate them because they are used to determine
>> if the agents are still alive for scheduling purposes. Did you have
>> something else in mind to determine if an agent is alive?
>>
>>
>> Put each agent in a tooz[1] group; have each agent periodically
>> heartbeat[2], have whoever needs to schedule read the active members of
>> that group (or use [3] to get notified via a callback), profit...
>>
>> Pick from your favorite (supporting) driver at:
>>
>> http://docs.openstack.org/__developer/tooz/compatibility.__html
>> <http://docs.openstack.org/developer/tooz/compatibility.html>
>>
>> [1]
>> http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping
>> <http://docs.openstack.org/developer/tooz/compatibility.html#grouping>
>> [2]
>> https://github.com/openstack/__tooz/blob/0.13.1/tooz/__
>> coordination.py#L315
>> <https://github.com/openstack/tooz/blob/0.13.1/tooz/coordination.py#L315>
>> [3]
>> http://docs.openstack.org/__developer/tooz/tutorial/group_
>> __membership.html#watching-__group-changes
>> <http://docs.openstack.org/developer/tooz/tutorial/group_
>> membership.html#watching-group-changes>
>>
>>
>> ____________________________________________________________
>> __________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.__openstack.org?subject:__unsubscribe
>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>> ____________________________________________________________
>> ______________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:
>> unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
--
Kevin Benton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150412/c5c3d2dc/attachment-0001.html>
------------------------------
Message: 8
Date: Mon, 13 Apr 2015 10:43:57 +1200
From: Robert Collins <robertc at robertcollins.net>
To: OpenStack Development Mailing List
<openstack-dev at lists.openstack.org>
Subject: [openstack-dev] [all][pbr] splitting our deployment vs
install dependencies
Message-ID:
<CAJ3HoZ2JLwPJUBRgTLyBZ-z00sVO4TyOupmR4LprjkNMHg9o0g at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Right now we do something that upstream pip considers wrong: we make
our requirements.txt be our install_requires.
Upstream there are two separate concepts.
install_requirements, which are meant to document what *must* be
installed to import the package, and should encode any mandatory
version constraints while being as loose as otherwise possible. E.g.
if package A depends on package B version 1.5 or above, it should say
B>=1.5 in A's install_requires. They should not specify maximum
versions except when that is known to be a problem: they shouldn't
borrow trouble.
deploy requirements - requirements.txt - which are meant to be *local
to a deployment*, and are commonly expected to specify very narrow (or
even exact fit) versions.
What pbr, which nearly if not all OpenStack projects use, does, is to
map the contents of requirements.txt into install_requires. And then
we use the same requirements.txt in our CI to control whats deployed
in our test environment[*]. and there we often have tight constraints
like seen here -
http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n63
I'd like to align our patterns with those of upstream, so that we're
not fighting our tooling so much.
Concretely, I think we need to:
- teach pbr to read in install_requires from setup.cfg, not requirements.txt
- when there are requirements in setup.cfg, stop reading requirements.txt
- separate out the global intall_requirements from the global CI
requirements, and update our syncing code to be aware of this
Then, setup.cfg contains more open requirements suitable for being on
PyPI, requirements.txt is the local CI set we know works - and can be
much more restrictive as needed.
Thoughts? If there's broad apathy-or-agreement I can turn this into a
spec for fine coverage of ramifications and corner cases.
-Rob
--
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud
------------------------------
Message: 9
Date: Sun, 12 Apr 2015 19:12:38 -0400
From: Monty Taylor <mordred at inaugust.com>
To: openstack-dev at lists.openstack.org
Subject: Re: [openstack-dev] [all][pbr] splitting our deployment vs
install dependencies
Message-ID: <552AFBE6.4010606 at inaugust.com>
Content-Type: text/plain; charset=windows-1252
On 04/12/2015 06:43 PM, Robert Collins wrote:
> Right now we do something that upstream pip considers wrong: we make
> our requirements.txt be our install_requires.
>
> Upstream there are two separate concepts.
>
> install_requirements, which are meant to document what *must* be
> installed to import the package, and should encode any mandatory
> version constraints while being as loose as otherwise possible. E.g.
> if package A depends on package B version 1.5 or above, it should say
> B>=1.5 in A's install_requires. They should not specify maximum
> versions except when that is known to be a problem: they shouldn't
> borrow trouble.
>
> deploy requirements - requirements.txt - which are meant to be *local
> to a deployment*, and are commonly expected to specify very narrow (or
> even exact fit) versions.
tl;dr - I'm mostly in support of what you're saying - but I'm going to
bludgeon it some.
To be either fair or unfair, depending on how you look at it - some
people upstream consider those two to be a pattern, but it is not
encoded anywhere except in hidden lore that is shared between secret
people. Upstream's tools have bumpkiss for support for this, and if we
hadn't drawn a line in the sand encoding SOME behavior there would still
be nothing.
Nor, btw, is it the right split. It optimizes for the wrong thing.
rust gets it wright. There is a Cargo.toml and a Cargo.lock, which are
understood by the tooling in a manner similar to what you have
described, and it is not just understood but DOCUMENTED that an
_application_ should ship with a Cargo.lock and a _library_ should not.
Without the library/application distinction, the effort in
differentiating is misplaced, I believe - because it's libraries that
need flexible depends - and applications where the specific set of
depends that were tested in CI become important to consumers.
> What pbr, which nearly if not all OpenStack projects use, does, is to
> map the contents of requirements.txt into install_requires. And then
> we use the same requirements.txt in our CI to control whats deployed
> in our test environment[*]. and there we often have tight constraints
> like seen here -
> http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n63
That is, btw, because that's what the overwhelming majority of consumers
assume those files mean. I take "overwhelming majority" from the days
when we had files but did not process them automatically and everyone
was confused.
> I'd like to align our patterns with those of upstream, so that we're
> not fighting our tooling so much.
Ok. I mean, they don't have a better answer, but if it makes "python"
hate us less, sweet.
> Concretely, I think we need to:
> - teach pbr to read in install_requires from setup.cfg, not requirements.txt
> - when there are requirements in setup.cfg, stop reading requirements.txt
> - separate out the global intall_requirements from the global CI
> requirements, and update our syncing code to be aware of this
>
> Then, setup.cfg contains more open requirements suitable for being on
> PyPI, requirements.txt is the local CI set we know works - and can be
> much more restrictive as needed.
>
> Thoughts? If there's broad apathy-or-agreement I can turn this into a
> spec for fine coverage of ramifications and corner cases.
I'm concerned that it adds a layer of difference that is confusing to
people for the sole benefit of pleasing someone else's pedantic worldview.
I'm also concerned that dstufft is actively wanting to move towards a
world where the build tooling is not needed or used as part of the
install pipeline (metadata 2.0) -- so I'm concerned that we're aligning
with a pattern that isn't very robust and isn't supported by tooling
directly and that we're going to need to change understood usage
patterns across a large developer based to chase something that _still_
isn't going to be "how people do it"
I'm concerned that "how people do it" is a myth not worth chasing.
I'm not _opposed_ to making this richer and more useful for people. I
just don't know what's broken currently for us.
Monty
------------------------------
Message: 10
Date: Mon, 13 Apr 2015 10:01:44 +1000
From: James Polley <jp at jamezpolley.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [all][pbr] splitting our deployment vs
install dependencies
Message-ID:
<CAPtRfUEoLc0KQoqPaHffxXevGwCiXaExm3Kx2oJWzNGFjL3NWA at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
On Mon, Apr 13, 2015 at 9:12 AM, Monty Taylor <mordred at inaugust.com> wrote:
> On 04/12/2015 06:43 PM, Robert Collins wrote:
> > Right now we do something that upstream pip considers wrong: we make
> > our requirements.txt be our install_requires.
> >
> > Upstream there are two separate concepts.
> >
> > install_requirements, which are meant to document what *must* be
> > installed to import the package, and should encode any mandatory
> > version constraints while being as loose as otherwise possible. E.g.
> > if package A depends on package B version 1.5 or above, it should say
> > B>=1.5 in A's install_requires. They should not specify maximum
> > versions except when that is known to be a problem: they shouldn't
> > borrow trouble.
> >
> > deploy requirements - requirements.txt - which are meant to be *local
> > to a deployment*, and are commonly expected to specify very narrow (or
> > even exact fit) versions.
>
That sounds, to me, very similar to a discussion we had a few weeks ago in
the context of our stable branches.
In that context, we have two competing requirements. One requirement is
that our CI system wants a very tightly pinned requirements, as do
downstream CI systems and deployers that want to test and deploy exact
known-tested versions of things. On the other hand, downstream distributors
(including OS packagers) need to balance OpenStack's version requirements
with version requirements from all the other packages in their
distribution; the tighter the requirements we list are, the harder it is
for the requirements to work with the requirements of other packages in the
distribution.
> tl;dr - I'm mostly in support of what you're saying - but I'm going to
> bludgeon it some.
>
> To be either fair or unfair, depending on how you look at it - some
> people upstream consider those two to be a pattern, but it is not
> encoded anywhere except in hidden lore that is shared between secret
> people. Upstream's tools have bumpkiss for support for this, and if we
> hadn't drawn a line in the sand encoding SOME behavior there would still
> be nothing.
>
> Nor, btw, is it the right split. It optimizes for the wrong thing.
>
> rust gets it wright. There is a Cargo.toml and a Cargo.lock, which are
> understood by the tooling in a manner similar to what you have
> described, and it is not just understood but DOCUMENTED that an
> _application_ should ship with a Cargo.lock and a _library_ should not.
>
This sounds similar to a solution that was proposed for the stable
branches: a requirements.in with mandatory version constraints while being
as loose as otherwise possible, which is used to generate a
requirements.txt which has the "local to deployment" exact versions that
are used in our CI. The details of the proposal are in
https://review.openstack.org/#/c/161047/
> Without the library/application distinction, the effort in
> differentiating is misplaced, I believe - because it's libraries that
> need flexible depends - and applications where the specific set of
> depends that were tested in CI become important to consumers.
>
> > What pbr, which nearly if not all OpenStack projects use, does, is to
> > map the contents of requirements.txt into install_requires. And then
> > we use the same requirements.txt in our CI to control whats deployed
> > in our test environment[*]. and there we often have tight constraints
> > like seen here -
> >
> http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n63
>
> That is, btw, because that's what the overwhelming majority of consumers
> assume those files mean. I take "overwhelming majority" from the days
> when we had files but did not process them automatically and everyone
> was confused.
>
> > I'd like to align our patterns with those of upstream, so that we're
> > not fighting our tooling so much.
>
> Ok. I mean, they don't have a better answer, but if it makes "python"
> hate us less, sweet.
>
> > Concretely, I think we need to:
> > - teach pbr to read in install_requires from setup.cfg, not
> requirements.txt
> > - when there are requirements in setup.cfg, stop reading
> requirements.txt
> > - separate out the global intall_requirements from the global CI
> > requirements, and update our syncing code to be aware of this
> >
> > Then, setup.cfg contains more open requirements suitable for being on
> > PyPI, requirements.txt is the local CI set we know works - and can be
> > much more restrictive as needed.
> >
> > Thoughts? If there's broad apathy-or-agreement I can turn this into a
> > spec for fine coverage of ramifications and corner cases.
>
> I'm concerned that it adds a layer of difference that is confusing to
> people for the sole benefit of pleasing someone else's pedantic worldview.
>
> I'm also concerned that dstufft is actively wanting to move towards a
> world where the build tooling is not needed or used as part of the
> install pipeline (metadata 2.0) -- so I'm concerned that we're aligning
> with a pattern that isn't very robust and isn't supported by tooling
> directly and that we're going to need to change understood usage
> patterns across a large developer based to chase something that _still_
> isn't going to be "how people do it"
>
> I'm concerned that "how people do it" is a myth not worth chasing.
>
> I'm not _opposed_ to making this richer and more useful for people. I
> just don't know what's broken currently for us.
>
To be clear, I don't mean to suggest that the solution proposed in
https://review.openstack.org/#/c/161047/ is necessarily the correct
solution to this problem - but I do think that it is illustrative of at
last one thing that's currently broken for us.
> Monty
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/4e267c09/attachment-0001.html>
------------------------------
Message: 11
Date: Sun, 12 Apr 2015 20:53:50 -0400
From: Monty Taylor <mordred at inaugust.com>
To: openstack-dev at lists.openstack.org
Subject: Re: [openstack-dev] [all][pbr] splitting our deployment vs
install dependencies
Message-ID: <552B139E.5010007 at inaugust.com>
Content-Type: text/plain; charset=windows-1252
On 04/12/2015 08:01 PM, James Polley wrote:
> On Mon, Apr 13, 2015 at 9:12 AM, Monty Taylor <mordred at inaugust.com> wrote:
>
>> On 04/12/2015 06:43 PM, Robert Collins wrote:
>>> Right now we do something that upstream pip considers wrong: we make
>>> our requirements.txt be our install_requires.
>>>
>>> Upstream there are two separate concepts.
>>>
>>> install_requirements, which are meant to document what *must* be
>>> installed to import the package, and should encode any mandatory
>>> version constraints while being as loose as otherwise possible. E.g.
>>> if package A depends on package B version 1.5 or above, it should say
>>> B>=1.5 in A's install_requires. They should not specify maximum
>>> versions except when that is known to be a problem: they shouldn't
>>> borrow trouble.
>>>
>>> deploy requirements - requirements.txt - which are meant to be *local
>>> to a deployment*, and are commonly expected to specify very narrow (or
>>> even exact fit) versions.
>>
>
> That sounds, to me, very similar to a discussion we had a few weeks ago in
> the context of our stable branches.
>
> In that context, we have two competing requirements. One requirement is
> that our CI system wants a very tightly pinned requirements, as do
> downstream CI systems and deployers that want to test and deploy exact
> known-tested versions of things. On the other hand, downstream distributors
> (including OS packagers) need to balance OpenStack's version requirements
> with version requirements from all the other packages in their
> distribution; the tighter the requirements we list are, the harder it is
> for the requirements to work with the requirements of other packages in the
> distribution.
This is not accurate. During distro packaging activities, pbr does not
process dependencies at all. So no matter how we pin things in
OpenStack, it does not make it harder for the distros.
>> tl;dr - I'm mostly in support of what you're saying - but I'm going to
>> bludgeon it some.
>>
>> To be either fair or unfair, depending on how you look at it - some
>> people upstream consider those two to be a pattern, but it is not
>> encoded anywhere except in hidden lore that is shared between secret
>> people. Upstream's tools have bumpkiss for support for this, and if we
>> hadn't drawn a line in the sand encoding SOME behavior there would still
>> be nothing.
>>
>> Nor, btw, is it the right split. It optimizes for the wrong thing.
>>
>> rust gets it wright. There is a Cargo.toml and a Cargo.lock, which are
>> understood by the tooling in a manner similar to what you have
>> described, and it is not just understood but DOCUMENTED that an
>> _application_ should ship with a Cargo.lock and a _library_ should not.
>>
>
> This sounds similar to a solution that was proposed for the stable
> branches: a requirements.in with mandatory version constraints while being
> as loose as otherwise possible, which is used to generate a
> requirements.txt which has the "local to deployment" exact versions that
> are used in our CI. The details of the proposal are in
> https://review.openstack.org/#/c/161047/
I disagree with this proposal. It's also not helping any users. Because
of what I said above, there is no flexibility that we lose downstream by
being strict and pedantic with our versions. So, having the "lose" and
the "strict" file just gets us two files and doubles the confusion.
Having a list of what we know the state to be is great. We should give
that to users. If they want to use something other than pip to install,
awesome - the person in charge of curating that content can test the
version interactions in their environment.
What we have in the gate is the thing that produces the artifacts that
someone installing using the pip tool would get. Shipping anything with
those artifacts other that a direct communication of what we tested is
just mean to our end users.
>
>> Without the library/application distinction, the effort in
>> differentiating is misplaced, I believe - because it's libraries that
>> need flexible depends - and applications where the specific set of
>> depends that were tested in CI become important to consumers.
>>
>>> What pbr, which nearly if not all OpenStack projects use, does, is to
>>> map the contents of requirements.txt into install_requires. And then
>>> we use the same requirements.txt in our CI to control whats deployed
>>> in our test environment[*]. and there we often have tight constraints
>>> like seen here -
>>>
>> http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n63
>>
>> That is, btw, because that's what the overwhelming majority of consumers
>> assume those files mean. I take "overwhelming majority" from the days
>> when we had files but did not process them automatically and everyone
>> was confused.
>>
>>> I'd like to align our patterns with those of upstream, so that we're
>>> not fighting our tooling so much.
>>
>> Ok. I mean, they don't have a better answer, but if it makes "python"
>> hate us less, sweet.
>>
>>> Concretely, I think we need to:
>>> - teach pbr to read in install_requires from setup.cfg, not
>> requirements.txt
>>> - when there are requirements in setup.cfg, stop reading
>> requirements.txt
>>> - separate out the global intall_requirements from the global CI
>>> requirements, and update our syncing code to be aware of this
>>>
>>> Then, setup.cfg contains more open requirements suitable for being on
>>> PyPI, requirements.txt is the local CI set we know works - and can be
>>> much more restrictive as needed.
>>>
>>> Thoughts? If there's broad apathy-or-agreement I can turn this into a
>>> spec for fine coverage of ramifications and corner cases.
>>
>> I'm concerned that it adds a layer of difference that is confusing to
>> people for the sole benefit of pleasing someone else's pedantic worldview.
>>
>> I'm also concerned that dstufft is actively wanting to move towards a
>> world where the build tooling is not needed or used as part of the
>> install pipeline (metadata 2.0) -- so I'm concerned that we're aligning
>> with a pattern that isn't very robust and isn't supported by tooling
>> directly and that we're going to need to change understood usage
>> patterns across a large developer based to chase something that _still_
>> isn't going to be "how people do it"
>>
>> I'm concerned that "how people do it" is a myth not worth chasing.
>>
>> I'm not _opposed_ to making this richer and more useful for people. I
>> just don't know what's broken currently for us.
>>
>
> To be clear, I don't mean to suggest that the solution proposed in
> https://review.openstack.org/#/c/161047/ is necessarily the correct
> solution to this problem - but I do think that it is illustrative of at
> last one thing that's currently broken for us.
I disagree that anything is broken for us that is not caused by our
inability to remember that distro packaging concerns are not the same as
our concerns, and that the mechanism already exists for distro pacakgers
to do what they want. Furthermore, it is caused by an insistence that we
need to keep versions "open" for some ephemeral reason such as "upstream
might release a bug fix" Since we all know that "if it's not tested,
it's broken" - any changes to upstream software should be considered
broken until proven otherwise. History over the last 5 years has shown
this to be accurate more than the other thing.
If we pin the stable branches with hard pins of direct and indirect
dependencies, we can have our stable branch artifacts be installable.
Thats awesome. IF there is a bugfix release or a security update to a
dependent library - someone can propose it. Otherwise, the stable
release should not be moving.
Monty
------------------------------
Message: 12
Date: Mon, 13 Apr 2015 13:02:13 +1200
From: Robert Collins <robertc at robertcollins.net>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [all][pbr] splitting our deployment vs
install dependencies
Message-ID:
<CAJ3HoZ2N9+EvsN0WkO2tiQLPKDi9q2tX_Qk8JRskdz+EX5+vBw at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
On 13 April 2015 at 12:01, James Polley <jp at jamezpolley.com> wrote:
>
>
> That sounds, to me, very similar to a discussion we had a few weeks ago in
> the context of our stable branches.
>
> In that context, we have two competing requirements. One requirement is that
> our CI system wants a very tightly pinned requirements, as do downstream CI
> systems and deployers that want to test and deploy exact known-tested
> versions of things. On the other hand, downstream distributors (including OS
> packagers) need to balance OpenStack's version requirements with version
> requirements from all the other packages in their distribution; the tighter
> the requirements we list are, the harder it is for the requirements to work
> with the requirements of other packages in the distribution.
They are analogous yes.
..
>> rust gets it wright. There is a Cargo.toml and a Cargo.lock, which are
>> understood by the tooling in a manner similar to what you have
>> described, and it is not just understood but DOCUMENTED that an
>> _application_ should ship with a Cargo.lock and a _library_ should not.
>
> This sounds similar to a solution that was proposed for the stable branches:
> a requirements.in with mandatory version constraints while being as loose as
> otherwise possible, which is used to generate a requirements.txt which has
> the "local to deployment" exact versions that are used in our CI. The
> details of the proposal are in https://review.openstack.org/#/c/161047/
That proposal is still under discussion, and seems stuck between the
distro and -infra folk. *if* it ends up doing the transitive thing, I
think that that would make a sensible requirements.txt, yes. However
see the spec for that thread of discussion.
.
>> I'm also concerned that dstufft is actively wanting to move towards a
>> world where the build tooling is not needed or used as part of the
>> install pipeline (metadata 2.0) -- so I'm concerned that we're aligning
>> with a pattern that isn't very robust and isn't supported by tooling
>> directly and that we're going to need to change understood usage
>> patterns across a large developer based to chase something that _still_
>> isn't going to be "how people do it"
Monty:
So wheels are already in that space. metadata-2.0 is about bringing
that declarative stuff forward in the pipeline, from binary packages
to source packages. I'm currently using frustration based development
to bring it in at the start - for developers, in the lead-in to source
packages.
So you're concerned - but about what specifically? What goes wrong
with wheels (not wheels with C code). Whats not robust about the
pattern? The cargo package manager you referred to is entirely
declarative....
James: I don't think the binary distribution stuff is relevant to my
discussion, since I'm talking entirely about 'using-pip' use cases,
whereas dpkg and rpm packages don't use that at all.
-Rob
--
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud
------------------------------
Message: 13
Date: Mon, 13 Apr 2015 13:09:44 +1200
From: Robert Collins <robertc at robertcollins.net>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [all][pbr] splitting our deployment vs
install dependencies
Message-ID:
<CAJ3HoZ0_O6y3tmc15xaO-P_04R8zFGUceQEUmd_6w9Q7Bk_vOg at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
On 13 April 2015 at 12:53, Monty Taylor <mordred at inaugust.com> wrote:
> What we have in the gate is the thing that produces the artifacts that
> someone installing using the pip tool would get. Shipping anything with
> those artifacts other that a direct communication of what we tested is
> just mean to our end users.
Actually its not.
What we test is point in time. At 2:45 UTC on Monday installing this
git ref of nova worked.
Noone can reconstruct that today.
I entirely agree with the sentiment you're expressing, but we're not
delivering that sentiment today.
We need to balance the inability to atomically update things - which
forces a degree of freedom on install_requires - with being able to
give someone the same install that we tested.
That is the fundamental tension that we're not handling well, nor have
I seen a proposal to tackle it so far.
I'll have to spend some time noodling on this, but one of the clear
constraints is that install_requires cannot both be:
- flexible enough to permit us to upgrade requirements across many
git based packages [because we could do coordinated releases of sdists
to approximate atomic bulk changes]
- tight enough enough to give the next person trying to run that ref
of the package the same things we installed in CI.
-> I think we need something other than install_requires
..
> I disagree that anything is broken for us that is not caused by our
> inability to remember that distro packaging concerns are not the same as
> our concerns, and that the mechanism already exists for distro pacakgers
> to do what they want. Furthermore, it is caused by an insistence that we
> need to keep versions "open" for some ephemeral reason such as "upstream
> might release a bug fix" Since we all know that "if it's not tested,
> it's broken" - any changes to upstream software should be considered
> broken until proven otherwise. History over the last 5 years has shown
> this to be accurate more than the other thing.
This seems like a strong argument for really being able to reconstruct
what was in CI.
> If we pin the stable branches with hard pins of direct and indirect
> dependencies, we can have our stable branch artifacts be installable.
> Thats awesome. IF there is a bugfix release or a security update to a
> dependent library - someone can propose it. Otherwise, the stable
> release should not be moving.
Can we do that in stable branches? We've still got the problem of
bumping dependencies across multiple packages.
-Rob
--
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud
------------------------------
Message: 14
Date: Sun, 12 Apr 2015 20:14:00 -0500
From: Brant Knudson <blk at acm.org>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: [openstack-dev] [all] Problems with keystoneclient stable
branch (and maybe yours too)
Message-ID:
<CAHjeE=Qw1E=bKfKCuEvqckgZpG0VOjAsjYpdVJjOF0fBtVTEJA at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
There were several problems with the keystoneclient stable/juno branch that
have been or are in the process of being fixed since its creation.
Hopefully this note will be useful to other projects that create stable
branches for their libraries.
1) Unit tests didn't pass with earlier packages
The supported versions of several of the packages in requirements.txt in
the stable branch are in the process of being capped[0], so that the tests
are now running with older versions of the packages. Since we don't
normally test with the older packages we didn't know that the
keystoneclient unit tests don't actually pass with the old version of the
package. This is fixed by correcting the tests to work with the older
versions of the packages.[1][2]
[0] https://review.openstack.org/#/c/172220/
[1] https://review.openstack.org/#/c/172655/
[2] https://review.openstack.org/#/c/172256/
It would be great if we were testing with the minimum versions of the
packages that we say we support somehow since that would have caught this.
2) Incorrect cap in requirements.txt
python-keystoneclient in stable/juno was capped at <=1.1.0, and 1.1.0 is
the version tagged for the stable branch. When you create a review in
stable/juno it installs python-keystoneclient and now the system has got a
version like 1.1.0.post1, which is >1.1.0, so now python-keystoneclient
doesn't match the requirements and swift-proxy fails to start (swift-proxy
is very good at catching this problem for whatever reason). The cap should
have been <1.2.0 so that we can propose patches and also make fix releases
(1.1.1, 1.1.2, etc.).[3]
[3] https://review.openstack.org/#/c/172718/
I tried to recap all of the clients but that didn't pass Jenkins, probably
because one or more clients didn't use semver correctly and have
requirements updates in a micro release.[4]
[4] https://review.openstack.org/#/c/172719/
3) Unsupported functional tests
We added support for functional tests (tox -e functional) in K, but Jenkins
was configured to run the functional job on all branches and it fails when
the tox target doesn't exist. The fix was to exclude the current stable/
branches for keystoneclient.[5]
[5] https://review.openstack.org/#/c/172658/
4) Tempest -juno job?
For some reason keystoneclient has 2 tempest-neutron jobs:
gate-tempest-dsvm-neutron-src-python-keystoneclient and
..-keystoneclient-juno , and the -juno job is failing in stable/juno. It
didn't make sense to me that we needed to run both in python-keystoneclient
stable/juno. I was told that we didn't need the -juno one anymore on any
branch since we have a stable/juno branch, so that job is removed.[6]
[6] https://review.openstack.org/#/c/172662/
Hopefully with these changes the python-keystoneclient stable/juno branch
will be working.
- Brant
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150412/2a0e52d7/attachment-0001.html>
------------------------------
Message: 15
Date: Mon, 13 Apr 2015 11:33:25 +1000
From: Michael Still <mikal at stillhq.com>
To: OpenStack Development Mailing List
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] In loving memory of Chris Yeoh
Message-ID:
<CAEd1pt7j2JutUjihkx1-S1ikZ_mZ_QHrMEhNMsUTkx7G7gY-GQ at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Hi, as promised I now have details of a charity for people to donate
to in Chris' memory:
http://participate.freetobreathe.org/site/TR?px=1582460&fr_id=2710&pg=personal#.VSscH5SUd90
In the words of the family:
"We would prefer that people donate to lung cancer research in lieu of
flowers. Lung cancer has the highest mortality rate out of all the
cancers, and the lowest funding out of all the cancers. There is a
stigma attached that lung cancer is a smoker's disease, and that
sufferers deserve their fate. They bring it on through lifestyle
choice. Except that Chris has never smoked in his life, like a
surprisingly large percentage of lung cancer sufferers. These people
suffer for the incorrect beliefs of the masses, and those that are
left behind are equally innocent. We shouldn't be doing this now. He
shouldn't be gone. We need to do more to fix this. There will be
charity envelopes available at the funeral, or you can choose your
preferred research to fund, should you wish to do so. You have our
thanks."
Michael
On Wed, Apr 8, 2015 at 2:49 PM, Michael Still <mikal at stillhq.com> wrote:
> It is my sad duty to inform the community that Chris Yeoh passed away this
> morning. Chris leaves behind a daughter Alyssa, aged 6, who I hope will
> remember Chris as the clever and caring person that I will remember him as.
> I haven?t had a chance to confirm with the family if they want flowers or a
> donation to a charity. As soon as I know those details I will reply to this
> email.
>
> Chris worked on open source for a very long time, with OpenStack being just
> the most recent in a long chain of contributions. He worked tirelessly on
> his contributions to Nova, including mentoring other developers. He was
> dedicated to the cause, with a strong vision of what OpenStack could become.
> He even named his cat after the project.
>
> Chris might be the only person to have ever sent an email to his coworkers
> explaining what his code review strategy would be after brain surgery. It
> takes phenomenal strength to carry on in the face of that kind of adversity,
> but somehow he did. Frankly, I think I would have just sat on the beach.
>
> Chris was also a contributor to the Linux Standards Base (LSB), where he
> helped improve the consistency and interoperability between Linux
> distributions. He ran the ?Hackfest? programming contests for a number of
> years at Australia?s open source conference -- linux.conf.au. He supported
> local Linux user groups in South Australia and Canberra, including
> involvement at installfests and speaking at local meetups. He competed in a
> programming challenge called Loki Hack, and beat out the world to win the
> event[1].
>
> Alyssa?s memories of her dad need to last her a long time, so we?ve decided
> to try and collect some fond memories of Chris to help her along the way. If
> you feel comfortable doing so, please contribute a memory or two at
> https://docs.google.com/forms/d/1kX-ePqAO7Cuudppwqz1cqgBXAsJx27GkdM-eCZ0c1V8/viewform
>
> Chris was humble, helpful and honest. The OpenStack and broader Open Source
> communities are poorer for his passing.
>
> Michael
>
> [1] http://www.lokigames.com/hack/
--
Rackspace Australia
------------------------------
Message: 16
Date: Mon, 13 Apr 2015 13:53:36 +1200
From: Robert Collins <robertc at robertcollins.net>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [all][pbr] splitting our deployment vs
install dependencies
Message-ID:
<CAJ3HoZ3RKSi8TQLnbS=eWFekOxztVtkE2EE+xFoXWvVnKUoBWw at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
On 13 April 2015 at 13:09, Robert Collins <robertc at robertcollins.net> wrote:
> On 13 April 2015 at 12:53, Monty Taylor <mordred at inaugust.com> wrote:
>
>> What we have in the gate is the thing that produces the artifacts that
>> someone installing using the pip tool would get. Shipping anything with
>> those artifacts other that a direct communication of what we tested is
>> just mean to our end users.
>
> Actually its not.
>
> What we test is point in time. At 2:45 UTC on Monday installing this
> git ref of nova worked.
>
> Noone can reconstruct that today.
>
> I entirely agree with the sentiment you're expressing, but we're not
> delivering that sentiment today.
This observation led to yet more IRC discussion and eventually
https://etherpad.openstack.org/p/stable-omg-deps
In short, the proposal is that we:
- stop trying to use install_requires to reproduce exactly what
works, and instead use it to communicate known constraints (> X, Y is
broken etc).
- use a requirements.txt file we create *during* CI to capture
exactly what worked, and also capture the dpkg and rpm versions of
packages that were present when it worked, and so on. So we'll build a
git tree where its history is an audit trail of exactly what worked
for everything that passed CI, formatted to make it really really easy
for other people to consume.
-Rob
--
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud
------------------------------
Message: 17
Date: Mon, 13 Apr 2015 02:17:22 +0000
From: joehuang <joehuang at huawei.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID:
<5E7A3D1BF5FD014E86E5F971CF446EFF54256EDB at szxema505-mbs.china.huawei.com>
Content-Type: text/plain; charset="utf-8"
Hi, Kevin and Joshua,
As my understanding, Tooz only addresses the issue of agent status management, but how to solve the concurrent dynamic load impact on large scale ( for example 100k managed nodes with the dynamic load like security goup rule update, routers_updated, etc )
And one more question is, if we have 100k managed nodes, how to do the partition? Or all nodes will be managed by one Tooz service, like Zookeeper? Can Zookeeper manage 100k nodes status?
Best Regards
Chaoyi Huang ( Joe Huang )
From: Kevin Benton [mailto:blak111 at gmail.com]
Sent: Monday, April 13, 2015 3:52 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
>Timestamps are just one way (and likely the most primitive), using redis (or memcache) key/value and expiry are another (and letting memcache or redis expire using its own internal algorithms), using zookeeper ephemeral nodes[1] are another... The point being that its backend specific and tooz supports varying backends.
Very cool. Is the backend completely transparent so a deployer could choose a service they are comfortable maintaining, or will that change the properties WRT to resiliency of state on node restarts, partitions, etc?
The Nova implementation of Tooz seemed pretty straight-forward, although it looked like it had pluggable drivers for service management already. Before I dig into it much further I'll file a spec on the Neutron side to see if I can get some other cores onboard to do the review work if I push a change to tooz.
On Sun, Apr 12, 2015 at 9:38 AM, Joshua Harlow <harlowja at outlook.com<mailto:harlowja at outlook.com>> wrote:
Kevin Benton wrote:
So IIUC tooz would be handling the liveness detection for the agents.
That would be nice to get ride of that logic in Neutron and just
register callbacks for rescheduling the dead.
Where does it store that state, does it persist timestamps to the DB
like Neutron does? If so, how would that scale better? If not, who does
a given node ask to know if an agent is online or offline when making a
scheduling decision?
Timestamps are just one way (and likely the most primitive), using redis (or memcache) key/value and expiry are another (and letting memcache or redis expire using its own internal algorithms), using zookeeper ephemeral nodes[1] are another... The point being that its backend specific and tooz supports varying backends.
However, before (what I assume is) the large code change to implement
tooz, I would like to quantify that the heartbeats are actually a
bottleneck. When I was doing some profiling of them on the master branch
a few months ago, processing a heartbeat took an order of magnitude less
time (<50ms) than the 'sync routers' task of the l3 agent (~300ms). A
few query optimizations might buy us a lot more headroom before we have
to fall back to large refactors.
Sure, always good to avoid prematurely optimizing things...
Although this is relevant for u I think anyway:
https://review.openstack.org/#/c/138607/ (same thing/nearly same in nova)...
https://review.openstack.org/#/c/172502/ (a WIP implementation of the latter).
[1] https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Ephemeral+Nodes
Kevin Benton wrote:
One of the most common is the heartbeat from each agent. However, I
don't think we can't eliminate them because they are used to determine
if the agents are still alive for scheduling purposes. Did you have
something else in mind to determine if an agent is alive?
Put each agent in a tooz[1] group; have each agent periodically
heartbeat[2], have whoever needs to schedule read the active members of
that group (or use [3] to get notified via a callback), profit...
Pick from your favorite (supporting) driver at:
http://docs.openstack.org/__developer/tooz/compatibility.__html
<http://docs.openstack.org/developer/tooz/compatibility.html>
[1]
http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping
<http://docs.openstack.org/developer/tooz/compatibility.html#grouping>
[2]
https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315
<https://github.com/openstack/tooz/blob/0.13.1/tooz/coordination.py#L315>
[3]
http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes
<http://docs.openstack.org/developer/tooz/tutorial/group_membership.html#watching-group-changes>
______________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request at lists.__openstack.org?subject:__unsubscribe<http://openstack.org?subject:__unsubscribe>
<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Kevin Benton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/710773cf/attachment-0001.html>
------------------------------
Message: 18
Date: Sun, 12 Apr 2015 20:06:02 -0700
From: Joshua Harlow <harlowja at outlook.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID: <BLU436-SMTP198723039606360764ABB95D8E70 at phx.gbl>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Kevin Benton wrote:
> >Timestamps are just one way (and likely the most primitive), using
> redis (or memcache) key/value and expiry are another (and letting
> memcache or redis expire using its own internal algorithms), using
> zookeeper ephemeral nodes[1] are another... The point being that its
> backend specific and tooz supports varying backends.
>
> Very cool. Is the backend completely transparent so a deployer could
> choose a service they are comfortable maintaining, or will that change
> the properties WRT to resiliency of state on node restarts, partitions, etc?
Of course... we tried to make it 'completely' transparent, but in
reality certain backends (zookeeper which uses a paxos-like algorithm
and redis with sentinel support...) are better (more resilient, more
consistent, handle partitions/restarts better...) than others (memcached
is after all just a distributed cache). This is just the nature of the
game...
>
> The Nova implementation of Tooz seemed pretty straight-forward, although
> it looked like it had pluggable drivers for service management already.
> Before I dig into it much further I'll file a spec on the Neutron side
> to see if I can get some other cores onboard to do the review work if I
> push a change to tooz.
Sounds good to me.
>
>
> On Sun, Apr 12, 2015 at 9:38 AM, Joshua Harlow <harlowja at outlook.com
> <mailto:harlowja at outlook.com>> wrote:
>
> Kevin Benton wrote:
>
> So IIUC tooz would be handling the liveness detection for the
> agents.
> That would be nice to get ride of that logic in Neutron and just
> register callbacks for rescheduling the dead.
>
> Where does it store that state, does it persist timestamps to the DB
> like Neutron does? If so, how would that scale better? If not,
> who does
> a given node ask to know if an agent is online or offline when
> making a
> scheduling decision?
>
>
> Timestamps are just one way (and likely the most primitive), using
> redis (or memcache) key/value and expiry are another (and letting
> memcache or redis expire using its own internal algorithms), using
> zookeeper ephemeral nodes[1] are another... The point being that its
> backend specific and tooz supports varying backends.
>
>
> However, before (what I assume is) the large code change to
> implement
> tooz, I would like to quantify that the heartbeats are actually a
> bottleneck. When I was doing some profiling of them on the
> master branch
> a few months ago, processing a heartbeat took an order of
> magnitude less
> time (<50ms) than the 'sync routers' task of the l3 agent
> (~300ms). A
> few query optimizations might buy us a lot more headroom before
> we have
> to fall back to large refactors.
>
>
> Sure, always good to avoid prematurely optimizing things...
>
> Although this is relevant for u I think anyway:
>
> https://review.openstack.org/#__/c/138607/
> <https://review.openstack.org/#/c/138607/> (same thing/nearly same
> in nova)...
>
> https://review.openstack.org/#__/c/172502/
> <https://review.openstack.org/#/c/172502/> (a WIP implementation of
> the latter).
>
> [1]
> https://zookeeper.apache.org/__doc/trunk/__zookeeperProgrammers.html#__Ephemeral+Nodes
> <https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Ephemeral+Nodes>
>
>
> Kevin Benton wrote:
>
>
> One of the most common is the heartbeat from each agent.
> However, I
> don't think we can't eliminate them because they are used
> to determine
> if the agents are still alive for scheduling purposes. Did
> you have
> something else in mind to determine if an agent is alive?
>
>
> Put each agent in a tooz[1] group; have each agent periodically
> heartbeat[2], have whoever needs to schedule read the active
> members of
> that group (or use [3] to get notified via a callback), profit...
>
> Pick from your favorite (supporting) driver at:
>
> http://docs.openstack.org/____developer/tooz/compatibility.____html
> <http://docs.openstack.org/__developer/tooz/compatibility.__html>
> <http://docs.openstack.org/__developer/tooz/compatibility.__html
> <http://docs.openstack.org/developer/tooz/compatibility.html>>
>
> [1]
> http://docs.openstack.org/____developer/tooz/compatibility.____html#grouping
> <http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping>
> <http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping
> <http://docs.openstack.org/developer/tooz/compatibility.html#grouping>>
> [2]
> https://github.com/openstack/____tooz/blob/0.13.1/tooz/____coordination.py#L315
> <https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315>
> <https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315
> <https://github.com/openstack/tooz/blob/0.13.1/tooz/coordination.py#L315>>
> [3]
> http://docs.openstack.org/____developer/tooz/tutorial/group_____membership.html#watching-____group-changes
> <http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes>
> <http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes
> <http://docs.openstack.org/developer/tooz/tutorial/group_membership.html#watching-group-changes>>
>
>
> __________________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.____openstack.org?subject:____unsubscribe
> <http://openstack.org?subject:__unsubscribe>
> <http://OpenStack-dev-request@__lists.openstack.org?subject:__unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>>
> http://lists.openstack.org/____cgi-bin/mailman/listinfo/____openstack-dev
> <http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev>
> <http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>>
>
> ______________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.__openstack.org?subject:__unsubscribe <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
>
> ______________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.__openstack.org?subject:__unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
>
>
>
> --
> Kevin Benton
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
------------------------------
Message: 19
Date: Sun, 12 Apr 2015 20:10:41 -0700
From: Joshua Harlow <harlowja at outlook.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID: <BLU437-SMTP392487FC50F749804A13DDD8E70 at phx.gbl>
Content-Type: text/plain; charset="UTF-8"; format=flowed
joehuang wrote:
> Hi, Kevin and Joshua,
>
> As my understanding, Tooz only addresses the issue of agent status
> management, but how to solve the concurrent dynamic load impact on large
> scale ( for example 100k managed nodes with the dynamic load like
> security goup rule update, routers_updated, etc )
Yes, that is correct, let's not confuse status/liveness management with
updates... since IMHO they are to very different things (the latter can
be eventually consistent IMHO will the liveness 'question' probably
should not be...).
>
> And one more question is, if we have 100k managed nodes, how to do the
> partition? Or all nodes will be managed by one Tooz service, like
> Zookeeper? Can Zookeeper manage 100k nodes status?
I can get u some data/numbers from some studies I've seen, but what u
are talking about is highly specific as to what u are doing with
zookeeper... There is no one solution for all the things IMHO; choose
what's best from your tool-belt for each problem...
>
> Best Regards
>
> Chaoyi Huang ( Joe Huang )
>
> *From:*Kevin Benton [mailto:blak111 at gmail.com]
> *Sent:* Monday, April 13, 2015 3:52 AM
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [neutron] Neutron scaling datapoints?
>
>>Timestamps are just one way (and likely the most primitive), using redis
> (or memcache) key/value and expiry are another (and letting memcache or
> redis expire using its own internal algorithms), using zookeeper
> ephemeral nodes[1] are another... The point being that its backend
> specific and tooz supports varying backends.
>
> Very cool. Is the backend completely transparent so a deployer could
> choose a service they are comfortable maintaining, or will that change
> the properties WRT to resiliency of state on node restarts, partitions, etc?
>
> The Nova implementation of Tooz seemed pretty straight-forward, although
> it looked like it had pluggable drivers for service management already.
> Before I dig into it much further I'll file a spec on the Neutron side
> to see if I can get some other cores onboard to do the review work if I
> push a change to tooz.
>
> On Sun, Apr 12, 2015 at 9:38 AM, Joshua Harlow <harlowja at outlook.com
> <mailto:harlowja at outlook.com>> wrote:
>
> Kevin Benton wrote:
>
> So IIUC tooz would be handling the liveness detection for the agents.
> That would be nice to get ride of that logic in Neutron and just
> register callbacks for rescheduling the dead.
>
> Where does it store that state, does it persist timestamps to the DB
> like Neutron does? If so, how would that scale better? If not, who does
> a given node ask to know if an agent is online or offline when making a
> scheduling decision?
>
>
> Timestamps are just one way (and likely the most primitive), using redis
> (or memcache) key/value and expiry are another (and letting memcache or
> redis expire using its own internal algorithms), using zookeeper
> ephemeral nodes[1] are another... The point being that its backend
> specific and tooz supports varying backends.
>
>
> However, before (what I assume is) the large code change to implement
> tooz, I would like to quantify that the heartbeats are actually a
> bottleneck. When I was doing some profiling of them on the master branch
> a few months ago, processing a heartbeat took an order of magnitude less
> time (<50ms) than the 'sync routers' task of the l3 agent (~300ms). A
> few query optimizations might buy us a lot more headroom before we have
> to fall back to large refactors.
>
>
> Sure, always good to avoid prematurely optimizing things...
>
> Although this is relevant for u I think anyway:
>
> https://review.openstack.org/#/c/138607/ (same thing/nearly same in nova)...
>
> https://review.openstack.org/#/c/172502/ (a WIP implementation of the
> latter).
>
> [1]
> https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Ephemeral+Nodes
> <https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Ephemeral+Nodes>
>
>
> Kevin Benton wrote:
>
>
> One of the most common is the heartbeat from each agent. However, I
> don't think we can't eliminate them because they are used to determine
> if the agents are still alive for scheduling purposes. Did you have
> something else in mind to determine if an agent is alive?
>
>
> Put each agent in a tooz[1] group; have each agent periodically
> heartbeat[2], have whoever needs to schedule read the active members of
> that group (or use [3] to get notified via a callback), profit...
>
> Pick from your favorite (supporting) driver at:
>
> http://docs.openstack.org/__developer/tooz/compatibility.__html
> <http://docs.openstack.org/developer/tooz/compatibility.html>
>
> [1]
> http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping
> <http://docs.openstack.org/developer/tooz/compatibility.html#grouping>
> [2]
> https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315
> <https://github.com/openstack/tooz/blob/0.13.1/tooz/coordination.py#L315>
> [3]
> http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes
> <http://docs.openstack.org/developer/tooz/tutorial/group_membership.html#watching-group-changes>
>
>
> ______________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.__openstack.org?subject:__unsubscribe
> <http://openstack.org?subject:__unsubscribe>
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
>
> Kevin Benton
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
------------------------------
Message: 20
Date: Sun, 12 Apr 2015 20:14:51 -0700
From: Joshua Harlow <harlowja at outlook.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID: <BLU436-SMTP1166EE6F77C4A1152DAEC4AD8E70 at phx.gbl>
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Joshua Harlow wrote:
> Kevin Benton wrote:
>> >Timestamps are just one way (and likely the most primitive), using
>> redis (or memcache) key/value and expiry are another (and letting
>> memcache or redis expire using its own internal algorithms), using
>> zookeeper ephemeral nodes[1] are another... The point being that its
>> backend specific and tooz supports varying backends.
>>
>> Very cool. Is the backend completely transparent so a deployer could
>> choose a service they are comfortable maintaining, or will that change
>> the properties WRT to resiliency of state on node restarts,
>> partitions, etc?
>
> Of course... we tried to make it 'completely' transparent, but in
> reality certain backends (zookeeper which uses a paxos-like algorithm
> and redis with sentinel support...) are better (more resilient, more
> consistent, handle partitions/restarts better...) than others (memcached
> is after all just a distributed cache). This is just the nature of the
> game...
>
And for some more reading fun:
https://aphyr.com/posts/315-call-me-maybe-rabbitmq
https://aphyr.com/posts/291-call-me-maybe-zookeeper
https://aphyr.com/posts/283-call-me-maybe-redis
https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul
.. (aphyr.com has alot of these neat posts)...
>>
>> The Nova implementation of Tooz seemed pretty straight-forward, although
>> it looked like it had pluggable drivers for service management already.
>> Before I dig into it much further I'll file a spec on the Neutron side
>> to see if I can get some other cores onboard to do the review work if I
>> push a change to tooz.
>
> Sounds good to me.
>
>>
>>
>> On Sun, Apr 12, 2015 at 9:38 AM, Joshua Harlow <harlowja at outlook.com
>> <mailto:harlowja at outlook.com>> wrote:
>>
>> Kevin Benton wrote:
>>
>> So IIUC tooz would be handling the liveness detection for the
>> agents.
>> That would be nice to get ride of that logic in Neutron and just
>> register callbacks for rescheduling the dead.
>>
>> Where does it store that state, does it persist timestamps to the DB
>> like Neutron does? If so, how would that scale better? If not,
>> who does
>> a given node ask to know if an agent is online or offline when
>> making a
>> scheduling decision?
>>
>>
>> Timestamps are just one way (and likely the most primitive), using
>> redis (or memcache) key/value and expiry are another (and letting
>> memcache or redis expire using its own internal algorithms), using
>> zookeeper ephemeral nodes[1] are another... The point being that its
>> backend specific and tooz supports varying backends.
>>
>>
>> However, before (what I assume is) the large code change to
>> implement
>> tooz, I would like to quantify that the heartbeats are actually a
>> bottleneck. When I was doing some profiling of them on the
>> master branch
>> a few months ago, processing a heartbeat took an order of
>> magnitude less
>> time (<50ms) than the 'sync routers' task of the l3 agent
>> (~300ms). A
>> few query optimizations might buy us a lot more headroom before
>> we have
>> to fall back to large refactors.
>>
>>
>> Sure, always good to avoid prematurely optimizing things...
>>
>> Although this is relevant for u I think anyway:
>>
>> https://review.openstack.org/#__/c/138607/
>> <https://review.openstack.org/#/c/138607/> (same thing/nearly same
>> in nova)...
>>
>> https://review.openstack.org/#__/c/172502/
>> <https://review.openstack.org/#/c/172502/> (a WIP implementation of
>> the latter).
>>
>> [1]
>> https://zookeeper.apache.org/__doc/trunk/__zookeeperProgrammers.html#__Ephemeral+Nodes
>>
>> <https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Ephemeral+Nodes>
>>
>>
>>
>> Kevin Benton wrote:
>>
>>
>> One of the most common is the heartbeat from each agent.
>> However, I
>> don't think we can't eliminate them because they are used
>> to determine
>> if the agents are still alive for scheduling purposes. Did
>> you have
>> something else in mind to determine if an agent is alive?
>>
>>
>> Put each agent in a tooz[1] group; have each agent periodically
>> heartbeat[2], have whoever needs to schedule read the active
>> members of
>> that group (or use [3] to get notified via a callback), profit...
>>
>> Pick from your favorite (supporting) driver at:
>>
>> http://docs.openstack.org/____developer/tooz/compatibility.____html
>> <http://docs.openstack.org/__developer/tooz/compatibility.__html>
>> <http://docs.openstack.org/__developer/tooz/compatibility.__html
>> <http://docs.openstack.org/developer/tooz/compatibility.html>>
>>
>> [1]
>> http://docs.openstack.org/____developer/tooz/compatibility.____html#grouping
>>
>> <http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping>
>>
>> <http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping
>> <http://docs.openstack.org/developer/tooz/compatibility.html#grouping>>
>> [2]
>> https://github.com/openstack/____tooz/blob/0.13.1/tooz/____coordination.py#L315
>>
>> <https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315>
>>
>> <https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315
>>
>> <https://github.com/openstack/tooz/blob/0.13.1/tooz/coordination.py#L315>>
>>
>> [3]
>> http://docs.openstack.org/____developer/tooz/tutorial/group_____membership.html#watching-____group-changes
>>
>> <http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes>
>>
>> <http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes
>>
>> <http://docs.openstack.org/developer/tooz/tutorial/group_membership.html#watching-group-changes>>
>>
>>
>>
>> __________________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.____openstack.org?subject:____unsubscribe
>> <http://openstack.org?subject:__unsubscribe>
>> <http://OpenStack-dev-request@__lists.openstack.org?subject:__unsubscribe
>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>>
>> http://lists.openstack.org/____cgi-bin/mailman/listinfo/____openstack-dev
>> <http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev>
>> <http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>>
>>
>> ______________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.__openstack.org?subject:__unsubscribe
>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>> ______________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.__openstack.org?subject:__unsubscribe
>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>>
>>
>> --
>> Kevin Benton
>>
>> __________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
------------------------------
Message: 21
Date: Sun, 12 Apr 2015 23:23:13 -0400
From: Emilien Macchi <emilien at redhat.com>
To: puppet-openstack at puppetlabs.com, openstack-dev at lists.openstack.org
Subject: Re: [openstack-dev] [puppet] Puppet PTL
Message-ID: <552B36A1.6030009 at redhat.com>
Content-Type: text/plain; charset="utf-8"
On 04/10/2015 06:58 PM, Colleen Murphy wrote:
> Just to make it official: since we only had one nominee for PTL, we will
> go ahead and name Emilien Macchi as our new PTL without proceeding with
> an election process. Thanks, Emilien, for all your hard work and for
> taking on this responsibility!
Well, I think this is a great opportunity to me to say something here.
First of all, thank you for your trust and I'll do my best to succeed in
this new position. You know me enough I'm always open to any feedback,
so please do.
Also, I would like to thank our whole community (core & non-core) for
the hard work we all did these years.
I truly think we will succeed under the big tent only if we continue to
work *together* as a team. But I'm confident, this is part of our DNA
and this is why we are at this stage today.
I'm really looking forward to seeing you at the next Summit.
Thanks,
>
> Colleen (crinkle)
>
> --
>
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-openstack+unsubscribe at puppetlabs.com
> <mailto:puppet-openstack+unsubscribe at puppetlabs.com>.
--
Emilien Macchi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150412/4e811715/attachment-0001.pgp>
------------------------------
Message: 22
Date: Mon, 13 Apr 2015 05:03:52 +0000
From: Gary Kotton <gkotton at vmware.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] In loving memory of Chris Yeoh
Message-ID: <D15127AD.74647%gkotton at vmware.com>
Content-Type: text/plain; charset="Windows-1252"
Hi,
I am very saddened to read this. Not only will Chris be missed on a
professional level but on a personal level. He was a real mensh
(http://www.thefreedictionary.com/mensh). He was always helpful and
supportive. Wishing his family a long life.
Thanks
Gary
On 4/13/15, 4:33 AM, "Michael Still" <mikal at stillhq.com> wrote:
>Hi, as promised I now have details of a charity for people to donate
>to in Chris' memory:
>
>
>https://urldefense.proofpoint.com/v2/url?u=http-3A__participate.freetobrea
>the.org_site_TR-3Fpx-3D1582460-26fr-5Fid-3D2710-26pg-3Dpersonal-23.VSscH5S
>Ud90&d=AwIGaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzk
>WT5jqz9JYBk8YTeq9N3-diTlNj4GyNc&m=IFwED7YYaddl7JbqZ5OLChF6gtEGxYkxfFHwjWRm
>sD8&s=B3EgunFqBdY8twmv-iJ7G7xvKZ4Th48oB4HKSv2uGKg&e=
>
>In the words of the family:
>
>"We would prefer that people donate to lung cancer research in lieu of
>flowers. Lung cancer has the highest mortality rate out of all the
>cancers, and the lowest funding out of all the cancers. There is a
>stigma attached that lung cancer is a smoker's disease, and that
>sufferers deserve their fate. They bring it on through lifestyle
>choice. Except that Chris has never smoked in his life, like a
>surprisingly large percentage of lung cancer sufferers. These people
>suffer for the incorrect beliefs of the masses, and those that are
>left behind are equally innocent. We shouldn't be doing this now. He
>shouldn't be gone. We need to do more to fix this. There will be
>charity envelopes available at the funeral, or you can choose your
>preferred research to fund, should you wish to do so. You have our
>thanks."
>
>Michael
>
>On Wed, Apr 8, 2015 at 2:49 PM, Michael Still <mikal at stillhq.com> wrote:
>> It is my sad duty to inform the community that Chris Yeoh passed away
>>this
>> morning. Chris leaves behind a daughter Alyssa, aged 6, who I hope will
>> remember Chris as the clever and caring person that I will remember him
>>as.
>> I haven?t had a chance to confirm with the family if they want flowers
>>or a
>> donation to a charity. As soon as I know those details I will reply to
>>this
>> email.
>>
>> Chris worked on open source for a very long time, with OpenStack being
>>just
>> the most recent in a long chain of contributions. He worked tirelessly
>>on
>> his contributions to Nova, including mentoring other developers. He was
>> dedicated to the cause, with a strong vision of what OpenStack could
>>become.
>> He even named his cat after the project.
>>
>> Chris might be the only person to have ever sent an email to his
>>coworkers
>> explaining what his code review strategy would be after brain surgery.
>>It
>> takes phenomenal strength to carry on in the face of that kind of
>>adversity,
>> but somehow he did. Frankly, I think I would have just sat on the beach.
>>
>> Chris was also a contributor to the Linux Standards Base (LSB), where he
>> helped improve the consistency and interoperability between Linux
>> distributions. He ran the ?Hackfest? programming contests for a number
>>of
>> years at Australia?s open source conference -- linux.conf.au. He
>>supported
>> local Linux user groups in South Australia and Canberra, including
>> involvement at installfests and speaking at local meetups. He competed
>>in a
>> programming challenge called Loki Hack, and beat out the world to win
>>the
>> event[1].
>>
>> Alyssa?s memories of her dad need to last her a long time, so we?ve
>>decided
>> to try and collect some fond memories of Chris to help her along the
>>way. If
>> you feel comfortable doing so, please contribute a memory or two at
>>
>>https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_form
>>s_d_1kX-2DePqAO7Cuudppwqz1cqgBXAsJx27GkdM-2DeCZ0c1V8_viewform&d=AwIGaQ&c=
>>Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzkWT5jqz9JYBk8YTe
>>q9N3-diTlNj4GyNc&m=IFwED7YYaddl7JbqZ5OLChF6gtEGxYkxfFHwjWRmsD8&s=iihsaOMe
>>lNeIR3VZapWKjr5KLgMQArZ3nifKDo1yy8o&e=
>>
>> Chris was humble, helpful and honest. The OpenStack and broader Open
>>Source
>> communities are poorer for his passing.
>>
>> Michael
>>
>> [1]
>>https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lokigames.com_hac
>>k_&d=AwIGaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzkW
>>T5jqz9JYBk8YTeq9N3-diTlNj4GyNc&m=IFwED7YYaddl7JbqZ5OLChF6gtEGxYkxfFHwjWRm
>>sD8&s=9SJI7QK-jzCsVUN2hTXSthqiXNEbq2Fvl9JqQiX9tfo&e=
>
>
>
>--
>Rackspace Australia
>
>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
------------------------------
Message: 23
Date: Mon, 13 Apr 2015 02:21:59 -0400 (EDT)
From: Attila Fazekas <afazekas at redhat.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID:
<147828721.17010859.1428906119294.JavaMail.zimbra at redhat.com>
Content-Type: text/plain; charset=utf-8
----- Original Message -----
> From: "Kevin Benton" <blak111 at gmail.com>
> To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org>
> Sent: Sunday, April 12, 2015 4:17:29 AM
> Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
>
>
>
> So IIUC tooz would be handling the liveness detection for the agents. That
> would be nice to get ride of that logic in Neutron and just register
> callbacks for rescheduling the dead.
>
> Where does it store that state, does it persist timestamps to the DB like
> Neutron does? If so, how would that scale better? If not, who does a given
> node ask to know if an agent is online or offline when making a scheduling
> decision?
>
You might find interesting the proposed solution in this bug:
https://bugs.launchpad.net/nova/+bug/1437199
> However, before (what I assume is) the large code change to implement tooz, I
> would like to quantify that the heartbeats are actually a bottleneck. When I
> was doing some profiling of them on the master branch a few months ago,
> processing a heartbeat took an order of magnitude less time (<50ms) than the
> 'sync routers' task of the l3 agent (~300ms). A few query optimizations
> might buy us a lot more headroom before we have to fall back to large
> refactors.
> Kevin Benton wrote:
>
>
>
> One of the most common is the heartbeat from each agent. However, I
> don't think we can't eliminate them because they are used to determine
> if the agents are still alive for scheduling purposes. Did you have
> something else in mind to determine if an agent is alive?
>
> Put each agent in a tooz[1] group; have each agent periodically heartbeat[2],
> have whoever needs to schedule read the active members of that group (or use
> [3] to get notified via a callback), profit...
>
> Pick from your favorite (supporting) driver at:
>
> http://docs.openstack.org/ developer/tooz/compatibility. html
>
> [1] http://docs.openstack.org/ developer/tooz/compatibility. html#grouping
> [2] https://github.com/openstack/ tooz/blob/0.13.1/tooz/ coordination.py#L315
> [3] http://docs.openstack.org/ developer/tooz/tutorial/group_
> membership.html#watching- group-changes
>
>
> ______________________________ ______________________________ ______________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists. openstack.org?subject: unsubscribe
> http://lists.openstack.org/ cgi-bin/mailman/listinfo/ openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
------------------------------
Message: 24
Date: Sun, 12 Apr 2015 23:26:10 -0700
From: Kevin Benton <blak111 at gmail.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [Neutron] Regarding neutron bug # 1432582
Message-ID:
<CAO_F6JP673t3trD4QudmHZKgUekPGVtqihg3zDHyhGf=7HpQrQ at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
I would like to see some form of this merged at least as an error message.
If a server has a bad CMOS battery and suffers a power outage, it's clock
could easily be several years behind. In that scenario, the NTP daemon
could refuse to sync due to a sanity check.
On Wed, Apr 8, 2015 at 10:46 AM, Sudipto Biswas <sbiswas7 at linux.vnet.ibm.com
> wrote:
> Hi Guys, I'd really appreciate your feedback on this.
>
> Thanks,
> Sudipto
>
>
> On Monday 30 March 2015 12:11 PM, Sudipto Biswas wrote:
>
>> Someone from my team had installed the OS on baremetal with a wrong 'date'
>> When this node was added to the Openstack controller, the logs from the
>> neutron-agent on the compute node showed - "AMQP connected". But the
>> neutron
>> agent-list command would not list this agent at all.
>>
>> I could figure out the problem when the neutron-server debug logs were
>> enabled
>> and it vaguely pointed at the rejection of AMQP connections due to a
>> timestamp
>> miss match. The neutron-server was treating these requests as stale due
>> to the
>> timestamp of the node being behind the neutron-server. However, there's no
>> good way to detect this if the agent runs on a node which is ahead of
>> time.
>>
>> I recently raised a bug here: https://bugs.launchpad.net/
>> neutron/+bug/1432582
>>
>> And tried to resolve this with the review:
>> https://review.openstack.org/#/c/165539/
>>
>> It went through quite a few +2s after 15 odd patch sets but we still are
>> not
>> in common ground w.r.t addressing this situation.
>>
>> My fix tries to log better and throw up an exception to the neutron agent
>> on
>> FIRST time boot of the agent for better detection of the problem.
>>
>> I would like to get your thoughts on this fix. Whether this seems legit
>> to have
>> the fix per the patch OR could you suggest a approach to tackle this OR
>> suggest
>> just abandoning the change.
>>
>>
>>
>> __________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:
>> unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
--
Kevin Benton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150412/b8eef87a/attachment-0001.html>
------------------------------
Message: 25
Date: Mon, 13 Apr 2015 14:51:05 +0800
From: "=?gb18030?B?NDQ5MTcxMzQy?=" <449171342 at qq.com>
To: "=?gb18030?B?b3BlbnN0YWNrLWRldg==?="
<openstack-dev at lists.openstack.org>
Subject: [openstack-dev] ?magnum?About clean none use container imag
Message-ID: <tencent_1B594E50513691B07D1B8F9E at qq.com>
Content-Type: text/plain; charset="gb18030"
From now on magnum had container create and delete api .The container create api will pull docker image from docker-registry.But the container delete api didn't delete image.It will let the image remain even though didn't had container use it.Is it much better we can clear the image in someway?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/7072441b/attachment-0001.html>
------------------------------
Message: 26
Date: Mon, 13 Apr 2015 15:09:18 +0800
From: Jay Lau <jay.lau.513 at gmail.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] ?magnum?About clean none use container
imag
Message-ID:
<CAFyztAG6EOd97Ho5zir7U0Gmywq=J=AErH9WJAi-2yswx2E5Ng at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Interesting topic, pulling image is time consuming, so someone might not
want to delete the container; But for some cases, if the image was not
used, then it is better to remove them from disk to release space. You may
want to send out an email to [openstack][magnum] ML to get more feedback ;-)
2015-04-13 14:51 GMT+08:00 449171342 <449171342 at qq.com>:
>
>
> From now on magnum had container create and delete api .The container create api will pull docker image from docker-registry.But the container delete api didn't delete image.It will let the image remain even though didn't had container use it.Is it much better we can clear the image in someway?
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
--
Thanks,
Jay Lau (Guangya Liu)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/38091051/attachment-0001.html>
------------------------------
Message: 27
Date: Mon, 13 Apr 2015 03:18:34 -0400 (EDT)
From: Attila Fazekas <afazekas at redhat.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID:
<1115924511.17061105.1428909514804.JavaMail.zimbra at redhat.com>
Content-Type: text/plain; charset=utf-8
----- Original Message -----
> From: "joehuang" <joehuang at huawei.com>
> To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org>
> Sent: Sunday, April 12, 2015 1:20:48 PM
> Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
>
>
>
> Hi, Kevin,
>
>
>
> I assumed that all agents are connected to same IP address of RabbitMQ, then
> the connection will exceed the port ranges limitation.
>
https://news.ycombinator.com/item?id=1571300
"TCP connections are identified by the (src ip, src port, dest ip, dest port) tuple."
"The server doesn't need multiple IPs to handle > 65535 connections. All the server connections to a given IP are to the same port. For a given client, the unique key for an http connection is (client-ip, PORT, server-ip, 80). The only number that can vary is PORT, and that's a value on the client. So, the client is limited to 65535 connections to the server. But, a second client could also have another 65K connections to the same server-ip:port."
>
> For a RabbitMQ cluster, for sure the client can connect to any one of member
> in the cluster, but in this case, the client has to be designed in fail-safe
> manner: the client should be aware of the cluster member failure, and
> reconnect to other survive member. No such mechnism has been implemented
> yet.
>
>
>
> Other way is to use LVS or DNS based like load balancer, or something else.
> If you put one load balancer ahead of a cluster, then we have to take care
> of the port number limitation, there are so many agents will require
> connection concurrently, 100k level, and the requests can not be rejected.
>
>
>
> Best Regards
>
>
>
> Chaoyi Huang ( joehuang )
>
>
>
> From: Kevin Benton [blak111 at gmail.com]
> Sent: 12 April 2015 9:59
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
>
> The TCP/IP stack keeps track of connections as a combination of IP + TCP
> port. The two byte port limit doesn't matter unless all of the agents are
> connecting from the same IP address, which shouldn't be the case unless
> compute nodes connect to the rabbitmq server via one IP address running port
> address translation.
>
> Either way, the agents don't connect directly to the Neutron server, they
> connect to the rabbit MQ cluster. Since as many Neutron server processes can
> be launched as necessary, the bottlenecks will likely show up at the
> messaging or DB layer.
>
> On Sat, Apr 11, 2015 at 6:46 PM, joehuang < joehuang at huawei.com > wrote:
>
>
>
>
>
> As Kevin talking about agents, I want to remind that in TCP/IP stack, port (
> not Neutron Port ) is a two bytes field, i.e. port ranges from 0 ~ 65535,
> supports maximum 64k port number.
>
>
>
> " above 100k managed node " means more than 100k L2 agents/L3 agents... will
> be alive under Neutron.
>
>
>
> Want to know the detail design how to support 99.9% possibility for scaling
> Neutron in this way, and PoC and test would be a good support for this idea.
>
>
>
> "I'm 99.9% sure, for scaling above 100k managed node,
> we do not really need to split the openstack to multiple smaller openstack,
> or use significant number of extra controller machine."
>
>
>
> Best Regards
>
>
>
> Chaoyi Huang ( joehuang )
>
>
>
> From: Kevin Benton [ blak111 at gmail.com ]
> Sent: 11 April 2015 12:34
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
>
> Which periodic updates did you have in mind to eliminate? One of the few
> remaining ones I can think of is sync_routers but it would be great if you
> can enumerate the ones you observed because eliminating overhead in agents
> is something I've been working on as well.
>
> One of the most common is the heartbeat from each agent. However, I don't
> think we can't eliminate them because they are used to determine if the
> agents are still alive for scheduling purposes. Did you have something else
> in mind to determine if an agent is alive?
>
> On Fri, Apr 10, 2015 at 2:18 AM, Attila Fazekas < afazekas at redhat.com >
> wrote:
>
>
> I'm 99.9% sure, for scaling above 100k managed node,
> we do not really need to split the openstack to multiple smaller openstack,
> or use significant number of extra controller machine.
>
> The problem is openstack using the right tools SQL/AMQP/(zk),
> but in a wrong way.
>
> For example.:
> Periodic updates can be avoided almost in all cases
>
> The new data can be pushed to the agent just when it needed.
> The agent can know when the AMQP connection become unreliable (queue or
> connection loose),
> and needs to do full sync.
> https://bugs.launchpad.net/neutron/+bug/1438159
>
> Also the agents when gets some notification, they start asking for details
> via the
> AMQP -> SQL. Why they do not know it already or get it with the notification
> ?
>
>
> ----- Original Message -----
> > From: "Neil Jerram" < Neil.Jerram at metaswitch.com >
> > To: "OpenStack Development Mailing List (not for usage questions)" <
> > openstack-dev at lists.openstack.org >
> > Sent: Thursday, April 9, 2015 5:01:45 PM
> > Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
> >
> > Hi Joe,
> >
> > Many thanks for your reply!
> >
> > On 09/04/15 03:34, joehuang wrote:
> > > Hi, Neil,
> > >
> > > From theoretic, Neutron is like a "broadcast" domain, for example,
> > > enforcement of DVR and security group has to touch each regarding host
> > > where there is VM of this project resides. Even using SDN controller, the
> > > "touch" to regarding host is inevitable. If there are plenty of physical
> > > hosts, for example, 10k, inside one Neutron, it's very hard to overcome
> > > the "broadcast storm" issue under concurrent operation, that's the
> > > bottleneck for scalability of Neutron.
> >
> > I think I understand that in general terms - but can you be more
> > specific about the broadcast storm? Is there one particular message
> > exchange that involves broadcasting? Is it only from the server to
> > agents, or are there 'broadcasts' in other directions as well?
> >
> > (I presume you are talking about control plane messages here, i.e.
> > between Neutron components. Is that right? Obviously there can also be
> > broadcast storm problems in the data plane - but I don't think that's
> > what you are talking about here.)
> >
> > > We need layered architecture in Neutron to solve the "broadcast domain"
> > > bottleneck of scalability. The test report from OpenStack cascading shows
> > > that through layered architecture "Neutron cascading", Neutron can
> > > supports up to million level ports and 100k level physical hosts. You can
> > > find the report here:
> > > http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers
> >
> > Many thanks, I will take a look at this.
> >
> > > "Neutron cascading" also brings extra benefit: One cascading Neutron can
> > > have many cascaded Neutrons, and different cascaded Neutron can leverage
> > > different SDN controller, maybe one is ODL, the other one is
> > > OpenContrail.
> > >
> > > ----------------Cascading Neutron-------------------
> > > / \
> > > --cascaded Neutron-- --cascaded Neutron-----
> > > | |
> > > ---------ODL------ ----OpenContrail--------
> > >
> > >
> > > And furthermore, if using Neutron cascading in multiple data centers, the
> > > DCI controller (Data center inter-connection controller) can also be used
> > > under cascading Neutron, to provide NaaS ( network as a service ) across
> > > data centers.
> > >
> > > ---------------------------Cascading Neutron--------------------------
> > > / | \
> > > --cascaded Neutron-- -DCI controller- --cascaded Neutron-----
> > > | | |
> > > ---------ODL------ | ----OpenContrail--------
> > > |
> > > --(Data center 1)-- --(DCI networking)-- --(Data center 2)--
> > >
> > > Is it possible for us to discuss this in OpenStack Vancouver summit?
> >
> > Most certainly, yes. I will be there from mid Monday afternoon through
> > to end Friday. But it will be my first summit, so I have no idea yet as
> > to how I might run into you - please can you suggest!
> >
> > > Best Regards
> > > Chaoyi Huang ( Joe Huang )
> >
> > Regards,
> > Neil
> >
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Kevin Benton
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> --
> Kevin Benton
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
------------------------------
Message: 28
Date: Mon, 13 Apr 2015 15:23:21 +0800
From: wu jiang <wingwj at gmail.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] In loving memory of Chris Yeoh
Message-ID:
<CAEfJYtTacCxQ2MQXzqQQ2b_VoX6o_Pxr=WmoaXB=HDp5EmZ92w at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
What bad news.. Chris helped me a lot, we lost a mentor and friend.
May God bless his/her soul.
WingWJ
On Mon, Apr 13, 2015 at 1:03 PM, Gary Kotton <gkotton at vmware.com> wrote:
> Hi,
> I am very saddened to read this. Not only will Chris be missed on a
> professional level but on a personal level. He was a real mensh
> (http://www.thefreedictionary.com/mensh). He was always helpful and
> supportive. Wishing his family a long life.
> Thanks
> Gary
>
> On 4/13/15, 4:33 AM, "Michael Still" <mikal at stillhq.com> wrote:
>
> >Hi, as promised I now have details of a charity for people to donate
> >to in Chris' memory:
> >
> >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__participate.freetobrea
> >the.org_site_TR-3Fpx-3D1582460-26fr-5Fid-3D2710-26pg-3Dpersonal-23.VSscH5S
> >Ud90&d=AwIGaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzk
> >WT5jqz9JYBk8YTeq9N3-diTlNj4GyNc&m=IFwED7YYaddl7JbqZ5OLChF6gtEGxYkxfFHwjWRm
> >sD8&s=B3EgunFqBdY8twmv-iJ7G7xvKZ4Th48oB4HKSv2uGKg&e=
> >
> >In the words of the family:
> >
> >"We would prefer that people donate to lung cancer research in lieu of
> >flowers. Lung cancer has the highest mortality rate out of all the
> >cancers, and the lowest funding out of all the cancers. There is a
> >stigma attached that lung cancer is a smoker's disease, and that
> >sufferers deserve their fate. They bring it on through lifestyle
> >choice. Except that Chris has never smoked in his life, like a
> >surprisingly large percentage of lung cancer sufferers. These people
> >suffer for the incorrect beliefs of the masses, and those that are
> >left behind are equally innocent. We shouldn't be doing this now. He
> >shouldn't be gone. We need to do more to fix this. There will be
> >charity envelopes available at the funeral, or you can choose your
> >preferred research to fund, should you wish to do so. You have our
> >thanks."
> >
> >Michael
> >
> >On Wed, Apr 8, 2015 at 2:49 PM, Michael Still <mikal at stillhq.com> wrote:
> >> It is my sad duty to inform the community that Chris Yeoh passed away
> >>this
> >> morning. Chris leaves behind a daughter Alyssa, aged 6, who I hope will
> >> remember Chris as the clever and caring person that I will remember him
> >>as.
> >> I haven?t had a chance to confirm with the family if they want flowers
> >>or a
> >> donation to a charity. As soon as I know those details I will reply to
> >>this
> >> email.
> >>
> >> Chris worked on open source for a very long time, with OpenStack being
> >>just
> >> the most recent in a long chain of contributions. He worked tirelessly
> >>on
> >> his contributions to Nova, including mentoring other developers. He was
> >> dedicated to the cause, with a strong vision of what OpenStack could
> >>become.
> >> He even named his cat after the project.
> >>
> >> Chris might be the only person to have ever sent an email to his
> >>coworkers
> >> explaining what his code review strategy would be after brain surgery.
> >>It
> >> takes phenomenal strength to carry on in the face of that kind of
> >>adversity,
> >> but somehow he did. Frankly, I think I would have just sat on the beach.
> >>
> >> Chris was also a contributor to the Linux Standards Base (LSB), where he
> >> helped improve the consistency and interoperability between Linux
> >> distributions. He ran the ?Hackfest? programming contests for a number
> >>of
> >> years at Australia?s open source conference -- linux.conf.au. He
> >>supported
> >> local Linux user groups in South Australia and Canberra, including
> >> involvement at installfests and speaking at local meetups. He competed
> >>in a
> >> programming challenge called Loki Hack, and beat out the world to win
> >>the
> >> event[1].
> >>
> >> Alyssa?s memories of her dad need to last her a long time, so we?ve
> >>decided
> >> to try and collect some fond memories of Chris to help her along the
> >>way. If
> >> you feel comfortable doing so, please contribute a memory or two at
> >>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_form
> >>s_d_1kX-2DePqAO7Cuudppwqz1cqgBXAsJx27GkdM-2DeCZ0c1V8_viewform&d=AwIGaQ&c=
> >>Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzkWT5jqz9JYBk8YTe
> >>q9N3-diTlNj4GyNc&m=IFwED7YYaddl7JbqZ5OLChF6gtEGxYkxfFHwjWRmsD8&s=iihsaOMe
> >>lNeIR3VZapWKjr5KLgMQArZ3nifKDo1yy8o&e=
> >>
> >> Chris was humble, helpful and honest. The OpenStack and broader Open
> >>Source
> >> communities are poorer for his passing.
> >>
> >> Michael
> >>
> >> [1]
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lokigames.com_hac
> >>k_&d=AwIGaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzkW
> >>T5jqz9JYBk8YTeq9N3-diTlNj4GyNc&m=IFwED7YYaddl7JbqZ5OLChF6gtEGxYkxfFHwjWRm
> >>sD8&s=9SJI7QK-jzCsVUN2hTXSthqiXNEbq2Fvl9JqQiX9tfo&e=
> >
> >
> >
> >--
> >Rackspace Australia
> >
> >__________________________________________________________________________
> >OpenStack Development Mailing List (not for usage questions)
> >Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/9e436bfd/attachment-0001.html>
------------------------------
Message: 29
Date: Mon, 13 Apr 2015 04:03:49 -0400 (EDT)
From: Victor Stinner <vstinner at redhat.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully
Python 3 compatible
Message-ID:
<868807730.15816798.1428912229323.JavaMail.zimbra at redhat.com>
Content-Type: text/plain; charset=utf-8
> Worth noting we've already switched to using PyMySQL in nodepool,
> storyboard and some of the subunit2sql tooling. It's been working
> out great so far.
Great. Did you notice a performance regression? Mike wrote that PyMySQL is much slower than MySQL-Python.
Victor
------------------------------
Message: 30
Date: Mon, 13 Apr 2015 11:22:38 +0300
From: Anastasia Urlapova <aurlapova at mirantis.com>
To: Andrey Sledzinskiy <asledzinskiy at mirantis.com>,
"mos-qa at mirantis.com" <mos-qa at mirantis.com>, "OpenStack Development
Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: [openstack-dev] [Fuel] Nominate Andrey Skedzinskiy for
fuel-qa(devops) core
Message-ID:
<CAC+Xjbbpm9jGap+j=qYayiJY8teqbRkJPjvZ0qieYVDDUFphOg at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Guys,
I would like to nominate Andrey Skedzinskiy[1] for
fuel-qa[2]/fuel-devops[3] core team.
Andrey is one of the strongest reviewers, under his watchful eye are such
features as:
- updrade/rollback master node
- collect usage information
- OS patching
- UI tests
and others
Please vote for Andrey!
Nastya.
[1]http://stackalytics.com/?project_type=stackforge&user_id=asledzinskiy
[2]https://github.com/stackforge/fuel-qa
[3]https://github.com/stackforge/fuel-devops
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/92ee8866/attachment-0001.html>
------------------------------
Message: 31
Date: Mon, 13 Apr 2015 11:37:44 +0300
From: Alexander Kislitsky <akislitsky at mirantis.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Cc: "mos-qa at mirantis.com" <mos-qa at mirantis.com>
Subject: Re: [openstack-dev] [Fuel] Nominate Andrey Skedzinskiy for
fuel-qa(devops) core
Message-ID:
<CAHWr6fkksK1-i3vSKvC01X5F7sGBQAgBt9W6q4CJjtQqNFLv=g at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Andrew shows great attention to the details. +1 for him.
On Mon, Apr 13, 2015 at 11:22 AM, Anastasia Urlapova <aurlapova at mirantis.com
> wrote:
> Guys,
> I would like to nominate Andrey Skedzinskiy[1] for
> fuel-qa[2]/fuel-devops[3] core team.
>
> Andrey is one of the strongest reviewers, under his watchful eye are such
> features as:
> - updrade/rollback master node
> - collect usage information
> - OS patching
> - UI tests
> and others
>
> Please vote for Andrey!
>
>
> Nastya.
>
> [1]http://stackalytics.com/?project_type=stackforge&user_id=asledzinskiy
> [2]https://github.com/stackforge/fuel-qa
> [3]https://github.com/stackforge/fuel-devops
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/8239f0df/attachment-0001.html>
------------------------------
Message: 32
Date: Mon, 13 Apr 2015 10:52:11 +0200
From: Miguel Angel Ajo Pelayo <majopela at redhat.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [all][pbr] splitting our deployment vs
install dependencies
Message-ID: <818AF012-6625-4766-B883-712D36A5851C at redhat.com>
Content-Type: text/plain; charset=us-ascii
> On 13/4/2015, at 3:53, Robert Collins <robertc at robertcollins.net> wrote:
>
> On 13 April 2015 at 13:09, Robert Collins <robertc at robertcollins.net> wrote:
>> On 13 April 2015 at 12:53, Monty Taylor <mordred at inaugust.com> wrote:
>>
>>> What we have in the gate is the thing that produces the artifacts that
>>> someone installing using the pip tool would get. Shipping anything with
>>> those artifacts other that a direct communication of what we tested is
>>> just mean to our end users.
>>
>> Actually its not.
>>
>> What we test is point in time. At 2:45 UTC on Monday installing this
>> git ref of nova worked.
>>
>> Noone can reconstruct that today.
>>
>> I entirely agree with the sentiment you're expressing, but we're not
>> delivering that sentiment today.
>
> This observation led to yet more IRC discussion and eventually
> https://etherpad.openstack.org/p/stable-omg-deps
>
> In short, the proposal is that we:
> - stop trying to use install_requires to reproduce exactly what
> works, and instead use it to communicate known constraints (> X, Y is
> broken etc).
> - use a requirements.txt file we create *during* CI to capture
> exactly what worked, and also capture the dpkg and rpm versions of
> packages that were present when it worked, and so on. So we'll build a
> git tree where its history is an audit trail of exactly what worked
> for everything that passed CI, formatted to make it really really easy
> for other people to consume.
>
That sounds like a very neat idea, this way we could look back, and backtrack
to discover which package version change breaks the system.
Miguel Angel Ajo
------------------------------
Message: 33
Date: Mon, 13 Apr 2015 08:51:39 +0000
From: Wangbibo <wangbibo at huawei.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: [openstack-dev] ??: [neutron] Neutron scaling datapoints?
Message-ID:
<EA5BE4191F3B3F4AB624BDC5B27A31A74E45D0A8 at nkgeml504-mbs.china.huawei.com>
Content-Type: text/plain; charset="utf-8"
Hi Kevin,
Totally agree with you that heartbeat from each agent is something that we cannot eliminate currently. Agent status depends on it, and further scheduler and HA depends on agent status.
I proposed a Liberty spec for introducing open framework/pluggable agent status drivers.[1][2] It allows us to use some other 3rd party backend to monitor agent status, such as zookeeper, memcached. Meanwhile, it guarantees backward compatibility so that users could still use db-based status monitoring mechanism as their default choice.
Base on that, we may do further optimization on issues Attila and you mentioned. Thanks.
[1] BP - https://blueprints.launchpad.net/neutron/+spec/agent-group-and-status-drivers
[2] Liberty Spec proposed - https://review.openstack.org/#/c/168921/
Best,
Robin
???: Kevin Benton [mailto:blak111 at gmail.com]
????: 2015?4?11? 12:35
???: OpenStack Development Mailing List (not for usage questions)
??: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Which periodic updates did you have in mind to eliminate? One of the few remaining ones I can think of is sync_routers but it would be great if you can enumerate the ones you observed because eliminating overhead in agents is something I've been working on as well.
One of the most common is the heartbeat from each agent. However, I don't think we can't eliminate them because they are used to determine if the agents are still alive for scheduling purposes. Did you have something else in mind to determine if an agent is alive?
On Fri, Apr 10, 2015 at 2:18 AM, Attila Fazekas <afazekas at redhat.com<mailto:afazekas at redhat.com>> wrote:
I'm 99.9% sure, for scaling above 100k managed node,
we do not really need to split the openstack to multiple smaller openstack,
or use significant number of extra controller machine.
The problem is openstack using the right tools SQL/AMQP/(zk),
but in a wrong way.
For example.:
Periodic updates can be avoided almost in all cases
The new data can be pushed to the agent just when it needed.
The agent can know when the AMQP connection become unreliable (queue or connection loose),
and needs to do full sync.
https://bugs.launchpad.net/neutron/+bug/1438159
Also the agents when gets some notification, they start asking for details via the
AMQP -> SQL. Why they do not know it already or get it with the notification ?
----- Original Message -----
> From: "Neil Jerram" <Neil.Jerram at metaswitch.com<mailto:Neil.Jerram at metaswitch.com>>
> To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
> Sent: Thursday, April 9, 2015 5:01:45 PM
> Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
>
> Hi Joe,
>
> Many thanks for your reply!
>
> On 09/04/15 03:34, joehuang wrote:
> > Hi, Neil,
> >
> > From theoretic, Neutron is like a "broadcast" domain, for example,
> > enforcement of DVR and security group has to touch each regarding host
> > where there is VM of this project resides. Even using SDN controller, the
> > "touch" to regarding host is inevitable. If there are plenty of physical
> > hosts, for example, 10k, inside one Neutron, it's very hard to overcome
> > the "broadcast storm" issue under concurrent operation, that's the
> > bottleneck for scalability of Neutron.
>
> I think I understand that in general terms - but can you be more
> specific about the broadcast storm? Is there one particular message
> exchange that involves broadcasting? Is it only from the server to
> agents, or are there 'broadcasts' in other directions as well?
>
> (I presume you are talking about control plane messages here, i.e.
> between Neutron components. Is that right? Obviously there can also be
> broadcast storm problems in the data plane - but I don't think that's
> what you are talking about here.)
>
> > We need layered architecture in Neutron to solve the "broadcast domain"
> > bottleneck of scalability. The test report from OpenStack cascading shows
> > that through layered architecture "Neutron cascading", Neutron can
> > supports up to million level ports and 100k level physical hosts. You can
> > find the report here:
> > http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers
>
> Many thanks, I will take a look at this.
>
> > "Neutron cascading" also brings extra benefit: One cascading Neutron can
> > have many cascaded Neutrons, and different cascaded Neutron can leverage
> > different SDN controller, maybe one is ODL, the other one is OpenContrail.
> >
> > ----------------Cascading Neutron-------------------
> > / \
> > --cascaded Neutron-- --cascaded Neutron-----
> > | |
> > ---------ODL------ ----OpenContrail--------
> >
> >
> > And furthermore, if using Neutron cascading in multiple data centers, the
> > DCI controller (Data center inter-connection controller) can also be used
> > under cascading Neutron, to provide NaaS ( network as a service ) across
> > data centers.
> >
> > ---------------------------Cascading Neutron--------------------------
> > / | \
> > --cascaded Neutron-- -DCI controller- --cascaded Neutron-----
> > | | |
> > ---------ODL------ | ----OpenContrail--------
> > |
> > --(Data center 1)-- --(DCI networking)-- --(Data center 2)--
> >
> > Is it possible for us to discuss this in OpenStack Vancouver summit?
>
> Most certainly, yes. I will be there from mid Monday afternoon through
> to end Friday. But it will be my first summit, so I have no idea yet as
> to how I might run into you - please can you suggest!
>
> > Best Regards
> > Chaoyi Huang ( Joe Huang )
>
> Regards,
> Neil
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Kevin Benton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/2569d73e/attachment.html>
------------------------------
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
End of OpenStack-dev Digest, Vol 36, Issue 32
*********************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150414/1f08a6b8/attachment-0001.html>
More information about the OpenStack-dev
mailing list