- openstack-discuss - lists.openstack.org

[openlab] November Update
by Melvin Hillsman 30 Nov '18

30 Nov '18

Hi everyone, We should have been doing this - totally my fault here - and hope you find updates at least monthly re OpenLab useful. In order to allow all people participating in OpenLab to provide details on the updates we are using etherpad: https://etherpad.openlabtesting.org/p/november2018update Thanks to all the community members helping to make OpenLab successful! -- Kind regards, Melvin Hillsman mrhillsman(a)gmail.com mobile: (832) 264-2646

2 1

[ironic] Finding a time to discuss cleaning/deployment steps visibility
by Julia Kreger 30 Nov '18

30 Nov '18

Greetings everyone, As discussed previously, I'd like to find a time when we can get on a higher bandwidth medium to discuss making clean and deployment steps visible. Generally we've struggled to gain consensus and I'd like for us to focus on trying to reach a minimally viable feature. I've created a doodle at: https://doodle.com/poll/yan4wyvztf7mpq46 Most recent specification: https://review.openstack.org/#/c/606199/ -Julia

1 0

[all] [infra] StoryBoard Migration: The Remaining Blockers Summary
by Kendall Nelson 30 Nov '18

30 Nov '18

Hello! As with all of my other summaries, if you are looking for the long etherpad version, you can find it here[1]. Somewhere along the way I used the wrong etherpad for notes and so we ended up doing the StoryBoard things in the etherpad originally meant for another session so pardon the name. Anywho. Overall the session went really well, there weren’t any huge glaring issues that anyone brought up. The majority of things were all smaller usability things. Just about all of which are definitely doable. The only slightly larger feature request (that we hadn't previously discussed) was a separate interface for filing stories- similar to the one LP provides. Something simple so the user doesn’t feel like they need to fill in all this information, but can just provide the minimum. I haven’t quite gotten around to making stories for all of the things discussed in the etherpad, but should have them up soon! You will be able to see them here[2], once they’ve been created. If you have opinions about any of what we discussed or have something you wanted to bring up but could not attend the session please drop into #storyboard and chat with us, or join us at our weekly meeting on Wednesday at 19:00 UTC. -Kendall (diablo_rojo) [1]https://etherpad.openstack.org/p/BER-Contrib-Portal-Feedback (yes, I know the name doesn’t seem right..) [2] https://storyboard.openstack.org/#!/story/list?status=active&project_group_…

2 2

[tripleo] gate is backed up (again)
by Alex Schultz 29 Nov '18

29 Nov '18

Ahoy folks. We're running into a massive backlog and it's messing with everyone again. As stop gap, we're clearing the gate and will need to land the patches to switch them back to non-voting and remove the scenarios[0] from the gate in stable branches. We had switched them back to voting ~2 weeks ago because they were still in the gate (but non-voting). I thought we weren't going to run into as many issues in the stable branches but we've been backporting a bit more lately. We are proposing this because in the gate we've had a bunch of scenario failures[1] not related to code but just time outs. Additionally we're making progress on converting the master scenario jobs to standalone versions which may be backportable to stable/rocky. So in the mean time, we're asking that you not approve or recheck anything until the CI has stabilized. We'll send out a note when we're clear. Thanks, -Alex [0] https://review.openstack.org/#/q/topic:scenarios-nv+(status:open+OR+status:… [1] http://zuul.openstack.org/builds?pipeline=gate&result=failure&result=timed_…

1 0

Re: [Openstack-operators] Fwd: Nova hypervisor uuid
by Ignazio Cassano 29 Nov '18

29 Nov '18

Hi Matt, sorry but I lost your answer and Gianpiero forwarded it to me. I am sure kvm nodes names are note changed. Tables where uuid are duplicated are: dataresource_providers in nova_api db compute_nodes in nova db Regards Ignazio Il 28/Nov/2018 11:09 AM, "Gianpiero Ardissono" <gp.ardissono(a)gmail.com> ha scritto: > > ---------- Forwarded message --------- > From: Matt Riedemann <mriedemos(a)gmail.com> > Date: mar 27 nov 2018, 19:03 > Subject: Re: [Openstack-operators] Nova hypervisor uuid > To: <openstack-operators(a)lists.openstack.org> > > > On 11/27/2018 11:32 AM, Ignazio Cassano wrote: > > Hi All, > > Please anyone know where hypervisor uuid is retrived? > > Sometime updating kmv nodes with yum update it changes and in nova > > database 2 uuids are assigned to the same node. > > regards > > Ignazio > > > > > > > > > > _______________________________________________ > > OpenStack-operators mailing list > > OpenStack-operators(a)lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > > > > To be clear, do you mean the computes_nodes.uuid column value in the > cell database? Which is also used for the GET /os-hypervisors response > 'id' value if using microversion >= 2.53. If so, that is generated > randomly* when the compute_nodes table record is created: > > > https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d6… > > > https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d6… > > When you hit this problem, are you sure the hostname on the compute host > is not changing? Because when nova-compute starts up, it should look for > the existing compute node record by host name and node name, which for > the libvirt driver should be the same. That lookup code is here: > > > https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d6… > > So the only way nova-compute should create a new compute_nodes table > record for the same host is if the host/node name changes during the > upgrade. Is the deleted value in the database the same (0) for both of > those records? > > * The exception to this is for the ironic driver which re-uses the > ironic node uuid as of this change: > https://review.openstack.org/#/c/571535/ > > -- > > Thanks, > > Matt > > _______________________________________________ > OpenStack-operators mailing list > OpenStack-operators(a)lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > _______________________________________________ OpenStack-operators mailing list OpenStack-operators(a)lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

2 6

[release][ptl] Holding all release activity
by Sean McGinnis 29 Nov '18

29 Nov '18

Hey everyone, There have been a rash of Pypi and other issues lately that have led to release job failures. We will be holding off on processing any more releases until we've seen some evidence that things have been resolved. I would guess this means that there will be no releases done until early next week. If you have a critical need for a release, please come talk to us in the #openstack-release channel and we can see if there is anything we can do in the interim. Thanks! Sean

1 1

[glance][ops][users]OpenStack Glance default visibility survey
by ekuvaja＠redhat.com 29 Nov '18

29 Nov '18

I've invited you to fill out the following form: OpenStack Glance default visibility survey To fill it out, visit: https://docs.google.com/forms/d/e/1FAIpQLScbbmeLQrx34P84U-Wr2RoLOpXdWMEZg-X… Hello all, Time for another very quick survey from Glance team. This time we would like to get your feedback on if we should set the default visibility of newly created images to "private" or not. Please note this survey closes in a week and we will take the action based on the results provided. Erno jokke Kuvaja I've invited you to fill out a form: Google Forms: Create and analyze surveys.

1 0

[release] Release countdown for week R-18, Dec 3-7
by Sean McGinnis 29 Nov '18

29 Nov '18

Welcome back from the Summit and holidays. Back to our regularly scheduled countdown emails. Development Focus ----------------- With the longer development cycle we are still a few weeks out until milestone 2. Hopefully folks got some good input and feedback and the Summit and Forum. Teams should be taking input and adjusting any earlier plans as needed based on this. General Information ------------------- Now may be a good time for teams to check on any stable releases that need to be done for your deliverables. If you have bugfixes that have been backported, but no stable release getting those. If you are unsure what is out there committed but not released, in the openstack/releases repo, running the command "tools/list_stable_unreleased_changes.sh <cycle_name>" gives a nice report. Upcoming Deadlines & Dates -------------------------- Start using openstack-discuss ML: November 19 (in case you missed it) Stein-2 Milestone: January 10 -- Sean McGinnis (smcginnis)

1 0

[dev] [tc][all] TC office hours is started now on #openstack-tc
by Ghanshyam Mann 29 Nov '18

29 Nov '18

Hi All, TC office hour is started on #openstack-tc channel. Feel free to reach to us for anything you want discuss/input/feedback/help from TC. - TC

1 0

Re: [openstack-dev] [TripleO][Edge] Reduce base layer of containers for security and size of images (maintenance) sakes
by Bogdan Dobrelya 29 Nov '18

29 Nov '18

On 11/29/18 2:00 PM, Jay Pipes wrote: > On 11/29/2018 04:28 AM, Bogdan Dobrelya wrote: >> On 11/28/18 8:55 PM, Doug Hellmann wrote: >>> I thought the preferred solution for more complex settings was config >>> maps. Did that approach not work out? >>> >>> Regardless, now that the driver work is done if someone wants to take >>> another stab at etcd integration it’ll be more straightforward today. >>> >>> Doug >>> >> >> While sharing configs is a feasible option to consider for large scale >> configuration management, Etcd only provides a strong consistency, >> which is also known as "Unavailable" [0]. For edge scenarios, to >> configure 40,000 remote computes over WAN connections, we'd rather >> want instead weaker consistency models, like "Sticky Available" [0]. >> That would allow services to fetch their configuration either from a >> central "uplink" or locally as well, when the latter is not accessible >> from remote edge sites. Etcd cannot provide 40,000 local endpoints to >> fit that case I'm afraid, even if those would be read only replicas. >> That is also something I'm highlighting in the paper [1] drafted for >> ICFC-2019. >> >> But had we such a sticky available key value storage solution, we >> would indeed have solved the problem of multiple configuration >> management system execution for thousands of nodes as James describes it. > > It's not that etcd is incapable of providing something like this. It's > that a *single* etcd KVS used by 40K compute nodes across a > disaggregated control plane would not be available to all of those nodes > simultaneously. > > But you could certainly use etcd as the data store to build a sticky > available configuration data store. If, for example, you had many local > [1] etcd KVS that stored local data and synchronized the local data set > with other etcd KVS endpoints when a network partition was restored, you > could get such a system that was essentially "sticky available" for all > intents and purposes. > > Come to think of it, you could do the same with a SQLite DB, ala Swift's > replication of SQLite DBs via rsync. > > But, at the risk of sounding like a broken record, at the end of the > day, many of OpenStack's core services -- notably Nova -- were not > designed for disaggregated control planes. They were designed for the > datacenter, with tightly-packed compute resources and low-latency links > for the control plane. > > The entire communication bus and state management system would need to > be redesigned from the nova-compute to the nova-conductor for (far) edge > case clouds to be a true reality. > > Instead of sending all data updates synchronously from each nova-compute > to nova-conductor, the communication bus needs to be radically > redesigned so that the nova-compute uses a local data store *as its > primary data storage* and then asynchronously sends batched updates to > known control plane endpoints when those regular network partitions > correct themselves. > > The nova-compute manager will need to be substantially hardened to keep > itself up and running (and writing to that local state storage) for long > periods of time and contain all the logic to resync itself when network > uplinks become available again. > > Finally, if those local nova-computes need to actually *do* anything > other than keep existing VMs/baremetal machines up and running, then a > local Compute API service needs to be made available in the far edge > sites themselves -- offering some subset of Compute API functionality to > control the VMs in that local site. Otherwise, the whole "multiple > department stores running an edge OpenStack site that can tolerate the > Mother Ship being down" isn't a thing that will work. > > Like I said, pretty much a complete redesign of the nova control plane... We derived a little bit off the topic... but that all is valid for the post-MVP Edge architecture phases [0] targeted for multiple (aka disaggregated/autonomous/local vs central) control planes, indeed. Although there are more options than that complete redesign. IIUC, does the latter assume supporting alternative to SQL/AMQP-ish data/messaging backends for Nova and OpenStack in general? That is only an option (see such backends examples [1][2]), though I love it the most :) Other options may be creating client libraries acting on top of APIs or existing DB/MQ backends and performing low-level data synchronization, or acting as an API re-translators, over multiple control planes. And AFAICT that would *not* require complete redesign of supported backends nor types of transactions in Nova et al. And for MQ, a brokerless qdr or something (there was a nice presentation at the summit)... But in the end, indeed, it is kinda proved in multiple R&D papers, like [3],[4] that only causal sticky consistent synchronization with advanced conflicts resolving [5] is the best Edge-y/Fog-y choice for both such client libraries and causal consistent DB/KVS/MQ backends. I think that is something similar what you (Jay) diescribed for multiple Etcd cluster exchanging its data? So for that example, such client libraries should be maintaining sticky sessions to groups of those Etcd clusters and replicate data around doing it the best of causal consistent ways. PS. That a nice SQLight & rsync combo would not provide us the best of eventual consistency world, no, it would rather be something of a "Total Available" [6] thing, the lowest of it, like Read Uncommited or Monotonic Writes, and would be a very (very) poor choice IMO. [0] https://wiki.openstack.org/w/index.php?title=OpenStack_Edge_Discussions_Dub… [1] https://www.ronpub.com/OJDB_2015v2i1n02_Elbushra.pdf [2] http://rainbowfs.lip6.fr/data/RainbowFS-2016-04-12.pdf [3] https://www.cs.cmu.edu/~dga/papers/cops-sosp2011.pdf [4] http://www.cs.cornell.edu/lorenzo/papers/cac-tr.pdf [5] https://ieeexplore.ieee.org/document/8104644 [6] https://jepsen.io/consistency > > Best, > -jay > > [1] or local-ish, think POPs or even local to the compute node itself... > >> [0] https://jepsen.io/consistency >> [1] >> https://github.com/bogdando/papers-ieee/blob/master/ICFC-2019/LaTeX/positio… -- Best regards, Bogdan Dobrelya, Irc #bogdando

1 0