- openstack-discuss - lists.openstack.org

[release] Release countdown for week R-5, March 4-8
by Sean McGinnis 28 Feb '19

28 Feb '19

We are getting close to the end of the cycle. Please review some important deadlines and information as we approach feature freeze and the cycle ending activities. Development Focus ----------------- Teams should be focused on implementing planned work. Work should be wrapping up on client libraries to meet the client lib deadline Thursday, the 7th. This is also Feature Freeze, so please review anything to determine if should still be targeted for Stein or will need to be deferred to Train. General Information ------------------- The final client library release is on Thursday the 7th. Releases will only be allowed for critical fixes in libraries after this point as we stabilize requirements and give time for any unforeseen impacts from lib changes to trickle through. If release critical library or client library releases are needed for Stein past the freeze dates, you must request a Feature Freeze Exception (FFE) from the requirements team before we can do a new release to avoid having something released in Rocky that is not actually usable. This is done by posting to the openstack-discuss mailing list with a subject line similar to: [$PROJECT][requirements] FFE requested for $PROJECT_LIB Include justification/reasoning for why a FFE is needed for this lib. If/when the requirements team OKs the post-freeze update, we can then process a new release. Including a link to the FFE in the release request is not required, but would be helpful in making sure we are clear to do a new release. When requesting these library releases, you should also include the stable branching request with the review (as an example, see the "branches" section here: http://git.openstack.org/cgit/openstack/releases/tree/deliverables/pike/os-…) Cycle-trailing projects are reminded that all reviews to the requirements project will have a procedural -2 unless it recieves a FFE until stable/stein is branched. This is a good point to start to think about communicating your project's updates for Stein. There are two venues for this that should be kept in mind: 1. The Prelude section of reno release notes is rendered as the top level overview for the release. Any important overall messaging for Stein changes can be added there to make sure the consumers of your release notes see them. 2. Through the releases repo, teams should be adding cycle-highlights to their deliverables. This is the direct input into marketing that helps shape what is communicated in press releases and other release activity at the end of the cycle. These are not per-deliverable, but rolled up per-team. So any significant work your team has done that would be good for marketing, please include these by the RC1 deadline. Upcoming Deadlines & Dates -------------------------- Stein-3 milestone: March 7 RC1 deadline: March 21 -- Sean McGinnis (smcginnis)

1 0

Re: [placement][tripleo][infra] zuul job dependencies for greater good?
by Bogdan Dobrelya 28 Feb '19

28 Feb '19

> On Mon, 2019-02-25 at 19:42 -0500, Clark Boylan wrote: >> On Mon, Feb 25, 2019, at 12:51 PM, Ben Nemec wrote: >> > > snip > >> That said, I wouldn't push too hard in either direction until someone >> crunched the numbers and figured out how much time it would have saved >> to not run long tests on patch sets with failing unit tests. I feel like >> it's probably possible to figure that out, and if so then we should do >> it before making any big decisions on this. > > For numbers the elastic-recheck tool [0] gives us fairly accurate tracking of which issues in the system cause tests to fail. You can use this as a starting point to potentially figure out how expensive indentation errors caught by the pep8 jobs ends up being or how often unittests fail. You probably need to tweak the queries there to get that specific though. > > Periodically I also dump node resource utilization by project, repo, and job [1]. I haven't automated this because Tobiash has written a much better thing that has Zuul inject this into graphite and we should be able to set up a grafana dashboard for that in the future instead. > > These numbers won't tell a whole story, but should paint a fairly accurate high level picture of the types of things we should look at to be more node efficient and "time in gate" efficient. Looking at these two really quickly myself it seems that job timeouts are a big cost (anyone looking into why our jobs timeout?). > > [0] http://status.openstack.org/elastic-recheck/index.html > [1] http://paste.openstack.org/show/746083/ > > Hope this helps, > Clark Here is some numbers [0] extracted via elastic-recheck console queries. It shows 6% of wasted failures because of tox issues in general, and 3% for tripleo projects in particular. My final take is, given some middle-ground solution, like I illustrated earlier this sub-thread, it might be worth it, and the effort for boosting up the total throughput of openstack CI system by a 6% is not so bad idea. [0] http://paste.openstack.org/show/746503/ -- Best regards, Bogdan Dobrelya, Irc #bogdando

3 5

[nova] stein-rc-potential tag and etherpad
by Matt Riedemann 28 Feb '19

28 Feb '19

There is one week to feature freeze and a lot going on, so I created an etherpad to help track things that we don't want to forget about before RC1 on March 21: https://etherpad.openstack.org/p/nova-stein-rc-potential I also created the official "stein-rc-potential" tag for nova in launchpad so you can tag bugs with it if you think they are install/upgrade/major regressions that need to be sorted out prior to releasing Stein. -- Thanks, Matt

1 0

[all] [forum] Forum Submissions are open!
by Jimmy McArthur 28 Feb '19

28 Feb '19

Hi Everyone - A quick reminder that we are accepting Forum [1] submissions for the 2019 Open Infrastructure Summit in Denver [2]. Please submit your ideas through the Summit CFP tool [3] through March 8th. Don't forget to put your brainstorming etherpad up on the Denver Forum page [4]. This is not a classic conference track with speakers and presentations. OSF community members (participants in development teams, operators, working groups, SIGs, and other interested individuals) discuss the topics they want to cover and get alignment on and we welcome your participation. The Forum is your opportunity to help shape the development of future project releases. More information about the Forum [1]. If you have questions or concerns, please reach out to speakersupport(a)openstack.org <mailto:speakersupport@openstack.org>. Cheers, Jimmy [1] https://wiki.openstack.org/wiki/Forum [2] https://www.openstack.org/summit/denver-2019/ [3] https://www.openstack.org/summit/denver-2019/call-for-presentations [4] https://wiki.openstack.org/wiki/Forum/Denver2019 ___________________________________________

8 16

[dev][oslo] oslo.cache and dogpile 0.7.0+ cache errors
by Herve Beraud 28 Feb '19

28 Feb '19

Hi, Just a heads up that the latest version of dogpile (0.7.0 onwards) have become incompatible with oslo.cache. This is causing a few issues for jobs. It's a little complex due to functional code and many decorated functions. The error you will see is: *oslo_cache.**tests.test_**cache.CacheRegi**onTest.**test_function_* *key_generator_* *with_kwargs -------**-------**-------**-------**-------**-------**-------* *-------**-------**-------**-------**------* *Captured traceback: ~~~~~~~~~~~~~~~~~~~ b'Traceback (most recent call last):' b' File "/tmp/oslo.**cache/oslo_**cache/tests/**test_cache.**py", line 324, in test_function_**key_generator_* *with_kwargs' b' value=self.**test_value)* *' b' File "/tmp/oslo.**cache/.**tox/py37/**lib/python3.**7/site-* *packages/**testtools/**testcase.* *py", line 485, in assertRaises' b' self.assertThat* *(our_callable, matcher)' b' File "/tmp/oslo.**cache/.**tox/py37/* *lib/python3.**7/site-**packages/**testtools/**testcase.* *py", line 498, in assertThat' b' raise mismatch_error' b'testtools* *.matchers.**_impl.MismatchE**rror: <function CacheRegionTest* *._get_cacheable**_function.**<locals>**.cacheable_**function at 0x7fec3f795400> returned <oslo_cache.**tests.test_**cache.TestProxy**Value object at 0x7fec3f792550>'* The problem appear since we uncap dogpile.cache on oslo.cache: https://github.com/openstack/oslo.cache/commit/62b53099861134859482656dc92d… The following unit test fail since we uncap dogpile => https://github.com/openstack/oslo.cache/blob/master/oslo_cache/tests/test_c… The problem was introduced by: https://gerrit.sqlalchemy.org/#/c/sqlalchemy/dogpile.cache/+/996/ Your main issue on oslo.cache side is that keyword arguments are tranformed in positionnal arguments when we use dogpile.cache.region.cache_on_arguments. I've try to revert the changes introduced by the previous dogpile.cache change and everything works fine on the oslo.cache side when changes was reverted (reverted to revision https://github.com/sqlalchemy/dogpile.cache/blob/2762ada1f5e43075494d91c512… ). The expected behavior is that dogpile.cache.util.function_key_generator raise a ValueError if **kwargs founds, but our kwargs is empty and our `value=self.test_value was` is recognized as a positionnal argument. Our unit test looking for an assertRaise(ValueError) on cachable decorated function when we pass kwargs but it doesn't happen due to empty kwargs. For these reasons we guess that is an dogpile.cache issue and not an oslo.cache issue due to the changes introduced by `decorator` module. The following are related: - https://github.com/openstack/oslo.cache/blob/master/oslo_cache/tests/test_c… : unit test where the problem occure - https://review.openstack.org/#/c/638788/ : possible fix but we don't think that is the right way - https://review.openstack.org/#/c/638732/ : possible remove of the unit test who fail The issue is being tracked in: https://bugs.launchpad.net/oslo.cache/+bug/1817032 If some dogpile expert can take a look and send feedback on this thread you are welcome. Thanks, -- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

3 6

Issue with ceph and vault
by Wouter van Bommel 28 Feb '19

28 Feb '19

Hi, I currently bumped into a situation in a cloud using both ceph and vault. While adding a new node to the cloud all seemed fine, until I noticed that on one node juju failed with the message: hook failed: "secrets-storage-relation-changed" Upon investigation it turned out that this node had a broken network configuration that was not picket up earlier. The problem was, that the vault instance could not be reached. This was fixed, but I could not un-wedge the error. A juju resolve on the node does not seem to have any effect. Using the debug-log option from juju I do see the following lines that do indicate the problem. INFO:vaultlocker.dmcrypt:LUKS formatting /dev/disk/by-dname/osd1-part1 using UUID:4448f9fa-d291-403e-9826-7318e57cf1a4 Cannot format device /dev/disk/by-dname/osd1-part1 which is still in use. vaultlocker: Command '['cryptsetup', '--batch-mode', '--uuid', '4448f9fa-d291-403e-9826-7318e57cf1a4', '--key-file', '-', 'luksFormat', '/dev/disk/by-dname/osd1-part1']' returned non-zero exit status 5 So the disk is in use. Which looks strange to me, as this machine could never complete the hooks as the vault was unreachable. I would like to get some advice on how to proceed. Regrds, Wouter

2 1

[tc][election] campaign question: team approval criteria
by Doug Hellmann 28 Feb '19

28 Feb '19

One of the key responsibilities of the Technical Committee is still evaluating projects and teams that want to become official OpenStack projects. The Foundation Open Infrastructure Project approval process has recently produced a different set of criteria for the Board to use for approving projects [1] than the TC uses for approving teams [2]. What parts, if any, of the OIP approval criteria do you think should apply to OpenStack teams? What other changes, if any, would you propose to the official team approval process or criteria? Are we asking the right questions and setting the minimum requirements high enough? Are there any criteria that are too hard to meet? How would you apply those rule changes to existing teams? [1] http://lists.openstack.org/pipermail/foundation/2019-February/002708.html [2] https://governance.openstack.org/tc/reference/new-projects-requirements.html -- Doug

7 12

Re: [placement][TripleO][infra] zuul job dependencies for greater good?
by Bogdan Dobrelya 28 Feb '19

28 Feb '19

I think we can still consider the middle-ground, where only deprecated multinode jobs, which tripleo infra team is in progress of migrating into standalone jobs, could be made depending on unit and pep8 checks? And some basic jobs will keep being depending on nothing. I expanded that idea in WIP topic [0]. Commit messages explain how the ordering was reworked. PS. I'm sorry I missed the submitted stats for zuul projects posted earlier in this topic, I'll take a look into that. [0] https://review.openstack.org/#/q/topic:ci_pipelines+(status:open+OR+status:… > Bogdan Dobrelya <bdobreli at redhat.com> writes: >> On 26.02.2019 17:53, James E. Blair wrote: >>> Bogdan Dobrelya <bdobreli at redhat.com> writes: >>> >>>> I attempted [0] to do that for tripleo-ci, but zuul was (and still >>>> does) complaining for some weird graphs building things :/ >>>> >>>> See also the related topic [1] from the past. >>>> >>>> [0] https://review.openstack.org/#/c/568543 >>>> [1] http://lists.openstack.org/pipermail/openstack-dev/2018-March/127869.html >>> >>> Thank you for linking to [1]. It's worth re-reading. Especially the >>> part at the end. >>> >>> -Jim >>> >> > > Yes, the part at the end is the best indeed. > I'd amend the time priorities graph though like that: > > CPU-time < a developer time < developers time > > That means burning some CPU and nodes in a pool for a waste might > benefit a developer, but saving some CPU and nodes in a pool would > benefit *developers* in many projects as they'd get the jobs results off > the waiting check queues faster :) -- Best regards, Bogdan Dobrelya, Irc #bogdando

1 2

[ceilometer] radosgw pollster
by Florian Engelmann 28 Feb '19

28 Feb '19

Hi, I failed to poll any usage data from our radosgw. I get 2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.containers.objects in the context of radosgw_300s_pollsters 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent pollster radosgw.containers.objects from polling [<Project description=, domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True, [...] id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self': u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source radosgw_300s_pollsters anymore!: PollsterPermanentError Configurations like: cat polling.yaml --- sources: - name: radosgw_300s_pollsters interval: 300 meters: - radosgw.usage - radosgw.objects - radosgw.objects.size - radosgw.objects.containers - radosgw.containers.objects - radosgw.containers.objects.size Also tried radosgw.api.requests instead of radowsgw.usage. ceilometer.conf [...] [service_types] radosgw = object-store [rgw_admin_credentials] access_key = xxxxx0Z0xxxxxxxxxxxx secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx [rgw_client] implicit_tenants = true Endpoints: | xxxxxxx | region | swift | object-store | True | admin | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | internal | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | public | https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s | Ceilometer user: { "user_id": "ceilometer", "display_name": "ceilometer", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "ceilometer", "access_key": "xxxxxxxxxxxxxxxxxx", "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx" } ], "swift_keys": [], "caps": [ { "type": "buckets", "perm": "read" }, { "type": "metadata", "perm": "read" }, { "type": "usage", "perm": "read" }, { "type": "users", "perm": "read" } ], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" } radosgw config: [client.rgw.xxxxxxxxxxx] host = somehost rgw frontends = "civetweb port=7480 num_threads=512" rgw num rados handles = 8 rgw thread pool size = 512 rgw cache enabled = true rgw dns name = s3.xxxxxx.xxx rgw enable usage log = true rgw usage log tick interval = 30 rgw realm = public rgw zonegroup = xxx rgw zone = xxxxx rgw resolve cname = False rgw usage log flush threshold = 1024 rgw usage max user shards = 1 rgw usage max shards = 32 rgw_keystone_url = https://keystone.xxxxxxxxxxxxx rgw_keystone_admin_domain = default rgw_keystone_admin_project = service rgw_keystone_admin_user = swift rgw_keystone_admin_password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx rgw_keystone_accepted_roles = member,_member_,admin rgw_keystone_accepted_admin_roles = admin rgw_keystone_api_version = 3 rgw_keystone_verify_ssl = false rgw_keystone_implicit_tenants = true rgw_keystone_admin_tenant = default rgw_keystone_revocation_interval = 0 rgw_keystone_token_cache_size = 0 rgw_s3_auth_use_keystone = true rgw_max_attr_size = 1024 rgw_max_attrs_num_in_req = 32 rgw_max_attr_name_len = 64 rgw_swift_account_in_url = true rgw_swift_versioning_enabled = true rgw_enable_apis = s3,swift,swift_auth,admin rgw_swift_enforce_content_length = true Any idea whats going on? All the best, Florian

3 7

[nova][dev] Any VMware resource pool and shares kind of feature available in openstack nova?
by Sanjay K 28 Feb '19

28 Feb '19

Hi Matt, I will define/derive priority based on the which sub network the VM belongs to - mostly Production or Development. From this, the Prod VMs will have higher resource allocation criteria than other normal VMs and these can be calculated at runtime when a VM is also rebooted like how VMware resource pools and shares features work. I appreciate any other suggestions. thanks and regards,

4 3