August 2019 - openstack-discuss - lists.openstack.org

Re: [openstack-dev] Dropping lazy translation support
by Shengjing Zhu 28 Aug '19

28 Aug '19

Sorry for replying the old mail, and please cc me when reply. Matt Riedemann <mriedemos at gmail.com> writes: > This is a follow up to a dev ML email [1] where I noticed that some > implementations of the upgrade-checkers goal were failing because some > projects still use the oslo_i18n.enable_lazy() hook for lazy log message > translation (and maybe API responses?). > > The very old blueprints related to this can be found here [2][3][4]. > > If memory serves me correctly from my time working at IBM on this, this > was needed to: > > 1. Generate logs translated in other languages. > > 2. Return REST API responses if the "Accept-Language" header was used > and a suitable translation existed for that language. > > #1 is a dead horse since I think at least the Ocata summit when we > agreed to no longer translate logs since no one used them. > > #2 is probably something no one knows about. I can't find end-user > documentation about it anywhere. It's not tested and therefore I have no > idea if it actually works anymore. > > I would like to (1) deprecate the oslo_i18n.enable_lazy() function so > new projects don't use it and (2) start removing the enable_lazy() usage > from existing projects like keystone, glance and cinder. > > Are there any users, deployments or vendor distributions that still rely > on this feature? If so, please speak up now. I was pointed to this discussion when I tried to fix this feature in keystone, https://review.opendev.org/677117 For #2 translated API response, this feature probably hasn't been working for some time, but it's still a valid user case. Has the decision been settled? -- Regards, Shengjing Zhu

3 4

[openstack-ansible] weekly update
by Mohammed Naser 28 Aug '19

28 Aug '19

Hi everyone, Here’s the update of what happened in this week’s OpenStack Ansible Office Hours. - The uwsgi role is mostly done but there’s an issue with ansible-lint checks so we’re thinking of moving it to the integrated repository and towards using collections in the long term. - We’re still working on the os_murano role. - We finally fixed Rocky POST_FAILURES (thanks infra!) - Rabbit and Galera are still stuck on bind-to-mgmt work.. - We will also need new code in order to build images in test jobs for Octavia since right now they’re only usable in Master. Thanks! Regards, Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. http://vexxhost.com

1 0

No manila meeting 29 August 2019
by Tom Barron 28 Aug '19

28 Aug '19

We have quite a few folks unavailable this Thursday and will cancel this week's Manila community meeting. The next meeting will be the following week, 5 September, at 1500 UTC as usual.

1 0

[nova] ComputeDriver.spawn getting a power_on kwarg
by Matt Riedemann 28 Aug '19

28 Aug '19

This is just an FYI to out of tree virt driver maintainers that the ComputeDriver.spawn interface is getting a new power_on kwarg with this chnage: https://review.opendev.org/#/c/642590/ -- Thanks, Matt

1 0

FWAAS V2 doesn't work with DVR
by Salman Khan 28 Aug '19

28 Aug '19

Hi Guys, I asked this question over #openstack-neutron channel but didn't get any answer, so asking here in a hope that someone might read this email and reply. The problem is: I have enabled FWAAS_V2 with DVR and that doesn't seem to work. I debugged things down to router namespaces and it looks like iptables rules are applied to rfp-<network-id> interface which doesn't exist in that namespace. So rules are completely wrong as they are applied to an interface that doesn't exist, I mean there is rfp-* interface but the <network-id> that fwaas expecting is not what it should be. I tried applying the rules to qr-* interfaces in the namespace but that didn't work as well, packets are dropping on "invalid" state rule. That's probably because of nat rules from dvr. Can someone please help me to understand this behaviour. Is it really suppose to work or not. If there is any bug or fix pending or there is any work ongoing to support this. Regards, Salman

2 2

[scientific-sig] SIG IRC meeting today 1100 UTC
by Stig Telfer 28 Aug '19

28 Aug '19

Hi All - We have an IRC meeting today at 1100 UTC (about an hour’s time) in channel #openstack-meeting. Everyone is welcome. Today’s agenda is here: https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_August_28th_2019 <https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_August_28th_2019> Cheers, Stig

1 0

[glance][neutron] Glance migrations broken for postgresql; impact to Neutron periodic jobs
by Nate Johnston 28 Aug '19

28 Aug '19

Glance developers, I am investigating failures with the neutron-tempest-postgres-full periodic job in Zuul, and it looks like the failing step is actually a database migration for Glance that occurs during devstack setup. Pasted below is a representative example [1]. You can see the same issue causing failures for this neutron periodic job by looking at the Zuul failures [2]; the error is in logs/devstacklog.txt.gz. The last successful job was 2019-08-15T06:07:06; the next one on 2019-08-16 failed. But there were no changes merged in glance during that time period [3]. So if you could help debug the source of this issue then we can hopefully get this job - and probably whatever others depend on postgresql - back working. 2019-08-27 06:35:44.963 | INFO [alembic.runtime.migration] Running upgrade rocky_expand02 -> train_expand01, empty expand for symmetry with train_contract01 2019-08-27 06:35:44.967 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl. 2019-08-27 06:35:44.967 | INFO [alembic.runtime.migration] Will assume transactional DDL. 2019-08-27 06:35:44.970 | Upgraded database to: train_expand01, current revision(s): train_expand01 2019-08-27 06:35:44.971 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl. 2019-08-27 06:35:44.971 | INFO [alembic.runtime.migration] Will assume transactional DDL. 2019-08-27 06:35:44.975 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl. 2019-08-27 06:35:44.975 | INFO [alembic.runtime.migration] Will assume transactional DDL. 2019-08-27 06:35:44.995 | CRITI [glance] Unhandled error 2019-08-27 06:35:44.996 | Traceback (most recent call last): 2019-08-27 06:35:44.996 | File "/usr/local/bin/glance-manage", line 10, in <module> 2019-08-27 06:35:44.996 | sys.exit(main()) 2019-08-27 06:35:44.996 | File "/opt/stack/new/glance/glance/cmd/manage.py", line 563, in main 2019-08-27 06:35:44.996 | return CONF.command.action_fn() 2019-08-27 06:35:44.996 | File "/opt/stack/new/glance/glance/cmd/manage.py", line 395, in sync 2019-08-27 06:35:44.996 | self.command_object.sync(CONF.command.version) 2019-08-27 06:35:44.996 | File "/opt/stack/new/glance/glance/cmd/manage.py", line 166, in sync 2019-08-27 06:35:44.996 | self.migrate(online_migration=False) 2019-08-27 06:35:44.996 | File "/opt/stack/new/glance/glance/cmd/manage.py", line 294, in migrate 2019-08-27 06:35:44.996 | if data_migrations.has_pending_migrations(db_api.get_engine()): 2019-08-27 06:35:44.996 | File "/opt/stack/new/glance/glance/db/sqlalchemy/alembic_migrations/data_migrations/__init__.py", line 61, in has_pending_migrations 2019-08-27 06:35:44.997 | return any([x.has_migrations(engine) for x in migrations]) 2019-08-27 06:35:44.997 | File "/opt/stack/new/glance/glance/db/sqlalchemy/alembic_migrations/data_migrations/train_migrate01_backend_to_store.py", line 28, in has_migrations 2019-08-27 06:35:44.997 | metadata_backend = con.execute(sql_query) 2019-08-27 06:35:44.997 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 982, in execute 2019-08-27 06:35:44.997 | return self._execute_text(object_, multiparams, params) 2019-08-27 06:35:44.997 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1155, in _execute_text 2019-08-27 06:35:44.997 | parameters, 2019-08-27 06:35:44.997 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context 2019-08-27 06:35:44.997 | e, statement, parameters, cursor, context 2019-08-27 06:35:44.997 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1464, in _handle_dbapi_exception 2019-08-27 06:35:44.997 | util.raise_from_cause(newraise, exc_info) 2019-08-27 06:35:44.997 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause 2019-08-27 06:35:44.997 | reraise(type(exception), exception, tb=exc_tb, cause=cause) 2019-08-27 06:35:44.997 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context 2019-08-27 06:35:44.998 | cursor, statement, parameters, context 2019-08-27 06:35:44.998 | File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 552, in do_execute 2019-08-27 06:35:44.998 | cursor.execute(statement, parameters) 2019-08-27 06:35:44.998 | DBError: (psycopg2.errors.UndefinedFunction) function instr(text, unknown) does not exist 2019-08-27 06:35:44.998 | LINE 1: select meta_data from image_locations where INSTR(meta_data,... 2019-08-27 06:35:44.998 | ^ 2019-08-27 06:35:44.998 | HINT: No function matches the given name and argument types. You might need to add explicit type casts. Thanks, Nate [1] also http://paste.openstack.org/show/765665/ if that is an easier view [2] http://zuul.openstack.org/builds?project=openstack%2Fneutron&job_name=neutr… [3] https://review.opendev.org/#/q/project:openstack/glance+status:merged

2 1

Re: [ops] [glance] Need to move images to a different backend
by Massimo Sgaravatto 28 Aug '19

28 Aug '19

Hi Abhishek Yes, before trying the: glance location-add --url rbd://8162f291-00b6-4b40-a8b4-1981a8c09b64/images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628/snap 6bcc4eab-ed35-42dc-88bd-1d45de73b628 command, I uploaded the image to ceph. And indeed I can see it: [root@ceph-mon-01 ~]# rbd info images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628 rbd image '6bcc4eab-ed35-42dc-88bd-1d45de73b628': size 10GiB in 1280 objects order 23 (8MiB objects) block_name_prefix: rbd_data.b7b56aa1e4f0bc format: 2 features: layering flags: create_timestamp: Tue Aug 27 11:42:32 2019 [root@ceph-mon-01 ~]# rbd snap ls images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628 SNAPID NAME SIZE TIMESTAMP 455500 snap 10GiB Tue Aug 27 11:46:37 2019 [root@ceph-mon-01 ~]# Cheers, Massimo On Tue, Aug 27, 2019 at 2:41 PM Abhishek Kekane <akekane(a)redhat.com> wrote: > Hi Massimo, > > I need to reproduce this first, but pretty much sure this issue is not > because you configured multiple backends. What you are doing is downloading > existing image to your local storage and then uploading it to glance using > add-location operation. > > So before adding this using add-location operation have you manually > uploaded this image to ceph/rbd? if not then how you are building your > location url which you are mentioning in the add-location command? If this > location is not existing then it might be a problem. > > Thank you, > > Abhishek > > On Tue, 27 Aug 2019 at 5:09 PM, Massimo Sgaravatto < > massimo.sgaravatto(a)gmail.com> wrote: > >> I have a Rocky installation where glance is configured with multiple >> backends. This [*] is the relevant part of the glance configuration. >> >> I now need to dismiss the file backend and move the images-snapshots >> stored there to rbd. >> >> >> I had in mind to follow this procedure (already successfully tested with >> an older version of OpenStack): >> >> 1) download the image from the file backend (glance image-download --file >> <file> <imageid> >> 2) upload the file <file> to rbd, relying on this function: >> https://github.com/openstack/glance_store/blob/stable/rocky/glance_store/_d… >> 3) glance location-add --url rbd://xyz <imageid> >> 4) glance location-delete --url file://abc <imageid> >> >> I have problems with 3: >> >> # glance --debug location-add --url >> rbd://8162f291-00b6-4b40-a8b4-1981a8c09b64/images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628/snap >> 6bcc4eab-ed35-42dc-88bd-1d45de73b628 >> ... >> ... >> DEBUG:keystoneauth.session:PATCH call to image for >> https://cloud-areapd.pd.infn.it:9292/v2/images/6bcc4eab-ed35-42dc-88bd-1d45… >> used request id req-6c0598cc-582c-4ce7-a14c-5d1bb6ec4f14 >> Request returned failure status 400. >> DEBUG:glanceclient.common.http:Request returned failure status 400. >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line >> 687, in main >> OpenStackImagesShell().main(argv) >> File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line >> 591, in main >> args.func(client, args) >> File "/usr/lib/python2.7/site-packages/glanceclient/v2/shell.py", line >> 749, in do_location_add >> image = gc.images.add_location(args.id, args.url, metadata) >> File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line >> 448, in add_location >> response = self._send_image_update_request(image_id, add_patch) >> File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", >> line 598, in inner >> return RequestIdProxy(wrapped(*args, **kwargs)) >> File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line >> 432, in _send_image_update_request >> data=json.dumps(patch_body)) >> File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line >> 340, in patch >> return self.request(url, 'PATCH', **kwargs) >> File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", >> line 377, in request >> return self._handle_response(resp) >> File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", >> line 126, in _handle_response >> raise exc.from_response(resp, resp.content) >> HTTPBadRequest: 400 Bad Request: Invalid location (HTTP 400) >> 400 Bad Request: Invalid location (HTTP 400) >> >> >> As far as I can see with the "openstack" client there is not something to >> add/delete a location. >> So I guess it is necessary to change the 'direct_url' and 'locations' >> properties. >> >> If I try to change the direct_url property: >> >> # openstack image set --property >> direct_url='rbd://8162f291-00b6-4b40-a8b4-1981a8c09b64/images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628/snap' >> 6bcc4eab-ed35-42dc-88bd-1d45de73b628 >> 403 Forbidden: Attribute 'direct_url' is read-only. (HTTP 403) >> >> Any hints ? >> Thanks, Massimo >> >> [*] >> >> [default] >> ... >> enabled_backends = file:file,http:http,rbd:rbd >> show_image_direct_url = true >> show_multiple_locations = true >> >> [glance_store] >> default_backend = rbd >> >> [file] >> filesystem_store_datadir = /var/lib/glance/images/ >> >> >> [rbd] >> rbd_store_chunk_size = 8 >> rbd_store_ceph_conf = /etc/ceph/ceph.conf >> rbd_store_user = glance-prod >> rbd_store_pool = images-prod >> >> -- > Thanks & Best Regards, > > Abhishek Kekane >

2 3

Zuul log location changing
by corvus＠inaugust.com 28 Aug '19

28 Aug '19

Hi, We've been working for some time[1] to retire our current static log server in favor of storing Zuul job logs in Swift. We're just about ready to do that. This means that not only do we get to use another great OpenStack project, but we also can stop worrying about fscking our 14TB log partition whenever it gets hit with a comsic ray. The change will happen in two phases, only the first of which should be readily apparent. Phase 1: On Monday August 12, 2019 we will change the URL that Zuul reports back to Gerrit. Instead of being a direct link to the log server, it will be a link to the Zuul Build page for the job. This is part of Zuul's web interface which has been around for a while, but isn't well known since we haven't linked to it yet. The build page shows a summary of information about the build, including output snippets for failed tasks. Next to the "Summary" tab, you'll find the "Logs" tab. This contains an expandable index of all the log files uploaded for the build. If they are text files, they can be rendered in-app with line-number hyperlinks and severity filtering. There are also direct links to the files on the log server. Links to preview sites (e.g., for docs builds) will show up in the "Artifacts" section. We also plan to further enhance the build page with additional features. Here are some links to sample build pages so you can see what it's like: https://zuul.opendev.org/t/openstack/build/a6e13a8098fc4a1fbff43d8f2c27ad29 https://zuul.opendev.org/t/openstack/build/75d1e8d4ffaf477db00520d7bfd77246 This step is necessary because our static log server implements a number of features as WSGI middleware in Apache. We have re-implemented the these on the Zuul build page, so there should be no loss in functionality (in fact, we think this is an improvement). Once in place, we can change the backend storage options without impacting the user interface. Phase 2: Shortly afterwards (depending on how phase 1 goes), we will configure jobs to upload logs to Swift instead of the static log server. At this point, there should be no user-visible change, since the main interface for interacting with logs is now the Zuul build page. However, you may notice that log urls have changed from our static log server to one of six different Swift instances. The Swift instance used to store the logs for any given build is chosen at random from among our providers, and is yet another really cool multi-cloud feature we get by using OpenStack. Thanks to our amazing providers and all of the folks who have helped with this effort over the years[1]. Please let us know if you have any questions or encounter any issues, either here, or in #openstack-infra on IRC. -Jim [1] Years. So. Many. Years.

5 7

[placement][plt][elections] Non-nomination for U
by Chris Dent 28 Aug '19

28 Aug '19

tl;dr: I'd prefer not to be Placement PTL for the U cycle. I've accomplished most of what I set out to do and I think we have other members of the team who would make excellent PTLs (notably tetsuro and gibi). Longer version: Getting placement standing on its own feet, with a mostly reasonable architecture, reasonable performance, and a fairly complete set of features has pretty much been my single focus since the start of 2016. From both a personal and professional standpoint I need to focus some energy elsewhere, lest the deep ruts in my brain become permanent. I think it is important to not be shy about the goal of that three and a half year process: Get placement out of nova and get nova out of placement. The architectural differences necessary to make placement work well, be maintainable, and easily deployable [1] we're difficult to do within nova (for many reasons), but once extracted took a matter of days. Nova, neutron, blazar, zun, watcher, and cyborg all use, or are in the process of starting to use, placement. Ironic is thinking about it. So: some success. But it's not all gravy, there are still some things left to do: While the fundamental concept of placement (presenting and keeping track of inventory of resources) is pretty straightforward, using it most effectively can be complicated, especially if nested resource providers are involved. There's a lot of work to be done to document how to best model data and queries. As the number of resources in the cloud grows it becomes increasingly important to "ask better questions"; knowing how to do that shouldn't require 3 years of experience. The "consumer types" feature is in flight. It will allow placement to be used to keep track of resource usage for some types of quota, which makes good sense: If we're already tracking "user X is using N VCPU" in placement it's wasteful tracking it somewhere else as well. Placement sharding, so that one placement can track multiple domains, is a feature that we've discussed in the last couple of years, thinking perhaps it could be useful to edge setups. Until the recent performance improvements it wouldn't have worked well, but now it might. However, until real users present themselves with a need for this feature, there's not much point moving forward with it. We should continue to do benchmarking and profiling. That work has done wonders to reveal unexpected areas for improvement. Being willing and able to analyse and adjust the system and the code, continuously, is key to ensuring the project does not become moribund or calcified. Thanks to everyone who has helped get placement to a healthy place. [1] For some examples: We've taken a common benchmark from 17s to .5s. It's now possible to run placement without a config file or without a manual database sync. Every module has a maintainability index of "A". Placement doesn't use eventlet (and is not accidentally infected by it) so it can benefit from whatever threading or multi-processing model whatever chosen WSGI server provides. Placement is a simple web app over a (not so simple) persistence and query layer. The web app and the associated database server(s) can be scaled independently, up and out. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent

3 2