- openstack-discuss - lists.openstack.org

queens heat db deadlock
by Ignazio Cassano 02 Jan '19

02 Jan '19

Hi All, I installed queens on centos 7. Heat seems to work fine with templates I wrote but when I create magnum cluster I often face with db deadlock in heat-engine log: 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource Traceback (most recent call last): 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 918, in _action_recorder 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource yield 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2035, in delete 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource *action_args) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 346, in wrapper 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource step = next(subtask) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 977, in action_handler_task 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource done = check(handler_data) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 587, in check_delete_complete 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource return self._check_status_complete(self.DELETE) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 454, in _check_status_complete 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource action=action) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource ResourceFailure: resources[0]: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') (Background on this error at: http://sqlalche.me/e/2j85) 2018-12-13 11:48:46.016 89597 ERROR heat.engine.resource 2018-12-13 11:48:46.030 89597 INFO heat.engine.stack [req-9a43184f-77d8-4fad-ab9e-2b9826c10b70 - admin - default default] Stack DELETE FAILED (swarm-clustergp27-ebhsalhb4bop-swarm_primary_master-tzgkh3ncmymw): Resource DELETE failed: resources[0]: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') (Background on this error at: http://sqlalche.me/e/2j85) 2018-12-13 11:48:46.844 89595 INFO heat.engine.resource [req-9a43184f-77d8-4fad-ab9e-2b9826c10b70 - admin - default default] DELETE: ResourceGroup "swarm_primary_master" [6984db3c-3ac1-4afc-901f-21b3e7f230a7] Stack "swarm-clustergp27-ebhsalhb4bop" [b43256fa-52e2-4613-ac15-63fd9340a8be] I read this issue was solved in heat version 10 but seems not. Regards Ignazio

5 14

Re: [heat] Bug : Heat cannot create Octavia Load Balancer
by Jeff Yang 02 Jan '19

02 Jan '19

Please confirm whether "auth_strategy" is set as ''keystone" in configure file. I remember that the value of "auth_strategy" is set as "noauth" in "/usr/share/octavia/octavia-dist.conf" default if you install octavia by rpm. If the value was set as "noauth", you must manually specify "project_id" for octavia. Jeff Yang <yjf1970231893(a)gmail.com> 于2019年1月2日周三下午12:26写道： > Please confirm whether "auth_strategy" is set as ''keystone" in > configure file. I remember that the value of "auth_strategy" is set as > "noauth" in "/usr/share/octavia/octavia-dist.conf" default if you install > octavia by rpm. If the value was set as "noauth", you must manually specify > "project_id" for octavia. > > Zane Bitter <zbitter(a)redhat.com> 于2019年1月2日周三上午11:23写道： > >> On 2/01/19 1:34 AM, Adam Harwell wrote: >> > I'm just on my phone over the holidays, but it kinda looks like the >> code >> > for this was just updated 12 days ago: >> > https://review.openstack.org/#/c/619577/ >> > >> > If you're using that new code, I imagine it's possible there could be a >> > bug that wasn't yet caught... If you're NOT using that code, maybe try >> > it and see if it helps? I'm guessing it's related one way or another. >> If >> > you come to the #openstack-lbaas channel once more people are around >> > (later this week?), we can probably take a look. >> >> That's only the example template (previously it was an example for >> LBaaSv2; now it's an example for Octavia); there's been no recent change >> to the code. >> >> > --Adam Harwell (rm_work) >> > >> > On Sun, Dec 30, 2018, 03:37 Zufar Dhiyaulhaq <zufardhiyaulhaq(a)gmail.com >> > <mailto:zufardhiyaulhaq@gmail.com>> wrote: >> > >> > I have try creating load balancer with Heat. but always get this >> error : >> > >> > Resource CREATE failed: OctaviaClientException: >> > resources.loadbalancer: Validation failure: Missing project ID in >> > request where one is required. (HTTP 400) (Request-ID: >> > req-b45208e1-a200-47f9-8aad-b130c4c12272) >> > >> > OctaviaClientException: resources.loadbalancer: Validation failure: >> > Missing project ID in request where one is required. (HTTP 400) >> > (Request-ID: req-b45208e1-a200-47f9-8aad-b130c4c12272) >> >> What version of OpenStack are you using? >> >> The issue is that Heat is sending a "tenant_id" but Octavia wants a >> "project_id", which is the new name for the same thing. (I think you >> likely modified that template after trying it but before uploading it, >> because there is no "project_id" property in Heat's >> OS::Octavia::LoadBalancer resource type.) >> >> This bug has been reported and there is a patch up for review in Heat: >> https://storyboard.openstack.org/#!/story/2004650 >> >> There was a change to Octavia in Pike >> (https://review.openstack.org/455442) to add backwards compatibility, >> but it was either incomplete or the problem reoccurred and was fixed >> again in Rocky (https://review.openstack.org/569881) My guess is that >> it's likely broken in Pike and Queens. >> >> I'd certainly have expected Heat's gate tests to pick up the problem, >> and it's a bit of a mystery why they didn't. Perhaps we're not >> exercising the case where a project_id is required (using it at all is >> an admin-only feature, so that's not too surprising I guess; it's >> actually more surprising that there's a case where it's _required_). >> >> cheers, >> Zane. >> >> > I create 2 openstack environment : >> > >> > * Heat with Octavia (Octavia Heat Template : >> > http://paste.opensuse.org/view//33592182 ) >> > * Heat with Neutron Lbaasv2 (Neutron LBaaSv2 Heat Template : >> > http://paste.opensuse.org/view//71741503) >> > >> > But always error when creating with octavia : >> > >> > * Octavia Log (https://imgur.com/a/EsuWvla) >> > * LBaaS v2 (https://imgur.com/a/BqNGRPH) >> > >> > Are Heat code is broken to create Octavia Load Balancer? >> > >> > Best Regards, >> > Zufar Dhiyaulhaq >> > >> >> >>

2 1

[heat] Bug : Heat cannot create Octavia Load Balancer
by Zufar Dhiyaulhaq 02 Jan '19

02 Jan '19

I have try creating load balancer with Heat. but always get this error : Resource CREATE failed: OctaviaClientException: resources.loadbalancer: Validation failure: Missing project ID in request where one is required. (HTTP 400) (Request-ID: req-b45208e1-a200-47f9-8aad-b130c4c12272) OctaviaClientException: resources.loadbalancer: Validation failure: Missing project ID in request where one is required. (HTTP 400) (Request-ID: req-b45208e1-a200-47f9-8aad-b130c4c12272) I create 2 openstack environment : - Heat with Octavia (Octavia Heat Template : http://paste.opensuse.org/view//33592182 ) - Heat with Neutron Lbaasv2 (Neutron LBaaSv2 Heat Template : http://paste.opensuse.org/view//71741503) But always error when creating with octavia : - Octavia Log (https://imgur.com/a/EsuWvla) - LBaaS v2 (https://imgur.com/a/BqNGRPH) Are Heat code is broken to create Octavia Load Balancer? Best Regards, Zufar Dhiyaulhaq

3 2

[Heat][Octavia] Is autoscaling feature missing?
by Viktor Shulhin 02 Jan '19

02 Jan '19

Hi all, I am trying to create Heat template with autoscaling and loadbalancing. I didn't find any similar Heat template examples. Individually, loadbalancing and autoscaling work well, but loadbalancing OS::Octavia::PoolMember can be added only manually. Is there any way to use OS::Heat::AutoScalingGroup as server pool for loadbalancing? Regards, Viktor.

3 2

[Cyborg] no irc meeting this week
by Li Liu 01 Jan '19

01 Jan '19

Hi Team, Since it's public holiday for folks in US and Canada, we will not have the irc meeting this week. Enjoy the new year guys :) Thank you Regards Li Liu

1 0

[dev][karbor]No meeting today
by 焦鹏举 01 Jan '19

01 Jan '19

Hi Karbor Team, We will skip karbor weekly meeting today due to holidays. Next meeting will come on 15 January. Thanks. Pengju Jiao

1 0

[chef] State of the Kitchen: 8th Edition
by Samuel Cassiba 01 Jan '19

01 Jan '19

It has been some time since the last State of the Kitchen. Since the release of Chef OpenStack 17 (Queens), we have been operating in a minimal churn mode to give people time to test/upgrade deployments and handle any regressions that emerge. There are three main areas that are still in progress upstream from Chef OpenStack, but can affect its cadence regardless. As a result, this update focuses more on those areas. Consider this more of a year-end review. ### Important Happenings * *fog-openstack*[^1] Beginning in August, we started receiving reports of breakage due to changes in fog-core. As a reactionary measure, we implemented upper-level constraints in the client resource cookbook to maintain a consistent outcome. The fog-openstack library has continued to receive changes to further align with fog-core, and we are following its progress to find a good time to move ChefDK and Chef OpenStack to a post-1.0 release of fog-openstack. We are targeting the 18th release of Chef OpenStack, due to Keystone endpoint changes that need to happen. * *Sous Chefs*[^2] One of the biggest strengths of Chef and OpenStack is the collective outcome of their unique communities. Within the Chef ecosystem, the Sous Chefs group was formed in response to a need for the continued existence of Chef components, libraries, and utilities that need a long-term home. Across the globe, Sous Chefs work to keep some of the most heavily used cookbooks in existence, such as [apache2](https://supermarket.chef.io/cookbooks/apache2), [mysql](https://supermarket.chef.io/cookbooks/mysql) (and [mariadb](https://supermarket.chef.io/cookbooks/mariadb)!), [postgresql](https://supermarket.chef.io/cookbooks/postgresql), as well as [redisio](https://supermarket.chef.io/cookbooks/redisio), and many more. Chef OpenStack depends on MariaDB, Apache, and their related cookbooks, for compatibility without operators needing to plumb those resources internally. * *poise-python*[^3] In early October, pip 18.1 was released, which made some additional waves in the ecosystem. Workarounds were devised and implemented to limit the fallout. Currently, the fix has been merged to poise-python's master, but cannot be released safely due to CI changes in the current workflow. There are limitations on what the Sous Chefs can reasonably maintain. The maintenance of poise is rather beyond that boundary, not to discount or disparage anyone involved. Anyone interested with spare cycles over the holiday season might consider joining the conversation. ### Upcoming Changes * In Chef OpenStack 18... - The MariaDB version will default to 10.3, consistent with the default in the 2.0 version of the cookbook. Please plan accordingly. - Keystone's endpoint will be changing to drop the hardcoded API version - the cloud primitives (client) cookbook is in the process of migrating from cookbook-openstackclient to cookbook-openstack-client (named openstack_client, to conform with current best practices in the Chef community) - Ubuntu will be upgraded from 16.04 to 18.04, and as such we will be gating against Bionic at that time. Plainly put, previous Chef OpenStack releases will not be moving to Bionic jobs, and will continue to work at best effort until they succumb to the detritus of time. ### Meetings Since the Summit, a few people have reached out through various means about Chef OpenStack and how to work together to improve the outcome. As a result, I would like to propose holding regular meetings for Chef OpenStack once more set aside a dedicated period where we can come together and talk about food, or other things. We have the IRC channel, but IRC has proven less effective for a small group to dedicate time consistently, so I would something more high bandwidth for technical conversations, such as video with a publicized method for joining and viewing. I will follow up with a more expanded proposal outside this update. ### On The Menu This would not be a State of the Kitchen without something to eat. My partner and I try to cook with recipes that are not overly complicated, but can be infinitely complex with just the right nudge. Sometimes we incorporate our own opinions into someone else's recipe to make it our own thing, and sometimes they're great just as they come. *Dat Dough, Doe* * 170g / 6oz grated mozzarella or Edam, or another mild cheese with similar melting consistency * 85g / 3oz almond meal/flour * 28g / 2 tbsp cream cheese or Neufchatel * 1 egg * pinch of salt to taste 1. Mix the shredded/grated cheese and the almond meal in a microwaveable bowl, then add the cream cheese. Microwave on high for 1 minute. Stir the mixture, then microwave on high for another 30 seconds. 2. Add egg, salt, additional spices or flavorings, and mix or fold gently. 3. Shape using parchment paper into the desired outcome, be it flat like a disc or rounded, like a boule. 4. Create vents to ensure that the finished product cooks evenly. 5. Fry, bake, broil or grill as desired. Lipids can be friends here. More commonly known as the "Fat Head" dough, out of these few ingredients, one can make food that can taste every bit like pizza, pasta, bread, even pão de queijo. Or, perhaps, cinnamon rolls, or danishes, as one might consider making. With these basic suggestions, one can apply their own opinions and set of requirements to create complex pieces of work, which can taste every bit like an artform and a science. See you in 2019! Your humble pastry chef, -scas [^1]: https://github.com/fog/fog-openstack/issues/434 [^2]: https://sous-chefs.org/ [^3]: https://github.com/poise/poise-python/issues/133

1 0

[Mistral]
by Deep C 31 Dec '18

31 Dec '18

Hi Team, Could anyone please help me fixing the below issue - Mistral server throws error when i execute workflows - 2018-12-31 00:49:40.025 59372 ERROR mistral.notifiers.default_notifier Exception: [9ad86428-59ca-4823-87ad-61915f61410d] Unable to publish event because st2 returned status code 500. { 2018-12-31 00:49:40.025 59372 ERROR mistral.notifiers.default_notifier "faultstring": "Internal Server Error" 2018-12-31 00:49:40.025 59372 ERROR mistral.notifiers.default_notifier } 2018-12-31 00:49:40.025 59372 ERROR mistral.notifiers.default_notifier Bug also raised for more details - https://bugs.launchpad.net/mistral/+bug/1808668 Please suggest the solution .

1 0

openstack magnum on queens: please help me
by Ignazio Cassano 30 Dec '18

30 Dec '18

Hello Everybody, I created a kubernetes cluster with mugnum under queens. After that I ran the folloquing commands: mkdir -p /root/clusters/kubecertslb/config export KUBECONFIG=/root/clusters/kubecertslb/config $(openstack coe cluster config kubecertslb --dir ~/clusters/kubecertslb) Then the command "kubectl -n kube-system get po" reports: NAME READY STATUS RESTARTS AGE coredns-5864cfd79d-76889 1/1 Running 0 18h heapster-68b976dd7-7vdlq 1/1 Running 0 18h kubernetes-dashboard-846b8b6844-cjp2f 1/1 Running 0 18h the command "kubectl cluster-info" reports: Kubernetes master is running at https://10.102.186.126:6443 Heapster is running at https://10.102.186.126:6443/api/v1/namespaces/kube-system/services/heapster… CoreDNS is running at https://10.102.186.126:6443/api/v1/namespaces/kube-system/services/kube-dns… To further debug and diagnose cluster problems, use 'kubectl cluster-info dump' I deployed nginx: kubectl run nginx --image=nginx --replicas=2 --port=80 Then the command "kubectl get po" reports: NAME READY STATUS RESTARTS AGE nginx-7587c6fdb6-chnw6 1/1 Running 0 18h nginx-7587c6fdb6-m7fr2 1/1 Running 0 18h Exposing nginx service with type NodePort works fine. Exposing with type LoadBalancer does not work: kubectl expose deployment nginx --target-port=80 --type=LoadBalancer kubectl get svc: NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 19h nginx LoadBalancer 10.254.97.40 <pending> 80:31568/TCP 165m No load balancer is created. Im not using octavia but standard lbaas v2. Lbaas is working fine because I tried a configuration between two ssh vm. I read somting about the magnum.conf trust section: someone suggested to enable "cluster_user_trust = True" I tried it but with this parameter the kubernetes cluster does not finish: the heat template remains in pending ang goes in timeout during the master creation. I read also something aboud kubernetes cloud provider must be "external" or "openstack" but I think it should the automatically configured by heat. Any help, please ? Thanks and Regards Ignazio

1 0

[Openstack-discuss][Horizon] Domain input in the login page disappeared
by Minjun Hong 30 Dec '18

30 Dec '18

Hi. I'm replacing existing controller node with a new server. After the setup step, I found something weird. The domain input in the horizon login page disappeared as following link https://imgur.com/Vuj7uNT I'm not sure that is normal situation but, horizon login page in the previous system always consisted of Domain, User Name and Password. So, is there anyone who has experienced this problem? Thanks. Regards

1 0