<div dir="ltr">Hi Swogat,<div>Thanks once again for your input it really helped much. </div><div><br></div><div>instead of running mentioned those three provisioning steps i used alternate method and passed directly in deploy command. now my current deploy command is:</div><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div>openstack overcloud deploy --templates \<br>--networks-file /home/stack/templates/custom_network_data.yaml \<br> --vip-file /home/stack/templates/custom_vip_data.yaml \<br> --baremetal-deployment /home/stack/templates/overcloud-baremetal-deploy.yaml \<br> --network-config \<br> -e /home/stack/templates/environment.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-conductor.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-overcloud.yaml \<br> -e /home/stack/templates/ironic-config.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \<br> -e /home/stack/containers-prepare-parameter.yaml<br></div></blockquote>The files as suggested by you are well created.</div><div><br></div><div>But once we run the deployment I get this error for all nodes:</div><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div> 0:00:16.064781 | 1.16s<br>2022-05-31 19:13:00.276954 | 525400ef-b928-9ded-fecc-000000000094 | TASK | Run tripleo_os_net_config_module with network_config<br>2022-05-31 19:40:30.061582 | 525400ef-b928-9ded-fecc-000000000094 | FATAL | Run tripleo_os_net_config_module with network_config | overcloud-controller-1 | error={"msg": "Data could not be sent to remote host \"30.30.30.117\". Make sure this host can be reached over ssh: ssh: connect to host 30.30.30.117 port 22: No route to host\r\n"}<br>2022-</div></blockquote><br></div><div>Baremetal node list are showing in as active.</div><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div>(undercloud) [stack@undercloud ~]$ openstack baremetal node list<br>/usr/lib64/python3.6/site-packages/_yaml/__init__.py:23: DeprecationWarning: The _yaml extension module is now located at yaml._yaml and its location is subject to change. To use the LibYAML-based parser and emitter, import from `yaml`: `from yaml import CLoader as Loader, CDumper as Dumper`.<br> DeprecationWarning<br>+--------------------------------------+--------+--------------------------------------+-------------+--------------------+-------------+<br>| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |<br>+--------------------------------------+--------+--------------------------------------+-------------+--------------------+-------------+<br>| 1a4d873c-f9f7-4504-a3af-92c11f954171 | node-a | 901453a1-183f-4de8-aaab-0f38be2be455 | power on | active | False |<br>| d18610fc-9532-410c-918e-8efc326c89f8 | node-b | d059b94a-8357-4f8e-a0d8-15a24b0c1afe | power on | active | False |<br>| b69f2d5a-5b18-4453-8843-15c6af79aca0 | node-c | f196ef3a-7950-47b9-a5ae-751f06b18f75 | power on | active | False |<br>| 8a38c584-f812-4ebc-a0b1-4299f0917637 | node-d | 1636517c-2ab2-43d7-8205-9f02c5290207 | power on | active | False |<br>+--------------------------------------+--------+--------------------------------------+-------------+--------------------+-------------+<br></div></blockquote></div><div>Some config is missing it seems, please check once and advise. </div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 31, 2022 at 11:40 AM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com">swogatpradhan22@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Lokendra,<div>You need to generate another file also in the following step: <span style="color:rgb(80,0,80)">openstack overcloud node provision --stack overcloud --overcloud-ssh-key /home/stack/sshkey/id_rsa overcloud-baremetal-deploy.</span><span style="color:rgb(80,0,80)">yaml also you need to pass another parameter --network-config.</span></div><div><span style="color:rgb(80,0,80)">example:</span></div><div>openstack overcloud node provision --stack overcloud --overcloud-ssh-key /home/stack/sshkey/id_rsa <b>--network-config</b> <b>--output overcloud-baremetal-deployed.yaml</b> overcloud-baremetal-deploy.yaml<span style="color:rgb(80,0,80)"><br></span></div><div><br></div><div>And then all these output files will be passed on to the openstack overcloud deploy command.</div><div>NOTE: when passing --network-config parameter in node provision step, it creates a directory in /etc/os-net-config and in it creates a file config.yaml, do check the indentation of that file. (in my case the indentation was wrong when i was using bondind everytime, so i had to manually change the script and run a while loop and then my node provision step was successful)</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 31, 2022 at 8:59 AM Lokendra Rathour <<a href="mailto:lokendrarathour@gmail.com" target="_blank">lokendrarathour@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Hi Swogat,<div>I tried generating the scripts as used by you in your deployments using the</div><div><br></div><div><div><br>#openstack overcloud network provision --stack overcloud --output networks-deployed-environment.yaml custom_network_data.yaml</div><div># openstack overcloud network vip provision --stack overcloud --output vip-deployed-environment.yaml custom_vip_data.yaml</div><div># openstack overcloud node provision --stack overcloud --overcloud-ssh-key /home/stack/sshkey/id_rsa overcloud-baremetal-deploy.yaml</div><div><br></div></div><div>and used the first two in the final deployment script, but it gives the error:</div><div><br></div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div> heatclient.exc.HTTPInternalServerError: ERROR: Internal Error<br>2022-05-30 14:14:39.772 479668 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):\n', ' File "/usr/lib/python3.6/ted_stack\n nested_stack.validate()\n', ' File "/usr/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper\n result = f(*args, ine 969, in validate\n result = res.validate()\n', ' File "/usr/lib/python3.6/site-packages/heat/engine/resources/openstack/neutron/port.py", line 454site-packages/heat/engine/resources/openstack/neutron/neutron.py", line 43, in validate\n res = super(NeutronResource, self).validate()\n', ' File "/un return self.validate_template()\n', ' File "/usr/lib/python3.6/site-packages/heat/engine/resource.py", line 1882, in validate_template\n self.t.rpy", line 200, in _validate_service_availability\n raise ex\n', 'heat.common.exception.ResourceTypeUnavailable: HEAT-E99001 Service neutron is not avaieutron network endpoint is not in service catalog.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recens/stack_resource.py", line 75, in validate_nested_stack\n nested_stack.validate()\n', ' File "/usr/lib/python3.6/site-packages/osprofiler/profiler.py"thon3.6/site-packages/heat/engine/stack.py", line 969, in validate\n result = res.validate()\n', ' File "/usr/lib/python3.6/site-packages/heat/engine/ateResource, self).validate()\n', ' File "/usr/lib/python3.6/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate\n self.validources/stack_resource.py", line 81, in validate_nested_stack\n ex, path=[self.stack.t.RESOURCES, path])\n', 'heat.common.exception.StackValidationFaileeploy/overcloud/tripleo-heat-templates/deployed-server/deployed-server.yaml>: HEAT-E99001 Service neutron is not available for resource type OS::TripleO::vice catalog.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/usr/lib/pline 320, in validate_nested_stack\n nested_stack.validate()\n', ' File "/usr/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrappe/heat/engine/stack.py", line 969, in validate\n result = res.validate()\n', ' File "/usr/lib/python3.6/site-packages/heat/engine/resources/template_relidate()\n', ' File "/usr/lib/python3.6/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate\n self.validate_nested_stack()\n'.py", line 81, in validate_nested_stack\n ex, path=[self.stack.t.RESOURCES, path])\n', 'heat.common.exception.StackValidationFailed: ResourceTypeUnavaimplates/puppet/compute-role.yaml>.resources.NovaCompute<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployed-server/deployed-servpleO::DeployedServer::ControlPlanePort, reason: neutron network endpoint is not in service catalog.\n', '\nDuring handling of the above exception, another/lib/python3.6/site-packages/heat/common/context.py", line 416, in wrapped\n return func(self, ctx, *args, **kwargs)\n', ' File "/usr/lib/python3.6/sirce_name, template_id)\n', ' File "/usr/lib/python3.6/site-packages/heat/engine/service.py", line 756, in _parse_template_and_validate_stack\n stack.v line 160, in wrapper\n result = f(*args, **kwargs)\n', ' File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 969, in validate\n resesources/stack_resource.py", line 65, in validate\n self.validate_nested_stack()\n', ' File "/usr/lib/python3.6/site-packages/heat/engine/resources/oph=[self.stack.t.RESOURCES, path])\n', 'heat.common.exception.StackValidationFailed: ResourceTypeUnavailable: resources.Compute<nested_stack>.resources.0<fte-role.yaml>.resources.NovaCompute<file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/deployed-server/deployed-server.yaml>: HEAT-E9900::ControlPlanePort, reason: neutron network endpoint is not in service catalog.\n'].</div><div><br></div></blockquote>Request you to check once, please. </div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 30, 2022 at 11:06 AM Lokendra Rathour <<a href="mailto:lokendrarathour@gmail.com" target="_blank">lokendrarathour@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Swogat,</div><div>Thanks once again. </div><div><br></div><div>with the files as shown below I am running the overcloud deploy for wallaby using this command:</div><div><br></div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>(undercloud) [stack@undercloud ~]$ cat deploy_overcloud_working_1.sh<br>openstack overcloud deploy --templates \<br> -n /home/stack/templates/network_data.yaml \<br> -r /home/stack/templates/roles_data.yaml \<br> -e /home/stack/templates/environment.yaml \<br> -e /home/stack/templates/environments/network-isolation.yaml \<br> -e /home/stack/templates/environments/network-environment.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-conductor.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-overcloud.yaml \<br> -e /home/stack/templates/ironic-config.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \<br> -e /home/stack/containers-prepare-parameter.yaml<br>(undercloud) [stack@undercloud ~]$</div></blockquote></div><div><br></div><div>This deployment is on ipv6 using triple0 wallaby, templates, as mentioned below, are generated using rendering steps and the network_data.yaml the roles_data.yaml</div><div>Steps used to render the templates:</div><div> cd /usr/share/openstack-tripleo-heat-templates/<br>./tools/process-templates.py -o ~/openstack-tripleo-heat-templates-rendered_at_wallaby -n /home/stack/templates/network_data.yaml -r /home/stack/templates/roles_data.yaml<br></div><div><b><br></b></div><div><b>Now if i try adding the related to VIP port I do get the error as:</b></div><div><br></div><div>2022-05-30 10:37:12.792 979387 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template to file: /home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/puppet/controller-role.yaml<br>2022-05-30 10:37:12.792 979387 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template to file: /home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/puppet/compute-role.yaml<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: ValueError: The environment is not a valid YAML mapping data type.<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args)<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args)<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1189, in take_action<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud stack, parsed_args, new_tht_root, user_tht_root)<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 227, in create_env_files<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud created_env_files, parsed_args, new_tht_root, user_tht_root)<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 204, in build_image_params<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud cleanup=(not parsed_args.no_cleanup))<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 1929, in process_multiple_environments<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud env_path=env_path, include_env_in_files=include_env_in_files)<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/heatclient/common/template_utils.py", line 326, in process_environment_and_files<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud env = environment_format.parse(raw_env)<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/heatclient/common/environment_format.py", line 50, in parse<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise ValueError(_('The environment is not a valid '<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud ValueError: The environment is not a valid YAML mapping data type.<br>2022-05-30 10:37:14.455 979387 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud<br>2022-05-30 10:37:14.457 979387 ERROR openstack [-] The environment is not a valid YAML mapping data type.<br>2022-05-30 10:37:14.457 979387 INFO osc_lib.shell [-] END return value: 1<br>(undercloud) [stack@undercloud ~]$<br></div><div><br></div><div>This is more of a syntax error where it is not able to understand the passed VIP data file:</div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>undercloud) [stack@undercloud ~]$ cat /home/stack/templates/vip-data-default-network-isolation.yaml<br>-<br> dns_name: overcloud<br> network: internal_api<br>-<br> dns_name: overcloud<br> network: external<br>-<br> dns_name: overcloud<br> network: ctlplane<br>-<br> dns_name: overcloud<br> network: oc_provisioning<br>-<br> dns_name: overcloud<br> network: j3mgmt<br></div></blockquote><br></div><div>Please advise, also please note that similar templates generated in prior releases such as train/ussuri works perfectly.</div><div><br></div><div><br></div><div><br></div><div>Please check the list of <b>templates </b>files:</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>drwxr-xr-x. 2 stack stack 68 May 30 09:22 environments</div><div>-rw-r--r--. 1 stack stack 265 May 27 13:47 environment.yaml</div><div>-rw-rw-r--. 1 stack stack 297 May 27 13:47 init-repo.yaml</div><div>-rw-r--r--. 1 stack stack 570 May 27 13:47 ironic-config.yaml</div><div>drwxrwxr-x. 4 stack stack 4096 May 27 13:53 network</div><div>-rw-r--r--. 1 stack stack 6370 May 27 14:26 network_data.yaml</div><div>-rw-r--r--. 1 stack stack 11137 May 27 13:53 roles_data.yaml</div><div>-rw-r--r--. 1 stack stack 234 May 30 09:23 vip-data-default-network-isolation.yaml</div></blockquote><div><br></div><div><br></div><div>(undercloud) [stack@undercloud templates]$ cat environment.yaml<br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>parameter_defaults:</div><div> OvercloudControllerFlavor: control</div><div> OvercloudComputeFlavor: compute</div><div> ControllerCount: 3</div><div> ComputeCount: 1</div><div> TimeZone: 'Asia/Kolkata'</div><div> NtpServer: ['30.30.30.3']</div><div> NeutronBridgeMappings: datacentre:br-tenant</div><div> NeutronFlatNetworks: datacentre</div><div>(undercloud) [stack@undercloud templates]$</div></blockquote><div><br></div><div><br></div><div>(undercloud) [stack@undercloud templates]$ cat ironic-config.yaml<br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>parameter_defaults:</div><div> NovaSchedulerDefaultFilters:</div><div> - AggregateInstanceExtraSpecsFilter</div><div> - AvailabilityZoneFilter</div><div> - ComputeFilter</div><div> - ComputeCapabilitiesFilter</div><div> - ImagePropertiesFilter</div><div> IronicEnabledHardwareTypes:</div><div> - ipmi</div><div> - redfish</div><div> IronicEnabledPowerInterfaces:</div><div> - ipmitool</div><div> - redfish</div><div> IronicEnabledManagementInterfaces:</div><div> - ipmitool</div><div> - redfish</div><div> IronicCleaningDiskErase: metadata</div><div> IronicIPXEEnabled: true</div><div> IronicInspectorSubnets:</div><div> - ip_range: 172.23.3.100,172.23.3.150</div></blockquote><div>(undercloud) [stack@undercloud templates]$ cat network_data.yaml<br></div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>- name: J3Mgmt</div><div> name_lower: j3mgmt</div><div> vip: true</div><div> vlan: 400</div><div> ipv6: true</div><div> ipv6_subnet: 'fd80:fd00:fd00:4000::/64'</div><div> ipv6_allocation_pools: [{'start': 'fd80:fd00:fd00:4000::10', 'end': 'fd80:fd00:fd00:4000:ffff:ffff:ffff:fffe'}]</div><div> mtu: 9000</div></blockquote><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><br></div><div><br></div><div>- name: InternalApi</div><div> name_lower: internal_api</div><div> vip: true</div><div> vlan: 418</div><div> ipv6: true</div><div> ipv6_subnet: 'fd00:fd00:fd00:2000::/64'</div><div> ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2000::10', 'end': 'fd00:fd00:fd00:2000:ffff:ffff:ffff:fffe'}]</div><div> mtu: 9000</div></blockquote></div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>- name: External</div><div> vip: true</div><div> name_lower: external</div><div> vlan: 408</div><div> ipv6: true</div><div> gateway_ipv6: 'fd00:fd00:fd00:9900::1'</div><div> ipv6_subnet: 'fd00:fd00:fd00:9900::/64'</div><div> ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:9900::10', 'end': 'fd00:fd00:fd00:9900:ffff:ffff:ffff:fffe'}]</div><div> mtu: 9000</div></blockquote><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>- name: OCProvisioning</div><div> vip: true</div><div> name_lower: oc_provisioning</div><div> vlan: 412</div><div> ip_subnet: '<a href="http://172.23.3.0/24" target="_blank">172.23.3.0/24</a>'</div><div> allocation_pools: [{'start': '172.23.3.10', 'end': '172.23.3.50'}]</div><div> mtu: 9000</div></blockquote><div><br></div><div><br></div><div><br></div><div>(undercloud) [stack@undercloud templates]$ cat roles_data.yaml<br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>###############################################################################</div><div># File generated by TripleO</div><div>###############################################################################</div><div>###############################################################################</div><div># Role: Controller #</div><div>###############################################################################</div><div>- name: Controller</div><div> description: |</div><div> Controller role that has all the controller services loaded and handles</div><div> Database, Messaging, and Network functions.</div><div> CountDefault: 1</div><div> tags:</div><div> - primary</div><div> - controller</div><div> # Create external Neutron bridge for SNAT (and floating IPs when using</div><div> # ML2/OVS without DVR)</div><div> - external_bridge</div><div> networks:</div><div> External:</div><div> subnet: external_subnet</div><div> InternalApi:</div><div> subnet: internal_api_subnet</div><div> OCProvisioning:</div><div> subnet: oc_provisioning_subnet</div><div> J3Mgmt:</div><div> subnet: j3mgmt_subnet</div></blockquote><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div> # For systems with both IPv4 and IPv6, you may specify a gateway network for</div><div> # each, such as ['ControlPlane', 'External']</div><div> default_route_networks: ['External']</div><div> HostnameFormatDefault: '%stackname%-controller-%index%'</div><div> RoleParametersDefault:</div><div> OVNCMSOptions: "enable-chassis-as-gw"</div><div> # Deprecated & backward-compatible values (FIXME: Make parameters consistent)</div><div> # Set uses_deprecated_params to True if any deprecated params are used.</div><div> uses_deprecated_params: True</div><div> deprecated_param_extraconfig: 'controllerExtraConfig'</div><div> deprecated_param_flavor: 'OvercloudControlFlavor'</div><div> deprecated_param_image: 'controllerImage'</div><div> deprecated_nic_config_name: 'controller.yaml'</div><div> update_serial: 1</div><div> ServicesDefault:</div><div> - OS::TripleO::Services::Aide</div><div> - OS::TripleO::Services::AodhApi</div><div> - OS::TripleO::Services::AodhEvaluator</div><div><br></div><div>..</div><div>.</div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>..###############################################################################</div># Role: Compute #<br>###############################################################################<br>- name: Compute<br> description: |<br> Basic Compute Node role<br> CountDefault: 1<br> # Create external Neutron bridge (unset if using ML2/OVS without DVR)<br> tags:<br> - compute<br> - external_bridge<br> networks:<br> InternalApi:<br> subnet: internal_api_subnet<br> J3Mgmt:<br> subnet: j3mgmt_subnet<br> HostnameFormatDefault: '%stackname%-novacompute-%index%'<br> RoleParametersDefault:<br> FsAioMaxNumber: 1048576<br> TunedProfileName: "virtual-host"<br> # Deprecated & backward-compatible values (FIXME: Make parameters consistent)<br> # Set uses_deprecated_params to True if any deprecated params are used.<br> # These deprecated_params only need to be used for existing roles and not for<br> # composable roles.<br> uses_deprecated_params: True<br> deprecated_param_image: 'NovaImage'<br> deprecated_param_extraconfig: 'NovaComputeExtraConfig'<br> deprecated_param_metadata: 'NovaComputeServerMetadata'<br> deprecated_param_scheduler_hints: 'NovaComputeSchedulerHints'<br> deprecated_param_ips: 'NovaComputeIPs'<br> deprecated_server_resource_name: 'NovaCompute'</blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"> deprecated_nic_config_name: 'compute.yaml'<br> update_serial: 25<br> ServicesDefault:<br> - OS::TripleO::Services::Aide<br> - OS::TripleO::Services::AuditD<br> - OS::TripleO::Services::BootParams</blockquote><div><br></div><div>(undercloud) [stack@undercloud templates]$ cat environments/network-environment.yaml<br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>#This file is an example of an environment file for defining the isolated</div><div>#networks and related parameters.</div><div>resource_registry:</div><div> # Network Interface templates to use (these files must exist). You can</div><div> # override these by including one of the net-*.yaml environment files,</div><div> # such as net-bond-with-vlans.yaml, or modifying the list here.</div><div> # Port assignments for the Controller</div><div> OS::TripleO::Controller::Net::SoftwareConfig: OS::Heat::None</div><div> # Port assignments for the Compute</div><div> OS::TripleO::Compute::Net::SoftwareConfig: OS::Heat::None</div></blockquote><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>parameter_defaults:</div><div> # This section is where deployment-specific configuration is done</div><div> #</div><div> ServiceNetMap:</div><div> IronicApiNetwork: oc_provisioning</div><div> IronicNetwork: oc_provisioning</div></blockquote><div><br><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div> # This section is where deployment-specific configuration is done</div><div> ControllerNetworkConfigTemplate: 'templates/bonds_vlans/bonds_vlans.j2'</div><div> ComputeNetworkConfigTemplate: 'templates/bonds_vlans/bonds_vlans.j2'</div></blockquote><div><br><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div> # Customize the IP subnet to match the local environment</div><div> J3MgmtNetCidr: 'fd80:fd00:fd00:4000::/64'</div><div> # Customize the IP range to use for static IPs and VIPs</div><div> J3MgmtAllocationPools: [{'start': 'fd80:fd00:fd00:4000::10', 'end': 'fd80:fd00:fd00:4000:ffff:ffff:ffff:fffe'}]</div><div> # Customize the VLAN ID to match the local environment</div><div> J3MgmtNetworkVlanID: 400</div></blockquote><div><br><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div> # Customize the IP subnet to match the local environment</div><div> InternalApiNetCidr: 'fd00:fd00:fd00:2000::/64'</div><div> # Customize the IP range to use for static IPs and VIPs</div><div> InternalApiAllocationPools: [{'start': 'fd00:fd00:fd00:2000::10', 'end': 'fd00:fd00:fd00:2000:ffff:ffff:ffff:fffe'}]</div><div> # Customize the VLAN ID to match the local environment</div><div> InternalApiNetworkVlanID: 418</div></blockquote><div><br><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div> # Customize the IP subnet to match the local environment</div><div> ExternalNetCidr: 'fd00:fd00:fd00:9900::/64'</div><div> # Customize the IP range to use for static IPs and VIPs</div><div> # Leave room if the external network is also used for floating IPs</div><div> ExternalAllocationPools: [{'start': 'fd00:fd00:fd00:9900::10', 'end': 'fd00:fd00:fd00:9900:ffff:ffff:ffff:fffe'}]</div><div> # Gateway router for routable networks</div><div> ExternalInterfaceDefaultRoute: 'fd00:fd00:fd00:9900::1'</div><div> # Customize the VLAN ID to match the local environment</div><div> ExternalNetworkVlanID: 408</div></blockquote><div><br><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div> # Customize the IP subnet to match the local environment</div><div> OCProvisioningNetCidr: '<a href="http://172.23.3.0/24" target="_blank">172.23.3.0/24</a>'</div><div> # Customize the IP range to use for static IPs and VIPs</div><div> OCProvisioningAllocationPools: [{'start': '172.23.3.10', 'end': '172.23.3.50'}]</div><div> # Customize the VLAN ID to match the local environment</div><div> OCProvisioningNetworkVlanID: 412</div></blockquote><div><br><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div> # List of Neutron network types for tenant networks (will be used in order)</div><div> NeutronNetworkType: 'geneve,vlan'</div><div> # Neutron VLAN ranges per network, for example 'datacentre:1:499,tenant:500:1000':</div><div> NeutronNetworkVLANRanges: 'datacentre:1:1000'</div><div> # Customize bonding options, e.g. "mode=4 lacp_rate=1 updelay=1000 miimon=100"</div><div> # for Linux bonds w/LACP, or "bond_mode=active-backup" for OVS active/backup.</div><div> BondInterfaceOvsOptions: "bond_mode=active-backup"</div></blockquote><div>(undercloud) [stack@undercloud templates]$<br></div><div><br></div><div><br></div><div>(undercloud) [stack@undercloud templates]$ cat environments/network-isolation.yaml<br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div># NOTE: This template is now deprecated, and is only included for compatibility</div><div># when upgrading a deployment where this template was originally used. For new</div><div># deployments, set "ipv6: true" on desired networks in network_data.yaml, and</div><div># include network-isolation.yaml.</div><div>#</div><div># Enable the creation of Neutron networks for isolated Overcloud</div><div># traffic and configure each role to assign ports (related</div><div># to that role) on these networks.</div><div>resource_registry:</div><div> # networks as defined in network_data.yaml</div><div> OS::TripleO::Network::J3Mgmt: ../network/j3mgmt_v6.yaml</div><div> OS::TripleO::Network::InternalApi: ../network/internal_api_v6.yaml</div><div> OS::TripleO::Network::External: ../network/external_v6.yaml</div><div> OS::TripleO::Network::OCProvisioning: ../network/oc_provisioning.yaml</div></blockquote><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div> # Port assignments for the VIPs</div><div> OS::TripleO::Network::Ports::J3MgmtVipPort: ../network/ports/j3mgmt_v6.yaml</div><div> OS::TripleO::Network::Ports::InternalApiVipPort: ../network/ports/internal_api_v6.yaml</div><div> OS::TripleO::Network::Ports::ExternalVipPort: ../network/ports/external_v6.yaml</div><div> OS::TripleO::Network::Ports::OCProvisioningVipPort: ../network/ports/oc_provisioning.yaml</div></blockquote><div><br><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><br></div><div> # Port assignments by role, edit role definition to assign networks to roles.</div><div> # Port assignments for the Controller</div><div> OS::TripleO::Controller::Ports::J3MgmtPort: ../network/ports/j3mgmt_v6.yaml</div><div> OS::TripleO::Controller::Ports::InternalApiPort: ../network/ports/internal_api_v6.yaml</div><div> OS::TripleO::Controller::Ports::ExternalPort: ../network/ports/external_v6.yaml</div><div> OS::TripleO::Controller::Ports::OCProvisioningPort: ../network/ports/oc_provisioning.yaml</div><div> # Port assignments for the Compute</div><div> OS::TripleO::Compute::Ports::J3MgmtPort: ../network/ports/j3mgmt_v6.yaml</div><div> OS::TripleO::Compute::Ports::InternalApiPort: ../network/ports/internal_api_v6.yaml</div></blockquote></div><div><br><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>parameter_defaults:</div><div> # Enable IPv6 environment for Manila</div><div> ManilaIPv6: True</div></blockquote><div>(undercloud) [stack@undercloud templates]$<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 24, 2022 at 5:04 PM Lokendra Rathour <<a href="mailto:lokendrarathour@gmail.com" target="_blank">lokendrarathour@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks, I'll check them out. <br><div>will let you know in case it works out.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 24, 2022 at 2:37 PM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div>Please find the below templates:</div><div>These are for openstack wallaby release:</div><div><br></div><div>(undercloud) [stack@hkg2director workplace]$ cat custom_network_data.yaml<br>- name: Storage<br> name_lower: storage<br> vip: true<br> mtu: 1500<br> subnets:<br> storage_subnet:<br> ip_subnet: <a href="http://172.25.202.0/26" target="_blank">172.25.202.0/26</a><br> allocation_pools:<br> - start: 172.25.202.6<br> end: 172.25.202.20<br> vlan: 1105<br>- name: StorageMgmt<br> name_lower: storage_mgmt<br> vip: true<br> mtu: 1500<br> subnets:<br> storage_mgmt_subnet:<br> ip_subnet: <a href="http://172.25.202.64/26" target="_blank">172.25.202.64/26</a><br> allocation_pools:<br> - start: 172.25.202.72<br> end: 172.25.202.87<br> vlan: 1106<br>- name: InternalApi<br> name_lower: internal_api<br> vip: true<br> mtu: 1500<br> subnets:<br> internal_api_subnet:<br> ip_subnet: <a href="http://172.25.201.192/26" target="_blank">172.25.201.192/26</a><br> allocation_pools:<br> - start: 172.25.201.198<br> end: 172.25.201.212<br> vlan: 1104<br>- name: Tenant<br> vip: false # Tenant network does not use VIPs<br> mtu: 1500<br> name_lower: tenant<br> subnets:<br> tenant_subnet:<br> ip_subnet: <a href="http://172.25.202.128/26" target="_blank">172.25.202.128/26</a><br> allocation_pools:<br> - start: 172.25.202.135<br> end: 172.25.202.150<br> vlan: 1108<br>- name: External<br> name_lower: external<br> vip: true<br> mtu: 1500<br> subnets:<br> external_subnet:<br> ip_subnet: <a href="http://172.25.201.128/26" target="_blank">172.25.201.128/26</a><br> allocation_pools:<br> - start: 172.25.201.135<br> end: 172.25.201.150<br> gateway_ip: 172.25.201.129<br> vlan: 1103<br></div><div><br></div><div>(undercloud) [stack@hkg2director workplace]$ cat custom_vip_data.yaml<br>- network: ctlplane<br> #dns_name: overcloud<br> ip_address: 172.25.201.91<br> subnet: ctlplane-subnet<br>- network: external<br> #dns_name: overcloud<br> ip_address: 172.25.201.150<br> subnet: external_subnet<br>- network: internal_api<br> #dns_name: overcloud<br> ip_address: 172.25.201.250<br> subnet: internal_api_subnet<br>- network: storage<br> #dns_name: overcloud<br> ip_address: 172.25.202.50<br> subnet: storage_subnet<br>- network: storage_mgmt<br> #dns_name: overcloud<br> ip_address: 172.25.202.90<br> subnet: storage_mgmt_subnet<br></div><div><br></div><div>(undercloud) [stack@hkg2director workplace]$ cat overcloud-baremetal-deploy.yaml<br>- name: Controller<br> count: 4<br> defaults:<br> networks:<br> - network: ctlplane<br> vif: true<br> - network: external<br> subnet: external_subnet<br> - network: internal_api<br> subnet: internal_api_subnet<br> - network: storage<br> subnet: storage_subnet<br> - network: storage_mgmt<br> subnet: storage_mgmt_subnet<br> - network: tenant<br> subnet: tenant_subnet<br> network_config:<br> template: /home/stack/templates/controller.j2<br> default_route_network:<br> - external<br> instances:<br> - hostname: overcloud-controller-0<br> name: dc1-controller2<br> #provisioned: false<br> - hostname: overcloud-controller-1<br> name: dc2-controller2<br> #provisioned: false<br> - hostname: overcloud-controller-2<br> name: dc1-controller1<br> #provisioned: false<br> - hostname: overcloud-controller-no-ceph-3<br> name: dc2-ceph2<br> #provisioned: false<br> #- hostname: overcloud-controller-3<br> #name: dc2-compute3<br> #provisioned: false<br><br>- name: Compute<br> count: 5<br> defaults:<br> networks:<br> - network: ctlplane<br> vif: true<br> - network: internal_api<br> subnet: internal_api_subnet<br> - network: tenant<br> subnet: tenant_subnet<br> - network: storage<br> subnet: storage_subnet<br> network_config:<br> template: /home/stack/templates/compute.j2<br> instances:<br> - hostname: overcloud-novacompute-0<br> name: dc2-compute1<br> #provisioned: false<br> - hostname: overcloud-novacompute-1<br> name: dc2-ceph1<br> #provisioned: false<br> - hostname: overcloud-novacompute-2<br> name: dc1-compute1<br> #provisioned: false<br> - hostname: overcloud-novacompute-3<br> name: dc1-compute2<br> #provisioned: false<br> - hostname: overcloud-novacompute-4<br> name: dc2-compute3<br> #provisioned: false<br><br>- name: CephStorage<br> count: 4<br> defaults:<br> networks:<br> - network: ctlplane<br> vif: true<br> - network: internal_api<br> subnet: internal_api_subnet<br> - network: storage<br> subnet: storage_subnet<br> - network: storage_mgmt<br> subnet: storage_mgmt_subnet<br> network_config:<br> template: /home/stack/templates/ceph-storage.j2<br> instances:<br> - hostname: overcloud-cephstorage-0<br> name: dc2-controller1<br> #provisioned: false<br> # - hostname: overcloud-cephstorage-1<br> # name: dc2-ceph2<br> - hostname: overcloud-cephstorage-1<br> name: dc1-ceph1<br> # provisioned: false<br> - hostname: overcloud-cephstorage-2<br> name: dc1-ceph2<br> #provisioned: false<br> - hostname: overcloud-cephstorage-3<br> name: dc2-compute2<br> #provisioned: false<br></div><div><br></div><div><br></div><div>You must use these templates to provision network, vip and nodes.</div><div>You must use the output files generated during the provisioning step in openstack overcloud deploy command using -e parameter.</div><div><br></div><div>With regards,</div><div>Swogat Pradhan</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 23, 2022 at 8:33 PM Lokendra Rathour <<a href="mailto:lokendrarathour@gmail.com" target="_blank">lokendrarathour@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Hi Swogat,<div dir="auto">I tried checking your solution and my templates but could not relate much.</div><div dir="auto">But issue seems the same</div><div dir="auto"><a href="http://lists.openstack.org/pipermail/openstack-discuss/2022-May/028401.html" target="_blank">http://lists.openstack.org/pipermail/openstack-discuss/2022-May/028401.html</a><br></div><div dir="auto"><br></div><div dir="auto">I tried somemore ways but looks like some issue with templates.</div><div dir="auto">Can you please share the templates used to deploy the overcloud.</div><div dir="auto"><br></div><div dir="auto">Mysetup have 3 controller and 1 compute.</div><div dir="auto"><br></div><div dir="auto">Thanks once again for reading my mail.</div><div dir="auto"><br></div><div dir="auto">Waiting for your reply.</div><div dir="auto"><br></div><div dir="auto">-Lokendra </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 20 May 2022, 08:25 Swogat Pradhan, <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div>Yes I was able to find the issue and fix it.</div><div>The issue was with the overcloud-baremetal-deployed.yaml file i was trying to provision controller-0, controller-1 and controller-3 and kept controller-2 aside for later, but the tripleo scripts are built in such a way that they were taking controller- 0, 1 and 2 inplace of controller-3, so the network ports and vip were created for controller 0,1 and 2 but not for 3 , so this error was popping off. Also i would request you to check the jinja nic templates and once the node provisioning is done check the /etc/os-net-config/config.json/yaml file for syntax if using bonded nic template.</div><div>If you need any more infor please let me know.<br></div><div><br></div><div>With regards,</div><div>Swogat Pradhan</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 20, 2022 at 8:01 AM Lokendra Rathour <<a href="mailto:lokendrarathour@gmail.com" rel="noreferrer" target="_blank">lokendrarathour@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Swogat,<div>Thanks for raising this issue. <br><div>Did you find any solution? to this problem ?</div><div><br></div><div>Please let me know it might be helpful</div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 19, 2022 at 12:43 PM Swogat Pradhan <<a href="mailto:swogatpradhan22@gmail.com" rel="noreferrer" target="_blank">swogatpradhan22@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<br><div>I am currently trying to deploy openstack wallaby using tripleo arch.</div><div>I created the network jinja templates, ran the following commands also:</div><div><br></div><div>#openstack overcloud network provision --stack overcloud --output networks-deployed-environment.yaml custom_network_data.yaml</div><div># openstack overcloud network vip provision --stack overcloud --output vip-deployed-environment.yaml custom_vip_data.yaml</div><div># openstack overcloud node provision --stack overcloud --overcloud-ssh-key /home/stack/sshkey/id_rsa overcloud-baremetal-deploy.yaml</div><div><br></div><div>and used the environment files in the openstack overcloud deploy command:</div><div><br></div><div>(undercloud) [stack@hkg2director ~]$ cat deploy.sh<br>#!/bin/bash<br>THT=/usr/share/openstack-tripleo-heat-templates/<br>CNF=/home/stack/<br>openstack overcloud deploy --templates $THT \<br>-r $CNF/templates/roles_data.yaml \<br>-n $CNF/workplace/custom_network_data.yaml \<br>-e ~/containers-prepare-parameter.yaml \<br>-e $CNF/templates/node-info.yaml \<br>-e $CNF/templates/scheduler-hints.yaml \<br>-e $CNF/workplace/networks-deployed-environment.yaml \<br>-e $CNF/workplace/vip-deployed-environment.yaml \<br>-e $CNF/workplace/overcloud-baremetal-deployed.yaml \<br>-e $CNF/workplace/custom-net-bond-with-vlans.yaml<br></div><div><br></div><div>Now when i run the ./deploy.sh script i encounter an error stating:</div><div><br></div><div>ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command.: tripleoclient.exceptions.InvalidConfiguration: Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command.<br>2022-04-19 13:47:16.582 735924 INFO osc_lib.shell [-] END return value: 1<br></div><div><br></div><div>Can someone tell me where the mistake is?</div><div><br></div><div>With regards,</div><div>Swogat Pradhan</div></div>
</blockquote></div><br clear="all"><div><br></div></blockquote></div></blockquote></div></blockquote></div></blockquote></div></blockquote></div></div>
</blockquote></div><br clear="all"><div><br></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr">~ Lokendra</div><div dir="ltr"><a href="http://www.inertiaspeaks.com" target="_blank">www.inertiaspeaks.com</a></div><div dir="ltr"><a href="http://www.inertiagroups.com" target="_blank">www.inertiagroups.com</a></div><div>skype: lokendrarathour</div><div dir="ltr"><img width="200" height="41" src="https://ci3.googleusercontent.com/mail-sig/AIorK4zyd6LpJOGqagxmzUlY59eMQx0-FN0t8HtjdtGE7VLZSKIxBUz3bI7z-MBqbgDVg1-XbtvHgN_ATJ10N6bonyO-JSGTtl5s_mNSbDoXBg"><br></div><div dir="ltr"><br></div></div></div></div></div>