<div dir="ltr">Hi Smooney,<div>Please ignore that mail, please check the other emails after that.</div><div>I can see other ports like 5900 and 5901 are being exposed and for the controller node i am able to telnet also to the said ports.</div><div>But for some reason i am sometimes able to access the console of the vm's and sometimes not.</div><div>I want to find a way to fix this issue.</div><div><br></div><div>With regards,</div><div>Swogat Pradhan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 12, 2023 at 12:31 AM <<a href="mailto:smooney@redhat.com">smooney@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Tue, 2023-07-11 at 23:40 +0530, Swogat Pradhan wrote:<br>
> Hi Melanie,<br>
> After searching through i found that no service is exposing port 5901.<br>
> Only when I migrate the instance or do any kind of activity port 5901 is<br>
> exposed on the said hypervisor node and the vm console is accessible.<br>
> <br>
> So, I believe your point 'Something about redoing the port bindings is<br>
> helping the situation sometimes.' is correct.<br>
> <br>
> But how do I troubleshoot this?<br>
that does nto really make sense to me 5901-n are used by qemu to expose the vnc console fo the vm<br>
but that has nothign to do with neutorn.<br>
<br>
each vm will have a specific vnc port in the xml if you do a virsh isntance list.<br>
that can also change on migrations form host ot host.<br>
<br>
it is allcoated by libvirt and the prot range is configureed in the libvirt qemu config<br>
<br>
the port range is contoled by remote_display_port_min and remote_display_port_max<br>
in /etc/libvirt/qemu.conf<br>
"""<br>
# Override the port for creating both VNC and SPICE sessions (min).<br>
# This defaults to 5900 and increases for consecutive sessions<br>
# or when ports are occupied, until it hits the maximum.<br>
#<br>
# Minimum must be greater than or equal to 5900 as lower number would<br>
# result into negative vnc display number.<br>
#<br>
# Maximum must be less than 65536, because higher numbers do not make<br>
# sense as a port number.<br>
#<br>
#remote_display_port_min = 5900<br>
#remote_display_port_max = 65535<br>
"""<br>
<br>
libvirt just increment the number form the min value for each instance on the host.<br>
<br>
you might need to check your firewall config to ensure that an approare range is openend for each host<br>
typically you do not have more then a hounded vms on a give host but a common default is to open 1024<br>
ports to not hit the limit.<br>
tripleo opens "5900-6923"<br>
<a href="https://github.com/openstack/tripleo-heat-templates/blob/1393d39be367db3acb02508e0e858395a4e4fefa/deployment/nova/nova-modular-libvirt-container-puppet.yaml#L391" rel="noreferrer" target="_blank">https://github.com/openstack/tripleo-heat-templates/blob/1393d39be367db3acb02508e0e858395a4e4fefa/deployment/nova/nova-modular-libvirt-container-puppet.yaml#L391</a><br>
so you need to ensure that those ports are acceable in the host in the dcn site form the cental site where the nova vnc<br>
proxy is located.<br>
<br>
if you are seeing errors in virtqemud.log related ot Cannot write data: Broken pipe<br>
that might point to a differnt error related to virtlogd so you shoudl also see fi there are any related issues there.<br>
for example its a know limitation taht if virtlogd is restarted it will break the console<br>
<a href="https://bugs.launchpad.net/tripleo/+bug/1838272" rel="noreferrer" target="_blank">https://bugs.launchpad.net/tripleo/+bug/1838272</a><br>
<a href="https://review.opendev.org/c/openstack/puppet-tripleo/+/787771" rel="noreferrer" target="_blank">https://review.opendev.org/c/openstack/puppet-tripleo/+/787771</a> simple ensure tripleo did not restart the virtlogd<br>
contaienr but if anythying else did it would break the console fo runing vms untill they are restart, or moved to a new<br>
host. so perhaps you are hitting something similar?<br>
<br>
<br>
> <br>
> With regards,<br>
> Swogat Pradhan<br>
> <br>
> <br>
> On Fri, Jun 30, 2023 at 12:29 AM melanie witt <<a href="mailto:melwittt@gmail.com" target="_blank">melwittt@gmail.com</a>> wrote:<br>
> <br>
> > On 06/25/23 03:50, Swogat Pradhan wrote:<br>
> > > Hi,<br>
> > > After doing a console url show after migration, I am still unable to<br>
> > > access the console.<br>
> > > <br>
> > > My site consists of 1 central site and 2 DCN sites. Consoles for<br>
> > > central and DCN02 are working fine without any issues.<br>
> > > But when i am creating an instance for DCN01 the console for the<br>
> > > instance is not coming up (attached image for reference).<br>
> > > <br>
> > > Today I created 3 different VM's using the same flavor, image, security<br>
> > > group, the instances were created in the same compute host. Console was<br>
> > > not accessible, so I shelved and unshelved all 3 instances, after which<br>
> > > I was able to access the console for 2 of those VM's and am still unable<br>
> > > to access the console of the 3rd VM no matter what I do.<br>
> > <br>
> > Apologies for the delayed reply.<br>
> > <br>
> > It sounds like there may be some type of problem with regard to the<br>
> > network connection from the novnc console proxy service to the DCN01<br>
> > site rather than something with the console proxy itself given that<br>
> > things work fine with DCN02 and central. You're seeing "Cannot write<br>
> > data: Broken pipe" in the connection between the compute host and the<br>
> > console proxy showing a connection being broken after being established.<br>
> > <br>
> > As for why shelve and unshelve sometimes helps, it may be because the<br>
> > network port bindings are updated to inactive during the shelve and then<br>
> > they are updated to active during the unshelve. Something about redoing<br>
> > the port bindings is helping the situation sometimes.<br>
> > <br>
> > It may be worthwhile to check if there is anything different between the<br>
> > networks/ports the instances with working consoles have vs what the<br>
> > instances with non-working consoles have.<br>
> > <br>
> > -melwitt<br>
> > <br>
> > > On Sat, Jun 24, 2023 at 2:00 AM melanie witt <<a href="mailto:melwittt@gmail.com" target="_blank">melwittt@gmail.com</a><br>
> > > <mailto:<a href="mailto:melwittt@gmail.com" target="_blank">melwittt@gmail.com</a>>> wrote:<br>
> > > <br>
> > > On 06/22/23 20:07, Swogat Pradhan wrote:<br>
> > > > Hi Mel,<br>
> > > > Thank you for your response.<br>
> > > > I am facing issues with the instance console (vnc) in the<br>
> > openstack<br>
> > > > dashboard, Most of the time I shelve the instance and unshelve the<br>
> > > > instance to get the console.<br>
> > > > But there are some VM's I created which are not working even after<br>
> > > > shelve/unshelve.<br>
> > > ><br>
> > > > I have used the same director to deploy a total of a central and<br>
> > > 2 edge<br>
> > > > sites.<br>
> > > > This issue is happening on a single edge site.<br>
> > > > Cold Migration also helps in some situations.<br>
> > > <br>
> > > OK, you didn't mention whether requesting a new console 'openstack<br>
> > > console url show --vnc <server>' gets you a working console after a<br>
> > > migration (or other event where you see the console stop working).<br>
> > I'm<br>
> > > trying to determine whether the behavior you're seeing is expected<br>
> > or a<br>
> > > bug. After an instance is moved to a different compute node than the<br>
> > > one<br>
> > > it was on when the console was started, that console is not expected<br>
> > to<br>
> > > work anymore. And a new console needs to be started.<br>
> > > <br>
> > > Can you give steps for reproducing the issue? Maybe that will provide<br>
> > > more clarity.<br>
> > > <br>
> > > -melwitt<br>
> > > <br>
> > > > On Fri, Jun 23, 2023 at 12:42 AM melanie witt <<a href="mailto:melwittt@gmail.com" target="_blank">melwittt@gmail.com</a><br>
> > > <mailto:<a href="mailto:melwittt@gmail.com" target="_blank">melwittt@gmail.com</a>><br>
> > > > <mailto:<a href="mailto:melwittt@gmail.com" target="_blank">melwittt@gmail.com</a> <mailto:<a href="mailto:melwittt@gmail.com" target="_blank">melwittt@gmail.com</a>>>> wrote:<br>
> > > ><br>
> > > > On 06/22/23 01:08, Swogat Pradhan wrote:<br>
> > > > > Hi,<br>
> > > > > Please find the below log:<br>
> > > > > [root@dcn01-hci-1 libvirt]# cat virtqemud.log<br>
> > > > > 2023-06-22 07:40:01.575+0000: 350319: error :<br>
> > > > virNetSocketReadWire:1804<br>
> > > > > : End of file while reading data: Input/output error<br>
> > > > > 2023-06-22 07:40:01.575+0000: 350319: error :<br>
> > > > virNetSocketWriteWire:1844<br>
> > > > > : Cannot write data: Broken pipe<br>
> > > > ><br>
> > > > > I think this is causing the problem of not getting the<br>
> > > instance<br>
> > > > console.<br>
> > > ><br>
> > > > When you say "instance console" are you referring to an<br>
> > > interactive<br>
> > > > console like VNC or are you talking about the console log for<br>
> > the<br>
> > > > instance?<br>
> > > ><br>
> > > > If it's the interactive console, if you have a console open<br>
> > > and then<br>
> > > > migrate the instance, that console will not be moved along<br>
> > > with the<br>
> > > > instance. When a user requests a console, the console proxy<br>
> > > service<br>
> > > > establishes a connection to the compute host where the<br>
> > > instance is<br>
> > > > located. The proxy doesn't know when an instance has been<br>
> > > moved though,<br>
> > > > so if the instance is moved, the user will need to request a<br>
> > new<br>
> > > > console<br>
> > > > (which will establish a new connection to the new compute<br>
> > host).<br>
> > > ><br>
> > > > Is that the behavior you are seeing?<br>
> > > ><br>
> > > > -melwitt<br>
> > > ><br>
> > > > > On Fri, Jun 2, 2023 at 11:27 AM Swogat Pradhan<br>
> > > > > <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a><br>
> > > <mailto:<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> <mailto:<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a><br>
> > > <mailto:<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>>><br>
> > > > <mailto:<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a><br>
> > > <mailto:<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>><br>
> > > > <mailto:<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a><br>
> > > <mailto:<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>>>>> wrote:<br>
> > > > ><br>
> > > > > Update:<br>
> > > > > If the i am performing any activity like migration or<br>
> > > resize<br>
> > > > of an<br>
> > > > > instance whose console is accessible, the console<br>
> > becomes<br>
> > > > > inaccessible giving out the following error :<br>
> > > something went<br>
> > > > wrong,<br>
> > > > > connection is closed<br>
> > > > ><br>
> > > > > The was 1 other instance whose console was not<br>
> > > accessible and<br>
> > > > i did<br>
> > > > > a shelve and unshelve and suddenly the instance<br>
> > > console became<br>
> > > > > accessible.<br>
> > > > ><br>
> > > > > This is a peculiar behavior and i don't understand<br>
> > > where is<br>
> > > > the issue .<br>
> > > > ><br>
> > > > > With regards,<br>
> > > > > Swogat Pradhan<br>
> > > > ><br>
> > > > > On Fri, Jun 2, 2023 at 11:19 AM Swogat Pradhan<br>
> > > > > <<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a><br>
> > > <mailto:<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>> <mailto:<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a><br>
> > > <mailto:<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>>><br>
> > > > <mailto:<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a><br>
> > > <mailto:<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>><br>
> > > > <mailto:<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a><br>
> > > <mailto:<a href="mailto:swogatpradhan22@gmail.com" target="_blank">swogatpradhan22@gmail.com</a>>>>> wrote:<br>
> > > > ><br>
> > > > > Hi,<br>
> > > > > I am creating instances in my DCN site and i am<br>
> > > unable to get<br>
> > > > > the console sometimes, error: something went wrong,<br>
> > > > connection<br>
> > > > > is closed<br>
> > > > ><br>
> > > > > I have 3 instances now running on my hci02 node<br>
> > > and there is<br>
> > > > > console access on 1 of the vm's and the rest two i<br>
> > > am not<br>
> > > > > getting the console, i have used the same flavor,<br>
> > > same image<br>
> > > > > same security group for the VM's.<br>
> > > > ><br>
> > > > > Please suggest what can be done.<br>
> > > > ><br>
> > > > > With regards,<br>
> > > > > Swogat Pradhan<br>
> > > > ><br>
> > > ><br>
> > > <br>
> > <br>
> > <br>
<br>
</blockquote></div>