instance console something went wrong, connection is closed | Wallaby DCN
smooney at redhat.com
smooney at redhat.com
Tue Jul 11 19:01:00 UTC 2023
On Tue, 2023-07-11 at 23:40 +0530, Swogat Pradhan wrote:
> Hi Melanie,
> After searching through i found that no service is exposing port 5901.
> Only when I migrate the instance or do any kind of activity port 5901 is
> exposed on the said hypervisor node and the vm console is accessible.
>
> So, I believe your point 'Something about redoing the port bindings is
> helping the situation sometimes.' is correct.
>
> But how do I troubleshoot this?
that does nto really make sense to me 5901-n are used by qemu to expose the vnc console fo the vm
but that has nothign to do with neutorn.
each vm will have a specific vnc port in the xml if you do a virsh isntance list.
that can also change on migrations form host ot host.
it is allcoated by libvirt and the prot range is configureed in the libvirt qemu config
the port range is contoled by remote_display_port_min and remote_display_port_max
in /etc/libvirt/qemu.conf
"""
# Override the port for creating both VNC and SPICE sessions (min).
# This defaults to 5900 and increases for consecutive sessions
# or when ports are occupied, until it hits the maximum.
#
# Minimum must be greater than or equal to 5900 as lower number would
# result into negative vnc display number.
#
# Maximum must be less than 65536, because higher numbers do not make
# sense as a port number.
#
#remote_display_port_min = 5900
#remote_display_port_max = 65535
"""
libvirt just increment the number form the min value for each instance on the host.
you might need to check your firewall config to ensure that an approare range is openend for each host
typically you do not have more then a hounded vms on a give host but a common default is to open 1024
ports to not hit the limit.
tripleo opens "5900-6923"
https://github.com/openstack/tripleo-heat-templates/blob/1393d39be367db3acb02508e0e858395a4e4fefa/deployment/nova/nova-modular-libvirt-container-puppet.yaml#L391
so you need to ensure that those ports are acceable in the host in the dcn site form the cental site where the nova vnc
proxy is located.
if you are seeing errors in virtqemud.log related ot Cannot write data: Broken pipe
that might point to a differnt error related to virtlogd so you shoudl also see fi there are any related issues there.
for example its a know limitation taht if virtlogd is restarted it will break the console
https://bugs.launchpad.net/tripleo/+bug/1838272
https://review.opendev.org/c/openstack/puppet-tripleo/+/787771 simple ensure tripleo did not restart the virtlogd
contaienr but if anythying else did it would break the console fo runing vms untill they are restart, or moved to a new
host. so perhaps you are hitting something similar?
>
> With regards,
> Swogat Pradhan
>
>
> On Fri, Jun 30, 2023 at 12:29 AM melanie witt <melwittt at gmail.com> wrote:
>
> > On 06/25/23 03:50, Swogat Pradhan wrote:
> > > Hi,
> > > After doing a console url show after migration, I am still unable to
> > > access the console.
> > >
> > > My site consists of 1 central site and 2 DCN sites. Consoles for
> > > central and DCN02 are working fine without any issues.
> > > But when i am creating an instance for DCN01 the console for the
> > > instance is not coming up (attached image for reference).
> > >
> > > Today I created 3 different VM's using the same flavor, image, security
> > > group, the instances were created in the same compute host. Console was
> > > not accessible, so I shelved and unshelved all 3 instances, after which
> > > I was able to access the console for 2 of those VM's and am still unable
> > > to access the console of the 3rd VM no matter what I do.
> >
> > Apologies for the delayed reply.
> >
> > It sounds like there may be some type of problem with regard to the
> > network connection from the novnc console proxy service to the DCN01
> > site rather than something with the console proxy itself given that
> > things work fine with DCN02 and central. You're seeing "Cannot write
> > data: Broken pipe" in the connection between the compute host and the
> > console proxy showing a connection being broken after being established.
> >
> > As for why shelve and unshelve sometimes helps, it may be because the
> > network port bindings are updated to inactive during the shelve and then
> > they are updated to active during the unshelve. Something about redoing
> > the port bindings is helping the situation sometimes.
> >
> > It may be worthwhile to check if there is anything different between the
> > networks/ports the instances with working consoles have vs what the
> > instances with non-working consoles have.
> >
> > -melwitt
> >
> > > On Sat, Jun 24, 2023 at 2:00 AM melanie witt <melwittt at gmail.com
> > > <mailto:melwittt at gmail.com>> wrote:
> > >
> > > On 06/22/23 20:07, Swogat Pradhan wrote:
> > > > Hi Mel,
> > > > Thank you for your response.
> > > > I am facing issues with the instance console (vnc) in the
> > openstack
> > > > dashboard, Most of the time I shelve the instance and unshelve the
> > > > instance to get the console.
> > > > But there are some VM's I created which are not working even after
> > > > shelve/unshelve.
> > > >
> > > > I have used the same director to deploy a total of a central and
> > > 2 edge
> > > > sites.
> > > > This issue is happening on a single edge site.
> > > > Cold Migration also helps in some situations.
> > >
> > > OK, you didn't mention whether requesting a new console 'openstack
> > > console url show --vnc <server>' gets you a working console after a
> > > migration (or other event where you see the console stop working).
> > I'm
> > > trying to determine whether the behavior you're seeing is expected
> > or a
> > > bug. After an instance is moved to a different compute node than the
> > > one
> > > it was on when the console was started, that console is not expected
> > to
> > > work anymore. And a new console needs to be started.
> > >
> > > Can you give steps for reproducing the issue? Maybe that will provide
> > > more clarity.
> > >
> > > -melwitt
> > >
> > > > On Fri, Jun 23, 2023 at 12:42 AM melanie witt <melwittt at gmail.com
> > > <mailto:melwittt at gmail.com>
> > > > <mailto:melwittt at gmail.com <mailto:melwittt at gmail.com>>> wrote:
> > > >
> > > > On 06/22/23 01:08, Swogat Pradhan wrote:
> > > > > Hi,
> > > > > Please find the below log:
> > > > > [root at dcn01-hci-1 libvirt]# cat virtqemud.log
> > > > > 2023-06-22 07:40:01.575+0000: 350319: error :
> > > > virNetSocketReadWire:1804
> > > > > : End of file while reading data: Input/output error
> > > > > 2023-06-22 07:40:01.575+0000: 350319: error :
> > > > virNetSocketWriteWire:1844
> > > > > : Cannot write data: Broken pipe
> > > > >
> > > > > I think this is causing the problem of not getting the
> > > instance
> > > > console.
> > > >
> > > > When you say "instance console" are you referring to an
> > > interactive
> > > > console like VNC or are you talking about the console log for
> > the
> > > > instance?
> > > >
> > > > If it's the interactive console, if you have a console open
> > > and then
> > > > migrate the instance, that console will not be moved along
> > > with the
> > > > instance. When a user requests a console, the console proxy
> > > service
> > > > establishes a connection to the compute host where the
> > > instance is
> > > > located. The proxy doesn't know when an instance has been
> > > moved though,
> > > > so if the instance is moved, the user will need to request a
> > new
> > > > console
> > > > (which will establish a new connection to the new compute
> > host).
> > > >
> > > > Is that the behavior you are seeing?
> > > >
> > > > -melwitt
> > > >
> > > > > On Fri, Jun 2, 2023 at 11:27 AM Swogat Pradhan
> > > > > <swogatpradhan22 at gmail.com
> > > <mailto:swogatpradhan22 at gmail.com> <mailto:swogatpradhan22 at gmail.com
> > > <mailto:swogatpradhan22 at gmail.com>>
> > > > <mailto:swogatpradhan22 at gmail.com
> > > <mailto:swogatpradhan22 at gmail.com>
> > > > <mailto:swogatpradhan22 at gmail.com
> > > <mailto:swogatpradhan22 at gmail.com>>>> wrote:
> > > > >
> > > > > Update:
> > > > > If the i am performing any activity like migration or
> > > resize
> > > > of an
> > > > > instance whose console is accessible, the console
> > becomes
> > > > > inaccessible giving out the following error :
> > > something went
> > > > wrong,
> > > > > connection is closed
> > > > >
> > > > > The was 1 other instance whose console was not
> > > accessible and
> > > > i did
> > > > > a shelve and unshelve and suddenly the instance
> > > console became
> > > > > accessible.
> > > > >
> > > > > This is a peculiar behavior and i don't understand
> > > where is
> > > > the issue .
> > > > >
> > > > > With regards,
> > > > > Swogat Pradhan
> > > > >
> > > > > On Fri, Jun 2, 2023 at 11:19 AM Swogat Pradhan
> > > > > <swogatpradhan22 at gmail.com
> > > <mailto:swogatpradhan22 at gmail.com> <mailto:swogatpradhan22 at gmail.com
> > > <mailto:swogatpradhan22 at gmail.com>>
> > > > <mailto:swogatpradhan22 at gmail.com
> > > <mailto:swogatpradhan22 at gmail.com>
> > > > <mailto:swogatpradhan22 at gmail.com
> > > <mailto:swogatpradhan22 at gmail.com>>>> wrote:
> > > > >
> > > > > Hi,
> > > > > I am creating instances in my DCN site and i am
> > > unable to get
> > > > > the console sometimes, error: something went wrong,
> > > > connection
> > > > > is closed
> > > > >
> > > > > I have 3 instances now running on my hci02 node
> > > and there is
> > > > > console access on 1 of the vm's and the rest two i
> > > am not
> > > > > getting the console, i have used the same flavor,
> > > same image
> > > > > same security group for the VM's.
> > > > >
> > > > > Please suggest what can be done.
> > > > >
> > > > > With regards,
> > > > > Swogat Pradhan
> > > > >
> > > >
> > >
> >
> >
More information about the openstack-discuss
mailing list