instance console something went wrong, connection is closed | Wallaby DCN
Swogat Pradhan
swogatpradhan22 at gmail.com
Wed Jul 12 08:49:31 UTC 2023
Hi Smooney,
Please ignore that mail, please check the other emails after that.
I can see other ports like 5900 and 5901 are being exposed and for the
controller node i am able to telnet also to the said ports.
But for some reason i am sometimes able to access the console of the vm's
and sometimes not.
I want to find a way to fix this issue.
With regards,
Swogat Pradhan
On Wed, Jul 12, 2023 at 12:31 AM <smooney at redhat.com> wrote:
> On Tue, 2023-07-11 at 23:40 +0530, Swogat Pradhan wrote:
> > Hi Melanie,
> > After searching through i found that no service is exposing port 5901.
> > Only when I migrate the instance or do any kind of activity port 5901 is
> > exposed on the said hypervisor node and the vm console is accessible.
> >
> > So, I believe your point 'Something about redoing the port bindings is
> > helping the situation sometimes.' is correct.
> >
> > But how do I troubleshoot this?
> that does nto really make sense to me 5901-n are used by qemu to expose
> the vnc console fo the vm
> but that has nothign to do with neutorn.
>
> each vm will have a specific vnc port in the xml if you do a virsh
> isntance list.
> that can also change on migrations form host ot host.
>
> it is allcoated by libvirt and the prot range is configureed in the
> libvirt qemu config
>
> the port range is contoled by remote_display_port_min and
> remote_display_port_max
> in /etc/libvirt/qemu.conf
> """
> # Override the port for creating both VNC and SPICE sessions (min).
> # This defaults to 5900 and increases for consecutive sessions
> # or when ports are occupied, until it hits the maximum.
> #
> # Minimum must be greater than or equal to 5900 as lower number would
> # result into negative vnc display number.
> #
> # Maximum must be less than 65536, because higher numbers do not make
> # sense as a port number.
> #
> #remote_display_port_min = 5900
> #remote_display_port_max = 65535
> """
>
> libvirt just increment the number form the min value for each instance on
> the host.
>
> you might need to check your firewall config to ensure that an approare
> range is openend for each host
> typically you do not have more then a hounded vms on a give host but a
> common default is to open 1024
> ports to not hit the limit.
> tripleo opens "5900-6923"
>
> https://github.com/openstack/tripleo-heat-templates/blob/1393d39be367db3acb02508e0e858395a4e4fefa/deployment/nova/nova-modular-libvirt-container-puppet.yaml#L391
> so you need to ensure that those ports are acceable in the host in the dcn
> site form the cental site where the nova vnc
> proxy is located.
>
> if you are seeing errors in virtqemud.log related ot Cannot write data:
> Broken pipe
> that might point to a differnt error related to virtlogd so you shoudl
> also see fi there are any related issues there.
> for example its a know limitation taht if virtlogd is restarted it will
> break the console
> https://bugs.launchpad.net/tripleo/+bug/1838272
> https://review.opendev.org/c/openstack/puppet-tripleo/+/787771 simple
> ensure tripleo did not restart the virtlogd
> contaienr but if anythying else did it would break the console fo runing
> vms untill they are restart, or moved to a new
> host. so perhaps you are hitting something similar?
>
>
> >
> > With regards,
> > Swogat Pradhan
> >
> >
> > On Fri, Jun 30, 2023 at 12:29 AM melanie witt <melwittt at gmail.com>
> wrote:
> >
> > > On 06/25/23 03:50, Swogat Pradhan wrote:
> > > > Hi,
> > > > After doing a console url show after migration, I am still unable to
> > > > access the console.
> > > >
> > > > My site consists of 1 central site and 2 DCN sites. Consoles for
> > > > central and DCN02 are working fine without any issues.
> > > > But when i am creating an instance for DCN01 the console for the
> > > > instance is not coming up (attached image for reference).
> > > >
> > > > Today I created 3 different VM's using the same flavor, image,
> security
> > > > group, the instances were created in the same compute host. Console
> was
> > > > not accessible, so I shelved and unshelved all 3 instances, after
> which
> > > > I was able to access the console for 2 of those VM's and am still
> unable
> > > > to access the console of the 3rd VM no matter what I do.
> > >
> > > Apologies for the delayed reply.
> > >
> > > It sounds like there may be some type of problem with regard to the
> > > network connection from the novnc console proxy service to the DCN01
> > > site rather than something with the console proxy itself given that
> > > things work fine with DCN02 and central. You're seeing "Cannot write
> > > data: Broken pipe" in the connection between the compute host and the
> > > console proxy showing a connection being broken after being
> established.
> > >
> > > As for why shelve and unshelve sometimes helps, it may be because the
> > > network port bindings are updated to inactive during the shelve and
> then
> > > they are updated to active during the unshelve. Something about redoing
> > > the port bindings is helping the situation sometimes.
> > >
> > > It may be worthwhile to check if there is anything different between
> the
> > > networks/ports the instances with working consoles have vs what the
> > > instances with non-working consoles have.
> > >
> > > -melwitt
> > >
> > > > On Sat, Jun 24, 2023 at 2:00 AM melanie witt <melwittt at gmail.com
> > > > <mailto:melwittt at gmail.com>> wrote:
> > > >
> > > > On 06/22/23 20:07, Swogat Pradhan wrote:
> > > > > Hi Mel,
> > > > > Thank you for your response.
> > > > > I am facing issues with the instance console (vnc) in the
> > > openstack
> > > > > dashboard, Most of the time I shelve the instance and
> unshelve the
> > > > > instance to get the console.
> > > > > But there are some VM's I created which are not working even
> after
> > > > > shelve/unshelve.
> > > > >
> > > > > I have used the same director to deploy a total of a central
> and
> > > > 2 edge
> > > > > sites.
> > > > > This issue is happening on a single edge site.
> > > > > Cold Migration also helps in some situations.
> > > >
> > > > OK, you didn't mention whether requesting a new console
> 'openstack
> > > > console url show --vnc <server>' gets you a working console
> after a
> > > > migration (or other event where you see the console stop
> working).
> > > I'm
> > > > trying to determine whether the behavior you're seeing is
> expected
> > > or a
> > > > bug. After an instance is moved to a different compute node than
> the
> > > > one
> > > > it was on when the console was started, that console is not
> expected
> > > to
> > > > work anymore. And a new console needs to be started.
> > > >
> > > > Can you give steps for reproducing the issue? Maybe that will
> provide
> > > > more clarity.
> > > >
> > > > -melwitt
> > > >
> > > > > On Fri, Jun 23, 2023 at 12:42 AM melanie witt <
> melwittt at gmail.com
> > > > <mailto:melwittt at gmail.com>
> > > > > <mailto:melwittt at gmail.com <mailto:melwittt at gmail.com>>>
> wrote:
> > > > >
> > > > > On 06/22/23 01:08, Swogat Pradhan wrote:
> > > > > > Hi,
> > > > > > Please find the below log:
> > > > > > [root at dcn01-hci-1 libvirt]# cat virtqemud.log
> > > > > > 2023-06-22 07:40:01.575+0000: 350319: error :
> > > > > virNetSocketReadWire:1804
> > > > > > : End of file while reading data: Input/output error
> > > > > > 2023-06-22 07:40:01.575+0000: 350319: error :
> > > > > virNetSocketWriteWire:1844
> > > > > > : Cannot write data: Broken pipe
> > > > > >
> > > > > > I think this is causing the problem of not getting the
> > > > instance
> > > > > console.
> > > > >
> > > > > When you say "instance console" are you referring to an
> > > > interactive
> > > > > console like VNC or are you talking about the console log
> for
> > > the
> > > > > instance?
> > > > >
> > > > > If it's the interactive console, if you have a console
> open
> > > > and then
> > > > > migrate the instance, that console will not be moved along
> > > > with the
> > > > > instance. When a user requests a console, the console
> proxy
> > > > service
> > > > > establishes a connection to the compute host where the
> > > > instance is
> > > > > located. The proxy doesn't know when an instance has been
> > > > moved though,
> > > > > so if the instance is moved, the user will need to
> request a
> > > new
> > > > > console
> > > > > (which will establish a new connection to the new compute
> > > host).
> > > > >
> > > > > Is that the behavior you are seeing?
> > > > >
> > > > > -melwitt
> > > > >
> > > > > > On Fri, Jun 2, 2023 at 11:27 AM Swogat Pradhan
> > > > > > <swogatpradhan22 at gmail.com
> > > > <mailto:swogatpradhan22 at gmail.com> <mailto:
> swogatpradhan22 at gmail.com
> > > > <mailto:swogatpradhan22 at gmail.com>>
> > > > > <mailto:swogatpradhan22 at gmail.com
> > > > <mailto:swogatpradhan22 at gmail.com>
> > > > > <mailto:swogatpradhan22 at gmail.com
> > > > <mailto:swogatpradhan22 at gmail.com>>>> wrote:
> > > > > >
> > > > > > Update:
> > > > > > If the i am performing any activity like migration
> or
> > > > resize
> > > > > of an
> > > > > > instance whose console is accessible, the console
> > > becomes
> > > > > > inaccessible giving out the following error :
> > > > something went
> > > > > wrong,
> > > > > > connection is closed
> > > > > >
> > > > > > The was 1 other instance whose console was not
> > > > accessible and
> > > > > i did
> > > > > > a shelve and unshelve and suddenly the instance
> > > > console became
> > > > > > accessible.
> > > > > >
> > > > > > This is a peculiar behavior and i don't understand
> > > > where is
> > > > > the issue .
> > > > > >
> > > > > > With regards,
> > > > > > Swogat Pradhan
> > > > > >
> > > > > > On Fri, Jun 2, 2023 at 11:19 AM Swogat Pradhan
> > > > > > <swogatpradhan22 at gmail.com
> > > > <mailto:swogatpradhan22 at gmail.com> <mailto:
> swogatpradhan22 at gmail.com
> > > > <mailto:swogatpradhan22 at gmail.com>>
> > > > > <mailto:swogatpradhan22 at gmail.com
> > > > <mailto:swogatpradhan22 at gmail.com>
> > > > > <mailto:swogatpradhan22 at gmail.com
> > > > <mailto:swogatpradhan22 at gmail.com>>>> wrote:
> > > > > >
> > > > > > Hi,
> > > > > > I am creating instances in my DCN site and i am
> > > > unable to get
> > > > > > the console sometimes, error: something went
> wrong,
> > > > > connection
> > > > > > is closed
> > > > > >
> > > > > > I have 3 instances now running on my hci02 node
> > > > and there is
> > > > > > console access on 1 of the vm's and the rest
> two i
> > > > am not
> > > > > > getting the console, i have used the same
> flavor,
> > > > same image
> > > > > > same security group for the VM's.
> > > > > >
> > > > > > Please suggest what can be done.
> > > > > >
> > > > > > With regards,
> > > > > > Swogat Pradhan
> > > > > >
> > > > >
> > > >
> > >
> > >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230712/3534f67f/attachment-0001.htm>
More information about the openstack-discuss
mailing list