Open Stack

I am getting the following error in nova-novncproxy.log in controller node:

2023-07-11 18:28:50.464 1080 INFO nova.console.websocketproxy
[req-847a31bd-b896-4535-a4b3-6e7d163f0ec8 - - - - -] handler exception:
[Errno 107] Transport endpoint is not connected

On Wed, Jul 12, 2023 at 12:01 AM Swogat Pradhan <swogatpradhan22 at gmail.com>
wrote:

> Please ignore the previous email about the port information.
>
> So, I believe your point 'Something about redoing the port bindings is
> helping the situation sometimes.' is correct.
>
> But how do I troubleshoot this?
>
> With regards,
> Swogat Pradhan
>
> On Tue, Jul 11, 2023 at 11:40 PM Swogat Pradhan <swogatpradhan22 at gmail.com>
> wrote:
>
>> Hi Melanie,
>> After searching through i found that no service is exposing port 5901.
>> Only when I migrate the instance or do any kind of activity port 5901 is
>> exposed on the said hypervisor node and the vm console is accessible.
>>
>> So, I believe your point 'Something about redoing the port bindings is
>> helping the situation sometimes.' is correct.
>>
>> But how do I troubleshoot this?
>>
>> With regards,
>> Swogat Pradhan
>>
>>
>> On Fri, Jun 30, 2023 at 12:29 AM melanie witt <melwittt at gmail.com> wrote:
>>
>>> On 06/25/23 03:50, Swogat Pradhan wrote:
>>> > Hi,
>>> > After doing a console url show after migration, I am still unable to
>>> > access the console.
>>> >
>>> >   My site consists of 1 central site and 2 DCN sites. Consoles for
>>> > central and DCN02 are working fine without any issues.
>>> > But when i am creating an instance for DCN01 the console for the
>>> > instance is not coming up (attached image for reference).
>>> >
>>> > Today I created 3 different VM's using the same flavor, image,
>>> security
>>> > group, the instances were created in the same compute host.
>>> Console was
>>> > not accessible, so I shelved and unshelved all 3 instances, after
>>> which
>>> > I was able to access the console for 2 of those VM's and am still
>>> unable
>>> > to access the console of  the 3rd VM no matter what I do.
>>>
>>> Apologies for the delayed reply.
>>>
>>> It sounds like there may be some type of problem with regard to the
>>> network connection from the novnc console proxy service to the DCN01
>>> site rather than something with the console proxy itself given that
>>> things work fine with DCN02 and central. You're seeing "Cannot write
>>> data: Broken pipe" in the connection between the compute host and the
>>> console proxy showing a connection being broken after being established.
>>>
>>> As for why shelve and unshelve sometimes helps, it may be because the
>>> network port bindings are updated to inactive during the shelve and then
>>> they are updated to active during the unshelve. Something about redoing
>>> the port bindings is helping the situation sometimes.
>>>
>>> It may be worthwhile to check if there is anything different between the
>>> networks/ports the instances with working consoles have vs what the
>>> instances with non-working consoles have.
>>>
>>> -melwitt
>>>
>>> > On Sat, Jun 24, 2023 at 2:00 AM melanie witt <melwittt at gmail.com
>>> > <mailto:melwittt at gmail.com>> wrote:
>>> >
>>> >     On 06/22/23 20:07, Swogat Pradhan wrote:
>>> >      > Hi Mel,
>>> >      > Thank you for your response.
>>> >      > I am facing issues with the instance console (vnc) in the
>>> openstack
>>> >      > dashboard, Most of the time I shelve the instance and unshelve
>>> the
>>> >      > instance to get the console.
>>> >      > But there are some VM's I created which are not working even
>>> after
>>> >      > shelve/unshelve.
>>> >      >
>>> >      > I have used the same director to deploy a total of a central and
>>> >     2 edge
>>> >      > sites.
>>> >      > This issue is happening on a single edge site.
>>> >      > Cold Migration also helps in some situations.
>>> >
>>> >     OK, you didn't mention whether requesting a new console 'openstack
>>> >     console url show --vnc <server>' gets you a working console after a
>>> >     migration (or other event where you see the console stop working).
>>> I'm
>>> >     trying to determine whether the behavior you're seeing is expected
>>> or a
>>> >     bug. After an instance is moved to a different compute node than
>>> the
>>> >     one
>>> >     it was on when the console was started, that console is not
>>> expected to
>>> >     work anymore. And a new console needs to be started.
>>> >
>>> >     Can you give steps for reproducing the issue? Maybe that will
>>> provide
>>> >     more clarity.
>>> >
>>> >     -melwitt
>>> >
>>> >      > On Fri, Jun 23, 2023 at 12:42 AM melanie witt <
>>> melwittt at gmail.com
>>> >     <mailto:melwittt at gmail.com>
>>> >      > <mailto:melwittt at gmail.com <mailto:melwittt at gmail.com>>> wrote:
>>> >      >
>>> >      >     On 06/22/23 01:08, Swogat Pradhan wrote:
>>> >      >      > Hi,
>>> >      >      > Please find the below log:
>>> >      >      > [root at dcn01-hci-1 libvirt]# cat virtqemud.log
>>> >      >      > 2023-06-22 07:40:01.575+0000: 350319: error :
>>> >      >     virNetSocketReadWire:1804
>>> >      >      > : End of file while reading data: Input/output error
>>> >      >      > 2023-06-22 07:40:01.575+0000: 350319: error :
>>> >      >     virNetSocketWriteWire:1844
>>> >      >      > : Cannot write data: Broken pipe
>>> >      >      >
>>> >      >      > I think this is causing the problem of not getting the
>>> >     instance
>>> >      >     console.
>>> >      >
>>> >      >     When you say "instance console" are you referring to an
>>> >     interactive
>>> >      >     console like VNC or are you talking about the console log
>>> for the
>>> >      >     instance?
>>> >      >
>>> >      >     If it's the interactive console, if you have a console open
>>> >     and then
>>> >      >     migrate the instance, that console will not be moved along
>>> >     with the
>>> >      >     instance. When a user requests a console, the console proxy
>>> >     service
>>> >      >     establishes a connection to the compute host where the
>>> >     instance is
>>> >      >     located. The proxy doesn't know when an instance has been
>>> >     moved though,
>>> >      >     so if the instance is moved, the user will need to request
>>> a new
>>> >      >     console
>>> >      >     (which will establish a new connection to the new compute
>>> host).
>>> >      >
>>> >      >     Is that the behavior you are seeing?
>>> >      >
>>> >      >     -melwitt
>>> >      >
>>> >      >      > On Fri, Jun 2, 2023 at 11:27 AM Swogat Pradhan
>>> >      >      > <swogatpradhan22 at gmail.com
>>> >     <mailto:swogatpradhan22 at gmail.com> <mailto:
>>> swogatpradhan22 at gmail.com
>>> >     <mailto:swogatpradhan22 at gmail.com>>
>>> >      >     <mailto:swogatpradhan22 at gmail.com
>>> >     <mailto:swogatpradhan22 at gmail.com>
>>> >      >     <mailto:swogatpradhan22 at gmail.com
>>> >     <mailto:swogatpradhan22 at gmail.com>>>> wrote:
>>> >      >      >
>>> >      >      >     Update:
>>> >      >      >     If the i am performing any activity like migration or
>>> >     resize
>>> >      >     of an
>>> >      >      >     instance whose console is accessible, the console
>>> becomes
>>> >      >      >     inaccessible giving out the following error :
>>> >     something went
>>> >      >     wrong,
>>> >      >      >     connection is closed
>>> >      >      >
>>> >      >      >     The was 1 other instance whose console was not
>>> >     accessible and
>>> >      >     i did
>>> >      >      >     a shelve and unshelve and suddenly the instance
>>> >     console became
>>> >      >      >     accessible.
>>> >      >      >
>>> >      >      >     This is a peculiar behavior and i don't understand
>>> >     where is
>>> >      >     the issue .
>>> >      >      >
>>> >      >      >     With regards,
>>> >      >      >     Swogat Pradhan
>>> >      >      >
>>> >      >      >     On Fri, Jun 2, 2023 at 11:19 AM Swogat Pradhan
>>> >      >      >     <swogatpradhan22 at gmail.com
>>> >     <mailto:swogatpradhan22 at gmail.com> <mailto:
>>> swogatpradhan22 at gmail.com
>>> >     <mailto:swogatpradhan22 at gmail.com>>
>>> >      >     <mailto:swogatpradhan22 at gmail.com
>>> >     <mailto:swogatpradhan22 at gmail.com>
>>> >      >     <mailto:swogatpradhan22 at gmail.com
>>> >     <mailto:swogatpradhan22 at gmail.com>>>> wrote:
>>> >      >      >
>>> >      >      >         Hi,
>>> >      >      >         I am creating instances in my DCN site and i am
>>> >     unable to get
>>> >      >      >         the console sometimes, error: something went
>>> wrong,
>>> >      >     connection
>>> >      >      >         is closed
>>> >      >      >
>>> >      >      >         I have 3 instances now running on my hci02 node
>>> >     and there is
>>> >      >      >         console access on 1 of the vm's and the rest two
>>> i
>>> >     am not
>>> >      >      >         getting the console, i have used the same flavor,
>>> >     same image
>>> >      >      >         same security group for the VM's.
>>> >      >      >
>>> >      >      >         Please suggest what can be done.
>>> >      >      >
>>> >      >      >         With regards,
>>> >      >      >         Swogat Pradhan
>>> >      >      >
>>> >      >
>>> >
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230712/59f290fd/attachment-0001.htm>