instance console something went wrong, connection is closed | Wallaby DCN
Hi, I am creating instances in my DCN site and i am unable to get the console sometimes, error: something went wrong, connection is closed I have 3 instances now running on my hci02 node and there is console access on 1 of the vm's and the rest two i am not getting the console, i have used the same flavor, same image same security group for the VM's. Please suggest what can be done. With regards, Swogat Pradhan
Update: If the i am performing any activity like migration or resize of an instance whose console is accessible, the console becomes inaccessible giving out the following error : something went wrong, connection is closed The was 1 other instance whose console was not accessible and i did a shelve and unshelve and suddenly the instance console became accessible. This is a peculiar behavior and i don't understand where is the issue . With regards, Swogat Pradhan On Fri, Jun 2, 2023 at 11:19 AM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi, I am creating instances in my DCN site and i am unable to get the console sometimes, error: something went wrong, connection is closed
I have 3 instances now running on my hci02 node and there is console access on 1 of the vm's and the rest two i am not getting the console, i have used the same flavor, same image same security group for the VM's.
Please suggest what can be done.
With regards, Swogat Pradhan
Hi, Please find the below log: [root@dcn01-hci-1 libvirt]# cat virtqemud.log 2023-06-22 07:40:01.575+0000: 350319: error : virNetSocketReadWire:1804 : End of file while reading data: Input/output error 2023-06-22 07:40:01.575+0000: 350319: error : virNetSocketWriteWire:1844 : Cannot write data: Broken pipe I think this is causing the problem of not getting the instance console. With regards, Swogat Pradhan On Fri, Jun 2, 2023 at 11:27 AM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Update: If the i am performing any activity like migration or resize of an instance whose console is accessible, the console becomes inaccessible giving out the following error : something went wrong, connection is closed
The was 1 other instance whose console was not accessible and i did a shelve and unshelve and suddenly the instance console became accessible.
This is a peculiar behavior and i don't understand where is the issue .
With regards, Swogat Pradhan
On Fri, Jun 2, 2023 at 11:19 AM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi, I am creating instances in my DCN site and i am unable to get the console sometimes, error: something went wrong, connection is closed
I have 3 instances now running on my hci02 node and there is console access on 1 of the vm's and the rest two i am not getting the console, i have used the same flavor, same image same security group for the VM's.
Please suggest what can be done.
With regards, Swogat Pradhan
On 06/22/23 01:08, Swogat Pradhan wrote:
Hi, Please find the below log: [root@dcn01-hci-1 libvirt]# cat virtqemud.log 2023-06-22 07:40:01.575+0000: 350319: error : virNetSocketReadWire:1804 : End of file while reading data: Input/output error 2023-06-22 07:40:01.575+0000: 350319: error : virNetSocketWriteWire:1844 : Cannot write data: Broken pipe
I think this is causing the problem of not getting the instance console.
When you say "instance console" are you referring to an interactive console like VNC or are you talking about the console log for the instance? If it's the interactive console, if you have a console open and then migrate the instance, that console will not be moved along with the instance. When a user requests a console, the console proxy service establishes a connection to the compute host where the instance is located. The proxy doesn't know when an instance has been moved though, so if the instance is moved, the user will need to request a new console (which will establish a new connection to the new compute host). Is that the behavior you are seeing? -melwitt
On Fri, Jun 2, 2023 at 11:27 AM Swogat Pradhan <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>> wrote:
Update: If the i am performing any activity like migration or resize of an instance whose console is accessible, the console becomes inaccessible giving out the following error : something went wrong, connection is closed
The was 1 other instance whose console was not accessible and i did a shelve and unshelve and suddenly the instance console became accessible.
This is a peculiar behavior and i don't understand where is the issue .
With regards, Swogat Pradhan
On Fri, Jun 2, 2023 at 11:19 AM Swogat Pradhan <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>> wrote:
Hi, I am creating instances in my DCN site and i am unable to get the console sometimes, error: something went wrong, connection is closed
I have 3 instances now running on my hci02 node and there is console access on 1 of the vm's and the rest two i am not getting the console, i have used the same flavor, same image same security group for the VM's.
Please suggest what can be done.
With regards, Swogat Pradhan
Hi Mel, Thank you for your response. I am facing issues with the instance console (vnc) in the openstack dashboard, Most of the time I shelve the instance and unshelve the instance to get the console. But there are some VM's I created which are not working even after shelve/unshelve. I have used the same director to deploy a total of a central and 2 edge sites. This issue is happening on a single edge site. Cold Migration also helps in some situations. With regards, Swogat Pradhan On Fri, Jun 23, 2023 at 12:42 AM melanie witt <melwittt@gmail.com> wrote:
On 06/22/23 01:08, Swogat Pradhan wrote:
Hi, Please find the below log: [root@dcn01-hci-1 libvirt]# cat virtqemud.log 2023-06-22 07:40:01.575+0000: 350319: error : virNetSocketReadWire:1804 : End of file while reading data: Input/output error 2023-06-22 07:40:01.575+0000: 350319: error : virNetSocketWriteWire:1844 : Cannot write data: Broken pipe
I think this is causing the problem of not getting the instance console.
When you say "instance console" are you referring to an interactive console like VNC or are you talking about the console log for the instance?
If it's the interactive console, if you have a console open and then migrate the instance, that console will not be moved along with the instance. When a user requests a console, the console proxy service establishes a connection to the compute host where the instance is located. The proxy doesn't know when an instance has been moved though, so if the instance is moved, the user will need to request a new console (which will establish a new connection to the new compute host).
Is that the behavior you are seeing?
-melwitt
On Fri, Jun 2, 2023 at 11:27 AM Swogat Pradhan <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>> wrote:
Update: If the i am performing any activity like migration or resize of an instance whose console is accessible, the console becomes inaccessible giving out the following error : something went wrong, connection is closed
The was 1 other instance whose console was not accessible and i did a shelve and unshelve and suddenly the instance console became accessible.
This is a peculiar behavior and i don't understand where is the issue .
With regards, Swogat Pradhan
On Fri, Jun 2, 2023 at 11:19 AM Swogat Pradhan <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>> wrote:
Hi, I am creating instances in my DCN site and i am unable to get the console sometimes, error: something went wrong, connection is closed
I have 3 instances now running on my hci02 node and there is console access on 1 of the vm's and the rest two i am not getting the console, i have used the same flavor, same image same security group for the VM's.
Please suggest what can be done.
With regards, Swogat Pradhan
On 06/22/23 20:07, Swogat Pradhan wrote:
Hi Mel, Thank you for your response. I am facing issues with the instance console (vnc) in the openstack dashboard, Most of the time I shelve the instance and unshelve the instance to get the console. But there are some VM's I created which are not working even after shelve/unshelve.
I have used the same director to deploy a total of a central and 2 edge sites. This issue is happening on a single edge site. Cold Migration also helps in some situations.
OK, you didn't mention whether requesting a new console 'openstack console url show --vnc <server>' gets you a working console after a migration (or other event where you see the console stop working). I'm trying to determine whether the behavior you're seeing is expected or a bug. After an instance is moved to a different compute node than the one it was on when the console was started, that console is not expected to work anymore. And a new console needs to be started. Can you give steps for reproducing the issue? Maybe that will provide more clarity. -melwitt
On Fri, Jun 23, 2023 at 12:42 AM melanie witt <melwittt@gmail.com <mailto:melwittt@gmail.com>> wrote:
On 06/22/23 01:08, Swogat Pradhan wrote: > Hi, > Please find the below log: > [root@dcn01-hci-1 libvirt]# cat virtqemud.log > 2023-06-22 07:40:01.575+0000: 350319: error : virNetSocketReadWire:1804 > : End of file while reading data: Input/output error > 2023-06-22 07:40:01.575+0000: 350319: error : virNetSocketWriteWire:1844 > : Cannot write data: Broken pipe > > I think this is causing the problem of not getting the instance console.
When you say "instance console" are you referring to an interactive console like VNC or are you talking about the console log for the instance?
If it's the interactive console, if you have a console open and then migrate the instance, that console will not be moved along with the instance. When a user requests a console, the console proxy service establishes a connection to the compute host where the instance is located. The proxy doesn't know when an instance has been moved though, so if the instance is moved, the user will need to request a new console (which will establish a new connection to the new compute host).
Is that the behavior you are seeing?
-melwitt
> On Fri, Jun 2, 2023 at 11:27 AM Swogat Pradhan > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>> wrote: > > Update: > If the i am performing any activity like migration or resize of an > instance whose console is accessible, the console becomes > inaccessible giving out the following error : something went wrong, > connection is closed > > The was 1 other instance whose console was not accessible and i did > a shelve and unshelve and suddenly the instance console became > accessible. > > This is a peculiar behavior and i don't understand where is the issue . > > With regards, > Swogat Pradhan > > On Fri, Jun 2, 2023 at 11:19 AM Swogat Pradhan > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>> wrote: > > Hi, > I am creating instances in my DCN site and i am unable to get > the console sometimes, error: something went wrong, connection > is closed > > I have 3 instances now running on my hci02 node and there is > console access on 1 of the vm's and the rest two i am not > getting the console, i have used the same flavor, same image > same security group for the VM's. > > Please suggest what can be done. > > With regards, > Swogat Pradhan >
Hi, After doing a console url show after migration, I am still unable to access the console. My site consists of 1 central site and 2 DCN sites. Consoles for central and DCN02 are working fine without any issues. But when i am creating an instance for DCN01 the console for the instance is not coming up (attached image for reference). Today I created 3 different VM's using the same flavor, image, security group, the instances were created in the same compute host. Console was not accessible, so I shelved and unshelved all 3 instances, after which I was able to access the console for 2 of those VM's and am still unable to access the console of the 3rd VM no matter what I do. With regards, Swogat Pradhan On Sat, Jun 24, 2023 at 2:00 AM melanie witt <melwittt@gmail.com> wrote:
On 06/22/23 20:07, Swogat Pradhan wrote:
Hi Mel, Thank you for your response. I am facing issues with the instance console (vnc) in the openstack dashboard, Most of the time I shelve the instance and unshelve the instance to get the console. But there are some VM's I created which are not working even after shelve/unshelve.
I have used the same director to deploy a total of a central and 2 edge sites. This issue is happening on a single edge site. Cold Migration also helps in some situations.
OK, you didn't mention whether requesting a new console 'openstack console url show --vnc <server>' gets you a working console after a migration (or other event where you see the console stop working). I'm trying to determine whether the behavior you're seeing is expected or a bug. After an instance is moved to a different compute node than the one it was on when the console was started, that console is not expected to work anymore. And a new console needs to be started.
Can you give steps for reproducing the issue? Maybe that will provide more clarity.
-melwitt
On Fri, Jun 23, 2023 at 12:42 AM melanie witt <melwittt@gmail.com <mailto:melwittt@gmail.com>> wrote:
On 06/22/23 01:08, Swogat Pradhan wrote: > Hi, > Please find the below log: > [root@dcn01-hci-1 libvirt]# cat virtqemud.log > 2023-06-22 07:40:01.575+0000: 350319: error : virNetSocketReadWire:1804 > : End of file while reading data: Input/output error > 2023-06-22 07:40:01.575+0000: 350319: error : virNetSocketWriteWire:1844 > : Cannot write data: Broken pipe > > I think this is causing the problem of not getting the instance console.
When you say "instance console" are you referring to an interactive console like VNC or are you talking about the console log for the instance?
If it's the interactive console, if you have a console open and then migrate the instance, that console will not be moved along with the instance. When a user requests a console, the console proxy service establishes a connection to the compute host where the instance is located. The proxy doesn't know when an instance has been moved though, so if the instance is moved, the user will need to request a new console (which will establish a new connection to the new compute host).
Is that the behavior you are seeing?
-melwitt
> On Fri, Jun 2, 2023 at 11:27 AM Swogat Pradhan > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>> wrote: > > Update: > If the i am performing any activity like migration or resize of an > instance whose console is accessible, the console becomes > inaccessible giving out the following error : something went wrong, > connection is closed > > The was 1 other instance whose console was not accessible and i did > a shelve and unshelve and suddenly the instance console became > accessible. > > This is a peculiar behavior and i don't understand where is the issue . > > With regards, > Swogat Pradhan > > On Fri, Jun 2, 2023 at 11:19 AM Swogat Pradhan > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>> wrote: > > Hi, > I am creating instances in my DCN site and i am unable to get > the console sometimes, error: something went wrong, connection > is closed > > I have 3 instances now running on my hci02 node and there is > console access on 1 of the vm's and the rest two i am not > getting the console, i have used the same flavor, same image > same security group for the VM's. > > Please suggest what can be done. > > With regards, > Swogat Pradhan >
Nova-novncproxy.log: 2023-06-25 10:55:32.487 515433 INFO nova.console.websocketproxy [-] 172.25.201.205 - - [25/Jun/2023 10:55:32] 172.25.201.205: Plain non-SSL (ws://) WebSocket connection 2023-06-25 10:55:32.489 515433 INFO nova.console.websocketproxy [-] 172.25.201.205 - - [25/Jun/2023 10:55:32] 172.25.201.205: Path: '/?token=f985582a-47ae-48f6-bfb3-477dae02c369' 2023-06-25 10:55:33.131 515433 INFO nova.console.websocketproxy [req-ee483ce0-1d1a-446e-8e03-da371daa977d - - - - -] 34368: connect info: ConsoleAuthToken(access_url_base=' https://bolt.bdxworld.com:13080/vnc_auto.html ',console_type='novnc',created_at=2023-06-25T10:55:31Z,host='172.25.221.228',id=3579,instance_uuid=cdcfe890-4f15-4091-abc1-9fe0cf7eff85,internal_access_path=None,port=5900,token='***',updated_at=None) 2023-06-25 10:55:33.135 515433 INFO nova.console.websocketproxy [req-ee483ce0-1d1a-446e-8e03-da371daa977d - - - - -] 34368: connecting to: 172.25.221.228:5900 2023-06-25 10:55:33.419 515433 INFO nova.console.securityproxy.rfb [req-ee483ce0-1d1a-446e-8e03-da371daa977d - - - - -] Finished security handshake, resuming normal proxy mode using secured socket 2023-06-25 10:55:33.456 515433 INFO nova.console.websocketproxy [req-ee483ce0-1d1a-446e-8e03-da371daa977d - - - - -] handler exception: [Errno 107] Transport endpoint is not connected 2023-06-25 10:55:34.830 515436 INFO nova.console.websocketproxy [-] 172.25.201.205 - - [25/Jun/2023 10:55:34] 172.25.201.205: Plain non-SSL (ws://) WebSocket connection 2023-06-25 10:55:34.832 515436 INFO nova.console.websocketproxy [-] 172.25.201.205 - - [25/Jun/2023 10:55:34] 172.25.201.205: Path: '/?token=f985582a-47ae-48f6-bfb3-477dae02c369' 2023-06-25 10:55:35.458 515436 INFO nova.console.websocketproxy [req-c9f5d4de-a4e3-4040-9fcf-dc42d5f0b0d3 - - - - -] 34369: connect info: ConsoleAuthToken(access_url_base=' https://bolt.bdxworld.com:13080/vnc_auto.html ',console_type='novnc',created_at=2023-06-25T10:55:31Z,host='172.25.221.228',id=3579,instance_uuid=cdcfe890-4f15-4091-abc1-9fe0cf7eff85,internal_access_path=None,port=5900,token='***',updated_at=None) 2023-06-25 10:55:35.461 515436 INFO nova.console.websocketproxy [req-c9f5d4de-a4e3-4040-9fcf-dc42d5f0b0d3 - - - - -] 34369: connecting to: 172.25.221.228:5900 2023-06-25 10:55:35.753 515436 INFO nova.console.securityproxy.rfb [req-c9f5d4de-a4e3-4040-9fcf-dc42d5f0b0d3 - - - - -] Finished security handshake, resuming normal proxy mode using secured socket 2023-06-25 10:55:35.790 515436 INFO nova.console.websocketproxy [req-c9f5d4de-a4e3-4040-9fcf-dc42d5f0b0d3 - - - - -] handler exception: [Errno 107] Transport endpoint is not connected On Sun, Jun 25, 2023 at 4:20 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi, After doing a console url show after migration, I am still unable to access the console.
My site consists of 1 central site and 2 DCN sites. Consoles for central and DCN02 are working fine without any issues. But when i am creating an instance for DCN01 the console for the instance is not coming up (attached image for reference).
Today I created 3 different VM's using the same flavor, image, security group, the instances were created in the same compute host. Console was not accessible, so I shelved and unshelved all 3 instances, after which I was able to access the console for 2 of those VM's and am still unable to access the console of the 3rd VM no matter what I do.
With regards, Swogat Pradhan
On Sat, Jun 24, 2023 at 2:00 AM melanie witt <melwittt@gmail.com> wrote:
On 06/22/23 20:07, Swogat Pradhan wrote:
Hi Mel, Thank you for your response. I am facing issues with the instance console (vnc) in the openstack dashboard, Most of the time I shelve the instance and unshelve the instance to get the console. But there are some VM's I created which are not working even after shelve/unshelve.
I have used the same director to deploy a total of a central and 2 edge sites. This issue is happening on a single edge site. Cold Migration also helps in some situations.
OK, you didn't mention whether requesting a new console 'openstack console url show --vnc <server>' gets you a working console after a migration (or other event where you see the console stop working). I'm trying to determine whether the behavior you're seeing is expected or a bug. After an instance is moved to a different compute node than the one it was on when the console was started, that console is not expected to work anymore. And a new console needs to be started.
Can you give steps for reproducing the issue? Maybe that will provide more clarity.
-melwitt
On Fri, Jun 23, 2023 at 12:42 AM melanie witt <melwittt@gmail.com <mailto:melwittt@gmail.com>> wrote:
On 06/22/23 01:08, Swogat Pradhan wrote: > Hi, > Please find the below log: > [root@dcn01-hci-1 libvirt]# cat virtqemud.log > 2023-06-22 07:40:01.575+0000: 350319: error : virNetSocketReadWire:1804 > : End of file while reading data: Input/output error > 2023-06-22 07:40:01.575+0000: 350319: error : virNetSocketWriteWire:1844 > : Cannot write data: Broken pipe > > I think this is causing the problem of not getting the instance console.
When you say "instance console" are you referring to an interactive console like VNC or are you talking about the console log for the instance?
If it's the interactive console, if you have a console open and then migrate the instance, that console will not be moved along with the instance. When a user requests a console, the console proxy service establishes a connection to the compute host where the instance is located. The proxy doesn't know when an instance has been moved though, so if the instance is moved, the user will need to request a new console (which will establish a new connection to the new compute host).
Is that the behavior you are seeing?
-melwitt
> On Fri, Jun 2, 2023 at 11:27 AM Swogat Pradhan > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>> wrote: > > Update: > If the i am performing any activity like migration or resize of an > instance whose console is accessible, the console becomes > inaccessible giving out the following error : something went wrong, > connection is closed > > The was 1 other instance whose console was not accessible and i did > a shelve and unshelve and suddenly the instance console became > accessible. > > This is a peculiar behavior and i don't understand where is the issue . > > With regards, > Swogat Pradhan > > On Fri, Jun 2, 2023 at 11:19 AM Swogat Pradhan > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com
<mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>> wrote: > > Hi, > I am creating instances in my DCN site and i am unable to get > the console sometimes, error: something went wrong, connection > is closed > > I have 3 instances now running on my hci02 node and there is > console access on 1 of the vm's and the rest two i am not > getting the console, i have used the same flavor, same image > same security group for the VM's. > > Please suggest what can be done. > > With regards, > Swogat Pradhan >
On 06/25/23 03:50, Swogat Pradhan wrote:
Hi, After doing a console url show after migration, I am still unable to access the console.
My site consists of 1 central site and 2 DCN sites. Consoles for central and DCN02 are working fine without any issues. But when i am creating an instance for DCN01 the console for the instance is not coming up (attached image for reference).
Today I created 3 different VM's using the same flavor, image, security group, the instances were created in the same compute host. Console was not accessible, so I shelved and unshelved all 3 instances, after which I was able to access the console for 2 of those VM's and am still unable to access the console of the 3rd VM no matter what I do.
Apologies for the delayed reply. It sounds like there may be some type of problem with regard to the network connection from the novnc console proxy service to the DCN01 site rather than something with the console proxy itself given that things work fine with DCN02 and central. You're seeing "Cannot write data: Broken pipe" in the connection between the compute host and the console proxy showing a connection being broken after being established. As for why shelve and unshelve sometimes helps, it may be because the network port bindings are updated to inactive during the shelve and then they are updated to active during the unshelve. Something about redoing the port bindings is helping the situation sometimes. It may be worthwhile to check if there is anything different between the networks/ports the instances with working consoles have vs what the instances with non-working consoles have. -melwitt
On Sat, Jun 24, 2023 at 2:00 AM melanie witt <melwittt@gmail.com <mailto:melwittt@gmail.com>> wrote:
On 06/22/23 20:07, Swogat Pradhan wrote: > Hi Mel, > Thank you for your response. > I am facing issues with the instance console (vnc) in the openstack > dashboard, Most of the time I shelve the instance and unshelve the > instance to get the console. > But there are some VM's I created which are not working even after > shelve/unshelve. > > I have used the same director to deploy a total of a central and 2 edge > sites. > This issue is happening on a single edge site. > Cold Migration also helps in some situations.
OK, you didn't mention whether requesting a new console 'openstack console url show --vnc <server>' gets you a working console after a migration (or other event where you see the console stop working). I'm trying to determine whether the behavior you're seeing is expected or a bug. After an instance is moved to a different compute node than the one it was on when the console was started, that console is not expected to work anymore. And a new console needs to be started.
Can you give steps for reproducing the issue? Maybe that will provide more clarity.
-melwitt
> On Fri, Jun 23, 2023 at 12:42 AM melanie witt <melwittt@gmail.com <mailto:melwittt@gmail.com> > <mailto:melwittt@gmail.com <mailto:melwittt@gmail.com>>> wrote: > > On 06/22/23 01:08, Swogat Pradhan wrote: > > Hi, > > Please find the below log: > > [root@dcn01-hci-1 libvirt]# cat virtqemud.log > > 2023-06-22 07:40:01.575+0000: 350319: error : > virNetSocketReadWire:1804 > > : End of file while reading data: Input/output error > > 2023-06-22 07:40:01.575+0000: 350319: error : > virNetSocketWriteWire:1844 > > : Cannot write data: Broken pipe > > > > I think this is causing the problem of not getting the instance > console. > > When you say "instance console" are you referring to an interactive > console like VNC or are you talking about the console log for the > instance? > > If it's the interactive console, if you have a console open and then > migrate the instance, that console will not be moved along with the > instance. When a user requests a console, the console proxy service > establishes a connection to the compute host where the instance is > located. The proxy doesn't know when an instance has been moved though, > so if the instance is moved, the user will need to request a new > console > (which will establish a new connection to the new compute host). > > Is that the behavior you are seeing? > > -melwitt > > > On Fri, Jun 2, 2023 at 11:27 AM Swogat Pradhan > > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>>> wrote: > > > > Update: > > If the i am performing any activity like migration or resize > of an > > instance whose console is accessible, the console becomes > > inaccessible giving out the following error : something went > wrong, > > connection is closed > > > > The was 1 other instance whose console was not accessible and > i did > > a shelve and unshelve and suddenly the instance console became > > accessible. > > > > This is a peculiar behavior and i don't understand where is > the issue . > > > > With regards, > > Swogat Pradhan > > > > On Fri, Jun 2, 2023 at 11:19 AM Swogat Pradhan > > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>>> wrote: > > > > Hi, > > I am creating instances in my DCN site and i am unable to get > > the console sometimes, error: something went wrong, > connection > > is closed > > > > I have 3 instances now running on my hci02 node and there is > > console access on 1 of the vm's and the rest two i am not > > getting the console, i have used the same flavor, same image > > same security group for the VM's. > > > > Please suggest what can be done. > > > > With regards, > > Swogat Pradhan > > >
Hi Melanie, After searching through i found that no service is exposing port 5901. Only when I migrate the instance or do any kind of activity port 5901 is exposed on the said hypervisor node and the vm console is accessible. So, I believe your point 'Something about redoing the port bindings is helping the situation sometimes.' is correct. But how do I troubleshoot this? With regards, Swogat Pradhan On Fri, Jun 30, 2023 at 12:29 AM melanie witt <melwittt@gmail.com> wrote:
On 06/25/23 03:50, Swogat Pradhan wrote:
Hi, After doing a console url show after migration, I am still unable to access the console.
My site consists of 1 central site and 2 DCN sites. Consoles for central and DCN02 are working fine without any issues. But when i am creating an instance for DCN01 the console for the instance is not coming up (attached image for reference).
Today I created 3 different VM's using the same flavor, image, security group, the instances were created in the same compute host. Console was not accessible, so I shelved and unshelved all 3 instances, after which I was able to access the console for 2 of those VM's and am still unable to access the console of the 3rd VM no matter what I do.
Apologies for the delayed reply.
It sounds like there may be some type of problem with regard to the network connection from the novnc console proxy service to the DCN01 site rather than something with the console proxy itself given that things work fine with DCN02 and central. You're seeing "Cannot write data: Broken pipe" in the connection between the compute host and the console proxy showing a connection being broken after being established.
As for why shelve and unshelve sometimes helps, it may be because the network port bindings are updated to inactive during the shelve and then they are updated to active during the unshelve. Something about redoing the port bindings is helping the situation sometimes.
It may be worthwhile to check if there is anything different between the networks/ports the instances with working consoles have vs what the instances with non-working consoles have.
-melwitt
On Sat, Jun 24, 2023 at 2:00 AM melanie witt <melwittt@gmail.com <mailto:melwittt@gmail.com>> wrote:
On 06/22/23 20:07, Swogat Pradhan wrote: > Hi Mel, > Thank you for your response. > I am facing issues with the instance console (vnc) in the openstack > dashboard, Most of the time I shelve the instance and unshelve the > instance to get the console. > But there are some VM's I created which are not working even after > shelve/unshelve. > > I have used the same director to deploy a total of a central and 2 edge > sites. > This issue is happening on a single edge site. > Cold Migration also helps in some situations.
OK, you didn't mention whether requesting a new console 'openstack console url show --vnc <server>' gets you a working console after a migration (or other event where you see the console stop working). I'm trying to determine whether the behavior you're seeing is expected or a bug. After an instance is moved to a different compute node than the one it was on when the console was started, that console is not expected to work anymore. And a new console needs to be started.
Can you give steps for reproducing the issue? Maybe that will provide more clarity.
-melwitt
> On Fri, Jun 23, 2023 at 12:42 AM melanie witt <melwittt@gmail.com <mailto:melwittt@gmail.com> > <mailto:melwittt@gmail.com <mailto:melwittt@gmail.com>>> wrote: > > On 06/22/23 01:08, Swogat Pradhan wrote: > > Hi, > > Please find the below log: > > [root@dcn01-hci-1 libvirt]# cat virtqemud.log > > 2023-06-22 07:40:01.575+0000: 350319: error : > virNetSocketReadWire:1804 > > : End of file while reading data: Input/output error > > 2023-06-22 07:40:01.575+0000: 350319: error : > virNetSocketWriteWire:1844 > > : Cannot write data: Broken pipe > > > > I think this is causing the problem of not getting the instance > console. > > When you say "instance console" are you referring to an interactive > console like VNC or are you talking about the console log for the > instance? > > If it's the interactive console, if you have a console open and then > migrate the instance, that console will not be moved along with the > instance. When a user requests a console, the console proxy service > establishes a connection to the compute host where the instance is > located. The proxy doesn't know when an instance has been moved though, > so if the instance is moved, the user will need to request a new > console > (which will establish a new connection to the new compute host). > > Is that the behavior you are seeing? > > -melwitt > > > On Fri, Jun 2, 2023 at 11:27 AM Swogat Pradhan > > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>>> wrote: > > > > Update: > > If the i am performing any activity like migration or resize > of an > > instance whose console is accessible, the console becomes > > inaccessible giving out the following error : something went > wrong, > > connection is closed > > > > The was 1 other instance whose console was not accessible and > i did > > a shelve and unshelve and suddenly the instance console became > > accessible. > > > > This is a peculiar behavior and i don't understand where is > the issue . > > > > With regards, > > Swogat Pradhan > > > > On Fri, Jun 2, 2023 at 11:19 AM Swogat Pradhan > > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>>> wrote: > > > > Hi, > > I am creating instances in my DCN site and i am unable to get > > the console sometimes, error: something went wrong, > connection > > is closed > > > > I have 3 instances now running on my hci02 node and there is > > console access on 1 of the vm's and the rest two i am not > > getting the console, i have used the same flavor, same image > > same security group for the VM's. > > > > Please suggest what can be done. > > > > With regards, > > Swogat Pradhan > > >
Please ignore the previous email about the port information. So, I believe your point 'Something about redoing the port bindings is helping the situation sometimes.' is correct. But how do I troubleshoot this? With regards, Swogat Pradhan On Tue, Jul 11, 2023 at 11:40 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi Melanie, After searching through i found that no service is exposing port 5901. Only when I migrate the instance or do any kind of activity port 5901 is exposed on the said hypervisor node and the vm console is accessible.
So, I believe your point 'Something about redoing the port bindings is helping the situation sometimes.' is correct.
But how do I troubleshoot this?
With regards, Swogat Pradhan
On Fri, Jun 30, 2023 at 12:29 AM melanie witt <melwittt@gmail.com> wrote:
On 06/25/23 03:50, Swogat Pradhan wrote:
Hi, After doing a console url show after migration, I am still unable to access the console.
My site consists of 1 central site and 2 DCN sites. Consoles for central and DCN02 are working fine without any issues. But when i am creating an instance for DCN01 the console for the instance is not coming up (attached image for reference).
Today I created 3 different VM's using the same flavor, image, security group, the instances were created in the same compute host. Console was not accessible, so I shelved and unshelved all 3 instances, after which I was able to access the console for 2 of those VM's and am still unable to access the console of the 3rd VM no matter what I do.
Apologies for the delayed reply.
It sounds like there may be some type of problem with regard to the network connection from the novnc console proxy service to the DCN01 site rather than something with the console proxy itself given that things work fine with DCN02 and central. You're seeing "Cannot write data: Broken pipe" in the connection between the compute host and the console proxy showing a connection being broken after being established.
As for why shelve and unshelve sometimes helps, it may be because the network port bindings are updated to inactive during the shelve and then they are updated to active during the unshelve. Something about redoing the port bindings is helping the situation sometimes.
It may be worthwhile to check if there is anything different between the networks/ports the instances with working consoles have vs what the instances with non-working consoles have.
-melwitt
On Sat, Jun 24, 2023 at 2:00 AM melanie witt <melwittt@gmail.com <mailto:melwittt@gmail.com>> wrote:
On 06/22/23 20:07, Swogat Pradhan wrote: > Hi Mel, > Thank you for your response. > I am facing issues with the instance console (vnc) in the openstack > dashboard, Most of the time I shelve the instance and unshelve the > instance to get the console. > But there are some VM's I created which are not working even after > shelve/unshelve. > > I have used the same director to deploy a total of a central and 2 edge > sites. > This issue is happening on a single edge site. > Cold Migration also helps in some situations.
OK, you didn't mention whether requesting a new console 'openstack console url show --vnc <server>' gets you a working console after a migration (or other event where you see the console stop working). I'm trying to determine whether the behavior you're seeing is expected or a bug. After an instance is moved to a different compute node than the one it was on when the console was started, that console is not expected to work anymore. And a new console needs to be started.
Can you give steps for reproducing the issue? Maybe that will provide more clarity.
-melwitt
> On Fri, Jun 23, 2023 at 12:42 AM melanie witt < melwittt@gmail.com <mailto:melwittt@gmail.com> > <mailto:melwittt@gmail.com <mailto:melwittt@gmail.com>>> wrote: > > On 06/22/23 01:08, Swogat Pradhan wrote: > > Hi, > > Please find the below log: > > [root@dcn01-hci-1 libvirt]# cat virtqemud.log > > 2023-06-22 07:40:01.575+0000: 350319: error : > virNetSocketReadWire:1804 > > : End of file while reading data: Input/output error > > 2023-06-22 07:40:01.575+0000: 350319: error : > virNetSocketWriteWire:1844 > > : Cannot write data: Broken pipe > > > > I think this is causing the problem of not getting the instance > console. > > When you say "instance console" are you referring to an interactive > console like VNC or are you talking about the console log for the > instance? > > If it's the interactive console, if you have a console open and then > migrate the instance, that console will not be moved along with the > instance. When a user requests a console, the console proxy service > establishes a connection to the compute host where the instance is > located. The proxy doesn't know when an instance has been moved though, > so if the instance is moved, the user will need to request a new > console > (which will establish a new connection to the new compute host). > > Is that the behavior you are seeing? > > -melwitt > > > On Fri, Jun 2, 2023 at 11:27 AM Swogat Pradhan > > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> <mailto: swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>>> wrote: > > > > Update: > > If the i am performing any activity like migration or resize > of an > > instance whose console is accessible, the console becomes > > inaccessible giving out the following error : something went > wrong, > > connection is closed > > > > The was 1 other instance whose console was not accessible and > i did > > a shelve and unshelve and suddenly the instance console became > > accessible. > > > > This is a peculiar behavior and i don't understand where is > the issue . > > > > With regards, > > Swogat Pradhan > > > > On Fri, Jun 2, 2023 at 11:19 AM Swogat Pradhan > > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> <mailto: swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>>> wrote: > > > > Hi, > > I am creating instances in my DCN site and i am unable to get > > the console sometimes, error: something went wrong, > connection > > is closed > > > > I have 3 instances now running on my hci02 node and there is > > console access on 1 of the vm's and the rest two i am not > > getting the console, i have used the same flavor, same image > > same security group for the VM's. > > > > Please suggest what can be done. > > > > With regards, > > Swogat Pradhan > > >
I am getting the following error in nova-novncproxy.log in controller node: 2023-07-11 18:28:50.464 1080 INFO nova.console.websocketproxy [req-847a31bd-b896-4535-a4b3-6e7d163f0ec8 - - - - -] handler exception: [Errno 107] Transport endpoint is not connected On Wed, Jul 12, 2023 at 12:01 AM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Please ignore the previous email about the port information.
So, I believe your point 'Something about redoing the port bindings is helping the situation sometimes.' is correct.
But how do I troubleshoot this?
With regards, Swogat Pradhan
On Tue, Jul 11, 2023 at 11:40 PM Swogat Pradhan <swogatpradhan22@gmail.com> wrote:
Hi Melanie, After searching through i found that no service is exposing port 5901. Only when I migrate the instance or do any kind of activity port 5901 is exposed on the said hypervisor node and the vm console is accessible.
So, I believe your point 'Something about redoing the port bindings is helping the situation sometimes.' is correct.
But how do I troubleshoot this?
With regards, Swogat Pradhan
On Fri, Jun 30, 2023 at 12:29 AM melanie witt <melwittt@gmail.com> wrote:
On 06/25/23 03:50, Swogat Pradhan wrote:
Hi, After doing a console url show after migration, I am still unable to access the console.
My site consists of 1 central site and 2 DCN sites. Consoles for central and DCN02 are working fine without any issues. But when i am creating an instance for DCN01 the console for the instance is not coming up (attached image for reference).
Today I created 3 different VM's using the same flavor, image, security group, the instances were created in the same compute host. Console was not accessible, so I shelved and unshelved all 3 instances, after which I was able to access the console for 2 of those VM's and am still unable to access the console of the 3rd VM no matter what I do.
Apologies for the delayed reply.
It sounds like there may be some type of problem with regard to the network connection from the novnc console proxy service to the DCN01 site rather than something with the console proxy itself given that things work fine with DCN02 and central. You're seeing "Cannot write data: Broken pipe" in the connection between the compute host and the console proxy showing a connection being broken after being established.
As for why shelve and unshelve sometimes helps, it may be because the network port bindings are updated to inactive during the shelve and then they are updated to active during the unshelve. Something about redoing the port bindings is helping the situation sometimes.
It may be worthwhile to check if there is anything different between the networks/ports the instances with working consoles have vs what the instances with non-working consoles have.
-melwitt
On Sat, Jun 24, 2023 at 2:00 AM melanie witt <melwittt@gmail.com <mailto:melwittt@gmail.com>> wrote:
On 06/22/23 20:07, Swogat Pradhan wrote: > Hi Mel, > Thank you for your response. > I am facing issues with the instance console (vnc) in the openstack > dashboard, Most of the time I shelve the instance and unshelve the > instance to get the console. > But there are some VM's I created which are not working even after > shelve/unshelve. > > I have used the same director to deploy a total of a central and 2 edge > sites. > This issue is happening on a single edge site. > Cold Migration also helps in some situations.
OK, you didn't mention whether requesting a new console 'openstack console url show --vnc <server>' gets you a working console after a migration (or other event where you see the console stop working). I'm trying to determine whether the behavior you're seeing is expected or a bug. After an instance is moved to a different compute node than the one it was on when the console was started, that console is not expected to work anymore. And a new console needs to be started.
Can you give steps for reproducing the issue? Maybe that will provide more clarity.
-melwitt
> On Fri, Jun 23, 2023 at 12:42 AM melanie witt < melwittt@gmail.com <mailto:melwittt@gmail.com> > <mailto:melwittt@gmail.com <mailto:melwittt@gmail.com>>> wrote: > > On 06/22/23 01:08, Swogat Pradhan wrote: > > Hi, > > Please find the below log: > > [root@dcn01-hci-1 libvirt]# cat virtqemud.log > > 2023-06-22 07:40:01.575+0000: 350319: error : > virNetSocketReadWire:1804 > > : End of file while reading data: Input/output error > > 2023-06-22 07:40:01.575+0000: 350319: error : > virNetSocketWriteWire:1844 > > : Cannot write data: Broken pipe > > > > I think this is causing the problem of not getting the instance > console. > > When you say "instance console" are you referring to an interactive > console like VNC or are you talking about the console log for the > instance? > > If it's the interactive console, if you have a console open and then > migrate the instance, that console will not be moved along with the > instance. When a user requests a console, the console proxy service > establishes a connection to the compute host where the instance is > located. The proxy doesn't know when an instance has been moved though, > so if the instance is moved, the user will need to request a new > console > (which will establish a new connection to the new compute host). > > Is that the behavior you are seeing? > > -melwitt > > > On Fri, Jun 2, 2023 at 11:27 AM Swogat Pradhan > > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> <mailto: swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>>> wrote: > > > > Update: > > If the i am performing any activity like migration or resize > of an > > instance whose console is accessible, the console becomes > > inaccessible giving out the following error : something went > wrong, > > connection is closed > > > > The was 1 other instance whose console was not accessible and > i did > > a shelve and unshelve and suddenly the instance console became > > accessible. > > > > This is a peculiar behavior and i don't understand where is > the issue . > > > > With regards, > > Swogat Pradhan > > > > On Fri, Jun 2, 2023 at 11:19 AM Swogat Pradhan > > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> <mailto: swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>>> wrote: > > > > Hi, > > I am creating instances in my DCN site and i am unable to get > > the console sometimes, error: something went wrong, > connection > > is closed > > > > I have 3 instances now running on my hci02 node and there is > > console access on 1 of the vm's and the rest two i am not > > getting the console, i have used the same flavor, same image > > same security group for the VM's. > > > > Please suggest what can be done. > > > > With regards, > > Swogat Pradhan > > >
Hi Melanie, After searching through i found that no service is exposing port 5901. Only when I migrate the instance or do any kind of activity port 5901 is exposed on the said hypervisor node and the vm console is accessible.
So, I believe your point 'Something about redoing the port bindings is helping the situation sometimes.' is correct.
But how do I troubleshoot this?
On Tue, 2023-07-11 at 23:40 +0530, Swogat Pradhan wrote: that does nto really make sense to me 5901-n are used by qemu to expose the vnc console fo the vm but that has nothign to do with neutorn. each vm will have a specific vnc port in the xml if you do a virsh isntance list. that can also change on migrations form host ot host. it is allcoated by libvirt and the prot range is configureed in the libvirt qemu config the port range is contoled by remote_display_port_min and remote_display_port_max in /etc/libvirt/qemu.conf """ # Override the port for creating both VNC and SPICE sessions (min). # This defaults to 5900 and increases for consecutive sessions # or when ports are occupied, until it hits the maximum. # # Minimum must be greater than or equal to 5900 as lower number would # result into negative vnc display number. # # Maximum must be less than 65536, because higher numbers do not make # sense as a port number. # #remote_display_port_min = 5900 #remote_display_port_max = 65535 """ libvirt just increment the number form the min value for each instance on the host. you might need to check your firewall config to ensure that an approare range is openend for each host typically you do not have more then a hounded vms on a give host but a common default is to open 1024 ports to not hit the limit. tripleo opens "5900-6923" https://github.com/openstack/tripleo-heat-templates/blob/1393d39be367db3acb0... so you need to ensure that those ports are acceable in the host in the dcn site form the cental site where the nova vnc proxy is located. if you are seeing errors in virtqemud.log related ot Cannot write data: Broken pipe that might point to a differnt error related to virtlogd so you shoudl also see fi there are any related issues there. for example its a know limitation taht if virtlogd is restarted it will break the console https://bugs.launchpad.net/tripleo/+bug/1838272 https://review.opendev.org/c/openstack/puppet-tripleo/+/787771 simple ensure tripleo did not restart the virtlogd contaienr but if anythying else did it would break the console fo runing vms untill they are restart, or moved to a new host. so perhaps you are hitting something similar?
With regards, Swogat Pradhan
On Fri, Jun 30, 2023 at 12:29 AM melanie witt <melwittt@gmail.com> wrote:
On 06/25/23 03:50, Swogat Pradhan wrote:
Hi, After doing a console url show after migration, I am still unable to access the console.
My site consists of 1 central site and 2 DCN sites. Consoles for central and DCN02 are working fine without any issues. But when i am creating an instance for DCN01 the console for the instance is not coming up (attached image for reference).
Today I created 3 different VM's using the same flavor, image, security group, the instances were created in the same compute host. Console was not accessible, so I shelved and unshelved all 3 instances, after which I was able to access the console for 2 of those VM's and am still unable to access the console of the 3rd VM no matter what I do.
Apologies for the delayed reply.
It sounds like there may be some type of problem with regard to the network connection from the novnc console proxy service to the DCN01 site rather than something with the console proxy itself given that things work fine with DCN02 and central. You're seeing "Cannot write data: Broken pipe" in the connection between the compute host and the console proxy showing a connection being broken after being established.
As for why shelve and unshelve sometimes helps, it may be because the network port bindings are updated to inactive during the shelve and then they are updated to active during the unshelve. Something about redoing the port bindings is helping the situation sometimes.
It may be worthwhile to check if there is anything different between the networks/ports the instances with working consoles have vs what the instances with non-working consoles have.
-melwitt
On Sat, Jun 24, 2023 at 2:00 AM melanie witt <melwittt@gmail.com <mailto:melwittt@gmail.com>> wrote:
On 06/22/23 20:07, Swogat Pradhan wrote: > Hi Mel, > Thank you for your response. > I am facing issues with the instance console (vnc) in the openstack > dashboard, Most of the time I shelve the instance and unshelve the > instance to get the console. > But there are some VM's I created which are not working even after > shelve/unshelve. > > I have used the same director to deploy a total of a central and 2 edge > sites. > This issue is happening on a single edge site. > Cold Migration also helps in some situations.
OK, you didn't mention whether requesting a new console 'openstack console url show --vnc <server>' gets you a working console after a migration (or other event where you see the console stop working). I'm trying to determine whether the behavior you're seeing is expected or a bug. After an instance is moved to a different compute node than the one it was on when the console was started, that console is not expected to work anymore. And a new console needs to be started.
Can you give steps for reproducing the issue? Maybe that will provide more clarity.
-melwitt
> On Fri, Jun 23, 2023 at 12:42 AM melanie witt <melwittt@gmail.com <mailto:melwittt@gmail.com> > <mailto:melwittt@gmail.com <mailto:melwittt@gmail.com>>> wrote: > > On 06/22/23 01:08, Swogat Pradhan wrote: > > Hi, > > Please find the below log: > > [root@dcn01-hci-1 libvirt]# cat virtqemud.log > > 2023-06-22 07:40:01.575+0000: 350319: error : > virNetSocketReadWire:1804 > > : End of file while reading data: Input/output error > > 2023-06-22 07:40:01.575+0000: 350319: error : > virNetSocketWriteWire:1844 > > : Cannot write data: Broken pipe > > > > I think this is causing the problem of not getting the instance > console. > > When you say "instance console" are you referring to an interactive > console like VNC or are you talking about the console log for the > instance? > > If it's the interactive console, if you have a console open and then > migrate the instance, that console will not be moved along with the > instance. When a user requests a console, the console proxy service > establishes a connection to the compute host where the instance is > located. The proxy doesn't know when an instance has been moved though, > so if the instance is moved, the user will need to request a new > console > (which will establish a new connection to the new compute host). > > Is that the behavior you are seeing? > > -melwitt > > > On Fri, Jun 2, 2023 at 11:27 AM Swogat Pradhan > > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>>> wrote: > > > > Update: > > If the i am performing any activity like migration or resize > of an > > instance whose console is accessible, the console becomes > > inaccessible giving out the following error : something went > wrong, > > connection is closed > > > > The was 1 other instance whose console was not accessible and > i did > > a shelve and unshelve and suddenly the instance console became > > accessible. > > > > This is a peculiar behavior and i don't understand where is > the issue . > > > > With regards, > > Swogat Pradhan > > > > On Fri, Jun 2, 2023 at 11:19 AM Swogat Pradhan > > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>>> wrote: > > > > Hi, > > I am creating instances in my DCN site and i am unable to get > > the console sometimes, error: something went wrong, > connection > > is closed > > > > I have 3 instances now running on my hci02 node and there is > > console access on 1 of the vm's and the rest two i am not > > getting the console, i have used the same flavor, same image > > same security group for the VM's. > > > > Please suggest what can be done. > > > > With regards, > > Swogat Pradhan > > >
Hi Smooney, Please ignore that mail, please check the other emails after that. I can see other ports like 5900 and 5901 are being exposed and for the controller node i am able to telnet also to the said ports. But for some reason i am sometimes able to access the console of the vm's and sometimes not. I want to find a way to fix this issue. With regards, Swogat Pradhan On Wed, Jul 12, 2023 at 12:31 AM <smooney@redhat.com> wrote:
Hi Melanie, After searching through i found that no service is exposing port 5901. Only when I migrate the instance or do any kind of activity port 5901 is exposed on the said hypervisor node and the vm console is accessible.
So, I believe your point 'Something about redoing the port bindings is helping the situation sometimes.' is correct.
But how do I troubleshoot this?
On Tue, 2023-07-11 at 23:40 +0530, Swogat Pradhan wrote: that does nto really make sense to me 5901-n are used by qemu to expose the vnc console fo the vm but that has nothign to do with neutorn.
each vm will have a specific vnc port in the xml if you do a virsh isntance list. that can also change on migrations form host ot host.
it is allcoated by libvirt and the prot range is configureed in the libvirt qemu config
the port range is contoled by remote_display_port_min and remote_display_port_max in /etc/libvirt/qemu.conf """ # Override the port for creating both VNC and SPICE sessions (min). # This defaults to 5900 and increases for consecutive sessions # or when ports are occupied, until it hits the maximum. # # Minimum must be greater than or equal to 5900 as lower number would # result into negative vnc display number. # # Maximum must be less than 65536, because higher numbers do not make # sense as a port number. # #remote_display_port_min = 5900 #remote_display_port_max = 65535 """
libvirt just increment the number form the min value for each instance on the host.
you might need to check your firewall config to ensure that an approare range is openend for each host typically you do not have more then a hounded vms on a give host but a common default is to open 1024 ports to not hit the limit. tripleo opens "5900-6923"
https://github.com/openstack/tripleo-heat-templates/blob/1393d39be367db3acb0... so you need to ensure that those ports are acceable in the host in the dcn site form the cental site where the nova vnc proxy is located.
if you are seeing errors in virtqemud.log related ot Cannot write data: Broken pipe that might point to a differnt error related to virtlogd so you shoudl also see fi there are any related issues there. for example its a know limitation taht if virtlogd is restarted it will break the console https://bugs.launchpad.net/tripleo/+bug/1838272 https://review.opendev.org/c/openstack/puppet-tripleo/+/787771 simple ensure tripleo did not restart the virtlogd contaienr but if anythying else did it would break the console fo runing vms untill they are restart, or moved to a new host. so perhaps you are hitting something similar?
With regards, Swogat Pradhan
On Fri, Jun 30, 2023 at 12:29 AM melanie witt <melwittt@gmail.com>
wrote:
On 06/25/23 03:50, Swogat Pradhan wrote:
Hi, After doing a console url show after migration, I am still unable to access the console.
My site consists of 1 central site and 2 DCN sites. Consoles for central and DCN02 are working fine without any issues. But when i am creating an instance for DCN01 the console for the instance is not coming up (attached image for reference).
Today I created 3 different VM's using the same flavor, image,
group, the instances were created in the same compute host. Console was not accessible, so I shelved and unshelved all 3 instances, after which I was able to access the console for 2 of those VM's and am still unable to access the console of the 3rd VM no matter what I do.
Apologies for the delayed reply.
It sounds like there may be some type of problem with regard to the network connection from the novnc console proxy service to the DCN01 site rather than something with the console proxy itself given that things work fine with DCN02 and central. You're seeing "Cannot write data: Broken pipe" in the connection between the compute host and the console proxy showing a connection being broken after being established.
As for why shelve and unshelve sometimes helps, it may be because the network port bindings are updated to inactive during the shelve and
they are updated to active during the unshelve. Something about redoing the port bindings is helping the situation sometimes.
It may be worthwhile to check if there is anything different between
networks/ports the instances with working consoles have vs what the instances with non-working consoles have.
-melwitt
On Sat, Jun 24, 2023 at 2:00 AM melanie witt <melwittt@gmail.com <mailto:melwittt@gmail.com>> wrote:
On 06/22/23 20:07, Swogat Pradhan wrote: > Hi Mel, > Thank you for your response. > I am facing issues with the instance console (vnc) in the openstack > dashboard, Most of the time I shelve the instance and unshelve the > instance to get the console. > But there are some VM's I created which are not working even after > shelve/unshelve. > > I have used the same director to deploy a total of a central and 2 edge > sites. > This issue is happening on a single edge site. > Cold Migration also helps in some situations.
OK, you didn't mention whether requesting a new console 'openstack console url show --vnc <server>' gets you a working console after a migration (or other event where you see the console stop working). I'm trying to determine whether the behavior you're seeing is expected or a bug. After an instance is moved to a different compute node than
one it was on when the console was started, that console is not
expected to
work anymore. And a new console needs to be started.
Can you give steps for reproducing the issue? Maybe that will
more clarity.
-melwitt
> On Fri, Jun 23, 2023 at 12:42 AM melanie witt <
melwittt@gmail.com
<mailto:melwittt@gmail.com> > <mailto:melwittt@gmail.com <mailto:melwittt@gmail.com>>>
wrote:
> > On 06/22/23 01:08, Swogat Pradhan wrote: > > Hi, > > Please find the below log: > > [root@dcn01-hci-1 libvirt]# cat virtqemud.log > > 2023-06-22 07:40:01.575+0000: 350319: error : > virNetSocketReadWire:1804 > > : End of file while reading data: Input/output error > > 2023-06-22 07:40:01.575+0000: 350319: error : > virNetSocketWriteWire:1844 > > : Cannot write data: Broken pipe > > > > I think this is causing the problem of not getting the instance > console. > > When you say "instance console" are you referring to an interactive > console like VNC or are you talking about the console log
for the
> instance? > > If it's the interactive console, if you have a console
open
and then > migrate the instance, that console will not be moved along with the > instance. When a user requests a console, the console
security then the the provide proxy
service > establishes a connection to the compute host where the instance is > located. The proxy doesn't know when an instance has been moved though, > so if the instance is moved, the user will need to
request a new
> console > (which will establish a new connection to the new compute
host).
> > Is that the behavior you are seeing? > > -melwitt > > > On Fri, Jun 2, 2023 at 11:27 AM Swogat Pradhan > > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> <mailto:
swogatpradhan22@gmail.com
<mailto:swogatpradhan22@gmail.com>> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>>> wrote: > > > > Update: > > If the i am performing any activity like migration
or
resize > of an > > instance whose console is accessible, the console
becomes
> > inaccessible giving out the following error : something went > wrong, > > connection is closed > > > > The was 1 other instance whose console was not accessible and > i did > > a shelve and unshelve and suddenly the instance console became > > accessible. > > > > This is a peculiar behavior and i don't understand where is > the issue . > > > > With regards, > > Swogat Pradhan > > > > On Fri, Jun 2, 2023 at 11:19 AM Swogat Pradhan > > <swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> <mailto:
swogatpradhan22@gmail.com
<mailto:swogatpradhan22@gmail.com>> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com> > <mailto:swogatpradhan22@gmail.com <mailto:swogatpradhan22@gmail.com>>>> wrote: > > > > Hi, > > I am creating instances in my DCN site and i am unable to get > > the console sometimes, error: something went
wrong,
> connection > > is closed > > > > I have 3 instances now running on my hci02 node and there is > > console access on 1 of the vm's and the rest
two i
am not > > getting the console, i have used the same
flavor,
same image > > same security group for the VM's. > > > > Please suggest what can be done. > > > > With regards, > > Swogat Pradhan > > >
participants (3)
-
melanie witt
-
smooney＠redhat.com
-
Swogat Pradhan