[kolla][nova][neutron] Access to VMs is slow when running on a remote compute host

Giuseppe Sannino km.giuseppesannino at gmail.com
Mon Jul 15 16:34:36 UTC 2019


Hi Sean,
the ssh to localhost is slow as well.
"telnet localhost" is also slow.

/Giuseppe

On Mon, 15 Jul 2019 at 18:18, Sean Mooney <smooney at redhat.com> wrote:

> On Mon, 2019-07-15 at 16:29 +0200, Giuseppe Sannino wrote:
> > Hi!
> > first of all, thanks for the fast replies. I do appreciate that.
> >
> > I did some more test trying to figure out the issue.
> > - Set UseDNS to "no" in sshd_config  => Issue persists
> > - Installed and configured Telnet => Telnet login is slow as well
> >
> > From the "top" or "auth.log"nothing specific popped up. I can sshd taking
> > some cpu for a short while but nothing more than that.
> >
> > Once logged in the VM is not too slow. CLI doesn't get stuck or similar.
> > One thing worthwhile to mention, it seems like the writing throughput on
> > the disk is a bit slow: 67MB/s  wrt around 318MB/s of another VM running
> on
> > a "datacenter" Openstack installation.
> unless you see iowait in the guest its likely not related to the disk
> speed.
> you might be able to improve the disk performace by changeing the chache
> mode
> but unless you are seeing io wait that is just an optimisation to try
> later.
>
> when you are logged into the vm have you tried ssh again via localhost to
> determin if the long login time is related to the network or the vm.
>
> if its related to the network it will be fast over localhost
> if its related to the vm, e.g. because of disk, cpu load, memory load or
> ssh server configuration
> then the local ssh will be slow.
>
> >
> > The Cinder Volume docker is running on the Compute Host and Cinder is
> using
> > the filesystem as backend.
> >
> > BR
> > /Giuseppe
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, 12 Jul 2019 at 17:41, Slawek Kaplonski <skaplons at redhat.com>
> wrote:
> >
> > > Hi,
> > >
> > > I suspect some problems with names resolving. Can You check if You have
> > > also such delay when doing e.g. “sudo” commands after You ssh to the
> > > instance?
> > >
> > > > On 12 Jul 2019, at 16:23, Brian Haley <haleyb.dev at gmail.com> wrote:
> > > >
> > > >
> > > >
> > > > On 7/12/19 9:57 AM, Giuseppe Sannino wrote:
> > > > > Hi community,
> > > > > I need your help ,tips, advices.
> > > > > *> Environment <*
> > > > > I have deployed Openstack "Stein" using the latest kolla-ansible
> on the
> > >
> > > following deployment topology:
> > > > > 1) OS Controller running as VM on a "cloud" location
> > > > > 2) OS Compute running on a baremetal server remotely (wrt OS
> > >
> > > Controller) location
> > > > > 3) Network node running on the Compute host
> > > > > As per the above info, Controller and compute run on two different
> > >
> > > networks.
> > > > > Kolla-Ansible is not really designed for such scenario but after
> > >
> > > manipulating the globals.yml and the inventory files (basically I had
> to
> > > move node specific network settings from the globals to the inventory
> > > file), eventually the deployment works fine.
> > > > > *> Problem <*
> > > > > I have no specific issue working with this deployment except the
> > >
> > > following:
> > > > > "SSH connection to the VM is quite slow".
> > > > > It takes around 20 seconds for me to log into the VM (Ubuntu,
> CentOS,
> > >
> > > whatever).
> > > >
> > > > But once logged-in things are OK?  For example, an scp stalls the
> same
> > >
> > > way, but the transfer is fast?
> > > >
> > > > > *> Observations <*
> > > > >  * Except for the slowness during the SSH login, I don't have any
> > > > >    further specific issue working with this envirorment
> > > > >  * With the Network on the Compute I can turn the OS controller off
> > > > >    with no impact on the VM. Still the connection is slow
> > > > >  * I tried different type of images (Ubuntu, CentOS, Windows)
> always
> > > > >    with the same result.
> > > > >  * SSH connection is slow even if I try to login into the VM
> within the
> > > > >    IP Namespace
> > > > > From the ssh -vvv, I can see that the authentication gets stuck
> here:
> > > > > debug1: Authentication succeeded (publickey).
> > > > > Authenticated to *****
> > > > > debug1: channel 0: new [client-session]
> > > > > debug3: ssh_session2_open: channel_new: 0
> > > > > debug2: channel 0: send open
> > > > > debug3: send packet: type 90
> > > > > debug1: Requesting no-more-sessions at openssh.com <mailto:
> > >
> > > no-more-sessions at openssh.com>
> > > > > debug3: send packet: type 80
> > > > > debug1: Entering interactive session.
> > > > > debug1: pledge: network
> > > > > > > > > > 10 to 15 seconds later
> > > >
> > > > What is sshd doing at this time?  Have you tried enabling debug or
> > >
> > > running tcpdump when a new connection is attempted?  At first glance
> I'd
> > > say it's a DNS issue since it eventually succeeds, the logs would help
> to
> > > point in a direction.
> > > >
> > > > -Brian
> > > >
> > > >
> > > > > debug3: receive packet: type 80
> > > > > debug1: client_input_global_request: rtype hostkeys-00 at openssh.com
> > >
> > > <mailto:hostkeys-00 at openssh.com> want_reply 0
> > > > > debug3: receive packet: type 91
> > > > > debug2: callback start
> > > > > debug2: fd 3 setting TCP_NODELAY
> > > > > debug3: ssh_packet_set_tos: set IP_TOS 0x10
> > > > > debug2: client_session2_setup: id 0
> > > > > debug2: channel 0: request pty-req confirm 1
> > > > > Have you ever experienced such issue ?
> > > > > Any suggestion?
> > > > > Many thanks
> > > > > /Giuseppe
> > >
> > > —
> > > Slawek Kaplonski
> > > Senior software engineer
> > > Red Hat
> > >
> > >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190715/9c19735d/attachment-0001.html>


More information about the openstack-discuss mailing list