Open Stack

On Thu, Oct 26, 2017 at 10:23 PM, Matt Riedemann <mriedemos at gmail.com>
wrote:

> Nova has had this long-standing known performance issue if you're
> filtering a large number of instances by IP. The instance IPs are stored in
> a JSON blob in the database so we don't do filtering in SQL. We pull the
> instances out of the database, deserialize the JSON and then apply a regex
> filter match in the nova-api python code.
>
> At the Queens PTG we talked about possible ways to fix this and came up
> with this nova spec:
>
> https://specs.openstack.org/openstack/nova-specs/specs/queen
> s/approved/improve-filter-instances-by-ip-performance.html
>
> The idea is to have nova get ports from neutron and apply the IP filter in
> neutron to whittle down the ports, then from that list of ports get the
> instances to pull out of the nova database.
>
> One issue that has come up with this is neutron does not currently support
> regex filters when listing ports. There is an RFE for adding that:
>
> https://bugs.launchpad.net/neutron/+bug/1718605
>
> The proposed neutron implementation is to just do SQL LIKE substring
> matching in the database.
>
> However, one issue that has come up is that the compute API accepts a
> python regex filter and uses re.match():
>
> https://github.com/openstack/nova/blob/16.0.0/nova/compute/api.py#L2469
>
> At least one good thing about that is match() only matches from the
> beginning of the string unlike search().
>
> So for example I can filter on "192.16.*[1-5]$" if I wanted to, but that's
> not going to work with just a LIKE substring filter in SQL.
>
> The question is, does anyone actually do more than basic substring
> matching with the IP filter today? Because if we started using neutron,
> that behavior would be broken. We've never actually documented the match
> restrictions on the IP filter, but that's not a good reason to break it.
>

The use-case for us is that it helps us easily identify or find VMs which
we get any abuse reports for (or anything we see malicious traffic going
to/from).  We usually search for an *exact* match of the IP address as we
are simply trying to perform a lookup of instance ID based on the IP
address.  Regex matching isn't important in our case.


> One option is to make this configurable such that deployments which rely
> on the complicated pattern matching can just use the existing nova code
> despite performance issues. However, that's not interoperable, I hate
> config-driven API behavior, and it would mean maintaining two code paths in
> nova, which is also terrible.
>
> I was trying to think of a way to determine if the IP filter passed to
> nova is basic or a complicated pattern match and let us decide that way,
> but I'm not sure if there are good ways to detect that - maybe by simply
> looking for special characters like (, ), - and $? But then there is [] and
> we have an IPv6 filter, so that gets messy too...
>
> For now I'd just like to know if people rely on the regex match or not.
> Other ideas on how to handle this are appreciated.
>
> --
>
> Thanks,
>
> Matt
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20171026/dfe908e8/attachment.html>