Open Stack

On Tue, Apr 29, 2014 at 9:48 AM, Mark McLoughlin <markmc at redhat.com> wrote:
> Hey
>
> In this patch:
>
>   https://review.openstack.org/83681
>
> by Ghanshyam Mann, we encountered an unusual situation where a timestamp
> in the returned XML looked like this:
>
>   2014-04-08 09:00:14.399708+00:00
>
> What appeared to be unusual was that the timestamp had both sub-second
> time resolution and timezone information. It was felt that this wasn't a
> valid timestamp format and then some debate about how to 'fix' it:
>
>   https://review.openstack.org/87563
>
> Anyway, this lead me down a bit of a rabbit hole, so I'm going to
> attempt to document some findings.
>
> Firstly, some definitions:
>
>   - Python's datetime module talk about datetime objects being 'naive'
>     or 'aware'
>
>         https://docs.python.org/2.7/library/datetime.html
>
>        "A datetime object d is aware if d.tzinfo is not None and
>         d.tzinfo.utcoffset(d) does not return None. If d.tzinfo is None,
>         or if d.tzinfo is not None but d.tzinfo.utcoffset(d) returns
>         None, d is naive."
>
>     (Most people will have encountered this already, but I'm including
>     it for completeness)
>
>   - The ISO8601 time and date format specifies timestamps like this:
>
>         2014-04-29T11:37:00Z
>
>     with many variations. One distinguishing aspect of the ISO8601
>     format is the 'T' separating date and time. RFC3339 is very closely
>     related and serves as easily accessible documentation of the format:
>
>        http://www.ietf.org/rfc/rfc3339.txt
>
>   - The Python iso8601 library allows parsing this time format, but
>     also allows subtle variations that don't conform to the standard
>     like omitting the 'T' separator:
>
>       >>> import iso8601
>       >>> iso8601.parse_date('2014-04-29 11:37:00Z')
>       datetime.datetime(2014, 4, 29, 11, 37, tzinfo=<iso8601.iso8601.Utc object at 0x214b050>)
>
>     Presumably this is for the pragmatic reason that when you stringify
>     a datetime object, the resulting string uses ' ' as a separator:
>
>       >>> import datetime
>       >>> str(datetime.datetime(2014, 4, 29, 11, 37))
>       '2014-04-29 11:37:00'
>
> And now some observations on what's going on in Nova:
>
>   - We don't store timezone information in the database, but all our
>     timestamps are relative to UTC nonetheless.
>
>   - The objects code automatically adds the UTC to naive datetime
>     objects:
>
>         if value.utcoffset() is None:
>             value = value.replace(tzinfo=iso8601.iso8601.Utc())
>
>     so code that is ported to objects may now be using aware datetime
>     objects where they were previously using naive objects.
>
>   - Whether we store sub-second resolution timestamps in the database
>     appears to be database specific. In my quick tests, we store that
>     information in sqlite but not MySQL.
>
>   - However, timestamps added by SQLAlchemy when you do e.g. save() do
>     include sub-second information, so some DB API calls may return
>     sub-second timestamps even when that information isn't stored in
>     the database.
>
> In our REST APIs, you'll essentially see one of three time formats. I'm
> calling them 'isotime', 'strtime' and 'xmltime':
>
>   - 'isotime' - this is the result from timeutils.isotime(). It
>     includes timezone information (i.e. a 'Z' prefix) but not
>     microseconds. You'll see this in places where we stringify the
>     datetime objects in the API layer using isotime() before passing
>     them to the JSON/XML serializers.
>
>   - 'strtime' - this is the result from timeutils.strtime(). It doesn't
>     include timezone information but does include decimal seconds. This
>     is what jsonutils.dumps() uses when we're serializing API responses
>
>   - 'xmltime' or 'str(datetime)' format - this is just what you get
>     when you stringify a datetime using str(). If the datetime is tz
>     aware or includes non-zero microseconds, then that information will
>     be included in the result. This is a significant different versus
>     the other two formats where it is clear whether tz and microsecond
>     information is included in the string.
>
> but there are some caveats:
>
>   - I don't know how significant it is these days, but timestamps will
>     be serialized to strtime format when going over RPC, but won't be
>     de-serialized on the remote end. This could lead to a situation
>     where the API layer tries and stringify a strtime formatted string
>     using timeutils.isotime(). (see below for a description of those
>     formats)
>
>   - In at least one place - e.g. the 'updated' timestamp for v2
>     extensions - we hardcode the timestamp as strings in the code and
>     don't currently use one of the formats above.
>
>
> My conclusions from all that:
>
>   1) This sucks
>
>   2) At the very least, we should be clear in our API samples tests
>      which of the three formats we expect - we should only change the
>      format used in a given part of the API after considering any
>      compatibility considerations
>
>   3) We should unify on a single format in the v3 API - IMHO, we should
>      be explicit about use of the UTC timezone and we should avoid
>      including microseconds unless there's a clear use case. In other
>      words, we should use the 'isotime' format.

This seems like a situation where we should try take the needs of all
projects into account, so we converge on a consistent timestamp format
across APIs. For example, ceilometer's database schema was updated to
support sub-second timestamps to differentiate between events coming
in very close together
(https://bugs.launchpad.net/ceilometer/+bug/1215676). I don't know if
Marconi uses timestamps in their API, but if so that's another case
where I would expect the caller to want sub-second precision, at least
sometimes.

Doug

>
>   4) The 'xmltime' format is just a dumb historical mistake and since
>      XML support is now firmly out of favor, let's not waste time
>      improving the timestamp situation in XML.
>
>   5) We should at least consider moving to a single format in the v2
>      (JSON) API. IMHO, moving from strtime to isotime for fields like
>      created_at and updated_at would be highly unlikely to cause any
>      real issues for API users.
>
> (Following up this email with some patches that I'll link to, but I want
> to link to this email from the patches themselves)
>
> Mark.
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev