Open Stack

On Sun, Jun 28, 2015 at 10:37 PM, Russell Sim <russell.sim at gmail.com> wrote:

>
>
> On 29 June 2015 at 11:30, Anne Gentle <annegentle at justwriteclick.com>
> wrote:
>
>>
>> On Sat, Jun 27, 2015 at 10:43 PM, Russell Sim <russell.sim at gmail.com>
>> wrote:
>>
>>>
>>> Raw swaggerish files http://fairy-slipper.russellsim.org/ you can copy
>>> and paste the urls into the swagger UI's above to view them.
>>>
>>> I have picked up some inconsistencies in the WADL files that I have been
>>> gradually fixing these. The following 2 outstanding reviews are all that's
>>> needed to fix things well enough to make the dumb converter that I've
>>> written work.
>>> https://review.openstack.org/#/c/195364/
>>> https://review.openstack.org/#/c/196407/
>>>
>>>
>> Both merged, cool.
>>
>
> Yep, i must say that ppl are very responsive with reviews :)  many thanks
> to the docs team!
>
>
>> I keep saying Swaggerish, because the swagger output isn't actually
>>> swagger, it's modified to support multiple request/response types at the
>>> same URL.  Basically, because the OpenStack API isn't really REST. :(
>>>
>>>
>> Heh, it's a lot to document any way you look at it.
>>
>> But it's cool you got the multiple request/response worked out. An
>> example for those who are reading along is the Orchestration actions
>> resource:
>>
>> http://developer.openstack.org/api-ref-orchestration-v1.html#stack_action_suspend
>>
>>
>> http://developer.openstack.org/api-ref-orchestration-v1.html#stack_action_resume
>>
>
> The worst offender is the
> http://developer.openstack.org/api-ref-compute-v2.html#compute_server-actions
>
> But there are more subtle examples.  Tom assigned me bug
> https://bugs.launchpad.net/openstack-manuals/+bug/1338131 it's about
> confusing documentation of request and response object. So where a request
> may take several different request objects and return different response
> objects. Perhaps the keystone v3 token url should actually just be
> represented like the actions endpoint.  So one request object token per
> auth type?
>
> The code repository I'm using is https://github.com/russell/fairy-slipper
>>> but beware, the code is a disgrace.  I tried using a proper WADL parser,
>>> but I couldn't get it to work, so instead I have a horrid SAX based parser
>>> that drunkenly fumbles through the files trying really hard not to crash.
>>>
>>> What's next.
>>> - Improve the Docbok/WADL parsers, so that it picks up the methods that
>>> are currently missing.
>>>
>>
>> I want to help with this, is the best way to side-by-side compare the
>> output with the current WADL (or HTML output from WADL)?
>>
>
> Funny you should ask so there are actually a number of minor problem which
> the parser has picked up in the existing WADL.
>
> http://fairy-slipper.russellsim.org/logs/wadl_to_swagger.log
>
> There are some of false positivities, because a WADL file may be getting
> used my multiple doc book files.  I'll look at cleaning it up tonight.
>
> But the log reads like, parses json docbook description, then parses each
> referenced WADL file.
>
> 2015-06-29 02:14:23,545 /opt/fairy-slipper/tools/wadl_to_swagger.py INFO Reading API description from /var/www/html/api-ref/api-ref-orchestration-v1.json
> 2015-06-29 02:14:23,546 /opt/fairy-slipper/tools/wadl_to_swagger.py INFO Parsing /opt/api-site/api-ref/src/wadls/orchestration-api/src/v1/orchestration-api.wadl
> 2015-06-29 02:14:23,589 /opt/fairy-slipper/tools/wadl_to_swagger.py WARNING No tags for method resource_type_list
> 2015-06-29 02:14:23,589 /opt/fairy-slipper/tools/wadl_to_swagger.py WARNING No tags for method resource_type_get
> 2015-06-29 02:14:23,589 /opt/fairy-slipper/tools/wadl_to_swagger.py WARNING No tags for method resource_type_template
>
> If i were a betting man I would think that there is no documentation for
> resource type list/get/template in
> http://developer.openstack.org/api-ref-orchestration-v1.html What has
> happened is that the docbook file doesn't reference these methods.  But
> someone did actually document the methods in the WADL.
>
> Any service with extensions will probably suffer from the false positives,
> but as another simple example.
>
> 2015-06-29 02:37:42,065 /opt/fairy-slipper/tools/wadl_to_swagger.py INFO Reading API description from /var/www/html/api-ref/api-ref-image-v1.json
> 2015-06-29 02:37:42,086 /opt/fairy-slipper/tools/wadl_to_swagger.py INFO Parsing /opt/api-site/api-ref/src/wadls/image-api/src/v1/os-image-1.0.wadl
> 2015-06-29 02:37:42,110 /opt/fairy-slipper/tools/wadl_to_swagger.py WARNING No tags for method listImage-v1
>
> Sure enough there is only the list image detailed documented in http://developer.openstack.org/api-ref-image-v1.html
>
> You can ignore,  lines like  "Can't find method setUserEnabled" i have
> patches for both of those but i have only created a review for one.
> https://review.openstack.org/#/c/194859/
>
> Any method that is missing tags will not be ported into Swagger at the
> moment.
>
>
> In regards to helping with checking the conversion.  I have started
> creating ReST output http://fairy-slipper.russellsim.org/rst/ that would
> be the best thing to check against.  I would look at the html doc and
> compare it to the ReST.  That should be the easiest way to confirm that
> things are being translated correctly, return status codes and other bits
> are still missing, but the main method documentation including formatting
> is there.  I'm not converting the chapter documentation to ReST, but I'll
> get that done tonight.
>
>
>>
>>> - Convert Swaggerish into ReST/JSONSchema (started, but nowhere near
>>> complete)
>>>
>>
>> Where should the JSON Schema live, in the fairy-slipper repo for now??
>>
>
> My thoughts are that we keep everything in fairy-slipper, organised
> appropriately by service.  We can then submit reviews to get it included
> into the projects that are willing.  As it gets migrated to the services,
> we can remove it from fairy-slipper and pull it from the service API
> directly.  What that actually means in practice isn't decided, but those
> are my thoughts.
>
>
>  - Write webserver to render ReST/JSONSchema to Swaggerish (started, but
>> nowhere near complete)
>>
>> Anyone interested in this one?
>>
>
> I'm actually happy to carry the weight.  I would rather get a framework of
> a service completed before asking people to contribute.  Not that i have
> any super cool ideas, but at the same time like any programmer I'm
> massively opinionated about 'the right way' to do things.  (Like that term
> has any real meaning) But the primary concern is consistency.  I think I
> would be better if i can try to get things into a reasonably consistent
> state where we can extend in a consistent way.
>
> Also, I see that as the fun part :D
>
>
>>
>>
>>> - Update Swagger UI to include Tag documentation. (currently swagger has
>>> no understanding of the documentation at the top of each chapter.  It's
>>> included in the swaggerish files but it's not rendered in the client
>>> presently)
>>>
>>>
>> I'd like to pull out the descriptive info into RST files or some such.
>> I'm guessing it's text like what's in the Servers info here?
>> http://developer.openstack.org/api-ref-compute-v2.html#compute_servers
>>
> If it's something else, let me know.
>>
>
> Ah, glad you asked that is exactly the documentation that is included in a
> Tag.  When you view the demos above each of the API endpoints are grouped
> by Tag.  These tags don't support documentation in Swagger by default.  But
> i'm including it in the schema so we can reproduce exactly the
> documentation you mention above.  At the conference ppl mentioned narrative
> documentation, is this what that term describes?
>
> The other thing to consider is that the Tag concept is reasonably
> flexible.  We could have a method in multiple Tags?  We could develop a Tag
> hierarchy with a naming scheme like:
>
> server
> server::actions
> server::metadata
> server::actions
>
> We could then perhaps develop a better way of browsing the API using the
> Tags?
>
> I originally had really high hopes of converting the doc and pushing it
>>> into the OpenStack services, I've scoped this back a bit initially because
>>> I think it would be better to convert the current documentation into the
>>> new formats first.  We will need a way to host this on the openstack.org
>>> website which may not have access to a running OpenStack deployment.  Once
>>> I have this in a form that is acceptable, then I'll look at how this will
>>> be best integrated the documentation into each of the OpenStack services.
>>>
>>
>> Yep, I'm good with this approach. Should we patch the openstack/api-site
>> repo in the interim for testing purposes? Or just leave it in fairy-slipper
>> while we keep working?
>>
>
> My thoughts are that we keep the WADL for the time being.  Once we have
> basically a viable solution to replace the current documentation then I
> will bundle it up as a review?  The other option would be once the proof of
> concept is usable as a complete system?  At the moment things are in quite
> a state of flux.
>
> I'll update the spec at https://review.openstack.org/#/c/177934/ and
>> merge it pretty quickly as I think the approach is sound and we should
>> start dividing the work as we can.
>>
>> Thanks so much for working on this! I know we can share some tasks so it
>> doesn't feel like you're eating an elephant!
>>
>
> I realise that i'm probably coming across as quite conservative.  This is
> not really true, I am concerned about having other people work on this
> while there are no tests and there is no real structure or defined way to
> generate the files.  I guess once it's reasonably automated (gimme 1 more
> week :))  then we can probably look at how others can help extend and
> develop the functionality.  But at the moment I don't even really have the
> skeleton walking.  It's about half way there, perhaps 1/3.  But what we
> should do is plan the things that aren't yet defined.
>
> So yes I am keen to have other people contribute code, but just not with
> code this very second.
>
>
So I know it doesn't look like I've been doing anything but I have been
trying to figure out JSON schema and fairy-slipper. :) I also went ahead
and merged the spec since we have a good plan and can update it as needed
from here on.

For one task, which is creating a JSON Schema we can use to validate the
output, I think I need to study this line:
https://github.com/russell/fairy-slipper/blob/master/tools/wadl_to_swagger.py#L448

then write a JSON Schema file that is identical to the Swagger 2.0 spec
except for allowing an array for GET and POST and then moving the
resources. Is that correct? Should I do a PR to the fairy-slipper repo with
a JSON Schema file? I was thinking I'd put it in /tools then we can
validate against it if needed.

Also I attended the nova team's API meeting last week and they are excited
to help.


> I am curious about how the migration will work:
> When this is proposed as a patch will we delete the WADL files
> immediately?  or will they hang around?
>

I'd love to delete the WADL files as a patch that depends on the patch that
replaces it all. They'll be retrievable in the git log if we ever need them
"one day." I'm good with that.


> Perhaps we could use a feature branch?  That way the OPs team can have a
> chance to prep the openstack.org website for the transition and we can
> maintain the WADL files until we are ready to launch?
>

A feature branch is exactly what we can do (or the dependent branch I
mentioned above).

I was targeting developer.openstack.org for the content, and the current
job is a file copy through FTP. So to prepare developer.openstack.org I
guess we'll need the web server ready? It's currently Cloud Sites on
Rackspace if my memory serves. We'll need to get a real web server. Can you
describe the specs, as in, what would the implementation require? We'd
probably need to puppet-ize the build? Hopefully Infra-team can offer us
input.


>
> What are we going to do about the frontend?  That is an area that will
> require some thought about how to present the information.  I personally
> don't like either example above.  I don't like the current API doc, too
> much scrolling, no navigation.  jensoleg's is crap because of too much
> scrolling, also features confusing amounts of scrolling for large response
> examples.  The default swagger interface sucks because it assumes I am a
> robot?  why is the URL more important that what the URL does?  Basically i
> think we can do better.  I think this is probably the best interface for
> API documentation i have seen so far http://docs.apimatic.apiary.io/ but
> if people can suggest other sites we can draw ideas from that would be
> great.
>

Agreed on the large response examples from jensoleg's Stripe example but I
like the side-by-side (and Apiary does something similar). At Rackspace we
are working on new layouts as well, so maybe we could work together on
interfaces, let me know and I'll connect you.

Thanks!
Anne




>
> --
> Cheers,
> Russell Sim
>



-- 
Anne Gentle
Rackspace
Principal Engineer
www.justwriteclick.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-docs/attachments/20150713/47e764bb/attachment-0001.html>