<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jun 28, 2015 at 10:37 PM, Russell Sim <span dir="ltr"><<a href="mailto:russell.sim@gmail.com" target="_blank">russell.sim@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On 29 June 2015 at 11:30, Anne Gentle <span dir="ltr"><<a href="mailto:annegentle@justwriteclick.com" target="_blank">annegentle@justwriteclick.com</a>></span> wrote:<br></span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote"><span><span>On Sat, Jun 27, 2015 at 10:43 PM, Russell Sim <span dir="ltr"><<a href="mailto:russell.sim@gmail.com" target="_blank">russell.sim@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br></div></blockquote></span></span><span><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div></div><div>Raw swaggerish files <a href="http://fairy-slipper.russellsim.org/" target="_blank">http://fairy-slipper.russellsim.org/</a> you can copy and paste the urls into the swagger UI's above to view them.</div><div><br></div><div>I have picked up some inconsistencies in the WADL files that I have been gradually fixing these. The following 2 outstanding reviews are all that's needed to fix things well enough to make the dumb converter that I've written work.</div><div><a href="https://review.openstack.org/#/c/195364/" target="_blank">https://review.openstack.org/#/c/195364/</a><br></div><div><a href="https://review.openstack.org/#/c/196407/" target="_blank">https://review.openstack.org/#/c/196407/</a><br></div><div><br></div></div></div></blockquote><div><br></div></span><div>Both merged, cool.</div></span></div></div></div></blockquote><div><br></div><div>Yep, i must say that ppl are very responsive with reviews :) many thanks to the docs team!</div><span><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div></div><div>I keep saying Swaggerish, because the swagger output isn't actually swagger, it's modified to support multiple request/response types at the same URL. Basically, because the OpenStack API isn't really REST. :(</div><div><br></div></div></div></blockquote><div><br></div></span><div>Heh, it's a lot to document any way you look at it. </div><div><br></div><div>But it's cool you got the multiple request/response worked out. An example for those who are reading along is the Orchestration actions resource:</div><div><a href="http://developer.openstack.org/api-ref-orchestration-v1.html#stack_action_suspend" target="_blank">http://developer.openstack.org/api-ref-orchestration-v1.html#stack_action_suspend</a> <br></div><div><a href="http://developer.openstack.org/api-ref-orchestration-v1.html#stack_action_resume" target="_blank">http://developer.openstack.org/api-ref-orchestration-v1.html#stack_action_resume</a></div></div></div></div></blockquote><div><br></div></span><div>The worst offender is the <a href="http://developer.openstack.org/api-ref-compute-v2.html#compute_server-actions" target="_blank">http://developer.openstack.org/api-ref-compute-v2.html#compute_server-actions</a></div><div><br></div><div>But there are more subtle examples. Tom assigned me bug <a href="https://bugs.launchpad.net/openstack-manuals/+bug/1338131" target="_blank">https://bugs.launchpad.net/openstack-manuals/+bug/1338131</a> it's about confusing documentation of request and response object. So where a request may take several different request objects and return different response objects. Perhaps the keystone v3 token url should actually just be represented like the actions endpoint. So one request object token per auth type?</div><span><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div></div><div>The code repository I'm using is <a href="https://github.com/russell/fairy-slipper" target="_blank">https://github.com/russell/fairy-slipper</a> but beware, the code is a disgrace. I tried using a proper WADL parser, but I couldn't get it to work, so instead I have a horrid SAX based parser that drunkenly fumbles through the files trying really hard not to crash.</div><div><br></div><div>What's next.</div><div>- Improve the Docbok/WADL parsers, so that it picks up the methods that are currently missing.</div></div></div></blockquote><div><br></div></span><div>I want to help with this, is the best way to side-by-side compare the output with the current WADL (or HTML output from WADL)? </div></div></div></div></blockquote><div><br></div></span><div>Funny you should ask so there are actually a number of minor problem which the parser has picked up in the existing WADL.</div><div><br></div><div><a href="http://fairy-slipper.russellsim.org/logs/wadl_to_swagger.log" target="_blank">http://fairy-slipper.russellsim.org/logs/wadl_to_swagger.log</a> <br></div><div><br></div><div>There are some of false positivities, because a WADL file may be getting used my multiple doc book files. I'll look at cleaning it up tonight.</div><div><br></div><div>But the log reads like, parses json docbook description, then parses each referenced WADL file.</div><div><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">2015-06-29 02:14:23,545 /opt/fairy-slipper/tools/wadl_to_swagger.py INFO Reading API description from /var/www/html/api-ref/api-ref-orchestration-v1.json
2015-06-29 02:14:23,546 /opt/fairy-slipper/tools/wadl_to_swagger.py INFO Parsing /opt/api-site/api-ref/src/wadls/orchestration-api/src/v1/orchestration-api.wadl
2015-06-29 02:14:23,589 /opt/fairy-slipper/tools/wadl_to_swagger.py WARNING No tags for method resource_type_list
2015-06-29 02:14:23,589 /opt/fairy-slipper/tools/wadl_to_swagger.py WARNING No tags for method resource_type_get
2015-06-29 02:14:23,589 /opt/fairy-slipper/tools/wadl_to_swagger.py WARNING No tags for method resource_type_template</pre></div><div>If i were a betting man I would think that there is no documentation for resource type list/get/template in <a href="http://developer.openstack.org/api-ref-orchestration-v1.html" target="_blank">http://developer.openstack.org/api-ref-orchestration-v1.html</a> What has happened is that the docbook file doesn't reference these methods. But someone did actually document the methods in the WADL.</div><div><br></div><div>Any service with extensions will probably suffer from the false positives, but as another simple example.</div><div><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">2015-06-29 02:37:42,065 /opt/fairy-slipper/tools/wadl_to_swagger.py INFO Reading API description from /var/www/html/api-ref/api-ref-image-v1.json
2015-06-29 02:37:42,086 /opt/fairy-slipper/tools/wadl_to_swagger.py INFO Parsing /opt/api-site/api-ref/src/wadls/image-api/src/v1/os-image-1.0.wadl
2015-06-29 02:37:42,110 /opt/fairy-slipper/tools/wadl_to_swagger.py WARNING No tags for method listImage-v1</pre><pre style="word-wrap:break-word"><font color="#000000" face="arial, helvetica, sans-serif"><span style="white-space:pre-wrap">Sure enough there is only the list image detailed documented in <a href="http://developer.openstack.org/api-ref-image-v1.html" target="_blank">http://developer.openstack.org/api-ref-image-v1.html</a></span></font></pre></div><div>You can ignore, lines like <font color="#000000"><span style="white-space:pre-wrap"> "Can't find method setUserEnabled" i have patches for both of those but i have only created a review for one. <a href="https://review.openstack.org/#/c/194859/" target="_blank">https://review.openstack.org/#/c/194859/</a> </span></font></div><div><br></div><div>Any method that is missing tags will not be ported into Swagger at the moment.</div><div><br></div><div><br></div><div>In regards to helping with checking the conversion. I have started creating ReST output <a href="http://fairy-slipper.russellsim.org/rst/" target="_blank">http://fairy-slipper.russellsim.org/rst/</a> that would be the best thing to check against. I would look at the html doc and compare it to the ReST. That should be the easiest way to confirm that things are being translated correctly, return status codes and other bits are still missing, but the main method documentation including formatting is there. I'm not converting the chapter documentation to ReST, but I'll get that done tonight.</div><span><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div>- Convert Swaggerish into ReST/JSONSchema (started, but nowhere near complete)</div></div></div></blockquote><div><br></div></span><div>Where should the JSON Schema live, in the fairy-slipper repo for now??</div></div></div></div></blockquote><div><br></div></span><div>My thoughts are that we keep everything in fairy-slipper, organised appropriately by service. We can then submit reviews to get it included into the projects that are willing. As it gets migrated to the services, we can remove it from fairy-slipper and pull it from the service API directly. What that actually means in practice isn't decided, but those are my thoughts.</div><span><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><div> - Write webserver to render ReST/JSONSchema to Swaggerish (started, but nowhere near complete)</div><div><br></div></span><div>Anyone interested in this one?</div></div></div></div></blockquote><div><br></div></span><div>I'm actually happy to carry the weight. I would rather get a framework of a service completed before asking people to contribute. Not that i have any super cool ideas, but at the same time like any programmer I'm massively opinionated about 'the right way' to do things. (Like that term has any real meaning) But the primary concern is consistency. I think I would be better if i can try to get things into a reasonably consistent state where we can extend in a consistent way.</div><div><br></div><div>Also, I see that as the fun part :D</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div>- Update Swagger UI to include Tag documentation. (currently swagger has no understanding of the documentation at the top of each chapter. It's included in the swaggerish files but it's not rendered in the client presently)</div><div><br></div></div></div></blockquote><div><br></div></span><div>I'd like to pull out the descriptive info into RST files or some such. I'm guessing it's text like what's in the Servers info here?</div><div><a href="http://developer.openstack.org/api-ref-compute-v2.html#compute_servers" target="_blank">http://developer.openstack.org/api-ref-compute-v2.html#compute_servers</a></div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>If it's something else, let me know. </div></div></div></div></blockquote><div> </div></span><div>Ah, glad you asked that is exactly the documentation that is included in a Tag. When you view the demos above each of the API endpoints are grouped by Tag. These tags don't support documentation in Swagger by default. But i'm including it in the schema so we can reproduce exactly the documentation you mention above. At the conference ppl mentioned narrative documentation, is this what that term describes? <br></div><div><br></div><div>The other thing to consider is that the Tag concept is reasonably flexible. We could have a method in multiple Tags? We could develop a Tag hierarchy with a naming scheme like:</div><div><br></div><div>server</div><div>server::actions</div><div>server::metadata</div><div>server::actions</div><div><br></div><div>We could then perhaps develop a better way of browsing the API using the Tags?</div><span><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div>I originally had really high hopes of converting the doc and pushing it into the OpenStack services, I've scoped this back a bit initially because I think it would be better to convert the current documentation into the new formats first. We will need a way to host this on the <a href="http://openstack.org" target="_blank">openstack.org</a> website which may not have access to a running OpenStack deployment. Once I have this in a form that is acceptable, then I'll look at how this will be best integrated the documentation into each of the OpenStack services.</div></div></div></blockquote><div><br></div></span><div>Yep, I'm good with this approach. Should we patch the openstack/api-site repo in the interim for testing purposes? Or just leave it in fairy-slipper while we keep working? </div></div></div></div></blockquote><div><br></div></span><div>My thoughts are that we keep the WADL for the time being. Once we have basically a viable solution to replace the current documentation then I will bundle it up as a review? The other option would be once the proof of concept is usable as a complete system? At the moment things are in quite a state of flux.</div><span><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>I'll update the spec at <a href="https://review.openstack.org/#/c/177934/" target="_blank">https://review.openstack.org/#/c/177934/</a> and merge it pretty quickly as I think the approach is sound and we should start dividing the work as we can.<br></div><div><br></div><div>Thanks so much for working on this! I know we can share some tasks so it doesn't feel like you're eating an elephant!</div></div></div></div></blockquote><div> </div></span><div>I realise that i'm probably coming across as quite conservative. This is not really true, I am concerned about having other people work on this while there are no tests and there is no real structure or defined way to generate the files. I guess once it's reasonably automated (gimme 1 more week :)) then we can probably look at how others can help extend and develop the functionality. But at the moment I don't even really have the skeleton walking. It's about half way there, perhaps 1/3. But what we should do is plan the things that aren't yet defined. </div><div><br></div><div>So yes I am keen to have other people contribute code, but just not with code this very second.</div><div><br></div></div></div></div></blockquote><div><br></div><div><div>So I know it doesn't look like I've been doing anything but I have been trying to figure out JSON schema and fairy-slipper. :) I also went ahead and merged the spec since we have a good plan and can update it as needed from here on.</div><div><br></div><div>For one task, which is creating a JSON Schema we can use to validate the output, I think I need to study this line:</div><div><a href="https://github.com/russell/fairy-slipper/blob/master/tools/wadl_to_swagger.py#L448" target="_blank">https://github.com/russell/fairy-slipper/blob/master/tools/wadl_to_swagger.py#L448</a><br></div><div><br></div><div>then write a JSON Schema file that is identical to the Swagger 2.0 spec except for allowing an array for GET and POST and then moving the resources. Is that correct? Should I do a PR to the fairy-slipper repo with a JSON Schema file? I was thinking I'd put it in /tools then we can validate against it if needed.</div><div><br></div><div>Also I attended the nova team's API meeting last week and they are excited to help. </div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div>I am curious about how the migration will work:</div><div>When this is proposed as a patch will we delete the WADL files immediately? or will they hang around? </div></div></div></div></blockquote><div><br></div><div>I'd love to delete the WADL files as a patch that depends on the patch that replaces it all. They'll be retrievable in the git log if we ever need them "one day." I'm good with that.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Perhaps we could use a feature branch? That way the OPs team can have a chance to prep the <a href="http://openstack.org" target="_blank">openstack.org</a> website for the transition and we can maintain the WADL files until we are ready to launch?</div></div></div></div></blockquote><div><br></div><div>A feature branch is exactly what we can do (or the dependent branch I mentioned above). </div><div><br></div><div>I was targeting <a href="http://developer.openstack.org" target="_blank">developer.openstack.org</a> for the content, and the current job is a file copy through FTP. So to prepare <a href="http://developer.openstack.org" target="_blank">developer.openstack.org</a> I guess we'll need the web server ready? It's currently Cloud Sites on Rackspace if my memory serves. We'll need to get a real web server. Can you describe the specs, as in, what would the implementation require? We'd probably need to puppet-ize the build? Hopefully Infra-team can offer us input.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>What are we going to do about the frontend? That is an area that will require some thought about how to present the information. I personally don't like either example above. I don't like the current API doc, too much scrolling, no navigation. jensoleg's is crap because of too much scrolling, also features confusing amounts of scrolling for large response examples. The default swagger interface sucks because it assumes I am a robot? why is the URL more important that what the URL does? Basically i think we can do better. I think this is probably the best interface for API documentation i have seen so far <a href="http://docs.apimatic.apiary.io/" target="_blank">http://docs.apimatic.apiary.io/</a> but if people can suggest other sites we can draw ideas from that would be great.</div></div></div></div></blockquote><div><br></div><div>Agreed on the large response examples from jensoleg's Stripe example but I like the side-by-side (and Apiary does something similar). At Rackspace we are working on new layouts as well, so maybe we could work together on interfaces, let me know and I'll connect you.</div><div><br></div><div>Thanks!</div><div>Anne</div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><span><font color="#888888"><div><br></div>-- <br><div><div dir="ltr">Cheers,<br>Russell Sim</div></div>
</font></span></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr"><div>Anne Gentle</div><div>Rackspace</div><div>Principal Engineer</div><div><a href="http://www.justwriteclick.com" target="_blank">www.justwriteclick.com</a></div></div></div>
</div></div>