Open Stack

Two things of note, having been doing heavy javascript development over the past couple years:

1) NPM actually resolves conflicts in a dependency tree. Unlike Python, it can ensure that if different packages declare conflicting versions, each one gets the version it requested. And conflicting dependencies are very common in JS packages. That's gonna be a nightmare for more "traditional" packaging systems if you try to shoehorn them together.

2) The OpenStack community is overwhelmingly fond of reinventing wheels. The JavaScript community has the opposite habit, and tends to create a package for every last function (that's why some of these seemingly single-purpose tools have 50+ dependencies). I caution against putting in place a system that penalizes developers for trying to use established tools or to use packages that do the job and do it well (such as forcing them to deal with xstatic packaging). While I respect the licensing and packaging concerns involved in large dependency trees, we need to stop encouraging people to reinvent wheels.

I think most folks at this point agree that the future of Horizon is to become a JavaScript-driven web app. It's the way the industry is going this point, and it's the way to satisfy what users want and expect. Provided we accept that future, we should strive to embrace the best practices of that development community, not to tell them we don't like it or it doesn't fit how OpenStack does things.

On a historical note, OpenStack had a very rocky relationship with the broader Python community in its early days. Since then, thanks to great efforts on both sides, that relationship has gotten much better. Let's try not to replay history by doing the same thing with the JavaScript/Node community.

We have to attract great developers so we can produce the best possible OpenStack. I remind people to think about how we can do *that* first and how we can deal with distro requirements to support the process second.

    - Gabriel


-----Original Message-----
From: Thomas Goirand [mailto:zigo at debian.org] 
Sent: Wednesday, November 12, 2014 5:30 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Horizon] the future of angularjs development in Horizon

On 11/12/2014 11:12 PM, Jiri Tomasek wrote:
> As Monty Taylor said, nodejs itself is not a blocker as multiple 
> versions of it should not be needed by our tools. (That's also what 
> npm and bower are taking care of, right?) Only thing that is required 
> is that all tools/js libs we want to use would eventually have to be 
> packaged. This is just a bunch of work for packagers.
> 
> Approach on using Xstatic packages vs Js tooling:
> 
> As only problem with using js tooling should be just actual packaging 
> of it, I think it makes sense to use these tools and make development 
> simpler then going other way around and using Xstatic packages - which 
> means devs would have to care about getting stuff packaged as xstatic 
> and added to the code, while maintaining proper versions and making 
> sure that they work ok together. NPM and Bower do this for us. Common 
> sense tells me packagers should take care of packaging.
> Packaging of these tools will have to get resolved somehow anyway, as 
> there will be rise in requirements of using them not just from Horizon...

I see an obvious issue with your approach.

Just like with pip and PyPi, it's going to be just one line of patch for Horizon people to add "yet-another-javascript-dependency". It's already a dependency hell. I have created 21 new python-xstatic packages in Debian, and at least, for half a dozen of them (maybe more), I also packaged the libjs-* packages. It took a long time to have them in order, and it just got stabilized for Juno (there's still something to fix in Jessie, but I don't really care as it's Icehouse there...).

Now, your motivation to go into the direction of using tooling seems to be motivated so that it's super easy to add more. And more... And more again. This leads me to believe that it's going to be even more crazy.
When is this going to stop? We're definitively enlarging the potential surface of attack and potential security breaches. And OpenStack at large is not at all doing great in terms of security. [1]

I just don't understand why all of this is needed. I did some JS programming myself back in the days (10 years ago, using PHP...). I could do Ajax by hand, without using a single library. What is it that you're doing in Horizon that needs so many libs? When I see Horizon in Juno, I don't even get it. Or is it because you just want to have fancy animation? Frankly, I don't care these. I care a lot more that we're not adding new potential security problems.

On 11/13/2014 12:18 AM, Julie Pichon wrote:
> There are instructions already on how to create xstatic packages [1], 
> it's not very complicated and just takes some review time.

Exactly!!!

And if someone can't make the effort to package something as an XStatic lib, then he doesn't deserve the rights to add a new libjs depends on the project. :)

Also, why do we need to:
1/ Change our procedure starting on each new cycle 2/ Constantly reinvent the wheel 3/ Make my life miserable... :)

On 11/13/2014 01:23 AM, Jiri Tomasek wrote:
> I might have expressed myself wrong about XStatic packages. But as you 
> say as well, to use XStatic packages, we need to most often create 
> them and maintain the correct versions we require in Horizon and they 
> don't help to packagers either.

They do! Well, except Ubuntu guys who found it funny to re-embed all the Javascript back in Horizon, but for all sane people doing packaging in this world, it's very helpful!!!

> Similar thing is npm and bower doing on the Angular side except for we 
> don't have to create these libraries as they already exist. NPM and 
> Bower are taking care of including the right versions of js libs our 
> dev env and our application needs. They use similar description files 
> as requirements.txt in Django.

How are you going to express the changes in the global-requirements.txt?
Is this just going to live within Horizon? How about projects like tuskar-ui, or murano-dashboard? There simply wont be a single unique place for me to look at to see what work I need to process. And no unique place either to have an overview and see how bad we're doing in terms of dependency (ie: we have a way too many...).

> It makes no sense not to use them and try to include js libraries 
> using XStatic packages and listing them in requirements.txt because 
> this way we don't know which version of js lib to use etc.

Not correct. XStatic package versions are following the one of upstream libjs packages. For example, python-xstatic-jsencrypt in Debian has version 2.0.0.2, and depends on version 2.0.0 of jsencrypt. If there's evolution of the dependency, then the XStatic package version will also change. I think it's very easy to follow, and a very good idea.

The thing is, using all these NodeJS stuff, we'll end-up depending on Node so much, that it's going to pull about a 100 more dependencies (or even more). Maybe it's going to be only build-depends, but that's the same from my point of view: it's going to be packages that we'll need to take care of in the distributions, while it's not needed with XStatic packages.

Last, the very good thing that XStatic packages offer for distributions, is the possibility to *not* package libjs packages separately, and just use whatever the XStatic package embeds. I did that for bootstrap 3, because I didn't want to do the packaging work for the ruby part of it (which I didn't want to care / maintain). I think that's a very nice feature, which I would like to keep. With your tooling method, this wont be possible anymore.

On 11/12/2014 09:35 PM, Monty Taylor wrote:
> That a bunch of javascript libraries will need to be distro pacakged 
> should not be a blocker (although I don't think that anyone is saying 
> it is) That is, after all, the important work the distros do. At this 
> point, given the popularity of javascript and javascript tooling, I'm 
> pretty sure the problem is going to have to be solved at some point.

There's no way the NodeJS and javascript library version and dependency hell can be solved anytime soon. Have you ever consider that we have about 90 reverse dependencies on jquery in Debian? Guess what happens if we upgrade to the latest version... Is OpenStack as a project ready to make the transitions at the same time as the distributions for these libraries? If that's the path we're choosing, I hope so. And I hope also that I wont be the only one working on these issues (just like I did with the Django 1.7 transition).

On this last release cycle, Horizon has been, by large, the most demanding project of all of OpenStack for me. I just hope this doesn't happen again, because that way, I can't focus on other important things (like ease of deployment, validating with tempest using packages, etc.).

On 11/12/2014 09:31 PM, Monty Taylor wrote:
>> jshint is NOT free software.
>> >
>> > https://github.com/jshint/jshint/blob/master/src/jshint.js#L19
> Reasonable people disagree on this point.

Feel free to have this debate with the entire Debian community. When you're done, then come back to us, and we can use jshint. In the mean while, let's not use it. (and by the way, read again the Debian Free Software Guideline and especially point number 5 and 6, I'm not saying that this is *my* point of argumentation, but it has been so within some threads in Debian).

On 11/12/2014 04:28 PM, Matthias Runge wrote:
> Looking at es5-shim, it pulls in additional 28 dependent packages,
> json3 has 12 dependencies (including a circular dependency, one 
> circular depencency in dependencies),
>
> Not counting dependencies in dependent packages, I can only suspect, 
> this will bring us at least 100 new packages required.

PLEASE DON'T DO THAT !!!
Or should I kill myself just right away now? :)

Yes, I did read the self-reply of Matthias. However, what he described above is exactly what I fear will happen if we don't take enough care.

Cheers,

Thomas Goirand (zigo)

[1] We're currently at the rate of 1 CVE per week. If you don't believe me, just count by yourself!!!


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev