Thomas, I think that's something that downstream maintainers will eventually need to seriously consider and figure out, because any modern front end service these days will end up being built with npm pieces, and various js libraries. Horizon is sadly a terrible example of a 'modern' front end service, and it will either need to eventually pivot to being a JS based front end, or replaced by one.

The mindset of excluding or avoiding npm dependencies will cripple any attempt at any serious javascript development, and given the modern web is all JS interfaces, that doesn't particularly bode well for OpenStack from a visual interface perspective if the mindset is very anti-js. :(

On 21/06/20 9:46 am, Thomas Goirand wrote:

On 6/19/20 10:03 PM, Mohammed Naser wrote:

...

Hi everyone,

I was wondering if anyone in the community has explored the idea of implementing PatternFly inside Horizon.  It seems like it shares a lot of the similar ideas that we use and we could really benefit from using a common library that already exists with a lot of good UX thought behind it.

I know it's based on React which is a bit of a leap from where Horizon is today.  However, I'd be curious if the Horizon team is interested in figuring out a plan to make a migration to something like this happen.

Personally, I think I would be able to provide resources to have someone do this work.  However, if this seems like a huge stretch and architecture change where it actually makes more sense to stand this up from scratch (and implement an architecture where the dashboard talks directly to the APIs?), perhaps we should explore that.

I also would like to hear more from our extended community too, as I think we really need to improve our user interface experience.

Thanks, Mohammed

-- Mohammed Naser VEXXHOST, Inc.

Stuff based on npm are *very* difficult to maintain on downstream distributions, because of the way apps are getting dependencies (ie: by the 100s of dependencies, each of them having to be packaged separately as separate libraries).

So, before considering any solution for the web, please consider the amount of work necessary to do the packaging. For example, I would *not* have the bandwidth to package 100s of nmp components.

These are just general remarks, I don't know any specifics about this particular library (just saw its package.json which is as horrifying (from a package maintainer perspective) as any other npm app...).

Cheers,

Thomas Goirand (zigo)

On 22/06/20 10:52 pm, Sean Mooney wrote:

On Mon, 2020-06-22 at 16:42 +1200, Adrian Turjak wrote:

...

Thomas, I think that's something that downstream maintainers will eventually need to seriously consider and figure out, because any modern front end service these days will end up being built with npm pieces, and various js libraries.

well not nessisarly you are assuming the only way to build a moderen frontend is via javascript. there are valid alternitives like dart/typescript for frontends. django is still a vlaid choice too alsthough to play devils advocate the same packaging proble that exitsts with js and npm once existed with python and pip but over time distros have packaged python dependcies. im not that famialr with npm and ho prolific the random dependencies might be any web dev i have done has been based on php and dart weher we had few dependcies most of which were free standing so i have never looked at an npm based project with lot and lots of deps, that just sounds like poor engnierring but both concerns are valid. e.g. not being able to use a modern tool set and the packaging of the same.

Dart and Typescript are still ultimately javascript when built for a web interface. :P

And Typescript is often used with the majority of projects already if building deps from npm. I doubt many people would do React without TS these days, same with Vue.

Don't get me wrong, I do a lot of Django work, and have built more than a few websites with it using their templates for html, but the web as a whole is heavily moving to dynamic JS based frontends that talk to APIs. Almost all the major CMSs are supporting headless deployments, and for good reason.

The issue with something like Horizon is that it's built on some truly amazing APIs, and something like a cloud dashboard lends itself so perfectly to being entirely dynamic js.

...

Horizon is sadly a terrible example of a 'modern' front end service, and it will either need to eventually pivot to being a JS based front end, or replaced by one.

i think part of the larger problem would be porting all the existing horizon plugins for the differnt services like manila or octavia that our outside of horizon core, also from a donw stream product perspective horizon is proably the component with the least investment and while we have a theme that is applied to give it a RedHat look and feel i dont think porting that to work with a new js based implemation would be high on our product teams backlog so if the burden was too high it could even bring into question if its worth continuting to maintain in a downstream product.

Horizon has already done one attempt at migrating to a JS framework, and that went badly because it was done as a gradual thing that never finished, but many of the plugins did start migrating to it. So there is definitely precedent for them to build plugin variants to support any new Horizon/Dashboard initiative.

As someone who does maintain a few Horizon plugins, I can't exactly say the current system is all that good either and it limits what I can achieve a hell of a lot. I'd honestly love an excuse to build them all better, but can't when the core framework behind Horizon is not ideal. My options right now are stick with Django/Horizon's framework (which I mostly have) and deal with a lot of pain based on how the application limits what kind of interfaces I actually want to build, or rebuild with AngularJS and waste my life learning a js framework that has no future. I'm sure you can see that neither option is ideal. :(

...

The mindset of excluding or avoiding npm dependencies will cripple any attempt at any serious javascript development, and given the modern web is all JS interfaces, that doesn't particularly bode well for OpenStack from a visual interface perspective if the mindset is very anti-js. :(

well to a certin degree if you are using horizon as your way to deploy application and manage them one openstack you are doing it wrong. you really should be automaing it with ansible or something else on the other hand i use horizon daliy when im doing dev because im lazy and while i know how to use the cli i often just fell like using the gui to quickly launch the test vms because my terminal are monitoring logs and i dont feel like opening yet another ssh seesion.

so dont get me wrong i am glad horizon exits but my requirements for it are basicly that its fast even on broswer that are running over ssh forwarding from a vm with no gpu accleration. it shoudl provie a clean workflow to just boot a vm and perferm simple admin task like creating flavor and image or live migration. the current version of horizon more or less does that.

In our case, we use Horizon for a public cloud, and there are a lot of other institutions that use it as a core interface as well. Yes most sane users will build the base of their infrastructure using Ansible, or Terraform, but there are many interactions that are easier and nicer via a good dashboard.

While none of us have the funding power of Google, when compared to their cloud interfaces, Horizon is awful. In some regards it's better than AWS, but only superficially, and even Digital Ocean's rather simple interface is nicer. We have a lot to learn and improve on.

For many people the first interaction they ever have with an OpenStack deployment is Horizon, and that doesn't always leave a good impression.

On 22/06/20 11:31 pm, Thomas Goirand wrote:

On 6/22/20 6:42 AM, Adrian Turjak wrote:

...

Thomas, I think that's something that downstream maintainers will eventually need to seriously consider and figure out, because any modern front end service these days will end up being built with npm pieces, and various js libraries.

That's not like if we don't want to do it, it's just that it's kind of very hard considering the amount of work that it implies.

So what would be your suggestion to fix the problem exactly?

I don't have one. :(

Then again I'm also not your target audience, because I'd never deploy Horizon from an OS deb package. I'd either build it myself into a container (which is what we do), or much like the Gitlab omnibus container, use an existing official upstream container that met my requirements if it existed, and trust their final deliverable.

Unless you are reviewing all the code yourself before packaging it (which sounds impossible), the most you can really do is follow: https://docs.npmjs.com/packages-and-modules/securing-your-code Or any other vuln scanning toolkits.

And that can be built into the build pipeline of your js blob.

My own opinion on this, is that the Javascript "ecosystem" is completely broken, and another approach should be considered than the one of blindly fetching dependencies. The current npm way doesn't feel safe at all for something to put in production, as it could be the source of security problems. How, for example, does one ensures that none of the dependency contains malicious code that is gathering browsing information? How do you ensure there's no privacy breach? How do you know none of the upstream libraries are malicious?

In Python, at least, there's a central repository, with administrator (trying to?) keeping things safe, and the chain of dependency isn't THAT insane.

Oh the js ecosystem is totally broken, but that's what we have to work with, that's where the world of front end interfaces is right now. The best we can do is work with it/around it. :(

But really pip isn't much better. The only difference is the dependencies are less convoluted. You still really want to be auditing your pip and npm libraries versions against known vulnerabilities.

I don't have a solution, but packaging each npm dep for an application that is ultimately a js blob that can be compiled anywhere feels kind of redundant to me. I can understand the need for golang or c libraries that need to be compiled locally because a binary compiled somewhere else may not work depending on host differences, but js is js, and where it is compiled doesn't matter one bit really for where it will run (the browser). You can/should audit the dependencies at build time, but I think the js blob is ultimately what should already be built somewhere else rather than on the host.

Disclaimer though, I know little to nothing about OS packaging, so take anything I say with a grain of salt, and apologies for any flagrant ignorance.

The biggest thing for me is that Horizon or any eventual future dashboard for OpenStack needs to adopt modern js standards, frameworks, and development processes to remain relevant. Having downstream maintainers as one of the reason against ever going in that direction, makes ever reaching that future much harder. :(

Hi everyone,

I decided to take some break after the initial mail in this thread to get my minds together about the current state and the future of Horizon.

I was asked about Reactjs/Angular/Vue/etc framework in Horizon several times during the past few years. My usual answer is something like: it’s great to have it but we don’t have the capacity to do it right now. We’ve got experience with Horizon angularization in the past. It led to missed some features, slower development and outdated JavaScript dependencies. I really love that effort even while I’m not a fan of AngularJS 1.x. New panels look better from a user point of view even if we missed some features we’d got in legacy panels.

Horizon’s mission is to provide an extensible unified web-based user interface for all OpenStack services. We want to have good UX for everybody and cover as many features as possible. We should care about plugins too. Unfortunately, plugins teams are missing contributors each cycle too. So we need to have as simple solution as possible.

We can’t ignore the JavaScript part of our project. It’s not an option to have some classic Django-based application with very few JavaScript dependencies. Users want to have a modern web application with good UX.

Sooner or later we will switch to some NPM-based application. It's the only way modern web applications could exist. I do understand all the reasons why operators and packagers don’t like it. Sometimes, I don't even like it either. Maybe it will be good timing to move to container-based deployment or use something like ‘nodeenv’.

The reasons why we need NPM packages it’s JavaScript development infrastructure. Any new library is packaged for NPM. You can not find any good manual on how to do something avoiding NPM at all. If we want to attract new UI developers, we have to switch to new technologies which are a standard in a world of web development.

Speaking about Horizon's future, there is something I would like to have some time. It’s only my personal opinion, so I’m open for other suggestions too.

I would like to have a Horizon project as a framework for OpenStack Web UI. It should contain some really simple mechanism to generate views for API resources using OpenstackSDK inside. It means we can easily add new pages for new resources just by adding some JSON/YAML config file to describe new entities. Such auto-generated views could be customized with HTML/CSS/JS using whatever framework we choose (React, Vue, etc). Only “edit” and “create” features should be implemented for this scenario. If you care about performance, we should support Server Side Rendering (SSR) to allow Horizon work on any device or a VM.

I don’t think Horizon can drop it’s “backend” to allow UI call APIs itself. Sometimes we have to do some preliminary processing for API responses, in other cases, it could depend on networking configuration and security reasons. We can argue if such a solution is fast enough or not but it’s something we need to have to provide everything we need for UI.

Of course, we should need to care about plugins too. Once we decide to implement Horizon Next, we have to re-implement each plugin too.

It’s a huge effort and I hope someday the community will be able to do it.

Regards, Ivan Kolodyazhny, http://blog.e0ne.info/

On Tue, Jun 23, 2020 at 4:21 PM Thomas Goirand zigo@debian.org wrote:

On 6/23/20 2:32 AM, Adrian Turjak wrote:

...

Unless you are reviewing all the code yourself before packaging it (which sounds impossible)

Since I'm packaging all of OpenStack, no, that's not possible! Though I'm hoping that everyone that's actually using a particular piece of code in OpenStack is potentially someone who could raise a security concern in this list. There's also the Debian security team which constantly reviews CVEs, and fills bugs against my packages when needed.

All of this would go away if we switch to "npm install ... and forget" kind of model.

...

the most you can really do is follow: https://docs.npmjs.com/packages-and-modules/securing-your-code Or any other vuln scanning toolkits.

The part here that says:

"Auditing package dependencies for security vulnerabilities"

would clearly not be possible with 600+ npm dependencies (like gitlab does, for example).

...

...

In Python, at least, there's a central repository, with administrator (trying to?) keeping things safe, and the chain of dependency isn't THAT insane.

Oh the js ecosystem is totally broken, but that's what we have to work with, that's where the world of front end interfaces is right now. The best we can do is work with it/around it. :(

Yeah. We absolutely *DO NOT* need to adopt all the bad practices, and the best we can do, is work around them. This includes avoiding the dependency hell by well selecting which libraries to depend on.

...

But really pip isn't much better.

In Javascript, even if a function ABI is backward compatible, there's no guarantee of what that function/method will return. The result is unnoticed brokenness at many levels.

...

You still really want to be auditing your pip and npm libraries versions against known vulnerabilities.

Yeah, just like any other language. Any given program is as weak as its weakest dependency.

...

I don't have a solution, but packaging each npm dep for an application that is ultimately a js blob that can be compiled anywhere feels kind of redundant to me. I can understand the need for golang or c libraries that need to be compiled locally because a binary compiled somewhere else may not work depending on host differences, but js is js, and where it is compiled doesn't matter one bit really for where it will run (the browser).

In this decade, mostly everyone is actually compiling JS and concatenating libraries into a huge file. In this regard, JS is no longer any different from other languages. I'd say it's comparable to Go, since every program is actually bundling (which is equivalent to statically linking) all the libraries. I'm super satisfied that for Horizon, we found a nice way to do that (ie: at package install, and using triggers whenever a JS dependency package is updated, so that it can be rebuilt).

Do you think JS are a special case? They also *must* be free software, and easily patch-able, just like any other piece of code.

...

The biggest thing for me is that Horizon or any eventual future dashboard for OpenStack needs to adopt modern js standards, frameworks,

Yes, but not at all costs.

...

and development processes to remain relevant.

If by "development processes" you are referring to the "npm install and don't care" pattern I described above, then hopefully, that's not what's going to happen.

...

Having downstream maintainers as one of the reason against ever going in that direction, makes ever reaching that future much harder. :(

Who said security and maintainability is easy? :)

Last word (and repeating to make sure my ideas are reaching): this is *NOT* a downstream distribution concern *ONLY*. :P

Cheers,

Thomas Goirand (zigo)

On 24/06/20 10:51 am, Thomas Goirand wrote:

...

If we want to attract new UI developers, we have to switch to new technologies which are a standard in a world of web development.

I'm really not convinced by that point of argumentation. It has been written in this tread multiple times, but I don't see why switching technology spawns new contributors out of nowhere.

There aren't many AngularJS devs around anymore, and the two external contractors we (Catalyst) brought in to do work with some Horizon plugins for us had a hell of a time with the codebase. Not to mention I bet very few of the existing devs who touch Horizon (myself included) want to seriously learn AngularJS given it's an EoL framework. Plus I avoid some features I want to do purely because it's so hard given the current codebase.

I've found ways to work with Horizon that make my life, and other devs working with me a little easier, but right now it's still not great, and that itself is a barrier to entry. It's very very hard to get anyone really enthusiastic about working on Horizon. :(

Though, as an operator, I really would prefer the switch from local_settings.py to a normal .ini to move forward. That's IMO more important than any internal redesign. Horizon is a real pain to configure (from both the package maintainer perspective with plugins, and from the configuration management side (puppet in my case)). Do we have any hope with this for Victoria?

I would be the first person to raise my hand and argue against that. A lot of the settings Horizon (and Django) has are nested dictionaries, or structures which do not lend themselves at all to .ini style settings files. Not to mention in our own deployment of Horizon our custom local_settings.py has logic in it, and reads most config from envvars with internal conditions for some settings based on what is read in.

I built Adjutant using Django, and while a python file as config is common for Django applications, I did opt for using .yaml (and soon toml as well) instead, with my django settings.py file pulling values from my config layer that Django needs. Horizon 'could' do something similar, but it would likely be best to do .yaml or .toml, not .ini simply because the lack of good dict support.

But of all the things to change, the config layer is the least of our problems in Horizon, and with a good erb template (or jinja with ansible) I have never had much pain with it. If it really is that painful for you, it is just python, you can easily enough make a custom local_settings.py that imports your primary settings from an .ini file of your choice/design! :P

On 24/06/20 8:34 pm, Thomas Goirand wrote:

On 6/24/20 4:40 AM, Adrian Turjak wrote:

...

But of all the things to change, the config layer is the least of our problems in Horizon, and with a good erb template (or jinja with ansible) I have never had much pain with it.

The biggest issue with a .erb template, is that it quickly gets outdated, and needs care for each new version of Horizon. I felt that pain multiple times recently.

But that happens for every service adding new config really. The pain for a total rework for Horizon's config layer would be felt by everyone, and not everyone would want it.

...

If it really is that painful for you, it is just python, you can easily enough make a custom local_settings.py that imports your primary settings from an .ini file of your choice/design! :P

A custom thing is really not what we want/need. I am using puppet-openstack (and contribute to it), and I do expect it to be able to work the same way on both CentOS and Debian/Ubuntu. So, I'd like Horizon to behave just like the rest of OpenStack.

I do understand the point about nested config. For these, probably we could keep these in separate .py files, or even yaml as you suggested, but this doesn't prevent from fixing everything that doesn't need nested config.

I like the ability to make it custom! But then again I'm used to Django configs.

What really bugs me with Horizon is the enabled files for managing panels/plugins etc. That's a nightmare to deal with depending on where you deploy it and what panels you want. These days we build one big Horizon container (all panels, all themes), and then on start up it reads our custom config and copies/deletes the enabled files you want: http://paste.openstack.org/show/795183/

We used to control plugins at build time, but that was worse. Not being able to change which plugins were enabled when starting/restarting the service was painful. With this, we can stop the container, update the config, and on the next start it will setup the panels/plugins we want. This used to be a NIGHTMARE with puppet.

Better config in general is definitely something to think about if a New Horizon ever happens, although non '.ini', because that's too limited a config format.

A rewrite is definitely the direction I think we need to go, without abandoning Horizon as it stands right now of course.

While I understand the packaging pain, I worry about points 1 and 2, because I fear that we have no real choice but to use npm, and some JS frameworks and libraries regardless of what we do. We can try and keep the list of dependencies small, but that's not something anyone can promise really. That's just the reality of modern front end development these days and we will just have to work around it.

That said, take my support or opinion of this with a grain of salt. I'm not a Horizon core dev, I do maintain Catalyst Cloud's custom plugins, and our internal fork/customisations, so I know the codebase, but mostly I'm somehow who works with Horizon and is frustrated by it, and has been vocal about it (and the 'need' for a rewrite).

Although I wish we could, Catalyst Cloud doesn't right now have the resources to put towards helping push this, but I'd still love to help out if I could. Plus the second we have any front end resources, I'd push internally to have them help with a New Horizon.

As corny as it sounds, I think the first step is admitting we have a problem, and getting a group of interested parties involved to actually design/build something better by broadcasting a call to arms of sorts.

On 26/06/20 8:24 am, Mohammed Naser wrote:

This has been a really interesting discussion but it seems like we might be hitting a few things

1. Distributions are happy in the way that Horizon is deployed today
2. Distributions don't want to have a front-end JS based thingamajig
3. Users, operators and deployers want to improve the UI experience of Horizon
4. There is so much _historic_ stuff stuffed into Horizon that

rewriting something from scratch is easier than refactoring Horizon 5. The path to improving UI inside OpenStack is "starting from scratch".

From my side, I think for the future of OpenStack, simplicity of our users and our operators that are involved here, it's time for a rewrite. I feel like Adrian is probably supportive of this too. I think it's too late to "cleanup" and the progress we can make by iterating on new technologies can get us there way quicker.

On Fri, Jun 26, 2020 at 9:28 AM Thomas Goirand zigo@debian.org wrote:

On 6/25/20 2:00 AM, Adrian Turjak wrote:

...

On 24/06/20 8:34 pm, Thomas Goirand wrote:

...

On 6/24/20 4:40 AM, Adrian Turjak wrote:

...

But of all the things to change, the config layer is the least of our problems in Horizon, and with a good erb template (or jinja with ansible) I have never had much pain with it.

The biggest issue with a .erb template, is that it quickly gets outdated, and needs care for each new version of Horizon. I felt that pain multiple times recently.

But that happens for every service adding new config really.

No. Other services using .ini have a default option embedded it the code itself (ie: that's how oslo.config works...), so if the option isn't there in the config file, it continues to work. This isn't the case with Horizon.

Wait, what? Are you sure we are talking about the same Horizon? It has defaults exactly the same way, and it literally works with an empty local_settings.py.

On Fri, Jun 26, 2020 at 12:50 PM Sean Mooney smooney@redhat.com wrote:

just as an aside if the direction of this tread is to start a parralel project to create a more moderen openstack dashboard. before selecting any framework i would suggest doing a review of several options and if that is done i would suggest that flutter https://flutter.dev/ be consider since iw toul provide a way to have a singel codebase that will work acorss mobile(ios and android), https://flutter.dev/web and eventually https://flutter.dev/desktop. react vue.js patternfly all have there merrits too but if you are going to embark on this it might be nice to create something that is accessble across many devices at the same time if it does not significantly increase the burden to do so.

Thank you for your helpful comment, however you seem to be misunderstanding the situation. Nobody is going to be doing any reviews of available frameworks or selecting them. If you want a Horizon replacement to be written in a particular framework, you can simply sit down and write it in that framework, it's that simple. Nobody is going to tell you what to use, the choice is entirely yours. Conversely, if you don't want to write the code, then telling the people who want to write it what to use is pretty pointless, especially since there doesn't seem to be any such people around.

The current Horizon developers are not going to do it. We are too busy maintaining the released versions of Horizon and trying to keep up with new features being added to OpenStack, and that situation is unlikely to change for at least several years from now, even if a replacement for Horizon with full feature parity was released tomorrow — we will still need to support the released versions of Horizon. We are happy to help with advice and explanations of how current Horizon implementation works — but we won't write the code. You have to start that project yourself or find other developers willing to work on it, who won't mind you telling them which framework to use.

Good luck.

Radomir Dopieralski wrote:

[...] Thank you for your helpful comment, however you seem to be misunderstanding the situation. Nobody is going to be doing any reviews of available frameworks or selecting them. If you want a Horizon replacement to be written in a particular framework, you can simply sit down and write it in that framework, it's that simple. Nobody is going to tell you what to use, the choice is entirely yours. Conversely, if you don't want to write the code, then telling the people who want to write it what to use is pretty pointless, especially since there doesn't seem to be any such people around.

I agree that the first step is to form a group of volunteers interested in working on such a new project. That group can discuss their scope and objectives, then which framework is the best match for their skills and goals.

Historically we have struggled to attract JS developers into the OpenStack community, our core being Python-based. We've had the same issue for years attracting developers to help improve Storyboard, after its original JS developer left (amusingly enough, after having pushed an Angular rewrite over the original Django app).

From the outside of this hypothetical team, the Horizon team says that it will continue to maintain the distro-friendly Django-based Horizon code, and that as a team it has no interest/bandwidth in driving a rewrite internally. That means a new team needs to form up (even if obviously some people can be member of both teams).

Now from the outside of this hypothetical new team, the TC can say if it supports the idea of a more-JS-native harder-to-package framework and would gladly add it the list of official OpenStack project teams as an alternative solution to Horizon. That may encourage some to step up.

On Tue, Jun 23, 2020 at 9:55 AM Jeremy Stanley fungi@yuggoth.org wrote:

On 2020-06-23 09:25:40 +0900 (+0900), Akihiro Motoki wrote: [...]

...

Regarding a topic of packaging raised by zigo, I am not sure it is a problem the OpenStack community should solve. I know legacy packagers would like to use single versions of libraries in their contents at the same time so they don't like npm/yarn based JS dependencies.

[...]

It's rather insulting to refer to distribution package maintainers as "legacy packagers" implying that distros are somehow outmoded. I'm sure you're a fan of just cramming random source into a container and crossing your fingers, but what non-distro sort of platform do you build and start that container on? Something which doesn't use "legacy packages" I suppose?

I think it's perfectly valid to want software which is mature enough to have ABI versioning and generous compatibility ranges for its dependency chain, and which sticks to a minimal set off well-established requirements rather than just grabbing whatever's shiny and new. Traditional Linux distributions and Unix derivatives understandably struggle with the chaos in programming language ecosystems like Javascript and Golang which are still very much in infancy and haven't settled into particularly mature and supportable patterns yet. This sort of fast and loose mentality may be great for prototyping new software, but the constant churn in dependencies and frameworks makes such solutions essentially unsupportable except by their upstream developers. For now you basically get to decide between being technical support for anyone who wants to use your software, or building it with languages and dependencies which can actually be supported by "legacy packagers." -- Jeremy Stanley

I am really sorry for my poor word selection. I didn't have an intention to give such negative impression. It is totally my bad.

I think I understand what distributions provide. They provide working sets of softwares verified from various perspectives. We cannot setup most deployments without distributions easily.

The point I would like to mention is why we need to avoid using the well-known dependency management systems in the JavaScript community from the beginning. I think the problem is not which system(s) we use for dependency management. What we need is to define a proper versioning strategy in the project at some point of maturity of a software.

I think we are in a dilemma between the current state of JavaScript community and supportability (packaging perspective highlights it first). Minimum set of dependencies would be great and it would help maintaining a software, but it is one of the considerations when selecting technologies. I am also worrying about the number´of developers in the dashboard. While I am not sure that adoption of recent technologies attract more developers or not, I would not block this kind of efforts to improve the GUI. 10 years passed since OpenStack started, so JavaScript world changes a lot and things are done more easily in recent technologies.

Back to packaging, I am also not sure whether the xstatic mechanism should continue. All updates in their upstreams need to be checked (manually now) and we usually tend to be behind it even if there is a security update. "Double packaging" is not efficient. Horizon upstream packages JS libraries into xstatic and distros packages it into their packages.

--amotoki

Thomas, please find a couple of comments inlined below.

On Tue, Jun 23, 2020 at 15:05 Thomas Goirand wrote: [...]

Hopefully, you understand the problem isn't specific to distributions, but also to any sysadmin which is pushing such a library bundle online.

We need to avoid NPM because it's broken by design. Not really because of NPM itself, but because of how it's used in the JS community: clarelessly.

Every language has its own management system, each being worse than the other. By far, I would consider NPM the most horrible one I even met. The result is always that:

• NPM gets multiple versions of the same library (because that's how

things are expressed)

• NPM discards the older one randomly (because only one version can be

there), and...

NPM can nest dependencies within a dependency when there are unsolvable conflicts, resulting in many node_modules directories. There are options to force a flat install, without nested node_modules, but you may get incompatible versions resulting in a broken build.

Well I've tried that without success to package the >1600 nodejs dependencies required by grafana-5.2.3 .

• the result gets compress and bundled with a tool that isn't even safe,

into a single file (which is faster to load).

Yes, I do consider nodejs-uglify as something dangerous that one should try to avoid: there's been proven security hole with it (please don't trust my words, and do your own research on it, it's easy to find, a few years ago someone even opened a thread in the debian-devel@lists.debian.org list about it).

On 6/23/20 5:51 AM, Akihiro Motoki wrote:

...

I think the problem is not which system(s) we use for dependency management. What we need is to define a proper versioning strategy in the project at some point of maturity of a software.

The dependency management tool you're going to use is going to have a huge influence on how you manage the dependencies. With NPM, one clearly decides the way of: "I don't care how or what, I just care about the shiny page result".

...

Back to packaging, I am also not sure whether the xstatic mechanism should continue. All updates in their upstreams need to be checked (manually now) and we usually tend to be behind it even if there is a security update. "Double packaging" is not efficient. Horizon upstream packages JS libraries into xstatic and distros packages it into their packages.

The effort using the XStatic thing was to be able to de-embed javascript from the rest of Horizon. I wouldn't mind changing to another system, but that system *MUST* provide the same enhancements we saw with XStatic:

1/ Enable de-embedding of JS libraries from Horizon (and plugins). 2/ Being easily fetched for testing in our (python based) gate. 3/ Artifacts can easily be replaced by the distro-specific version.

I currently don't see another way than XStatic. If you do, please share. We've discussed multiple times that it is contributor time consuming, we're all aware of it but found no other approach so far.

Could XStatic somehow works for typescript (Angular) or JSX (React) ? If I understand correctly, those frameworks require a toolchain to transpile the code to JS.

Otherwise it seems like the packaging concern you are raising may be too much of an issue as such frameworks are quite an important requirements for many modern JS applications.

Considering how entangled the JS requirements are, is it really worth it to bother with individual packages compared to a bundle produced from a lock file where a bot like dependabot can propose updates when a dependency has an known issue?

Another solution would be to avoid JS entirely and to use a safer language like Elm or PureScript which seems to have a saner approach to dependencies, but that is not an option for existing JS code base :-)

Regards, -Tristan

On 24/06/20 1:05 am, Thomas Goirand wrote:

Anyone dismissing how huge of a problem this is, isn't doing serious programming, for serious production use. That person would just be doing script kiddy work in the playground. Yes, it works, yes, it's shiny and all. The upstream code may even be super nice, well written and all. But it's *NOT* serious to put such JS bundling approach in production.

And yet people are running huge projects in production like this just fine. So clearly people are finding sane practices around it that give them enough security to feel safe that don't involve packaging each npm requirement as an OS package. How exactly are all the huge powerhouses doing it then when most of the internet's front end is giant js bundles built from npm dependencies? How does gitlab do it for their omnibus?

From a cursory glance it did seem like they did use npm, and had a rake

job to compile the js. Gitlab most definitely isn't "script kiddy work".

I'm mostly a python dev, so I don't deal with npm often. When it comes to python though, other than underlying OS packages for python/pip itself, I use pip for installing my versions (in a venv or container). I've had too many painful cases of weird OS package versions, and I dislike the idea of relying on the OS when there is a perfectly valid and working package management system for my application requirements. I can audit the versions installed against known CVEs, and because I control the requirements, I can ensure I'm never using out of date libraries.

Javascript and npm is only different because the sheer number of dependencies. Which is terrifying, don't get me wrong, but you can lock versions, you can audit them against CVEs, you can be warned if they are out of date. How other than by sheer scale is it really worse than pip if you follow some standards and a consistent process?

Yes, I do consider nodejs-uglify as something dangerous that one should try to avoid: there's been proven security hole with it (please don't trust my words, and do your own research on it, it's easy to find, a few years ago someone even opened a thread in the debian-devel@lists.debian.org list about it).

Are we talking specifically about uglify-js, or minification in general? Because there is nothing reportedly wrong with uglify-js itself that hasn't been fixed: https://snyk.io/node-js/uglify-js https://www.cvedetails.com/vulnerability-list/vendor_id-16037/Uglifyjs-Proje...

I can understand the worry about potentially not trusting a minified bundle from an external source, but if you control that minification process it's less worrying.

I currently don't see another way than XStatic. If you do, please share. We've discussed multiple times that it is contributor time consuming, we're all aware of it but found no other approach so far.

Xstatic feels like an awful approach to this problem. I get it in a way, but just the idea of using python pip packages to wrap js code, which I guess then also someone bundles into OS packages feels crazy.

On Wed, Jun 24, 2020 at 02:07:06PM +1200, Adrian Turjak wrote:

On 24/06/20 1:05 am, Thomas Goirand wrote:

...

Anyone dismissing how huge of a problem this is, isn't doing serious programming, for serious production use. That person would just be doing script kiddy work in the playground. Yes, it works, yes, it's shiny and all. The upstream code may even be super nice, well written and all. But it's *NOT* serious to put such JS bundling approach in production.

And yet people are running huge projects in production like this just fine. So clearly people are finding sane practices around it that give them enough security to feel safe that don't involve packaging each npm requirement as an OS package. How exactly are all the huge powerhouses doing it then when most of the internet's front end is giant js bundles built from npm dependencies? How does gitlab do it for their omnibus? From a cursory glance it did seem like they did use npm, and had a rake job to compile the js. Gitlab most definitely isn't "script kiddy work".

I'm mostly a python dev, so I don't deal with npm often. When it comes to python though, other than underlying OS packages for python/pip itself, I use pip for installing my versions (in a venv or container). I've had too many painful cases of weird OS package versions, and I dislike the idea of relying on the OS when there is a perfectly valid and working package management system for my application requirements. I can audit the versions installed against known CVEs, and because I control the requirements, I can ensure I'm never using out of date libraries.

Javascript and npm is only different because the sheer number of dependencies. Which is terrifying, don't get me wrong, but you can lock versions, you can audit them against CVEs, you can be warned if they are out of date. How other than by sheer scale is it really worse than pip if you follow some standards and a consistent process?

What Thomas is trying to say, and I think other people in this thread also agreed with, is that it's not "only" because of the sheer number of dependencies. My personal opinion is that the Javascript ecosystem is currently where Perl/CPAN was 25 years ago, Python was between 15 and 20 years ago, and Ruby was 10-15 years ago: quite popular, attracting many people who "just want to write a couple of lines of code to solve this simple task", and, as a very logical consequence, full of small libraries that various people developed to fix their own itches and just released out into the wild without very much thought of long-term maintenance. Now, this has several consequences (most of them have been pointed out already):

- there are many (not all, but many) developers who do not even try to keep their own libraries backwards-compatible

- there are many (not all, but many) developers who, once they have written a piece of code that uses three libraries from other people, do not really bother to follow the development of those libraries and try to make their own piece of code compatible with their new versions (this holds even more if there are not three, but fifteen libraries from other people; it can be a bit hard to keep up with them all if their authors do not care about API stability)

- there are many libraries that lock the versions of their dependencies, thus bringing back what was once known as "DLL hell", over and over and over again (and yes, this happens in other languages, too)

- there are many, many, *many* libraries that solve the same problems over and over again in subtly different ways, either because their authors were not aware of the other implementations or because said other implementations could not exactly scratch the author's itch and it was easier to write their own instead of spend some more time trying to adapt the other one and propose changes to its author (and, yes, I myself have been guilty of this in C, Perl, and Python projects in the past; NIH is a very, very easy slope to slide down along)

I *think* that, with time, many Javascript developers will realize that this situation is unsustainable in the long term, and, one by one, they will start doing what C/C++, Perl, Python, and Ruby people have been doing for some time now:

- start thinking about backwards compatibility, think really hard before making an incompatible change and, if they really have to, use something like semantic versioning (not necessarily exactly semver, but something similar) to signal the API breakage

- once the authors of the libraries they depend on start doing this, start encoding loose version requirements (not strictly pinned), such as "dep >= 1.2.1, dep < 3". This is already done in many Python packages, and OpenStack's upper-constraints machinery is a wonderful example of how this can be maintained in a conservative manner that virtually guarantees that the end result will work.

- start wondering whether it is really worth it to maintain their own pet implementation instead of extending a more-widely-used one, thus eventually having the community settle on a well-known set of more-or-less comprehensive and very widely tested packages for most tasks. Once this happens, the authors of these widely-used libraries absolutely *have* to keep some degree of backwards compatibility and some kind of reasonable versioning scheme to signal changes.

So, I'm kind of optimistic and I believe that, with time, the Javascript ecosystem will become better. Unfortunately, this process has taken many years for the other languages I've mentioned, and is not really fully complete in any of them: any module repository has its share of mostly-maintained reimplementations of various shapes and sizes of the wheel. So I guess the point of all this was mostly to explain the problem (once again) more than propose any short-term solutions :/

G'luck, Peter