PLIP: Simplify the Resource Registry

[hvelarde]

PLIP: Simplify the Resource Registry

Responsible Persons

Proposer: @hvelarde

Seconder:

Abstract

Plone 5 modernized the CSS and JS development experience by incorporating tools like Bower, Grunt, RequireJS and LESS into it.

In Plone 5, CSS and JS resources for core Plone and add-on packages are managed in a new and completely rewritten Resource Registry that has proven buggy, difficult to maintain, and hard to understand for integrators and non-core developers.

We need to change that story by simplifying it.

Motivation

While the motivation of the rewritten was very well justified, the final implementation has shown some problems that we need to address ASAP:

• The revolutionary approach chosen (rewrite everything) has proven buggy and difficult to maintain, as the many issues reported over the last months seem to prove
• Some of the JavaScript tools used (Bower, Grunt, RequireJS…) seem to be less attractive nowadays than more modern options (Webpack…)
• Resource bundling was a workaround for a limitation of the HTTP/1.1 protocol and not a CMS feature; HTTP/2 makes it unnecessary and even undesirable
• LESS variable control panel configlet is useless in the context of the resource registry (PLIP: Theme LESS variables management #1677)

Some of these issues have discouraged integrators and add-on developers on using Plone 5.

Assumptions

Plone Framework Team has to discuss around the current toolset, and make a recommendation on its extended usage or its deprecation in favor of more modern options.

That could be difficult, given the current state of JS develoopment.

Proposal & Implementation

The proposal is to completely remove the bundling feature from the Plone core backend (probably including all the recently created ++production++ and ++unique++ namespaces) and UI (configlet).

The final result should be something like this:

• Plone core should enforce a standardized JS dependency management (currently, by using RequireJS)
• Plone should recommend integrators and add-on developers to follow embrace this, but should not enforce them to use it; the only supported mechanism should remain the one selected by the framework team and it should be very clear that integrators and add-on developers would be at their own if they fail to follow this recommendation
• Plone should provide bundled core CSS and JS resources for authenticated and anonymous users only and should not try to bundle third party resources in any way
• integrators and add-on developers are not enforce to bind resources on any way; that should be their choice
• Plone should render references to resources in the order they were registered in the registry
• Resource references could be rendered in development or production mode: development mode should render a reference to their name unchanged; production mode should render a reference using a hash (based on last time of cooking) instead, to avoid issues with intermediate proxy servers
• Registration of third party CSS and JS resources must fire automatically a cooking process to calculate the hash to be used in production mode
• Site managers and administrators must be able to cook production hashes also using an option in the configlet
• There should be a way to manually register new CSS/JS resources in the UI of the new refactored resource registry available to administrators only

The following example shows how this could work in practice:

• A user visits a Plone site as anonymous; the browser will make 2 requests to get the resources: one for CSS, another for JS
• The user logs in; the browser will make 2 additional requests to get the resources for logged-in users: one for CSS, another for JS (4 in total)
• The user installs a new add-on that includes bundled CSS and JS resources; the browser will make 2 additional requests to get those resources: one for CSS, another for JS (6 in total for authenticated users, 4 in total for anonymous users)
• The user installs another add-on that includes 2 new unbundled JS resources; the browser will make 2 additional requests to get those new resources (8 in total for authenticated users, 6 in total for anonymous users)
• so on…
• For the sake of simplicity, this example is not taking into account other resources that must require request also, like background images referenced in CSS files

From the add-on developer point of view this is how this should work:

• An add-on developer are recommended to use standardized JS dependency management (currently, by using RequireJS), but are free not to follow this recommendation or not
• Add-on developers and integrators must be aware that failing to comply with the previous recommendation is not supported and they will be on their own in case of issues
• Static resources should be available using the standard ++resource++ namespace
• Add on developers are totally responsible of taking care of the quality of their CSS and JS resources (combination, minification, etc.), the same way they are responsible for other static resources like images
• Resource registration/removal is simply an operation over the Plone registry using simplified version of the current record entry removing all unnecessary (current syntax should be available for backwards compatibility until Plone 6 is available)
• Add-on developers must have a way to cook their resources after an upgrade

This reflects more or less the current situation on Plone 4 on that aspect.

Deliverables

• A new, updated and simplified Resource Registry backend without the needless bundling feature
• A new, updated and simplified Resource Registry configlet without the needless bundling feature
• An upgrade step to transform resources already registered into the new format; this should include the plone, plone-legacy and plone logged-in bundles
• Updated documentation on how developers must install, test and uninstall CSS and JS resources
• Updated documentation on how developers should handle JS dependencies (RequireJS or similar)
• Documentation on how developers can use the selected resource bundler (Webpack or similar)
• Documentation on how resource overriden should be done

Risks

Just another change on the way we do things could discourage core and add-on developers, unless we convince them this time will be different and life is going to be easier for them.

The Plone legacy bundle must be removed and a upgrade step must be created to deal with resources already there.

Third party resources bundled in the plone and plone logged-in bundles must be decoupled from there.

What to do with the other existing bundles in Plone 5 (resourceregistry, thememapper…)? Do they still make sense? Probably just in the context of Plone core. Do we need to be able to manage more bundles in the resource registry UI? I don't know.

Currently there is no way to manage conditions per resource, do we want that back?

What about the overrides?

This PLIP is complex and has many obscure aspects that will be evident only after the work is started.

Participants

• Rodrigo Ferreira de Souza (@rodfersou)
• Your name here!

[rodfersou]

I think this is the biggest challenge we have with Plone 5 right now.

We don't need a tool that works TTW, we need a tool that just work and be as simple as possible.

Count me in!

[rodfersou]

To be simple we could use a configuration over convention trick, all addons will provide a ++resource++<addon_name>/main.min.js and ++resource++<addon_name>/main.min.css files.

These files will be generated by grunt, gulp, webpack, name your cool js bundle tool or even a simple file

[gforcada]

The more the JS tooling is evolving, the more I think that JS & CSS for Plone should be managed completely separated from Plone itself, i.e. just like we fetch a new package from pypi to get X functionality, we should just do the same with JS, go to npm and install it, bundle it as you see fit and ship it to your users via diazo (i.e. with the theme).

So in a way, as we can upload zipped theme files, if your JS tooling already generates everything, I guess is not that difficult that they generate a zip file as well, ready to upload :-)

Of course, this approach means that any TTW customization is gone, specially regarding CSS. Some good documentation, tutorials and so on could mitigate that shortcoming, although nowadays the average design firm should most probably be well versed with JS tooling to generate some CSS and JS files.

[keul]

I'll not miss the TTW customization of JS and CSS as far as we still give a simple way to do a ploneCustom.css TTW.

I also share the idea that webpack (and CSS and JS outside Plone) is probably the way to go (at least for 6 months, before JS world will find something new 😃). Let be honest: we have a new resource registry but some top Plone dev today are not using it.

But there's still something that really scare me about this approach: what about add-on development? I've the feeling that releasing Plone add-on with JavaScript and CSS (and JavaScript is every day more important) will be impossible to be done.

And let me say that Plone without add-on is dead.

[tkimnguyen]

Nicely thought out and well written. Can't argue with the motivation, though I lack the detailed understanding of the innards.

[gforcada]

@keul the add-on python part would be released to pypi and the JS/CSS part on npm, then you will have to update buildout.cfg with the add-on as well as the packages.json :-)

At the end, specially regarding JS/CSS you most probably want to customize them anyway.

[rnixx]

@hvelarde

Plone core and add-on developers should enforce a standardized JS dependency management (currently, by using RequireJS)

That's one of the most problematic points. We definitely MUST not enforce anyone doing any JS "the right way" (although I agree that Plone core JS must follow some conventions) All the AMD stuff will vanish into the air as soon as EcmaScript will provide proper dependency management OOTB. By trying to force Plone devs doing JS the one or the other way we'll end up in the same discussion in a year or two again.

The Plone JS story in the past consolidated to the availability of jQuery in Plone 4 and I think this should become the default again without the need of AMD integration simply to make Add-on migration easy while keeping B/C to Plone 4.

The biggest problem with the old RR was dependency management. Let's face it not from the technical JS POV. Dependency management was implemented by inserting resources before or after another resource by name. While this is not really reliable the initial problem would have been solved by giving resources an id and building a dependency tree.

The new RR on the other side solved the problem of technology agnostic dependency management while making assumptions about HOW you need to code JS and which tools you need to use then - which is the main cause of the big JS depression.

So retrospective it might have been better to stick to the old RR, implement proper dependency declarations and provide a way to export this dependency tree as JSON - which then can be used by bower, grunt and so on for bundling... Ok, this possibility is just gone ;) - But we can think about this approach given the new RR declarations from the registry.

@gforcada

Using npm as JS delivery system like we do with pypi for python has some going for it, anyway it also just adds a needless level of complexity to the whole story.

And - way more important - we need to recover a maintainable way for migrating Add-ons from 4 to 5. This implies the need of NOT touching any JS at all in the best case.

[hvelarde]

I've rewritten the PLIP giving more detailed information of what I think must be done.

@gforcada I'm not propossing to deploy add-ons code at two places now; that's ubercomplicated and not sane, IMO. we must continue using the current approach of installing add-ons because it works; let's change one thing at a time and wait to see how things evolve.

@keul third party add-ons will include their CSS and JS code as they do right now; the only change is we need to handle dependencies in a sane way (currently by using RequireJS) and there's no obligation for resource bundling.

@rnixx I think the problem with the new resource registry was the bundling and not the usage of RequireJS, which is not, per se, big deal.

anyway, as I said before I'm not an expert on the topic, but I think AMDification of all JS code on third party add-ons must be a precondition before moving an add-on to Plone 5; AMD-compatible JS just work OOTB in Plone 4 and is not difficult to implement.

here is an explanation of all those technologies: http://stackoverflow.com/a/16522990/644075

removal of RequireJS at this point is out of the discussion, IMO, at least for the Plone core.

when Plone 6 arrives, and ECMAScript 6 is a reality, we will not have to discuss over all of this again, but just clean up and remove all the crap.

[rnixx]

@hvelarde

I think AMDification of all JS code on third party add-ons must be a precondition before moving an add-on to Plone 5; AMD-compatible JS just work OOTB in Plone 4 and is not difficult to implement.

Right now it is a precondition, or at least it's a hack if you want to avoid it. Have you thought about JS used in Plone context but amongst others, skipping AMD deliberately? Why do you want to force ppl having to maintain another version of their JS? I have made a proposal how this can be handled here #1529 (comment) and here #1707 (comment)

when Plone 6 arrives, and ECMAScript 6 is a reality, we will not have to discuss over all of this again, but just clean up and remove all the crap.

So we integrate AMD all over the place now to remove it again in the next major release of Plone? Really? Why not make things work next to each other? Count the repos in collective and do an estimation how much work it is to migrate the repos 2 times vs. make the framework deal with the options.

problem with the new resource registry was the bundling

Why? I think the information is good, the way bundle information is resolved and bundles are delivered is bad.

[hvelarde]

@rnixx here are a couple of documents explaining why AMD is better than the module pattern:

• http://www.requirejs.org/docs/whyamd.html
• http://dojotoolkit.org/documentation/tutorials/1.10/modules/

adding a couple of lines to the JS code now (and remove them 2 years later) seems to me no big deal regardless the number of add-ons we maintain, and compared to the time I have to spend in all other things in order to make a package compatible with Plone 5.

having said that, I'm not against what you are saying, I just want to keep the scope of this PLIP limited to get rid of the resource registry bundling as it's currently broken, overrated and completely unnecessary in the short term.

you can create a new PLIP so we can discuss around JS modules if you like.

[rnixx]

@hvelarde

here are a couple of documents explaining why AMD is better

It's exactly this attitude why we ended up with the current mess.

Even if AMD is better, non AMD might be the reality of others for a good reason and this reality not changes by ignorance.

You may want to read again my linked comments about how AMD and non AMD stuff can coexist, this PLIP is exactly the right place for dealing with this issue.

[keul]

Any chance that developers that nowaday are not using RR can jump in there? @hvelarde or @datakurre IIRC?

Sorry but I share the fear of @rnixx about taking the wrong choice.. I suffer the "World if going to jQuery, we choose KSS" syndrome 😃)

[datakurre]

@keul Once I moved to webpack I kind of gave up on "activate add-on and go"-approach. And now that I battle with webpack anyway, I'm ok with any JS that webpack can chew.

[vangheem]

AMD is not just going to vanish--it's a fine method for defining modules.

• RR aimed to provide a TTW way to define javascript modules and do dependency management
• with this, having a unified way to build js means you aren't including modules multiple times and you have a way to do dependency resolution on your modules
• The major requirement that was put on @bloodbare and I was that whatever we did must be TTW compatible. If we're really okay dropping this, that makes things much easier. I know @tkimnguyen was one of the main people who pushed this though(and others).

"Resource bundling was a workaround for a limitation of the HTTP/1.1 protocol and not a CMS feature; HTTP/2 makes it unnecessary and even undesirable"

I'm not convinced of this. Modern websites bundle 100s, maybe 1000s of files together to build everything. Think of just a theme, various less/scss files, various icons--bundling puts that all together. Now think of an angular app...

Do we really think that there isn't going to be a performance impact on requesting all those resources from Plone opposed to just a couple? Maybe if all those resources are served at a proxy/CDN everything will still be alright. I hope you're right but am waiting to see how it pans out...

But I'm going to try and stay out of it from here. I'm glad there are some taking ownership of this.

[djay]

@vangheem not ok with a non TTW way to include js in your theme but the the current RR TTW compilation only seems to work sometimes. Some of the time if I want a js in my theme I just include and bypass the registries.

[seanupton]

@hvelarde

AMDification of all JS code on third party add-ons must be a precondition before moving an add-on to Plone 5;

As an add-on author, I agree with this, with two possible exceptions:

1. I still want to be able to use macro slots or viewlets to inject one-off JS (not in registry) that do not depend on anything other than jQuery (via global namespace). For example, I have a ES6-based data-viz toolkit built using webpack/babel, and I want to just include the built asset in only relevant pages/views.
2. There are still bound to be JS assets that have minimal dependencies that should just be injected as distinct <script> tags in some "legacy" configuration of some sort or another. This should not be advertised as ideal, but it it should be possible with a YMMV disclaimer in the docs. For these, I do not care about bundling, cooking (other than cache busting), dependency management, etc. I just want to put a script on all pages in the site.

That said, I am cool to force use of AMD for all JS that depends on Plone core JS libs/tools other than jQuery.

[hvelarde]

@vangheem I'm not proposing not to bundle resources from a single source (Plone core, third party add-ons…), but not to bundle the resulting bundled/unbundled resources as one huge bundle on Plone.

Plone should bundle its CSS/JS resources in 2 batches, one for anonymous users and one for authenticated ones; @plone/framework-team should decide what to use and what to recommend for third party developers, but not enforce them to use it; please, read again the example to understand what I'm proposing. the whole thing is about removing the compilation/bundling part that seems to be a collection of hacks/hooks that are not working anyway.

BTW, @bloodbare don't you have any comments on this?

@djay I think I forgot to mention that there should be an option to register resources TTW as we currently have.

I get the point from @rnixx and @seanupton that we could have to option to include pure JS without requiring its AMDfication.

thanks for your reflections, guys; I'll review this and I'll mark it as ready to see if we can fix this during Sorrento's Open Garden. also, note that I need seconders for this proposal.

[hvelarde]

I rewrote the PLIP taking into account your suggestions and, if there are no more comments, I think I've finished it.

I would like to call the @plone/framework-team to discuss it on the next meeting.

[ebrehault]

I copy here an extract of FWT meeting minutes (see https://community.plone.org/t/framework-team-meeting-minutes-2017-02-07/3418 ):

We agree the JS story might be simplified. This PLIP is about many different aspects and some are not totally clear (it assumes some complex aspects of the core JS resources management have to be mandatorily applied to any resource, which is not the case).

And most of the features described here are already existing (but lot of people are not aware).

Nevertheless we are opened to specific PLIPs or PRs if you think it could improve the situation:

• HTTP 2 is not there yet, we think we can not abandon bundles for now (maybe in 2 or 3 years), but we could imagine a PLIP to disable meta-bundling in production mode (note: it can already be done by just setting merge_with=”” on each bundle registry entry, but maybe you imagine a global setting ?)
• Ordering resources according the registry order could be done via a PR (no need for a PLIP here). It is obviously not useless for RequireJS compliant resources, but it might be useful for non-RequireJS ones.
• Almost all the points about managing individual resources apart from core bundles are already possible as far as those resources are directly declared as bundles. If the “bundle” name is confusing here, we could rename “bundle” into “resource” and “resource” into “resource item” (or “resource component”), and maybe it would help.
• It is already possible to create a resource without using RequireJS but the non-ordering might be a problem, so the previously mentioned PR might be enough.
• The simplified version of the registry records might be interesting but should be detailed in a dedicated PR.

[hvelarde]

thanks for your feedback; I'm closing this as seems to me there is no interest on addressing this issue.

[gforcada]

@hvelarde there is interest in addressing the issue, but as there are different approaches... as long as we only discuss and no one implements it can be an endless discussion, so who is up to work on it?

@hvelarde I just re-read the whole PLIP proposal and that's a really nice write up, but as you say it's already quite complex. We could do a plone.api approach here: write the docs on how things should behave (writing some more paragraphs to each of the bullets points you listed there) and then try to split that work into easier to digest PLIPs...

I would be tempted to hit the reopen button...

[hvelarde]

@gforcada thanks for your comments; I think the PLIP should be rewritten to reduce the scope in case the @plone/framework-team changes its mind.

[gforcada]

@hvelarde as it's a very complex topic we most probably have to discuss a few more times, addressing some usages are probably easier than others (integrators vs TTW-only for example). We can speed up that as long as there's interest in working on the changes :-)

[tkimnguyen]

Sounds like the sort of discussion that could go on at PLOG ... (ducks)