J3.10.0 gives false positives for J3 extensions compatibility with j4

[rogercreagh]

Steps to reproduce the issue

install an old j3 extension which uses the update system but does not have a j4 compatible update element specified.
(ie only <targetplatform name="joomla" version="3.*"/> is present.

Expected result

extension should be reported as either incompatible or unknown

Actual result

The checker reports that the extension is compatible:

"The extension developer states that the currently installed version is compatible."

when in fact the extension developer states no such thing.

System information (as much as possible)

J3.10.0 alpha 3

Additional comments

The J4/3.10 release news states:
"Once your website is updated to 3.10, the pre-update checker will allow you to check the compatibility with the Joomla 4.0 minimum requirements in terms of the current hosting environment and, more importantly, your installed extensions. This will help you migrate to Joomla 4.0 smoothly."

This is not the case. It is giving false positives.

It is illogical that a well formed j3 extension needs to be modified before it will be correctly reported as incompatible.

If there is no mention of j4 on the update xml then the extension cannot be safely reported as j4 compatible.

[zero-24]

Thanks for reporting can you send us an example extension so we can try to debug the root cause? The expected behavir would be indeed to show that extension under the "incompatible or unknown" section.

[rogercreagh]

OK I have identified the issue. The problem arises where the update xml file identifies the target platform as 3.*. Assuming the installed version is up-to-date as per the update xml, changing the target platform to 3.9 changes the listing to update information unavailable, whereas changing it to 3.10 makes it No update required.

Surely this is illogical,
If the update xml specifies the target platform as 3.* or 3.10 then that is NOT saying that the extension is J4 compatible.

Only if there is an update entry specifying target platform as 4.x should the extension be identified as compatible. If there are update entries for both 3.10 and 4.x then the developer is saying that the extension is J4 compatible - there is a j3 version and a j4 version. If the version numbers for both are the same, and match the installed version on j3.10 then it is safe to say that no update is required.

If the j4 update entry has a different version number to the j3.10 (or j3.*) one, then the installed version will require update but may not be backward compatible with j3.10

The fact that a j3 extension runs on 3.10 ok doesn't necessarily mean that it is j4 compatible does it???

[zero-24]

Agree as mention above this sounds like a bug to me. Can you provide the sample extension that you are working with?

[rogercreagh]

Agree as mention above this sounds like a bug to me. Can you provide the sample extension that you are working with?

The one I'm working on is only on my local network test server (its far from ready), but you can achieve the same effect by changing the update.xml file for any extension you have access to the update server for to read
<targetplatform name="joomla" version="3.*"/>
or
<targetplatform name="joomla" version="3.10"/>

There is a further quirk that the no update required details info appears to be taken from the version_id in the #__schemas table for the extension. This only seems to reflect version where any database updates were last made - in this case (below) v0.5.5.
Versions 0.5.6, 0.5.7, and the current one didn't update any database tables so there was no sql update file - although they may have made other changes towards j4 compatibility. So there is potential for confusion there as, for example, 0.5.5 may have been incompatible but 0.5.6 might have been the first j4 compatible version.

Looking at the #__schemas table on some live sites it is clear that not many extensions that create their own database tables follow the recommendation that every update should include a admin/sql/updates/mysql/x.y.z.sql file even if it is empty because there are no database changes. So the schema version_id will lag behind the installed version id. Which makes sense in terms of tracking which version last changed the database structure (which wouldn't happen if dummy .sql files were included), but is not very helpful for the j4 update checker which reports:

Extension Name | Extension Type | Update Compatible Version | Currently Compatible Version | Installed Version
xb-Books | Component | 0.5.5 | 0.5.5 | 0.5.8b

It is not at all clear what is meant by the labels "Update Compatible Version" and "Currently Compatible Version" but probably not what it says for version_id in the #__schemas table.

This is probably a separate issue, as it seems to only affect the details display, not the false positives.

[rogercreagh]

In conclusion I suggest that the only authoritative source for j4 compatibility is the update.xml file containing a
<targetplatform name="joomla" version="4.0"/>
or higher tag. The highest version number with a j4 target will be the currently compatible version.
To track what the first j4 compatible version was, you need to parse any earlier versions in the update.xml file to determine what the first j4 compatible one was (Update Compatible Version).

There is no other reliable source of info from the component developer or from within the installed component. The only think that can be extracted there is the currently installed version.
Table #__schemas shouldn't enter into it.

[zero-24]

There is a further quirk that the no update required details info appears to be taken from the version_id in the #__schemas table for the extension. This only seems to reflect version where any database updates were last made - in this case (below) v0.5.5.
Versions 0.5.6, 0.5.7, and the current one didn't update any database tables so there was no sql update file - although they may have made other changes towards j4 compatibility. So there is potential for confusion there as, for example, 0.5.5 may have been incompatible but 0.5.6 might have been the first j4 compatible version.

Can you please share a version even dev version with me maybe via mail? I'm happy to look at it but debugging an issue that did not encurred to me is hard. Specificly the #__schemas issue you raised here.

Taking the version form the #__schemas table seems to be wrong to me as well. And creating empty update sqls are not an recommendation, at least I'm not aware of, where is that recommendation comming from?

In conclusion I suggest that the only authoritative source for j4 compatibility is the update.xml file containing a

Well yes that is the plan everything where the regex applies and includes 4.0 that be shown as compatible. Maybe a quick look at the update XML files could give a glips too? Can you share that I mean the URL to the full XML file?

I have just tried with on of my own dummy extensions and can not reproduce the issue mention here please try this dummy extension:
plg_system_dummy_extension.zip

That plugin is pointing to: https://www.jah-tz.de/downloads/core/extension/dummy/extension_4.xml and shows correctly as "you have an update" in the Installer (so that 3.* works):

But without compatible versions aviable for 4.x as expected:

So please share with me your dev version and write down the steps you took to get to the point at best based on a backup or an clean install as you described above. If you want you can send it to tobias.zulauf[at]community.joomla.org or upload it here and I'm happy to check that issue so we make sure that case is covered too.

Thanks

[rogercreagh]

Firstly I now discover that a couple of my previous extensions exhibit the same behaviour. Try these:
Autosave: http://www.crosborne.co.uk/downloads?download=19 (it adds timed autosave functionality to JCE/TinyMCE)
Sunclock: http://www.crosborne.co.uk/index.php/downloads?download=4 (time since sunrise etc)

Regarding your dummy extension - the installer is version 1.0.0 and the update xml says that 1.0.1 is available so on my system it is appearing in the Update Required section (fresh install of J3.10.0 Alpha 2 shortly after it was released, updated to 3.10.0 alpha 3 last week.) NOT as per your screenshot which is very strange.

There isn't actually a 1.0.1 available on the download url (it is 1.0.0) so updating it on the installer page has no effect.
But if I change the version number in the plg_system_dummy zip file dummy.xml to 1.0.1 which then matches the update xml it appeasr in the No Uppdate required section.

So I am getting a different result to you. We seem to have the same version of 3.10 installed so something else is affecting it.

I am running joomla 3.10 on localhost on ubuntu 20.0.4, so there is no ssl but I can''t see why that would make a difference here. Using the Brave browser (which is a Chrome fork)

Incidentally none of those extensions listed as not requiring an update are J4 ready. It looks like on my system the update checker is merely doing the same as the extension manager update checker and reporting the existence of J3 updates. Why???

[zero-24]

Ok I'm sorry that it took that long to come back to you on this issue.

I have now looked a bit deeper into it and now i can explain whats happening here.

Your extension mod_sunclock has the following update server: https://www.crosborne.co.uk/phocadownload/jupdates/sunclock_updater.xml

Based on the informations provided there the latest version of your extension is 1.1.0 and that is only compatible within 3.x but the developer has provided no compatibility information. for Joomla 4 yet.

So the outcome is expected and it is also expected that this extension shows up in the Update Information Unavailable section given that the information what version of that extension is compatible with Joomla 4 has not been published yet.

But I agree that the text is not good as it is now. I have prepared an PR #32078 that improves the texts and explains that all "3.x only" extensions will be listed there too.

Thanks for beeing so patient on this topic :)

[rogercreagh]

Thanks for trying @zero-24 but you've missed the actual error. The commit by @richard67 appears to only correct the text under the red bar at the top of the group of extensions that are not providing any j4 comaptibility information.

The problem is that extensions that target 3.* in their update file (as might be typical for some old extensions - it has been recommended in various tutorial files in the past) are not being listed under the red bar group, they are being listed under the green "no update required" group.

If you explicitly target anything up to and including 3.10 ( eg version="3.(8|9|10)" ) then you will correctly be listed under the orange "update required" for J4.

If there has been an announcement to the effect that from J3.11 all extensions must be J4 compatible, then apologies I must have missed it. In any event detecting 3.* as implying J4 compatible seems illogical.

In the screen shot below the extensions xbBooks and xbFilms use version="3.(9.10)" and are correctly listed as needing update to be J4 compatible. The extensions xbAutoSave and SunClock are old ones that are using version="3.*" and are very far from j4 compatible but they are listed under the green bar.

I propose two changes are required:

1. if the update.xml file does not explicitly mention version 4 compatiblity then the extension must be listed under the orange bar.
2. Improve the title on the bars to make it clear they are talking about J4 compatibility:
   Orange Bar: "Update required for Joomla 4"
   Green Bar: "No update required for Joomla 4"

Alternatively the page needs to specify which the "selected target version of Joomla" actually is - nowhere does it actually tell you this. I understood, possibly wrongly, that this update checker was primarily for checking whether your installed extensions are J4 ready.

The red bar group is routinely listing extensions which DO use the update system but which simply do not say 3.10 or higher (including 3.*). This is confusing. Surely if an extension specifies 3.9 compatibility only then it should be listed in orange group (Update Required) even if there is no update for 3.10 or 4.x available yet. The normal extension update check page (and the control panel) tells you if there is an update available for a particular extension. The red group should only be extensions which do not use the update system. I find it all very confusing at present.

Otherwise there are going to be a lot of false positives with extensions that state 3.* compatibility and pissed off/confused users when they update to J4 and it all falls apart.

I appreciate that perhaps it is now technically wrong to specify version="3.*" in the update file, but I suspect there must be others like me who have used that format in the past and are unaware of this issue. Naturally I will update all of my update files appropriately.

Suggest re-opening this ticket?
Many thanks for your work so far.
Roger

[zero-24]

Thanks for trying @zero-24 but you've missed the actual error. The commit by @richard67 appears to only correct the text under the red bar at the top of the group of extensions that are not providing any j4 comaptibility information.

Thats correct. This message is not specific to Joomla 4 and for that also not mention there. The aim is to list all extensions where manually steps are required.

The problem is that extensions that target 3.* in their update file (as might be typical for some old extensions - it has been recommended in various tutorial files in the past) are not being listed under the red bar group, they are being listed under the green "no update required" group.

I can not reproduce this issue when pointing my system to the Joomla 4 update server. It is correct that this specific extension would be listed as "no update required" in the case that we are pointing the system to an 3.10 update as the update server clearly states that.

If you explicitly target anything up to and including 3.10 ( eg version="3.(8|9|10)" ) then you will correctly be listed under the orange "update required" for J4.

I can not reproduce this kind of issues. Can you please provide more infos like the exact version you are running on and the settings you put in com_joomlaupdate.

Mine are Joomla! 3.10.0-alpha4 and

If there has been an announcement to the effect that from J3.11 all extensions must be J4 compatible, then apologies I must have missed it. In any event detecting 3.* as implying J4 compatible seems illogical.

There are no plans for anything beyond 3.10 and no 3.x extension is required to be Joomla 4 compatible.
3.* is intended to indicate only 3.x compatibility and to be not compatible with 4.0 and following.

In the screen shot below the extensions xbBooks and xbFilms use version="3.(9.10)" and are correctly listed as needing update to be J4 compatible. The extensions xbAutoSave and SunClock are old ones that are using version="3.*" and are very far from j4 compatible but they are listed under the green bar.

Hmm that does not look correct to me. But i can not reproduce this issue with the extension you mention and the settings above. Lets check whether we use the same settings so we can debug and check this.

if the update.xml file does not explicitly mention version 4 compatiblity then the extension must be listed under the orange bar.

I disagree as long as there is no indication for Joomla 4 compatiblity the extension should be listed under the "red" box. As this is the "you have to check them before any upgrade attempt" extensions. They are threated as not compatible as there is no compatible update published. Once there is a compatible update published than the extension should be moved to the orange box indecating that the upgrade has to be installed before or after the joomla core update, depending on the documentation and upgrade process implemented by the extension developer.

Improve the title on the bars to make it clear they are talking about J4 compatibility:
Orange Bar: "Update required for Joomla 4"
Green Bar: "No update required for Joomla 4"

This pre upgrade checker is not only used to check Joomla 4 compatiblity but all updates starting with 3.10.0 and Joomla 4. For that reason we choose to have that exact wording :)

Alternatively the page needs to specify which the "selected target version of Joomla" actually is - nowhere does it actually tell you this.

Agree do you have a suggestion where it would make sense in your opinion?

I understood, possibly wrongly, that this update checker was primarily for checking whether your installed extensions are J4 ready.

Yes and no. I would expect the most time you have to use this is within the 3.10 to 4.x migration (or any future major version) but this is not specificly build for Joomla 4.

The red bar group is routinely listing extensions which DO use the update system but which simply do not say 3.10 or higher (including 3.*). This is confusing. Surely if an extension specifies 3.9 compatibility only then it should be listed in orange group (Update Required) even if there is no update for 3.10 or 4.x available yet. The normal extension update check page (and the control panel) tells you if there is an update available for a particular extension. The red group should only be extensions which do not use the update system. I find it all very confusing at present.

Yes thats the reason I did the PR to extend the list of reasons to be listed as it lists all extensions that have not provided any information about that they are compatible.

The orange box is intended for all extensions that have acutally provided a update to joomla 4 whether you have to install that before or after the core update. And that is also a difference between the pre upgrade checker and the controll panel.
The control panel will you only show updates that can be applied to the current 3.x system. But it is also totally valid to just publish an 4.x only compatible version of your extension. So you can not apply the update in 3.x.
Such extensions are not listed within the control panel as that update can not not be applied to Joomla 3.x but only 4.x in this case.

Otherwise there are going to be a lot of false positives with extensions that state 3.* compatibility and pissed off/confused users when they update to J4 and it all falls apart.

Agree thats the reason we show it this way. The other way we would also piss of / confuse extension developers when the users contact them on an update that joomla proposed that they have not provided yet.

I appreciate that perhaps it is now technically wrong to specify version="3.*" in the update file, but I suspect there must be others like me who have used that format in the past and are unaware of this issue. Naturally I will update all of my update files appropriately.

It is 100% technically valid. I personally choose to specificly list the Joomla Versions i have tested this version on (example). Using this I can make sure no update ins provided to systems that i have not under support as well as not tested my code one. For example some of my extensions use code that only runs on 3.9+ so 3.* would be wrong in that case but it is 100% valid for extensions that can run and be installed on 3.0.0 up to 3.10

Suggest re-opening this ticket?

Will do for the both questions raised above.

Many thanks for your work so far.

Thanks for your feedback :)

[rogercreagh]

Thanks @zero-24 .
Firstly possibly the reason you couldn't reproduce the problem in my screenshot was that after my last post I changed the upgrade version in the update xml to 3.9 - I have put them back to 3.*.

Can I take a step back and review the situation as I understand it should be:
There is one other factor that we haven't mentioned yet and that is the update channel setting in options.

I have J3.10.0 alpha4 installed on my test site
If I select the update channel (JoomlaUpdate|Options) to Default then the Joomla Update page tells me:

If I change it to Joomla Next it tells me this:

presumably once there is a release of J4 this will change - and a compatibility tab will appear.
If I select Testing as the update channel with minimal stability set to beta, release candidate, or stable I get this:

which makes sense for release candidate and stable, but doesn't seem right for beta as the minimum stability since we are already on J4beta6
(Yes I have cleared caches and clicked the check for updates button after each change of update channel)
It is only if I select Testing and Alpha that I get the pre-update check tab. It gives me the same message as before on the Live Update tab telling me there is no update available.

What it is not doing is telling me on the pre-update check page what version it is checking for - this would be a really useful addition at the top of the page.

Reading between the lines of the last post I guess that currently the update checker is not actually capable of telling that there is a J4 option (beta) available and so is ONLY reporting on J3.10+ compatibility. Which explains why minimum stabilty beta doesnt't show it.

Hence we get the following results:
xml version - update check result
version="3.9" - Red
version="3.(8|9)" - Red
version="3.(9|10)"- Green
version="3.*" - Green
version="((3.10)|(4.0))" - Green
version="4.0" - Red
version="((3.9)|(4.0))" - Red

Understanding that it is only testing for J3.10 compatibility then this seems ok. It just needs to be made very clear what version is being checked against at the top of the page. At present this page does NOT indicate anything about J4 compatibility of extensions.

However the last item in the list does explain why many existing J4 compatible extensions are appearing in the red area - they are not specifying 3.10 compatibility, just 3.9 and 4.0
Examples include Akeeba Backup Pro version="((3\.9)|(4\.0))" (ie the same update section for both j3 and j4) and JCE version="3.[6789]" for J3 and a separate section with version="4.[0123456789]" (ie separate entries in the update xml but both pointing to the same download.

The other area for improvement would be to separate the red area into two distinct sections:
Red a - extensions which do not provide update xml files
and
Red b for extensions which provide update xml but do not state they are compatible with J3.10 despite clearly being installed on J3.10. These should really be green or perhaps blue to indicate that you seem to have installed them despite them not claiming to work on j3.10. Perhaps this section should have a Blue header bar.

Finally the update options need to be modified so that selecting Joomla Next channel would provide a sub option for minimum stability as with Testing channel.

Then all the confusion would disappear. If I selecteded Joomla Next as the channel with beta or alpha as the minimum stability I would get a Joomla 4 compatibility checker page - essentially the same as now but checking for 4.0+ instead of 3.10

[zero-24]

Firstly possibly the reason you couldn't reproduce the problem in my screenshot was that after my last post I changed the upgrade version in the update xml to 3.9 - I have put them back to 3.*.

I changed the update server to point to a local copy which i tested all possible scenarios with.

If I select the update channel (JoomlaUpdate|Options) to Default then the Joomla Update page tells me:

Yes correct given that Joomla 4 is not publised yet on the Stable / Default channel.

If I change it to Joomla Next it tells me this:

Yes correct given than Joomla 4 is not yet published on that channel too.

If I select Testing as the update channel with minimal stability set to beta, release candidate, or stable I get this:

Correct as the upgrade path is not yet published on the test update servers.

What it is not doing is telling me on the pre-update check page what version it is checking for - this would be a really useful addition at the top of the page.

Yes as mention above do you have any suggestion where we could add that too?

Reading between the lines of the last post I guess that currently the update checker is not actually capable of telling that there is a J4 option (beta) available and so is ONLY reporting on J3.10+ compatibility. Which explains why minimum stabilty beta doesnt't show it.

Yes and no it is capable of telling Joomla 4 compatibility but you have to use Custom with this update server: https://update.joomla.org/core/test/310to4_list.xml as mentiond in the 3.10/4.0 beta release notes: https://www.joomla.org/announcements/release-news/5832-joomla-4-0-0-beta6-and-joomla-3-10-alpha4.html

Understanding that it is only testing for J3.10 compatibility then this seems ok. It just needs to be made very clear what version is being checked against at the top of the page. At present this page does NOT indicate anything about J4 compatibility of extensions.

Yes are you are testing against 3.10 and not 4.x yet ;)

Finally the update options need to be modified so that selecting Joomla Next channel would provide a sub option for minimum stability as with Testing channel.

The Joomla Next channel brings you allways the the latest stable major once released. The default setting will keep you within the current stable release.
The Joomla Next is not intended to ship non stable releases for that reason it is intended that there is no minimum compatiblitly setting on that channel.

Then all the confusion would disappear. If I selecteded Joomla Next as the channel with beta or alpha as the minimum stability I would get a Joomla 4 compatibility checker page - essentially the same as now but checking for 4.0+ instead of 3.10

Thats true once we have 4.0 stable out for the time 4.0 beeing in development mode you have to use the update server settings mentiond above and in the release notes.

The other area for improvement would be to separate the red area into two distinct sections:
Red a - extensions which do not provide update xml files
and
Red b for extensions which provide update xml but do not state they are compatible with J3.10 despite clearly being installed o n J3.10. These should really be green or perhaps blue to indicate that you seem to have installed them despite them not claiming to work on j3.10. Perhaps this section should have a Blue header bar.

Hmm well since ages we have the JED requirement to provide an update server, primarly this "red box" mentions all extensions that you have manuall steps to do before you even attempt to do an upgrade. Maybe we can change the title to something like Manuall checking required or something similiar? Any suggestions?

[zero-24]

I have just noticed the code was already there but the language string is not using it. Working on a PR now.

[zero-24]

PR is here: #32098

[ghost]

@rogercreagh please close this issue as there is a pr

[rogercreagh]

Very many thanks to @zero-24 for his patience in working through my ramblings.

Yes that PR looks like it should largely fix the problem by making things clearer. Lets close this issue and see what it looks like once implemented.
It is not clear from the screenshots whether this also repeats the information about which channel you are on. If it doesn't then I'll open a fresh issue when we see it working if needed.

I had completely missed the information about having to use custom channel and specify the update server to get v4 checking - I do apologise for that. It would certainly help if the screen told you that you were only on the Testing channel so the information below relates to 3.10 compatibility, or the Custom channel in which case it relates to J4 compatibility - and if you are Joomla Next channel then I think it should check J4 compatibility even though J4 is not released yet. (In other words before release if you are on Joomla Next channel the update checker should appear and tell you what is currently compatible or not even though you can't yet do a Live Update - that would be very useful, and enable us to see what needed to be done before release without having to enter the custom channel url.

Thanks again for your assistance. Closing this issue now.
Roger

[zero-24]

No problem feedback like yours helps massively to improve the system. Thanks for that!

That channel stuff is noted somewhere but maybe "just" within the live update Tab right now.