[bbc] Fix BBCCoUkIPlayerPlaylistIE

[dirkf]

Please follow the guide below

• You will be asked some questions, please read them carefully and answer honestly
• Put an x into all the boxes [ ] relevant to your pull request (like that [x])
• Use Preview tab to see how your pull request will actually look like

---

Before submitting a pull request make sure you have:

• Searched the bugtracker for similar pull requests
• Read adding new extractor tutorial
• Read youtube-dl coding conventions and adjusted the code to meet them
• Covered the code with tests (note that PRs without tests will be REJECTED)
• Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

• I am the original author of this code and I am willing to release it under Unlicense
• I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

• Bug fix
• Improvement
• New extractor
• New feature

---

Description of your pull request and other information

The BBCCoUkIPlayerPlaylistIE extractor was failing, whereas BBCCoUkPlaylistIE still worked for the same pids: eg, the _TESTS.

In the iPlayer series pages, the playlist metadata is now sent in a JSON expression within a <script> element, and this patch extracts the title and description from it, and modifies the _VIDEO_ID_TEMPLATE to match the episode links within the entities subobject of the JSON.

Also, the regex used to get the next page link for BBCCoUkPlaylistBaseIE didn't allow for a newline that is now sent between tags.

For multi-series iPlayer playlists, each series page is also processed.

For multi-page iPlayer playlists, the pagination regex is extended to match the "Next Page" link on these pages.

[remitamine]

do mix multiple changes in a single PR.

[remitamine]

• does not handle programmes/series with multiple seasons.
• does not handle seasons/groups with multiple pages.

[dirkf]

The current version should deal with your comments, but the download tests aren't yet validated: CI is failing because Certificate did not match expected hostname: files.pythonhosted.org when installing nose with pip install nose in the Jython configurations.

[remitamine]

as i was looking at this, i think there is a better way to make an optimized version that would reduce the number of requests and still support all the expected features in addition to extracting more metadata for --flat-playlist option, i will try to implement it.

[dirkf]

Feel free to use the test URLs from this PR for that. See also PR #23438 for a previous solution and set of test cases.

Generally, even though the BBC site support is split into different extractors, the BBCIE extractor in particular implements multiple extraction tactics and it's difficult to know which of these are still valid -- perhaps some telemetry would help, as well as refactoring the over-sized _real_extract() method.

It seems that two types of tactics are currently valid:

• find some JSON in JS that is part of the page; extract links and metadata from the JSON;
• find certain links, normally with a regex search, but mostly trying to replicate a DOM search similar to JS document.querySelectorAll() without the overhead of DOM.

Maybe generic methods can be created to support these tactics?

I have further updates to fix some of the failing test cases but will hold off if you're reimplementing.

[remitamine]

Generally, even though the BBC site support is split into different extractors, the BBCIE extractor in particular implements multiple extraction tactics and it's difficult to know which of these are still valid -- perhaps some telemetry would help, as well as refactoring the over-sized _real_extract() method.

It seems that two types of tactics are currently valid:

• find some JSON in JS that is part of the page; extract links and metadata from the JSON;
• find certain links, normally with a regex search, but mostly trying to replicate a DOM search similar to JS document.querySelectorAll() without the overhead of DOM.

Maybe generic methods can be created to support these tactics?

the main BBC Extractor has to be split in the future, this can be done by slowly extracting code that is related to a specific URL pattern into it's own extractor, it would certainly make the code more manageable.

I have further updates to fix some of the failing test cases but will hold off if you're reimplementing.

it always depend on whether there is a better way of doing thing, in the case of BBC Reel code, even thought i would preferred to introduce the code in a different way, but as the code can be reused in the future, i did accept it with only small changes, but in this case, the method that i added is quite different from the code here, so there aren't much code that can be reused(except for the tests).

[dirkf]

The changes I mentioned aren't specific to the iPlayer extractor (fixes to tests, morph playlists, support simorgh playlists).

Regarding the new code, please see #15710 (comment) for a case where what started out as an iPlayer URL redirected to a programmes page in which _call_api() returned None and caused a crash.