Full screen bitmap smpte-tt subtitle not rendered correctly

[szaboa]

[REQUIRED] Issue description

Support for image based SMPTE-TT subtitles was implemented in #1583, however it's not working quite well for "full screen" subtitles as it renders these out of the screen and with wrong height. By full screen I mean when the subtitle should take size of the surface. This is the case when tt tts:extent = region tts:extent in case of pixel based regions.

[REQUIRED] Reproduction steps

It can be reproduced with the ExoPlayer demo app, just needs a stream with image based SMPTE-TT subtitle track where the Base64 encoded bitmap is "full screen".

[REQUIRED] Link to test content

Email

[REQUIRED] A full bug report captured from the device

No logs required.

[REQUIRED] Version of ExoPlayer being used

ExoPlayer 2.9.3 - 2.9.6

[REQUIRED] Device(s) and version(s) of Android being used

It can be reproduced every time and not device dependent.
In my tests, I've used Pixel 1.

[szaboa]

Example:
The manifest contains in tt tag let's say tts:extent="720px 576px" and there are regions like <region xml:id="region_0" tts:extent="720px 576px" tts:origin="0px 0px"/>.

Two problems causing this issue:

1. Incorrect height calculation
In the example above the bitmap will take the width of the parent correctly, but the height is scaled up to keep the bitmap's aspect ration.

Height calculation in com.google.android.exoplayer2.ui.SubtitlePainter#setupBitmapLayout:

int height = cueBitmapHeight != Cue.DIMEN_UNSET ? Math.round(parentHeight * cueBitmapHeight)
        : Math.round(width * ((float) cueBitmap.getHeight() / cueBitmap.getWidth()));

The cueBitmapHeight will be Cue.DIMEN_UNSET because that's what we pass in com.google.android.exoplayer2.text.ttml.TtmlNode#getCues (line 238).

This logic is OK for non-full screen subtitles, but for a full screen one it doesn't force the bitmap to fit the parent size. This results incorrect positioning.

Inside com.google.android.exoplayer2.text.ttml.TtmlDecoder#parseRegionAttributes the width and height calculations are OK and both of them are 1 (which means 100% width and 100% height). But the height is not passed to the TtmlRegion.

Proposed solution:

Pass the height from parseRegionAttributes to TtmlRegion and inside com.google.android.exoplayer2.text.ttml.TtmlNode#getCues pass this height to Cue (line 238) or pass it only if both of the region's width and height is 1, this would only affect the full screen bitmaps. In this way the bitmap's height would be set to the parent height in com.google.android.exoplayer2.ui.SubtitlePainter#setupBitmapLayout.

2. Incorrect position anchor

Inside com.google.android.exoplayer2.text.ttml.TtmlNode#getCues we pass Cue.ANCHOR_TYPE_MIDDLE as a default position anchor for image based subtitles. This will render a full screen image based subtitle offscreen (top half of the bitmap is outside of the view).

Again, because the y calculation in com.google.android.exoplayer2.ui.SubtitlePainter#setupBitmapLayout:

int y = Math.round(cuePositionAnchor == Cue.ANCHOR_TYPE_END ? (anchorY - height)
        : cuePositionAnchor == Cue.ANCHOR_TYPE_MIDDLE ? (anchorY - (height / 2)) : anchorY);

Proposed solution:

Change the default position anchor to Cue.ANCHOR_TYPE_START or do not set anything, this way it will fall back to anchor start.

@ojw28
What do you think? Once we agree on a solution, I'll fix this.

[szaboa]

Can someone have a look on this please?

[AquilesCanta]

@szaboa, have you shared a stream with us? We will need sample media before we look into this. If you think the solution is straightforward, you might want to send a pull request directly: But most likely, we will still need a sample stream.

[szaboa]

I will try to get a sample stream as soon as I can.

Actually, I was who implemented the support for image based subtitles (smpte-tt) and I didn't pay enough attention to this use case. That's why I proposed the fix.

[szaboa]

Sorry, accidentally closed the issue, miss click :)

[szaboa]

@AquilesCanta, sorry for the late response :)

Updated ticket's description with sample streams. During testing make sure you pick the second Norwegian subtitle track and note that you might need to wait a little for the subs to show up (probably not synced correctly in this test content).

I've also created a pull request with the proposed solution (#5711).

---

And to make sure nothing changed for regular (not full screen) image based subtitles, I've also tested with this sample stream (which I used as a reference when I implemented support for image based subs at first place): https://livesim.dashif.org/dash/vod/testpic_2s/img_subs.mpd

---

@ojw28, you might want to have a look too on the proposed solution, as we have discussed this feature quite a lot at the beginning :)

[szaboa]

Any update on this? :)

[szaboa]

@ojw28, I would really appreciate if someone could take a look on this.

[AquilesCanta]

I'll look into it this week.

[szaboa]

Thank you!

[AquilesCanta]

Hi, I'm getting a 404 now. Can you please make sure the streams are available?

[szaboa]

Hi, will send the new ones via email shortly.
Edit. I will be able to send it only on Monday (27.05).

[AquilesCanta]

Ok. Please post below once you do. I suppose I'll get to this on Tuesday.

[szaboa]

@AquilesCanta
Ok, so no need for streams actually. I've created a TTML subtitle file:
https://drive.google.com/uc?export=download&id=1Rf7sWPgnDs7Ax7XOcFlNXwu2dNj8s8Pv

Which contains a cue that is full screen (big red rectangle) and it should cover the whole screen, as it has a region 1280px 720px and the root tt value is also 1280px 720px (which gives the window reference).

Please side load this subtitle to any of the streams inside the exo's demo application.

Before fix:

After fix:

[AquilesCanta]

We will be pushing a fix soon. Please try it out once a commit appears referenced below so that you can make sure it works before it reaches a release version.

[szaboa]

Alright, thanks.

[szaboa]

@AquilesCanta
The fix seems to be working fine, thank you 👍

[AquilesCanta]

Glad to hear that. Let me know if you run into any other issues.