Support SEI - Mainly for Film Grain and Field Frame #257

QSXW · 2024-08-06T18:24:58Z

use -vf tinterlace filter to generate the same yuv as VTM
passed files:
FIELD_A_Panasonic_4.bit
FIELD_B_Panasonic_2.bit

VVCSoftware_VTM

FFmpeg

nuomi2021

Thank you for the patch.

nuomi2021 · 2024-08-11T13:34:24Z

libavcodec/cbs_h266.h

@@ -862,6 +862,14 @@ typedef struct CodedBitstreamH266Context {
    H266RawVPS  *vps[VVC_MAX_VPS_COUNT]; ///< RefStruct references
    H266RawSPS  *sps[VVC_MAX_SPS_COUNT]; ///< RefStruct references
    H266RawPPS  *pps[VVC_MAX_PPS_COUNT]; ///< RefStruct references
+
+    // The currently active parameter sets.  These are updated when any


vvc spec did not define the active parameter set.
the parameter set will be active by the H266RawPictureHeader

Okay. Will check that.

Yeah. Remove it and move the film grain to the h274 sei syntax template also.

nuomi2021 · 2024-08-11T13:40:08Z

libavcodec/cbs_h266.h

@@ -848,6 +848,27 @@ typedef struct H266RawSlice {
    int          data_bit_start;
 } H266RawSlice;

+typedef struct H266RawSEIFilmGrainCharacteristics {


could we reuse AVFilmGrainH274Params?

better not to use here. It's used to align for h264/h265/h266 syntax_template.

nuomi2021 · 2024-08-11T13:50:04Z

libavcodec/vvc/sei.h

@@ -31,8 +31,19 @@
 #include "libavcodec/sei.h"
 #include "libavcodec/vvc.h"

+typedef struct VVCSEIPictureHash {


could we define this in h274.h? and follow film grain things

We can, but film grain is exposed to the API user with the AV prefix. It's used to align with hevc/sei.h

but the vvc sei defined in h274, right?

Yeah. Will do.

nuomi2021 · 2024-08-11T13:53:41Z

libavcodec/cbs.c

@@ -158,7 +158,7 @@ av_cold void ff_cbs_close(CodedBitstreamContext **ctx_ptr)
    av_freep(ctx_ptr);
 }

-static void cbs_unit_uninit(CodedBitstreamUnit *unit)
+void ff_cbs_unit_uninit(CodedBitstreamUnit *unit)


why do we need this? better give some explain on the commit log

Will it be better to change it to ff_cbs_unref_unit?

I mean, why we need to expose this to public.
can we just take a reference to unit->content_ref, like slice_start

Goot catch. I use the same way to add a reference to the unit content_ref and remove the redundant commit for exposing ref unit.

nuomi2021 · 2024-08-15T13:20:15Z

libavcodec/vvc/sei.h

 typedef struct VVCSEI {
 	H2645SEI common;
    VVCSEIPictureHash picture_hash;
+    VVCSEIFrameFieldInfo frame_field_info;


this can be defined in h274.h too, right?

Sure. Update VVCSET to H274SEI and move to h274.h

…istics Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

QSXW · 2024-08-27T20:15:18Z

Should we also move to verify picture hash to ff_h274_verify_decoded_picture_hash like ff_h274_apply_film_grain?

nuomi2021

thank you for the patch.
comment inline

nuomi2021 · 2024-08-31T12:37:59Z

libavcodec/vvc/sei.h

+#include "libavcodec/h274.h"
+
+typedef struct VVCSEI {
+	H2645SEI common;


wrong indent??

Yeah. It is a tab and replace it with spaces.

nuomi2021 · 2024-08-31T12:42:18Z

libavcodec/vvc/sei.c

+            h->comp_model_present_flag[c] = s->fg_comp_model_present_flag[c];
+
+        for (int c = 0; c < 3; c++) {
+            if (h->comp_model_present_flag[c]) {


can we combine the previous with this?

Yes sure. Good point.

nuomi2021 · 2024-08-31T12:55:04Z

libavcodec/cbs_sei_syntax_template.c

+            flags(fg_comp_model_present_flag[c], 1, c);
+
+        for (c = 0; c < 3; c++) {
+            if (current->fg_comp_model_present_flag[c]) {


how about:
const int bd = (c ? current->fg_bit_depth_chroma_minus8 ? current->fg_bit_depth_luma_minus8) + 8;
const int min = 0 - current->fg_model_id * (1 << (bd - 1));
const int max =((1 << bd) - 1) - current->fg_model_id * (1 << (bd - 1))

nuomi2021 · 2024-08-31T13:03:05Z

libavcodec/cbs_sei_syntax_template.c

+            flag(ffi_top_field_first_flag);
+        u(8, ffi_display_elemental_periods_minus1, 0, 0xff);
+    }
+    u(2, ffi_source_scan_type, 0, 3);


ub(2, ffi_source_scan_type); ?

The range of ffi_source_scan_type is from 0~3. Shouldn't we add it?

Do two bits mean it ranges from 0~3

nuomi2021 · 2024-08-31T13:16:51Z

libavcodec/cbs_h2645.c

+        SEI_MESSAGE_RW(sei, film_grain_characteristics),
+    },
+    {
+        SEI_TYPE_DISPLAY_ORIENTATION,


It's not included in h274, right?

Nope, it is. See 8.25 Display orientation SEI message

8.25? not in vvc spec, right

nuomi2021 · 2024-08-31T13:40:25Z

libavcodec/vvc/dec.c

@@ -623,6 +665,10 @@ static av_cold int frame_context_init(VVCFrameContext *fc, AVCodecContext *avctx
        fc->DPB[j].frame = av_frame_alloc();
        if (!fc->DPB[j].frame)
            return AVERROR(ENOMEM);
+
+        fc->DPB[j].frame_grain = av_frame_alloc();


DPB is not a good place for frame_grain. frame_grain is not included in dpb process.

I reference it from hevcdec.

for (i = 0; i < FF_ARRAY_ELEMS(s->DPB); i++) { ff_hevc_unref_frame(&s->DPB[i], ~0); av_frame_free(&s->DPB[i].frame_grain); }

May you help check where is a better place to put on?

let me try, how to test frame_grain?

nuomi2021 · 2024-08-31T13:40:48Z

libavcodec/vvc/dec.c

@@ -887,16 +999,192 @@ static int set_output_format(const VVCContext *s, const AVFrame *output)
    return 0;
 }

+static int verify_md5(VVCContext *s, VVCFrameContext *fc)


maybe we can move this to h274.c

Yeah! Moved.

nuomi2021 · 2024-08-31T13:41:29Z

libavcodec/vvc/sei.c

+#include "sei.h"
+#include "dec.h"
+
+static int decode_film_grain_characteristics(const VVCFrameContext *fc, H2645SEIFilmGrainCharacteristics *h, void *payload)


this needed by all h274 users, right?

It needs a valid sps, so it's better to keep it for codec specific.

how about put
luma_bit_depth, chroma_bit_depth, vui_full_range_flag, vui_colour_primaries, vui_transfer_characteristics, vui_matrix_coeffs in params list

nuomi2021 · 2024-08-31T13:47:27Z

libavcodec/vvc/dec.c

+
+    if (fc->ref->needs_fg) {
+        av_assert0(fc->ref->frame_grain->buf[0]);
+        fgp = av_film_grain_params_select(fc->ref->frame);


fgp can be NULL,

Yeah. It is checked check_film_grain.

if (fc->ref->needs_fg && (fc->sei.common.film_grain_characteristics.present && !ff_h274_film_grain_params_supported(fc->sei.common.film_grain_characteristics.model_id, fc->ref->frame->format) || !av_film_grain_params_select(fc->ref->frame))) { av_log_once(s->avctx, AV_LOG_WARNING, AV_LOG_DEBUG, &s->film_grain_warning_shown, "Unsupported film grain parameters. Ignoring film grain.\n"); fc->ref->needs_fg = 0; }

but not in this function, right?

nuomi2021 · 2024-08-31T13:54:20Z

libavcodec/vvc/dec.c

-        if (!ret)
-            *got_output = 1;
+    if (!ret) {
+        ret = frame_end(s, delayed);


do we need to call it export_sei and move it to " if (delayed->output_frame->buf[0] && output) {"

Maybe not. We already have a function set_side_data for export sei.

I tried add it to if (delayed->output_frame->buf[0] && output) { at my first version but it doesn't work.

Could you have a check?

nuomi2021 · 2024-08-31T13:56:06Z

Should we also move to verify picture hash to ff_h274_verify_decoded_picture_hash like ff_h274_apply_film_grain?

Yes, if a function is needed by all H.274 users, we should move it to h274.c

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

passed files: FIELD_A_Panasonic_4.bit FIELD_B_Panasonic_2.bit Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

The type of data of AVBufferRef is uint8_t * and the name can be various types of pointer, so cast it to void * before the assignment to silence the compiling warning. Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

The sei parsing doesn't influent the decoding, so skip the corrupt sei. The buffering period SEI of the following clip also failed to parse on VVCSoftware_VTM: https://www.itu.int/wftp3/av-arch/jvet-site/bitstream_exchange/VVC/under_test/VTM-15.0/HRD_B_Fujitsu_2.zip Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

Buffering Period SEI message will be used by other SEI messages like Picture Timing and DU information. Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

…ms_present_flag Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

QSXW force-pushed the feat/FilmGrain branch from 7023f9f to 9af43f1 Compare August 6, 2024 18:27

QSXW mentioned this pull request Aug 6, 2024

Support film grain #253

Open

QSXW force-pushed the feat/FilmGrain branch 3 times, most recently from 9205eea to 4024167 Compare August 10, 2024 18:50

nuomi2021 requested changes Aug 11, 2024

View reviewed changes

QSXW changed the title ~~Support Film Grain~~ Support Film Grain and Field Frame Aug 14, 2024

QSXW force-pushed the feat/FilmGrain branch from baa5957 to bee3360 Compare August 14, 2024 19:21

QSXW changed the title ~~Support Film Grain and Field Frame~~ Support Versatile supplemental enhancement information messages - Mainly for Film Grain and Field Frame Aug 14, 2024

QSXW changed the title ~~Support Versatile supplemental enhancement information messages - Mainly for Film Grain and Field Frame~~ Support SEI - Mainly for Film Grain and Field Frame Aug 14, 2024

QSXW force-pushed the feat/FilmGrain branch from bee3360 to 7752279 Compare August 14, 2024 19:56

QSXW mentioned this pull request Aug 14, 2024

Support fields #6

Open

QSXW self-assigned this Aug 14, 2024

QSXW added the new feature label Aug 14, 2024

nuomi2021 reviewed Aug 15, 2024

View reviewed changes

avcodec/cbs_sei_syntax_template: add sei message film_grain_character…

46547f9

…istics Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

QSXW force-pushed the feat/FilmGrain branch 2 times, most recently from bc2d955 to 69a02c9 Compare August 27, 2024 20:14

QSXW requested a review from nuomi2021 August 27, 2024 20:14

nuomi2021 reviewed Aug 31, 2024

View reviewed changes

QSXW force-pushed the feat/FilmGrain branch from 69a02c9 to 42f67f8 Compare September 5, 2024 17:07

QSXW added 7 commits September 6, 2024 01:15

avcodec/vvc: support decoding prefix and suffix nal units

d7eb4de

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/vvc/sei: add decode_film_grain_characteristics

d0e35db

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/vvc/dec: export sei to the frame when the frame starts

212d613

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/vvc/dec: support applying film grain by the decoder

9d07a00

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/vvc/dec: support removing film grain params from side data

e8f4182

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/h274: add H274SEIPictureHash struct

3f4e970

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/vvc/sei: add decode_decoded_picture_hash

d1bf0b9

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

QSXW and others added 13 commits September 6, 2024 01:21

avcodec/h274: add ff_h274_verify_decoded_picture_hash

cb98fb1

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/cbs_sei_syntax_template: add sei message sei_display_orientation

72b881a

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/vvc/sei: add decode_display_orientation

6e37383

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/vvc/sei: add decode_content_light_level_info

4c21a89

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/cbs_sei_syntax_template: add sei message frame_field_information

3e1348c

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/h274: add H274SEIFrameFieldInfo struct

80a5566

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/vvc/sei: add decode_frame_field_info

14a2da0

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/vvc: support fields

2c7d931

passed files: FIELD_A_Panasonic_4.bit FIELD_B_Panasonic_2.bit Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/vvc/sei: add decode_ambient_viewing_environment

b75017d

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/vvc/sei: add decode_mastering_display_colour_volume

4a34af7

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

add github workflow

2bb8d4b

Add fuzz regression test

15e164b

Fix fuzz test path

fe0ea5e

QSXW force-pushed the feat/FilmGrain branch from 42f67f8 to fe0ea5e Compare September 5, 2024 17:33

QSXW requested a review from nuomi2021 September 5, 2024 17:39

QSXW added 6 commits September 9, 2024 03:38

avcodec/cbs_sei_syntax_template: add sei message buffering_period

9751fdd

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/cbs_h2645: silence conversion warning

fc7796e

The type of data of AVBufferRef is uint8_t * and the name can be various types of pointer, so cast it to void * before the assignment to silence the compiling warning. Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/cbs_h266: add RefStruct reference of Buffering Period

8d14344

Buffering Period SEI message will be used by other SEI messages like Picture Timing and DU information. Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/cbs_sei_syntax_template: add sei message picture_timing

cd69ad7

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/cbs_sei_syntax_template: add sei message decoding_unit_info

f28f8dd

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

QSXW force-pushed the feat/FilmGrain branch from abb06b0 to 6d97e3d Compare September 9, 2024 20:17

QSXW added 2 commits September 10, 2024 04:18

avcodec/cbs_h266: cache prefix sei

1583889

Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

avcodec/cbs_h266_syntax_template: add sanity check for bp_du_hrd_para…

5800f8d

…ms_present_flag Signed-off-by: Wu Jianhua <toqsxw@outlook.com>

QSXW force-pushed the feat/FilmGrain branch from 6d97e3d to 5800f8d Compare September 9, 2024 20:18

Support SEI - Mainly for Film Grain and Field Frame #257

Are you sure you want to change the base?

Support SEI - Mainly for Film Grain and Field Frame #257

Conversation

QSXW commented Aug 6, 2024 • edited Loading

nuomi2021 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

QSXW Aug 27, 2024 • edited Loading

Choose a reason for hiding this comment

QSXW commented Aug 27, 2024

nuomi2021 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

QSXW Sep 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nuomi2021 commented Aug 31, 2024 • edited Loading

QSXW commented Aug 6, 2024 •

edited

Loading

QSXW Aug 27, 2024 •

edited

Loading

QSXW Sep 5, 2024 •

edited

Loading

nuomi2021 commented Aug 31, 2024 •

edited

Loading