Introduction

This is the github repo of our recently published work Probabilistic Debiasing of Scene Graphs at CVPR 2023. We address the long-tailed distibution of scene graphs through within-triplet debiasing of measurement triplets. We perform experiments on Visual Genome and GQA dataset and achieve state-of-the-art debiased scene graphs. The codebase is primarily developed in MATLAB. The measurement results of the baseline models are extracted using python code and stored as MAT file. Afterwards, we perform the inference, evaluate the scene graphs, and plot the recall of each relationship with MATLAB.

Baseline Models

We generate measurements from the following publicly available baselines.

1. IMP, MOTIF, VCTree, Causal-MOTIF (TDE-MOTIF) -- Released by Tang et al. (Unbiased scene graph generation from biased training). The github link is https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch
2. DLFE_MOTIF -- Released by Chiou et al. (Recovering the Unbiased Scene Graphs from the Biased Ones). The github link is https://github.com/coldmanck/recovering-unbiased-scene-graphs
3. BGNN -- Released by Li et al. (Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generation). The github link is https://github.com/SHTUPLUS/PySGG

Uncertain evidence from baseline models

You can download our extracted evidence data for the above-mentioned baselines here. You can also use the following steps for extracting evidence data from the baseline data by yourself.

1. Go to "Gen_data_evidence/"
2. Copy "generate_meas_infer_triplet.py" in the run directory of the baseline (see github directories above). It will generate measurement results of testing images with associated measurement probabilities for specified baseline (imp, motif, vctree, ...), setting (predcls, sgcls, sgdet), and dataset (vg,gqa) . The MATLAB data file will have the name "data_rel_meas_infer_baseline_setting_dataset.mat". An example is "data_rel_meas_infer_vctree_sgcls_vg.mat"
3. Copy and run "generate_tripelts_ground.py" in the same dir. It will generate ground truth annotations for testing data in VG. An example is "data_rel_ground_vctree_sgcls_vg.mat"

Datasets (Visual Genome and GQA)

The baseline github repos already cover Visual Genome. However, they do not utilize GQA dataset. You can follow our DATASET.MD for creating GQA dataset.

Performing Inference

1. Go insider folder "Post_infer_evi_prior/"
2. You can select the model, setting, and dataset in the "run_post_inference.m".
3. Run "run_post_inference.m". It will take the measurements as input and produce inferred triplets with BN learnt from original and augmented samples. It will also calculate and plot the mean recall improvement.

Citation

If you find our work useful, please cite our paper

@InProceedings{Biswas_2023_CVPR,
    author    = {Biswas, Bashirul Azam and Ji, Qiang},
    title     = {Probabilistic Debiasing of Scene Graphs},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {10429-10438}
}