a small bug in file "models/detector.py"

[AllenDun]

nice work! I did some tests and found some codes in file "models/detector.py" could cause some problem:

        global_entity_list = [] # save all the entity type name for each sentence.
        for entity_str in extracted_entities:
            # border case: nothing to extract
            if 'none' in entity_str.lower():
                continue
            entity_list = entity_str.split('.')
            for ent in entity_list:
                global_entity_dict.setdefault(ent, {}).setdefault('total_count', 0)
                global_entity_dict.setdefault(ent, {}).setdefault('crop_path', [])
                global_entity_dict.setdefault(ent, {}).setdefault('bbox', [])
            global_entity_list.append(entity_list)

when an entity is 'none', the 'global_entity_list' will not include the 'none' entity, which will result in wrong index order in 'global_entity_list' and 'sample['split_sents']' in file "models/questioner.py":

def generate_questions(self, sample: Dict):
        sentences = sample['split_sents']
        global_entity_dict = sample['entity_info']
        global_entity_list = sample['entity_list']
        qs_list = []
        num_calls = len(sentences)
        print(f'generate ques will call llm {num_calls} times')
        for ent_list, sent in zip(global_entity_list, sentences):
            exist_entity = [ent for ent in ent_list if ent in global_entity_dict and global_entity_dict[ent]['total_count'] > 0]
            # border case: no detection result for any entity. no question asked.
            if len(exist_entity)==0 :
                qs_list.append([])
                continue
            questions = get_res(self.nlp, '.'.join(exist_entity), sent)
            qs_list.append(questions)

by the way, how will the performance of VQA model affect the woodpecker performance? I changed the GPT-3.5 to llama3 and I understand the llm model will play an import role. But for the VQA model, did try other models? @xjtupanda @BradyFU