Finetuning Tinychart

[ViCtOr-dev13]

Hello, I would like to fine-tune or train TinyChart to improve its summarization skills. I have the impression that it doesn't capture all the data during summarization, whereas it does during data conversion into tables. I can't find a fine-tuning template code specifically for TinyChart. Can you please help me?

[zhangliang-04]

Hi @ViCtOr-dev13 ,
You are right. The chart-to-table task requires the model to extract all the data presented in the chart, but the summarization task doesn't. This is related to the text style of the summaries in the training data. We plan to release the training code in early June. If you are in urgent demand, you can refer to the training code of TinyLlava, which is the repo we are based on.

[zhangliang-04]

Hi @ViCtOr-dev13,
We just released the training code in this repo. Please try to pull the latest code. Have fun!

[ViCtOr-dev13]

Hello @zhangliang-04 ,
I saw you released the training code, thank you. I've tried it but I'm dealing with some issues. I'm looking for finetuning your model so I uncomment this part in train.sh

#!/bin/bash
TRAIN_DATA=../data_test/train_finetune_captions.json
TEST_DATA=../data_test/test_finetune_captions.json

#LLM_PATH=bczhou/TinyLLaVA-3.1B
#VIT_PATH=pretrained_models/TinyLLaVA-3.1B-SigLIP

#If you want to fine-tune TinyChart-3B-768:
LLM_PATH=mPLUG/TinyChart-3B-768
VIT_PATH=mPLUG/TinyChart-3B-768-siglip

First does it possible to finetune like this and also does a T4 GPU 15g RAM is enough
thank you for your time.