[Bug]:

[loyal-ikun]

Describe the bug

test commend result is different from Testing stage result of train commend.

Dataset

Folder

Model

GANomaly

Steps to reproduce the behavior

run test.py train.py respectively

OS information

OS information:

• OS: [e.g. Ubuntu 18.04]
• Python version: [e.g. 3.8.10]
• Anomalib version: [e.g. 0.3.6]
• PyTorch version: [e.g. 1.9.0]
• CUDA/cuDNN version: [e.g. 11.1]
• GPU models and configuration: [e.g. 2x GeForce RTX 3090]
• Any other relevant information: [e.g. I'm using a custom dataset]

Expected behavior

test commend result is different from Testing stage result of train commend.

Screenshots

No response

Pip/GitHub

pip

What version/branch did you use?

No response

Configuration YAML

none

Logs

none

Code of Conduct

• I agree to follow this project's Code of Conduct

[loyal-ikun]

can someone answer my question？ I've been trying for several days.

[ashwinvaidya17]

@loyal-ikun can you try the main branch and see if results are still different. Also, can you share the commands you used for training and testing the model?

[loyal-ikun]

my testing commend is python tools/test.py --config /home/lian/data/code/results/ganomaly/tor/logs/lightning_logs/version_3/hparams.yaml

only ganomaly model can't reproduce the training results.

[loyal-ikun]

this my yaml file：

dataset:
name: #options: [mvtec, btech, folder]
format: folder
path:
normal_dir: normal_train
abnormal_dir: abnormal #abnormal
normal_test_dir: normal_test #normal_test
task: classification
mask: null
extensions: null
split_ratio: null
image_size: 64
train_batch_size: 256
test_batch_size: 256

inference_batch_size: 32

num_workers: 8
transform_config:
train: null
val: null
create_validation_set: false
tiling:
apply: true
tile_size: 64
stride: null
remove_border_count: 0
use_random_tiling: False
random_tile_count: 16

model:
name: ganomaly
latent_vec_size: 130 #100 800 also ok
n_features: 64 #64
extra_layers: 0
add_final_conv: true
early_stopping:
patience: 3
metric: image_AUROC
mode: max
lr: 0.0002 #0.0002 0.0001ok
beta1: 0.5 #0.5
beta2: 0.999 #0.999
wadv: 1 #1
wcon: 50 #50
wenc: 1 #1
normalization_method: none

metrics:
image:
- F1Score
- AUROC
- Accuracy
- Precision
- Recall
threshold:
image_default: 0
adaptive: true

visualization:
show_images: False # show images on the screen
save_images: False # save images to the file system
log_images: False # log images to the available loggers (if any)
image_save_path: null # path to which images will be saved
mode: full # options: ["full", "simple"]

project:
seed: 42
path: ./results

logging:
logger: [csv] # options: [comet, tensorboard, wandb, csv] or combinations.
log_graph: false # Logs the model graph to respective logger.

optimization:
export_mode: ""

PL Trainer Args. Don't add extra parameter here.

trainer:
accelerator: auto # <"cpu", "gpu", "tpu", "ipu", "hpu", "auto">
accumulate_grad_batches: 1
amp_backend: native
auto_lr_find: false
auto_scale_batch_size: false
auto_select_gpus: false
benchmark: false
check_val_every_n_epoch: 2
default_root_dir: null
detect_anomaly: false
deterministic: false
devices: 1
enable_checkpointing: true
enable_model_summary: true
enable_progress_bar: true
fast_dev_run: false
gpus: null # Set automatically
gradient_clip_val: 0
ipus: null
limit_predict_batches: 1.0
limit_test_batches: 1.0
limit_train_batches: 1.0
limit_val_batches: 1.0
log_every_n_steps: 50
log_gpu_memory: null
max_epochs: 100 #100
max_steps: null
min_epochs: null
min_steps: null
move_metrics_to_cpu: false
multiple_trainloader_mode: max_size_cycle
num_nodes: 1
num_processes: null
num_sanity_val_steps: 0
overfit_batches: 0.0
plugins: null
precision: 32
profiler: null
reload_dataloaders_every_n_epochs: 0
replace_sampler_ddp: true
strategy: null
sync_batchnorm: false
tpu_cores: null
track_grad_norm: -1
val_check_interval: 1.0

[loyal-ikun]

@ashwinvaidya17 my commend and yaml file, as seen above.

[ashwinvaidya17]

Ideally the lightning_logs folder should not be generated. I just tested on main and explicitly passing the model weights to the test.py script works. Can you try this python tools/test.py --config <path_to_training_config> --weight_file <results/.../model.ckpt> Replace the path based on your model config and weights location.

[loyal-ikun]

I have solved this problem according to your method.Thank you.