Val_mssim very low

[nguyenanhtuan1008]

@AdneneBoumessouer Thank for your hard work.
I try to run your code but I saw that accuracy very low. I don't know what happen but I try with 3 ways and all given not good result.

python3 train.py -d mvtec/pill -a baselineCAE -b 32 -l mssim -c rgb (I also tested with ssim, l2, grayscale)
python3 train.py -d mvtec/pill -a mvtecCAE -b 32 -l mssim -c rgb
python3 train.py -d mvtec/pill -a inceptionCAE -b 32 -l mssim -c rgb

-> Epoch 00013: Reducing Max LR on Plateau: new max lr will be 0.08725637942552567 (if not early_stopping).
Restoring model weights from the end of the best epoch.
8/8 [==============================] - 10s 1s/step - loss: 0.4127 - mssim: 0.8223 - val_loss: 0.8789 - val_mssim: 0.3548
Epoch 00013: early stopping
Weights from best epoch have been loaded into model.

INFO:autoencoder.autoencoder:loss_plot.png successfully saved. INFO:autoencoder.autoencoder:lr_schedule_plot.png successfully saved. INFO:autoencoder.autoencoder:training history has been successfully saved as csv file. INFO:autoencoder.autoencoder:training files have been successfully saved at: /content/drive/Shared drives/1_New/Acuity/MVTec-Anomaly-Detection/saved_models/mvtec/pill/baselineCAE/mssim/13-10-2020_11-35-32 INFO:__main__:done.

And then, using finetune:
python3 finetune.py -p "saved_models/mvtec/pill/baselineCAE/mssim/13-10-2020_11-35-32/baselineCAE_b32_e0.hdf5" -m ssim -t float64

Last, using test.py:
python3 test.py -p "saved_models/mvtec/pill/baselineCAE/mssim/13-10-2020_11-35-32/baselineCAE_b32_e0.hdf5"

              filenames  predictions  truth  accurate_predictions

0 color/000.png 0 1 False
1 color/001.png 0 1 False
2 color/002.png 0 1 False
3 color/003.png 1 1 True
4 color/004.png 0 1 False
5 color/005.png 0 1 False
6 color/006.png 0 1 False
7 color/007.png 1 1 True
8 color/008.png 1 1 True
9 color/009.png 1 1 True
10 color/010.png 0 1 False
11 color/011.png 0 1 False
12 color/012.png 0 1 False
13 color/013.png 0 1 False
14 color/014.png 0 1 False
15 color/015.png 0 1 False
16 color/016.png 0 1 False
17 color/017.png 0 1 False
18 color/018.png 1 1 True
19 color/019.png 1 1 True
20 color/020.png 0 1 False
21 color/021.png 0 1 False
22 color/022.png 0 1 False
23 color/023.png 0 1 False
24 color/024.png 0 1 False
25 combined/000.png 0 1 False
26 combined/001.png 1 1 True
27 combined/002.png 0 1 False
28 combined/003.png 0 1 False
29 combined/004.png 0 1 False
30 combined/005.png 0 1 False
31 combined/006.png 1 1 True
32 combined/007.png 1 1 True
33 combined/008.png 1 1 True
34 combined/009.png 1 1 True
35 combined/010.png 1 1 True
36 combined/011.png 1 1 True
37 combined/012.png 1 1 True
38 combined/013.png 0 1 False
39 combined/014.png 0 1 False
40 combined/015.png 1 1 True
41 combined/016.png 0 1 False
42 contamination/000.png 0 1 False
43 contamination/001.png 0 1 False
44 contamination/002.png 0 1 False
45 contamination/003.png 0 1 False
46 contamination/004.png 0 1 False
47 contamination/005.png 0 1 False
48 contamination/006.png 0 1 False
49 contamination/007.png 0 1 False
50 contamination/008.png 0 1 False
51 contamination/009.png 0 1 False
52 contamination/010.png 0 1 False
53 contamination/011.png 1 1 True
54 contamination/012.png 1 1 True
55 contamination/013.png 0 1 False
56 contamination/014.png 0 1 False
57 contamination/015.png 0 1 False
58 contamination/016.png 0 1 False
59 contamination/017.png 0 1 False
60 contamination/018.png 1 1 True
61 contamination/019.png 0 1 False
62 contamination/020.png 1 1 True
63 crack/000.png 0 1 False
64 crack/001.png 0 1 False
65 crack/002.png 1 1 True
66 crack/003.png 0 1 False
67 crack/004.png 0 1 False
68 crack/005.png 0 1 False
69 crack/006.png 1 1 True
70 crack/007.png 0 1 False
71 crack/008.png 0 1 False
72 crack/009.png 0 1 False
73 crack/010.png 1 1 True
74 crack/011.png 1 1 True
75 crack/012.png 0 1 False
76 crack/013.png 1 1 True
77 crack/014.png 0 1 False
78 crack/015.png 0 1 False
79 crack/016.png 0 1 False
80 crack/017.png 0 1 False
81 crack/018.png 0 1 False
82 crack/019.png 1 1 True
83 crack/020.png 0 1 False
84 crack/021.png 0 1 False
85 crack/022.png 1 1 True
86 crack/023.png 1 1 True
87 crack/024.png 0 1 False
88 crack/025.png 0 1 False
89 faulty_imprint/000.png 0 1 False
90 faulty_imprint/001.png 0 1 False
91 faulty_imprint/002.png 1 1 True
92 faulty_imprint/003.png 0 1 False
93 faulty_imprint/004.png 0 1 False
94 faulty_imprint/005.png 0 1 False
95 faulty_imprint/006.png 1 1 True
96 faulty_imprint/007.png 1 1 True
97 faulty_imprint/008.png 0 1 False
98 faulty_imprint/009.png 0 1 False
99 faulty_imprint/010.png 0 1 False
100 faulty_imprint/011.png 0 1 False
101 faulty_imprint/012.png 1 1 True
102 faulty_imprint/013.png 0 1 False
103 faulty_imprint/014.png 0 1 False
104 faulty_imprint/015.png 0 1 False
105 faulty_imprint/016.png 1 1 True
106 faulty_imprint/017.png 0 1 False
107 faulty_imprint/018.png 1 1 True
108 good/000.png 0 0 True
109 good/001.png 0 0 True
110 good/002.png 0 0 True
111 good/003.png 0 0 True
112 good/004.png 0 0 True
113 good/005.png 0 0 True
114 good/006.png 0 0 True
115 good/007.png 0 0 True
116 good/008.png 0 0 True
117 good/009.png 0 0 True
118 good/010.png 0 0 True
119 good/011.png 0 0 True
120 good/012.png 1 0 False
121 good/013.png 0 0 True
122 good/014.png 1 0 False
123 good/015.png 0 0 True
124 good/016.png 0 0 True
125 good/017.png 1 0 False
126 good/018.png 0 0 True
127 good/019.png 0 0 True
128 good/020.png 0 0 True
129 good/021.png 0 0 True
130 good/022.png 0 0 True
131 good/023.png 0 0 True
132 good/024.png 0 0 True
133 good/025.png 0 0 True
134 pill_type/000.png 0 1 False
135 pill_type/001.png 0 1 False
136 pill_type/002.png 0 1 False
137 pill_type/003.png 0 1 False
138 pill_type/004.png 0 1 False
139 pill_type/005.png 0 1 False
140 pill_type/006.png 0 1 False
141 pill_type/007.png 1 1 True
142 pill_type/008.png 1 1 True
143 scratch/000.png 1 1 True
144 scratch/001.png 0 1 False
145 scratch/002.png 0 1 False
146 scratch/003.png 0 1 False
147 scratch/004.png 1 1 True
148 scratch/005.png 0 1 False
149 scratch/006.png 0 1 False
150 scratch/007.png 0 1 False
151 scratch/008.png 0 1 False
152 scratch/009.png 0 1 False
153 scratch/010.png 0 1 False
154 scratch/011.png 0 1 False
155 scratch/012.png 0 1 False
156 scratch/013.png 0 1 False
157 scratch/014.png 0 1 False
158 scratch/015.png 1 1 True
159 scratch/016.png 1 1 True
160 scratch/017.png 0 1 False
161 scratch/018.png 0 1 False
162 scratch/019.png 0 1 False
163 scratch/020.png 0 1 False
164 scratch/021.png 1 1 True
165 scratch/022.png 1 1 True
166 scratch/023.png 0 1 False
test results: {'min_area': 770, 'threshold': 0.27000000000000013, 'TPR': 0.2907801418439716, 'TNR': 0.8846153846153846, 'score': 0.587697763229678, 'method': 'ssim', 'dtype': 'float64'}

The result is not good at all.
Can you give me an advice?

[AdneneBoumessouer]

Rerun your experiments with mvtecCAE ,baselineCAE and inceptionCAE but this time by adding some variability to your dataset by increasing the data augmentation parameters in config.py, for example with the following values:
ROT_ANGLE = 30
W_SHIFT_RANGE = 0.1
H_SHIFT_RANGE = 0.1
FILL_MODE = "nearest"
BRIGHTNESS_RANGE = [0.90, 1.1]
VAL_SPLIT = 0.1
Use the flag --inspect in train.py to generate inspection plots in order to assess the quality of the reconstructions.

If the results are not satisfactory, try using resnetCAE with a batch size of 8, the ssim loss and grayscale and set the flag --inspect:
python3 train.py -d mvtec/pill -a resnetCAE -b 8 -l ssim -c grayscale --inspect

[nguyenanhtuan1008]

I try all mvtec, baseline, inception with your config, the val_mssim increased in training but the testing result not good.

• test results: {'min_area': 920, 'threshold': 0.11400000000000002, 'TPR': 0.3120567375886525, 'TNR': 0.8846153846153846, 'score': 0.5983360611020185, 'method': 'ssim', 'dtype': 'float64'} --> mvtecCAE

• test results: {'min_area': 785, 'threshold': 0.15400000000000005, 'TPR': 0.41134751773049644, 'TNR': 1.0, 'score': 0.7056737588652482, 'method': 'ssim', 'dtype': 'float64'} --> baselineCAE

• inceptionCAE got error:
  Finetune.py:

Traceback (most recent call last):
File "finetune.py", line 360, in
main(args)
File "finetune.py", line 78, in main
preprocessing_function = get_preprocessing_function(architecture)
File "MVTec-Anomaly-Detection/processing/preprocessing.py", line 149, in get_preprocessing_function
return preprocessing_function
UnboundLocalError: local variable 'preprocessing_function' referenced before assignment

• resnetCAE got error when training:

Epoch 9/10
30/30 [==============================] - 8s 260ms/step - loss: nan - ssim: nan

done.
Visually inspect loss plot and select learning rate associated with falling loss

INFO:autoencoder.autoencoder:lr with minimum loss divided by 10: 6.31E-02 INFO:autoencoder.autoencoder:lr with minimum numerical gradient: 1.45E-04 MVTec-Anomaly-Detection/autoencoder/autoencoder.py:231: RuntimeWarning: invalid value encountered in less self.lr_opt_i = np.argwhere(segment < optimal_loss)[0][0] Traceback (most recent call last): File "train.py", line 238, in main(args) File "train.py", line 81, in main autoencoder.find_lr_opt(train_generator, validation_generator) File "MVTec-Anomaly-Detection/autoencoder/autoencoder.py", line 194, in find_lr_opt self.custom_lr_estimate() File "MVTec-Anomaly-Detection/autoencoder/autoencoder.py", line 231, in custom_lr_estimate self.lr_opt_i = np.argwhere(segment < optimal_loss)[0][0] IndexError: index 0 is out of bounds for axis 0 with size 0

[AdneneBoumessouer]

Did you encounter the same problems for the other categories of the MVTec Dataset ?

[nguyenanhtuan1008]

Did you encounter the same problems for the other categories of the MVTec Dataset ?

yes. got the same problem!

[chenghao618]

对于 MVTec 数据集的其他类别，您是否遇到过同样的问题？

是的。遇到了同样的问题！
Did you solve this problem？ I saw that accuracy very low too.