Problem with AMD benchmark

[computingdolas]

I am getting the 1/10 flops/s on the AMD Vega architecture as compared to one mentioned in the results folder. Anybody know why ???

[sunway513]

Hi @computingdolas , could you help provide the versions on your ROCm software environment?

[computingdolas]

Thanks @sunway513 for your response. Here is the what rocminfo says 👍

=====================
HSA System Attributes

Runtime Version: 1.1
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (number of timestamp)
Machine Model: LARGE
System Endianness: LITTLE

[sunway513]

Thanks @computingdolas , can you share more information? e.g. the log of:
apt --installed list | grep rocm
Or the following in centos:
rpm -qa | grep rocm

[computingdolas]

Hey @sunway513

See this :)

rocm-clang-ocl-0.3.0_7997136-1.x86_64
rocm-device-libs-0.0.1-1.x86_64
rocm-dev-1.8.192-1.x86_64
rocminfo-1.0.0-1.x86_64
rocm-amdgpu-pro-icd-17.50-552542.el7.x86_64
rocm-opencl-1.2.0-2018071635.x86_64
rocm-libs-1.8.192-1.x86_64
rocm-smi-1.0.0_46_g81ef66f-1.x86_64
rocm-amdgpu-pro-17.50-552542.el7.x86_64
rocm-utils-1.8.192-1.x86_64
rocm-profiler-5.4.6878-g15f6673.x86_64
rocm-opencl-devel-1.2.0-2018071635.x86_64
rocm-dkms-1.8.192-1.x86_64
rocm-amdgpu-pro-opencl-17.50-552542.el7.x86_64

[sunway513]

Great, so you are on the latest ROCm, thanks :-)
I'll try to reproduce your result and update here.

[computingdolas]

Here are my results for gemm benchmark in code/amd folder. matrix flops approximately is 2mn*k and TFLOPS = (flops/time)/10^12. For the first case in this I am getting somewhere around 0.15 TFLOPs but I should according to results folder get 1.5 TFLOPs. Please find the data below 👍
m n k a_t b_t time (usec)
1760 16 1760 n n 644
1760 32 1760 n n 657
1760 64 1760 n n 690
1760 128 1760 n n 692
1760 7000 1760 n n 8626
2048 16 2048 n n 752
2048 32 2048 n n 774
2048 64 2048 n n 881
2048 128 2048 n n 933
2048 7000 2048 n n 11562
2560 16 2560 n n 939
2560 32 2560 n n 963
2560 64 2560 n n 1043
2560 128 2560 n n 1110
2560 7000 2560 n n 17520
4096 16 4096 n n 1523
4096 32 4096 n n 1556
4096 64 4096 n n 1860
4096 128 4096 n n 2314
4096 7000 4096 n n 48839
1760 16 1760 t n 1104
1760 32 1760 t n 1114
1760 64 1760 t n 1173
1760 128 1760 t n 1307
1760 7000 1760 t n 10391
2048 16 2048 t n 1375
2048 32 2048 t n 1404
2048 64 2048 t n 1889
2048 128 2048 t n 2131
2048 7000 2048 t n 14931
2560 16 2560 t n 1853
2560 32 2560 t n 1889
2560 64 2560 t n 2146
2560 128 2560 t n 2324
2560 7000 2560 t n 21081
4096 16 4096 t n 3368
4096 32 4096 t n 3459
4096 64 4096 t n 3660
4096 128 4096 t n 12966
4096 7000 4096 t n 57209
1760 7133 1760 n t 7234
2048 7133 2048 n t 8275
2560 7133 2560 n t 13501
4096 7133 4096 n t 36544
5124 9124 1760 n n 32020
35 8457 1760 n n 985
5124 9124 2048 n n 41212
35 8457 2048 n n 2658
5124 9124 2560 n n 48522
35 8457 2560 n n 1729
5124 9124 4096 n n 82356
35 8457 4096 n n 4522
5124 9124 1760 t n 37142
35 8457 1760 t n 1438
5124 9124 2048 t n 46961
35 8457 2048 t n 2351
5124 9124 2560 t n 54639
35 8457 2560 t n 2441
5124 9124 4096 t n 92358
35 8457 4096 t n 4228
7680 16 2560 n n 989
7680 32 2560 n n 977
7680 64 2560 n n 1162
7680 128 2560 n n 1337
7680 16 2560 t n 2262
7680 32 2560 t n 2257
7680 64 2560 t n 3044
7680 128 2560 t n 3402
3072 16 1024 n n 389
3072 32 1024 n n 399
3072 64 1024 n n 496
3072 128 1024 n n 586
3072 16 1024 t n 882
3072 32 1024 t n 902
3072 64 1024 t n 1034
3072 128 1024 t n 1527
3072 7435 1024 n t 7455
7680 5481 2560 n t 34993
512 8 500000 n n 176088
1024 8 500000 n n 175514
512 16 500000 n n 178471
1024 16 500000 n n 177038
512 8 500000 t n 336153
1024 8 500000 t n 337628
512 16 500000 t n 337572
1024 16 500000 t n 336461
1024 700 512 n n 304
1024 700 512 t n 446
7680 24000 2560 n n 187187
6144 24000 2048 n n 124284
4608 24000 1536 n n 68394
8448 24000 2816 n n 225464
3072 24000 1024 n n 31407
7680 48000 2560 n n 380244
6144 48000 2048 n n 249364
4608 48000 1536 n n 137419
8448 48000 2816 n n 454826
3072 48000 1024 n n 65120

Thank you for your support @sunway513

[sunway513]

Hi @computingdolas, my numbers are very different (much faster) than yours, please find it here:
rocm-deepbench.log

To reproduce my number, please use the following command to run the docker image I've prepared:

alias drun='sudo docker run -it --network=host --device=/dev/kfd --device=/dev/dri --group-add video -v $HOME/dockerx:/dockerx'
drun rocm/deepbench:latest

[computingdolas]

Hey @sunway513 Thank you for your response. Those are nice numbers for AMD GPUs. Why I am getting this issue any idea ?

[computingdolas]

Just to confirm you are still using AMD Vega gfx900. I am using AMD Pro SSG ?

[computingdolas]

Is it the driver problem because I am really confused now ? I am looking non-docker solution. I want to know what happened that these numbers are so bad ?

[sunway513]

Hi @computingdolas , my test GPU is MI25, it's GFX900 based.
Could you firstly try with the docker? If you still get suboptimal performance data, that means your driver stack was not properly configured. If that can boost your performance, then it's a userland software issue -- we can take a further look from there.

[dagamayank]

@computingdolas

I am getting the 1/10 flops/s on the AMD Vega architecture as compared to one mentioned in the results folder.

Please also clarify which GPU are you really using. I saw some reference of AMD Pro SSG, and that is NOT one of the supported deep learning AMD GPUs.

[computingdolas]

hi @sunway513 Ok let's try the docker solution and I will update you in that :)

@dagamayank Hey I am using AMD SSG-PRO which is Vega 10 XT architecture. Are you saying we have good ROCm support for that GPU ? I saw the white paper and the data sheet and I saw many references where they mentioned about this capabilities for deep learning stuff. Can you let me know more about this.

Thanks :)

[computingdolas]

Correction @dagamayank Are you saying we don't have good ROCm support for this GPU ?

[computingdolas]

@sunway513 Is it possible to provide me remote access to your AMD mi25 GPU ?

[sunway513]

@computingdolas , I'm not able to provide public access to the MI25 node. However, you can alternatively try with third-party cloud services using VegaFE:
https://www.gpueater.com/

[computingdolas]

Hey @sunway513 but https://www.gpueater.com/ don't have MI25 GPUs although having Vega frontier edition

[sunway513]

Yes, VegaFE should run ROCm fine with the similar performance as what I've provided in my log for MI25.