[GPU/OpenCL] Initial version of Rotary Embedding Kernel for GPU and generalization via Attention Interface @ open sesame 10/18 11:30 #2721

yashSingh0723 · 2024-08-28T12:24:25Z

Added initial version of Rotary Embedding kernel for GPU. This includes both FP32 and FP16 implementation for GPU kernel.
Changes added with this PR:

Created attention_kernel_interface to handle pre processing.
Implemeneted OpenCL rotary_embedding kernel for GPU for both fp32 anf fp16 usecase.
Refactored apply_rotary_emb_cl and apply_rotary_emb_cl_fp16 for generalization.
Added unittest_attention_kernels_cl.cpp to test Attention kernels on GPU.
Added testing_rotary_emb to test rotary embedding on CPU.
Added attention_kernel_strings.h to handle attention kernels at one place.
Modified attention kernels to remove cl_context related dependencies.
Added initAttentionCLKernels function to register default attention kernels.

taos-ci · 2024-08-28T12:24:28Z

📝 TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #2721. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/.

taos-ci · 2024-08-28T12:24:32Z

cibot: @yashSingh0723, The last line of a text file must have a newline character. Please append a new line at the end of the line in nntrainer/tensor/cl_operations/attention_kernel_interface.cpp.

taos-ci

@yashSingh0723, 💯 All CI checkers are successfully verified. Thanks.

skykongkong8

Found some fixes like below:

skykongkong8 · 2024-08-29T01:12:17Z

nntrainer/tensor/cl_operations/attention_kernels.cpp

+                transformed_value = input[b * channel * height * width + c * height * width + h * width + span - half_];
+              }
+              value = value * cos_ptr[k] + transformed_value * sin_ptr[k];
+              // printf("GPU Batch: %u, Height: %u, Channel: %u, Width: %u, K: %u, Span: %u, Value: %f, Transformed Value: %f, cos_ptr[k]: %f, sin_ptr[k]: %f\n",  b, h, c, w, k, span, value, transformed_value, cos_ptr[k], sin_ptr[k]);


how about removing unnecessary commented print codes

Yes, thank you pointing it out. I've removed all the unecessary comments from the code.

skykongkong8 · 2024-08-29T01:13:47Z

nntrainer/tensor/cl_operations/testing_rotarty_emb.cpp

+#ifdef USE_NEON
+      nntrainer::calc_trigonometric_vals_dup(half_, freqs.data(), cos[i].data(),
+                                             sin[i].data(), i);
+#else


Is using NEON during cl operations possible?

No, actually I took this code for rotary emb from the felice repo. This file was just to test rotary emb on the cpu and to run the unittests. Regardless of it, I've updated and removed this code.

skykongkong8 · 2024-08-29T01:14:11Z

nntrainer/tensor/cl_operations/testing_rotarty_emb.cpp

+                // printf("CPU Batch: %u, Channel: %u, Height: %u, Width: %u, K:
+                // %u, Span: %u, Value: %f, Transformed Value: %f, cos_ptr[k]:
+                // %f, sin_ptr[k]: %f\n ",  b, c, h, w, k, span, value,
+                // transformed_value, cos_[k], sin_[k]);


Think this should be removed

Yes, removed.

skykongkong8 · 2024-08-29T01:14:33Z

test/unittest/unittest_attention_kernels_cl.cpp

+  // std::cout << "\nA_fp32 and B_fp32 before rotary embedding:" << std::endl;
+  // for (unsigned int i = 0; i < A_fp32.size(); ++i) {
+  //   std::cout << "Element " << i << " -> " << *(A_fp32.getData<float>() + i)
+  //   <<"\t"<<*(B_fp32.getData<float>() + i)<< std::endl;
+  // }


Think this should be removed

baek2sm · 2024-08-29T02:24:58Z

nntrainer/tensor/cl_operations/attention_kernels_fp16.cpp

+
+    result = cosBuf.WriteData(context.command_queue_inst_, cos_.data());
+    if (!result) {
+      printf("Failed to write cos data\n");


Like the "Failed to set ~" log, how about making the freqs_cosBuf and cosBuf logs different in the "Failed to write ~" log?

Yeah sure, Made the necessary changes in the latest commit. Thanks for pointing it out.

taos-ci · 2024-10-07T12:58:13Z

cibot: @yashSingh0723, A builder checker could not be completed because one of the checkers is not completed. In order to find out a reason, please go to http://ci.nnstreamer.ai/nntrainer/ci/repo-workers/pr-checker/2721-202410072116300.93544793128967-d8dc40410a4be4688abadedc5601c6551b5b4cef/.

taos-ci

@yashSingh0723, 💯 All CI checkers are successfully verified. Thanks.

taos-ci

@yashSingh0723, 💯 All CI checkers are successfully verified. Thanks.

taos-ci

@yashSingh0723, 💯 All CI checkers are successfully verified. Thanks.

jijoongmoon · 2024-10-16T02:16:35Z

nntrainer/tensor/cl_operations/testing_rotarty_emb.cpp

+/**
+ * Copyright (C) 2024 Yash Singh <yash.singh@samsung.com>
+ *
+ * @file	testing_rotary_emb.cpp


if this file is for the testing.. then it might be better to be in unittest dir.

Moved the file to unittest directory in the latest commit.

taos-ci

@yashSingh0723, 💯 All CI checkers are successfully verified. Thanks.

taos-ci

@yashSingh0723, 💯 All CI checkers are successfully verified. Thanks.

Added initial version of Rotary Embedding kernel for GPU. This includes both FP32 and FP16 implementation got GPU kernel. Signed-off-by: Yash Singh <yash.singh@samsung.com>

Added newline at the end in new files. Signed-off-by: Yash Singh <yash.singh@samsung.com>

Comments removed from the code. Signed-off-by: Yash Singh <yash.singh@samsung.com>

Kernel Optimized for GPU. Some trivial changes in code. Signed-off-by: Yash Singh <yash.singh@samsung.com>

…ependency Added registerCLKernel function to register custom OpenCL kernels as well as in-house kernels. Modified attention kernels to remove cl_context related dependencies. Added initAttentionCLKernels function to register default attention kernels. Modified unittest to remove layer_context dependency attention_kernel_strings.h added to handle attention kernels at one place. Rebased the PR with current log. Signed-off-by: Yash Singh <yash.singh@samsung.com>

taos-ci

@yashSingh0723, 💯 All CI checkers are successfully verified. Thanks.

Moved testing_rotary_emb.cpp to unittest Directory. Signed-off-by: Yash Singh <yash.singh@samsung.com>

taos-ci

@yashSingh0723, 💯 All CI checkers are successfully verified. Thanks.

jijoongmoon

We might need more code restructuring, but we could do it in the later PR. LGTM

yashSingh0723 requested review from myungjoo, jijoongmoon, again4you, jaeyun-jung, leemgs, wooksong, helloahn, kparichay, gichan-jang, anyj0527, zhoonit, lhs8928, songgot, jihochu, DonghakPark, SeoHyungjun, baek2sm, skykongkong8, djeong20, EunjuYang and a team as code owners August 28, 2024 12:24

github-actions bot added the Need Review label Aug 28, 2024

yashSingh0723 changed the title ~~[GPU/OpenCL] Initial version of Rotary Embedding Kernel for GPU and generalization via Attentiosn Kernel~~ [GPU/OpenCL] Initial version of Rotary Embedding Kernel for GPU and generalization via Attention Kernel Aug 28, 2024

yashSingh0723 changed the title ~~[GPU/OpenCL] Initial version of Rotary Embedding Kernel for GPU and generalization via Attention Kernel~~ [GPU/OpenCL] Initial version of Rotary Embedding Kernel for GPU and generalization via Attention Interface Aug 28, 2024

taos-ci approved these changes Aug 28, 2024

View reviewed changes

skykongkong8 reviewed Aug 29, 2024

View reviewed changes

baek2sm reviewed Aug 29, 2024

View reviewed changes

yashSingh0723 force-pushed the opencl_layer_rotaryEmb_kernel branch from 4a71cc7 to 9eaca55 Compare August 29, 2024 06:49

yashSingh0723 force-pushed the opencl_layer_rotaryEmb_kernel branch from 1b0be27 to d8dc404 Compare October 7, 2024 12:16

taos-ci approved these changes Oct 8, 2024

View reviewed changes

yashSingh0723 force-pushed the opencl_layer_rotaryEmb_kernel branch 2 times, most recently from 7c0fac7 to 3e0e6af Compare October 8, 2024 08:39

taos-ci approved these changes Oct 8, 2024

View reviewed changes

yashSingh0723 force-pushed the opencl_layer_rotaryEmb_kernel branch from 3e0e6af to 34dcdfb Compare October 15, 2024 06:40

taos-ci approved these changes Oct 15, 2024

View reviewed changes

jijoongmoon reviewed Oct 16, 2024

View reviewed changes

taos-ci approved these changes Oct 16, 2024

View reviewed changes

yashSingh0723 force-pushed the opencl_layer_rotaryEmb_kernel branch from 2ca11d0 to c018711 Compare October 16, 2024 10:27

taos-ci approved these changes Oct 16, 2024

View reviewed changes

yashSingh0723 added 5 commits October 17, 2024 11:45

[GPU/OpenCL] Initial version of Rotary Embedding with OpenCL ops

ab72f73

Added initial version of Rotary Embedding kernel for GPU. This includes both FP32 and FP16 implementation got GPU kernel. Signed-off-by: Yash Singh <yash.singh@samsung.com>

[Trivial] New line at the end

a23e29d

Added newline at the end in new files. Signed-off-by: Yash Singh <yash.singh@samsung.com>

[Trivial] Unnecessary comments Removed

473231c

Comments removed from the code. Signed-off-by: Yash Singh <yash.singh@samsung.com>

[GPU/OpenCl] Kernel optimization

46e17c1

Kernel Optimized for GPU. Some trivial changes in code. Signed-off-by: Yash Singh <yash.singh@samsung.com>

yashSingh0723 force-pushed the opencl_layer_rotaryEmb_kernel branch from c018711 to 813066f Compare October 17, 2024 06:23

yashSingh0723 changed the title ~~[GPU/OpenCL] Initial version of Rotary Embedding Kernel for GPU and generalization via Attention Interface~~ [GPU/OpenCL] Initial version of Rotary Embedding Kernel for GPU and generalization via Attention Interface @ open sesame 10/17 12:09 Oct 17, 2024

taos-ci approved these changes Oct 17, 2024

View reviewed changes

[Trivial] Moved Testing Rotary Embedding to Unittest Dir

2007b91

Moved testing_rotary_emb.cpp to unittest Directory. Signed-off-by: Yash Singh <yash.singh@samsung.com>

yashSingh0723 force-pushed the opencl_layer_rotaryEmb_kernel branch from 813066f to 2007b91 Compare October 17, 2024 08:25

taos-ci approved these changes Oct 17, 2024

View reviewed changes

jijoongmoon added BUILD/CI and removed rebase required labels Oct 18, 2024

jijoongmoon approved these changes Oct 20, 2024

View reviewed changes

jijoongmoon merged commit 89e8b7c into nnstreamer:main Oct 20, 2024
39 of 43 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GPU/OpenCL] Initial version of Rotary Embedding Kernel for GPU and generalization via Attention Interface @ open sesame 10/18 11:30 #2721

[GPU/OpenCL] Initial version of Rotary Embedding Kernel for GPU and generalization via Attention Interface @ open sesame 10/18 11:30 #2721

yashSingh0723 commented Aug 28, 2024 •

edited

Loading

taos-ci commented Aug 28, 2024

taos-ci commented Aug 28, 2024

taos-ci left a comment

skykongkong8 left a comment

skykongkong8 Aug 29, 2024

yashSingh0723 Aug 29, 2024

skykongkong8 Aug 29, 2024

yashSingh0723 Aug 29, 2024

skykongkong8 Aug 29, 2024

yashSingh0723 Aug 29, 2024

skykongkong8 Aug 29, 2024

baek2sm Aug 29, 2024

yashSingh0723 Aug 29, 2024

taos-ci commented Oct 7, 2024

taos-ci left a comment

taos-ci left a comment

taos-ci left a comment

jijoongmoon Oct 16, 2024

yashSingh0723 Oct 16, 2024

taos-ci left a comment

taos-ci left a comment

taos-ci left a comment

taos-ci left a comment

jijoongmoon left a comment

[GPU/OpenCL] Initial version of Rotary Embedding Kernel for GPU and generalization via Attention Interface @ open sesame 10/18 11:30 #2721

[GPU/OpenCL] Initial version of Rotary Embedding Kernel for GPU and generalization via Attention Interface @ open sesame 10/18 11:30 #2721

Conversation

yashSingh0723 commented Aug 28, 2024 • edited Loading

taos-ci commented Aug 28, 2024

taos-ci commented Aug 28, 2024

taos-ci left a comment

Choose a reason for hiding this comment

skykongkong8 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

taos-ci commented Oct 7, 2024

taos-ci left a comment

Choose a reason for hiding this comment

taos-ci left a comment

Choose a reason for hiding this comment

taos-ci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

taos-ci left a comment

Choose a reason for hiding this comment

taos-ci left a comment

Choose a reason for hiding this comment

taos-ci left a comment

Choose a reason for hiding this comment

taos-ci left a comment

Choose a reason for hiding this comment

jijoongmoon left a comment

Choose a reason for hiding this comment

yashSingh0723 commented Aug 28, 2024 •

edited

Loading