support more base Instructions and support resnet #41

zrr1999 · 2023-04-29T11:57:25Z

添加

LOAD_ATTR
LOAD_METHOD
CALL_METHOD
CALL_FUNCTION_KW
BUILD_xxx
BINARY_SUBSCR
STORE_SUBSCR

paddlefx.optimize 可以捕获ResNet模型，将其转为fx graph。

在_compile时跳过 paddle模块里的函数，从而可以支持paddle.add 这种，而不进入执行动态图和静态图的分支

SigureMo · 2023-05-04T05:02:00Z

examples/simple_dynamo.py

@@ -31,6 +31,7 @@ def func(a, b):
 in_a = paddle.rand([3, 4])
 in_b = paddle.rand([3, 4])
 out = paddle.add(in_a, in_b)
+# out = paddle.add(out, out)


无用的注释需要删掉，其他地方还有多处，以及无用的 print

好的，print和一些代码注释我都删掉啦

SigureMo · 2023-05-04T05:03:18Z

examples/simple_dynamo.py

+
+original_res = func(in_a, in_b)
+optimized_res = optimized_func(in_a, in_b)
+np.testing.assert_equal(original_res.numpy(), optimized_res.numpy())


据我所知，目前还是跑原来的字节码吧？所以这里对比貌似没啥意义？@gglin001

可以加一个 NOTE 或者 TODO 在这里～

目前还是跑原来的字节码吧？所以这里对比貌似没啥意义？

是的, 目前的对比只是确保返回了原始的 code, 不是trace 到后转换的 code

据我所知，目前还是跑原来的字节码吧？所以这里对比貌似没啥意义？@gglin001

可以加一个 NOTE 或者 TODO 在这里～

这块我加了一个# TODO(zrr1999): optimized_res is the result of running the converted bytecode in the future.

SigureMo · 2023-05-04T05:07:01Z

src/paddlefx/eval_frame.py

    code = frame.f_code
+    for paddle_module in paddle_modules:
+        if package_name.startswith(paddle_module):
+            return GuardedCode(code)


在_compile时跳过 paddle模块里的函数，从而可以支持paddle.add 这种，而不进入执行动态图和静态图的分支

同上，这是因为目前只能跑原来的字节码，如果跑转换后的字节码理应是不会进入这些函数的 Eval Frame 里的，不过这个 PR 用于验证 ResNet 所需要的字节码的支持完备性是可以暂时这样的～在之后跑转换后的字节码时这部分逻辑应该可以删掉

这块我也新加了个TODO，This part can be removed when running the converted bytecode in the future.

SigureMo · 2023-05-04T05:08:19Z

src/paddlefx/translator.py

+    def pop(self):
+        return self.stack.pop()
+
+    def append(self, item):


根据栈的语义，取名 push 会更好？

代码里若干处 self.stack.append 可统一替换为 self.push

好的，这部分已经替换完毕

SigureMo · 2023-05-04T05:10:37Z

src/paddlefx/translator.py

+    def IS_OP(self, inst: Instruction):
+        args = list(reversed([self.pop() for _ in range(2)]))
+        res = self.output.create_node('call_function', operator.is_, args, {})
+        self.stack.append(res)


这个是否可以统一到 BINARY_MAPPER 呢，看起来可以复用 _binary_constructor？

以及各处的 list(reversed([self.pop() for _ in range(n)])) 逻辑应该可以复用 self.popn(n)？

list(reversed([self.pop() for _ in range(n)]))替换成了self.popn(n, reverse=True)，IS_OP加入了BINARY_MAPPER

SigureMo · 2023-05-04T05:11:45Z

src/paddlefx/translator.py

+            if k == "self":
+                self.f_locals[k] = self.output._proxy_placeholder(k)
+            else:
+                self.f_locals[k] = self.output._proxy_placeholder(k)


emmm，这俩分支有区别嘛？我好像没看粗来:joy:？

这块之前是因为跟_generate_forward那块的self有些冲突，后来实验了很多次，然后忘了改回来，现在是直接在 _generate_forward 里判断有没有self

SigureMo

emmm，我本地尝试跑了一下，dynamo trace 得到的结果貌似有点奇怪？

my_compiler() called with FX graph:
opcode         name        target                             args                                 kwargs
-------------  ----------  ---------------------------------  -----------------------------------  --------
placeholder    self        self                               ()                                   {}
placeholder    x           x                                  ()                                   {}
call_function  getattr_1   <built-in function getattr>        (self, 'conv1')                      {}
call_function  getattr_2   <built-in function getattr>        (getattr_1, 'forward')               {}
call_function  getattr_3   <built-in function getattr>        (getattr_2, '__name__')              {}
call_function  getattr_3   Proxy(getattr_2)                   [Proxy(x)]                           {}
call_function  getattr_4   <built-in function getattr>        (self, 'bn1')                        {}
call_function  getattr_5   <built-in function getattr>        (getattr_4, 'forward')               {}
call_function  getattr_6   <built-in function getattr>        (getattr_5, '__name__')              {}
call_function  getattr_6   Proxy(getattr_5)                   [getattr_3]                          {}
call_function  getattr_7   <built-in function getattr>        (self, 'relu')                       {}
call_function  getattr_8   <built-in function getattr>        (getattr_7, 'forward')               {}
call_function  getattr_9   <built-in function getattr>        (getattr_8, '__name__')              {}
call_function  getattr_9   Proxy(getattr_8)                   [getattr_6]                          {}
call_function  getattr_10  <built-in function getattr>        (self, 'maxpool')                    {}
call_function  getattr_11  <built-in function getattr>        (getattr_10, 'forward')              {}
call_function  getattr_12  <built-in function getattr>        (getattr_11, '__name__')             {}
call_function  getattr_12  Proxy(getattr_11)                  [getattr_9]                          {}
call_function  getattr_13  <built-in function getattr>        (self, 'layer1')                     {}
call_function  getattr_14  <built-in function getattr>        (getattr_13, 'forward')              {}
call_function  getattr_15  <built-in function getattr>        (getattr_14, '__name__')             {}
call_function  getattr_15  Proxy(getattr_14)                  [getattr_12]                         {}
call_function  getattr_16  <built-in function getattr>        (self, 'layer2')                     {}
call_function  getattr_17  <built-in function getattr>        (getattr_16, 'forward')              {}
call_function  getattr_18  <built-in function getattr>        (getattr_17, '__name__')             {}
call_function  getattr_18  Proxy(getattr_17)                  [getattr_15]                         {}
call_function  getattr_19  <built-in function getattr>        (self, 'layer3')                     {}
call_function  getattr_20  <built-in function getattr>        (getattr_19, 'forward')              {}
call_function  getattr_21  <built-in function getattr>        (getattr_20, '__name__')             {}
call_function  getattr_21  Proxy(getattr_20)                  [getattr_18]                         {}
call_function  getattr_22  <built-in function getattr>        (self, 'layer4')                     {}
call_function  getattr_23  <built-in function getattr>        (getattr_22, 'forward')              {}
call_function  getattr_24  <built-in function getattr>        (getattr_23, '__name__')             {}
call_function  getattr_24  Proxy(getattr_23)                  [getattr_21]                         {}
call_function  getattr_25  <built-in function getattr>        (self, 'avgpool')                    {}
call_function  getattr_26  <built-in function getattr>        (getattr_25, 'forward')              {}
call_function  getattr_27  <built-in function getattr>        (getattr_26, '__name__')             {}
call_function  getattr_27  Proxy(getattr_26)                  [getattr_24]                         {}
call_function  gt          <built-in function gt>             [0, Proxy(getattr_31)]               {}
call_function  flatten_1   <function flatten at 0x125377280>  [1, getattr_27]                      {}
call_function  getattr_28  <built-in function getattr>        (self, 'fc')                         {}
call_function  getattr_29  <built-in function getattr>        (getattr_28, 'forward')              {}
call_function  getattr_30  <built-in function getattr>        (getattr_29, '__name__')             {}
call_function  getattr_30  Proxy(getattr_29)                  [flatten_1]                          {}
output         output      output                             [Proxy(getattr_32), gt, getattr_30]  {}
call_function  getattr_31  <built-in function getattr>        (self, 'num_classes')                {}
call_function  getattr_32  <built-in function getattr>        (self, 'with_pool')                  {}

好像和直接 FX trace 出来的差的有点多，my_compiler 里加上 gl.forward(paddle.rand([2, 3, 224, 224])) 好像也跑不起来？

而且为什么 args 里既有 tuple 又有 list 呢？

SigureMo · 2023-05-04T10:58:57Z

src/paddlefx/translator.py

+            self.push(None)
+
+    def CALL_FUNCTION(self, inst: Instruction):
+        args = [self.pop() for _ in range(inst.argval)]


这里是不是反了？貌似没有地方有 reverse=False 的需求？

比如对于如下 function call：

import dis def foo(): bar(1, 2, 3) dis.dis(foo)

字节码如下：

24 0 LOAD_GLOBAL 0 (bar) 2 LOAD_CONST 1 (1) 4 LOAD_CONST 2 (2) 6 LOAD_CONST 3 (3) 8 CALL_FUNCTION 3 10 POP_TOP 12 LOAD_CONST 0 (None) 14 RETURN_VALUE

参数 1、2、3 依次入栈，依次出栈将会是反的，因此需要 reverse 才可以，BUILD_TUPLE 等都是相同的，入栈是依次的，出栈后需要 reverse

细节可以参考 dis 文档、Python - ceval.c source

好的，这块我已经都改成统一的反过来出栈啦

zrr1999 · 2023-05-04T16:24:00Z

emmm，我本地尝试跑了一下，dynamo trace 得到的结果貌似有点奇怪？

my_compiler() called with FX graph:
opcode         name        target                             args                                 kwargs
-------------  ----------  ---------------------------------  -----------------------------------  --------
placeholder    self        self                               ()                                   {}
placeholder    x           x                                  ()                                   {}
call_function  getattr_1   <built-in function getattr>        (self, 'conv1')                      {}
call_function  getattr_2   <built-in function getattr>        (getattr_1, 'forward')               {}
call_function  getattr_3   <built-in function getattr>        (getattr_2, '__name__')              {}
call_function  getattr_3   Proxy(getattr_2)                   [Proxy(x)]                           {}
call_function  getattr_4   <built-in function getattr>        (self, 'bn1')                        {}
call_function  getattr_5   <built-in function getattr>        (getattr_4, 'forward')               {}
call_function  getattr_6   <built-in function getattr>        (getattr_5, '__name__')              {}
call_function  getattr_6   Proxy(getattr_5)                   [getattr_3]                          {}
call_function  getattr_7   <built-in function getattr>        (self, 'relu')                       {}
call_function  getattr_8   <built-in function getattr>        (getattr_7, 'forward')               {}
call_function  getattr_9   <built-in function getattr>        (getattr_8, '__name__')              {}
call_function  getattr_9   Proxy(getattr_8)                   [getattr_6]                          {}
call_function  getattr_10  <built-in function getattr>        (self, 'maxpool')                    {}
call_function  getattr_11  <built-in function getattr>        (getattr_10, 'forward')              {}
call_function  getattr_12  <built-in function getattr>        (getattr_11, '__name__')             {}
call_function  getattr_12  Proxy(getattr_11)                  [getattr_9]                          {}
call_function  getattr_13  <built-in function getattr>        (self, 'layer1')                     {}
call_function  getattr_14  <built-in function getattr>        (getattr_13, 'forward')              {}
call_function  getattr_15  <built-in function getattr>        (getattr_14, '__name__')             {}
call_function  getattr_15  Proxy(getattr_14)                  [getattr_12]                         {}
call_function  getattr_16  <built-in function getattr>        (self, 'layer2')                     {}
call_function  getattr_17  <built-in function getattr>        (getattr_16, 'forward')              {}
call_function  getattr_18  <built-in function getattr>        (getattr_17, '__name__')             {}
call_function  getattr_18  Proxy(getattr_17)                  [getattr_15]                         {}
call_function  getattr_19  <built-in function getattr>        (self, 'layer3')                     {}
call_function  getattr_20  <built-in function getattr>        (getattr_19, 'forward')              {}
call_function  getattr_21  <built-in function getattr>        (getattr_20, '__name__')             {}
call_function  getattr_21  Proxy(getattr_20)                  [getattr_18]                         {}
call_function  getattr_22  <built-in function getattr>        (self, 'layer4')                     {}
call_function  getattr_23  <built-in function getattr>        (getattr_22, 'forward')              {}
call_function  getattr_24  <built-in function getattr>        (getattr_23, '__name__')             {}
call_function  getattr_24  Proxy(getattr_23)                  [getattr_21]                         {}
call_function  getattr_25  <built-in function getattr>        (self, 'avgpool')                    {}
call_function  getattr_26  <built-in function getattr>        (getattr_25, 'forward')              {}
call_function  getattr_27  <built-in function getattr>        (getattr_26, '__name__')             {}
call_function  getattr_27  Proxy(getattr_26)                  [getattr_24]                         {}
call_function  gt          <built-in function gt>             [0, Proxy(getattr_31)]               {}
call_function  flatten_1   <function flatten at 0x125377280>  [1, getattr_27]                      {}
call_function  getattr_28  <built-in function getattr>        (self, 'fc')                         {}
call_function  getattr_29  <built-in function getattr>        (getattr_28, 'forward')              {}
call_function  getattr_30  <built-in function getattr>        (getattr_29, '__name__')             {}
call_function  getattr_30  Proxy(getattr_29)                  [flatten_1]                          {}
output         output      output                             [Proxy(getattr_32), gt, getattr_30]  {}
call_function  getattr_31  <built-in function getattr>        (self, 'num_classes')                {}
call_function  getattr_32  <built-in function getattr>        (self, 'with_pool')                  {}

好像和直接 FX trace 出来的差的有点多，my_compiler 里加上 gl.forward(paddle.rand([2, 3, 224, 224])) 好像也跑不起来？

而且为什么 args 里既有 tuple 又有 list 呢？

python codegen这块貌似还有些问题，我明天看看能不能搞明白，现在output后面还有其他的node就很奇怪。print_table我修复了一下，之前用成了call_function，现在是call_module应该是跟resnet_trace里的基本一致了，args的我在graph加上了转换tuple的代码

zrr1999 · 2023-05-04T17:12:05Z

emmm，我本地尝试跑了一下，dynamo trace 得到的结果貌似有点奇怪？

my_compiler() called with FX graph:
opcode         name        target                             args                                 kwargs
-------------  ----------  ---------------------------------  -----------------------------------  --------
placeholder    self        self                               ()                                   {}
placeholder    x           x                                  ()                                   {}
call_function  getattr_1   <built-in function getattr>        (self, 'conv1')                      {}
call_function  getattr_2   <built-in function getattr>        (getattr_1, 'forward')               {}
call_function  getattr_3   <built-in function getattr>        (getattr_2, '__name__')              {}
call_function  getattr_3   Proxy(getattr_2)                   [Proxy(x)]                           {}
call_function  getattr_4   <built-in function getattr>        (self, 'bn1')                        {}
call_function  getattr_5   <built-in function getattr>        (getattr_4, 'forward')               {}
call_function  getattr_6   <built-in function getattr>        (getattr_5, '__name__')              {}
call_function  getattr_6   Proxy(getattr_5)                   [getattr_3]                          {}
call_function  getattr_7   <built-in function getattr>        (self, 'relu')                       {}
call_function  getattr_8   <built-in function getattr>        (getattr_7, 'forward')               {}
call_function  getattr_9   <built-in function getattr>        (getattr_8, '__name__')              {}
call_function  getattr_9   Proxy(getattr_8)                   [getattr_6]                          {}
call_function  getattr_10  <built-in function getattr>        (self, 'maxpool')                    {}
call_function  getattr_11  <built-in function getattr>        (getattr_10, 'forward')              {}
call_function  getattr_12  <built-in function getattr>        (getattr_11, '__name__')             {}
call_function  getattr_12  Proxy(getattr_11)                  [getattr_9]                          {}
call_function  getattr_13  <built-in function getattr>        (self, 'layer1')                     {}
call_function  getattr_14  <built-in function getattr>        (getattr_13, 'forward')              {}
call_function  getattr_15  <built-in function getattr>        (getattr_14, '__name__')             {}
call_function  getattr_15  Proxy(getattr_14)                  [getattr_12]                         {}
call_function  getattr_16  <built-in function getattr>        (self, 'layer2')                     {}
call_function  getattr_17  <built-in function getattr>        (getattr_16, 'forward')              {}
call_function  getattr_18  <built-in function getattr>        (getattr_17, '__name__')             {}
call_function  getattr_18  Proxy(getattr_17)                  [getattr_15]                         {}
call_function  getattr_19  <built-in function getattr>        (self, 'layer3')                     {}
call_function  getattr_20  <built-in function getattr>        (getattr_19, 'forward')              {}
call_function  getattr_21  <built-in function getattr>        (getattr_20, '__name__')             {}
call_function  getattr_21  Proxy(getattr_20)                  [getattr_18]                         {}
call_function  getattr_22  <built-in function getattr>        (self, 'layer4')                     {}
call_function  getattr_23  <built-in function getattr>        (getattr_22, 'forward')              {}
call_function  getattr_24  <built-in function getattr>        (getattr_23, '__name__')             {}
call_function  getattr_24  Proxy(getattr_23)                  [getattr_21]                         {}
call_function  getattr_25  <built-in function getattr>        (self, 'avgpool')                    {}
call_function  getattr_26  <built-in function getattr>        (getattr_25, 'forward')              {}
call_function  getattr_27  <built-in function getattr>        (getattr_26, '__name__')             {}
call_function  getattr_27  Proxy(getattr_26)                  [getattr_24]                         {}
call_function  gt          <built-in function gt>             [0, Proxy(getattr_31)]               {}
call_function  flatten_1   <function flatten at 0x125377280>  [1, getattr_27]                      {}
call_function  getattr_28  <built-in function getattr>        (self, 'fc')                         {}
call_function  getattr_29  <built-in function getattr>        (getattr_28, 'forward')              {}
call_function  getattr_30  <built-in function getattr>        (getattr_29, '__name__')             {}
call_function  getattr_30  Proxy(getattr_29)                  [flatten_1]                          {}
output         output      output                             [Proxy(getattr_32), gt, getattr_30]  {}
call_function  getattr_31  <built-in function getattr>        (self, 'num_classes')                {}
call_function  getattr_32  <built-in function getattr>        (self, 'with_pool')                  {}

好像和直接 FX trace 出来的差的有点多，my_compiler 里加上 gl.forward(paddle.rand([2, 3, 224, 224])) 好像也跑不起来？

而且为什么 args 里既有 tuple 又有 list 呢？

现在应该可以了

gglin001

LGTM, 👍

SigureMo

LGTM~

translate more bytecode ops

ce34e9b

zrr1999 changed the title ~~translate more bytecode ops~~ support more base Instructions Apr 29, 2023

zrr1999 added 5 commits April 29, 2023 15:34

revert

2ac5907

add some instructions

9504f12

add test

8d22965

fix bug

f082955

pass paddle op

6f806e5

zrr1999 marked this pull request as ready for review April 30, 2023 07:49

zrr1999 mentioned this pull request Apr 30, 2023

[WIP] TODO list #2

Closed

16 tasks

zrr1999 added 2 commits April 30, 2023 09:23

replace self.stack.pop() to self.pop()

5f21cb1

pass paddle_modules

5598d96

zrr1999 marked this pull request as draft May 1, 2023 08:26

zrr1999 added 2 commits May 1, 2023 10:18

support module

590ebae

support resnet

75b8aac

zrr1999 marked this pull request as ready for review May 1, 2023 10:37

zrr1999 changed the title ~~support more base Instructions~~ support more base Instructions and support resnet May 1, 2023

fix bug

b561392

zrr1999 marked this pull request as draft May 1, 2023 10:49

zrr1999 marked this pull request as ready for review May 1, 2023 10:49

zrr1999 added 2 commits May 1, 2023 10:50

test

c7283d9

add pip update

dca017f

zrr1999 force-pushed the translate branch from 3a4a1e4 to dca017f Compare May 1, 2023 10:59

zrr1999 added 4 commits May 1, 2023 11:00

print

c0c7606

fix bug in codegen

b0c3759

fix bug

245ce49

fix bug when fn is a proxy

e584690

gglin001 requested a review from jzhang533 May 4, 2023 02:53

SigureMo reviewed May 4, 2023

View reviewed changes

zrr1999 added 2 commits May 4, 2023 09:57

unify pop and push

76c9739

merge IS_OP to BINARY_OP

f4c3e1e

zrr1999 added 3 commits May 4, 2023 10:06

remove unused print and note

53bde89

add todo

4e4ff49

fix typo

c644f49

zrr1999 requested a review from SigureMo May 4, 2023 10:26

SigureMo reviewed May 4, 2023

View reviewed changes

zrr1999 added 4 commits May 4, 2023 13:06

add call_module method in graph

0e2c2e6

update popn

d92f263

use call_module

ae09773

unify args to tuple

385ef3b

fix bug

f838b77

zrr1999 added 2 commits May 5, 2023 05:01

fix bug

71a76e0

fix bug in SUBSCR

fbcb26b

gglin001 approved these changes May 5, 2023

View reviewed changes

SigureMo approved these changes May 5, 2023

View reviewed changes

gglin001 merged commit 83b9fdc into PFCCLab:main May 5, 2023

Ligoml mentioned this pull request May 11, 2023

中国软件开源创新大赛：飞桨框架任务挑战赛（上） PaddlePaddle/Paddle#53172

Closed

This was referenced May 17, 2023

[WIP] 赛题一：实现飞桨框架的 python level jit compiler #45

Closed

任务一：添加更多字节码转换 #46

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support more base Instructions and support resnet #41

support more base Instructions and support resnet #41

zrr1999 commented Apr 29, 2023 •

edited

Loading

SigureMo May 4, 2023

zrr1999 May 4, 2023

SigureMo May 4, 2023

gglin001 May 4, 2023

zrr1999 May 4, 2023

SigureMo May 4, 2023

zrr1999 May 4, 2023

SigureMo May 4, 2023

zrr1999 May 4, 2023

SigureMo May 4, 2023

SigureMo May 4, 2023

zrr1999 May 4, 2023

SigureMo May 4, 2023

zrr1999 May 4, 2023

SigureMo left a comment

SigureMo May 4, 2023 •

edited

Loading

zrr1999 May 4, 2023

zrr1999 commented May 4, 2023

zrr1999 commented May 4, 2023 •

edited

Loading

gglin001 left a comment

SigureMo left a comment

support more base Instructions and support resnet #41

support more base Instructions and support resnet #41

Conversation

zrr1999 commented Apr 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SigureMo left a comment

Choose a reason for hiding this comment

SigureMo May 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zrr1999 commented May 4, 2023

zrr1999 commented May 4, 2023 • edited Loading

gglin001 left a comment

Choose a reason for hiding this comment

SigureMo left a comment

Choose a reason for hiding this comment

zrr1999 commented Apr 29, 2023 •

edited

Loading

SigureMo May 4, 2023 •

edited

Loading

zrr1999 commented May 4, 2023 •

edited

Loading