add netty4 metrics. #12695

hujun-w-2 · 2023-07-08T04:22:39Z

What is the purpose of the change

增加netty相关指标，详情：##12271

Brief changelog

监控指标类型

NettyAllocatorMetrics
NettyEventExecutorMetrics

指标详情

"netty.allocator.memory.used" - Size of memory used by the allocator, in bytes
Tags: "id" (unique id for the allocator), "allocator.type" (pooled, unpooled), "memory.type" (heap, direct)
"netty.allocator.memory.pinned" - Size of memory used by allocated buffers, in bytes.
Tags: "id" (unique id for the allocator), "allocator.type" (pooled, unpooled), "memory.type" (heap, direct)
"netty.allocator.pooled.arenas" - Number of Arenas for a pooled allocator.
Tags: "id" (unique id for the allocator), "allocator.type" (pooled, unpooled), "memory.type" (heap, direct)
"netty.allocator.pooled.cache.size" - Size of the cache for a pooled allocator, in bytes.
Tags: "id" (unique id for the allocator), "allocator.type" (pooled, unpooled), "cache.type" (normal, small)
"netty.allocator.pooled.threadlocal.caches" - Number of ThreadLocal caches for a pooled allocator.
Tags: "id" (unique id for the allocator), "allocator.type" (pooled, unpooled)
"netty.allocator.pooled.chunk.size" - Size of memory chunks for a pooled allocator, in bytes.
Tags: "id" (unique id for the allocator), "allocator.type" (pooled, unpooled)
"netty.eventexecutor.tasks.pending" - Number of pending tasks in the event executor.
Tags: "name" (unique name for the event executor)

Verifying this change

Checklist

Make sure there is a GitHub_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GitHub issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
Each commit in the pull request should have a meaningful subject line and body.
Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
Check if is necessary to patch to Dubbo 3 if you are work on Dubbo 2.7
Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
Add some description to dubbo-website project if you are requesting to add a feature.
GitHub Actions works fine on your own branch.
If this contribution is large, please follow the Software Donation Guide.

AlbumenJ

It is best to add a switch to control whether this ability is enabled

…o feature_netty_metrics

dubbo-common/src/main/java/org/apache/dubbo/config/MetricsConfig.java

codecov-commenter · 2023-07-13T14:20:20Z

Codecov Report

Merging #12695 (77da9c6) into 3.3 (6c5a876) will increase coverage by 2.32%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                3.3   #12695      +/-   ##
============================================
+ Coverage     65.81%   68.13%   +2.32%     
  Complexity        6        6              
============================================
  Files          1834     1704     -130     
  Lines         73496    70203    -3293     
  Branches      10437    10184     -253     
============================================
- Hits          48369    47836     -533     
+ Misses        20431    17721    -2710     
+ Partials       4696     4646      -50

see 187 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

AlbumenJ

Please rebase on 3.3 branch, this is a new feature

…o feature_netty_metrics

* RequestEvent optimization * RequestEvent optimization * RequestEvent optimization(apache#12655) * RequestEvent optimization(apache#12655) * fix conflict * code optimize * code optimize * optimize --------- Co-authored-by: songxiaosheng <songxiaosheng@elastic.link>

songxiaosheng · 2023-07-30T05:16:39Z

eartbeat=60000&interface=org.apache.dubbo.spring.boot.api.HelloService&methods=sayHello&pid=9&prefer.serialization=fastjson2,hessian2&qos.enable=false&release=3.2.5-SNAPSHOT&revision=1.0.0&service-name-mapping=true&side=provider&timestamp=1689776520246&token=d21bf92b-dc41-4ff6-a261-d8e8f10b5087&version=1.0.0) Failed to bind NettyServer on /0.0.0.0:20880, cause: io.netty.buffer.PooledByteBufAllocator.metric()Lio/netty/buffer/PooledByteBufAllocatorMetric;
2023-07-19T14:25:16.4850517Z at org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol.createServer(DubboProtocol.java:386) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4851247Z at org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol.openServer(DubboProtocol.java:351) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4852077Z at org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol.export(DubboProtocol.java:332) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4852827Z at org.apache.dubbo.rpc.protocol.ProtocolListenerWrapper.export(ProtocolListenerWrapper.java:73) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4853585Z at org.apache.dubbo.qos.protocol.QosProtocolWrapper.export(QosProtocolWrapper.java:80) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4854325Z at org.apache.dubbo.rpc.protocol.ProtocolSecurityWrapper.export(ProtocolSecurityWrapper.java:83) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4855111Z at org.apache.dubbo.rpc.cluster.filter.ProtocolFilterWrapper.export(ProtocolFilterWrapper.java:61) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4855918Z at org.apache.dubbo.rpc.protocol.ProtocolSerializationWrapper.export(ProtocolSerializationWrapper.java:47) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4856707Z at org.apache.dubbo.rpc.protocol.InvokerCountWrapper.export(InvokerCountWrapper.java:42) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4857356Z at org.apache.dubbo.rpc.Protocol$Adaptive.export(Protocol$Adaptive.java) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4858079Z at org.apache.dubbo.registry.integration.RegistryProtocol.lambda$doLocalExport$3(RegistryProtocol.java:319) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4858877Z at org.apache.dubbo.registry.integration.ExporterFactory.lambda$createExporter$0(ExporterFactory.java:32) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4859315Z ... 41 common frames omitted
2023-07-19T14:25:16.4859841Z Caused by: org.apache.dubbo.remoting.RemotingException: Failed to bind NettyServer on /0.0.0.0:20880, cause: io.netty.buffer.PooledByteBufAllocator.metric()Lio/netty/buffer/PooledByteBufAllocatorMetric;
2023-07-19T14:25:16.4860732Z at org.apache.dubbo.remoting.transport.AbstractServer.(AbstractServer.java:75) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4861421Z at org.apache.dubbo.remoting.transport.netty4.NettyServer.(NettyServer.java:89) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4862233Z at org.apache.dubbo.remoting.transport.netty4.NettyTransporter.bind(NettyTransporter.java:35) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4862936Z at org.apache.dubbo.remoting.Transporter$Adaptive.bind(Transporter$Adaptive.java) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4863556Z at org.apache.dubbo.remoting.Transporters.bind(Transporters.java:48) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4864286Z at org.apache.dubbo.remoting.exchange.support.header.HeaderExchanger.bind(HeaderExchanger.java:52) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4864998Z at org.apache.dubbo.remoting.exchange.Exchangers.bind(Exchangers.java:62) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4865702Z at org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol.createServer(DubboProtocol.java:384) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4866119Z ... 52 common frames omitted
2023-07-19T14:25:16.4866561Z Caused by: java.lang.NoSuchMethodError: io.netty.buffer.PooledByteBufAllocator.metric()Lio/netty/buffer/PooledByteBufAllocatorMetric;
2023-07-19T14:25:16.4867314Z at org.apache.dubbo.remoting.transport.netty4.NettyServer.lambda$doOpen$0(NettyServer.java:121) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4868028Z at org.apache.dubbo.metrics.event.MetricsEventBus.post(MetricsEventBus.java:69) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4868690Z at org.apache.dubbo.metrics.event.MetricsEventBus.post(MetricsEventBus.java:51) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4869374Z at org.apache.dubbo.remoting.transport.netty4.NettyServer.doOpen(NettyServer.java:119) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4870191Z at org.apache.dubbo.remoting.transport.AbstractServer.(AbstractServer.java:69) ~[dubbo-3.2.5-SNAPSHOT.jar:3.2.5-SNAPSHOT]
2023-07-19T14:25:16.4870585Z ... 59 common frames omitted
2023-07-19T14:25:16.4871621Z Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled.
2023-07-19T14:25:16.4872172Z 2023-07-19 14:22:01.284 ERROR 9 --- [ main] o.s.b.d.LoggingFailureAnalysisReporter :
2023-07-19T14:25:16.4872403Z
2023-07-19T14:25:16.4872485Z ***************************
2023-07-19T14:25:16.4872706Z APPLICATION FAILED TO START
2023-07-19T14:25:16.4872918Z ***************************
2023-07-19T14:25:16.4873034Z
2023-07-19T14:25:16.4873118Z Description:
2023-07-19T14:25:16.4873243Z
2023-07-19T14:25:16.4873710Z An attempt was made to call the method io.netty.buffer.PooledByteBufAllocator.metric()Lio/netty/buffer/PooledByteBufAllocatorMetric; but it does not exist. Its class, io.netty.buffer.PooledByteBufAllocator, is available from the following locations:
2023-07-19T14:25:16.4874180Z
2023-07-19T14:25:16.4874530Z jar:file:/usr/local/dubbo/app/dependency/netty-all-4.0.38.Final.jar!/io/netty/buffer/PooledByteBufAllocator.class
2023-07-19T14:25:16.4874798Z
2023-07-19T14:25:16.4874919Z It was loaded from the following location:
2023-07-19T14:25:16.4875080Z
2023-07-19T14:25:16.4875308Z file:/usr/local/dubbo/app/dependency/netty-all-4.0.38.Final.jar
2023-07-19T14:25:16.4875502Z
2023-07-19T14:25:16.4875509Z
2023-07-19T14:25:16.4875583Z Action:
2023-07-19T14:25:16.4875697Z

songxiaosheng · 2023-08-03T15:05:12Z

please fix conflits

# Conflicts: # dubbo-distribution/dubbo-bom/pom.xml # dubbo-metrics/pom.xml

songxiaosheng

LGTM

songxiaosheng · 2023-07-18T11:44:12Z

...bo-remoting-netty4/src/main/java/org/apache/dubbo/remoting/transport/netty4/NettyServer.java

+
+    private boolean isSupportMetrics() {
+        return ClassUtils.isPresent("io.netty.buffer.PooledByteBufAllocatorMetric", NettyServer.class.getClassLoader()) &&
+            ClassUtils.isPresent("io.netty.handler.codec.compression.EncoderUtil", NettyServer.class.getClassLoader());


这里为何要判断EncoderUtil这个类型，没看到后面指标使用了这个类型，

songxiaosheng · 2023-08-05T10:28:51Z

@AlbumenJ PTAL

sonarcloud · 2023-08-07T16:00:26Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
10 Code Smells

39.3% Coverage
0.0% Duplication

netty4 metrics.

094749c

hujun-w-2 changed the title ~~netty4 metrics.~~ add netty4 metrics. Jul 8, 2023

AlbumenJ reviewed Jul 8, 2023

View reviewed changes

hujun-w-2 added 9 commits July 10, 2023 17:18

netty metrics.

55b0e7e

netty4 metrics.

3ed099f

revert demo .

753b6b0

netty4 metrics.

4c057dc

Merge remote-tracking branch 'hujun-github/feature_netty_metrics' int…

357ba58

…o feature_netty_metrics

add module

706e28d

checkstyle

3509fab

new module config

d0380ff

netty dependency

ccba8a1

songxiaosheng requested changes Jul 12, 2023

View reviewed changes

dubbo-common/src/main/java/org/apache/dubbo/config/MetricsConfig.java Show resolved Hide resolved

hujun-w-2 added 5 commits July 13, 2023 13:00

dubbo xsd 更新

a41d69b

Merge branch '3.2' into feature_netty_metrics

fa7833b

netty 包冲突

027a59b

dependency

ed2fd7c

dependency

e7fb9b4

AlbumenJ requested changes Jul 14, 2023

View reviewed changes

songxiaosheng and others added 6 commits July 14, 2023 20:05

Merge branch '3.2' into feature_netty_metrics

99fa07d

Merge branch '3.2' into feature_netty_metrics

e95f142

remove judge

cb09737

Merge remote-tracking branch 'hujun-github/feature_netty_metrics' int…

d3ffc5b

…o feature_netty_metrics

Merge branch '3.2' into feature_netty_metrics

4ade742

test

66e6883

AlbumenJ changed the base branch from 3.2 to 3.3 July 30, 2023 12:22

hujun-w-2 and others added 2 commits August 4, 2023 09:24

Merge remote-tracking branch 'github/3.3' into feature_netty_metrics

77da9c6

# Conflicts: # dubbo-distribution/dubbo-bom/pom.xml # dubbo-metrics/pom.xml

Merge branch '3.3' into feature_netty_metrics

b49d1e8

songxiaosheng approved these changes Aug 4, 2023

View reviewed changes

Merge branch '3.3' into feature_netty_metrics

5a89435

AlbumenJ approved these changes Aug 9, 2023

View reviewed changes

AlbumenJ merged commit 04f3fe7 into apache:3.3 Aug 9, 2023
14 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add netty4 metrics. #12695

add netty4 metrics. #12695

hujun-w-2 commented Jul 8, 2023

AlbumenJ left a comment

codecov-commenter commented Jul 13, 2023 •

edited

Loading

AlbumenJ left a comment

songxiaosheng commented Jul 30, 2023 •

edited

Loading

songxiaosheng commented Aug 3, 2023

songxiaosheng left a comment

songxiaosheng Jul 18, 2023

songxiaosheng commented Aug 5, 2023

sonarcloud bot commented Aug 7, 2023

add netty4 metrics. #12695

add netty4 metrics. #12695

Conversation

hujun-w-2 commented Jul 8, 2023

What is the purpose of the change

Brief changelog

Verifying this change

Checklist

AlbumenJ left a comment

Choose a reason for hiding this comment

codecov-commenter commented Jul 13, 2023 • edited Loading

Codecov Report

AlbumenJ left a comment

Choose a reason for hiding this comment

songxiaosheng commented Jul 30, 2023 • edited Loading

songxiaosheng commented Aug 3, 2023

songxiaosheng left a comment

Choose a reason for hiding this comment

songxiaosheng Jul 18, 2023

Choose a reason for hiding this comment

songxiaosheng commented Aug 5, 2023

sonarcloud bot commented Aug 7, 2023

codecov-commenter commented Jul 13, 2023 •

edited

Loading

songxiaosheng commented Jul 30, 2023 •

edited

Loading