Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: DiskState parsing error #31

Merged
merged 3 commits into from
May 11, 2023
Merged

fix: DiskState parsing error #31

merged 3 commits into from
May 11, 2023

Conversation

chrisgacsal
Copy link
Collaborator

Q A
Bug fix? yes
New feature? no
API breaks? no
Deprecations? no
Related tickets -
License Apache 2.0

What's in this PR?

Calling the KafkaClusterLoad API with PopulateDiskInfo parameter set to true resulted the following error:

failed to convert HTTP response to API response: failed to parse JSON response: json: cannot unmarshal object into Go struct field BrokerLoadStats.loadAfterOptimization.brokers.DiskState

The reason of the error was that the BrokerLoadStats.DiskState had a slice type why it was actually a map. See the JSON response below:

JSON response
{
  "brokers": [
    {
      "FollowerNwInRate": 0.0873900055885315,
      "BrokerState": "ALIVE",
      "Broker": 0,
      "NwOutRate": 0.6877933293581009,
      "DiskState": {
        "/var/lib/kafka/data0": {
          "DiskPct": 0.00036533355712890623,
          "NumLeaderReplicas": 25,
          "DiskMB": 0.36533355712890625,
          "NumReplicas": 48
        },
        "/var/lib/kafka/data1": {
          "DiskPct": 0.0009332275390625,
          "NumLeaderReplicas": 24,
          "DiskMB": 0.9332275390625,
          "NumReplicas": 48
        }
      },
      "NumCore": 1,
      "Host": "kafka-0",
      "CpuPct": 1.7037400007247925,
      "Replicas": 96,
      "NetworkInCapacity": 10000,
      "Rack": "rack-0",
      "Leaders": 49,
      "DiskCapacityMB": 200000,
      "DiskMB": 1.2985610961914062,
      "PnwOutRate": 0.8443229794502258,
      "NetworkOutCapacity": 10000,
      "LeaderNwInRate": 0.5030473470687866,
      "DiskPct": 0.0006492805480957032
    },
    {
      "FollowerNwInRate": 0.28115568310022354,
      "BrokerState": "ALIVE",
      "Broker": 1,
      "NwOutRate": 0.15093320608139038,
      "DiskState": {
        "/var/lib/kafka/data0": {
          "DiskPct": 0.0009337587356567382,
          "NumLeaderReplicas": 26,
          "DiskMB": 0.9337587356567383,
          "NumReplicas": 49
        },
        "/var/lib/kafka/data1": {
          "DiskPct": 0.0001764554977416992,
          "NumLeaderReplicas": 23,
          "DiskMB": 0.17645549774169922,
          "NumReplicas": 48
        }
      },
      "NumCore": 1,
      "Host": "kafka-1",
      "CpuPct": 1.604756236076355,
      "Replicas": 97,
      "NetworkInCapacity": 10000,
      "Rack": "rack-1",
      "Leaders": 49,
      "DiskCapacityMB": 200000,
      "DiskMB": 1.1102142333984375,
      "PnwOutRate": 0.5013578087091446,
      "NetworkOutCapacity": 10000,
      "LeaderNwInRate": 0.07950638979673386,
      "DiskPct": 0.0005551071166992188
    },
    {
      "FollowerNwInRate": 0.309577576816082,
      "BrokerState": "ALIVE",
      "Broker": 2,
      "NwOutRate": 0.16242243349552155,
      "DiskState": {
        "/var/lib/kafka/data0": {
          "DiskPct": 0.0003588676452636719,
          "NumLeaderReplicas": 23,
          "DiskMB": 0.3588676452636719,
          "NumReplicas": 49
        },
        "/var/lib/kafka/data1": {
          "DiskPct": 0.00015853500366210937,
          "NumLeaderReplicas": 24,
          "DiskMB": 0.15853500366210938,
          "NumReplicas": 48
        }
      },
      "NumCore": 1,
      "Host": "kafka-2",
      "CpuPct": 1.5978903770446777,
      "Replicas": 97,
      "NetworkInCapacity": 10000,
      "Rack": "rack-2",
      "Leaders": 47,
      "DiskCapacityMB": 200000,
      "DiskMB": 0.5174026489257812,
      "PnwOutRate": 0.6566172242164612,
      "NetworkOutCapacity": 10000,
      "LeaderNwInRate": 0.09556946903467178,
      "DiskPct": 0.0002587013244628906
    }
  ],
  "hosts": [
    {
      "FollowerNwInRate": 0.0873900055885315,
      "NwOutRate": 0.6877933293581009,
      "NumCore": 1,
      "Host": "kafka-0",
      "CpuPct": 1.7037400007247925,
      "Replicas": 96,
      "NetworkInCapacity": 10000,
      "Rack": "rack-0",
      "Leaders": 49,
      "DiskCapacityMB": 200000,
      "DiskMB": 1.2985610961914062,
      "PnwOutRate": 0.8443229794502258,
      "NetworkOutCapacity": 10000,
      "LeaderNwInRate": 0.5030473470687866,
      "DiskPct": 0.0006492805480957032
    },
    {
      "FollowerNwInRate": 0.28115568310022354,
      "NwOutRate": 0.15093320608139038,
      "NumCore": 1,
      "Host": "kafka-1",
      "CpuPct": 1.604756236076355,
      "Replicas": 97,
      "NetworkInCapacity": 10000,
      "Rack": "rack-1",
      "Leaders": 49,
      "DiskCapacityMB": 200000,
      "DiskMB": 1.1102142333984375,
      "PnwOutRate": 0.5013578087091446,
      "NetworkOutCapacity": 10000,
      "LeaderNwInRate": 0.07950638979673386,
      "DiskPct": 0.0005551071166992188
    },
    {
      "FollowerNwInRate": 0.309577576816082,
      "NwOutRate": 0.16242243349552155,
      "NumCore": 1,
      "Host": "kafka-2",
      "CpuPct": 1.5978903770446777,
      "Replicas": 97,
      "NetworkInCapacity": 10000,
      "Rack": "rack-2",
      "Leaders": 47,
      "DiskCapacityMB": 200000,
      "DiskMB": 0.5174026489257812,
      "PnwOutRate": 0.6566172242164612,
      "NetworkOutCapacity": 10000,
      "LeaderNwInRate": 0.09556946903467178,
      "DiskPct": 0.0002587013244628906
    }
  ],
  "version": 1
}

Checklist

  • Implementation tested
  • Tested against Cruise Control version: x.y.z (if applicable)
  • User guide and development docs updated (if needed)

Add capacity information for all the brokers as having only default set
could result errors on CruiseControl end in certain scenarios.
@chrisgacsal chrisgacsal requested a review from a team as a code owner May 10, 2023 21:24
@chrisgacsal chrisgacsal self-assigned this May 10, 2023
@chrisgacsal chrisgacsal changed the title Fix broker stats fix: DiskState parsing error May 10, 2023
@bartam1
Copy link
Contributor

bartam1 commented May 11, 2023

LGTM! Thank you!

@chrisgacsal chrisgacsal merged commit b7bb540 into main May 11, 2023
@chrisgacsal chrisgacsal deleted the fix-broker-stats branch May 11, 2023 07:17
amuraru pushed a commit to adobe/go-cruise-control that referenced this pull request Dec 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants