-
Notifications
You must be signed in to change notification settings - Fork 581
REST APIs
- Asynchronous Endpoints
- GET Requests
-
POST Requests
- Trigger a workload balance
- Add a list of new brokers to Kafka Cluster
- Decommission a list of brokers from the Kafka cluster
- Fix offline replicas in Kafka cluster
- Demote a list of brokers from the Kafka cluster
- Stop the current proposal execution task
- Pause metrics load sampling
- Resume metrics load sampling
- Change Kafka topic configuration
- Change Cruise Control configuration
- 2-step Verification
- Rightsize the cluster with the Provisioner
Please ensure proper use of UUIDs or cookies to interact with async endpoints.
Selected Cruise Control (CC) endpoints accept async calls to avoid blocking more than a configured period of time
(via webserver.request.maxBlockTimeMs
configuration).
If the server-side processing of such requests takes more than this configured time, along with a with SC_ACCEPTED
code,
CC returns a progress response.
This response contains both (1) a sessionId
, and (2) a UUID corresponding to the client cookie and the request, respectively.
The completed response of an in-progress request can be retrieved within a predefined time.
Using cookies, this response can be retrieved before the timeout is configured via webserver.session.maxExpiryTimeMs
.
Using the UUID, the timeout is configured via relevant completed user task retention time configuration
(see completed.kafka.monitor.user.task.retention.time.ms
, completed.cruise.control.monitor.user.task.retention.time.ms
,
completed.kafka.admin.user.task.retention.time.ms
, completed.cruise.control.admin.user.task.retention.time.ms
,
and completed.user.task.retention.time.ms
).
Here is an example how to use UUID with requests using cURL
:
- Create a new request
curl -vv -X POST "http://CRUISE_CONTROL_HOST:9090/kafkacruisecontrol/remove_broker?brokerid=1234&dryrun=false"
-
Retrieve the
User-Task-ID
from response header, e.g.User-Task-ID: 5ce7c299-53b3-48b6-b72e-6623e25bd9a8
-
Specifying the
User-Task-ID
in request that has not completed
curl -vv -X POST -H "User-Task-ID: 5ce7c299-53b3-48b6-b72e-6623e25bd9a8" "http://CRUISE_CONTROL_HOST:9090/kafkacruisecontrol/remove_broker?brokerid=1234&dryrun=false"
Here is an example of how to use cookies with requests using cURL
:
- Create a cookie associated with a new request
curl -X POST -c /tmp/mycookie-jar.txt "http://CRUISE_CONTROL_HOST:9090/kafkacruisecontrol/remove_broker?brokerid=1234&dryrun=false"
- Use an existing cookie from the created file for a request that has not completed
curl -X POST -b /tmp/mycookie-jar.txt "http://CRUISE_CONTROL_HOST:9090/kafkacruisecontrol/remove_broker?brokerid=1234&dryrun=false"
Note that a User-Task-ID
or a sessionId
and is applicable for an entire URL
, including its parameters. Hence, the same endpoint with different parameters would create and use a different User-Task-Id
.
The GET requests in Kafka Cruise Control REST API are for read only operations, i.e. the operations that do not have any external impacts. The GET requests include the following operations:
- Query the state of Cruise Control
- Query the current cluster load
- Query partition resource utilization
- Query partition and replica state
- Get optimization proposals
- Query the user request result
User can query the state of Kafka Cruise Control at any time by issuing a HTTP GET request.
GET /kafkacruisecontrol/state
Supported parameters are:
PARAMETER | TYPE | DESCRIPTION | DEFAULT | OPTIONAL |
---|---|---|---|---|
substates | list | substates for which to retrieve state from cruise-control, available substates are analyzer , monitor , executor and anomaly_detector
|
all substates | yes |
json | boolean | return in JSON format or not | false | yes |
verbose | boolean | return detailed state information | false | yes |
super_verbose | boolean | return more detailed state information | false | yes |
doAs | string | propagated user by the trusted proxy service | null | yes |
reason | string | reason for the request | "No reason provided" | yes |
The returned state contains the following information:
-
Monitor State:
- State:
NOT_STARTED
/RUNNING
/SAMPLING
/PAUSED
/BOOTSTRAPPING
/TRAINING
/LOADING
, - Bootstrapping progress (If state is
BOOTSTRAPPING
) - Number of valid monitored windows / number of total monitored windows
- Number of valid partitions out of the total number of partitions
- Percentage of the partitions that are valid
- Time and reason of recently sampling task pause/resume
- State:
-
Executor State:
- State:
NO_TASK_IN_PROGRESS
/STARTING_EXECUTION
/INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS
/INTRA_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS
/LEADER_MOVEMENT_TASK_IN_PROGRESS
/STOPPING_EXECUTION
/INITIALIZING_PROPOSAL_EXECUTION
/GENERATING_PROPOSALS_FOR_EXECUTION
- Inter-broker replica movement progress (if state is
INTER_BROKER_REPLICA_MOVEMENT_IN_PROGRESS
) - Intra-broker replica movement progress (if state is
INTRA_BROKER_REPLICA_MOVEMENT_IN_PROGRESS
) - Leadership movement progress (if state is
LEADERSHIP_MOVEMENT
) - movement concurrency
- UUID triggers the execution
- State:
-
Analyzer State:
- isProposalReady: Is there a proposal cached
- ReadyGoals: A list of goals that are ready for running
-
Anomaly Detector State:
- selfHealingEnabled: Anomaly type for which self healing is enabled
- selfHealingDisabled: Anomaly type for which self healing is disabled
- recentGoalViolations: Recently detected goal violations
- recentBrokerFailures: Recently detected broker failures
- recentDiskFailures: Recently detected disk failures
- recentMetricAnomalies: Recently detected metric anomalies
If verbose
is set to true
, the details about monitored windows and goals will be displayed.
If super_verbose
is set to true
, the details about extrapolation made on metric samples will be displayed.
If substates
is not set, the full state will be displayed; if it is set to the specific substate(s), only state(s) of interest will be displayed and response will be returned faster.
Once Cruise Control Load Monitor shows it is in the RUNNING
state, Users can use the following HTTP GET to get the cluster load:
GET /kafkacruisecontrol/load
Supported parameters are:
PARAMETER | TYPE | DESCRIPTION | DEFAULT | OPTIONAL |
---|---|---|---|---|
start | long | start time of the cluster load | time of earliest valid window | yes |
end | long | end time of the cluster load | current system time | yes |
time | long | end time of the cluster load | current system time | yes |
allow_capacity_estimation | boolean | whether to allow broker capacity to be estimated from other broker in the cluster | true | yes |
populate_disk_info | boolean | whether show the load of each disk of broker | false | yes |
capacity_only | boolean | whether show only the cluster capacity or the utilization, as well. | false | yes |
json | boolean | return in JSON format or not | false | yes |
verbose | boolean | return detailed state information | false | yes |
doAs | string | propagated user by the trusted proxy service | null | yes |
reason | string | reason for the request | "No reason provided" | yes |
If the number of workload snapshots for the given timestamp is not sufficient to generate a good load model, an exception will be returned.
Timestamp for start
/end
/time
is in milliseconds since the epoch; what System.currentTimeMillis()
returns. The time zone is the time zone of the Cruise Control server.
If allow_capacity_estimation
is set to true
, for brokers missing capacity information Cruise Control will make estimations based on other brokers in the cluster; otherwise an IllegalStateException
will be thrown and shown in response.
The response contains both load-per-broker and load-per-host information. This is specifically useful when multiple brokers are hosted by the same machine.
NOTE: The load shown is only for the load from the valid partitions. i.e the partitions with enough metric samples. So please always check the LoadMonitor
's state(via State
endpoint) to decide whether the workload is representative enough.
The following GET request gives the partition load sorted by the utilization of a given resource:
GET /kafkacruisecontrol/partition_load
Supported parameters are:
PARAMETER | TYPE | DESCRIPTION | DEFAULT | OPTIONAL |
---|---|---|---|---|
resource | string | resource type to sort partition load, available resources are DISK /CPU /NW_IN /NW_OUT
|
DISK |
yes |
start | long | the timestamp in millisecond of the earliest metric sample use to generate load | time of earliest valid window | yes |
end | long | the timestamp in millisecond of the latest metric sample use to generate load | current system time | yes |
entries | integer | number of partition load entries to report in response | MAX_INT |
yes |
json | boolean | return in JSON format or not | false | yes |
allow_capacity_estimation | boolean | whether to allow capacity estimation when cruise-control is unable to obtain all per-broker capacity information | true | yes |
max_load | boolean | whether report the max load for partition in windows | false | yes |
avg_load | boolean | whether report the average load for partition in windows | false | yes |
topic | regex | regular expression to filter partition load to report based on partition's topic | null | yes |
partition | integer/range | partition number(e.g. 10) range(e.g. 1-10) to filter partition load to report | null | yes |
min_valid_partition_ratio | double | minimal valid partition ratio requirement for cluster model | null | yes |
brokerid | int | broker id to to filter partition load to report | null | yes |
doAs | string | propagated user by the trusted proxy service | null | yes |
reason | string | reason for the request | "No reason provided" | yes |
The returned result would be a partition list sorted by the utilization of the specified resource in the time range specified by start
and end
. The resource can be CPU
, NW_IN
, NW_OUT
and DISK
. By default, the start
is the earliest monitored time, the end
is current wall clock time, resource
is DISK
, and entries
is the all partitions in the cluster.
By specifying topic
,partition
and/or brokerid
parameter, client can filter returned partition entries.
The min_valid_partition_ratio
specifies minimal monitored valid partition percentage needed to calculate the partition load. If this parameter is not set in request, the config value min.valid.partition.ratio
will be used.
The max_load
parameter specifies whether report the maximal historical value or not. The avg_load
parameter specifies whether report the average historical value or not. If both are not specified or specified as false
, for DISK
resource, latest value will be reported; for NW_IN
/NW_OUT
/CPU
resource, average value will be reported.
The following GET request gives partition healthiness on the cluster:
GET /kafkacruisecontrol/kafka_cluster_state
Supported parameters are:
PARAMETER | TYPE | DESCRIPTION | DEFAULT | OPTIONAL |
---|---|---|---|---|
topic | regex | regular expression to filter partition state to report based on partition's topic | null | yes |
json | boolean | return in JSON format or not | false | yes |
verbose | boolean | return detailed state information | false | yes |
doAs | string | propagated user by the trusted proxy service | null | yes |
reason | string | reason for the request | "No reason provided" | yes |
The returned result contains the following information
-
For each broker
- Distribution of leader/follower/out-of-sync/offline replica information
- Online/offline disks
-
For each partition
- Distribution of leader/follower/in-sync/out-of-sync/offline replica information
The following GET request returns the optimization proposals generated based on the workload model of the given timestamp. The workload summary before and after the optimization will also be returned.
GET /kafkacruisecontrol/proposals
Supported parameters are:
PARAMETER | TYPE | DESCRIPTION | DEFAULT | OPTIONAL |
---|---|---|---|---|
ignore_proposal_cache | boolean | whether to ignore the cached proposal or not | false | yes |
data_from | string | whether to calculate proposal from available valid partitions or valid windows | VALID_WINDOWS |
yes |
goals | list | list of goals used to generate proposal | default goals | yes |
kafka_assigner | boolean | whether to use Kafka assigner mode to generate proposals | false | yes |
allow_capacity_estimation | boolean | whether to allow broker capacity to be estimated | true | yes |
excluded_topics | regex | regular expression to specify topics excluded from replica and leadership movement | null | yes |
use_ready_default_goals | boolean | whether to use only ready goals to generate proposals | false | yes |
exclude_recently_demoted_brokers | boolean | whether to allow leader replicas to be moved to recently demoted brokers | false | yes |
exclude_recently_removed_brokers | boolean | whether allow replicas to be moved to recently removed broker | false | yes |
destination_broker_ids | list | specify brokers to move replicas to | available brokers | yes |
rebalance_disk | boolean | whether to balance load between disks within brokers (requires JBOD Kafka deployment) | false | yes |
json | boolean | return in JSON format or not | false | yes |
verbose | boolean | return detailed state information | false | yes |
doAs | string | propagated user by the trusted proxy service | null | yes |
fast_mode | boolean | true to compute proposals in fast mode, false otherwise | true | yes |
reason | string | reason for the request | "No reason provided" | yes |
Proposal can be generated based on valid_window or valid_partitions.
valid_windows: rebalance the cluster based on the information in the available valid snapshot windows. A valid snapshot window is a windows whose valid monitored partitions coverage meets the requirements of all the goals. (This is the default behavior)
valid_partitions: rebalance the cluster based on all the available valid partitions. All the snapshot windows will be included in this case.
Users can only specify either valid_windows
or valid_partitions
, but not both.
Kafka cruise control tries to precompute the optimization proposal in the background and caches the best proposal to serve when user queries. If users want to have a fresh proposal without reading it from the proposal cache, set the ignore_proposal_cache
flag to true. The precomputing always uses available valid partitions to generate the proposals.
By default the proposal will be returned from the cache where all the pre-defined goals are used. Detailed information about the reliability of the proposals will also be returned. If users want to run with a different set of goals, they can specify the goals
parameter with the goal names (simple class name).
If verbose
is turned on, Cruise Control will return all the generated proposals. Otherwise a summary of the proposals will be returned.
If kafka_assigner
is turned on, the proposals will be generated in Kafka Assigner mode,i.e. Cruise Control behaves like kafka assigner tool. This mode performs optimizations using the goals specific to Kafka Assigner -- i.e. goals with name KafkaAssigner*
.
Users can specify excluded_topics
to prevent certain topics' replicas from moving in the generated proposals.
If use_ready_default_goals
is turned on, Cruise Control will use whatever ready goals(based on available metric data) to calculate the proposals.
The following get request allows user to get a full list of all the active/completed(and not recycled) tasks inside Cruise Control, with their initial request detail(request time/IP address/request URL and parameter) and UUID information. User can then use the returned UUID and URL to fetch the original final result of the specific request.
GET /kafkacruisecontrol/user_tasks
Supported parameters are:
PARAMETER | TYPE | DESCRIPTION | DEFAULT | OPTIONAL |
---|---|---|---|---|
user_task_ids | list | comma separated UUIDs to filter the task results Cruise Control report | all tasks | yes |
client_ids | list | comma separated IP addresses to filter the task results Cruise Control report | all users | yes |
entries | integer | number of partition load entries to report in response | MAX_INT |
yes |
endpoints | list | comma separated endpoints to filter the task results Cruise Control report | all endpoints | yes |
types | string | comma separated HTTP request types to filter the task results Cruise Control report | all request types | yes |
json | boolean | return in JSON format or not | false | yes |
fetch_completed_task | boolean | whether return the original request's final response | false | yes |
doAs | string | propagated user by the trusted proxy service | null | yes |
reason | string | reason for the request | "No reason provided" | yes |
User can use user_task_ids
/client_ids
/endpoints
/types
make Cruise Control only return requests they are interested. By default all the requests get returned.
If fetch_completed_task
is set to true
, the original response of each request will be returned. In the case where a task completed with errors the response will be CompletedWithError
.
The post requests of Kafka Cruise Control REST API are operations that will have impact on the Kafka cluster. The post operations include:
- Trigger a workload balance
- Add a list of new brokers to Kafka Cluster
- Decommission a list of brokers from the Kafka cluster
- Fix offline replicas in Kafka cluster
- Demote a list of brokers from the Kafka cluster
- Stop the current proposal execution task
- Pause metrics load sampling
- Resume metrics load sampling
- Change Kafka topic configuration
- Change Cruise Control configuration
Most of the POST actions has a dry-run mode, which only generate the proposals and estimated result but not really execute the proposals. To avoid accidentally triggering of data movement, by default all the POST actions are in dry run mode. To let Kafka Cruise Control actually move data, users need to explicitly set dryrun=false
.
The following POST request will let Kafka Cruise Control rebalance a Kafka cluster
POST /kafkacruisecontrol/rebalance
Supported parameters are:
PARAMETER | TYPE | DESCRIPTION | DEFAULT | OPTIONAL |
---|---|---|---|---|
dryrun | boolean | whether dry-run the request or not | true | yes |
data_from | string | whether generate proposal from available valid partitions or valid windows | VALID_WINDOW |
yes |
goals | list | list of goals used to generate proposal | default goals | yes |
kafka_assigner | boolean | whether to use Kafka assigner mode to generate proposals | false | yes |
allow_capacity_estimation | boolean | whether to allow broker capacity to be estimated | true | yes |
concurrent_partition_movements_per_broker | integer | upper bound of ongoing replica movements going into/out of each broker | null | yes |
max_partition_movements_in_cluster | integer | upper bound of maximum replica movements going into/out of brokers across cluster | default_config |
yes |
concurrent_intra_broker_partition_movements | integer | upper bound of ongoing replica movements between disks within each broker | null | yes |
concurrent_leader_movements | integer | upper bound of ongoing leadership movements | null | yes |
skip_hard_goal_check | boolean | Whether allow hard goals be skipped in proposal generation | false | yes |
excluded_topics | regex | regular expression to specify topics excluded from replica and leadership movement | null | yes |
use_ready_default_goals | boolean | whether only use ready goals to generate proposal | false | yes |
exclude_recently_demoted_brokers | boolean | whether to allow leader replicas to be moved to recently demoted brokers | false | yes |
exclude_recently_removed_brokers | boolean | whether allow replicas to be moved to recently removed broker | false | yes |
replica_movement_strategies | string | replica movement strategy to use | null | yes |
ignore_proposal_cache | boolean | whether to ignore the cached proposal or not | false | yes |
replication_throttle | long | upper bound on the bandwidth used to move replicas (in bytes per second) | null | yes |
destination_broker_ids | list | specify brokers to move replicas to | available brokers | yes |
rebalance_disk | boolean | whether to balance load between disks within brokers (requires JBOD Kafka deployment) | false | yes |
json | boolean | return in JSON format or not | false | yes |
verbose | boolean | return detailed state information | false | yes |
reason | string | reason for the request | "No reason provided" | yes |
doAs | string | propagated user by the trusted proxy service | null | yes |
fast_mode | boolean | true to compute proposals in fast mode, false otherwise | true | yes |
Similar to the GET interface for getting proposals, the rebalance can also be based on available valid windows or available valid partitions.
User can specify goals to use for rebalance via goals
parameter. When goals
is provided, the cached proposals will be ignored.
When rebalancing a cluster, all the brokers in the cluster(except recently removed/demoted brokers if exclude_recently_removed_brokers
/exclude_recently_demoted_brokers
set to true
) are eligible to give / receive replicas. All the brokers will be throttled during the partition movement. The throttling can be set in two ways.
- By throttling number of replica concurrently moving into a broker. It is gated by the request parameter
concurrent_partition_movements_per_broker
or config valuenum.concurrent.partition.movements.per.broker
(if request parameter is not set),concurrent_intra_partition_movements
or config valuenum.concurrent.intra.broker.partition.movements
. Similarly, the number of concurrent partition leadership changes is gated by request parameterconcurrent_leader_movements
or config valuenum.concurrent.leader.movements
. - By throttling the bandwidth used to move replicas into a broker. It is gated by the request parameter
replication_throttle
or config valuedefault.replication.throttle
(if request parameter is not set).
The following POST request adds the given brokers to the Kafka cluster
POST /kafkacruisecontrol/add_broker?brokerid=[id1,id2...]
Supported parameters are:
PARAMETER | TYPE | DESCRIPTION | DEFAULT | OPTIONAL |
---|---|---|---|---|
brokerid | list | list of ids of new broker added to the cluster | N/A | no |
dryrun | boolean | whether dry-run the request or not | true | yes |
data_from | string | whether generate proposal from available valid partitions or valid windows | VALID_WINDOWS |
yes |
goals | list | list of goals used to generate proposal | default goals | yes |
kafka_assigner | boolean | whether to use Kafka assigner mode to generate proposals | false | yes |
allow_capacity_estimation | boolean | whether to allow broker capacity to be estimated from other broker in the cluster | true | yes |
concurrent_partition_movements_per_broker | integer | upper bound of ongoing replica movements going into/out of each broker | null | yes |
max_partition_movements_in_cluster | integer | upper bound of maximum replica movements going into/out of brokers across cluster | default_config |
yes |
concurrent_leader_movements | integer | upper bound of ongoing leadership movements | null | yes |
skip_hard_goal_check | boolean | whether allow hard goals be skipped in proposal generation | false | yes |
excluded_topics | regex | regular expression to specify topics excluded from replica and leadership movement | null | yes |
use_ready_default_goals | boolean | whether only use ready goals to generate proposal | false | yes |
exclude_recently_demoted_brokers | boolean | whether to allow leader replicas to be moved to recently demoted brokers | false | yes |
exclude_recently_removed_brokers | boolean | whether allow replicas to be moved to recently removed broker | false | yes |
replica_movement_strategies | string | replica movement strategy to use | null | yes |
replication_throttle | long | Upper bound on the bandwidth used to move replicas (in bytes per second) | null | yes |
throttle_added_broker | boolean | whether throttle replica movement to new broker or not | true | yes |
json | boolean | return in JSON format or not | false | yes |
verbose | boolean | return detailed state information | false | yes |
reason | string | reason for the request | "No reason provided" | yes |
doAs | string | propagated user by the trusted proxy service | null | yes |
fast_mode | boolean | true to compute proposals in fast mode, false otherwise | true | yes |
When adding new brokers to a Kafka cluster, Cruise Control makes sure that the replicas will only be moved from the existing brokers to the provided new broker, but not moved among existing brokers.
Users can choose whether to throttle replica movement to the newly added broker via throttle_added_broker
, in either case, the replica movement out of current broker are throttled, and the throttling can be set in the same way as rebalance
request.
The following POST request removes a list of brokers from the Kafka cluster:
POST /kafkacruisecontrol/remove_broker?brokerid=[id1,id2...]
Supported parameters are:
PARAMETER | TYPE | DESCRIPTION | DEFAULT | OPTIONAL |
---|---|---|---|---|
brokerid | list | list of ids of broker to be removed from the cluster | N/A | no |
dryrun | boolean | whether dry-run the request or not | true | yes |
data_from | string | specify either generate proposal from available valid partitions or valid windows | VALID_WINDOW |
yes |
goals | list | list of goals used to generate proposal | default goals | yes |
kafka_assigner | boolean | whether to use Kafka assigner mode to generate proposals | false | yes |
allow_capacity_estimation | boolean | whether to allow broker capacity to be estimated from other brokers in the cluster | true | yes |
concurrent_partition_movements_per_broker | integer | upper bound of ongoing replica movements going into/out of each broker | null | yes |
max_partition_movements_in_cluster | integer | upper bound of maximum replica movements going into/out of brokers across cluster | default_config |
yes |
concurrent_leader_movements | integer | upper bound of ongoing leadership movements | null | yes |
skip_hard_goal_check | boolean | whether allow hard goals be skipped in proposal generation | false | yes |
excluded_topics | regex | regular expression to specify topics excluded from replica and leadership movement | null | yes |
use_ready_default_goals | boolean | whether only use ready goals to generate proposal | false | yes |
exclude_recently_demoted_brokers | boolean | whether to allow leader replicas to be moved to recently demoted brokers | false | yes |
exclude_recently_removed_brokers | boolean | whether allow replicas to be moved to recently removed broker | false | yes |
replica_movement_strategies | string | replica movement strategy to use | null | yes |
replication_throttle | long | upper bound on the bandwidth used to move replicas (in bytes per second) | false | yes |
throttle_removed_broker | boolean | whether throttle replica movement out of the removed broker or not | true | yes |
destination_broker_ids | list | specify brokers to move replicas to | available brokers | yes |
json | boolean | return in JSON format or not | false | yes |
verbose | boolean | return detailed state information | false | yes |
reason | string | reason for the request | "No reason provided" | yes |
doAs | string | propagated user by the trusted proxy service | null | yes |
fast_mode | boolean | true to compute proposals in fast mode, false otherwise | true | yes |
Similar to adding brokers to a cluster, removing brokers from a cluster will only move partitions from the brokers to be removed to the other existing brokers. There won't be partition movements among remaining brokers. And user can specify the destination broker for these replica movement via destination_broker_ids
parameter.
Like adding brokers, users can choose whether to throttle the removed broker during the partition movement. If the removed brokers are throttled, the number of partitions concurrently moving out of a broker, number of concurrent partition leadership changes and bandwidth used for replica movement are gated in the same way as add_broker above.
Note if the topics specified in excluded_topics
has replicas on the removed broker, the replicas will still get moved off the broker.
The following POST request moves all the offline replicas from dead disks/brokers. This endpoint is not available in kafka_0_11_and_1_0
branch.
POST /kafkacruisecontrol/fix_offline_replicas
Supported parameters are:
PARAMETER | TYPE | DESCRIPTION | DEFAULT | OPTIONAL |
---|---|---|---|---|
dryrun | boolean | whether dry-run the request or not | true | yes |
data_from | string | specify either generate proposal from available valid partitions or valid windows | VALID_WINDOW |
yes |
goals | list | list of goals used to generate proposal | default goals | yes |
kafka_assigner | boolean | whether to use Kafka assigner mode to generate proposals | false | yes |
allow_capacity_estimation | boolean | whether to allow broker capacity to be estimated from other brokers in the cluster | true | yes |
concurrent_partition_movements_per_broker | integer | upper bound of ongoing replica movements going into/out of each broker | null | yes |
max_partition_movements_in_cluster | integer | upper bound of maximum replica movements going into/out of brokers across cluster | default_config |
yes |
concurrent_leader_movements | integer | upper bound of ongoing leadership movements | null | yes |
skip_hard_goal_check | boolean | Whether allow hard goals be skipped in proposal generation | false | yes |
excluded_topics | regex | regular expression to specify topics excluded from replica and leadership movement | null | yes |
use_ready_default_goals | boolean | whether only use ready goals to generate proposal | false | yes |
exclude_recently_demoted_brokers | boolean | whether to allow leader replicas to be moved to recently demoted brokers | false | yes |
exclude_recently_removed_brokers | boolean | whether allow replicas to be moved to recently removed broker | false | yes |
replica_movement_strategies | string | replica movement strategy to use | null | yes |
replication_throttle | long | upper bound on the bandwidth used to move replicas (in bytes per second) | null | yes |
json | boolean | return in JSON format or not | false | yes |
verbose | boolean | return detailed state information | false | yes |
reason | string | reason for the request | "No reason provided" | yes |
doAs | string | propagated user by the trusted proxy service | null | yes |
fast_mode | boolean | true to compute proposals in fast mode, false otherwise | true | yes |
Likewise, users can throttle partition movement, the throttling can be set in the same way as rebalance
request.
Note if the topics specified in excluded_topics
has offline replicas, the replicas will still get moved to healthy brokers.
The following POST request moves all the leader replicas away from a list of brokers.
POST /kafkacruisecontrol/demote_broker?brokerid=[id1, id2...]
User can also request to move all the leader replicas away from the a list of disks via
POST /kafkacruisecontrol/demote_broker?brokerid_and_logdirs=[id1-logdir1, id2-logdir2...]
Supported parameters are:
PARAMETER | TYPE | DESCRIPTION | DEFAULT | OPTIONAL |
---|---|---|---|---|
brokerid | list | list of ids of broker to be demoted in the cluster | N/A | no if brokerid_and_logdirs is not set |
brokerid_and_logdirs | list | list of broker id and logdir pair to be demoted in the cluster | N/A | no if brokerids is not set |
dryrun | boolean | whether dry-run the request or not | true | yes |
allow_capacity_estimation | boolean | whether to allow broker capacity to be estimated | true | yes |
concurrent_leader_movements | integer | upper bound of ongoing leadership movements | null | yes |
skip_urp_demotion | boolean | whether skip demoting leader replicas for under replicated partitions | true | yes |
exclude_follower_demotion | boolean | whether skip demoting follower replicas on the broker to be demoted | true | yes |
exclude_recently_demoted_brokers | boolean | whether to allow leader replicas to be moved to recently demoted brokers | false | yes |
replica_movement_strategies | string | replica movement strategy to use | null | yes |
replication_throttle | long | upper bound on the bandwidth used to move replicas (in bytes per second) | null | yes |
json | boolean | return in JSON format or not | false | yes |
verbose | boolean | return detailed state information | false | yes |
reason | string | reason for the request | "No reason provided" | yes |
doAs | string | propagated user by the trusted proxy service | null | yes |
Demoting a broker/disk is consist of tow steps.
- Make all the replicas on given broker/disk the least preferred replicas for leadership election within their corresponding partitions
- Trigger a preferred leader election on the partitions to migrate the leader replicas off the broker/disk
Set skip_urp_demotion
to false will cancel outstanding operations if partitions stay under replicated; Set exclude_follower_demotion
will skip operations on the partitions which only have follower replicas on the brokers/disks to be demoted. The purpose of the former is to prevent the URP recovery process from blocking the demotion execution, the latter ensures that the demotion operation is limited to leaders.
The following POST request will let Kafka Cruise Control stop an ongoing rebalance
, add_broker
, remove_broker
, fix_offline_replica
, topic_configuration
or demote_broker
operation:
POST /kafkacruisecontrol/stop_proposal_execution
Supported parameters are:
PARAMETER | TYPE | DESCRIPTION | DEFAULT | OPTIONAL |
---|---|---|---|---|
force_stop | boolean | (not supported in Kafka 2.4 or above) stop an ongoing execution forcefully by deleting Kafka internal zNodes | false | yes |
stop_external_agent | boolean | (required Kafka 2.4 or above) stop an ongoing execution even if it is started by an external agent | true | yes |
review_id | integer | review id for 2-step verification | N/A | yes |
json | boolean | return in JSON format or not | false | yes |
get_response_schema | boolean | return JSON schema in response header or not | false | yes |
doAs | string | propagated user by the trusted proxy service | null | yes |
reason | string | reason for the request | "No reason provided" | yes |
Note that Cruise Control does not wait for the ongoing batch to finish when it stops execution, i.e. the in-progress batch may still be running after Cruise Control stops the execution.
The stop_external_agent
parameter is a no-op for CC versions before 2.4.73
and 2.5.65
.
The following POST request will let Kafka Cruise Control pause an ongoing metrics sampling process:
POST /kafkacruisecontrol/pause_sampling
Supported parameters are:
PARAMETER | TYPE | DESCRIPTION | DEFAULT | OPTIONAL |
---|---|---|---|---|
reason | string | reason to pause sampling | "No reason provided" | yes |
json | boolean | return in JSON format or not | false | yes |
doAs | string | propagated user by the trusted proxy service | null | yes |
The reason to pause sampling will be recorded and shows up in state
endpoint(under LoadMonitor
sub state).
The following POST request will let Kafka Cruise Control resume a paused metrics sampling process:
POST /kafkacruisecontrol/resume_sampling
PARAMETER | TYPE | DESCRIPTION | DEFAULT | OPTIONAL |
---|---|---|---|---|
reason | string | reason to resume sampling | "No reason provided" | yes |
json | boolean | return in JSON format or not | false | yes |
doAs | string | propagated user by the trusted proxy service | null | yes |
The reason to resume sampling will be recorded and shows up in state
endpoint(under LoadMonitor
sub state).
Currently Cruise Control only supports changing topic's replication factor via topic_configuration
endpoint. Ultimately we want make Cruise Control the central place to change any topic configurations(partition count, retention time etc.).
The following POST request can change topic's replication factor.
POST /kafkacruisecontrol/topic_configuration?topic=[topic_regex]&replication_factor=[target_replication_factor]
Supported parameters are:
PARAMETER | TYPE | DESCRIPTION | DEFAULT | OPTIONAL |
---|---|---|---|---|
dryrun | boolean | whether dry-run the request or not | true | yes |
topic | regex | regular expression to specify subject topics | N/A | no |
replication_factor | Integer | target replication factor | N/A | no |
data_from | string | whether generate proposal from available valid partitions or valid windows | VALID_WINDOW |
yes |
goals | list | list of goals used to generate proposal | default goals | yes |
allow_capacity_estimation | boolean | whether to allow broker capacity to be estimated | true | yes |
concurrent_partition_movements_per_broker | integer | upper bound of ongoing replica movements going into/out of each broker | null | yes |
max_partition_movements_in_cluster | integer | upper bound of maximum replica movements going into/out of brokers across cluster | default_config |
yes |
concurrent_leader_movements | integer | upper bound of ongoing leadership movements | null | yes |
skip_hard_goal_check | boolean | Whether allow hard goals be skipped in proposal generation | false | yes |
exclude_recently_demoted_brokers | Boolean | Whether to allow leader replicas to be moved to recently demoted brokers | false | yes |
exclude_recently_removed_brokers | Boolean | Whether allow replicas to be moved to recently removed broker | false | yes |
replica_movement_strategies | string | replica movement strategy to use | null | yes |
replication_throttle | long | upper bound on the bandwidth used to move replicas (in bytes per second) | null | yes |
json | boolean | return in JSON format or not | false | yes |
verbose | boolean | return detailed state information | false | yes |
reason | string | reason for the request | "No reason provided" | yes |
doAs | string | propagated user by the trusted proxy service | null | yes |
fast_mode | boolean | true to compute proposals in fast mode, false otherwise | true | yes |
Changing topic's replication factor will not move any existing replicas. goals
are used to determine which replica to be deleted(to decrease topic's replication factor) and which broker to assign new replica (to increase topic's replication factor).
Note sometimes the topic regex can be too long to put at POST request head, in this case user can specify topic regex and target replication factor pairs in POST request body. For details, check Change-topic-replication-factor-through-Cruise-Control wiki page.
Some Cruise Control configs can be changed dynamically via admin
endpoint, which includes
- Dynamically change the partition and leadership concurrency and the interval between checking and updating (if needed) the progress of an ongoing execution.
- Enable/disable self-healing for the specified anomaly types.
- Drop selected recently removed/demoted brokers.
- Enable/disable the specified concurrency adjusters.
- Enable/disable the (At/Under)MinISR-based concurrency adjustment.
Supported parameters are:
PARAMETER | TYPE | DESCRIPTION | DEFAULT | OPTIONAL |
---|---|---|---|---|
disable_self_healing_for | list | list of anomaly types to disable self-healing | N/A | yes |
enable_self_healing_for | list | list of anomaly types to enable self-healing | N/A | yes |
concurrent_partition_movements_per_broker | integer | upper bound of ongoing replica movements into/out of a broker | N/A | yes |
max_partition_movements_in_cluster | integer | upper bound of max replica movements into/out of brokers across cluster | N/A | yes |
concurrent_intra_broker_partition_movements | integer | upper bound of ongoing replica movements between disks within a broker | N/A | yes |
concurrent_leader_movements | integer | upper bound of ongoing leadership movements | N/A | yes |
drop_recently_removed_brokers | list | list of id of recently removed brokers to be dropped | N/A | yes |
drop_recently_demoted_brokers | list | list of id of recently demoted brokers to be dropped | N/A | yes |
doAs | string | propagated user by the trusted proxy service | null | yes |
json | boolean | return in JSON format or not | false | yes |
review_id | integer | review id for 2-step verification | N/A | yes |
execution_progress_check_interval_ms | long | execution progress check interval in milliseconds | N/A | yes |
get_response_schema | boolean | return JSON schema in response header or not | false | yes |
disable_concurrency_adjuster_for | list | disable concurrency adjuster for given concurrency types | N/A | yes |
enable_concurrency_adjuster_for | list | enable concurrency adjuster for given concurrency types | N/A | yes |
min_isr_based_concurrency_adjustment | boolean | enable (true) or disable (false) MinISR-based concurrency adjustment | N/A | yes |
reason | string | reason for the request | "No reason provided" | yes |
To Enable/disable self-healing, send POST request like:
POST /kafkacruisecontrol/admin?disable_self_healing_for=[anomaly_type]
To increase/decrease execution concurrency, send POST request like:
POST /kafkacruisecontrol/admin?concurrent_partition_movements_per_broker=[integer]
To drop recently removed/demoted brokers, send POST request like:
POSTs /kafkacruisecontrol/admin?drop_recently_removed_brokers=[broker_ids]
2-step verification aims to help users verify the command they (or their peers) intend to run by letting them review requests explicitly to approve or discard them, and enable execution of only the approved requests. Read 2 step verification for POST requests for more detail.
The following POST request can create a request to the provisioner to rightsize the broker or partition of a cluster.
POST /kafkacruisecontrol/rightsize
Supported parameters are:
PARAMETER | TYPE | DESCRIPTION | DEFAULT | OPTIONAL |
---|---|---|---|---|
num_brokers_to_add | integer | difference in broker count to rightsize towards | -1 | yes |
partition_count | integer | target number of partitions to rightsize towards | -1 | yes |
topic | regex | regular expression to specify subject topics | null | yes |
doAs | string | propagated user by the trusted proxy service | null | yes |
get_response_schema | boolean | return JSON schema in response header or not | false | yes |
json | boolean | return in JSON format or not | false | yes |
reason | string | reason for the request | "No reason provided" | yes |
Contents
- Cruise Control Wiki
- Overview
- Troubleshooting
- User Guide
- Python Client
- Developer Guide