Подключиться
Language / Region
© 2021 ООО «Яндекс.Облако»
Yandex Managed Service for Apache Kafka®

Cluster

Статья создана

A set of methods for managing Apache Kafka® clusters.

JSON Representation

{
  "id": "string",
  "folderId": "string",
  "createdAt": "string",
  "name": "string",
  "description": "string",
  "labels": "object",
  "environment": "string",
  "monitoring": [
    {
      "name": "string",
      "description": "string",
      "link": "string"
    }
  ],
  "config": {
    "version": "string",
    "kafka": {
      "resources": {
        "resourcePresetId": "string",
        "diskSize": "string",
        "diskTypeId": "string"
      },

      // `config.kafka` includes only one of the fields `kafkaConfig_2_1`, `kafkaConfig_2_6`, `kafkaConfig_2_8`
      "kafkaConfig_2_1": {
        "compressionType": "string",
        "logFlushIntervalMessages": "integer",
        "logFlushIntervalMs": "integer",
        "logFlushSchedulerIntervalMs": "integer",
        "logRetentionBytes": "integer",
        "logRetentionHours": "integer",
        "logRetentionMinutes": "integer",
        "logRetentionMs": "integer",
        "logSegmentBytes": "integer",
        "logPreallocate": true,
        "socketSendBufferBytes": "integer",
        "socketReceiveBufferBytes": "integer",
        "autoCreateTopicsEnable": true,
        "numPartitions": "integer",
        "defaultReplicationFactor": "integer"
      },
      "kafkaConfig_2_6": {
        "compressionType": "string",
        "logFlushIntervalMessages": "integer",
        "logFlushIntervalMs": "integer",
        "logFlushSchedulerIntervalMs": "integer",
        "logRetentionBytes": "integer",
        "logRetentionHours": "integer",
        "logRetentionMinutes": "integer",
        "logRetentionMs": "integer",
        "logSegmentBytes": "integer",
        "logPreallocate": true,
        "socketSendBufferBytes": "integer",
        "socketReceiveBufferBytes": "integer",
        "autoCreateTopicsEnable": true,
        "numPartitions": "integer",
        "defaultReplicationFactor": "integer"
      },
      "kafkaConfig_2_8": {
        "compressionType": "string",
        "logFlushIntervalMessages": "integer",
        "logFlushIntervalMs": "integer",
        "logFlushSchedulerIntervalMs": "integer",
        "logRetentionBytes": "integer",
        "logRetentionHours": "integer",
        "logRetentionMinutes": "integer",
        "logRetentionMs": "integer",
        "logSegmentBytes": "integer",
        "logPreallocate": true,
        "socketSendBufferBytes": "integer",
        "socketReceiveBufferBytes": "integer",
        "autoCreateTopicsEnable": true,
        "numPartitions": "integer",
        "defaultReplicationFactor": "integer"
      },
      // end of the list of possible fields`config.kafka`

    },
    "zookeeper": {
      "resources": {
        "resourcePresetId": "string",
        "diskSize": "string",
        "diskTypeId": "string"
      }
    },
    "zoneId": [
      "string"
    ],
    "brokersCount": "integer",
    "assignPublicIp": true,
    "unmanagedTopics": true,
    "schemaRegistry": true
  },
  "networkId": "string",
  "health": "string",
  "status": "string",
  "securityGroupIds": [
    "string"
  ],
  "hostGroupIds": [
    "string"
  ],
  "deletionProtection": true
}
Field Description
id string

ID of the Apache Kafka® cluster. This ID is assigned at creation time.
folderId string

ID of the folder that the Apache Kafka® cluster belongs to.
createdAt string (date-time)

Creation timestamp.

String in RFC3339 text format.
name string

Name of the Apache Kafka® cluster. The name must be unique within the folder. 1-63 characters long. Value must match the regular expression [a-zA-Z0-9_-]*.
description string

Description of the Apache Kafka® cluster. 0-256 characters long.
labels object

Custom labels for the Apache Kafka® cluster as key:value pairs. A maximum of 64 labels per resource is allowed.
environment string

Deployment environment of the Apache Kafka® cluster.

  • PRODUCTION: stable environment with a conservative update policy when only hotfixes are applied during regular maintenance.
  • PRESTABLE: environment with a more aggressive update policy when new versions are rolled out irrespective of backward compatibility.
monitoring[] object

Metadata of monitoring system.
monitoring[].
name		 string

Name of the monitoring system.
monitoring[].
description		 string

Description of the monitoring system.
monitoring[].
link		 string

Link to the monitoring system charts for the Apache Kafka® cluster.
config object

Configuration of the Apache Kafka® cluster.
config.
version		 string

Version of Apache Kafka® used in the cluster. Possible values: 2.1, 2.6.
config.
kafka		 object

Configuration and resource allocation for Kafka brokers.
config.
kafka.
resources		 object
config.
kafka.
resources.
resourcePresetId		 string

ID of the preset for computational resources available to a host (CPU, memory, etc.). All available presets are listed in the documentation.
config.
kafka.
resources.
diskSize		 string (int64)

Volume of the storage available to a host, in bytes.
config.
kafka.
resources.
diskTypeId		 string

Type of the storage environment for the host.
config.
kafka.
kafkaConfig_2_1		 object
config.kafka includes only one of the fields kafkaConfig_2_1, kafkaConfig_2_6, kafkaConfig_2_8

Kafka version 2.1 broker configuration.
config.
kafka.
kafkaConfig_2_1.
compressionType		 string

Cluster topics compression type.

  • COMPRESSION_TYPE_UNCOMPRESSED: no codec (uncompressed).
  • COMPRESSION_TYPE_ZSTD: Zstandard codec.
  • COMPRESSION_TYPE_LZ4: LZ4 codec.
  • COMPRESSION_TYPE_SNAPPY: Snappy codec.
  • COMPRESSION_TYPE_GZIP: GZip codec.
  • COMPRESSION_TYPE_PRODUCER: the codec to use is set by a producer (can be any of ZSTD, LZ4, GZIP or SNAPPY codecs).
config.
kafka.
kafkaConfig_2_1.
logFlushIntervalMessages		 integer (int64)

The number of messages accumulated on a log partition before messages are flushed to disk.

This is the global cluster-level setting that can be overridden on a topic level by using the flushMessages setting.
config.
kafka.
kafkaConfig_2_1.
logFlushIntervalMs		 integer (int64)

The maximum time (in milliseconds) that a message in any topic is kept in memory before flushed to disk. If not set, the value of logFlushSchedulerIntervalMs is used.

This is the global cluster-level setting that can be overridden on a topic level by using the flushMs setting.
config.
kafka.
kafkaConfig_2_1.
logFlushSchedulerIntervalMs		 integer (int64)

The frequency of checks (in milliseconds) for any logs that need to be flushed to disk. This check is done by the log flusher.
config.
kafka.
kafkaConfig_2_1.
logRetentionBytes		 integer (int64)

Partition size limit; Kafka will discard old log segments to free up space if delete cleanupPolicy is in effect. This setting is helpful if you need to control the size of a log due to limited disk space.

This is the global cluster-level setting that can be overridden on a topic level by using the retentionBytes setting.
config.
kafka.
kafkaConfig_2_1.
logRetentionHours		 integer (int64)

The number of hours to keep a log segment file before deleting it.
config.
kafka.
kafkaConfig_2_1.
logRetentionMinutes		 integer (int64)

The number of minutes to keep a log segment file before deleting it.

If not set, the value of logRetentionHours is used.
config.
kafka.
kafkaConfig_2_1.
logRetentionMs		 integer (int64)

The number of milliseconds to keep a log segment file before deleting it.

If not set, the value of logRetentionMinutes is used.

This is the global cluster-level setting that can be overridden on a topic level by using the retentionMs setting.
config.
kafka.
kafkaConfig_2_1.
logSegmentBytes		 integer (int64)

The maximum size of a single log file.

This is the global cluster-level setting that can be overridden on a topic level by using the segmentBytes setting.
config.
kafka.
kafkaConfig_2_1.
logPreallocate		 boolean (boolean)

Should pre allocate file when create new segment?

This is the global cluster-level setting that can be overridden on a topic level by using the preallocate setting.
config.
kafka.
kafkaConfig_2_1.
socketSendBufferBytes		 integer (int64)

The SO_SNDBUF buffer of the socket server sockets. If the value is -1, the OS default will be used.
config.
kafka.
kafkaConfig_2_1.
socketReceiveBufferBytes		 integer (int64)

The SO_RCVBUF buffer of the socket server sockets. If the value is -1, the OS default will be used.
config.
kafka.
kafkaConfig_2_1.
autoCreateTopicsEnable		 boolean (boolean)

Enable auto creation of topic on the server
config.
kafka.
kafkaConfig_2_1.
numPartitions		 integer (int64)

Default number of partitions per topic on the whole cluster
config.
kafka.
kafkaConfig_2_1.
defaultReplicationFactor		 integer (int64)

Default replication factor of the topic on the whole cluster
config.
kafka.
kafkaConfig_2_6		 object
config.kafka includes only one of the fields kafkaConfig_2_1, kafkaConfig_2_6, kafkaConfig_2_8

Kafka version 2.6 broker configuration.
config.
kafka.
kafkaConfig_2_6.
compressionType		 string

Cluster topics compression type.

  • COMPRESSION_TYPE_UNCOMPRESSED: no codec (uncompressed).
  • COMPRESSION_TYPE_ZSTD: Zstandard codec.
  • COMPRESSION_TYPE_LZ4: LZ4 codec.
  • COMPRESSION_TYPE_SNAPPY: Snappy codec.
  • COMPRESSION_TYPE_GZIP: GZip codec.
  • COMPRESSION_TYPE_PRODUCER: the codec to use is set by a producer (can be any of ZSTD, LZ4, GZIP or SNAPPY codecs).
config.
kafka.
kafkaConfig_2_6.
logFlushIntervalMessages		 integer (int64)

The number of messages accumulated on a log partition before messages are flushed to disk.

This is the global cluster-level setting that can be overridden on a topic level by using the flushMessages setting.
config.
kafka.
kafkaConfig_2_6.
logFlushIntervalMs		 integer (int64)

The maximum time (in milliseconds) that a message in any topic is kept in memory before flushed to disk. If not set, the value of logFlushSchedulerIntervalMs is used.

This is the global cluster-level setting that can be overridden on a topic level by using the flushMs setting.
config.
kafka.
kafkaConfig_2_6.
logFlushSchedulerIntervalMs		 integer (int64)

The frequency of checks (in milliseconds) for any logs that need to be flushed to disk. This check is done by the log flusher.
config.
kafka.
kafkaConfig_2_6.
logRetentionBytes		 integer (int64)

Partition size limit; Kafka will discard old log segments to free up space if delete cleanupPolicy is in effect. This setting is helpful if you need to control the size of a log due to limited disk space.

This is the global cluster-level setting that can be overridden on a topic level by using the retentionBytes setting.
config.
kafka.
kafkaConfig_2_6.
logRetentionHours		 integer (int64)

The number of hours to keep a log segment file before deleting it.
config.
kafka.
kafkaConfig_2_6.
logRetentionMinutes		 integer (int64)

The number of minutes to keep a log segment file before deleting it.

If not set, the value of logRetentionHours is used.
config.
kafka.
kafkaConfig_2_6.
logRetentionMs		 integer (int64)

The number of milliseconds to keep a log segment file before deleting it.

If not set, the value of logRetentionMinutes is used.

This is the global cluster-level setting that can be overridden on a topic level by using the retentionMs setting.
config.
kafka.
kafkaConfig_2_6.
logSegmentBytes		 integer (int64)

The maximum size of a single log file.

This is the global cluster-level setting that can be overridden on a topic level by using the segmentBytes setting.
config.
kafka.
kafkaConfig_2_6.
logPreallocate		 boolean (boolean)

Should pre allocate file when create new segment?

This is the global cluster-level setting that can be overridden on a topic level by using the preallocate setting.
config.
kafka.
kafkaConfig_2_6.
socketSendBufferBytes		 integer (int64)

The SO_SNDBUF buffer of the socket server sockets. If the value is -1, the OS default will be used.
config.
kafka.
kafkaConfig_2_6.
socketReceiveBufferBytes		 integer (int64)

The SO_RCVBUF buffer of the socket server sockets. If the value is -1, the OS default will be used.
config.
kafka.
kafkaConfig_2_6.
autoCreateTopicsEnable		 boolean (boolean)

Enable auto creation of topic on the server
config.
kafka.
kafkaConfig_2_6.
numPartitions		 integer (int64)

Default number of partitions per topic on the whole cluster
config.
kafka.
kafkaConfig_2_6.
defaultReplicationFactor		 integer (int64)

Default replication factor of the topic on the whole cluster
config.
kafka.
kafkaConfig_2_8		 object
config.kafka includes only one of the fields kafkaConfig_2_1, kafkaConfig_2_6, kafkaConfig_2_8

Kafka version 2.8 broker configuration.
config.
kafka.
kafkaConfig_2_8.
compressionType		 string

Cluster topics compression type.

  • COMPRESSION_TYPE_UNCOMPRESSED: no codec (uncompressed).
  • COMPRESSION_TYPE_ZSTD: Zstandard codec.
  • COMPRESSION_TYPE_LZ4: LZ4 codec.
  • COMPRESSION_TYPE_SNAPPY: Snappy codec.
  • COMPRESSION_TYPE_GZIP: GZip codec.
  • COMPRESSION_TYPE_PRODUCER: the codec to use is set by a producer (can be any of ZSTD, LZ4, GZIP or SNAPPY codecs).
config.
kafka.
kafkaConfig_2_8.
logFlushIntervalMessages		 integer (int64)

The number of messages accumulated on a log partition before messages are flushed to disk.

This is the global cluster-level setting that can be overridden on a topic level by using the flushMessages setting.
config.
kafka.
kafkaConfig_2_8.
logFlushIntervalMs		 integer (int64)

The maximum time (in milliseconds) that a message in any topic is kept in memory before flushed to disk. If not set, the value of logFlushSchedulerIntervalMs is used.

This is the global cluster-level setting that can be overridden on a topic level by using the flushMs setting.
config.
kafka.
kafkaConfig_2_8.
logFlushSchedulerIntervalMs		 integer (int64)

The frequency of checks (in milliseconds) for any logs that need to be flushed to disk. This check is done by the log flusher.
config.
kafka.
kafkaConfig_2_8.
logRetentionBytes		 integer (int64)

Partition size limit; Kafka will discard old log segments to free up space if delete cleanupPolicy is in effect. This setting is helpful if you need to control the size of a log due to limited disk space.

This is the global cluster-level setting that can be overridden on a topic level by using the retentionBytes setting.
config.
kafka.
kafkaConfig_2_8.
logRetentionHours		 integer (int64)

The number of hours to keep a log segment file before deleting it.
config.
kafka.
kafkaConfig_2_8.
logRetentionMinutes		 integer (int64)

The number of minutes to keep a log segment file before deleting it.

If not set, the value of logRetentionHours is used.
config.
kafka.
kafkaConfig_2_8.
logRetentionMs		 integer (int64)

The number of milliseconds to keep a log segment file before deleting it.

If not set, the value of logRetentionMinutes is used.

This is the global cluster-level setting that can be overridden on a topic level by using the retentionMs setting.
config.
kafka.
kafkaConfig_2_8.
logSegmentBytes		 integer (int64)

The maximum size of a single log file.

This is the global cluster-level setting that can be overridden on a topic level by using the segmentBytes setting.
config.
kafka.
kafkaConfig_2_8.
logPreallocate		 boolean (boolean)

Should pre allocate file when create new segment?

This is the global cluster-level setting that can be overridden on a topic level by using the preallocate setting.
config.
kafka.
kafkaConfig_2_8.
socketSendBufferBytes		 integer (int64)

The SO_SNDBUF buffer of the socket server sockets. If the value is -1, the OS default will be used.
config.
kafka.
kafkaConfig_2_8.
socketReceiveBufferBytes		 integer (int64)

The SO_RCVBUF buffer of the socket server sockets. If the value is -1, the OS default will be used.
config.
kafka.
kafkaConfig_2_8.
autoCreateTopicsEnable		 boolean (boolean)

Enable auto creation of topic on the server
config.
kafka.
kafkaConfig_2_8.
numPartitions		 integer (int64)

Default number of partitions per topic on the whole cluster
config.
kafka.
kafkaConfig_2_8.
defaultReplicationFactor		 integer (int64)

Default replication factor of the topic on the whole cluster
config.
zookeeper		 object

Configuration and resource allocation for ZooKeeper hosts.
config.
zookeeper.
resources		 object

Resources allocated to ZooKeeper hosts.
config.
zookeeper.
resources.
resourcePresetId		 string

ID of the preset for computational resources available to a host (CPU, memory, etc.). All available presets are listed in the documentation.
config.
zookeeper.
resources.
diskSize		 string (int64)

Volume of the storage available to a host, in bytes.
config.
zookeeper.
resources.
diskTypeId		 string

Type of the storage environment for the host.
config.
zoneId[]		 string

IDs of availability zones where Kafka brokers reside.
config.
brokersCount		 integer (int64)

The number of Kafka brokers deployed in each availability zone.
config.
assignPublicIp		 boolean (boolean)

The flag that defines whether a public IP address is assigned to the cluster. If the value is true, then Apache Kafka® cluster is available on the Internet via it's public IP address.
config.
unmanagedTopics		 boolean (boolean)

Allows to manage topics via AdminAPI
config.
schemaRegistry		 boolean (boolean)

Enables managed schema registry on cluster
networkId string

ID of the network that the cluster belongs to.
health string

Aggregated cluster health.

  • HEALTH_UNKNOWN: state of the cluster is unknown (health of all hosts in the cluster is UNKNOWN).
  • ALIVE: cluster is alive and well (health of all hosts in the cluster is ALIVE).
  • DEAD: cluster is inoperable (health of all hosts in the cluster is DEAD).
  • DEGRADED: cluster is in degraded state (health of at least one of the hosts in the cluster is not ALIVE).
status string

Current state of the cluster.

  • STATUS_UNKNOWN: cluster state is unknown.
  • CREATING: cluster is being created.
  • RUNNING: cluster is running normally.
  • ERROR: cluster encountered a problem and cannot operate.
  • UPDATING: cluster is being updated.
  • STOPPING: cluster is stopping.
  • STOPPED: cluster stopped.
  • STARTING: cluster is starting.
securityGroupIds[] string

User security groups
hostGroupIds[] string

Host groups hosting VMs of the cluster.
deletionProtection boolean (boolean)

Deletion Protection inhibits deletion of the cluster

Methods

Method Description
create Creates a new Apache Kafka® cluster in the specified folder.
delete Deletes the specified Apache Kafka® cluster.
get Returns the specified Apache Kafka® cluster.
list Retrieves the list of Apache Kafka® clusters that belong to the specified folder.
listHosts Retrieves a list of hosts for the specified Apache Kafka® cluster.
listLogs Retrieves logs for the specified Apache Kafka® cluster.
listOperations Retrieves the list of operations for the specified Apache Kafka® cluster.
move Moves the specified Apache Kafka® cluster to the specified folder.
start Starts the specified Apache Kafka® cluster.
stop Stops the specified Apache Kafka® cluster.
streamLogs Same as listLogs but using server-side streaming. Also allows for tail -f semantics.
update Updates the specified Apache Kafka® cluster.
Language / Region
© 2021 ООО «Яндекс.Облако»
В этой статье: