Command Line

cctools is command Line tool for helping on Kafka operations or migrations between different Kafka platforms. Cluster migrations are not a trivial task, and this tool aims to help on this process.

This CLI uses Kafka client and REST APIs to extract and export all the resources from the Source cluster in order to replicate them on the target cluster. It was tested with Confluent Platform and Confluent Cloud clusters.

It allows to export resources into different formats, that could be used as input for different tools like Confluent Cloud, Terraform, Confluent For Kubernetes or any other tool.

It provides a copy command to replicate cluster metadata, like topics or consumer groups, between clusters.

It is possible to import resources to a cluster.

Installation

Go to Releases and Download your OS distribution.

Last Release:

Example Usage

Export to JSON files all the Topics metadata from a Source Cluster. See Topic Exporters documentation for using other exporters.

Create a config.yaml file:

touch config.yaml

• Add the Source Cluster connection configuration, the source tag requires:

• Cluster: kafka section contains the source Kafka cluster connection configuration.

  • bootstrapServer: Source Cluster bootstrap server.
  • clientProps: Kafka client properties map.

Example for SASL_SSL (config.yaml):

source: 
  kafka:
    bootstrapServer: <bootstrap_server>
    clientProps:  
      - ssl.ca.location: "<path>/cacerts.pem" 
      - sasl.mechanisms: PLAIN
      - security.protocol: SASL_SSL
      - sasl.username: <admin_user>
      - sasl.password: <admin_password>

• Add the output path for the JSON files, all the Topics metadata will be exported to <export.output.path>/topics/json, one file per Topic.

export:
  output: <path>

• Filtering Topics. Exclude.

export:
  output: <path>
  topics:
    exclude: <topic_name_substring>  #_confluent

• Run the command:

cctools export topics --config config.yaml --output json

• Output folder. One JSON file per Topic exported.

JSON Topic example:

{
 "name": "demo.topic",
 "partitions": 4,
 "replicationFactor": 3,
 "minIsr": "2",
 "retentionTime": "604800000",
 "configs": [
  {
   "name": "confluent.stray.log.max.deletions.per.run",
   "value": "72"
  },
  {
   "name": "confluent.stray.log.delete.delay.ms",
   "value": "604800000"
  },
  ...
 ],
  "acls": [
  {
   "principal": "User:test",
   "host": "*",
   "operation": "ALL",
   "permission": "ALLOW",
   "resourceType": "TOPIC",
   "resourceName": "demo.topic",
   "patternType": "LITERAL"
  }
 ]

Configuration

See documentation for all the Configuration

Commands

Copy

cctools copy

A command to replicate cluster metadata, like topics or consumer groups, between clusters.

Usage: Copy

See documentation for all the Commands

Export

cctools export

A command to export cluster metadata to different formats with a single command.

Usage: Export

See documentation for all the Commands

Import

cctools import

A command to import metadata.

Usage: Import

See documentation for all the Commands

DEBUG

Enable Debug mode:LOG=DEBUG for extra logging.

LOG=DEBUG cctools <command>