使用Confluent for Kubernetes在AKS上配置Confluent Platform-CCC认证版
总结
Confluent for Kubernetes（CFK）是一个云原生的控制平台，用于在私有云环境（本例中为Azure Kubernetes Service（AKS））上部署和管理Confluent。它提供了一个基于声明式API的标准且简洁的界面，用于定制、部署和管理Confluent Platform。
使用CFK在Confluent Control Center（CCC）上进行登录认证的工作流程如下，以配置和部署Confluent Platform。
-
- 准备 Kubernetes 环境（通过预先准备完成）
-
- 部署 Confluent for Kubernetes（通过预先准备完成）
-
- 配置 Confluent Platform
-
- 部署 Confluent Platform
- 连接（登录）到 Confluent Control Center
本地环境
-
- macOS Monterey 12.3.1
-
- python 3.8.12
-
- Azure CLI 2.34.1
-
- helm v3.6.3
- kubectl v1.21.3
前期准备
-
- 执行此文，确保已构建 AKS 集群环境。
执行此文，确保使用 CFK 在 AKS 上配置和部署 Confluent Platform 的准备工作已经完成。
Confluent 平台的配置
创建基本认证密码（Basic Authentication Secret）
我会创建一个用于 CCC 登录时的基本认证的ID/密码定义文件。
c3admin: password1,Administrators
c3restricted: password2,Restricted
创建 Confluent Platform 组件的 yaml 文件。
-
- 从这里复制并创建一个定义了所有Confluent Platform组件的confluent_platform_ccc.yaml文件。
-
- 需要更改的部分包括命名空间（namespace）、URL等。
- 为了使用CCC进行登录认证，将”kind: ControlCenter”中的”spec: – authentication:”部分进行更改。
---
apiVersion: platform.confluent.io/v1beta1
kind: Zookeeper
metadata:
name: zookeeper
namespace: akscfk231
spec:
replicas: 3
image:
application: confluentinc/cp-zookeeper:7.1.0
init: confluentinc/confluent-init-container:2.3.0
dataVolumeCapacity: 10Gi
logVolumeCapacity: 10Gi
---
apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
name: kafka
namespace: akscfk231
spec:
replicas: 3
image:
application: confluentinc/cp-server:7.1.0
init: confluentinc/confluent-init-container:2.3.0
dataVolumeCapacity: 10Gi
metricReporter:
enabled: true
---
apiVersion: platform.confluent.io/v1beta1
kind: Connect
metadata:
name: connect
namespace: akscfk231
spec:
replicas: 1
image:
application: confluentinc/cp-server-connect:7.1.0
init: confluentinc/confluent-init-container:2.3.0
dependencies:
kafka:
bootstrapEndpoint: kafka:9071
---
apiVersion: platform.confluent.io/v1beta1
kind: KsqlDB
metadata:
name: ksqldb
namespace: akscfk231
spec:
replicas: 1
image:
application: confluentinc/cp-ksqldb-server:7.1.0
init: confluentinc/confluent-init-container:2.3.0
dataVolumeCapacity: 10Gi
---
apiVersion: platform.confluent.io/v1beta1
kind: ControlCenter
metadata:
name: controlcenter
namespace: akscfk231
spec:
replicas: 1
image:
application: confluentinc/cp-enterprise-control-center:7.1.0
init: confluentinc/confluent-init-container:2.3.0
dataVolumeCapacity: 10Gi
authentication:
type: basic
basic:
roles:
- Administrators
- Restricted
restrictedRoles:
- Restricted
secretRef: cccbasicsecret
dependencies:
schemaRegistry:
url: http://schemaregistry.akscfk231.svc.cluster.local:8081
ksqldb:
- name: ksqldb
url: http://ksqldb.akscfk231.svc.cluster.local:8088
connect:
- name: connect
url: http://connect.akscfk231.svc.cluster.local:8083
---
apiVersion: platform.confluent.io/v1beta1
kind: SchemaRegistry
metadata:
name: schemaregistry
namespace: akscfk231
spec:
replicas: 1
image:
application: confluentinc/cp-schema-registry:7.1.0
init: confluentinc/confluent-init-container:2.3.0
---
apiVersion: platform.confluent.io/v1beta1
kind: KafkaRestProxy
metadata:
name: kafkarestproxy
namespace: akscfk231
spec:
dependencies:
schemaRegistry:
url: http://schemaregistry.akscfk231.svc.cluster.local:8081
image:
application: confluentinc/cp-kafka-rest:7.1.0
init: confluentinc/confluent-init-container:2.3.0
replicas: 1
部署Confluent平台
在AKS环境中创建认证信息。
$ kubectl create secret generic cccbasicsecret --from-file=basic.txt=./cccbasic.txt
部署 Confluent 平台组件。
## すべての Confluent Platform コンポーネントをインストールします
kubectl apply -f confluent_platform_ccc.yaml
## Pod の確認
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
confluent-operator-76d7677b8c-q4ltf 1/1 Running 0 44m
connect-0 1/1 Running 1 (3m36s ago) 5m5s
controlcenter-0 1/1 Running 0 2m35s
kafka-0 1/1 Running 0 3m50s
kafka-1 1/1 Running 0 3m50s
kafka-2 1/1 Running 0 3m50s
kafkarestproxy-0 1/1 Running 0 2m35s
ksqldb-0 1/1 Running 1 (99s ago) 2m35s
schemaregistry-0 1/1 Running 0 2m35s
zookeeper-0 1/1 Running 0 5m5s
zookeeper-1 1/1 Running 0 5m5s
zookeeper-2 1/1 Running 0 5m5s
## デプロイされた Confluent Platform リソースの確認
$ kubectl get confluent
NAME REPLICAS READY STATUS AGE
kafkarestproxy.platform.confluent.io/kafkarestproxy 1 1 RUNNING 6m5s
NAME REPLICAS READY STATUS AGE
connect.platform.confluent.io/connect 1 1 RUNNING 6m5s
NAME REPLICAS READY STATUS AGE
controlcenter.platform.confluent.io/controlcenter 1 1 RUNNING 6m5s
NAME REPLICAS READY STATUS AGE
schemaregistry.platform.confluent.io/schemaregistry 1 1 RUNNING 6m5s
NAME REPLICAS READY STATUS AGE
zookeeper.platform.confluent.io/zookeeper 3 3 RUNNING 6m5s
NAME REPLICAS READY STATUS AGE
kafka.platform.confluent.io/kafka 3 3 RUNNING 6m5s
NAME REPLICAS READY STATUS AGE
ksqldb.platform.confluent.io/ksqldb 1 1 RUNNING 6m5s
## CCCの詳細ステータスの確認
$ kubectl describe controlcenter
Name: controlcenter
Namespace: akscfk231
Labels: <none>
Annotations: <none>
API Version: platform.confluent.io/v1beta1
Kind: ControlCenter
Metadata:
Creation Timestamp: 2022-06-23T05:33:44Z
Finalizers:
controlcenter.finalizers.platform.confluent.io
Generation: 1
Managed Fields:
API Version: platform.confluent.io/v1beta1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:authentication:
.:
f:basic:
.:
f:restrictedRoles:
f:roles:
f:secretRef:
f:type:
f:dataVolumeCapacity:
f:dependencies:
.:
f:connect:
f:ksqldb:
f:schemaRegistry:
.:
f:url:
f:image:
.:
f:application:
f:init:
f:replicas:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-06-23T05:33:44Z
API Version: platform.confluent.io/v1beta1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"controlcenter.finalizers.platform.confluent.io":
Manager: manager
Operation: Update
Time: 2022-06-23T05:33:44Z
API Version: platform.confluent.io/v1beta1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:clusterName:
f:clusterNamespace:
f:conditions:
f:controlCenterName:
f:currentReplicas:
f:id:
f:kafka:
.:
f:bootstrapEndpoint:
f:operatorVersion:
f:phase:
f:readyReplicas:
f:replicas:
f:restConfig:
.:
f:authenticationType:
f:internalEndpoint:
f:selector:
Manager: manager
Operation: Update
Subresource: status
Time: 2022-06-23T05:38:48Z
Resource Version: 16805
UID: b65d711f-d34e-4412-9c0c-d428bcf85270
Spec:
Authentication:
Basic:
Restricted Roles:
Restricted
Roles:
Administrators
Restricted
Secret Ref: basicsecret
Type: basic
Data Volume Capacity: 10Gi
Dependencies:
Connect:
Name: connect
URL: http://connect.akscfk231.svc.cluster.local:8083
Ksqldb:
Name: ksqldb
URL: http://ksqldb.akscfk231.svc.cluster.local:8088
Schema Registry:
URL: http://schemaregistry.akscfk231.svc.cluster.local:8081
Image:
Application: confluentinc/cp-enterprise-control-center:7.1.0
Init: confluentinc/confluent-init-container:2.3.0
Replicas: 1
Status:
Cluster Name: controlcenter
Cluster Namespace: akscfk231
Conditions:
Last Probe Time: 2022-06-23T05:36:14Z
Last Transition Time: 2022-06-23T05:38:48Z
Message: Deployment has minimum availability.
Reason: MinimumReplicasAvailable
Status: True
Type: platform.confluent.io/statefulset-available
Last Probe Time: 2022-06-23T05:36:14Z
Last Transition Time: 2022-06-23T05:38:48Z
Message: Kubernetes resources ready.
Reason: KubernetesResourcesReady
Status: True
Type: platform.confluent.io/resources-ready
Last Probe Time: 2022-06-23T05:36:14Z
Last Transition Time: 2022-06-23T05:36:14Z
Message: Cluster is not being garbage collected
Reason: Garbage Collection not triggered
Status: False
Type: platform.confluent.io/garbage-collecting
Control Center Name: _confluent-controlcenter
Current Replicas: 1
Id: 0
Kafka:
Bootstrap Endpoint: kafka.akscfk231.svc.cluster.local:9071
Operator Version: v0.435.23
Phase: RUNNING
Ready Replicas: 1
Replicas: 1
Rest Config:
Authentication Type: basic
Internal Endpoint: http://controlcenter.akscfk231.svc.cluster.local:9021
Selector: app=controlcenter,clusterId=akscfk231,confluent-platform=true,type=controlcenter
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Warning 4m23s (x11 over 6m38s) controlcenter waiting for at-least one kafka pod availability
Normal SuccessfulCreate 4m8s (x2 over 4m8s) controlcenter resource type *v1.Service successfully created
Normal SuccessfulCreate 4m8s (x2 over 4m8s) controlcenter resource type *v1.ConfigMap successfully created
Normal SuccessfulCreate 4m8s controlcenter resource type *v1.PersistentVolumeClaim successfully created
Normal SuccessfulCreate 4m8s controlcenter resource type *v1.StatefulSet successfully created
########## ちなみに、、、、エラーのときのイベント例.....
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning KeyInSecretRefIssue 37s (x32 over 15m) controlcenter required key [basic.txt] missing in secretRef [cccbasicsecret] for auth type [basic]
使用Confluent插件CLI工具进行确认。
## 使用方法の確認
$ kubectl confluent
## Confluent コンポーネントのバージョン確認
$ kubectl confluent version
COMPONENT NAME VERSION OPERATOR-VERSION
Zookeeper zookeeper 7.1.0 v0.435.23
Kafka kafka 7.1.0 v0.435.23
Connect connect 7.1.0 v0.435.23
SchemaRegistry schemaregistry 7.1.0 v0.435.23
KsqlDB ksqldb 7.1.0 v0.435.23
ControlCenter controlcenter 7.1.0 v0.435.23
## Confluent コンポーネントへのアクセスに使用できるエンドポイントの確認
$ kubectl confluent http-endpoints
COMPONENT NAME ACCESS ADDRESS AUTH AUTHORIZATION
Kafka kafka-rest INTERNAL http://kafka.akscfk231.svc.cluster.local:8090
Connect connect INTERNAL http://connect.akscfk231.svc.cluster.local:8083
SchemaRegistry schemaregistry INTERNAL http://schemaregistry.akscfk231.svc.cluster.local:8081
KsqlDB ksqldb INTERNAL http://ksqldb.akscfk231.svc.cluster.local:8088
ControlCenter controlcenter INTERNAL http://controlcenter.akscfk231.svc.cluster.local:9021 basic
连接至Confluent Control Center
$ kubectl confluent dashboard controlcenter
http://localhost:9021
处理后
Pod / secret / namespace 的卸载方式
## Pod : confluent-operator
$ helm delete confluent-operator
## Pod : confluent-platform
$ kubectl delete -f confluent_platform_ccc.yaml
## secret情報
$ kubectl delete secret cccbasicsecret
## namespace の削除方法（namespace配下のPodは全て削除される）
$ kubectl delete namespace akscfk231
停止和启动AKS集群
$ az aks stop -g rg_ituru_aks01 -n aks_ituru_cp01
$ az aks start -g rg_ituru_aks01 -n aks_ituru_cp01
总结
为了了解CFK的使用方法，首先从实施CCC的认证登录开始。我们确认了登录时认证顺利进行。
请提供相关信息。
我已参考了以下的信息。
Confluent for Kubernetes 概述
介绍 Confluent for Kubernetes
confluentinc/confluent-kubernetes-examples
