VSAM在z/OS上的复制-（3）与Kafka的协作
首先
我试过在z/OS上的VSAM文件转到Linux上的Db2进行复制的例子到这里。这次我会尝试改变目标，使用Kafka代替Db2来构建配置。换句话说，我将尝试将VSAM在z/OS上复制到Linux上的Kafka。
相关文章
VSAM on z/OS 的复制 – (1) 环境配置（到 Db2）
VSAM on z/OS 的复制 – (2) 复制配置（到 Db2）
VSAM on z/OS 的复制 – (3) 与 Kafka 联动
VSAM on z/OS 的复制 – (4) 添加 VSAM 文件（通过 Kafka 联动）/ 复制测试
整个景象
由于将Db2替换为Kafka，因此我们也将Target Engine替换为适用于Kafka的引擎。除此之外，我们将继续使用现有的构建。
搭建目标环境（Kafka）
请参考以下的CDC Replication Engine for Kafka。
卡夫卡（Confluent平台）
准备所需的Kafka环境。在这里，我们使用Confluent平台。由于只是用于测试，所以我们将使用单节点、单个Broker，并基本使用默认配置。
请参考以下具体的环境搭建步骤，详见Confluent平台备忘录。(1)环境搭建。
目标引擎（Kafka的CDC复制引擎）安装
创建管理员用户
我会创建一个名为cdckafka的管理用户，并设置一个密码。
[root@test12 ~]# useradd cdckafka
[root@test12 ~]# passwd cdckafka
ユーザー cdckafka のパスワードを変更。
新しいパスワード:
新しいパスワードを再入力してください:
passwd: すべての認証トークンが正しく更新できました。
安装
安装程序将使用与安装Db2引擎时相同的选项。
参考：（1）环境设置 – （6）目标引擎（CDC复制引擎适用于Db2 for LUW）- 安装
使用cdckafka用户执行安装程序的bin(setup-iidr-11.4.0.4-5602-linux-x86.bin)。
请按照指示进行安装。
准备安装
正在从安装程序存档中提取JRE……
正在解压JRE……
正在从安装程序存档中提取安装资源……
正在配置此系统环境的安装程序…正在启动安装程序…
===============================================================================
选择区域设置…
———-
1- 英语
->2- 日语
请输入区域设置号码进行选择: 2
===============================================================================
IBM InfoSphere Data Replication (由InstallAnywhere创建)
——————————————————————————-
准备进行控制台模式安装…
===============================================================================
概述
使用InstallAnywhere来安装IBM InfoSphere Data Replication 11.4.0.4。
响应每个提示以进行安装的下一步。
如果您希望返回上一步进行更改，请输入”back”。
输入”quit”以随时取消安装。
->1- 安装新产品
2- 升级现有产品
输入选择项的编号或按
接受默认值: 1
===============================================================================
选择数据存储类型
->1- IBM Db2 for z/OS
2- Kafka
3- IBM DB2
4- IBM InfoSphere DataStage (包括Apache Hadoop，WebHDFS，Cloudant)
5- FlexRep
6- IBM Netezza
7- Oracle
8- Oracle XStream
9- Oracle Trigger
10- Teradata
11- PostgreSQL
12- MySQL
输入选择项的编号或按
接受默认值: 2
===============================================================================
选择提供
如果您对符合资格的提供的判断感到困惑，
请与销售代表联系。
->1- IBM数据复制
2- IBM数据复制z/OS源目标
3- IBM InfoSphere数据复制非RDBMS目标
4- IBM InfoSphere数据复制
5- IBM InfoSphere变更数据传递
6- IBM InfoSphere变更数据传递用于PureData System for Analytics
7- IBM数据复制Db2 for z/OS远程源
8- IBM数据复制VSAM for z/OS远程源
输入选择项的编号或按
接受默认值: 4
===============================================================================
选择安装文件夹
—————
安装文件夹
默认安装文件夹: /opt/ibm/InfoSphereDataReplication/ReplicationEngineforKafka
输入绝对路径或按ENTER
使用默认值：
:
===============================================================================
许可证信息
除了IBM和客户事先达成一致的”程序”使用许可条件，
以下许可证信息根据以下条款使用。
如果客户未事先同意”程序”的有效使用许可条件，
则将适用”程序”的使用条件(Z125-3301-14)。
程序名称 (程序编号):
IBM InfoSphere数据复制V11.4.0.4 (5725-E30)
IBM InfoSphere非生产环境数据复制V11.
4.0.4 (5725-E30)
以下标准条件适用于被许可人使用的”程序”。
受限制的使用权
除套餐程序以外，根据授权提供给许可人的所有IBM
如果要继续阅读使用条款，请按Enter
如果同意，请按1，
如果不同意，请按2，
如果要打印，请按3，
如果要以英文查看，请按5。
如果要返回上一屏幕，请按99：1
===============================================================================
选择实例文件夹
—————
实例和产品日志将被创建在哪里？
输入绝对路径或按Enter
接受默认值
实例文件夹: (默认值:
/opt/ibm/InfoSphereDataReplication/ReplicationEngineforKafka):
===============================================================================
预安装摘要
———-
在继续之前，请确认以下内容。
产品名称：
IBM InfoSphere数据复制
安装文件夹：
/opt/ibm/InfoSphereDataReplication/ReplicationEngineforKafka
实例文件夹：
“/opt/ibm/InfoSphereDataReplication/ReplicationEngineforKafka”
数据存储类型：
Kafka
提供：
IBM InfoSphere数据复制
磁盘容量信息（安装目标）：
所需磁盘容量： 422,637,876 字节
可用磁盘容量：49,491,722,240 字节
按ENTER键继续：
===============================================================================
安装中…
———-
[==================|==================|==================|==================]
[——————|——————|——————|——————]
===============================================================================
安装已完成
————-
恭喜！IBM InfoSphere数据复制已成功安装在以下位置。
/opt/ibm/InfoSphereDataReplication/ReplicationEngineforKafka
您可以运行/opt/ibm/InfoSphereDataReplication/ReplicationEngineforKafka/bin/dmconfigurets
来
启动配置工具。
是否启动配置工具？（1=是，2=否）
（默认值: 1）：2
创建/启动 Target Engine 实例
使用cdckafka用户执行以下命令以启动配置工具。
（需要启动GUI工具，因此需要X Server。）
[cdckafka@test12 ~]$ /opt/ibm/InfoSphereDataReplication/ReplicationEngineforKafka/bin/dmconfigurets
在/conf/目录下已经创建了用于Producer和Consumer的属性文件。
#Default values are provided as below: 'retries' is used for retry times and 'retry.backoff.ms' is for sleep time during each retry
#Mon Aug 16 11:37:30 JST 2021
retries=3
retry.backoff.ms=500
batch.size=65536
#
#Mon Aug 16 11:37:30 JST 2021
这次我们就暂且保持默认设置吧。
启动/停止Target Engine实例。
启动命令: amdts64
停止命令: dmshutdown
从第二次开始，您无需启动GUI工具，只需使用命令即可启动/停止实例。
您可以使用admts64命令启动实例。（以cdckafka用户身份运行）
参考：启动CDC Replication Engine for Kafka。
[cdckafka@test12 ~]$ /opt/ibm/InfoSphereDataReplication/ReplicationEngineforKafka/bin/dmts64 -I cdc_kafka_test01 &
您可以使用dmshutdown命令启动实例。(在cdckafka用户下执行)
参考：停止CDC Replication Engine for Kafka
[cdckafka@test12 ~]$ /opt/ibm/InfoSphereDataReplication/ReplicationEngineforKafka/bin/dmshutdown -I cdc_kafka_test01
IBM InfoSphere Data Replication が正常にシャットダウンされました。
网络结构
我們將允許對於防火牆的11701端口進行訪問。
[root@test12 ~]# firewall-cmd --zone=public --add-port=11701/tcp --permanent
success
[root@test12 ~]# firewall-cmd --reload
success
复制配置
设置源代码
源端（VSAM）的设置将使用之前的文章中创建的设置。
参考：（2）复制配置-源端设置
目标设定
如果作为目标方拥有Kafka服务器和目标引擎的实例就好了。
复制设置（订阅）
通过管理控制台连接到访问服务器，并进行各种对象的定义。
创建适用于源码的数据存储
我会直接使用在上一篇文章中创建的内容。
参考：（2）复制配置 – 复制设置 – 创建源数据存储
创建目标数据存储库
用户的分配
对于先前创建的数据存储，进行用户的分配。
创建订阅
Kafka属性的设置
参考：指定Kafka适用的订阅的连接设置
按照上面显示的警告，右键单击刚创建的订阅-选择Kafka属性。
执行复制
刷新
首先，我们将现有的VSAM数据映射到Kafka上。
在此期间，将保持CICS中的VSAM处于关闭状态。
我們來查看 Kafka 上的主題列表。
[root@test12 /opt/confluent-6.2.0/bin]# ./kafka-topics --list --bootstrap-server localhost:9092
__consumer_offsets
_schemas
cdc_kafka_test01-VSAMT02-commitstream
cdc_kafka_test01.vsamt02.sourcedb.vsam.test01.filea
已创建两个主题：cdc_kafka_test01-VSAMT02-commitstream和cdc_kafka_test01.vsamt02.sourcedb.vsam.test01.filea。
使用kafka-avro-console-consumer命令查看Topiccdc_kafka_test01.vsamt02.sourcedb.vsam.test01.filea的消息。
[root@test12 /opt/confluent-6.2.0/bin]# ./kafka-avro-console-consumer --from-beginning --topic cdc_kafka_test01.vsamt02.sourcedb.vsam.test01.filea \
> --bootstrap-server localhost:9092 \
> --property print.key=true \
> --property print.schema.ids=true \
> --property schema.id.separator=: \
> --property schema.registry.url=http://localhost:8081
{"NUMB":{"string":"000001"}}:6 {"STAT":{"string":"U"},"NUMB":{"string":"000001"},"NAME":{"string":"AAAAAA "},"ADDRX":{"string":"AAA "},"PHONE":{"string":"1111 "},"DATEX":{"string":"11 "},"AMOUNT":{"string":"$0001.00"},"COMMENT":{"string":"AAAAAA "}}:5
{"NUMB":{"string":"000002"}}:6 {"STAT":{"string":"U"},"NUMB":{"string":"000002"},"NAME":{"string":"AAA "},"ADDRX":{"string":"TTT "},"PHONE":{"string":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},"DATEX":{"string":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},"AMOUNT":{"string":"$0001.00"},"COMMENT":{"string":"COMMENT "}}:5
{"NUMB":{"string":"000003"}}:6 {"STAT":{"string":"U"},"NUMB":{"string":"000003"},"NAME":{"string":"TTTTT "},"ADDRX":{"string":"XXX "},"PHONE":{"string":"333-333 "},"DATEX":{"string":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},"AMOUNT":{"string":"$3333.00"},"COMMENT":{"string":"CCCCC "}}:5
{"NUMB":{"string":"000004"}}:6 {"STAT":{"string":"A"},"NUMB":{"string":"000004"},"NAME":{"string":"DDDDD "},"ADDRX":{"string":"DDD "},"PHONE":{"string":"444444 "},"DATEX":{"string":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},"AMOUNT":{"string":"$4444.00"},"COMMENT":{"string":"DDDD "}}:5
{"NUMB":{"string":"000100"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"000100"},"NAME":{"string":"S. D. BORMAN "},"ADDRX":{"string":"SURREY, ENGLAND "},"PHONE":{"string":"32156778"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0100.11"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"000102"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"000102"},"NAME":{"string":"J. T. CZAYKOWSKI "},"ADDRX":{"string":"WARWICK, ENGLAND "},"PHONE":{"string":"98356183"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$1111.11"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"000104"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"000104"},"NAME":{"string":"M. B. DOMBEY "},"ADDRX":{"string":"LONDON,ENGLAND "},"PHONE":{"string":"12846293"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0999.99"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"000106"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"000106"},"NAME":{"string":"A. I. HICKSON "},"ADDRX":{"string":"CROYDON, ENGLAND "},"PHONE":{"string":"19485673"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0087.71"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"000111"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"000111"},"NAME":{"string":"ALAN TULIP "},"ADDRX":{"string":"SARATOGA,CALIFORNIA "},"PHONE":{"string":"46120753"},"DATEX":{"string":"01 02 74"},"AMOUNT":{"string":"$0111.11"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"000762"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"000762"},"NAME":{"string":"SUSAN MALAIKA "},"ADDRX":{"string":"SAN JOSE,CALIFORNIA "},"PHONE":{"string":"22312121"},"DATEX":{"string":"01 06 74"},"AMOUNT":{"string":"$0000.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"000983"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"000983"},"NAME":{"string":"J. S. TILLING "},"ADDRX":{"string":"WASHINGTON, DC "},"PHONE":{"string":"34512120"},"DATEX":{"string":"21 04 75"},"AMOUNT":{"string":"$9999.99"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"001222"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"001222"},"NAME":{"string":"D.J.VOWLES "},"ADDRX":{"string":"BOBLINGEN, GERMANY "},"PHONE":{"string":"70315551"},"DATEX":{"string":"10 04 73"},"AMOUNT":{"string":"$3349.99"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"001781"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"001781"},"NAME":{"string":"TINA J YOUNG "},"ADDRX":{"string":"SINDELFINGEN,GERMANY"},"PHONE":{"string":"70319990"},"DATEX":{"string":"21 06 77"},"AMOUNT":{"string":"$0009.99"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"003210"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"003210"},"NAME":{"string":"B.A. WALKER "},"ADDRX":{"string":"NICE, FRANCE "},"PHONE":{"string":"12345670"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$3349.99"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"003214"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"003214"},"NAME":{"string":"PHIL CONWAY "},"ADDRX":{"string":"SUNNYVALE, CAL. "},"PHONE":{"string":"34112120"},"DATEX":{"string":"00 06 73"},"AMOUNT":{"string":"$0009.99"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"003890"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"003890"},"NAME":{"string":"BRIAN HARDER "},"ADDRX":{"string":"NICE, FRANCE "},"PHONE":{"string":"00000000"},"DATEX":{"string":"28 05 74"},"AMOUNT":{"string":"$0009.99"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"004004"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"004004"},"NAME":{"string":"JANET FOUCHE "},"ADDRX":{"string":"DUBLIN, IRELAND "},"PHONE":{"string":"71112121"},"DATEX":{"string":"02 11 73"},"AMOUNT":{"string":"$1259.99"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"004445"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"004445"},"NAME":{"string":"DR. P. JOHNSON "},"ADDRX":{"string":"SOUTH BEND, S.DAK. "},"PHONE":{"string":"61212120"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0009.99"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"004878"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"004878"},"NAME":{"string":"ADRIAN JONES "},"ADDRX":{"string":"SUNNYVALE, CALIF. "},"PHONE":{"string":"32212120"},"DATEX":{"string":"10 06 73"},"AMOUNT":{"string":"$5399.99"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"005005"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"005005"},"NAME":{"string":"A. E. DALTON "},"ADDRX":{"string":"SAN FRANCISCO, CA. "},"PHONE":{"string":"00000001"},"DATEX":{"string":"01 08 73"},"AMOUNT":{"string":"$0009.99"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"005444"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"005444"},"NAME":{"string":"ROS READER "},"ADDRX":{"string":"SARATOGA, CALIF. "},"PHONE":{"string":"67712120"},"DATEX":{"string":"20 10 74"},"AMOUNT":{"string":"$0809.99"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"005581"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"005581"},"NAME":{"string":"PETE ROBBINS "},"ADDRX":{"string":"BOSTON, MASS. "},"PHONE":{"string":"41312120"},"DATEX":{"string":"11 04 74"},"AMOUNT":{"string":"$0259.99"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"006016"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"006016"},"NAME":{"string":"SIR MICHAEL ROBERTS "},"ADDRX":{"string":"NEW DELHI, INDIA "},"PHONE":{"string":"70331211"},"DATEX":{"string":"21 05 74"},"AMOUNT":{"string":"$0009.88"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"006670"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"006670"},"NAME":{"string":"IAN HALL "},"ADDRX":{"string":"NEW YORK, N.Y. "},"PHONE":{"string":"21212120"},"DATEX":{"string":"31 01 75"},"AMOUNT":{"string":"$3509.88"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"006968"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"006968"},"NAME":{"string":"J.A.L. STAINFORTH "},"ADDRX":{"string":"WARWICK, ENGLAND "},"PHONE":{"string":"56713821"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0009.88"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"007007"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"007007"},"NAME":{"string":"ANDREW WHARMBY "},"ADDRX":{"string":"STUTTGART, GERMANY "},"PHONE":{"string":"70311000"},"DATEX":{"string":"10 10 75"},"AMOUNT":{"string":"$5009.88"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"007248"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"007248"},"NAME":{"string":"M. J. AYRES "},"ADDRX":{"string":"REDWOOD CITY, CALF. "},"PHONE":{"string":"33312121"},"DATEX":{"string":"11 10 75"},"AMOUNT":{"string":"$0009.88"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"007779"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"007779"},"NAME":{"string":"MRS. A. STEWART "},"ADDRX":{"string":"SAN JOSE, CALIF. "},"PHONE":{"string":"41512120"},"DATEX":{"string":"03 01 75"},"AMOUNT":{"string":"$0009.88"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"009000"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"009000"},"NAME":{"string":"P. E. HAVERCAN "},"ADDRX":{"string":"WATERLOO, ONTARIO "},"PHONE":{"string":"09876543"},"DATEX":{"string":"21 01 75"},"AMOUNT":{"string":"$9000.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"100000"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"100000"},"NAME":{"string":"M. ADAMS "},"ADDRX":{"string":"TORONTO, ONTARIO "},"PHONE":{"string":"03415121"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0010.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"111111"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"111111"},"NAME":{"string":"C. BAKER "},"ADDRX":{"string":"OTTAWA, ONTARIO "},"PHONE":{"string":"51212003"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0011.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"200000"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"200000"},"NAME":{"string":"S. P. RUSSELL "},"ADDRX":{"string":"GLASGOW, SCOTLAND "},"PHONE":{"string":"63738290"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0020.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"222222"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"222222"},"NAME":{"string":"DR E. GRIFFITHS "},"ADDRX":{"string":"FRANKFURT, GERMANY "},"PHONE":{"string":"20034151"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0022.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"300000"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"300000"},"NAME":{"string":"V. J. HARRIS "},"ADDRX":{"string":"NEW YORK, U.S. "},"PHONE":{"string":"64739801"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0030.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"333333"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"333333"},"NAME":{"string":"J.D. HENRY "},"ADDRX":{"string":"CARDIFF, WALES "},"PHONE":{"string":"78493020"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0033.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"400000"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"400000"},"NAME":{"string":"C. HUNT "},"ADDRX":{"string":"MILAN, ITALY "},"PHONE":{"string":"25363738"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0040.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"444444"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"444444"},"NAME":{"string":"D. JACOBS "},"ADDRX":{"string":"CALGARY, ALBERTA "},"PHONE":{"string":"77889820"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0044.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"500000"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"500000"},"NAME":{"string":"P. KINGSLEY "},"ADDRX":{"string":"MADRID, SPAIN "},"PHONE":{"string":"44454640"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0000.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"555555"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"555555"},"NAME":{"string":"S.J. LAZENBY "},"ADDRX":{"string":"KINGSTON, N.Y. "},"PHONE":{"string":"39944420"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0005.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"600000"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"600000"},"NAME":{"string":"M.F. MASON "},"ADDRX":{"string":"DUBLIN, IRELAND "},"PHONE":{"string":"12398780"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0010.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"666666"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"666666"},"NAME":{"string":"R. F. WALLER "},"ADDRX":{"string":"LA HULPE, BRUSSELS "},"PHONE":{"string":"42983840"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0016.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"700000"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"700000"},"NAME":{"string":"M. BRANDON "},"ADDRX":{"string":"DALLAS, TEXAS "},"PHONE":{"string":"57984320"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0002.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"777777"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"777777"},"NAME":{"string":"L.A. FARMER "},"ADDRX":{"string":"WILLIAMSBURG, VIRG. "},"PHONE":{"string":"91876131"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0027.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"800000"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"800000"},"NAME":{"string":"P. LUPTON "},"ADDRX":{"string":"WESTEND, LONDON "},"PHONE":{"string":"24233389"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0030.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"888888"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"888888"},"NAME":{"string":"P. MUNDY "},"ADDRX":{"string":"NORTHAMPTON, ENG. "},"PHONE":{"string":"23691639"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0038.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"900000"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"900000"},"NAME":{"string":"D.S. RENSHAW "},"ADDRX":{"string":"TAMPA, FLA. "},"PHONE":{"string":"35668120"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0040.00"},"COMMENT":{"string":"*********"}}:5
{"NUMB":{"string":"999999"}}:6 {"STAT":{"string":" "},"NUMB":{"string":"999999"},"NAME":{"string":"ANJI STEVENS "},"ADDRX":{"string":"RALEIGH, N.Y. "},"PHONE":{"string":"84591639"},"DATEX":{"string":"26 11 81"},"AMOUNT":{"string":"$0049.00"},"COMMENT":{"string":"*********"}}:5
[root@test12 ~]# curl http://localhost:8081/schemas/ids/5
{“schema”:”{\”type\”:\”record\”,\”name\”:\”FILEA\”,\”namespace\”:\”value.SOURCEDB.VSAM.TEST01\”,\”fields\”:[{\”name\”:\”STAT\”,\”type\”:[{\”type\”:\”string\”,\”logicalType\”:\”CHARACTER\”,\”dbColumnName\”:\”STAT\”,\”length\”:1},\”null\”],\”doc\”:\”\”,\”default\”:\”\”},{\”name\”:\”NUMB\”,\”type\”:[{\”type\”:\”string\”,\”logicalType\”:\”CHARACTER\”,\”dbColumnName\”:\”NUMB\”,\”length\”:6},\”null\”],\”doc\”:\”\”,\”default\”:\”\”},{\”name\”:\”NAME\”,\”type\”:[{\”type\”:\”string\”,\”logicalType\”:\”CHARACTER\”,\”dbColumnName\”:\”NAME\”,\”length\”:20},\”null\”],\”doc\”:\”\”,\”default\”:\”\”},{\”name\”:\”ADDRX\”,\”type\”:[{\”type\”:\”string\”,\”logicalType\”:\”CHARACTER\”,\”dbColumnName\”:\”ADDRX\”,\”length\”:20},\”null\”],\”doc\”:\”\”,\”default\”:\”\”},{\”name\”:\”PHONE\”,\”type\”:[{\”type\”:\”string\”,\”logicalType\”:\”CHARACTER\”,\”dbColumnName\”:\”PHONE\”,\”length\”:8},\”null\”],\”doc\”:\”\”,\”default\”:\”\”},{\”name\”:\”DATEX\”,\”type\”:[{\”type\”:\”string\”,\”logicalType\”:\”CHARACTER\”,\”dbColumnName\”:\”DATEX\”,\”length\”:8},\”null\”],\”doc\”:\”\”,\”default\”:\”\”},{\”name\”:\”AMOUNT\”,\”type\”:[{\”type\”:\”string\”,\”logicalType\”:\”CHARACTER\”,\”dbColumnName\”:\”AMOUNT\”,\”length\”:8},\”null\”],\”doc\”:\”\”,\”default\”:\”\”},{\”name\”:\”COMMENT\”,\”type\”:[{\”type\”:\”string\”,\”logicalType\”:\”CHARACTER\”,\”dbColumnName\”:\”COMMENT\”,\”length\”:9},\”null\”],\”doc\”:\”\”,\”default\”:\”\”}]}}为了使其可读性更高，进行了整理
[root@test12 ~]# curl -s http://localhost:8081/schemas/ids/5 | jq .schema | sed -e ‘s/^”//’ -e ‘s/”$//’ -e ‘s/\\//g’ | jq .
{
“type”: “record”,
“name”: “FILEA”,
“namespace”: “value.SOURCEDB.VSAM.TEST01”,
“fields”: [
{
“name”: “STAT”,
“type”: [
{
“type”: “string”,
“logicalType”: “CHARACTER”,
“dbColumnName”: “STAT”,
“length”: 1
},
“null”
],
“doc”: “”,
“default”: “”
},
{
“name”: “NUMB”,
“type”: [
{
“type”: “string”,
“logicalType”: “CHARACTER”,
“dbColumnName”: “NUMB”,
“length”: 6
},
“null”
],
“doc”: “”,
“default”: “”
},
{
“name”: “NAME”,
“type”: [
{
“type”: “string”,
“logicalType”: “CHARACTER”,
“dbColumnName”: “NAME”,
“length”: 20
},
“null”
],
“doc”: “”,
“default”: “”
},
{
“name”: “ADDRX”,
“type”: [
{
“type”: “string”,
“logicalType”: “CHARACTER”,
“dbColumnName”: “ADDRX”,
“length”: 20
},
“null”
],
“doc”: “”,
“default”: “”
},
{
“name”: “PHONE”,
“type”: [
{
“type”: “string”,
“logicalType”: “CHARACTER”,
“dbColumnName”: “PHONE”,
“length”: 8
},
“null”
],
“doc”: “”,
“default”: “”
},
{
“name”: “DATEX”,
“type”: [
{
“type”: “string”,
“logicalType”: “CHARACTER”,
“dbColumnName”: “DATEX”,
“length”: 8
},
“null”
],
“doc”: “”,
“default”: “”
},
{
“name”: “AMOUNT”,
“type”: [
{
“type”: “string”,
“logicalType”: “CHARACTER”,
“dbColumnName”: “AMOUNT”,
“length”: 8
},
“null”
],
“doc”: “”,
“default”: “”
},
{
“name”: “COMMENT”,
“type”: [
{
“type”: “string”,
“logicalType”: “CHARACTER”,
“dbColumnName”: “COMMENT”,
“length”: 9
},
“null”
],
“doc”: “”,
“default”: “”
}
]
}
以下是对上述命令的简要说明：
从 Schema Registry 使用 curl 获取信息，将返回的 schema 的值作为一个字符串。
然后使用 jq 命令提取出 schema 元素的值。
再使用 sed 命令删除开头和结尾的双引号，并去除转义字符（斜线）。
最后再次使用 jq 命令进行格式化并显示。
在创建订阅时选择了“多个Kafka映射”，导致key变为null。
[root@test12 /opt/confluent-6.2.0/bin]# ./kafka-avro-console-consumer –from-beginning –topic cdc_kafka_test01.vsamt02.s
开始镜像
请确认插入
我将在CICS端打开该文件并执行更新该文件的操作。我将使用CICS提供的示例事务AMNU，尝试添加以下记录（插入）。
FILE ADD
NUMBER: 000005
NAME: EEEEE
ADDRESS: EEE
PHONE: 55555
DATE: 55
AMOUNT: $0055.00
COMMENT: DDDDD
ENTER DATA AND PRESS ENTER KEY
使用kafka-avro-console-consumer命令查看Topiccdc_kafka_test01.vsamt02.sourcedb.vsam.test01.filea的消息时，添加了以下消息。
{"NUMB":{"string":"000005"}}:6 {"STAT":{"string":"A"},"NUMB":{"string":"000005"},"NAME":{"string":"EEEEE "},"ADDRX":{"string":"EEE "},"PHONE":{"string":"55555 "},"DATEX":{"string":"55 "},"AMOUNT":{"string":"$0055.00"},"COMMENT":{"string":"DDDDD "}}:5
更新确认
我尝试更新CICS事务中NUMBER:000001的记录。
FILE UPDATE
NUMBER: 000001
NAME: TTTTTT
ADDRESS: TTT
PHONE: 9999
DATE: 11
AMOUNT: $0099.00
COMMENT: TTTTTT
CHANGE FIELDS AND PRESS ENTER
经过消费者的观察，可以确认以下消息已被添加。
{"NUMB":{"string":"000001"}}:6 {"STAT":{"string":"U"},"NUMB":{"string":"000001"},"NAME":{"string":"TTTTTT "},"ADDRX":{"string":"TTT "},"PHONE":{"string":"9999 "},"DATEX":{"string":"11 "},"AMOUNT":{"string":"$0099.00"},"COMMENT":{"string":"TTTTTT "}}:5
※第一个000001的消息保留不变，新的消息使用相同的键添加了进去。
刪除的確認
让我们从CICS事务中删除NUMBER:000005的记录。
从消费者的角度来看，我可以确认以下消息已经被添加了。
{"NUMB":{"string":"000005"}} null
在这种情况下，请务必删除kafka-avro-console-consumer命令的–property print.schema.ids=true、–property schema.id.separator=:选项！如果不这样做，由于缺少模式ID，消费者会崩溃！
