{"id":47192,"date":"2023-09-23T20:21:38","date_gmt":"2023-03-01T05:23:59","guid":{"rendered":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8databricks-delta-lake%e4%b8%ad%e7%ae%80%e5%8c%96change-data-capture\/"},"modified":"2024-05-04T01:13:11","modified_gmt":"2024-05-03T17:13:11","slug":"%e5%9c%a8databricks-delta-lake%e4%b8%ad%e7%ae%80%e5%8c%96change-data-capture","status":"publish","type":"post","link":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8databricks-delta-lake%e4%b8%ad%e7%ae%80%e5%8c%96change-data-capture\/","title":{"rendered":"\u5728Databricks Delta Lake\u4e2d\u7b80\u5316Change Data Capture"},"content":{"rendered":"<p>\u4f7f\u7528Databricks Delta\u7b80\u5316\u53d8\u66f4\u6570\u636e\u6355\u83b7 &#8211; \u8fd9\u662fDatabricks\u535a\u5ba2\u7684\u7ffb\u8bd1\u3002<\/p>\n<blockquote><p>\u5efa\u8bae\u9605\u8bfbDatabricks Delta Lake\u5173\u4e8e\u9ad8\u6548UPSERT\u548cDELETE\u7684MERGE\u547d\u4ee4\u7684\u4f7f\u7528\u65b9\u6cd5\uff0c\u4ee5\u5b9e\u73b0\u9ad8\u6548UPSERT\u3002<\/p><\/blockquote>\n<p>\u5728Databricks\u4e2d\uff0c\u5e38\u89c1\u7684\u7528\u4f8b\u662f\u5728\u5ba2\u6237\u7aef\u6267\u884c\u4ece\u591a\u4e2a\u6765\u6e90\u5230\u4e00\u7cfb\u5217Databricks Delta\u8868\u7684\u53d8\u66f4\u6570\u636e\u6355\u83b7(CDC)\u3002\u8fd9\u4e9b\u6765\u6e90\u53ef\u4ee5\u662f\u5728\u672c\u5730\u6216\u4e91\u4e0a\u5b58\u5728\u7684\u64cd\u4f5c\u6027\u6570\u636e\u5e93\u6216\u6570\u636e\u4ed3\u5e93\u3002\u4ee5\u4e0b\u662f\u8fde\u63a5\u5b83\u4eec\u7684\u7c98\u5408\u5242-\u901a\u8fc7\u4ee5\u4e0b\u65b9\u6cd5\u751f\u6210\u7684\u53d8\u66f4\u96c6\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">Oracle GoldenGate\u3084Informatica PowerExchange\u306e\u3088\u3046\u306aETL<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u30d9\u30f3\u30c0\u30fc\u304c\u63d0\u4f9b\u3059\u308b\u5909\u66f4\u30c6\u30fc\u30d6\u30eb(\u4f8b\uff1aOracle Change Data Capture)<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">insert\/update\/delete\u30c8\u30ea\u30ac\u30fc\u3092\u7528\u3044\u3066\u5909\u66f4\u3092\u30ad\u30e3\u30d7\u30c1\u30e3\u3059\u308b\u30e6\u30fc\u30b6\u30fc\u7ba1\u7406\u306e\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u30c6\u30fc\u30d6\u30eb<\/ul>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7d6a913a08637a69fbf2\/4-0.png\" alt=\"\" \/><\/div>\n<h1>\u80cc\u666f\uff08 \uff09<\/h1>\n<p>\u6570\u636e\u6355\u83b7\u548c\u53d8\u66f4\u6570\u636e\u6355\u83b7\uff08CDC\uff09\u662f\u6307\u6355\u6349\u4e00\u7cfb\u5217\u6570\u636e\u6e90\u4e2d\u7684\u53d8\u66f4\uff0c\u5e76\u5c06\u8fd9\u4e9b\u53d8\u66f4\u5408\u5e76\u5230\u76ee\u6807\u8868\u4e2d\uff0c\u901a\u5e38\u662f\u6570\u636e\u4ed3\u5e93\u7684\u8fc7\u7a0b\u3002\u8fd9\u4e9b\u8fc7\u7a0b\u901a\u5e38\u5728\u591c\u95f4\u3001\u5c0f\u65f6\u95f4\u9694\u6216\u66f4\u77ed\u7684\u5468\u671f\u5185\uff08\u4f8b\u5982\u6bcf15\u5206\u949f\uff09\u6267\u884c\u3002\u5728\u8fd9\u91cc\uff0c\u6211\u4eec\u5c06\u8fd9\u4e2a\u5468\u671f\u79f0\u4e3a\u5237\u65b0\u5468\u671f\u3002<\/p>\n<p>\u5728\u5237\u65b0\u5468\u671f\u5185\uff0c\u7279\u5b9a\u8868\u4e2d\u7684\u53d8\u66f4\u8bb0\u5f55\u96c6\u88ab\u79f0\u4e3a\u53d8\u66f4\u96c6\u3002\u6700\u540e\uff0c\u5177\u6709\u76f8\u540c\u4e3b\u952e\u7684\u4e00\u7cfb\u5217\u8bb0\u5f55\u5728\u53d8\u66f4\u96c6\u4e2d\u88ab\u79f0\u4e3a\u8bb0\u5f55\u96c6\u3002\u76f4\u89c2\u5730\u8bf4\uff0c\u5b83\u4eec\u88ab\u89c6\u4e3a\u9488\u5bf9\u6700\u7ec8\u8868\u4e2d\u540c\u4e00\u8bb0\u5f55\u7684\u4e0d\u540c\u66f4\u6539\u3002<\/p>\n<div>\n<div class=\"post-table\">FLAGIDVALUECDC_TIMESTAMPI1102018-01-01 16:02:00 \u200bU1112018-01-01 16:02:01D1112018-01-01 16:02:03U2202018-01-01 16:02:00D3302018-01-01 16:02:00<\/div>\n<\/div>\n<p>\u88681\uff1a\u8868\u683cT\u57282018\u5e7401\u670801\u65e517:00:00\u65f6\u7684\u53d8\u66f4\u96c6C<\/p>\n<p>\u88681\u5c55\u793a\u4e86\u7279\u5b9a\u65f6\u95f4\u70b9\u4e0a\u8868\u683cT\u7684\u53d8\u66f4\u96c6C\u3002\u53d8\u66f4\u96c6C\u67094\u4e2a\u5217\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">FLAG\u306f\u5909\u66f4\u306e\u30bf\u30a4\u30d7I\/U\/D(insert\/update\/delete)\u3092\u793a\u3057\u307e\u3059\u3002<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">ID\u306f\u30ec\u30b3\u30fc\u30c9\u3092\u4e00\u610f\u306b\u8b58\u5225\u3057\u307e\u3059\u3002<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">VALUE\u306f\u30ec\u30b3\u30fc\u30c9\u304c\u66f4\u65b0\u3055\u308c\u305f\u969b\u306e\u5024\u3092\u793a\u3057\u307e\u3059\u3002<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">CDC_TIMESTAMP\u306f\u3044\u3064\u30ec\u30b3\u30fc\u30c9\u304c\u8ffd\u52a0\u3001\u66f4\u65b0\u3001\u524a\u9664\u3055\u308c\u305f\u306e\u304b\u3092\u793a\u3057\u3066\u3044\u307e\u3059\u3002\u30bf\u30fc\u30b2\u30c3\u30c8\u30c6\u30fc\u30d6\u30ebT\u306fFLAG\u4ee5\u5916\u306f\u540c\u3058\u30b9\u30ad\u30fc\u30de\u3068\u306a\u3063\u3066\u3044\u307e\u3059\u3002<\/ul>\n<p>\u5728\u8fd9\u4e2a\u53d8\u66f4\u96c6\u4e2d\uff0cID1\u7684\u8bb0\u5f55\u8fdb\u884c\u4e86\u6dfb\u52a0\u3001\u66f4\u65b0\u548c\u5220\u9664\uff08\u4ece\u7b2c\u4e00\u884c\u5230\u7b2c\u4e09\u884c\uff09\u3002\u56e0\u6b64\uff0cID=1\u7684\u8bb0\u5f55\u96c6\u5305\u542b\u4e86\u4e09\u6761\u8bb0\u5f55\u3002ID2\u7684\u8bb0\u5f55\u4ec5\u8fdb\u884c\u4e86\u66f4\u65b0\uff0cID3\u7684\u8bb0\u5f55\u88ab\u5220\u9664\u3002\u53ef\u4ee5\u8ba4\u4e3a\u8bb0\u5f55ID2\u548cID3\u662f\u4e8b\u5148\u6dfb\u52a0\u7684\u3002<\/p>\n<h1>\u4ee5\u524d\u7684CDC\u662f\u6307Databricks Delta Lake\u4e4b\u524d\u7684\u53d8\u66f4\u6570\u636e\u6355\u83b7\u3002<\/h1>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7d6a913a08637a69fbf2\/14-0.png\" alt=\"\" \/><\/div>\n<ol>\u6211\u5011\u5df2\u7d93\u5728\u8d85\u8ca0\u8377\u7684Oracle\u5be6\u4f8b\u4e0a\u9032\u884c\u8ca0\u8f09\uff0c\u4e26\u4e14\u5c0d\u65bc\u4f55\u6642\u4ee5\u53ca\u5982\u4f55\u57f7\u884cETL\u5de5\u4f5c\u6709\u9650\u5236\u3002\u57fa\u65bc\u7d14\u7cb9\u7684Parquet\u8868\uff08\u5728Databricks Delta Lake\u4e4b\u524d\uff09\uff0c\u540c\u6642\u57f7\u884c\u7684\u80fd\u529b\u6709\u9650\uff0c\u56e0\u6b64\u591c\u9593\u7684\u5237\u65b0\u7387\u6700\u9069\u5408\u3002<\/ol>\n<h1>\u4f7f\u7528Databricks Delta Lake\u8fdb\u884cCDC\u3002<\/h1>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7d6a913a08637a69fbf2\/17-0.png\" alt=\"\" \/><\/div>\n<p>\u9488\u5bf9\u67d0\u4f4d\u5ba2\u6237\uff0c\u6211\u4eec\u5728ETL\u7ba1\u9053\u4e0a\u5b9e\u65bd\u4e86CDC\u6280\u672f\uff0c\u786e\u4fdd\u5176\u6700\u5927\u5316\u548c\u9ad8\u9891\u7387\u7684\u5237\u65b0\u3002\u6839\u636e\u8fd9\u4f4d\u5ba2\u6237\u7684\u60c5\u51b5\uff0cInformatica\u6bcf15\u5206\u949f\u5c0665\u4e2a\u8868\u4e2d\u7684\u53d8\u5316\u4f5c\u4e3a\u66f4\u6539\u96c6\u5199\u5165S3\u3002\u8fd9\u4e9b\u66f4\u6539\u96c6\u672c\u8eab\u5f88\u5c0f\uff08&lt;1000\u6761\u8bb0\u5f55\uff09\uff0c\u4f46\u76ee\u6807\u8868\u975e\u5e38\u5e9e\u5927\u3002\u5176\u4e2d\u5927\u7ea6\u67096\u4e2a\u8868\u5177\u67095000\u4e07\u81f31\u4ebf\u884c\u7684\u8bb0\u5f55\uff0c\u5176\u4f59\u7684\u8868\u884c\u6570\u5c0f\u4e8e5000\u4e07\u3002\u4e3a\u4e0eInformatica\u4fdd\u6301\u540c\u6b65\uff0c\u6211\u4eec\u5728Oracle\u4e2d\u6bcf15\u5206\u949f\u8fd0\u884c\u4e00\u6b21\u8be5\u7ba1\u9053\u3002\u5728Databricks Delta\u4e2d\uff0c\u6700\u521d\u6211\u4eec\u8ba4\u4e3a\u7531\u4e8eS3\u7684\u5ef6\u8fdf\u95ee\u9898\uff0c\u5904\u7406\u65f6\u95f4\u53ef\u80fd\u4f1a\u8fbe\u52301\u5c0f\u65f6\u5de6\u53f3\uff0c\u4f46\u4ee4\u4eba\u9ad8\u5174\u7684\u662f\uff0c\u901a\u8fc7\u8c03\u6574\u7fa4\u96c6\u7684\u89c4\u6a21\uff0c\u6211\u4eec\u6210\u529f\u5b9e\u73b0\u4e8630\u5206\u949f\u751a\u81f315\u5206\u949f\u7684\u5237\u65b0\u3002<\/p>\n<h2>\u4f7f\u7528Insert Overwrite\u547d\u4ee4<\/h2>\n<p>\u8fd9\u79cd\u65b9\u6cd5\u7684\u57fa\u672c\u601d\u60f3\u662f\u4fdd\u6301\u4e00\u4e2a\u7528\u4e8e\u7d2f\u79ef\u7279\u5b9a\u8bb0\u5f55\u96c6\u7684\u5206\u62e3\u8868\uff0c\u4ee5\u53ca\u4fdd\u7559\u7528\u6237\u53ef\u4ee5\u67e5\u8be2\u7684\u6700\u65b0\u5feb\u7167\u7684\u6700\u7ec8\u8868\u3002<\/p>\n<p><iframe><\/iframe><\/p>\n<p>\u9009\u9879\u4e00: \u56fe1\uff1a\u4eceInformatica\u7684\u6e90\u5230Databricks Delta\u4e91\u5b58\u50a8\u7684Insert Overwrite\u6d41\u7a0b<\/p>\n<p>\u5728\u6bcf\u4e2a\u5237\u65b0\u5faa\u73af\u4e2d\uff0cSpark\u7684\u4f5c\u4e1a\u5c06\u6267\u884c\u4e24\u4e2aINSERT\u64cd\u4f5c\u3002<\/p>\n<p>Insert(Insert 1): \u5f53\u8a72\u30ea\u30d5\u30ec\u30c3\u30b7\u30e5\u5468\u671f\u3067\u3001S3\u3084Kafka\u304b\u3089\u30c1\u30a7\u30f3\u30b8\u30bb\u30c3\u30c8\u3092\u8aad\u307f\u8fbc\u307f\u3001\u30b9\u30c6\u30fc\u30b8\u30f3\u30b0\u30c6\u30fc\u30d6\u30eb\u306b\u5909\u66f4\u3092INSERT\u3059\u308b\u3002<\/p>\n<p>Insert Overwrite(Insert 2): \u30b9\u30c6\u30fc\u30b8\u30f3\u30b0\u30c6\u30fc\u30d6\u30eb\u304b\u3089\u5168\u3066\u306e\u30ec\u30b3\u30fc\u30c9\u30bb\u30c3\u30c8\u306e\u73fe\u5728\u306e\u30d0\u30fc\u30b8\u30e7\u30f3\u3092\u53d6\u5f97\u3057\u3001\u30d5\u30a1\u30a4\u30ca\u30eb\u30c6\u30fc\u30d6\u30eb\u4e0a\u3067\u5bfe\u5fdc\u3059\u308b\u30ec\u30b3\u30fc\u30c9\u3092\u4e0a\u66f8\u304d\u3059\u308b\u3002<\/p>\n<p><iframe><\/iframe><\/p>\n<p>\u9009\u98791: \u56fe2\uff1a\u4eceKafka\u6e90\u7801\u5230Databricks Delta\u4e91\u5b58\u50a8\u7684Insert Overwrite\u6d41\u7a0b<\/p>\n<p>CDC\u5b9f\u8df5\u8005\u719f\u6089\u7684\u5206\u7c7b\u65b9\u6848\u662f\u4e00\u79cd\u5904\u7406\u4e0d\u540c\u7c7b\u578b\u66f4\u65b0\u7684\u65b9\u6cd5\uff0c\u88ab\u79f0\u4e3a&#8221;slowly changing dimensions&#8221;\uff08SCDs\uff09\u3002\u6211\u4eec\u7684\u6682\u5b58\u8868\u4e0eSCD Type2\u65b9\u6848\u7c7b\u4f3c\uff0c\u800c\u6700\u7ec8\u8868\u5219\u4e0eSCD Type1\u65b9\u6848\u7c7b\u4f3c\u3002<\/p>\n<h3>\u5b9e\u65bd<\/h3>\n<p>\u8ba9\u6211\u4eec\u6765\u8be6\u7ec6\u770b\u4e00\u4e0b\u8fd9\u4e24\u4e2a\u6b65\u9aa4\u3002\u9996\u5148\u662f\u7b2c\u4e00\u4e2a\u63d2\u5165\u6b65\u9aa4\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"o\">%<\/span><span class=\"n\">scala<\/span>\r\n<span class=\"k\">val<\/span> <span class=\"nv\">changeSets<\/span> <span class=\"k\">=<\/span> <span class=\"nc\">Array<\/span><span class=\"o\">(<\/span><span class=\"n\">file1<\/span><span class=\"o\">,<\/span> <span class=\"n\">file2<\/span><span class=\"o\">,<\/span> <span class=\"err\">\u2026<\/span><span class=\"o\">)<\/span>\r\n<span class=\"nv\">spark<\/span><span class=\"o\">.<\/span><span class=\"py\">read<\/span><span class=\"o\">.<\/span><span class=\"py\">parquet<\/span><span class=\"o\">(<\/span><span class=\"n\">changeSets<\/span> <span class=\"k\">:_<\/span><span class=\"kt\">*<\/span><span class=\"o\">).<\/span><span class=\"py\">createOrReplaceTempView<\/span><span class=\"o\">(<\/span><span class=\"s\">\"incremental\"<\/span><span class=\"o\">)<\/span>\r\n<\/code><\/pre>\n<pre class=\"post-pre\"><code><span class=\"o\">%<\/span><span class=\"k\">sql<\/span>\r\n<span class=\"k\">INSERT<\/span> <span class=\"k\">INTO<\/span> <span class=\"n\">T_STAGING<\/span>\r\n<span class=\"n\">PARTITION<\/span><span class=\"p\">(<\/span><span class=\"n\">CREATE_DATE_YEAR<\/span><span class=\"p\">)<\/span>\r\n<span class=\"k\">SELECT<\/span> <span class=\"n\">ID<\/span><span class=\"p\">,<\/span> <span class=\"n\">VALUE<\/span><span class=\"p\">,<\/span> <span class=\"n\">CDC_TIMESTAMP<\/span>\r\n <span class=\"k\">FROM<\/span> <span class=\"n\">INCREMENTAL<\/span>\r\n<\/code><\/pre>\n<p>\u5728\u8fd9\u91cc\uff0c\u6211\u4eec\u5b9a\u4e49\u4e86\u4e00\u4e2a\u7528\u4e8e\u5411\u7b2c\u4e8c\u4e2a\u547d\u4ee4\u7684INSERT INTO\u63d0\u4f9b\u53d8\u66f4\u96c6\u7684\u4e34\u65f6\u89c6\u56fe\u3002\u5173\u4e8eINSERT INTO\uff0c\u9664\u4e86PARTITION\u4e4b\u5916\uff0c\u5176\u4ed6\u90e8\u5206\u90fd\u76f8\u5bf9\u5bb9\u6613\u7406\u89e3\uff0c\u6211\u4eec\u6765\u8be6\u7ec6\u770b\u4e00\u4e0b\u8fd9\u4e2a\u3002<\/p>\n<p>\u5728\u4e91\u6570\u636e\u5b58\u50a8\u548cHDFS\u4e2d\uff0c\u8bb0\u5f55\u4fdd\u5b58\u5728\u6587\u4ef6\u4e2d\uff0c\u6240\u4ee5\u8bf7\u8bb0\u4f4f\u66f4\u65b0\u7684\u5355\u4f4d\u4e5f\u662f\u6587\u4ef6\u3002\u5bf9\u4e8eDatabricks Delta Lake\uff0c\u6b63\u5982\u5728\u8fd9\u7bc7\u6587\u7ae0\u4e2d\u6240\u8ff0\uff0c\u8fd9\u4e9b\u6587\u4ef6\u662fParquet\u6587\u4ef6\u3002\u5f53\u9700\u8981\u66f4\u65b0\u8bb0\u5f55\u65f6\uff0cSpark\u9700\u8981\u52a0\u8f7d\u6574\u4e2a\u6587\u4ef6\u5e76\u8fdb\u884c\u91cd\u65b0\u5199\u5165\u3002\u56e0\u6b64\uff0c\u5c06\u66f4\u65b0\u5c40\u90e8\u5316\u4ee5\u51cf\u5c11\u66f4\u65b0\u76ee\u6807\u6587\u4ef6\u7684\u6570\u91cf\u975e\u5e38\u91cd\u8981\u3002\u57fa\u4e8e\u4ee5\u4e0a\u539f\u56e0\uff0c\u6211\u4eec\u901a\u8fc7\u5728PARTITION\u5b50\u53e5(Azure|AWS)\u4e2d\u6307\u5b9a\u5206\u533a\u5217\uff0c\u5c06\u6682\u5b58\u8868\u548c\u6700\u7ec8\u8868\u90fd\u5206\u533a\u5230\u6700\u5c0f\u5316\u901a\u8fc7CDC\u64cd\u4f5c\u7684\u884c\u6570\u7684\u5217\u4e0a\uff0c\u4ee5\u4fbfDatabricks Delta Lake\u80fd\u5c06\u8bb0\u5f55\u6dfb\u52a0\u5230T_STAGING\u7684\u9002\u5f53\u5206\u533a\u3002<\/p>\n<p>\u63a5\u4e0b\u6765\uff0c\u8ba9\u6211\u4eec\u770b\u4e00\u4e0b\u7b2c\u4e8c\u4e2a\u63d2\u5165\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"o\">%<\/span><span class=\"k\">sql<\/span>\r\n<span class=\"k\">INSERT<\/span> <span class=\"n\">OVERWRITE<\/span> <span class=\"k\">TABLE<\/span> <span class=\"n\">T_FINAL<\/span>\r\n   <span class=\"n\">PARTITION<\/span><span class=\"p\">(<\/span><span class=\"n\">CREATE_DATE_YEAR<\/span><span class=\"p\">)<\/span>\r\n   <span class=\"k\">SELECT<\/span> <span class=\"n\">ID<\/span><span class=\"p\">,<\/span> <span class=\"n\">VALUE<\/span><span class=\"p\">,<\/span> <span class=\"n\">CDC_TIMESTAMP<\/span>  \r\n   <span class=\"k\">FROM<\/span> <span class=\"p\">(<\/span>\r\n      <span class=\"k\">SELECT<\/span> <span class=\"n\">A<\/span><span class=\"p\">.<\/span><span class=\"o\">*<\/span><span class=\"p\">,<\/span>\r\n             <span class=\"n\">RANK<\/span><span class=\"p\">()<\/span> <span class=\"n\">OVER<\/span> <span class=\"p\">(<\/span><span class=\"n\">PARTITION<\/span> <span class=\"k\">BY<\/span> <span class=\"n\">ID<\/span> <span class=\"k\">ORDER<\/span> <span class=\"k\">BY<\/span> <span class=\"n\">CDC_TIMESTAMP<\/span> <span class=\"k\">DESC<\/span><span class=\"p\">)<\/span> <span class=\"k\">AS<\/span> <span class=\"n\">RNK<\/span>\r\n        <span class=\"k\">FROM<\/span> <span class=\"n\">T_STAGING<\/span> <span class=\"n\">A<\/span><span class=\"p\">.<\/span><span class=\"o\">*<\/span>\r\n       <span class=\"k\">WHERE<\/span> <span class=\"n\">CREATE_DATE_YEAR<\/span> <span class=\"k\">IN<\/span> <span class=\"p\">(<\/span><span class=\"mi\">2018<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2016<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2015<\/span><span class=\"p\">)<\/span>\r\n   <span class=\"p\">)<\/span> <span class=\"n\">B<\/span>\r\n<span class=\"k\">WHERE<\/span> <span class=\"n\">B<\/span><span class=\"p\">.<\/span><span class=\"n\">RNK<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">1<\/span> <span class=\"k\">AND<\/span> <span class=\"n\">B<\/span><span class=\"p\">.<\/span><span class=\"n\">FLAG<\/span> <span class=\"o\">&lt;<\/span> <span class=\"o\">&gt;<\/span> <span class=\"s1\">'D'<\/span>\r\n<\/code><\/pre>\n<p>\u8ba9\u6211\u4eec\u4eceT_STAGING\u8bfb\u53d6\u7684\u5185\u90e8\u67e5\u8be2\u5f00\u59cb\u3002\u8bf7\u8bb0\u4f4f\uff0cstaging\u8868\u5305\u542b\u9488\u5bf9\u7279\u5b9a\u8bb0\u5f55\u96c6\u7684\u4efb\u610f\u6570\u91cf\u7684insert\u3001update\u548cdelete\u64cd\u4f5c\u3002\u7531\u4e8e\u8fd9\u4e9b\u66f4\u6539\u662f\u8de8\u591a\u4e2a\u5237\u65b0\u5468\u671f\u6dfb\u52a0\u5230staging\u8868\u4e2d\u7684\uff0c\u56e0\u6b64\u8fd9\u4e9b\u66f4\u6539\u53ef\u80fd\u662f\u7531\u7279\u5b9a\u7684change set\uff08\u4f8b\u5982\uff0c\u9488\u5bf9\u88681\u4e2dID=1\u76843\u4e2a\u66f4\u6539\uff09\u6216\u591a\u4e2achange set\u5f15\u8d77\u7684\u3002\u901a\u8fc7\u5185\u90e8RANK\u548c\u5916\u90e8\u7b5b\u9009\u5668B.RNK=1 and B.FLAG &lt;&gt; &#8216;D&#8217;\uff0c\u6211\u4eec\u53ef\u4ee5\u786e\u4fdd\u4ee5\u4e0b\u5185\u5bb9\u3002<\/p>\n<ol>\n<li style=\"list-style-type: none;\">\n<ol>\u53ea\u9009\u62e9\u7279\u5b9a\u8bb0\u5f55\u96c6\u4e2d\u7684\u6700\u65b0\u66f4\u6539\u3002<\/ol>\n<\/li>\n<\/ol>\n<p>&nbsp;<\/p>\n<ol>\u5982\u679c\u6700\u65b0\u66f4\u6539\u4e3a&#8217;D&#8217;\uff0c\u5219\u4e0d\u5c06\u6574\u4e2a\u8bb0\u5f55\u96c6\u6dfb\u52a0\u5230\u6700\u7ec8\u9636\u6bb5\uff0c\u4ee5\u5b9e\u73b0\u5220\u9664\u8bb0\u5f55\u3002<\/ol>\n<p>\u63a5\u4e0b\u6765\uff0c\u6211\u4eec\u5c06\u6ce8\u610f\u5230WHERE CREATE_DATE_YEAR IN (\u2026\uff09\u5b50\u53e5\u3002\u901a\u8fc7\u4e0e\u5916\u90e8\u67e5\u8be2\u7684PARTITION\uff08CREATE_DATE_YEAR\uff09\u914d\u5408\u4f7f\u7528\uff0cDatabricks Delta Lake\u5c06\u786e\u4fdd\u4ec5\u8986\u76d6\u7279\u5b9a\u7684\u5206\u533a\uff0c\u59822018\u5e74\u30012016\u5e74\u548c2015\u5e74\uff0c\u5e76\u4e14\u4e0d\u4f1a\u89e6\u53ca\u5176\u4ed6\u5206\u533a\u3002\u4e0a\u8ff0\u793a\u4f8b\u4e2d\u5c06\u5206\u533a\u786c\u7f16\u7801\u4ee5\u4fbf\u66f4\u6613\u7406\u89e3\uff0c\u4f46\u5728\u5b9e\u9645\u5b9e\u65bd\u4e2d\uff0c\u8fd9\u4e9b\u5206\u533a\u5c06\u4f5c\u4e3aScala\u5217\u8868\u63d0\u4f9b\uff0c\u5e76\u5c06\u52a8\u6001\u751f\u6210\u67e5\u8be2\u4e0e\u53d8\u66f4\u96c6\u76f8\u5173\u7684\u5185\u5bb9\uff0c\u5982\u4e0b\u6240\u793a\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"k\">val<\/span> <span class=\"nv\">partitionsToOverwrite<\/span> <span class=\"k\">=<\/span> <span class=\"nv\">spark<\/span><span class=\"o\">.<\/span><span class=\"py\">sql<\/span><span class=\"o\">(<\/span><span class=\"s\">\"select year(to_date(create_date, \"<\/span><span class=\"nc\">MM<\/span><span class=\"o\">\/<\/span><span class=\"n\">dd<\/span><span class=\"o\">\/<\/span><span class=\"n\">yyyy<\/span><span class=\"s\">\")) from incremental\"<\/span><span class=\"o\">)<\/span>\r\n<span class=\"o\">...<\/span>\r\n<span class=\"nv\">spark<\/span><span class=\"o\">.<\/span><span class=\"py\">sql<\/span><span class=\"o\">(<\/span><span class=\"n\">s<\/span><span class=\"s\">\"\"\"\r\nINSERT OVERWRITE T_FINAL\r\n...\r\nWHERE CREATE_DATE_YEAR IN ( ${partitionsToOverwrite.mkString(\",\") )\r\n...\r\n\"\"\"<\/span><span class=\"o\">)<\/span>\r\n<\/code><\/pre>\n<h3>\u6027\u80fd\u53ef\u4ee5\u88ab\u7406\u89e3\u4e3a\u4e00\u4e2a\u4ea7\u54c1\u6216\u7cfb\u7edf\u5728\u7279\u5b9a\u6761\u4ef6\u4e0b\u7684\u8868\u73b0\u3002<\/h3>\n<p>\u5982\u524d\u6240\u8ff0\uff0cDatabricks Delta Lake\u901a\u8fc7CDC\u7ba1\u9053\u4ee5\u5e76\u884c\u7684\u65b9\u5f0f\u6267\u884c\uff0c\u4f7f\u7528\u6237\u80fd\u591f\u5728\u67e5\u8be2\u4e2d\u5bf9\u6570\u636e\u8fdb\u884c\u4e00\u81f4\u6027\u7684\u89c6\u56fe\u3002\u5728\u8fd9\u91cc\uff0c\u6211\u4eec\u5c06\u4ecb\u7ecd\u7528\u4e8e\u4f18\u5316\u8bfb\u53d6\u5668\u548c\u5199\u5165\u5668\u7684\u4e24\u4e2a\u529f\u80fd\u3002<\/p>\n<p>\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3\u30d7\u30eb\u30fc\u30cb\u30f3\u30b0: \u4e0a\u306e2\u3064\u76ee\u306einsert(\u30e9\u30a4\u30bf\u30fc)\u306b\u304a\u3044\u3066\u306f\u3001Databricks Delta Lake\u306f\u3001\u66f4\u65b0\u3059\u308b\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3\u306e\u307f\u3092\u8aad\u307f\u53d6\u308a\u3001\u518d\u66f8\u304d\u8fbc\u307f\u3059\u308b\u305f\u3081\u306b\u3001PARTITION\u306e\u6307\u5b9a\u3068WHERE\u53e5\u306eIN\u30ea\u30b9\u30c8\u3092\u53c2\u7167\u3057\u307e\u3059\u3002\u5b9f\u969b\u3001\u3053\u308c\u306b\u3088\u3063\u3066\u64cd\u4f5c\u3055\u308c\u308b\u30c6\u30fc\u30d6\u30eb\u306e\u5272\u5408\u306f\u534a\u5206\u3001\u901a\u5e38\u306f\u305d\u308c\u4ee5\u4e0b\u3068\u306a\u308a\u30012\u3064\u76ee\u306einsert\u306b\u304a\u3051\u308bT_FINAL\u306e\u66f4\u65b0\u304a\u3088\u3073T_STAGING\u306eSELECT\u306e\u5c40\u6240\u5316\u3092\u6539\u5584\u3057\u307e\u3059\u3002<\/p>\n<p>\u30c7\u30fc\u30bf\u30b9\u30ad\u30c3\u30d4\u30f3\u30b0\/ZORDER\u30a4\u30f3\u30c7\u30c3\u30af\u30b9: T_FINAL\u306b\u30af\u30a8\u30ea\u30fc\u3092\u884c\u3046\u30e6\u30fc\u30b6\u30fc\u306f\u3001BI\u30c4\u30fc\u30eb\u304b\u3089\u30a2\u30c9\u30db\u30c3\u30af\u306eSQL\u30af\u30a8\u30ea\u30fc\u3068\u591a\u5c90\u306b\u6e21\u308a\u307e\u3059\u3002\u3053\u3053\u3067\u306f\u3001WHERE\u53e5\u306b\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3\u30ab\u30e9\u30e0CREATE_DATE_YEAR\u304c\u542b\u307e\u308c\u308b\u5834\u5408\u3082\u3042\u308c\u3070\u542b\u307e\u308c\u306a\u3044\u5834\u5408\u3082\u3042\u308a\u307e\u3059\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"o\">%<\/span><span class=\"k\">sql<\/span>\r\n<span class=\"k\">SELECT<\/span> <span class=\"err\">\u2026<\/span>\r\n<span class=\"k\">FROM<\/span> <span class=\"n\">T_FINAL<\/span>\r\n<span class=\"k\">WHERE<\/span> <span class=\"n\">COL1<\/span> <span class=\"o\">=<\/span> <span class=\"n\">val<\/span> <span class=\"k\">and<\/span> <span class=\"n\">COL2<\/span> <span class=\"o\">=<\/span> <span class=\"n\">val<\/span>\r\n<\/code><\/pre>\n<p>\u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\uff0cCOL1\u548cCOL2\u90fd\u4e0d\u5305\u542b\u5728\u5206\u533a\u5217\u4e2d\u3002\u4f46\u662f\uff0c\u7528\u6237\u53ef\u4ee5\u521b\u5efa\u9488\u5bf9\u8fd9\u4e24\u4e2a\u5217\u7684Z-order\u7d22\u5f15\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"n\">OPTIMIZE<\/span> <span class=\"n\">T_FINAL<\/span> <span class=\"n\">ZORDER<\/span> <span class=\"k\">BY<\/span> <span class=\"p\">(<\/span><span class=\"n\">COL1<\/span><span class=\"p\">,<\/span> <span class=\"n\">COL2<\/span><span class=\"p\">)<\/span>\r\n<\/code><\/pre>\n<p>\u5728\u5185\u90e8\uff0cDatabricks Delta Lake\u4f1a\u6839\u636eZ\u503c\u5c06Parquet\u6587\u4ef6\u8fdb\u884c\u805a\u7c7b\uff0c\u4ee5\u53ea\u8bbf\u95ee\u5305\u542bCOL1 = val\u548cCOL2 = val\u7684\u6587\u4ef6\uff0c\u5728\u4e0a\u8ff0\u67e5\u8be2\u4e2d\u3002<\/p>\n<p>\u5bf9\u4e8e\u53ef\u4ee5\u5229\u7528Z-order\u7d22\u5f15\u6765\u6269\u5c55\u4e00\u7cfb\u5217\u67e5\u8be2\u7684\u60c5\u51b5\uff0c\u6211\u4f1a\u63d0\u4f9b\u4e24\u70b9\u8bf4\u660e\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u4e0a\u8a18\u306e\u30b1\u30fc\u30b9\u306b\u304a\u3044\u3066\u306f\u3001COL1\u306e\u307f\uff08\u3042\u308b\u3044\u306fCOL2\u306e\u307f)\u306b\u30d5\u30a3\u30eb\u30bf\u30ea\u30f3\u30b0\u3092\u884c\u3046\u30af\u30a8\u30ea\u30fc\u3082\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u306e\u6069\u6075\u3092\u53d7\u3051\u308b\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002\u3053\u308c\u306f\u3001RDBMS\u306e\u8907\u5408\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u3068\u7570\u306a\u308a\u3001Z-order\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u306b\u304a\u3044\u3066\u306f\u3001\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u3055\u308c\u305f\u30ab\u30e9\u30e0\u30ea\u30b9\u30c8\u306e\u30d7\u30ec\u30d5\u30a3\u30af\u30b9\u306b\u57fa\u3065\u304f\u30d5\u30a3\u30eb\u30bf\u30fc\u306b\u5bfe\u3059\u308b\u30d0\u30a4\u30a2\u30b9\u304c\u7121\u3044\u305f\u3081\u3067\u3059\u3002<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\u4e0a\u306e\u30b1\u30fc\u30b9\u3068\u7570\u306a\u308a\u3001\u30af\u30a8\u30ea\u30fc\u304c\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3\u30ab\u30e9\u30e0\u306b\u5bfe\u3059\u308b\u30d5\u30a3\u30eb\u30bf\u30ea\u30f3\u30b0\u3092\u884c\u3046\u5834\u5408\u306b\u306f\u3001\u30af\u30a8\u30ea\u30fc\u5b9f\u884c\u306e\u969b\u306b\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3\u30d7\u30eb\u30fc\u30cb\u30f3\u30b0\u3068Z-order\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u306e\u4e21\u65b9\u304c\u5287\u7684\u306b\u51e6\u7406\u5bfe\u8c61\u306e\u30d5\u30a1\u30a4\u30eb\u306e\u6570\u3092\u524a\u6e1b\u3057\u307e\u3059\u3002<\/ul>\n<p>\u8bf7\u53c2\u8003\u8fd9\u7bc7\u4f18\u79c0\u7684\u6587\u7ae0\uff0c\u4e86\u89e3\u6570\u636e\u8df3\u8dc3\u548cZ-order\u7d22\u5f15\u4e0e\u5206\u533a\u4fee\u526a\u7684\u914d\u5408\u5de5\u4f5c\u65b9\u5f0f\u3002<\/p>\n<h3>\u540c\u65f6\u6709\u6548<\/h3>\n<p>\u6b63\u5982\u4e4b\u524d\u7684\u6587\u7ae0\u6240\u63d0\u5230\u7684\uff0cDatabricks Delta Lake\u5728\u4e91\u5b58\u50a8\u4e2d\u5f15\u5165\u4e86\u4e8b\u52a1\u529f\u80fd\u3002\u60a8\u53ef\u4ee5\u5982\u4e0b\u5229\u7528\u6b64\u529f\u80fd\uff1a\u5728\u8986\u76d6\u5206\u533a\u7684\u540c\u65f6\uff0cDatabricks Delta Lake\u4f1a\u521b\u5efa\u65b0\u7684Parquet\u6587\u4ef6\uff0c\u5e76\u5c06\u65e7\u7684Parquet\u6587\u4ef6\u4fdd\u7559\u7ed9\u6b63\u5728\u67e5\u8be2\u6570\u636e\u7684\u7528\u6237\u3002\u5728\u8986\u76d6\u64cd\u4f5c\u5b8c\u6210\u540e\u542f\u52a8\u7684\u67e5\u8be2\u5c06\u5f15\u7528\u65b0\u6570\u636e\u3002Delta\u5229\u7528\u4e8b\u52a1\u65e5\u5fd7\u4f7f\u67e5\u8be2\u5f15\u7528\u4e00\u81f4\u7684\u6570\u636e\u7248\u672c\u3002<\/p>\n<h3>\u7d27\u7f29\u548c\u6e05\u6d01<\/h3>\n<p>\u968f\u7740\u65f6\u95f4\u7684\u6d41\u901d\uff0c\u65e2\u8001\u53c8\u4e0d\u518d\u4f7f\u7528\u7684\u8bb0\u5f55\u4f1a\u5728T_STAGING\u548cT_FINAL\u4e2d\u79ef\u7d2f\u3002\u4f8b\u5982\uff0cT_STAGING\u4e2dRANK &gt; 1\u7684\u8bb0\u5f55\u6216\u88ab\u6807\u8bb0\u4e3a\u65e7\u7684T_FINAL\u8bb0\u5f55\u901a\u8fc7\u6587\u4ef6\u8986\u76d6\u800c\u53d8\u65e7\u3002\u8fd9\u4e9b\u8bb0\u5f55\u4e0d\u4f1a\u5f71\u54cd\u67e5\u8be2\u7684\u51c6\u786e\u6027\uff0c\u4f46\u968f\u7740\u65f6\u95f4\u7684\u63a8\u79fb\u4f1a\u4f7fCDC\u548c\u67e5\u8be2\u6027\u80fd\u4e0b\u964d\u3002\u5e78\u8fd0\u7684\u662f\uff0c\u5728Databricks Delta Lake\u4e2d\uff0c\u8fd9\u79cd\u7ef4\u62a4\u4efb\u52a1\u53d8\u5f97\u7b80\u5355\u3002\u5220\u9664T_FINAL\u4e2d\u65e7\u6587\u4ef6\u7684\u64cd\u4f5c\u5982\u4e0b\u6240\u793a\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"o\">%<\/span><span class=\"k\">sql<\/span>\r\n<span class=\"k\">VACUUM<\/span> <span class=\"n\">T_FINAL<\/span>\r\n<\/code><\/pre>\n<p>\u5982\u679c\u672a\u6307\u5b9a\u4fdd\u7559\u671f\u53c2\u6570\uff08\u8bf7\u53c2\u9605VACUUM\u6587\u6863\uff1aAzure|AWS\uff09\uff0c\u4e8b\u52a1\u65e5\u5fd7\u4e2d\u4e0d\u4f1a\u4fdd\u7559\u4efb\u4f55\u8d85\u8fc77\u5929\u7684\u6587\u4ef6\u3002\u8fd9\u4e2a\u65f6\u95f4\u8db3\u591f\u957f\uff0c\u8db3\u591f\u8ba4\u4e3a\u4e0d\u5b58\u5728\u4efb\u4f55\u8bbf\u95ee\u8fd9\u4e9b\u6587\u4ef6\u7684\u8bfb\u53d6\u5668\u3002<\/p>\n<p>\u5728T_STAGING\u4e2d\uff0c\u5bf9\u4e8e\u6392\u540d\u5927\u4e8e1\u7684\u6240\u6709\u8bb0\u5f55\uff0c\u5c06\u5176\u5168\u90e8\u5220\u9664\u3002\u4e00\u4e2a\u7b80\u5355\u7684\u65b9\u6cd5\u662f\u5c06T_FINAL\u590d\u5236\u5230T_STAGING\u4e2d\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"o\">%<\/span><span class=\"k\">sql<\/span>\r\n<span class=\"k\">INSERT<\/span> <span class=\"n\">OVERWRITE<\/span> <span class=\"n\">T_STAGING<\/span> <span class=\"k\">SELECT<\/span> <span class=\"o\">*<\/span> <span class=\"k\">FROM<\/span> <span class=\"n\">T_FINAL<\/span>\r\n<\/code><\/pre>\n<p>\u4f5c\u4e3a\u7ef4\u62a4\u4efb\u52a1\uff0c\u60a8\u53ef\u4ee5\u5c06\u4e0a\u8ff0\u4e24\u4e2a\u547d\u4ee4\u4ee5\u53ca\u4e0a\u8ff0\u8bb2\u89e3\u7684OPTIMIZE\u547d\u4ee4\u6574\u7406\u5230\u7b14\u8bb0\u672c\u4e2d\uff0c\u5e76\u5c06\u5176\u5b89\u6392\u4e3aDatabricks\u4f5c\u4e1a\u8fdb\u884c\u5b9a\u65f6\u6267\u884c\u3002<\/p>\n<blockquote><p>\u4e0d\u4e45\u5c06\u53d1\u5e03\uff082018\/10\u6587\u7ae0\uff09\u7684Databricks Runtime 5.0\u4e2d\uff0c\u53ef\u4ee5\u901a\u8fc7\u4f7f\u7528MERGE INTO\u6765\u6539\u5584Databricks Delta Lake\u7684\u6027\u80fd\uff0c\u5e76\u652f\u6301\u8bb0\u5f55\u7684\u5220\u9664(D)\u64cd\u4f5c\u3002<\/p><\/blockquote>\n<h1>\u7ba1\u9053\u7684\u5168\u9762\u8fd0\u8425<\/h1>\n<p>\u4f5c\u4e3a\u4e00\u4e2a\u5e73\u53f0\uff0cDatabricks\u4e0d\u4ec5\u53ef\u4ee5\u6784\u5efaETL\u7ba1\u9053\uff0c\u8fd8\u53ef\u4ee5\u7f29\u77ed\u5c06\u8fd9\u4e9b\u7ba1\u9053\u6295\u5165\u5b9e\u9645\u8fd0\u8425\u6240\u9700\u7684\u65f6\u95f4\u3002\u5728\u8fd9\u91cc\uff0c\u6211\u4eec\u5c06\u4ecb\u7ecd\u4e24\u4e2a\u53ef\u4ee5\u5728\u5b9e\u9645\u8fd0\u8425CDC\u7ba1\u9053\u65f6\u5229\u7528\u7684Apache Spark\u529f\u80fd\u548c\u6709\u7528\u6280\u672f\u3002<\/p>\n<h2>\u8bbe\u5b9a\u9a71\u52a8\u7f16\u7a0b<\/h2>\n<p>\u5728\u6784\u5efa\u5927\u89c4\u6a21\u5e94\u7528\u7a0b\u5e8f\u65f6\uff0c\u901a\u5e38\u91c7\u7528\u7684\u8bbe\u8ba1\u6a21\u5f0f\u662f\u901a\u8fc7\u914d\u7f6e\u6765\u63a7\u5236\u8f6f\u4ef6\u7684\u884c\u4e3a\uff08\u4f8b\u5982\uff1aYAML\u6216JSON\u914d\u7f6e\u6587\u4ef6\uff09\u3002\u901a\u8fc7Spark\u7684SQL\u3001Scala\u3001Python\u7b49\u901a\u7528\u7f16\u7a0b\u8bed\u8a00\u7684\u652f\u6301\uff0c\u53ef\u4ee5\u5c06\u914d\u7f6e\u4fdd\u5b58\u5728\u8868\u4e2d\uff0c\u5e76\u5b9e\u73b0\u57fa\u4e8e\u914d\u7f6e\u7684\u52a8\u6001SQL\u751f\u6210\uff0c\u56e0\u6b64\u53ef\u4ee5\u8bf4\u8fd9\u79cd\u8bbe\u8ba1\u6a21\u5f0f\u975e\u5e38\u9002\u7528\u4e8e\u6b64\u3002\u8ba9\u6211\u4eec\u6765\u770b\u770b\u8fd9\u5728CDC\u4e0a\u4e0b\u6587\u4e2d\u662f\u5982\u4f55\u5de5\u4f5c\u7684\u3002<\/p>\n<p>\u9996\u5148\uff0c\u8bf7\u8bb0\u4f4f\u6211\u4eec\u7684CDC\u7ba1\u9053\u4e2d\u670965\u4e2a\u8868\u3002\u6bcf\u4e2a65\u4e2a\u8868\u90fd\u662f\u4e00\u884c\uff0c\u5217\u4e2d\u5305\u542b\u4e86\u6784\u5efaCDC SQL\u8bed\u53e5\u6240\u9700\u7684\u4fe1\u606f\u7684CONFIG\u8868\u3002<\/p>\n<div>\n<div class=\"post-table\">TABLEPARTITION_COLUMN_EXPRESSIONPARTITION_COLUMN_ALIASRANK_EXPRESSIONIS_INSERT_ONLYT1year(to_date(create_date, \u201cMM\/dd\/yyyy\u201d))create_date_yearPARTITION BY ID ORDER BY CDC_TIMESTAMP DESCNT2year(to_date(transaction_date, \u201cMM\/dd\/yyyy\u201d))transaction_date_yearPARTITION BY ID1, ID2 ORDER BY CDC_TIMESTAMP DESCNT3nullnullnullYT4&#8230;&#8230;&#8230;&#8230;<\/div>\n<\/div>\n<p>\u88682 &#8211; \u7528\u4e8e\u6267\u884c\u4e00\u7cfb\u5217\u8868\u7684CDC\u7ba1\u9053\u7684\u8bbe\u7f6e\u8868\u683c<\/p>\n<p>\u4e3a\u4e86\u83b7\u53d6\u7279\u5b9a\u8868\u7684\u914d\u7f6e\u4fe1\u606f\uff0c\u5e76\u4e3a\u8be5\u8868\u5e94\u7528CDC\u903b\u8f91\uff0c\u53ef\u4ee5\u4f7f\u7528\u4ee5\u4e0b\u4ee3\u7801\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"k\">val<\/span> <span class=\"nv\">hiveDb<\/span> <span class=\"k\">=<\/span> <span class=\"err\">\u201c<\/span><span class=\"n\">mydb<\/span><span class=\"err\">\u201d<\/span>\r\n<span class=\"k\">val<\/span> <span class=\"nv\">CONFIG_TABLE<\/span> <span class=\"k\">=<\/span> <span class=\"err\">\u201c<\/span><span class=\"nc\">CONFIG<\/span><span class=\"err\">\u201d<\/span>\r\n\r\n<span class=\"c1\">\/\/ Table is a notebook input widget<\/span>\r\n<span class=\"k\">val<\/span> <span class=\"nv\">table<\/span><span class=\"k\">=<\/span><span class=\"n\">s<\/span><span class=\"s\">\"\"\"${dbutils.widgets.get(\"wTable\")}\"\"\"<\/span>\r\n<span class=\"nf\">val<\/span> <span class=\"o\">(<\/span><span class=\"n\">partitionColumnExpression<\/span><span class=\"o\">,<\/span> <span class=\"n\">partitionColumnAlias<\/span><span class=\"o\">,<\/span> <span class=\"n\">rankExpression<\/span><span class=\"o\">,<\/span> <span class=\"n\">isInsertOnly<\/span><span class=\"o\">)<\/span> <span class=\"k\">=<\/span> <span class=\"nv\">spark<\/span><span class=\"o\">.<\/span><span class=\"py\">sql<\/span><span class=\"o\">(<\/span><span class=\"n\">s<\/span><span class=\"s\">\"\"\"\r\nSELECT PARTITION_COLUMN_EXPRESSION, PARTITION_COLUMN_ALIAS, RANK_EXPRESSION, IS_INSERT_ONLY\r\n  FROM ${hiveDb}.${CONFIG_TABLE}\r\n WHERE TABLE_NAME=LOWER('$table')\r\n\"\"\"<\/span><span class=\"o\">).<\/span><span class=\"py\">as<\/span><span class=\"o\">[(<\/span><span class=\"kt\">String<\/span>, <span class=\"kt\">String<\/span>, <span class=\"kt\">String<\/span>, <span class=\"kt\">Boolean<\/span><span class=\"o\">)].<\/span><span class=\"py\">head<\/span>\r\n\r\n<span class=\"o\">...<\/span>\r\n\r\n<span class=\"cm\">\/*\r\n* Insert 1 above would look like following. Here, the table\r\n* variable is set to T1 or T2 from the config table\r\n*\/<\/span>\r\n<span class=\"nv\">spark<\/span><span class=\"o\">.<\/span><span class=\"py\">sql<\/span><span class=\"o\">(<\/span><span class=\"n\">s<\/span><span class=\"s\">\"\"\"\r\nINSERT INTO ${table}_STAGING\r\nPARTITION(${partitionColumnAlias)\r\nSELECT ${projectListFromIncremental}\r\n  FROM INCREMENTAL\r\n\"\"\"<\/span><span class=\"o\">)<\/span>\r\n\r\n<span class=\"o\">...<\/span>\r\n\r\n<span class=\"c1\">\/\/ Insert 2 could look like<\/span>\r\n<span class=\"k\">val<\/span> <span class=\"nv\">partitionsToOverwrite<\/span> <span class=\"k\">=<\/span> <span class=\"nv\">spark<\/span><span class=\"o\">.<\/span><span class=\"py\">sql<\/span><span class=\"o\">(<\/span><span class=\"n\">s<\/span><span class=\"s\">\"\"\"SELECT DISTINCT ${partitionColumnExpression} FROM INCREMENTAL\"\"\"<\/span><span class=\"o\">).<\/span><span class=\"py\">as<\/span><span class=\"o\">[<\/span><span class=\"kt\">String<\/span><span class=\"o\">].<\/span><span class=\"py\">collect<\/span>\r\n\r\n<span class=\"nv\">spark<\/span><span class=\"o\">.<\/span><span class=\"py\">sql<\/span><span class=\"o\">(<\/span><span class=\"n\">s<\/span><span class=\"s\">\"\"\"\r\nINSERT OVERWRITE TABLE ${table}_FINAL\r\nPARTITION(${partitionColumnAlias})\r\nSELECT ${projectListFromIncremental}\r\n  FROM (SELECT A.*, RANK() OVER (${rankExpression}) AS RNK\r\n          FROM ${table}_STAGING A.*\r\n         WHERE ${partitionColumnAlias} IN (${partitionsToOverwrite.mkString(\u201c,\u201d) )\r\n  ) B\r\nWHERE B.RNK = 1 AND B.FLAG &lt; &gt; \u2018D\u2019\r\n\"\"\"<\/span><span class=\"o\">)<\/span>\r\n<\/code><\/pre>\n<h2>\u7b14\u8bb0\u672c\u5de5\u4f5c\u6d41\u7a0b\u548c\u4efb\u52a1<\/h2>\n<p>\u5047\u8bbe\u4e0a\u8ff0\u5904\u7406\u5df2\u7ecf\u88ab\u5b9e\u73b0\u5728\u4e00\u4e2a\u540d\u4e3aProcessIncremental\u7684\u7b14\u8bb0\u672c\u4e2d\u3002\u5728\u8fd9\u91cc\uff0c\u6211\u4eec\u5c06\u4f7f\u7528\u7b14\u8bb0\u672c\u5de5\u4f5c\u6d41\u6765\u4e3a\u6bcf\u4e2a65\u4e2a\u8868\u6267\u884c\u5904\u7406\uff0c\u5e76\u68c0\u6d4b\u51fa\u503c\u5f97\u6ce8\u610f\u7684\u53d8\u66f4\u96c6\uff0c\u5e76\u521b\u5efa\u4e00\u4e2a\u63a7\u5236\u5668\u7b14\u8bb0\u672c\u6765\u5bf9\u5b83\u4eec\u6267\u884cProcessIncremental\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"k\">val<\/span> <span class=\"nv\">startDate<\/span> <span class=\"k\">=<\/span> <span class=\"err\">\u201c<\/span><span class=\"mi\">20180101<\/span><span class=\"err\">\u201d<\/span>\r\n<span class=\"k\">val<\/span> <span class=\"nv\">tables<\/span> <span class=\"k\">=<\/span> <span class=\"nv\">spark<\/span><span class=\"o\">.<\/span><span class=\"py\">sql<\/span><span class=\"o\">(<\/span><span class=\"n\">s<\/span><span class=\"s\">\"\"\"\r\nSELECT TABLE_NAME \r\n  FROM $hiveDb.$CONFIG_TABLE\r\n\"\"\"<\/span><span class=\"o\">).<\/span><span class=\"py\">as<\/span><span class=\"o\">[<\/span><span class=\"kt\">String<\/span><span class=\"o\">].<\/span><span class=\"py\">collect<\/span><span class=\"o\">.<\/span><span class=\"py\">map<\/span><span class=\"o\">(<\/span><span class=\"nv\">_<\/span><span class=\"o\">.<\/span><span class=\"py\">toLowerCase<\/span><span class=\"o\">)<\/span>\r\n\r\n<span class=\"nv\">tables<\/span><span class=\"o\">.<\/span><span class=\"py\">foreach<\/span> <span class=\"o\">{<\/span> <span class=\"n\">tbl<\/span> <span class=\"k\">=&gt;<\/span>   \r\n  <span class=\"k\">val<\/span> <span class=\"nv\">processTheseChangeSets<\/span> <span class=\"k\">=<\/span> <span class=\"nv\">dbutils<\/span><span class=\"o\">.<\/span><span class=\"py\">notebook<\/span><span class=\"o\">.<\/span><span class=\"py\">run<\/span><span class=\"o\">(<\/span><span class=\"s\">\"GetNextChangeSets\"<\/span><span class=\"o\">,<\/span> <span class=\"mi\">0<\/span><span class=\"o\">,<\/span> <span class=\"nc\">Map<\/span><span class=\"o\">(<\/span>\r\n          <span class=\"s\">\"wHiveDb\"<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">hiveDb<\/span><span class=\"o\">,<\/span>\r\n          <span class=\"s\">\"wTable\"<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">tbl<\/span><span class=\"o\">,<\/span>\r\n          <span class=\"s\">\"wStartDate\"<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">startDate<\/span>\r\n      <span class=\"o\">)<\/span>\r\n  <span class=\"o\">)<\/span>\r\n\r\n  <span class=\"nf\">if<\/span><span class=\"o\">(!<\/span><span class=\"nv\">processTheseChangeSets<\/span><span class=\"o\">.<\/span><span class=\"py\">isEmpty<\/span><span class=\"o\">)<\/span> <span class=\"o\">{<\/span>\r\n    <span class=\"k\">val<\/span> <span class=\"nv\">stats<\/span> <span class=\"k\">=<\/span> <span class=\"nv\">dbutils<\/span><span class=\"o\">.<\/span><span class=\"py\">notebook<\/span><span class=\"o\">.<\/span><span class=\"py\">run<\/span><span class=\"o\">(<\/span><span class=\"s\">\"ProcessIncremental\"<\/span><span class=\"o\">,<\/span> <span class=\"mi\">0<\/span><span class=\"o\">,<\/span> <span class=\"nc\">Map<\/span><span class=\"o\">(<\/span>\r\n        <span class=\"s\">\"wHiveDb\"<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">hiveDb<\/span><span class=\"o\">,<\/span>\r\n        <span class=\"s\">\"wIncrFiles\"<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">processTheseChangeSets<\/span><span class=\"o\">,<\/span>\r\n        <span class=\"s\">\"wTable\"<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">tbl<\/span><span class=\"o\">)<\/span>\r\n      <span class=\"o\">)<\/span>\r\n  <span class=\"o\">)<\/span>\r\n<span class=\"o\">}<\/span>\r\n<\/code><\/pre>\n<p>\u53ea\u9700\u8981\u4e00\u4e2a\u9009\u9879\uff0c\u4ee5\u4e0b\u662f\u5bf9\u539f\u6587\u7684\u6c49\u8bed\u672c\u5730\u5316\u6539\u5199\uff1a<\/p>\n<p>\u60a8\u53ef\u4ee5\u8f7b\u677e\u5730\u5c06\u63a7\u5236\u5668\u7b14\u8bb0\u672c\u4f5c\u4e3aDatabricks\u4f5c\u4e1a\u8fdb\u884c\u8c03\u5ea6\uff0c\u4ee5\u4fbf\u6309\u7167\u60a8\u559c\u6b22\u7684\u9891\u7387\u8fd0\u884cCDC\u7ba1\u9053\u3002\u6700\u540e\uff0c\u5c3d\u7ba1\u5faa\u73af\u662f\u4e32\u884c\u7684\uff0c\u4f46\u60a8\u53ef\u4ee5\u4f7f\u7528par\u547d\u4ee4\u6216Scala\u7684\u529f\u80fd\u5c06\u4e32\u884c\u96c6\u5408\u8f6c\u6362\u4e3a\u5e76\u884c\u96c6\u5408\uff0c\u4ece\u800c\u8f7b\u677e\u5730\u5c06\u5176\u66f4\u6539\u4e3a\u5e76\u884c\u5faa\u73af\u3002<\/p>\n<h1>\u603b\u7ed3<\/h1>\n<p>\u5728\u8fd9\u7bc7\u6587\u7ae0\u4e2d\uff0c\u6211\u4eec\u89e3\u91ca\u4e86\u5c06CDC\u5de5\u5177\uff08\u5982Oracle GoldenGate\u3001Informatica PowerExchange\uff09\u548c\u4f9b\u5e94\u5546\uff08\u5982Oracle Change Data Capture\uff09\u7ba1\u7406\u7684\u53d8\u66f4\u8868\u548c\u7531\u7528\u6237\u7684insert\/update\/delete\u89e6\u53d1\u5668\u7ba1\u7406\u7684\u53d8\u66f4\u8868\u4e0eDatabricks Delta Lake\u5408\u5e76\u65f6\u7684\u53c2\u8003\u67b6\u6784\u3002\u4e3a\u4e86\u4f18\u5316\u7528\u6237\u7684\u8bfb\u53d6\u4f53\u9a8c\uff0c\u6211\u4eec\u8ba8\u8bba\u4e86\u5728\u5c06\u8fd9\u4e9b\u53d8\u66f4\u53cd\u6620\u5230Databricks Delta Lake\u65f6\u4f7f\u7528\u7684Spark SQL\u7684\u6df1\u6316\u3001\u4ee5\u53ca\u4e24\u4e2a\u6027\u80fd\u65b9\u9762\u7684\u8003\u8651\uff08\u5206\u533a\u548cZ-order\u7d22\u5f15\uff09\u3001\u4ee5\u53ca\u4f34\u968f\u7684\u538b\u7f29\u548c\u6e05\u7406\u7b49\u8f85\u52a9\u90e8\u5206\u7684\u8003\u8651\u3002\u6b64\u5916\uff0c\u901a\u8fc7\u914d\u7f6e\u9a71\u52a8\u7684ETL\u6d41\u6c34\u7ebf\u6784\u5efa\u548c\u7b14\u8bb0\u672c\u5de5\u4f5c\u6d41\u4ee5\u53ca\u4f5c\u4e1a\u9a71\u52a8\u7684\u5de5\u4f5c\u6d41\u7684\u5168\u9762\u8fd0\u7528\uff0c\u6211\u4eec\u770b\u5230\u4e86Databricks\u5982\u4f55\u52a0\u901f\u5de5\u4f5c\u6d41\u7684\u6784\u5efa\u548c\u8fd0\u8425\u3002<\/p>\n<h3>Databricks \u514d\u8d39\u8bd5\u7528<\/h3>\n<p>Databricks \u514d\u8d39\u8bd5\u7528<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u4f7f\u7528Databricks Delta\u7b80\u5316\u53d8\u66f4\u6570\u636e\u6355\u83b7 &#8211; \u8fd9\u662fDatabricks\u535a\u5ba2\u7684\u7ffb\u8bd1\u3002 \u5efa [&hellip;]<\/p>\n","protected":false},"author":9,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-47192","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.5 (Yoast SEO v21.5) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>\u5728Databricks Delta Lake\u4e2d\u7b80\u5316Change Data Capture - Blog - Silicon Cloud<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.silicloud.com\/zh\/blog\/\u5728databricks-delta-lake\u4e2d\u7b80\u5316change-data-capture\/\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"\u5728Databricks Delta Lake\u4e2d\u7b80\u5316Change Data Capture\" \/>\n<meta property=\"og:description\" content=\"\u4f7f\u7528Databricks Delta\u7b80\u5316\u53d8\u66f4\u6570\u636e\u6355\u83b7 &#8211; \u8fd9\u662fDatabricks\u535a\u5ba2\u7684\u7ffb\u8bd1\u3002 \u5efa [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.silicloud.com\/zh\/blog\/\u5728databricks-delta-lake\u4e2d\u7b80\u5316change-data-capture\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog - Silicon Cloud\" \/>\n<meta property=\"article:published_time\" content=\"2023-03-01T05:23:59+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-05-03T17:13:11+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7d6a913a08637a69fbf2\/4-0.png\" \/>\n<meta name=\"author\" content=\"\u6e05, \u626c\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"\u6e05, \u626c\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8databricks-delta-lake%e4%b8%ad%e7%ae%80%e5%8c%96change-data-capture\/\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8databricks-delta-lake%e4%b8%ad%e7%ae%80%e5%8c%96change-data-capture\/\",\"name\":\"\u5728Databricks Delta Lake\u4e2d\u7b80\u5316Change Data Capture - Blog - Silicon Cloud\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\"},\"datePublished\":\"2023-03-01T05:23:59+00:00\",\"dateModified\":\"2024-05-03T17:13:11+00:00\",\"author\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/cb5556d2501da73d864cac945e8d9461\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8databricks-delta-lake%e4%b8%ad%e7%ae%80%e5%8c%96change-data-capture\/#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8databricks-delta-lake%e4%b8%ad%e7%ae%80%e5%8c%96change-data-capture\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8databricks-delta-lake%e4%b8%ad%e7%ae%80%e5%8c%96change-data-capture\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.silicloud.com\/zh\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"\u5728Databricks Delta Lake\u4e2d\u7b80\u5316Change Data Capture\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/\",\"name\":\"Blog - Silicon Cloud\",\"description\":\"\",\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/cb5556d2501da73d864cac945e8d9461\",\"name\":\"\u6e05, \u626c\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/32a4239de8ff29adace466261d309424a1e5fe9f7e3036bf89fe03f2e3dbe717?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/32a4239de8ff29adace466261d309424a1e5fe9f7e3036bf89fe03f2e3dbe717?s=96&d=mm&r=g\",\"caption\":\"\u6e05, \u626c\"},\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/author\/qingyang\/\"},{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8databricks-delta-lake%e4%b8%ad%e7%ae%80%e5%8c%96change-data-capture\/#local-main-organization-logo\",\"url\":\"\",\"contentUrl\":\"\",\"caption\":\"Blog - Silicon Cloud\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"\u5728Databricks Delta Lake\u4e2d\u7b80\u5316Change Data Capture - Blog - Silicon Cloud","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.silicloud.com\/zh\/blog\/\u5728databricks-delta-lake\u4e2d\u7b80\u5316change-data-capture\/","og_locale":"zh_CN","og_type":"article","og_title":"\u5728Databricks Delta Lake\u4e2d\u7b80\u5316Change Data Capture","og_description":"\u4f7f\u7528Databricks Delta\u7b80\u5316\u53d8\u66f4\u6570\u636e\u6355\u83b7 &#8211; \u8fd9\u662fDatabricks\u535a\u5ba2\u7684\u7ffb\u8bd1\u3002 \u5efa [&hellip;]","og_url":"https:\/\/www.silicloud.com\/zh\/blog\/\u5728databricks-delta-lake\u4e2d\u7b80\u5316change-data-capture\/","og_site_name":"Blog - Silicon Cloud","article_published_time":"2023-03-01T05:23:59+00:00","article_modified_time":"2024-05-03T17:13:11+00:00","og_image":[{"url":"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7d6a913a08637a69fbf2\/4-0.png"}],"author":"\u6e05, \u626c","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"\u6e05, \u626c","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"4 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8databricks-delta-lake%e4%b8%ad%e7%ae%80%e5%8c%96change-data-capture\/","url":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8databricks-delta-lake%e4%b8%ad%e7%ae%80%e5%8c%96change-data-capture\/","name":"\u5728Databricks Delta Lake\u4e2d\u7b80\u5316Change Data Capture - Blog - Silicon Cloud","isPartOf":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website"},"datePublished":"2023-03-01T05:23:59+00:00","dateModified":"2024-05-03T17:13:11+00:00","author":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/cb5556d2501da73d864cac945e8d9461"},"breadcrumb":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8databricks-delta-lake%e4%b8%ad%e7%ae%80%e5%8c%96change-data-capture\/#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8databricks-delta-lake%e4%b8%ad%e7%ae%80%e5%8c%96change-data-capture\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8databricks-delta-lake%e4%b8%ad%e7%ae%80%e5%8c%96change-data-capture\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.silicloud.com\/zh\/blog\/"},{"@type":"ListItem","position":2,"name":"\u5728Databricks Delta Lake\u4e2d\u7b80\u5316Change Data Capture"}]},{"@type":"WebSite","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website","url":"https:\/\/www.silicloud.com\/zh\/blog\/","name":"Blog - Silicon Cloud","description":"","inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/cb5556d2501da73d864cac945e8d9461","name":"\u6e05, \u626c","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/32a4239de8ff29adace466261d309424a1e5fe9f7e3036bf89fe03f2e3dbe717?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/32a4239de8ff29adace466261d309424a1e5fe9f7e3036bf89fe03f2e3dbe717?s=96&d=mm&r=g","caption":"\u6e05, \u626c"},"url":"https:\/\/www.silicloud.com\/zh\/blog\/author\/qingyang\/"},{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8databricks-delta-lake%e4%b8%ad%e7%ae%80%e5%8c%96change-data-capture\/#local-main-organization-logo","url":"","contentUrl":"","caption":"Blog - Silicon Cloud"}]}},"_links":{"self":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/47192","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/comments?post=47192"}],"version-history":[{"count":2,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/47192\/revisions"}],"predecessor-version":[{"id":95220,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/47192\/revisions\/95220"}],"wp:attachment":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/media?parent=47192"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/categories?post=47192"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/tags?post=47192"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}