{"id":47195,"date":"2023-04-16T17:14:31","date_gmt":"2023-03-25T15:12:48","guid":{"rendered":"https:\/\/www.silicloud.com\/zh\/blog\/%e7%bb%93%e6%9e%84%e5%8c%96%e6%b5%81%e5%aa%92%e4%bd%93%ef%bc%9a%e5%9b%9e%e9%a1%be2021%e5%b9%b4\/"},"modified":"2024-04-29T23:48:40","modified_gmt":"2024-04-29T15:48:40","slug":"%e7%bb%93%e6%9e%84%e5%8c%96%e6%b5%81%e5%aa%92%e4%bd%93%ef%bc%9a%e5%9b%9e%e9%a1%be2021%e5%b9%b4","status":"publish","type":"post","link":"https:\/\/www.silicloud.com\/zh\/blog\/%e7%bb%93%e6%9e%84%e5%8c%96%e6%b5%81%e5%aa%92%e4%bd%93%ef%bc%9a%e5%9b%9e%e9%a1%be2021%e5%b9%b4\/","title":{"rendered":"\u7ed3\u6784\u5316\u6d41\u5a92\u4f53\uff1a\u56de\u987e2021\u5e74"},"content":{"rendered":"

2021\u5e74\u4e3aDatabricks\u548cApache Spark\u5f00\u53d1\u7684\u6240\u6709\u65b0\u7ed3\u6784\u5316\u6d41\u5904\u7406\u529f\u80fd\u6982\u8ff0\u3002<\/p>\n

\u8fd9\u672c\u4e66\u662f\u7ffb\u8bd1\u7684\uff0c\u4e0d\u80fd\u4fdd\u8bc1\u5185\u5bb9\u7684\u51c6\u786e\u6027\u3002\u6709\u5173\u51c6\u786e\u7684\u5185\u5bb9\uff0c\u8bf7\u53c2\u8003\u539f\u6587\u3002<\/div>\n

2022\u5e74\u5230\u6765\u4e4b\u9645\uff0c\u6211\u4eec\u5728\u8fd9\u91cc\u56de\u987e\u4e00\u4e0bDatabricks\u4e0eApache Spark\u2122\u5728\u6d41\u5f0f\u5904\u7406\u65b9\u9762\u6240\u53d6\u5f97\u7684\u4f1f\u5927\u6210\u5c31\uff01\u57282021\u5e74\uff0c\u5de5\u7a0b\u56e2\u961f\u548c\u5f00\u6e90\u8d21\u732e\u8005\u4ee5\u4e09\u4e2a\u76ee\u6807\u4e3a\u6307\u5bfc\uff0c\u53d6\u5f97\u4e86\u8bb8\u591a\u91cd\u5927\u6210\u5c31\u3002<\/p>\n

    \n
  1. \n
      \u6539\u5584\u4f4e\u5ef6\u8fdf\u548c\u5177\u6709\u72b6\u6001\u7684\u5904\u7406<\/ol>\n<\/li>\n<\/ol>\n

       <\/p>\n

        \n
      1. \n
          \u6539\u5584Databricks\u548cSpark\u7ed3\u6784\u5316\u6d41\u5de5\u4f5c\u8d1f\u8f7d\u7684\u53ef\u89c2\u5bdf\u6027<\/ol>\n<\/li>\n<\/ol>\n

           <\/p>\n

            \u6539\u5584\u8d44\u6e90\u914d\u7f6e\u548c\u53ef\u6269\u5c55\u6027<\/ol>\n

            \u6700\u7ec8\u7684\u76ee\u6807\u662f\u901a\u8fc7\u4f7f\u56e2\u961f\u80fd\u591f\u5728Databricks\u548cSpark\u4e0a\u6267\u884c\u6d41\u5f0f\u5de5\u4f5c\u8d1f\u8f7d\uff0c\u4ee5\u5728Databricks\u4e0a\u8fd0\u884c\u5173\u952e\u4efb\u52a1\u7ea7\u522b\u7684\u6d41\u5f0f\u5e94\u7528\u7a0b\u5e8f\uff0c\u5e76\u540c\u65f6\u4f18\u5316\u6210\u672c\u6548\u76ca\u548c\u8d44\u6e90\u5229\u7528\u91cf\uff0c\u4ece\u800c\u5b9e\u73b0\u8fd9\u4e9b\u76ee\u6807\u80cc\u540e\u7684\u52a8\u673a\u3002<\/p>\n

            \u76ee\u6807\uff031\uff1a\u63d0\u9ad8\u4f4e\u5ef6\u8fdf\u548c\u72b6\u6001\u5904\u7406\u80fd\u529b\u3002<\/h1>\n

            \u6709\u4e24\u4e2a\u5173\u952e\u529f\u80fd\u65e8\u5728\u51cf\u5c11\u6709\u72b6\u6001\u64cd\u4f5c\u7684\u5ef6\u8fdf\u5e76\u6539\u5584\u6709\u72b6\u6001API\u3002\u7b2c\u4e00\u4e2a\u529f\u80fd\u662f\u5bf9\u5927\u89c4\u6a21\u6709\u72b6\u6001\u64cd\u4f5c\u7684\u5f02\u6b65\u68c0\u67e5\u70b9\uff0c\u8fd9\u6539\u5584\u4e86\u4e4b\u524d\u540c\u6b65\u8bbe\u8ba1\u5bfc\u81f4\u8f83\u9ad8\u5ef6\u8fdf\u7684\u95ee\u9898\u3002<\/p>\n

            \u975e\u540c\u6b65\u6aa2\u67e5\u9ede<\/h2>\n
            \"\"<\/div>\n

            \u5bf9\u4efb\u610f\u4e00\u79cd\u6709\u72b6\u6001\u64cd\u4f5c\u5458\u7684\u6539\u8fdb<\/h2>\n

            \u6211\u5011\u5728\u4ee5\u524d\u7684\u6587\u7ae0\u4e2d\u4ecb\u7d39\u4e86\u4f7f\u7528[flat]MapGroupsWithState\u9032\u884c\u7d50\u69cb\u5316\u4e32\u6d41\u7684\u4efb\u610f\u72c0\u614b\u8655\u7406\u3002\u4f7f\u7528\u9019\u4e9b\u904b\u7b97\u5b50\uff0c\u60a8\u53ef\u4ee5\u7372\u5f97\u8de8\u8d8a\u805a\u5408\u6846\u67b6\u7684\u8907\u96dc\u72c0\u614b\u64cd\u4f5c\u7684\u9748\u6d3b\u6027\u3002\u4ee5\u4e0b\u662f\u6211\u5011\u5c0d\u9019\u4e9b\u904b\u7b97\u5b50\u9032\u884c\u7684\u6539\u9032\u3002<\/p>\n

              \n
            • \n
                \u521d\u671f\u72b6\u614b\u306e\u8a31\u53ef\u3001\u3053\u308c\u306b\u3088\u3063\u3066\u304a\u4f7f\u3044\u306e\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u30c7\u30fc\u30bf\u5168\u3066\u306e\u518d\u51e6\u7406\u3092\u56de\u907f\u3059\u308b\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002<\/ul>\n<\/li>\n<\/ul>\n

                 <\/p>\n

                  \u65b0\u305f\u306bTestGroupState\u30a4\u30f3\u30bf\u30d5\u30a7\u30fc\u30b9\u3092\u63d0\u4f9b\u3059\u308b\u3053\u3068\u3067\u3001\u30ed\u30b8\u30c3\u30af\u306e\u30c6\u30b9\u30c8\u304c\u5bb9\u6613\u306b\u306a\u308a\u3001\u30e6\u30fc\u30b6\u30fc\u306fGroupState\u306e\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u3092\u4f5c\u6210\u3059\u308b\u3053\u3068\u3067\u8a2d\u5b9a\u3055\u308c\u305f\u5185\u90e8\u306e\u5024\u306b\u30a2\u30af\u30bb\u30b9\u3067\u304d\u3001\u72b6\u614b\u3092\u5909\u5316\u3055\u305b\u308b\u95a2\u6570\u306e\u30e6\u30cb\u30c3\u30c8\u30c6\u30b9\u30c8\u304c\u30b7\u30f3\u30d7\u30eb\u306b\u306a\u308a\u307e\u3059\u3002<\/ul>\n

                  \u521d\u671f\u9636\u6bb5\u7684\u8bb8\u53ef<\/h3>\n

                  \u8ba9\u6211\u4eec\u4ece\u4ee5\u4e0b\u7684flatMapGroupswithState\u64cd\u4f5c\u7b26\u5f00\u59cb\u3002<\/p>\n

                  def<\/span> flatMapGroupsWithState<\/span>[<\/span>S:<\/span> Encoder<\/span>, U:<\/span> Encoder<\/span>](<\/span>\r\n    outputMode<\/span>:<\/span> OutputMode<\/span>,<\/span>\r\n    timeoutConf<\/span>:<\/span> GroupStateTimeout<\/span>,<\/span>\r\n    initialState<\/span>:<\/span> KeyValueGroupedDataset<\/span>[<\/span>K<\/span>, S<\/span>])(<\/span>\r\n    func<\/span>:<\/span> (<\/span>K<\/span>,<\/span> Iterator<\/span>[<\/span>V<\/span>],<\/span> GroupState<\/span>[<\/span>S<\/span>])<\/span> =><\/span> Iterator<\/span>[<\/span>U<\/span>])<\/span>\r\n<\/code><\/pre>\n
                  \u8fd9\u4e2a\u81ea\u5b9a\u4e49\u72b6\u6001\u51fd\u6570\u5c06\u4fdd\u6301\u9047\u5230\u7684\u6c34\u679c\u6570\u91cf\u3002<\/p>\n
                  val<\/span> fruitCountFunc<\/span> =(<\/span>key<\/span>:<\/span> String<\/span>,<\/span> values<\/span>:<\/span> Iterator<\/span>[<\/span>String<\/span>],<\/span> state<\/span>:<\/span> GroupState<\/span>[<\/span>RunningCount<\/span>])<\/span> =><\/span> {<\/span>\r\n  val<\/span> count<\/span> =<\/span> state<\/span>.<\/span>getOption<\/span>.<\/span>map<\/span>(<\/span>_<\/span>.<\/span>count<\/span>).<\/span>getOrElse<\/span>(<\/span>0L<\/span>)<\/span> +<\/span> valList<\/span>.<\/span>size<\/span>\r\n  state<\/span>.<\/span>update<\/span>(<\/span>new<\/span> RunningCount<\/span>(<\/span>count<\/span>))<\/span>\r\n  Iterator<\/span>((<\/span>key<\/span>,<\/span> count<\/span>.<\/span>toString<\/span>))<\/span>\r\n}<\/span>\r\n<\/code><\/pre>\n
                  \u5728\u8fd9\u4e2a\u4f8b\u5b50\u4e2d\uff0c\u901a\u8fc7\u8bbe\u7f6e\u7279\u5b9a\u6c34\u679c\u7684\u521d\u59cb\u503c\u6765\u6307\u5b9a\u8fd9\u4e2a\u64cd\u4f5c\u5458\u7684\u521d\u59cb\u72b6\u6001\u3002<\/p>\n
                  val<\/span> fruitCountInitialDS<\/span>:<\/span> Dataset<\/span>[(<\/span>String<\/span>, RunningCount<\/span>)]<\/span> =<\/span> Seq<\/span>(<\/span>\r\n  (<\/span>\"apple\"<\/span>,<\/span> new<\/span> RunningCount<\/span>(<\/span>1<\/span>)),<\/span>\r\n  (<\/span>\"orange\"<\/span>,<\/span> new<\/span> RunningCount<\/span>(<\/span>2<\/span>)),<\/span>\r\n  (<\/span>\"mango\"<\/span>,<\/span> new<\/span> RunningCount<\/span>(<\/span>5<\/span>)),<\/span>\r\n).<\/span>toDS<\/span>()<\/span>\r\n\r\nval<\/span> fruitCountInitial<\/span> =<\/span> initialState<\/span>.<\/span>groupByKey<\/span>(<\/span>x<\/span> =><\/span> x<\/span>.<\/span>_1<\/span>).<\/span>mapValues<\/span>(<\/span>_<\/span>.<\/span>_2<\/span>)<\/span>\r\n\r\nfruitStream<\/span>\r\n  .<\/span>groupByKey<\/span>(<\/span>x<\/span> =><\/span> x<\/span>)<\/span>\r\n  .<\/span>flatMapGroupsWithState<\/span>(<\/span>Update<\/span>,<\/span> GroupStateTimeout<\/span>.<\/span>NoTimeout<\/span>,<\/span> fruitCountInitial<\/span>)(<\/span>fruitCountFunc<\/span>)<\/span>\r\n<\/code><\/pre>\n
                  \u4ee5\u4e00\u79cd\u7b80\u4fbf\u7684\u65b9\u5f0f\u8fdb\u884c\u903b\u8f91\u6d4b\u8bd5<\/h3>\n
                  \u6211\u4eec\u4e5f\u53ef\u4ee5\u4f7f\u7528 TestGroupState API \u6765\u6d4b\u8bd5\u72b6\u6001\u66f4\u65b0\u3002<\/p>\n
                  import<\/span> org.apache.spark.sql.streaming._<\/span>\r\nimport<\/span> org.apache.spark.api.java.Optional<\/span>\r\n\r\ntest<\/span>(<\/span>\"flatMapGroupsWithState's state update function\"<\/span>)<\/span> {<\/span>\r\n  var<\/span> prevState<\/span> =<\/span> TestGroupState<\/span>.<\/span>create<\/span>[<\/span>UserStatus<\/span>](<\/span>\r\n    optionalState<\/span> =<\/span> Optional<\/span>.<\/span>empty<\/span>[<\/span>UserStatus<\/span>],<\/span>\r\n    timeoutConf<\/span> =<\/span> GroupStateTimeout<\/span>.<\/span>EventTimeTimeout<\/span>,<\/span>\r\n    batchProcessingTimeMs<\/span> =<\/span> 1L<\/span>,<\/span>\r\n    eventTimeWatermarkMs<\/span> =<\/span> Optional<\/span>.<\/span>of<\/span>(<\/span>1L<\/span>),<\/span>\r\n    hasTimedOut<\/span> =<\/span> false<\/span>)<\/span>\r\n\r\n  val<\/span> userId<\/span>:<\/span> String<\/span> =<\/span> ...<\/span>\r\n  val<\/span> actions<\/span>:<\/span> Iterator<\/span>[<\/span>UserAction<\/span>]<\/span> =<\/span> ...<\/span>\r\n\r\n  assert<\/span>(!<\/span>prevState<\/span>.<\/span>hasUpdated<\/span>)<\/span>\r\n\r\n  updateState<\/span>(<\/span>userId<\/span>,<\/span> actions<\/span>,<\/span> prevState<\/span>)<\/span>\r\n\r\n  assert<\/span>(<\/span>prevState<\/span>.<\/span>hasUpdated<\/span>)<\/span>\r\n\r\n}<\/span>\r\n<\/code><\/pre>\n
                  \u60a8\u8fd8\u53ef\u4ee5\u5728Databricks\u7684\u6587\u6863\u4e2d\u67e5\u770b\u5176\u4ed6\u793a\u4f8b\u3002<\/p>\n
                  \u4f1a\u8bdd\u7a97\u53e3\u7684\u672c\u5730\u652f\u6301<\/h2>\n
                  \u5728\u7ed3\u6784\u5316\u6d41\u5904\u7406\u4e2d\uff0c\u6211\u4eec\u5f15\u5165\u4e86\u7528\u4e8e\u5728\u57fa\u4e8e\u4e8b\u4ef6\u65f6\u95f4\u7684\u56fa\u5b9a\u957f\u5ea6\u6eda\u52a8\u7a97\u53e3\u548c\u6ed1\u52a8\u7a97\u53e3\u4e2d\u8fdb\u884c\u805a\u5408\u7684\u529f\u80fd\u3002Spark 3.2\u5f15\u5165\u4e86\u5177\u6709\u52a8\u6001\u7a97\u53e3\u957f\u5ea6\u7684\u4f1a\u8bdd\u7a97\u53e3\u6982\u5ff5\u3002\u4ee5\u524d\u9700\u8981\u4f7f\u7528flatMapGroupsWithState\u6765\u5b9e\u73b0\u81ea\u5b9a\u4e49\u72b6\u6001\u64cd\u4f5c\u7b26\u3002<\/p>\n
                  \u6211\u5011\u5c07\u8209\u4e00\u500b\u4f7f\u7528\u52d5\u614b\u5dee\u8ddd\u7684\u4f8b\u5b50\u3002<\/p>\n
                  # Define the session window having dynamic gap duration based on eventType\r\n<\/span>session_window<\/span> expr<\/span> =<\/span> session_window<\/span>(<\/span>events<\/span>.<\/span>timestamp<\/span>,<\/span> \\\r\n    when<\/span>(<\/span>events<\/span>.<\/span>eventType<\/span> ==<\/span> \"type1\"<\/span>,<\/span> \"5 seconds\"<\/span>)<\/span> \\\r\n    .<\/span>when<\/span>(<\/span>events<\/span>.<\/span>eventType<\/span> ==<\/span> \"type2\"<\/span>,<\/span> \"20 seconds\"<\/span>)<\/span> \\\r\n    .<\/span>otherwise<\/span>(<\/span>\"5 minutes\"<\/span>))<\/span>\r\n\r\n# Group the data by session window and userId, and compute the count of each group\r\n<\/span>windowedCountsDF<\/span> =<\/span> events<\/span> \\\r\n    .<\/span>withWatermark<\/span>(<\/span>\"timestamp\"<\/span>,<\/span> \"10 minutes\"<\/span>)<\/span> \\\r\n    .<\/span>groupBy<\/span>(<\/span>events<\/span>.<\/span>userID<\/span>,<\/span> session_window_expr<\/span>)<\/span> \\\r\n    .<\/span>count<\/span>()<\/span>\r\n<\/code><\/pre>\n
                  \u76ee\u68072\uff1a\u63d0\u9ad8\u5bf9\u6d41\u5f0f\u5de5\u4f5c\u8d1f\u8f7d\u7684\u53ef\u89c2\u5bdf\u6027<\/h1>\n
                  \u901a\u8fc7\u4f7f\u7528 StreamingQueryListener API\uff0c\u60a8\u53ef\u4ee5\u5f02\u6b65\u76d1\u89c6 SparkSession \u4e2d\u7684\u67e5\u8be2\uff0c\u5e76\u5b9a\u4e49\u7528\u4e8e\u67e5\u8be2\u72b6\u6001\u3001\u8fdb\u5ea6\u548c\u505c\u6b62\u4e8b\u4ef6\u7684\u81ea\u5b9a\u4e49\u56de\u8c03\u51fd\u6570\u3002\u7136\u800c\uff0c\u5728\u5fae\u6279\u5904\u7406\u4e2d\u7406\u89e3\u80cc\u538b\u548c\u786e\u5b9a\u74f6\u9888\u4ecd\u7136\u662f\u56f0\u96be\u7684\u3002\u5728 Databricks Runtime 8.1 \u4e2d\uff0cStreamingQueryProgress \u5bf9\u8c61\u4f1a\u62a5\u544a Kafka\u3001Kinesis\u3001Delta Lake \u548c Auto Loader \u6570\u636e\u6e90\u7684\u80cc\u538b\u6307\u6807\u3002<\/p>\n
                  \u4ee5\u4e0b\u662f\u7531 Kafka \u63d0\u4f9b\u7684\u6307\u6807\u7684\u793a\u4f8b\u3002<\/p>\n
                  {\r\n  \"sources\" : [ {\r\n    \"description\" : \"KafkaV2[Subscribe[topic]]\",\r\n    \"metrics\" : {\r\n      \"avgOffsetsBehindLatest\" : \"4.0\",\r\n      \"maxOffsetsBehindLatest\" : \"4\",\r\n      \"minOffsetsBehindLatest\" : \"4\",\r\n      \"estimatedTotalBytesBehindLatest\" : \"80.0\"\r\n    },\r\n  } ]\r\n<\/code><\/pre>\n
                  Databricks\u8fd0\u884c\u65f68.3\u5c06\u5f15\u5165\u5b9e\u65f6\u6307\u6807\uff0c\u4ee5\u5e2e\u52a9\u7406\u89e3RocksDB\u72b6\u6001\u5b58\u50a8\u7684\u6027\u80fd\u548c\u8c03\u8bd5\u72b6\u6001\u64cd\u4f5c\u7684\u6027\u80fd\u3002\u8fd9\u6709\u52a9\u4e8e\u5728\u5f02\u6b65\u68c0\u67e5\u70b9\u4e2d\u786e\u5b9a\u76ee\u6807\u5de5\u4f5c\u8d1f\u8f7d\u3002<\/p>\n
                  \u6211\u5c06\u7ed9\u51fa\u4e00\u4e2a\u65b0\u7684\u72b6\u6001\u5b58\u50a8\u6307\u6807\u793a\u4f8b\u3002<\/p>\n
                  {\r\n  \"id\" : \"6774075e-8869-454b-ad51-513be86cfd43\",\r\n  \"runId\" : \"3d08104d-d1d4-4d1a-b21e-0b2e1fb871c5\",\r\n  \"batchId\" : 7,\r\n  \"stateOperators\" : [ {\r\n    \"numRowsTotal\" : 20000000,\r\n    \"numRowsUpdated\" : 20000000,\r\n    \"memoryUsedBytes\" : 31005397,\r\n    \"numRowsDroppedByWatermark\" : 0,\r\n    \"customMetrics\" : {\r\n      \"rocksdbBytesCopied\" : 141037747,\r\n      \"rocksdbCommitCheckpointLatency\" : 2,\r\n      \"rocksdbCommitCompactLatency\" : 22061,\r\n      \"rocksdbCommitFileSyncLatencyMs\" : 1710,\r\n      \"rocksdbCommitFlushLatency\" : 19032,\r\n      \"rocksdbCommitPauseLatency\" : 0,\r\n      \"rocksdbCommitWriteBatchLatency\" : 56155,\r\n      \"rocksdbFilesCopied\" : 2,\r\n      \"rocksdbFilesReused\" : 0,\r\n      \"rocksdbGetCount\" : 40000000,\r\n      \"rocksdbGetLatency\" : 21834,\r\n      \"rocksdbPutCount\" : 1,\r\n      \"rocksdbPutLatency\" : 56155599000,\r\n      \"rocksdbReadBlockCacheHitCount\" : 1988,\r\n      \"rocksdbReadBlockCacheMissCount\" : 40341617,\r\n      \"rocksdbSstFileSize\" : 141037747,\r\n      \"rocksdbTotalBytesReadByCompaction\" : 336853375,\r\n      \"rocksdbTotalBytesReadByGet\" : 680000000,\r\n      \"rocksdbTotalBytesReadThroughIterator\" : 0,\r\n      \"rocksdbTotalBytesWrittenByCompaction\" : 141037747,\r\n      \"rocksdbTotalBytesWrittenByPut\" : 740000012,\r\n      \"rocksdbTotalCompactionLatencyMs\" : 21949695000,\r\n      \"rocksdbWriterStallLatencyMs\" : 0,\r\n      \"rocksdbZipFileBytesUncompressed\" : 7038\r\n    }\r\n  } ],\r\n  \"sources\" : [ {\r\n  } ],\r\n  \"sink\" : {\r\n  }\r\n}\r\n<\/code><\/pre>\n
                  \u76ee\u68073\uff1a\u8d44\u6e90\u5206\u914d\u548c\u53ef\u6269\u5c55\u6027\u7684\u6539\u5584\u3002<\/h1>\n
                  Delta Live Tables\uff08DLT\uff09\u7684\u6d41\u5f0f\u81ea\u52a8\u7f29\u653e<\/h2>\n
                  \u6211\u4eec\u5728\u53bb\u5e74\u7684Data + AI Summit\u4e0a\u53d1\u5e03\u4e86Delta Live Tables\uff0c\u8fd9\u662f\u4e00\u4e2a\u80fd\u591f\u58f0\u660e\u6027\u5730\u6784\u5efa\u548c\u7f16\u6392\u6570\u636e\u7ba1\u9053\uff0c\u5e76\u6781\u5927\u5730\u62bd\u8c61\u5316\u4e86\u96c6\u7fa4\u548c\u8282\u70b9\u7c7b\u578b\u8bbe\u7f6e\u7684\u6846\u67b6\u3002\u6211\u4eec\u8fdb\u4e00\u6b65\u63a8\u52a8\u4e86\u8fd9\u4e00\u6846\u67b6\uff0c\u5e76\u6539\u8fdb\u4e86\u73b0\u6709\u7684Databricks\u4f18\u5316\u81ea\u52a8\u7f29\u653e\u529f\u80fd\uff0c\u5f15\u5165\u4e86\u5bf9\u6d41\u5f0f\u7ba1\u9053\u7684\u667a\u80fd\u89e3\u51b3\u65b9\u6848\u3002\u4ee5\u4e0b\u662f\u5b83\u7684\u4e00\u4e9b\u4f18\u70b9\uff1a<\/p>\n
                  \u30af\u30e9\u30b9\u30bf\u30fc\u4f7f\u7528\u7387\u306e\u6539\u5584: \u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u30ef\u30fc\u30af\u30ed\u30fc\u30c9\u3067\u8ca0\u8377\u306e\u5909\u52d5\u304c\u3042\u308b\u30b7\u30ca\u30ea\u30aa\u306b\u304a\u3044\u3066\u3001\u65b0\u305f\u306a\u30d0\u30c3\u30af\u30d7\u30ec\u30c3\u30b7\u30e3\u30fc\u30e1\u30c8\u30ea\u30af\u30b9\u3092\u6d3b\u7528\u3057\u3066\u30af\u30e9\u30b9\u30bf\u30fc\u306e\u30b5\u30a4\u30ba\u3092\u8abf\u6574\u3057\u3001\u30af\u30e9\u30b9\u30bf\u30fc\u4f7f\u7528\u7387\u3092\u6539\u5584\u3057\u307e\u3059\u3002<\/p>\n
                  \u30d7\u30ed\u30a2\u30af\u30c6\u30a3\u30d6\u306a\u30ef\u30fc\u30ab\u30fc\u306e\u30b0\u30ec\u30fc\u30b9\u30d5\u30eb\u30b7\u30e3\u30c3\u30c8\u30c0\u30a6\u30f3: \u65e2\u5b58\u306e\u30aa\u30fc\u30c8\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0\u306e\u30bd\u30ea\u30e5\u30fc\u30b7\u30e7\u30f3\u306f\u30ce\u30fc\u30c9\u304c\u30a2\u30a4\u30c9\u30eb\u72b6\u614b\u306e\u5834\u5408\u306b\u306e\u307f\u30ea\u30c8\u30e9\u30a4\u3092\u884c\u3044\u307e\u3059\u304c\u3001\u65b0\u305f\u306aDLT\u306e\u30aa\u30fc\u30c8\u30b9\u30b1\u30fc\u30e9\u30fc\u306f\u4f7f\u7528\u7387\u304c\u4f4e\u3044\u5834\u5408\u3001\u30b7\u30e3\u30c3\u30c8\u30c0\u30a6\u30f3\u306b\u3088\u3063\u3066\u30bf\u30b9\u30af\u304c\u5931\u6557\u3057\u306a\u3044\u3053\u3068\u3092\u4fdd\u8a3c\u3057\u3064\u3064\u3082\u3001\u30d7\u30ed\u30a2\u30af\u30c6\u30a3\u30d6\u306b\u9078\u629e\u3055\u308c\u305f\u30ce\u30fc\u30c9\u3092\u30b7\u30e3\u30c3\u30c8\u30c0\u30a6\u30f3\u3057\u307e\u3059\u3002<\/p>\n
                  \u8bf7\u5728\u4e0eDatabricks\u8d26\u6237\u56e2\u961f\u8054\u7cfb\u540e\u83b7\u53d6\u66f4\u591a\u8be6\u7ec6\u4fe1\u606f\uff0c\u56e0\u4e3a\u672c\u4e66\u7684\u64b0\u5199\u9636\u6bb5\uff0c\u6b64\u529f\u80fd\u4ecd\u5904\u4e8e\u79c1\u4eba\u9884\u89c8\u9636\u6bb5\u3002<\/p>\n
                  \u7acb\u5373\u89e6\u53d1<\/h2>\n
                  \u5728\u7ed3\u6784\u5316\u6d41\u5904\u7406\u4e2d\uff0c\u53ef\u4ee5\u901a\u8fc7\u4f7f\u7528\u89e6\u53d1\u5668\u6765\u5b9a\u4e49\u6d41\u5f0f\u67e5\u8be2\u7684\u6570\u636e\u5904\u7406\u65f6\u673a\u3002\u8fd9\u4e9b\u89e6\u53d1\u5668\u7c7b\u578b\u5305\u62ec\u5fae\u6279\u6b21\uff08\u9ed8\u8ba4\uff09\u3001\u56fa\u5b9a\u7684\u5fae\u6279\u6b21\u95f4\u9694\uff08Trigger.ProcessingTime\uff08””\uff09\uff09\u3001\u4ec5\u4e00\u6b21\u7684\u5fae\u6279\u6b21\uff08Trigger.Once\uff09\u548c\u8fde\u7eed\uff08Trigger.Continuous\uff09\u3002<\/p>\n
                  Databricks 10.1 \u53ca\u4ee5\u4e0a\u7684\u7248\u672c\u5f15\u5165\u4e86\u65b0\u578b\u89e6\u53d1\u5668\u3002Trigger.AvailableNow \u7c7b\u4f3c\u4e8e Trigger.Once\uff0c\u4f46\u63d0\u4f9b\u4e86\u66f4\u4f18\u79c0\u7684\u53ef\u6269\u5c55\u6027\u3002\u4e0e Trigger.Once \u7c7b\u4f3c\uff0c\u5b83\u4f1a\u5728\u67e5\u8be2\u505c\u6b62\u4e4b\u524d\u5904\u7406\u53ef\u7528\u7684\u6240\u6709\u6570\u636e\uff0c\u4f46\u4e0d\u662f\u5728\u4e00\u4e2a\u6279\u6b21\u4e2d\u5904\u7406\uff0c\u800c\u662f\u5728\u591a\u4e2a\u6279\u6b21\u4e2d\u5904\u7406\u3002\u8fd9\u53d7\u5230 Delta Lake \u548c Auto Loader \u7684\u6d41\u5f0f\u6570\u636e\u6e90\u7684\u652f\u6301\u3002<\/p>\n
                  \u8bf7\u7ed9\u51fa\u4e00\u4e2a\u6837\u4f8b\u3002<\/p>\n
                  spark<\/span>.<\/span>readStream<\/span>\r\n  .<\/span>format<\/span>(<\/span>\"delta\"<\/span>)<\/span>\r\n  .<\/span>option<\/span>(<\/span>\"maxFilesPerTrigger\"<\/span>,<\/span> \"1\"<\/span>)<\/span>\r\n  .<\/span>load<\/span>(<\/span>inputDir<\/span>)<\/span>\r\n  .<\/span>writeStream<\/span>\r\n  .<\/span>trigger<\/span>(<\/span>Trigger<\/span>.<\/span>AvailableNow<\/span>)<\/span>\r\n  .<\/span>option<\/span>(<\/span>\"checkpointLocation\"<\/span>,<\/span> checkpointDir<\/span>)<\/span>\r\n  .<\/span>start<\/span>()<\/span>\r\n<\/code><\/pre>\n
                  \u6458\u8981<\/h1>\n
                  \u57282022\u5e74\uff0c\u6211\u4eec\u5c06\u7ee7\u7eed\u52a0\u901f\u5728\u7ed3\u6784\u5316\u6d41\u5a92\u4f53\u9886\u57df\u7684\u521b\u65b0\uff0c\u8fdb\u4e00\u6b65\u63d0\u5347\u6027\u80fd\u3001\u51cf\u5c11\u5ef6\u8fdf\uff0c\u5e76\u63d0\u4f9b\u5168\u65b0\u7684\u7cbe\u5f69\u529f\u80fd\u3002\u671f\u5f85\u60a8\u5bf9\u4eca\u5e74\u7684\u8fdb\u5c55\u4fdd\u6301\u5173\u6ce8\uff01<\/p>\n
                  Databricks\u514d\u8d39\u8bd5\u7528<\/h3>\n
                  Databricks \u514d\u8d39\u8bd5\u7528<\/p>\n","protected":false},"excerpt":{"rendered":"
                  2021\u5e74\u4e3aDatabricks\u548cApache Spark\u5f00\u53d1\u7684\u6240\u6709\u65b0\u7ed3\u6784\u5316\u6d41\u5904\u7406\u529f\u80fd\u6982\u8ff0\u3002 \u8fd9\u672c\u4e66\u662f\u7ffb\u8bd1\u7684\uff0c […]<\/p>\n","protected":false},"author":12,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-47195","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"\n\u7ed3\u6784\u5316\u6d41\u5a92\u4f53\uff1a\u56de\u987e2021\u5e74 - Blog - Silicon Cloud<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.silicloud.com\/zh\/blog\/\u7ed3\u6784\u5316\u6d41\u5a92\u4f53\uff1a\u56de\u987e2021\u5e74\/\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"\u7ed3\u6784\u5316\u6d41\u5a92\u4f53\uff1a\u56de\u987e2021\u5e74\" \/>\n<meta property=\"og:description\" content=\"2021\u5e74\u4e3aDatabricks\u548cApache Spark\u5f00\u53d1\u7684\u6240\u6709\u65b0\u7ed3\u6784\u5316\u6d41\u5904\u7406\u529f\u80fd\u6982\u8ff0\u3002 \u8fd9\u672c\u4e66\u662f\u7ffb\u8bd1\u7684\uff0c […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.silicloud.com\/zh\/blog\/\u7ed3\u6784\u5316\u6d41\u5a92\u4f53\uff1a\u56de\u987e2021\u5e74\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog - Silicon Cloud\" \/>\n<meta property=\"article:published_time\" content=\"2023-03-25T15:12:48+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-04-29T15:48:40+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7d67913a08637a69fb5f\/8-1.png\" \/>\n<meta name=\"author\" content=\"\u9038, \u79d1\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"\u9038, \u79d1\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e7%bb%93%e6%9e%84%e5%8c%96%e6%b5%81%e5%aa%92%e4%bd%93%ef%bc%9a%e5%9b%9e%e9%a1%be2021%e5%b9%b4\/\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e7%bb%93%e6%9e%84%e5%8c%96%e6%b5%81%e5%aa%92%e4%bd%93%ef%bc%9a%e5%9b%9e%e9%a1%be2021%e5%b9%b4\/\",\"name\":\"\u7ed3\u6784\u5316\u6d41\u5a92\u4f53\uff1a\u56de\u987e2021\u5e74 - Blog - Silicon Cloud\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\"},\"datePublished\":\"2023-03-25T15:12:48+00:00\",\"dateModified\":\"2024-04-29T15:48:40+00:00\",\"author\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/85c1dae56e6ea1e695c73d33c684d487\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e7%bb%93%e6%9e%84%e5%8c%96%e6%b5%81%e5%aa%92%e4%bd%93%ef%bc%9a%e5%9b%9e%e9%a1%be2021%e5%b9%b4\/#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.silicloud.com\/zh\/blog\/%e7%bb%93%e6%9e%84%e5%8c%96%e6%b5%81%e5%aa%92%e4%bd%93%ef%bc%9a%e5%9b%9e%e9%a1%be2021%e5%b9%b4\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e7%bb%93%e6%9e%84%e5%8c%96%e6%b5%81%e5%aa%92%e4%bd%93%ef%bc%9a%e5%9b%9e%e9%a1%be2021%e5%b9%b4\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.silicloud.com\/zh\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"\u7ed3\u6784\u5316\u6d41\u5a92\u4f53\uff1a\u56de\u987e2021\u5e74\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/\",\"name\":\"Blog - Silicon Cloud\",\"description\":\"\",\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/85c1dae56e6ea1e695c73d33c684d487\",\"name\":\"\u9038, \u79d1\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c94f6d9cbbfbca863fab309840bd690c153c95f8490c290ad2ed54dd693dad16?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c94f6d9cbbfbca863fab309840bd690c153c95f8490c290ad2ed54dd693dad16?s=96&d=mm&r=g\",\"caption\":\"\u9038, \u79d1\"},\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/author\/keyi\/\"},{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e7%bb%93%e6%9e%84%e5%8c%96%e6%b5%81%e5%aa%92%e4%bd%93%ef%bc%9a%e5%9b%9e%e9%a1%be2021%e5%b9%b4\/#local-main-organization-logo\",\"url\":\"\",\"contentUrl\":\"\",\"caption\":\"Blog - Silicon Cloud\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"\u7ed3\u6784\u5316\u6d41\u5a92\u4f53\uff1a\u56de\u987e2021\u5e74 - Blog - Silicon Cloud","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.silicloud.com\/zh\/blog\/\u7ed3\u6784\u5316\u6d41\u5a92\u4f53\uff1a\u56de\u987e2021\u5e74\/","og_locale":"zh_CN","og_type":"article","og_title":"\u7ed3\u6784\u5316\u6d41\u5a92\u4f53\uff1a\u56de\u987e2021\u5e74","og_description":"2021\u5e74\u4e3aDatabricks\u548cApache Spark\u5f00\u53d1\u7684\u6240\u6709\u65b0\u7ed3\u6784\u5316\u6d41\u5904\u7406\u529f\u80fd\u6982\u8ff0\u3002 \u8fd9\u672c\u4e66\u662f\u7ffb\u8bd1\u7684\uff0c […]","og_url":"https:\/\/www.silicloud.com\/zh\/blog\/\u7ed3\u6784\u5316\u6d41\u5a92\u4f53\uff1a\u56de\u987e2021\u5e74\/","og_site_name":"Blog - Silicon Cloud","article_published_time":"2023-03-25T15:12:48+00:00","article_modified_time":"2024-04-29T15:48:40+00:00","og_image":[{"url":"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7d67913a08637a69fb5f\/8-1.png"}],"author":"\u9038, \u79d1","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"\u9038, \u79d1","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"2 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e7%bb%93%e6%9e%84%e5%8c%96%e6%b5%81%e5%aa%92%e4%bd%93%ef%bc%9a%e5%9b%9e%e9%a1%be2021%e5%b9%b4\/","url":"https:\/\/www.silicloud.com\/zh\/blog\/%e7%bb%93%e6%9e%84%e5%8c%96%e6%b5%81%e5%aa%92%e4%bd%93%ef%bc%9a%e5%9b%9e%e9%a1%be2021%e5%b9%b4\/","name":"\u7ed3\u6784\u5316\u6d41\u5a92\u4f53\uff1a\u56de\u987e2021\u5e74 - Blog - Silicon Cloud","isPartOf":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website"},"datePublished":"2023-03-25T15:12:48+00:00","dateModified":"2024-04-29T15:48:40+00:00","author":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/85c1dae56e6ea1e695c73d33c684d487"},"breadcrumb":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e7%bb%93%e6%9e%84%e5%8c%96%e6%b5%81%e5%aa%92%e4%bd%93%ef%bc%9a%e5%9b%9e%e9%a1%be2021%e5%b9%b4\/#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.silicloud.com\/zh\/blog\/%e7%bb%93%e6%9e%84%e5%8c%96%e6%b5%81%e5%aa%92%e4%bd%93%ef%bc%9a%e5%9b%9e%e9%a1%be2021%e5%b9%b4\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e7%bb%93%e6%9e%84%e5%8c%96%e6%b5%81%e5%aa%92%e4%bd%93%ef%bc%9a%e5%9b%9e%e9%a1%be2021%e5%b9%b4\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.silicloud.com\/zh\/blog\/"},{"@type":"ListItem","position":2,"name":"\u7ed3\u6784\u5316\u6d41\u5a92\u4f53\uff1a\u56de\u987e2021\u5e74"}]},{"@type":"WebSite","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website","url":"https:\/\/www.silicloud.com\/zh\/blog\/","name":"Blog - Silicon Cloud","description":"","inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/85c1dae56e6ea1e695c73d33c684d487","name":"\u9038, \u79d1","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/c94f6d9cbbfbca863fab309840bd690c153c95f8490c290ad2ed54dd693dad16?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c94f6d9cbbfbca863fab309840bd690c153c95f8490c290ad2ed54dd693dad16?s=96&d=mm&r=g","caption":"\u9038, \u79d1"},"url":"https:\/\/www.silicloud.com\/zh\/blog\/author\/keyi\/"},{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e7%bb%93%e6%9e%84%e5%8c%96%e6%b5%81%e5%aa%92%e4%bd%93%ef%bc%9a%e5%9b%9e%e9%a1%be2021%e5%b9%b4\/#local-main-organization-logo","url":"","contentUrl":"","caption":"Blog - Silicon Cloud"}]}},"_links":{"self":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/47195","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/users\/12"}],"replies":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/comments?post=47195"}],"version-history":[{"count":2,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/47195\/revisions"}],"predecessor-version":[{"id":88386,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/47195\/revisions\/88386"}],"wp:attachment":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/media?parent=47195"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/categories?post=47195"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/tags?post=47195"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}