{"id":47439,"date":"2023-01-10T01:22:03","date_gmt":"2023-04-16T05:36:35","guid":{"rendered":"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bb%80%e4%b9%88%e6%98%af%e6%b5%81%e5%a4%84%e7%90%86%ef%bc%9f%e4%bb%a5%e5%8f%8a2016%e5%b9%b4%e5%8f%91%e7%94%9f%e7%9a%84%e4%ba%8b%e4%bb%b6%e3%80%82\/"},"modified":"2024-04-29T09:17:08","modified_gmt":"2024-04-29T01:17:08","slug":"%e4%bb%80%e4%b9%88%e6%98%af%e6%b5%81%e5%a4%84%e7%90%86%ef%bc%9f%e4%bb%a5%e5%8f%8a2016%e5%b9%b4%e5%8f%91%e7%94%9f%e7%9a%84%e4%ba%8b%e4%bb%b6%e3%80%82","status":"publish","type":"post","link":"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bb%80%e4%b9%88%e6%98%af%e6%b5%81%e5%a4%84%e7%90%86%ef%bc%9f%e4%bb%a5%e5%8f%8a2016%e5%b9%b4%e5%8f%91%e7%94%9f%e7%9a%84%e4%ba%8b%e4%bb%b6%e3%80%82\/","title":{"rendered":"\u4ec0\u4e48\u662f\u6d41\u5904\u7406\uff1f\u4ee5\u53ca2016\u5e74\u53d1\u751f\u7684\u4e8b\u4ef6"},"content":{"rendered":"<h2>\u8fd9\u7bc7\u6587\u7ae0\u8bb2\u8ff0\u4e86\u4ec0\u4e48\u5185\u5bb9\uff1f<\/h2>\n<p>\u6d41\u5904\u7406\u6700\u521d\u5c31\u662f\u4ece\u67d0\u4e2a\u5730\u65b9\u5f00\u59cb\u7684\uff0c\u5b83\u662f\u5173\u4e8e\u5fc5\u8981\u8003\u8651\u4e8b\u9879\u7684\u4fe1\u606f\u7684\u603b\u7ed3\uff0c\u4f9b\u56de\u987e\u4f7f\u7528\u3002<br \/>\n\u6b64\u5916\uff0c\u6211\u8fd8\u603b\u7ed3\u4e86\u4eca\u5e74\u4e2a\u4eba\u8ba4\u4e3a\u5bf9\u8fd9\u4e2a\u9886\u57df\u5f71\u54cd\u5f88\u5927\u7684\u4e8b\u4ef6\u3002<\/p>\n<p>\u4e3a\u4e86\u4f5c\u4e3a\u89e3\u91ca\u7ed9\u5176\u4ed6\u4eba\u7684\u57fa\u7840\u800c\u603b\u7ed3\uff0c\u5bf9\u4e8e\u5df2\u7ecf\u719f\u6089\u8fd9\u4e2a\u9886\u57df\u7684\u4eba\u6765\u8bf4\uff0c\u53ef\u80fd\u4f1a\u6709\u5f88\u591a\u5197\u957f\u7684\u5185\u5bb9\u3002\u5982\u679c\u662f\u8fd9\u79cd\u60c5\u51b5\uff0c\u53ef\u4ee5\u9002\u5f53\u5730\u8df3\u8fc7\u9605\u8bfb\u3002\u6b64\u5916\uff0c\u6211\u8fd8\u4ece\u6211\u7684\u5176\u4ed6\u6587\u7ae0\u4e2d\u63d0\u53d6\u4e86\u4e00\u4e9b\u5185\u5bb9\uff0c\u6240\u4ee5\u53ef\u80fd\u4f1a\u6709\u4e00\u4e9b\u91cd\u590d\u3002\u7279\u522b\u662f\u5173\u4e8e\u8c37\u6b4c\u5bf9\u4e8e\u6d41\u6570\u636e\u5904\u7406\u7684\u60f3\u6cd5\uff0c\u53ef\u80fd\u4f1a\u6709\u4e9b\u91cd\u53e0\u7684\u76ee\u7684\u2026\u2026<\/p>\n<p>\u6211\u6253\u7b97\u5c3d\u91cf\u4e0d\u4f7f\u7528\u592a\u591a\u7279\u5b9a\u7684\u8bcd\u6c47\u6765\u5199\uff0c\u5982\u679c\u6709\u54ea\u4e2a\u5730\u65b9\u4e0d\u660e\u767d\uff0c\u8bf7\u60a8\u7559\u4e0b\u8bc4\u8bba\uff0c\u8c22\u8c22\u3002<\/p>\n<p>\u5728\u4e2d\u6587\u4e2d\uff0c\u53ea\u9700\u8981\u4e00\u4e2a\u9009\u9879\u5c06\u5176\u91cd\u8ff0\u4e3a\uff1a\u201c\u5f53\u53ea\u5199\u2018\u6d41\u5904\u7406\u2019\u65f6\uff0c\u5bb9\u6613\u4e0e\u6d41\u5a92\u4f53\u4f20\u8f93\u7b49\u6df7\u6dc6\uff0c\u4f46\u5728\u8fd9\u91cc\u6211\u4eec\u6307\u7684\u662f\u5904\u7406\u6570\u636e\u6d41\u7684\u6280\u672f\u96c6\u7fa4\uff0c\u6240\u4ee5\u6211\u4eec\u5c06\u5176\u63cf\u8ff0\u4e3a\u2018\u6d41\u5904\u7406\u2019\u3002\u201d<\/p>\n<h2>\u4ec5\u63d0\u4f9b\u4e00\u79cd\u4e2d\u6587\u7684\u9009\u62e9\u6765\u91ca\u4e49\u4ee5\u4e0b\u5185\u5bb9\uff1a \u53c2\u8003\u8d44\u6599<\/h2>\n<p>\u7531\u4e8e\u5728\u6587\u7ae0\u4e2d\u9002\u65f6\u63d2\u5165\u5f15\u7528\u8d44\u6599\u4f1a\u5bfc\u81f4\u6587\u7ae0\u53d8\u5f97\u6742\u4e71\u65e0\u7ae0\uff0c\u6240\u4ee5\u6211\u5c06\u5b83\u4eec\u7edf\u4e00\u653e\u5728\u5f00\u5934\u3002<br \/>\n\u5173\u4e8eGoogleCloudDataflow\u65b9\u9762\uff0c\u8bf4\u5b83\u662f\u5426\u901a\u7528\u6709\u4e9b\u5fae\u5999\uff0c<br \/>\n\u4f46\u4ece\u5c06\u6a21\u578b\u901a\u7528\u5316\u5e76\u878d\u5165\u5176\u4e2d\u7684\u610f\u4e49\u6765\u770b\uff0c\u53ef\u4ee5\u5c06\u5176\u5f52\u7c7b\u4e3a\u5982\u6b64\u3002<\/p>\n<h5>\u6d41\u5904\u7406\u7684\u901a\u7528\u6027<\/h5>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">The world beyond batch: Streaming 101<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">The world beyond batch: Streaming 102<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">The Evolution of Massive-Scale Data Processing<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">MillWheel: Fault-Tolerant Stream Processing at Internet Scale<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">Streaming Engines for Big Data<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">Introduction to Streaming Analytics<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">Stream Processing Myths Debunked:Six Common Streaming Misconceptions<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">A Practical Guide to Selecting a Stream Processing Technology<\/ul>\n<h5>\u963f\u5e15\u5947\u00b7\u5bbe Beam<\/h5>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">Apache Beam and Google Cloud Dataflow<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">The Beam Model<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">Dynamic work rebalancing in Apache Beam<\/ul>\n<h5>AWS (\u4e9a\u9a6c\u900a\u4e91\u670d\u52a1)<\/h5>\n<ul class=\"post-ul\">AWS\u4e0a\u3067\u306e\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u30c7\u30fc\u30bf\u5206\u6790\u5165\u9580<\/ul>\n<h5>\u5bf9\u6d41<\/h5>\n<ul class=\"post-ul\">Apache Flink and Apache Kafka Streams: a comparison and guideline for users<\/ul>\n<h5>\u6570\u636e\u6d2a\u6d41<\/h5>\n<ul class=\"post-ul\">THROUGHPUT, LATENCY, AND YAHOO! PERFORMANCE BENCHMARKS. IS THERE A WINNER?<\/ul>\n<h5>\u6570\u636e\u827a\u672f\u5bb6<\/h5>\n<ul class=\"post-ul\">Stream Processing as a Foundational Paradigm and Apache Flink&#8217;s Approach to It<\/ul>\n<h1>\u4ec0\u4e48\u662f\u6d41\u5904\u7406\uff1f<\/h1>\n<h2>\u5b9e\u9645\u4e0a\uff0c\u6307\u7684\u662f\u4ec0\u4e48\u4e1c\u897f\uff1f<\/h2>\n<p>\u6d41\u5904\u7406\u8fd9\u4e2a\u8bcd\u7528\u4e8e\u5404\u79cd\u4e0d\u540c\u7684\u5e94\u7528\u3002<br \/>\n\u56e0\u6b64\uff0c\u6211\u4eec\u5c06\u5176\u5b9a\u4e49\u4e3a\u5177\u6709\u4ee5\u4e0b\u7279\u6027\u7684\u5904\u7406\uff0c\u5373\u5e94\u7528\u548c\u5b9e\u9645\u7cfb\u7edf\u4e00\u822c\u5177\u6709\u7684\u5171\u540c\u70b9\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u7121\u9650\u306b\u767a\u751f\u3057\u7d9a\u3051\u308b\u30c7\u30fc\u30bf\u3092\u51e6\u7406<\/ul>\n<\/li>\n<\/ul>\n<p>\u5e38\u306b\u5897\u5927\u3057\u7d9a\u3051\u3001\u672c\u8cea\u7684\u306b\u306f\u7121\u9650\u306b\u767a\u751f\u3059\u308b\u30c7\u30fc\u30bf\u3092\u51e6\u7406\u3057\u307e\u3059\u3002<\/p>\n<p>\u30c7\u30fc\u30bf\u6e90\u81ea\u4f53\u3082\u3057\u3070\u3057\u3070\u300c\u30b9\u30c8\u30ea\u30fc\u30e0\u30c7\u30fc\u30bf\u300d\u3068\u547c\u3070\u308c\u307e\u3059\u3002<br \/>\n\u30b7\u30b9\u30c6\u30e0\u306e\u30ed\u30b0\u3001\u30bb\u30f3\u30b5\u30fc\u30c7\u30fc\u30bf\u3001\u4eba\u9593\u306e\u884c\u52d5\u30ed\u30b0\u7b49\u591a\u3005\u5b58\u5728\u3057\u307e\u3059\u3002<br \/>\n\u307e\u305f\u3001\u3042\u308b\u5de8\u5927\u30d5\u30a1\u30a4\u30eb\u3092\u5ef6\u3005\u8aad\u307f\u8fbc\u3080\u5834\u5408\u3082\u3001\u8aad\u8fbc\u307f\u7d50\u679c\u81ea\u4f53\u306f\u30b9\u30c8\u30ea\u30fc\u30e0\u30c7\u30fc\u30bf\u3068\u8a00\u3048\u307e\u3059\u3002<\/p>\n<p>\u51e6\u7406\u304c\u6c38\u7d9a\u7684\u306b\u7d99\u7d9a<\/p>\n<p>\u5e38\u306b\u5897\u5927\u3057\u7d9a\u3051\u3001\u672c\u8cea\u7684\u306b\u306f\u7121\u9650\u306b\u767a\u751f\u3059\u308b\u30c7\u30fc\u30bf\u3067\u3042\u308b\u305f\u3081\u3001\u51e6\u7406\u3082\u6c38\u7d9a\u7684\u306b\u5b9f\u884c\u3055\u308c\u307e\u3059\u3002<\/p>\n<p>\u3068\u306f\u3044\u3048\u3001\u5b9f\u969b\u306f\u4e0a\u8a18\u306e\u3088\u3046\u306a\u30c7\u30fc\u30bf\u3067\u3042\u3063\u3066\u3082\u4e00\u5b9a\u306e\u6642\u9593\u306e\u533a\u5207\u308a\u3092\u3059\u308c\u3070\u30d0\u30c3\u30c1\u51e6\u7406\u3067\u3082\u51e6\u7406\u306f\u53ef\u80fd\u3067\u3059\u3002<br \/>\n\u305d\u306e\u305f\u3081\u3001\u533a\u5225\u306e\u610f\u5473\u3067\u3082\u6c38\u7d9a\u7684\u306b\u7d99\u7d9a\u3059\u308b\u3088\u3046\u306a\u3082\u306e\u3092\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u3068\u547c\u3073\u307e\u3059\u3002<\/p>\n<p>\u5e38\u306b\u5b9f\u884c\u3057\u7d9a\u3051\u308b\u3068\u3044\u3046\u6027\u8cea\u4e0a\u3001\u30d0\u30c3\u30c1\u51e6\u7406\u4ee5\u4e0a\u306b\u6027\u80fd\u306e\u5b89\u5b9a\u6027\u3084\u3001\u30b9\u30d1\u30a4\u30af\u6642\u306b\u6b7b\u306a\u306a\u3044\u3053\u3068\u304c\u6c42\u3081\u3089\u308c\u307e\u3059\u3002<\/p>\n<p>\u3053\u3053\u304b\u3089\u3001BackPressure\u3068\u3044\u3046\u6027\u8cea\u304c\u751f\u307e\u308c\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u4f4e\u9045\u5ef6\u3001\u8fd1\u4f3c\u5024\uff06\u4e0d\u5b9a\u671f\u306a\u7d50\u679c\u51fa\u529b<\/p>\n<p>\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u306f\u30d0\u30c3\u30c1\u51e6\u7406\u306b\u6bd4\u3079\u3066\u9045\u5ef6\u304c\u5c0f\u3055\u304f\u306a\u308a\u307e\u3059\u3002<br \/>\n\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u304c\u751f\u6210\u3059\u308b\u7d50\u679c\u306f\u4ee5\u5f8c\u306b\u8aac\u660e\u3059\u308b\u6027\u8cea\u306e\u95a2\u4fc2\u4e0a\u3001\u3057\u3070\u3057\u3070\u8fd1\u4f3c\u306e\u5024\u3060\u3063\u305f\u308a\u3001\u4e0d\u5b9a\u671f\u306a\u51fa\u529b\u304c\u884c\u308f\u308c\u307e\u3059\u3002<\/p>\n<p>\u30d0\u30c3\u30c1\u51e6\u7406\u306e\u5834\u5408\u3001\u57fa\u672c\u306f\u751f\u6210\u3055\u308c\u308b\u306e\u306f\u5b8c\u5168\u306a\u30c7\u30fc\u30bf\u3067\u3001\u51fa\u529b\u3055\u308c\u308b\u306e\u306f\u30d0\u30c3\u30c1\u51e6\u7406\u304c\u5b9f\u884c\u5b8c\u4e86\u3057\u305f\u30bf\u30a4\u30df\u30f3\u30b0\u3068\u660e\u78ba\u3067\u3059\u306e\u3067\u3001\u305d\u308c\u3068\u306e\u5bfe\u6bd4\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n<p>\u7efc\u5408\u8d77\u6765\uff0c\u867d\u7136\u53d8\u5f97\u5f88\u957f\uff0c\u4f46\u53ea\u9700\u8981\u5c06\u5176\u770b\u4f5c\u662f\u5904\u7406\u65e0\u7a77\u65e0\u5c3d\u7684\u6570\u636e\u7684\u6a21\u578b\u5373\u53ef\u3002<\/p>\n<h2>\u8fd9\u662f\u7528\u6765\u505a\u4ec0\u4e48\u7684\uff1f<\/h2>\n<p>\u867d\u7136\u53ea\u662f\u4e00\u79cd\u4f8b\u5b50\uff0c\u4f46\u4f3c\u4e4e\u5728\u4ee5\u4e0b\u60c5\u51b5\u4e0b\u88ab\u4f7f\u7528\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u8ab2\u91d1\u51e6\u7406<\/ul>\n<\/li>\n<\/ul>\n<p>\u30af\u30e9\u30a6\u30c9\u7cfb\u30b5\u30fc\u30d3\u30b9\u306e\u5229\u7528\u6599\u8ab2\u91d1<br \/>\n\u643a\u5e2f\u96fb\u8a71\u306e\u901a\u4fe1\u6599\u8ab2\u91d1<\/p>\n<p>\u30e9\u30a4\u30d6\u8cbb\u7528\u898b\u7a4d<\/p>\n<p>\u30af\u30e9\u30a6\u30c9\u7cfb\u30b5\u30fc\u30d3\u30b9\u306e\u5229\u7528\u72b6\u6cc1\u304b\u3089\u306e\u8ab2\u91d1\u898b\u7a4d\u308a<br \/>\n\u643a\u5e2f\u96fb\u8a71\u306e\u901a\u4fe1\u91cf\u898b\u7a4d\u3082\u308a<\/p>\n<p>\u4e0d\u6b63\u30fb\u7570\u5e38\u691c\u51fa<\/p>\n<p>\u3042\u308b\u30b7\u30b9\u30c6\u30e0\u306b\u5bfe\u3059\u308b\u4e0d\u6b63\u30ed\u30b0\u30a4\u30f3\u691c\u77e5<br \/>\n\u30b7\u30b9\u30c6\u30e0\u306e\u7570\u5e38\u691c\u77e5<br \/>\n\u6c17\u8c61\u30c7\u30fc\u30bf\u306e\u7570\u5e38\u691c\u77e5<br \/>\n\u682a\u4fa1\u306e\u7570\u5e38\u632f\u308c\u5e45\u691c\u77e5<br \/>\n\u30e6\u30fc\u30b6\u306e\u884c\u52d5\u306b\u5bfe\u3059\u308b\u30ec\u30b3\u30e1\u30f3\u30c9<\/p>\n<p>\u4e0d\u6b63\u30fb\u7570\u5e38\u691c\u77e5\u3067\u306f\u306a\u3044\u3067\u3059\u304c\u3001\u7279\u5fb4\u7684\u306a\u884c\u52d5\u306b\u5bfe\u3059\u308b\u53cd\u5fdc\u3068\u3044\u3046\u3053\u3068\u3067\u3053\u3053\u306b\u6319\u3052\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u4e0d\u6b63\u691c\u51fa\u7d50\u679c\u5fa9\u65e7<\/p>\n<p>\u4e0d\u6b63\u30ed\u30b0\u30a4\u30f3\u691c\u77e5\u306e\u7d50\u679c\u5bfe\u5fdc<\/p>\n<p>\u4e0d\u6b63\u3092\u691c\u77e5\u3057\u305f\u30bf\u30a4\u30df\u30f3\u30b0\u3067\u4e00\u5ea6\u505c\u6b62\u5f8c\u3001\u518d\u958b\u306a\u3069\u306e\u5bfe\u5fdc<\/p>\n<p>\u30b7\u30b9\u30c6\u30e0\u306e\u7570\u5e38\u5fa9\u65e7<br \/>\n\u6c17\u8c61\u8b66\u5831\u5f8c\u306e\u7d4c\u904e\u901a\u77e5<br \/>\n\u7570\u5e38\u3092\u691c\u77e5\u3057\u305f\u7d50\u679c\u3001\u5373\u5bfe\u5fdc\u3067\u304d\u308b\u3082\u306e\u3092\u3057\u305f\u5f8c\u306e\u4e8b\u5f8c\u5bfe\u5fdc<\/p>\n<h2>\u7cfb\u7edf\u91c7\u7528\u4ec0\u4e48\u6837\u7684\u7ed3\u6784\uff1f<\/h2>\n<p>\u5728\u8fd9\u91cc\uff0c\u6211\u5c06\u8bf4\u660e\u4e00\u822c\u6d41\u5904\u7406\u7cfb\u7edf\u91c7\u7528\u7684\u7cfb\u7edf\u914d\u7f6e\u3002\u56e0\u4e3a\u5982\u679c\u4e0d\u5728\u8fd9\u91cc\u8bf4\u660e\uff0c\u540e\u9762\u7684\u60c5\u51b5\u53ef\u80fd\u65e0\u6cd5\u60f3\u8c61\u3002<\/p>\n<p>\u5728\u8fdb\u884c\u6d41\u5904\u7406\u7684\u7cfb\u7edf\u4e2d\uff0c\u5e38\u89c1\u7684\u914d\u7f6e\u5982\u4e0b\u6240\u793a\u3002<br \/>\n\u5173\u4e8e\u6bcf\u4e2a\u914d\u7f6e\u8981\u7d20\uff0c\u5c06\u5728\u201c\u5b9e\u9645\u7684\u6d41\u5904\u7406\u7cfb\u7edf\u662f\u4ec0\u4e48\u6837\u7684\uff1f\u201d\u4e2d\u8fdb\u884c\u518d\u6b21\u89e3\u91ca\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/30-0.png\" alt=\"StreamProcessingSystemArch.png\" \/><\/div>\n<p>\u6b63\u5982\u60a8\u6240\u89c1\uff0c\u6211\u4eec\u4f1a\u5c06\u6570\u636e\u4e34\u65f6\u7f13\u51b2\u5230\u6d88\u606f\u603b\u7ebf\u7684\u4f4d\u7f6e\uff0c\u5e76\u83b7\u53d6\u7f13\u51b2\u7684\u6570\u636e\u4ee5\u8fdb\u884c\u6d41\u5904\u7406\u3002<\/p>\n<h2>\u6279\u5904\u7406\u6709\u4ec0\u4e48\u4e0d\u540c\uff1f<\/h2>\n<p>\u6d41\u5f0f\u5904\u7406\u548c\u6279\u5904\u7406\u663e\u7136\u6709\u4e0d\u540c\u7684\u60c5\u51b5\uff0c\u9996\u5148\u8ba9\u6211\u4eec\u5217\u4e3e\u4e00\u4e9b\u5927\u6570\u636e\u7684\u4e3b\u8981\u5904\u7406\u6a21\u5f0f\u3002<\/p>\n<h3>\u5927\u6570\u636e\u7684\u5178\u578b\u5904\u7406\u6a21\u5f0f<\/h3>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/35-0.png\" alt=\"\u30d3\u30c3\u30b0\u30c7\u30fc\u30bf\u51e6\u7406\u4e00\u89a7.png\" \/><\/div>\n<h4>\u6279\u5904\u7406<\/h4>\n<p>\u628a\u5b58\u50a8\u5728\u6570\u636e\u5b58\u50a8\u4e2d\u7684\u6570\u636e\u8fdb\u884c\u6279\u91cf\u8f6c\u6362\u5e76\u8f93\u51fa\u7ed3\u679c\u7684\u5904\u7406\u6a21\u578b\u3002<br \/>\n\u7cfb\u7edf\u901a\u5e38\u91c7\u7528\u4ee5\u4e0b\u6784\u67b6\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/38-0.png\" alt=\"\u30d0\u30c3\u30c1\u51e6\u7406\u30e2\u30c7\u30eb.png\" \/><\/div>\n<h4>\u5bf9\u8bdd\u5f0f\u67e5\u8be2<\/h4>\n<p>\u6267\u884c\u67e5\u8be2\u5e76\u83b7\u53d6\u7ed3\u679c\u7684\u6a21\u578b\uff0c\u5bf9\u5b58\u50a8\u5728\u6570\u636e\u5b58\u50a8\u4e2d\u7684\u6570\u636e\u6267\u884c\u67e5\u8be2\u3002\u901a\u5e38\u91c7\u7528\u4ee5\u4e0b\u7cfb\u7edf\u914d\u7f6e\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/41-0.png\" alt=\"\u5bfe\u8a71\u578b\u30af\u30a8\u30ea\u30e2\u30c7\u30eb.png\" \/><\/div>\n<p>\u6839\u636e\u4e0a\u8ff0\u56fe\u6765\u770b\uff0c\u8fd9\u53ea\u662f\u6279\u5904\u7406\u7684\u4e00\u79cd\u53d8\u5f62\u3002\u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\uff0c&#8221;\u8f93\u51fa\u76ee\u6807&#8221;\u548c&#8221;\u6267\u884c\u6e90\u76f8\u540c&#8221;\u3002<\/p>\n<blockquote><p>\u5f53\u6211\u4eec\u770b\u5b9e\u9645\u7684\u6d41\u7a0b\u6a21\u578b\u65f6\uff0c\u6279\u5904\u7406\u548c\u6d41\u5904\u7406\u5728\u6267\u884c\u65f6\u95f4\u4e0a\u90e8\u7f72\u8fdb\u7a0b\uff0c\u800c\u4ea4\u4e92\u5f0f\u67e5\u8be2\u901a\u5e38\u4f1a\u5e38\u9a7b\u8fdb\u7a0b\u7b49\u65b9\u9762\u6709\u6240\u4e0d\u540c\uff0c\u6211\u4eec\u4e0d\u6df1\u5165\u8be6\u7ec6\u8ba8\u8bba\u8fd9\u4e9b\u5dee\u5f02\u7684\u7ec6\u8282\u3002\u4ece\u5bf9\u5df2\u5b58\u5728\u7684\u6570\u636e\u8fdb\u884c\u5206\u5e03\u5f0f\u5904\u7406\u5e76\u8f93\u51fa\uff08\u83b7\u53d6\uff09\u7ed3\u679c\u7684\u89d2\u5ea6\u6765\u770b\uff0c\u4e0e\u6279\u5904\u7406\u6ca1\u6709\u591a\u5927\u533a\u522b\u3002<\/p><\/blockquote>\n<p>\u7136\u800c\uff0c\u6700\u8fd1\u6d41\u884c\u7684\u201c\u6301\u7eed\u67e5\u8be2\u201d\u7684\u65b9\u5f0f\u5141\u8bb8\u4e8b\u5148\u8bbe\u5b9a\u67e5\u8be2\u6761\u4ef6\uff0c\u5e76\u5728\u5b9e\u65f6\u4e2d\u83b7\u53d6\u7b26\u5408\u6761\u4ef6\u7684\u6570\u636e\uff0c\u56e0\u6b64\u5c06\u6210\u4e3a\u6db5\u76d6\u6279\u5904\u7406\u548c\u6d41\u5904\u7406\u4e24\u79cd\u6a21\u578b\u7684\u7efc\u5408\u6a21\u578b\u3002<br \/>\n\u4e5f\u5c31\u662f\u8bf4\uff0c\u867d\u7136\u5728\u5178\u578b\u7684\u5904\u7406\u6a21\u5f0f\u4e2d\u53ea\u5c5e\u4e8e\u5176\u4e2d\u4e00\u9879\uff0c\u4f46\u5b9e\u9645\u4e0a\u53ef\u4ee5\u5305\u542b\u5728\u6279\u5904\u7406\u6216\u6d41\u5904\u7406\u4e2d\u7684\u4efb\u4f55\u4e00\u79cd\u3002<\/p>\n<p>\u56e0\u6b64\uff0c\u6211\u5011\u5c07\u5728\u5f8c\u7e8c\u7684\u89e3\u91cb\u4e2d\u7701\u7565\u6b64\u6a21\u578b\u3002<\/p>\n<h4>\u6d41\u5904\u7406<\/h4>\n<p>\u8bf7\u53c2\u8003\u4e4b\u524d\u7684\u8bf4\u660e\u3002\u5173\u4e8e\u7cfb\u7edf\u7684\u7ed3\u6784\uff0c\u8bf7\u53c2\u8003\u201c\u5b9e\u9645\u7684\u6d41\u5904\u7406\u7cfb\u7edf\u662f\u4ec0\u4e48\u6837\u7684\uff1f\u201d<\/p>\n<h3>\u6279\u5904\u7406\u548c\u6570\u636e\u6d41\u5904\u7406\u7684\u533a\u522b\u3002<\/h3>\n<p>\u5728\u6b64\u4e4b\u4e0a\uff0c\u6211\u4eec\u6765\u63d0\u53d6\u6279\u5904\u7406\u7684\u5047\u8bbe\u3002<br \/>\n\u6279\u5904\u7406\u662f\u5728\u4ee5\u4e0b\u57fa\u7840\u4e0a\u6784\u5efa\u7684\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u30d0\u30c3\u30c1\u51e6\u7406\u5b9f\u884c\u6642\u306b\u5168\u3066\u30c7\u30fc\u30bf\u304c\u63c3\u3063\u3066\u304a\u308a\u3001\u51e6\u7406\u5bfe\u8c61\u306e\u7bc4\u56f2\u304c\u660e\u78ba\u3067\u306a\u3044\u3068\u3044\u3051\u306a\u3044\u3002<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\u30d0\u30c3\u30c1\u3092\u307e\u305f\u3044\u3060\u7d50\u679c\u51fa\u529b\u306f\u57fa\u672c\u7684\u306b\u5bfe\u5fdc\u3057\u306a\u3044\u3002<\/ul>\n<p>\u6211\u8ba4\u4e3a\u7b2c\u4e00\u4e2a\u524d\u63d0\u662f\uff0c\u6279\u5904\u7406\u662f\u5c06\u539f\u59cb\u6570\u636e\u5b8c\u5168\u6574\u7406\u6210\u4ee5\u4e0b\u7684\u5f62\u5f0f\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/52-0.jpeg\" alt=\"Batch_01.JPG\" \/><\/div>\n<p>\u5982\u679c\u8981\u8f93\u51fa\u591a\u4e2a\u7ed3\u679c\uff0c\u5219\u9700\u8981\u6309\u7167\u4e0b\u9762\u7684\u65b9\u5f0f\u6267\u884c\u591a\u6b21\u6279\u5904\u7406\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/54-0.jpeg\" alt=\"Batch_02.JPG\" \/><\/div>\n<p>\u5982\u679c\u6309\u7167\u65f6\u95f4\u5206\u6bb5\u7ed3\u679c\u7684\u8bdd\uff0c\u6211\u4eec\u4f1a\u6839\u636e\u5305\u542b\u6240\u6709\u6570\u636e\u7684\u57fa\u7840\u4e0a\u6267\u884c\u6279\u5904\u7406\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/56-0.jpeg\" alt=\"Batch_03.JPG\" \/><\/div>\n<p>\u5982\u679c\u8981\u8f93\u51fa\u5173\u4e8e\u7b2c\u4e8c\u4e2a\u524d\u63d0\u7684\u56fe\u8868\uff08\u6307\u7684\u662f\u5728\u4e00\u5b9a\u65f6\u95f4\u6bb5\u4e2d\u8fde\u7eed\u8bbf\u95ee\u7684\u7528\u6237\u4f1a\u8bdd\uff09\uff0c\u53ef\u4ee5\u53c2\u8003\u4e0b\u9762\u7684\u56fe\u8868\u3002<br \/>\n\u5982\u679c\u5206\u522b\u8f93\u51fa\u661f\u671f\u4e8c\u548c\u661f\u671f\u4e09\u7684\u7ed3\u679c\uff0c\u90a3\u4e48\u8de8\u8d8a\u65e5\u671f\u7684\u4f1a\u8bdd\u5c06\u88ab\u5206\u5272\u3002<br \/>\n\u5982\u679c\u60f3\u8981\u5c06\u661f\u671f\u4e09\u7684\u7ed3\u679c\u4e0e\u661f\u671f\u4e8c\u8fde\u63a5\u8d77\u6765\uff0c\u53ef\u4ee5\u8bfb\u53d6\u661f\u671f\u4e8c\u7684\u7ed3\u679c\u5e76\u91cd\u65b0\u8f93\u51fa\u7ed3\u679c\uff0c\u4f46\u5b9e\u9645\u64cd\u4f5c\u8d77\u6765\u5e76\u4e0d\u73b0\u5b9e\uff0c\u56e0\u4e3a\u9700\u8981\u8bfb\u53d6\u6240\u6709\u8fc7\u53bb\u7684\u7ed3\u679c\u5e76\u8fdb\u884c\u8fde\u63a5\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/58-0.jpeg\" alt=\"Batch_04.JPG\" \/><\/div>\n<p>\u4ece\u8fd9\u91cc\u770b\uff0c\u53ef\u4ee5\u6ce8\u610f\u5230\u6279\u5904\u7406\u7684\u7b2c\u4e8c\u5f20\u56fe\u53ef\u4ee5\u6309\u7167\u4ee5\u4e0b\u65b9\u5f0f\u8fdb\u884c\u53d8\u5f62\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/60-0.png\" alt=\"BatchFiles.png\" \/><\/div>\n<p>\u8fd9\u5c31\u662f\u5c06\u65e0\u9650\u7684\u6d41\u6570\u636e\u6309\u7167\u56fa\u5b9a\u65f6\u95f4\u95f4\u9694\u5212\u5206\u5f00\u7684\u4e1c\u897f\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/62-0.png\" alt=\"BatchInputAsStream.png\" \/><\/div>\n<p>\u6362\u53e5\u8bdd\u8bf4\uff0c\u6279\u5904\u7406\u662f\u6d41\u5904\u7406\u6a21\u578b\u4e2d\u7684\u4e00\u79cd\u6709\u9650\u5904\u7406\u65b9\u5f0f\u3002<\/p>\n<p>\u7136\u800c\uff0c\u6d41\u5904\u7406\u4e0d\u53ef\u4ee5\u5e94\u7528\u4e8e\u4ee5\u4e0b\u793a\u4f8b\uff0c\u5b83\u9002\u7528\u4e8e\u6279\u91cf\u5904\u7406\u7684\u524d\u63d0\u6761\u4ef6\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u30d0\u30c3\u30c1\u51e6\u7406\u5b9f\u884c\u6642\u306b\u5168\u3066\u30c7\u30fc\u30bf\u304c\u63c3\u3063\u3066\u304a\u308a\u3001\u51e6\u7406\u5bfe\u8c61\u306e\u7bc4\u56f2\u304c\u660e\u78ba\u3067\u306a\u3044\u3068\u3044\u3051\u306a\u3044\u3002<\/ul>\n<\/li>\n<\/ul>\n<p>\uff1e\u3000\u7121\u9650\u306e\u30c7\u30fc\u30bf\u3067\u3042\u308a\u3001\u5e38\u6642\u30c7\u30fc\u30bf\u306f\u5230\u7740\u3057\u7d9a\u3051\u308b\u305f\u3081\u30c7\u30fc\u30bf\u304c\u63c3\u3046\u3068\u3044\u3046\u3053\u3068\u306f\u306a\u3044\u3002<\/p>\n<p>\u30d0\u30c3\u30c1\u3092\u307e\u305f\u3044\u3060\u7d50\u679c\u51fa\u529b\u306f\u57fa\u672c\u7684\u306b\u5bfe\u5fdc\u3057\u306a\u3044\u3002<\/p>\n<p>\uff1e\u3000\u305d\u3082\u305d\u3082\u7121\u9650\u306e\u30c7\u30fc\u30bf\u3092\u51e6\u7406\u3064\u3065\u3051\u308b\u3082\u306e\u306e\u305f\u3081\u3001\u524d\u63d0\u81ea\u4f53\u304c\u9055\u3046\u3002<\/p>\n<p>\u5982\u679c\u5728\u6267\u884c\u65f6\u6570\u636e\u90fd\u5df2\u7ecf\u5b58\u5728\uff08\u5b58\u5728\u6309\u65f6\u95f4\u5206\u9694\u7684\u6570\u636e\uff09\uff0c\u90a3\u4e48\u4e0e\u6279\u5904\u7406\u76f8\u540c\u7684\u64cd\u4f5c\u662f\u53ef\u4ee5\u6267\u884c\u7684\u3002<br \/>\n\u5728\u8fd9\u65b9\u9762\uff0c\u53ef\u4ee5\u4f53\u4f1a\u5230\u6279\u5904\u7406\u53ea\u662f\u6d41\u5904\u7406\u4e2d\u7684\u4e00\u4e2a\u6709\u9650\u5904\u7406\u6a21\u578b\u3002<br \/>\n\uff08\u203b\u7136\u800c\uff0c\u8fd9\u53ea\u662f\u5728\u529f\u80fd\u4e0a\u5982\u6b64\uff0c\u4eba\u7c7b\u7406\u89e3\u6613\u5ea6\u3001\u7ba1\u7406\u6613\u5ea6\u548c\u6027\u80fd\u6548\u7387\u7b49\u81ea\u7136\u9700\u8981\u5206\u522b\u8fdb\u884c\u6bd4\u8f83\u3002\uff09<\/p>\n<h2>\u5728\u6279\u5904\u7406\u4e2d\u6ca1\u6709\u7684\u65b0\u95ee\u9898\u662f\u4ec0\u4e48\uff1f<\/h2>\n<p>\u5728\u8fdb\u884c\u6d41\u5904\u7406\u5904\u7406\u65e0\u9650\u6570\u636e\u7684\u60c5\u51b5\u4e0b\uff0c\u4f1a\u51fa\u73b0\u65b0\u7684\u56f0\u6270\u3002<br \/>\n\u8fd9\u5c31\u662f\u6240\u8c13\u7684\u201c\u6570\u636e\u4e0d\u6309\u7167\u53d1\u751f\u987a\u5e8f\u5230\u8fbe\u7cfb\u7edf\u201d\uff08Out of order\uff09\u3002<\/p>\n<p>\u5982\u679c\u6267\u884c\u4ece\u667a\u80fd\u624b\u673a\u6536\u96c6\u6570\u636e\u7684\u6d41\u6570\u636e\u5904\u7406\u65f6\uff0c\u5982\u679c\u4fe1\u53f7\u65ad\u5f00\uff0c\u7136\u540e\u4e00\u6bb5\u65f6\u95f4\u540e\u4fe1\u53f7\u6062\u590d\uff0c\u90a3\u4e48\u4f1a\u6709\u201c\u4fe1\u53f7\u4e2d\u65ad\u65f6\u523b\u7684\u6570\u636e\u201d\u7a0d\u540e\u5230\u8fbe\u3002<\/p>\n<p>\u5173\u4e8e\u201c\u4e71\u5e8f\u201d\u9879\uff0c\u53ef\u4ee5\u8003\u8651\u5982\u4e0b\u5185\u5bb9\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u5927\u304d\u306a\u305a\u308c\u304c\u767a\u751f\u3059\u308b\u8981\u7d20<\/ul>\n<\/li>\n<\/ul>\n<p>\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u5207\u65ad<br \/>\n\u4e0a\u8a18\u306e\u30b9\u30de\u30fc\u30c8\u30d5\u30a9\u30f3\u306e\u4e8b\u4f8b\u3082\u305d\u3046\u3067\u3059\u306d\u3002<br \/>\n\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u9045\u5ef6<\/p>\n<p>\u5c0f\u3055\u306a\u305a\u308c\u304c\u767a\u751f\u3059\u308b\u8981\u7d20<\/p>\n<p>\u5206\u6563\u30b7\u30b9\u30c6\u30e0\u3067\u3042\u308b\u304c\u3086\u3048\u306b\u767a\u751f\u3059\u308b\u9045\u5ef6<br \/>\n\u3053\u308c\u306f\u8907\u6570\u306e\u30de\u30b7\u30f3\u3067\u69cb\u6210\u3055\u308c\u308b\u4ee5\u4e0a\u5fc5\u305a\u767a\u751f\u3059\u308b\u8981\u7d20\u3067\u3059\u3002<br \/>\n\u30de\u30b7\u30f3\u9593\u306e\u6642\u523b\u306e\u305a\u308c<\/p>\n<p>\u7531\u4e8e\u4e0a\u8ff0\u60c5\u51b5\u5b58\u5728\uff0c\u6d41\u5904\u7406\u4e2d\u5b58\u5728\u4ee5\u4e0b\u591a\u4e2a\u65f6\u95f4\u6982\u5ff5\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u30a4\u30d9\u30f3\u30c8\u6642\u523b(EventTime)<\/ul>\n<\/li>\n<\/ul>\n<p>\u5b9f\u969b\u306b\u30c7\u30fc\u30bf\u3092\u751f\u6210\u3059\u308b\u3053\u3068\u3068\u306a\u3063\u305f\u30a4\u30d9\u30f3\u30c8\u304c\u767a\u751f\u3057\u305f\u6642\u523b<\/p>\n<p>\u5230\u7740\u6642\u523b\uff08IngestTime\uff09<\/p>\n<p>\u30b7\u30b9\u30c6\u30e0\u306b\u30c7\u30fc\u30bf\u304c\u5230\u7740\u3057\u305f\u6642\u523b<\/p>\n<p>\u51e6\u7406\u6642\u523b(ProcessingTime)<\/p>\n<p>\u5b9f\u969b\u306b\u30b9\u30c8\u30ea\u30fc\u30e0\u30c7\u30fc\u30bf\u3092\u51e6\u7406\u3057\u305f\u6642\u523b<\/p>\n<p>\u7136\u800c\uff0c\u672c\u8d28\u4e0a\u91cd\u8981\u7684\u662f\u5b58\u5728\u591a\u4e2a\u65f6\u95f4\u70b9\u548c\u65e0\u5e8f\u6027\u3002\u56e0\u6b64\uff0c\u6211\u4eec\u5c06\u5728\u4e4b\u540e\u7684\u8ba8\u8bba\u4e2d\u4ec5\u8003\u8651\u4e8b\u4ef6\u65f6\u523b\uff08EventTime\uff09\u548c\u5904\u7406\u65f6\u523b\uff08ProcessingTime\uff09\u3002<\/p>\n<h2>\u4f60\u4e3a\u4ec0\u4e48\u4f1a\u56e0\u6b64\u800c\u82e6\u607c\u5462\uff1f<\/h2>\n<p>\u5373\u4f7f\u6570\u636e\u5230\u8fbe\u7684\u65f6\u95f4\u5ef6\u8fdf\uff0c\u53ea\u8981\u6570\u636e\u4e4b\u95f4\u6ca1\u6709\u5173\u8054\uff0c\u5c31\u53ef\u4ee5\u7406\u89e3\u4e3a\u201c\u7ed3\u679c\u662f\u8fd1\u4f3c\u503c\u201d\uff0c\u5728\u6784\u5efa\u7cfb\u7edf\u65f6\u7279\u522b\u56f0\u6270\u7684\u95ee\u9898\u5e76\u4e0d\u5b58\u5728\u3002<br \/>\n\u53ea\u8981\u6570\u636e\u5230\u8fbe\u7cfb\u7edf\uff0c\u5c31\u53ef\u4ee5\u7ee7\u7eed\u5904\u7406\u3002<br \/>\n\uff08\u203b\u5224\u65ad\u4f55\u65f6\u5168\u90e8\u5230\u8fbe\u7684\u65f6\u673a\u662f\u53e6\u5916\u4e00\u4e2a\u95ee\u9898\u3002\uff09<\/p>\n<p>\u7136\u800c\uff0c\u6b63\u5982\u521a\u624d\u5728\u6279\u5904\u7406\u5904\u7406\u6a21\u578b\u7684\u56fe\u8868\u4e2d\u6240\u793a\uff0c\u6d41\u5f0f\u5904\u7406\u4e2d\u53ef\u80fd\u9700\u8981\u4e00\u79cd\u79f0\u4e3a\u7a97\u53e3\u7684\u6982\u5ff5\u3002\u4f8b\u5982\uff0c\u5bf9\u4e8e\u5f02\u5e38\u68c0\u6d4b\uff0c\u4ec5\u901a\u8fc7\u4e00\u6b21\u8bbf\u95ee\u5f88\u96be\u68c0\u6d4b\u5230\u5f02\u5e38\u60c5\u51b5\uff0c\u56e0\u6b64\u4e3b\u8981\u662f\u901a\u8fc7\u4e0e\u524d\u540e\u6570\u636e\u7684\u76f8\u5173\u6027\u6765\u68c0\u6d4b\u5f02\u5e38\uff0c\u6bd4\u5982&#8221;\u5728\u77ed\u65f6\u95f4\u5185\u53d1\u751f\u4e86\u5927\u91cf\u767b\u5f55\u5931\u8d25&#8221;\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u56fa\u5b9a\u9577\u30a6\u30a3\u30f3\u30c9\u30a6(Tubling Window)<\/ul>\n<\/li>\n<\/ul>\n<p>\uff11\u6642\u9593\u3054\u3068\u3001\u7b49\u306e\u4e00\u5b9a\u306e\u6642\u9593\u3054\u3068\u306b\u533a\u5207\u3063\u305f\u7bc4\u56f2\u306e\u30a6\u30a3\u30f3\u30c9\u30a6<\/p>\n<p>\u30b9\u30e9\u30a4\u30c7\u30a3\u30f3\u30b0\u30a6\u30a3\u30f3\u30c9\u30a6(Sliding Window)<\/p>\n<p>\u6bce\u5206\u3001\u904e\u53bb\uff15\u5206\u9593\u5206\u306e\u7d50\u679c\u3092\u96c6\u8a08\u3057\u3066\u51fa\u529b\u3059\u308b\u3068\u3044\u3063\u305f\u7bc4\u56f2\u304c\u79fb\u52d5\u3059\u308b\u30a6\u30a3\u30f3\u30c9\u30a6<\/p>\n<p>\u30bb\u30c3\u30b7\u30e7\u30f3\u30a6\u30a3\u30f3\u30c9\u30a6(Session Window)<\/p>\n<p>\u4e00\u5b9a\u6642\u9593\u4ee5\u5185\u306b\u30a2\u30af\u30bb\u30b9\u304c\u9023\u7d9a\u3057\u305f\u5834\u5408\u306b\u305d\u306e\u30a2\u30af\u30bb\u30b9\u3092\u7d10\u3065\u3051\u308b\u3068\u3044\u3046\u9577\u3055\u304c\u56fa\u5b9a\u3055\u308c\u306a\u3044\u30a6\u30a3\u30f3\u30c9\u30a6<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/79-0.png\" alt=\"WindowVariation.png\" \/><\/div>\n<p>\u5728\u8fd9\u91cc\uff0c&#8221;\u6570\u636e\u4e0d\u662f\u6309\u7167\u53d1\u751f\u987a\u5e8f\u5230\u8fbe\u7cfb\u7edf&#8221;\u6210\u4e3a\u4e00\u4e2a\u91cd\u5927\u95ee\u9898\u3002\u56e0\u4e3a\uff0c\u5982\u679c\u5728\u8f93\u51fa\u4e86&#8221;17:00\uff5e17:59\u671f\u95f4\u53d1\u751f\u7684\u6570\u636e\u7684\u56fa\u5b9a\u957f\u5ea6\u7a97\u53e3\u805a\u5408\u7ed3\u679c&#8221;\u4e4b\u540e\uff0c&#8221;17:10\u53d1\u751f\u7684\u6570\u636e&#8221;\u5ef6\u8fdf\u5230\u8fbe\uff0c\u90a3\u4e48\u5c31\u4f1a\u51fa\u73b0\u95ee\u9898\u3002<\/p>\n<h2>\u5982\u4f55\u624d\u80fd\u89e3\u51b3\u65b0\u51fa\u73b0\u7684\u95ee\u9898\uff1f<\/h2>\n<p>\u5728\u6570\u636e\u4e0d\u662f\u6309\u7167\u4ea7\u751f\u987a\u5e8f\u5230\u8fbe\u7cfb\u7edf\u7684\u60c5\u51b5\u4e0b\uff0c\u901a\u8fc7\u7ec4\u5408\u4e09\u4e2a\u5143\u7d20\u6765\u66f4\u5bb9\u6613\u5e94\u5bf9\u8f93\u51fa\u7a97\u53e3\u7684\u7ed3\u679c\u3002<\/p>\n<h4>\u5904\u7406\u4e8b\u4ef6\u7684\u622a\u6b62\u65f6\u95f4\u70b9\uff08\u6c34\u5370\uff09<\/h4>\n<p>\u6c34\u5370\u662f\u4e00\u4e2a\u6982\u5ff5\uff0c\u7528\u4e8e\u8868\u793a\u5728\u4e8b\u4ef6\u65f6\u95f4\u57fa\u7840\u4e0a\u5904\u7406\u7684\u8fdb\u5ea6\u5230\u8fbe\u4e86\u591a\u8fdc\u3002<br \/>\n\u5982\u679c\u4e8b\u4ef6\u65f6\u95f4\u548c\u5904\u7406\u65f6\u95f4\u662f\u5206\u5f00\u5b58\u5728\u7684\uff0c\u5219\u5b9e\u9645\u5904\u7406\u65f6\u95f4\u548c\u4e8b\u4ef6\u65f6\u95f4\u7684\u5904\u7406\u8fdb\u5ea6\u4f1a\u6709\u6240\u504f\u5dee\uff0c\u5982\u4e0b\u56fe\u6240\u793a\u3002<br \/>\n\u56e0\u6b64\uff0c\u9700\u8981\u5f15\u5165\u6c34\u5370\u6982\u5ff5\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/85-0.png\" alt=\"Watermark.png\" \/><\/div>\n<p>\u4e3e\u4f8b\u6765\u8bf4\uff0c\u5f53Watermark\u53d6\u5f97X\u7684\u503c\u65f6\uff0c\u5b83\u8868\u793a\u201c\u65e9\u4e8eX\u7684\u4e8b\u4ef6\u65f6\u95f4\u7684\u6570\u636e\u5df2\u7ecf\u5168\u90e8\u5904\u7406\u5b8c\u6bd5\u201d\u3002\u6362\u53e5\u8bdd\u8bf4\uff0cWatermark\u5728\u5904\u7406\u65e0\u9650\u6570\u636e\u7684\u8fc7\u7a0b\u4e2d\uff0c\u4f5c\u4e3a\u4e00\u4e2a\u201c\u5206\u9694\u70b9\u201d\u6765\u6307\u793a\u5904\u7406\u7684\u8fdb\u5c55\u3002<br \/>\n\u4f46\u9700\u8981\u6ce8\u610f\u7684\u662f\uff0cWatermark\u6c38\u8fdc\u4e0d\u53ef\u80fd\u662f\u5b8c\u5168\u51c6\u786e\u7684\u3002\u56e0\u4e3a\u6570\u636e\u53ef\u80fd\u4f1a\u5ef6\u8fdf\u5230\u8fbe\uff0c\u6240\u4ee5Watermark\u53ea\u662f\u4e00\u4e2a\u201c\u5927\u81f4\u5ef6\u8fdf\u5230\u8fd9\u4e2a\u7a0b\u5ea6\u201d\u7684\u4f30\u7b97\u3002<br \/>\n\uff08\u203b\u4e5f\u8bb8\u4f1a\u6709\u4eba\u8d28\u7591\u4e3a\u4ec0\u4e48\u9700\u8981\u8fd9\u6837\u7684\u4f30\u7b97\uff0c\u4f46\u7531\u4e8e\u53ea\u6709\u8fd9\u6837\u624d\u80fd\u8fdb\u884c\u5904\u7406\uff0c\u6240\u4ee5\u5fc5\u987b\u8fdb\u884c\u4f30\u7b97\u3002\uff09<\/p>\n<h4>\u5b9a\u4e49\u4e00\u4e2a\u673a\u5236\uff08\u89e6\u53d1\u5668\uff09\uff0c\u7528\u4e8e\u786e\u5b9a\u5728\u4f55\u65f6\u8f93\u51fa\u7edf\u8ba1\u7ed3\u679c\u3002<\/h4>\n<p>Trigger\u673a\u5236\u662f\u6839\u636e\u60c5\u51b5\u6765\u5b9a\u4e49\u4f55\u65f6\u8f93\u51fa\u7a97\u53e3\u805a\u5408\u7ed3\u679c\u7684\u673a\u5236\u3002<br \/>\n\u901a\u8fc7\u5b9a\u4e49Trigger\uff0c\u53ef\u4ee5\u7075\u6d3b\u5730\u786e\u5b9a\u4f55\u65f6\u5e94\u8be5\u8f93\u51fa\u7a97\u53e3\u805a\u5408\u7ed3\u679c\u3002<br \/>\n\u6b64\u5916\uff0cTrigger\u673a\u5236\u8fd8\u53ef\u4ee5\u5b9e\u73b0\u5728\u6570\u636e\u66f4\u65b0\u65f6\u591a\u6b21\u8f93\u51fa\u7a97\u53e3\u805a\u5408\u7ed3\u679c\u3002<br \/>\n\u901a\u8fc7\u8fd9\u4e2a\u673a\u5236\uff0c\u4e00\u65e6Watermark\u5230\u8fbe\u56fa\u5b9a\u4f4d\u7f6e\u6216\u8005\u665a\u4e8eWatermark\u7684\u6570\u636e\u5230\u8fbe\u65f6\uff0c\u53ef\u4ee5\u53ca\u65f6\u8fdb\u884c\u6295\u673a\u5f0f\u7684\u8f93\u51fa\u3002<\/p>\n<p>\u4e3e\u4e2a\u4f8b\u5b50\uff0c\u5728Clouddataflow\u4e2d\uff0c\u5982\u679cWatermark\u8fbe\u5230\u4e86\u56fa\u5b9a\u957f\u5ea6\u7a97\u53e3\u7684\u6700\u540e\u4f4d\u7f6e\uff0c\u5c31\u53ef\u4ee5\u7528\u4e0b\u9762\u7684\u4ee3\u7801\u6765\u8f93\u51fa\u805a\u5408\u7ed3\u679c\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"nc\">PCollection<\/span><span class=\"o\">&lt;<\/span><span class=\"no\">KV<\/span><span class=\"o\">&lt;<\/span><span class=\"nc\">String<\/span><span class=\"o\">,<\/span> <span class=\"nc\">Integer<\/span><span class=\"o\">&gt;&gt;<\/span> <span class=\"n\">scores<\/span> <span class=\"o\">=<\/span> <span class=\"n\">input<\/span>\r\n  <span class=\"o\">.<\/span><span class=\"na\">apply<\/span><span class=\"o\">(<\/span><span class=\"nc\">Window<\/span><span class=\"o\">.<\/span><span class=\"na\">into<\/span><span class=\"o\">(<\/span><span class=\"nc\">FixedWindows<\/span><span class=\"o\">.<\/span><span class=\"na\">of<\/span><span class=\"o\">(<\/span><span class=\"nc\">Duration<\/span><span class=\"o\">.<\/span><span class=\"na\">standardMinutes<\/span><span class=\"o\">(<\/span><span class=\"mi\">2<\/span><span class=\"o\">)))<\/span>  <span class=\"c1\">\/\/ \u56fa\u5b9a\u9577\u30a6\u30a3\u30f3\u30c9\u30a6\u3092\u5ba3\u8a00<\/span>\r\n               <span class=\"o\">.<\/span><span class=\"na\">triggering<\/span><span class=\"o\">(<\/span><span class=\"nc\">AtWatermark<\/span><span class=\"o\">()))<\/span>                          <span class=\"c1\">\/\/ Watermark\u304c\u5230\u7740\u3057\u305f\u30bf\u30a4\u30df\u30f3\u30b0\u3067\u51fa\u529b<\/span>\r\n  <span class=\"o\">.<\/span><span class=\"na\">apply<\/span><span class=\"o\">(<\/span><span class=\"nc\">Sum<\/span><span class=\"o\">.<\/span><span class=\"na\">integersPerKey<\/span><span class=\"o\">());<\/span>\r\n<\/code><\/pre>\n<p>\u9664\u4e86\u4e0a\u8ff0\u7684\u60c5\u51b5\u4e4b\u5916\uff0c\u5982\u679c\u5ef6\u8fdf\u7684\u6570\u636e\u5230\u8fbe\u65f6\u9700\u8981\u91cd\u65b0\u8f93\u51fa\u7ed3\u679c\uff0c\u53ef\u4ee5\u6309\u7167\u4ee5\u4e0b\u65b9\u5f0f\u8fdb\u884c\u63cf\u8ff0\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"nc\">PCollection<\/span><span class=\"o\">&lt;<\/span><span class=\"no\">KV<\/span><span class=\"o\">&lt;<\/span><span class=\"nc\">String<\/span><span class=\"o\">,<\/span> <span class=\"nc\">Integer<\/span><span class=\"o\">&gt;&gt;<\/span> <span class=\"n\">scores<\/span> <span class=\"o\">=<\/span> <span class=\"n\">input<\/span>\r\n  <span class=\"o\">.<\/span><span class=\"na\">apply<\/span><span class=\"o\">(<\/span><span class=\"nc\">Window<\/span><span class=\"o\">.<\/span><span class=\"na\">into<\/span><span class=\"o\">(<\/span><span class=\"nc\">FixedWindows<\/span><span class=\"o\">.<\/span><span class=\"na\">of<\/span><span class=\"o\">(<\/span><span class=\"nc\">Duration<\/span><span class=\"o\">.<\/span><span class=\"na\">standardMinutes<\/span><span class=\"o\">(<\/span><span class=\"mi\">2<\/span><span class=\"o\">)))<\/span>  <span class=\"c1\">\/\/ \u56fa\u5b9a\u9577\u30a6\u30a3\u30f3\u30c9\u30a6\u3092\u5ba3\u8a00<\/span>\r\n               <span class=\"o\">.<\/span><span class=\"na\">triggering<\/span><span class=\"o\">(<\/span>\r\n                 <span class=\"nc\">AtWatermark<\/span><span class=\"o\">())<\/span>                                     <span class=\"c1\">\/\/ Watermark\u304c\u5230\u7740\u3057\u305f\u30bf\u30a4\u30df\u30f3\u30b0\u3067\u51fa\u529b<\/span>\r\n                   <span class=\"o\">.<\/span><span class=\"na\">withLateFirings<\/span><span class=\"o\">(<\/span><span class=\"nc\">AtCount<\/span><span class=\"o\">(<\/span><span class=\"mi\">1<\/span><span class=\"o\">))))<\/span>                   <span class=\"c1\">\/\/ \u9045\u308c\u30c7\u30fc\u30bf\u304c1\u4ef6\u5230\u7740\u3059\u308b\u3054\u3068\u306b\u51fa\u529b<\/span>\r\n  <span class=\"o\">.<\/span><span class=\"na\">apply<\/span><span class=\"o\">(<\/span><span class=\"nc\">Sum<\/span><span class=\"o\">.<\/span><span class=\"na\">integersPerKey<\/span><span class=\"o\">());<\/span>\r\n<\/code><\/pre>\n<p>\u6b64\u5916\uff0c\u5f53\u5ef6\u8fdf\u6570\u636e\u5230\u8fbe\u65f6\uff0c\u6211\u4eec\u5c06\u4fdd\u7559\u7a97\u53e3\u7684\u65f6\u95f4\u9650\u5236\u8bbe\u5b9a\u4e3a\u6700\u591a5\u5206\u949f\u7684\u5ef6\u8fdf\u3002\u5982\u679c\u6570\u636e\u5728\u6b64\u4e4b\u540e\u5230\u8fbe\uff0c\u5219\u53ef\u4ee5\u6309\u4ee5\u4e0b\u63cf\u8ff0\u8fdb\u884c\u5220\u9664\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"nc\">PCollection<\/span><span class=\"o\">&lt;<\/span><span class=\"no\">KV<\/span><span class=\"o\">&lt;<\/span><span class=\"nc\">String<\/span><span class=\"o\">,<\/span> <span class=\"nc\">Integer<\/span><span class=\"o\">&gt;&gt;<\/span> <span class=\"n\">scores<\/span> <span class=\"o\">=<\/span> <span class=\"n\">input<\/span>\r\n  <span class=\"o\">.<\/span><span class=\"na\">apply<\/span><span class=\"o\">(<\/span><span class=\"nc\">Window<\/span><span class=\"o\">.<\/span><span class=\"na\">into<\/span><span class=\"o\">(<\/span><span class=\"nc\">FixedWindows<\/span><span class=\"o\">.<\/span><span class=\"na\">of<\/span><span class=\"o\">(<\/span><span class=\"nc\">Duration<\/span><span class=\"o\">.<\/span><span class=\"na\">standardMinutes<\/span><span class=\"o\">(<\/span><span class=\"mi\">2<\/span><span class=\"o\">)))<\/span>  <span class=\"c1\">\/\/ \u56fa\u5b9a\u9577\u30a6\u30a3\u30f3\u30c9\u30a6\u3092\u5ba3\u8a00<\/span>\r\n               <span class=\"o\">.<\/span><span class=\"na\">triggering<\/span><span class=\"o\">(<\/span>\r\n                 <span class=\"nc\">AtWatermark<\/span><span class=\"o\">())<\/span>                                     <span class=\"c1\">\/\/ Watermark\u304c\u5230\u7740\u3057\u305f\u30bf\u30a4\u30df\u30f3\u30b0\u3067\u51fa\u529b<\/span>\r\n                   <span class=\"o\">.<\/span><span class=\"na\">withLateFirings<\/span><span class=\"o\">(<\/span><span class=\"nc\">AtCount<\/span><span class=\"o\">(<\/span><span class=\"mi\">1<\/span><span class=\"o\">)))<\/span>                    <span class=\"c1\">\/\/ \u9045\u308c\u30c7\u30fc\u30bf\u304c1\u4ef6\u5230\u7740\u3059\u308b\u3054\u3068\u306b\u51fa\u529b<\/span>\r\n               <span class=\"o\">.<\/span><span class=\"na\">withAllowedLateness<\/span><span class=\"o\">(<\/span><span class=\"nc\">Duration<\/span><span class=\"o\">.<\/span><span class=\"na\">standardMinutes<\/span><span class=\"o\">(<\/span><span class=\"mi\">5<\/span><span class=\"o\">)))<\/span>   <span class=\"c1\">\/\/ \u9045\u308c\u306e\u8a31\u5bb9\u30925\u5206\u9593\u307e\u3067<\/span>\r\n  <span class=\"o\">.<\/span><span class=\"na\">apply<\/span><span class=\"o\">(<\/span><span class=\"nc\">Sum<\/span><span class=\"o\">.<\/span><span class=\"na\">integersPerKey<\/span><span class=\"o\">());<\/span>\r\n<\/code><\/pre>\n<p>\u7b80\u800c\u8a00\u4e4b\uff0c\u6211\u4eec\u53ef\u4ee5\u901a\u8fc7\u89e6\u53d1\u5668\u7684\u673a\u5236\u6765\u63a7\u5236\u4e0b\u5217\u60c5\u51b5\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u96c6\u8a08\u7d50\u679c\u306e\u51fa\u529b\u30bf\u30a4\u30df\u30f3\u30b0<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">Out of order\u306a\u30c7\u30fc\u30bf\u3078\u306e\u5bfe\u5fdc<\/ul>\n<\/li>\n<\/ul>\n<p>\u51fa\u529b\u3092\u884c\u3046\u30bf\u30a4\u30df\u30f3\u30b0<br \/>\nEventTime\u3068ProcessingTime\u306e\u305a\u308c\u306e\u8a31\u5bb9\u671f\u9593<\/p>\n<h4>\u5728\u8f93\u51fa\u7edf\u8ba1\u7ed3\u679c\u65f6\u4f7f\u7528\u7684\u7d2f\u79ef\u8ba1\u7b97\u65b9\u6cd5\uff08\u7d2f\u79ef\uff09<\/h4>\n<p>Accumulation\u5b9a\u4e49\u4e86\u5728\u540c\u4e00\u4e2a\u7a97\u53e3\u4e2d\u591a\u6b21\u8f93\u51fa\u805a\u5408\u7ed3\u679c\u65f6\u7684\u5173\u7cfb\u548c\u64cd\u4f5c\u3002<br \/>\n\u8fd9\u53d6\u51b3\u4e8e\u805a\u5408\u7ed3\u679c\u7684\u8f93\u51fa\u7cfb\u7edf\uff0c\u4e5f\u5c31\u662f\u6839\u636e\u4f7f\u7528\u65b9\u5f0f\u800c\u5b9a\u3002<\/p>\n<p>\u5728\u8f93\u51fa\u7cfb\u7edf\u5c06\u4e4b\u524d\u7684\u6c47\u603b\u7ed3\u679c\u76f8\u52a0\u7684\u60c5\u51b5\u4e0b\uff0c\u5982\u679c\u5728\u4e00\u6b21\u7a97\u53e3\u4e2d\u8f93\u51fa\u6c47\u603b\u7ed3\u679c\uff0c\u5219\u8be5\u503c\u5c06\u88ab\u4e22\u5f03\uff0c\u4e0b\u4e00\u6b21\u8f93\u51fa\u5e94\u8be5\u662f\u65b0\u6570\u636e\u7684\u6c47\u603b\u7ed3\u679c\u3002<br \/>\n\u76f8\u53cd\u5730\uff0c\u5982\u679c\u8f93\u51fa\u7cfb\u7edf\u662f\u57fa\u4e8e\u7528\u6237ID\u548c\u65f6\u95f4\u7684Key-Value\u5b58\u50a8\u7b49\u6570\u636e\u7ed3\u6784\uff0c\u5219\u5373\u4f7f\u5728\u4e00\u6b21\u7a97\u53e3\u4e2d\u8f93\u51fa\u6c47\u603b\u7ed3\u679c\u4e4b\u540e\uff0c\u4ecd\u9700\u8981\u4fdd\u7559\u8be5\u503c\uff0c\u4e0b\u4e00\u6b21\u8f93\u51fa\u9700\u8981\u5c06\u73b0\u6709\u6570\u636e\u548c\u65b0\u6570\u636e\u7684\u6c47\u603b\u7ed3\u679c\u5408\u5e76\u8f93\u51fa\u3002<\/p>\n<p>Accumulation \u662f\u4e00\u79cd\u673a\u5236\uff0c\u7528\u4e8e\u5207\u6362\u5173\u4e8e\u5904\u7406\u7c7b\u4f3c\u4e8e\u4e0a\u8ff0\u7edf\u8ba1\u7ed3\u679c\u8f93\u51fa\u65f6\u7684\u503c\u7684\u65b9\u5f0f\u3002<\/p>\n<p>\u79ef\u7d2f\u6a21\u5f0f\u4e3b\u8981\u5b58\u5728\u4ee5\u4e0b\u4e09\u79cd\u65b9\u5f0f\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u7834\u68c4\u30e2\u30fc\u30c9(Discarding)<\/ul>\n<\/li>\n<\/ul>\n<p>\u30a6\u30a3\u30f3\u30c9\u30a6\u306e\u96c6\u8a08\u7d50\u679c\u3092\u51fa\u529b\u3057\u305f\u30bf\u30a4\u30df\u30f3\u30b0\u3067\u96c6\u8a08\u7d50\u679c\u306f\u7834\u68c4\u3055\u308c\u308b\u30e2\u30fc\u30c9<br \/>\n\u3064\u307e\u308a\u3001\u30a6\u30a3\u30f3\u30c9\u30a6\u306e\u96c6\u8a08\u7d50\u679c\u3092\u4e00\u5ea6\u51fa\u529b\u3057\u305f\u5834\u5408\u3001\u72b6\u614b\u306f\u30af\u30ea\u30a2\u3055\u308c\u3001\u6b21\u306e\u96c6\u8a08\u51fa\u529b\u306b\u304a\u3044\u3066\u306f\u3001\u51fa\u529b\u5f8c\u306b\u5230\u7740\u3057\u305f\u30c7\u30fc\u30bf\u306e\u307f\u306e\u96c6\u8a08\u7d50\u679c\u3092\u51fa\u529b\u3059\u308b\u52d5\u4f5c\u3068\u306a\u308a\u307e\u3059\u3002<br \/>\n\u3053\u306e\u30e2\u30fc\u30c9\u306f\u4e0b\u6d41\u306e\u30b7\u30b9\u30c6\u30e0\u304c\u81ea\u524d\u3067\u4f55\u304b\u3057\u3089\u306e\u7d2f\u7a4d\u8a08\u7b97\u306e\u6a5f\u69cb\u3092\u4fdd\u6301\u3057\u3066\u3044\u308b\u5834\u5408\u306b\u6709\u7528\u3067\u3059\u3002<\/p>\n<p>\u4f8b\u3048\u3070\u3001\u96c6\u8a08\u5024\u3068\u3057\u3066\u524d\u56de\u3068\u306e\u5dee\u5206\u3092\u9001\u4fe1\u3057\u3001\u5916\u90e8\u30b7\u30b9\u30c6\u30e0\u306f\u305d\u306e\u5024\u3092\u5408\u8a08\u3057\u3066\u6700\u7d42\u7d50\u679c\u3068\u3059\u308b\u306a\u3069\u3002<\/p>\n<p>\u7d2f\u7b97\u30e2\u30fc\u30c9(Accumulating)<\/p>\n<p>\u30a6\u30a3\u30f3\u30c9\u30a6\u306e\u96c6\u8a08\u7d50\u679c\u3092\u4fdd\u6301\u3057\u7d9a\u3051\u3001\u524d\u306e\u5024\u306b\u5bfe\u3057\u3066\u305d\u306e\u5f8c\u306e\u5024\u3092\u7d2f\u7b97\u3055\u305b\u3066\u5024\u3092\u51fa\u529b\u3059\u308b\u30e2\u30fc\u30c9<br \/>\n\u3053\u306e\u30e2\u30fc\u30c9\u306f\u65b0\u3057\u3044\u7d50\u679c\u3067\u53e4\u3044\u7d50\u679c\u3092\u4e0a\u66f8\u304d\u3059\u308c\u3070\u3044\u3044\u5834\u5408\u306b\u6709\u7528\u3067\u3059\u3002<\/p>\n<p>\u4f8b\u3048\u3070\u3001BigTable\u306e\u3088\u3046\u306aKVS\u306b\u7d50\u679c\u3092\u51fa\u529b\u3059\u308b\u30b1\u30fc\u30b9\u306a\u3069\u3002<\/p>\n<p>\u7d2f\u7b97\uff06\u5f8c\u9000\u30e2\u30fc\u30c9(Accumulating &amp; Retracting)<\/p>\n<p>\u7d2f\u7b97\u306e\u30e2\u30fc\u30c9\u306b\u4f3c\u3044\u307e\u3059\u304c\u3001\u65b0\u305f\u306a\u96c6\u8a08\u7d50\u679c\u3092\u51fa\u529b\u3059\u308b\u6642\u306b\u306f\u524d\u56de\u306e\u5024\u3092\u57fa\u306b\u3057\u305f\u76f8\u6bba\u3059\u308b\u305f\u3081\u306e\u5dee\u5206\u5024\uff08\u5f8c\u9000\u5024\uff09\u3092\u51fa\u529b\u3059\u308b\u30e2\u30fc\u30c9<br \/>\n\u5f8c\u9000\uff08\u524d\u306e\u5024\u3068\u4eca\u56de\u306e\u5024\u3092\u57fa\u306b\u3057\u305f\u7d50\u679c\uff09\u306e\u65b9\u5f0f\u3067\u57fa\u672c\u7684\u306a\u8003\u3048\u65b9\u3068\u3057\u3066\u3001\u300c\u4ee5\u524d\u306f\u7d50\u679c\u3092X\u3068\u3057\u305f\u304c\u3001\u305d\u308c\u306f\u9593\u9055\u3063\u3066\u3044\u305f\u3002X\u306e\u4ee3\u308f\u308a\u306bY\u3067\u66f4\u65b0\u3059\u308b\u3088\u3002\u300d\u3068\u306a\u308a\u307e\u3059\u3002<br \/>\n\u3053\u306e\u30e2\u30fc\u30c9\u306f\u4e0b\u8a18\u306e\u3088\u3046\u306a\u5834\u5408\u306b\u6709\u7528\u3060\u3068\u8003\u3048\u3089\u308c\u307e\u3059\u3002<\/p>\n<p>Session Window\u3092\u4f7f\u7528\u3059\u308b\u5834\u5408\u306b\u65b0\u3057\u3044\u5024\u306f\u5358\u306b\u524d\u306e\u5024\u3092\u66f4\u65b0\u3059\u308b\u3060\u3051\u3067\u306a\u304f\u3001\u524d\u306eWindow\u306e\u5024\u3092\u7d71\u5408\u3059\u308b\u306a\u3069<br \/>\n\u4e0b\u6d41\u306e\u30b7\u30b9\u30c6\u30e0\u304c\u8907\u6570\u5b58\u5728\u3057\u3001\u5404\u3005\u7570\u306a\u308b\u65b9\u5f0f\u3067\u96c6\u8a08\u3092\u884c\u3046\u306a\u3069<\/p>\n<p>\u901a\u8fc7\u5b9e\u9645\u6bd4\u8f83\u591a\u79cd\u6a21\u5f0f\u7684\u64cd\u4f5c\uff0c\u53ef\u4ee5\u6e05\u695a\u5730\u4e86\u89e3\u6bcf\u4e2a\u6a21\u5f0f\u4e4b\u95f4\u7684\u5dee\u5f02\u3002<\/p>\n<p>\u5047\u8bbe\u572812:00\u81f312:02\u4e4b\u95f4\u8fdb\u884c\u6570\u636e\u6c47\u603b\uff0c\u6570\u636e\u63a5\u6536\u7684\u987a\u5e8f\u5982\u4e0b\u3002<\/p>\n<div>\n<div class=\"post-table\">No\u51e6\u7406\u6642\u523b\u30a4\u30d9\u30f3\u30c8\u767a\u751f\u6642\u523b\u30a4\u30d9\u30f3\u30c8\u306e\u5024112:0512:027212:0612:007312:0712:018<\/div>\n<\/div>\n<p>\u7ecf\u8fc7\u8fd9\u6837\u7684\u64cd\u4f5c\uff0c\u4e0d\u8bba\u662f\u7d2f\u52a0\u4e0e\u56de\u9000\u6a21\u5f0f\u770b\u8d77\u6765\u663e\u5f97\u610f\u4e49\u96be\u61c2\uff0c\u4f46\u4ece\u8fd9\u6837\u770b\u6765\uff0c\u6700\u7ec8\u7684\u503c\u4e5f\u597d\uff0c\u603b\u8ba1\u7684\u503c\u4e5f\u597d\uff0c\u90fd\u80fd\u770b\u51fa\u4fdd\u6301\u4e86\u4e00\u5b9a\u7684\u6570\u503c\u3002<\/p>\n<div>\n<div class=\"post-table\">\n\u7834\u68c4\u30e2\u30fc\u30c9\u7d2f\u7b97\u30e2\u30fc\u30c9\u7d2f\u7b97\uff06\u5f8c\u9000\u30e2\u30fc\u30c9Event1777Event271414,-7Event382222,-14\u6700\u7d42\u502482222\u7dcf\u8a08\u5024224322<\/div>\n<\/div>\n<h2>\u5728\u5904\u7406\u8fd9\u4e2a\u95ee\u9898\u4e0a\uff0c\u80fd\u591f\u5bf9\u6240\u6709\u7684\u60c5\u51b5\u90fd\u505a\u51fa\u5e94\u5bf9\u662f\u4e0d\u53ef\u80fd\u7684\u5427\uff1f<\/h2>\n<p>\u5728\u5f15\u5165\u6c34\u5370\u3001\u89e6\u53d1\u5668\u548c\u7d2f\u79ef\u673a\u5236\u540e\uff0c\u4e0d\u610f\u5473\u7740\u6d41\u5904\u7406\u5c31\u53ef\u4ee5\u5e94\u5bf9\u6240\u6709\u60c5\u51b5\u3002<\/p>\n<p>\u56e0\u4e3a\u4f1a\u51fa\u73b0\u4ee5\u4e0b\u8fd9\u4e9b\u95ee\u9898\uff0c\u6240\u4ee5\u624d\u4f1a\u8fd9\u6837\u505a\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">Watermark\u3092\u5b9f\u6642\u523b\u304b\u3089\u3069\u308c\u304f\u3089\u3044\u9045\u3089\u305b\u3066\u8a2d\u5b9a\u3059\u308c\u3070\u3044\u3044\u306e\u304b\uff1f<\/ul>\n<\/li>\n<\/ul>\n<p>\u9045\u308c\u3092\u5927\u304d\u304f\u3059\u308c\u3070\u6b63\u78ba\u6027\u306f\u5897\u3057\u307e\u3059\u304c\u3001\u9045\u5ef6\u6642\u9593\u306f\u5927\u304d\u304f\u306a\u308a\u307e\u3059\u3002<\/p>\n<p>Accumulation\u306e\u305f\u3081\u306b\u30a6\u30a3\u30f3\u30c9\u30a6\u306e\u96c6\u8a08\u7d50\u679c\u3092\u3069\u308c\u3060\u3051\u4fdd\u6301\u3059\u308c\u3070\u3044\u3044\u306e\u304b\uff1f<\/p>\n<p>\u4fdd\u6301\u3059\u308b\u6642\u9593\u304c\u9577\u3044\u307b\u3069\u3001\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u3092\u884c\u3046\u30b7\u30b9\u30c6\u30e0\u306e\u30ea\u30bd\u30fc\u30b9\u304c\u5fc5\u8981\u3068\u306a\u308a\u307e\u3059\u3002<\/p>\n<p>\u6570\u636e\u5904\u7406\u7cfb\u7edf\uff08\u5305\u62ec\u6279\u5904\u7406\u548c\u6d41\u5904\u7406\uff09\u88ab\u8ba4\u4e3a\u6709\u4ee5\u4e0b\u4e09\u4e2a\u8981\u7d20\u7684\u6743\u8861\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u5b8c\u5168\u6027(Completeness)<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u4f4e\u9045\u5ef6(Low Latency)<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\u4f4e\u30b3\u30b9\u30c8(Low Cost)<\/ul>\n<p>\u65e0\u6cd5\u540c\u65f6\u6ee1\u8db3\u8fd9\u4e09\u4e2a\u8981\u7d20\uff0c\u6240\u6709\u7684\u6570\u636e\u5904\u7406\u7cfb\u7edf\u90fd\u662f\u7531\u8fd9\u4e09\u4e2a\u8981\u7d20\u7684\u5e73\u8861\u6784\u6210\u7684\u3002<br \/>\n\u5728\u8fd9\u91cc\uff0c\u6210\u672c\u4e0d\u4ec5\u6307\u6570\u636e\u5904\u7406\u7cfb\u7edf\u7684\u8d44\u6e90\uff0c\u8fd8\u5305\u62ec\u5bf9\u6536\u96c6\u6570\u636e\u7684\u8def\u5f84\u7b49\u7684\u6210\u672c\u3002<\/p>\n<p>\u4f8b\u5982\uff0c\u5bf9\u4e8e\u4ed8\u8d39\u5904\u7406\uff0c\u6211\u4eec\u5c06\u5b8c\u6574\u6027\u89c6\u4e3a\u91cd\u8981\uff0c\u5373\u4f7f\u4f1a\u7a0d\u5fae\u5ef6\u8fdf\u6216\u589e\u52a0\u4e00\u4e9b\u6210\u672c\u4e5f\u4e0d\u4f1a\u6210\u4e3a\u95ee\u9898\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/116-0.png\" alt=\"Billing.PNG\" \/><\/div>\n<p>\u7136\u800c\uff0c\u5728\u5f02\u5e38\u68c0\u6d4b\u7cfb\u7edf\u4e2d\uff0c\u4f4e\u5ef6\u8fdf\u662f\u6700\u4f18\u5148\u8003\u8651\u7684\uff0c\u5176\u4ed6\u8981\u7d20\u7684\u4f18\u5148\u7ea7\u76f8\u5bf9\u8f83\u4f4e\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/118-0.png\" alt=\"AbuseDetection.PNG\" \/><\/div>\n<p>\u7531\u4e8e\u5b58\u5728\u8fd9\u6837\u7684\u60c5\u51b5\uff0c\u6211\u4eec\u9700\u8981\u6839\u636e\u5b9e\u9645\u5f00\u53d1\u7684\u7cfb\u7edf\u6765\u786e\u5b9a\u9700\u8981\u4ec0\u4e48\uff0c\u5e76\u627e\u5230\u4e00\u4e2a\u5408\u7406\u7684\u5e73\u8861\u70b9\u3002<\/p>\n<h1>\u5b9e\u9645\u7684\u6d41\u5904\u7406\u7cfb\u7edf\u662f\u4ec0\u4e48\u6837\u7684\uff1f<\/h1>\n<p>\u7531\u4e8e\u5df2\u7ecf\u8ba8\u8bba\u8fc7\u8fc4\u4eca\u4e3a\u6b62\u7684\u6d41\u5904\u7406\u662f\u4ec0\u4e48\uff0c\u63a5\u4e0b\u6765\u5c06\u89e3\u91ca\u8fdb\u884c\u6d41\u5904\u7406\u7684\u6d41\u5904\u7406\u7cfb\u7edf\u4f1a\u662f\u4ec0\u4e48\u6837\u5b50\u3002<\/p>\n<h2>\u7cfb\u7edf\u914d\u7f6e<\/h2>\n<p>\u7cfb\u7edf\u914d\u7f6e\u4e4b\u524d\u4e5f\u51fa\u73b0\u8fc7\uff0c\u4f46\u5927\u591a\u6570\u60c5\u51b5\u4e0b\u4f7f\u7528\u5982\u4e0b\u56fe\u6240\u793a\u7684\u914d\u7f6e\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/124-0.png\" alt=\"StreamProcessingSystemArch.png\" \/><\/div>\n<p>\u6211\u5c06\u89e3\u91ca\u5404\u4e2a\u6784\u6210\u8981\u7d20\u7684\u89d2\u8272\u3002<\/p>\n<h3>\u6d88\u606f\u603b\u7ebf<\/h3>\n<p>\u7531\u4e8e\u6d41\u5904\u7406\u7cfb\u7edf\u662f\u5904\u7406\u6301\u7eed\u53d1\u751f\u7684\u65e0\u9650\u6570\u636e\u7684\u6e90\uff0c\u56e0\u6b64\u7ecf\u5e38\u4f1a\u7a81\u7136\u51fa\u73b0\u6570\u636e\u6d41\u91cf\u5267\u589e\u7684\u5cf0\u503c\u3002<\/p>\n<p>\u6b64\u5916\uff0c\u5728\u53d1\u751f\u969c\u788d\u65f6\uff0c\u8fd8\u6709\u53ef\u80fd\u9700\u8981\u4fdd\u8bc1\u6d88\u606f\u7684\u5904\u7406\uff0c\u5e76\u4e14\u53ef\u80fd\u9700\u8981\u591a\u6b21\u83b7\u53d6\u76f8\u540c\u7684\u6d88\u606f\u3002<\/p>\n<p>\u56e0\u6b64\uff0c\u901a\u5e38\u91c7\u7528\u5c06\u6570\u636e\u4e34\u65f6\u5b58\u50a8\u5728\u6d88\u606f\u603b\u7ebf\u4e2d\u7684\u65b9\u6cd5\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u4f7f\u7528\u3055\u308c\u308b\u4e3b\u8981\u306a\u30d7\u30ed\u30c0\u30af\u30c8<\/ul>\n<\/li>\n<\/ul>\n<p>Apache Kafka<br \/>\n(Distributed Log)<br \/>\n(Pulsar)<\/p>\n<p>\u4f7f\u7528\u3055\u308c\u308b\u4e3b\u8981\u306a\u30de\u30cd\u30fc\u30b8\u30c9\u30b5\u30fc\u30d3\u30b9<\/p>\n<p>Amazon Kinesis Stream<br \/>\nGoogle Cloud PubSub<br \/>\nAzure Service Bus<\/p>\n<h3>\u6d41\u5904\u7406\u57fa\u7840\u8bbe\u65bd<\/h3>\n<p>\u4ece\u6d88\u606f\u603b\u7ebf\u4e2d\u83b7\u53d6\u6570\u636e\u5e76\u8fdb\u884c\u6d41\u5904\u7406\u7684\u57fa\u7840\u90e8\u5206\u3002<br \/>\n\u7531\u4e8e\u59cb\u7ec8\u6301\u7eed\u8fd0\u884c\uff0c\u56e0\u6b64\u9700\u8981\u66f4\u9ad8\u7684\u8fdb\u7a0b\u7a33\u5b9a\u6027\uff0c\u4e0e\u6279\u5904\u7406\u76f8\u6bd4\uff0c\u901a\u5e38\u66f4\u52a0\u91cd\u8981\u3002<\/p>\n<p>\u8bf7\u786e\u8ba4\u6709\u5173\u88ab\u4f7f\u7528\u7684\u4ea7\u54c1\u548c\u670d\u52a1\u7684\u6570\u91cf\u548c\u60f3\u6cd5\u8f83\u591a\u7684\u95ee\u9898\uff0c\u5373\u201c\u6709\u54ea\u4e9b\u6d41\u5904\u7406\u57fa\u7840\u8bbe\u65bd\u53ef\u7528\uff1f\u201d<\/p>\n<h3>\u6570\u636e\u4f7f\u7528\u76ee\u7684<\/h3>\n<p>\u5c06\u5728\u6d41\u5904\u7406\u57fa\u7840\u8bbe\u65bd\u4e2d\u5904\u7406\u7684\u6570\u636e\u8f93\u51fa\uff0c\u5e76\u7528\u4e8e\u5b9e\u9645\u4f7f\u7528\u7684\u529f\u80fd\u3002<br \/>\n\u5bf9\u4e8e\u8fd9\u90e8\u5206\u6765\u8bf4\uff0c\u5b83\u5b8c\u5168\u4f9d\u8d56\u4e8e\u7cfb\u7edf\uff0c\u56e0\u6b64\u4e0d\u80fd\u4e00\u6982\u800c\u8bba\u3002<\/p>\n<p>\u8fd8\u6709\u4e00\u79cd\u6a21\u5f0f\u662f\u5c06\u6570\u636e\u4f7f\u7528\u7684\u76ee\u7684\u5730\u518d\u6b21\u53d8\u4e3a\u6d88\u606f\u603b\u7ebf\uff0c\u7136\u540e\u901a\u8fc7\u591a\u4e2a\u9636\u6bb5\u8fdb\u884c\u6d41\u5904\u7406\u3002<\/p>\n<h1>\u5728\u6d41\u5a92\u4f53\u5904\u7406\u57fa\u7840\u8bbe\u65bd\u4e2d\uff0c\u6709\u54ea\u4e9b\u9009\u9879\uff1f<\/h1>\n<p>\u6211\u5c06\u4ecb\u7ecd\u6709\u5173\u5728\u6d41\u5904\u7406\u7cfb\u7edf\u4e2d\u6267\u884c\u6570\u636e\u5904\u7406\u7684\u4ea7\u54c1\u548c\u670d\u52a1\uff0c\u4ee5\u5b9e\u73b0\u6d41\u5904\u7406\u57fa\u7840\u8bbe\u65bd\u3002<\/p>\n<h2>\u6d41\u5904\u7406\u57fa\u7840\u8bbe\u65bd\u4e4b\u8c31<\/h2>\n<p>\u9996\u5148\uff0c\u5982\u679c\u6211\u4eec\u67e5\u770b\u6d41\u5904\u7406\u5e73\u53f0\u7684\u5386\u53f2\uff0c\u7ed3\u679c\u5982\u4e0b\uff1a<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/141-0.png\" alt=\"StreamProcessingSeries.png\" \/><\/div>\n<p>\u7136\u800c\uff0c\u5373\u4f7f\u662f\u79f0\u4e4b\u4e3a\u6d41\u5904\u7406\u57fa\u7840\u67b6\u6784\uff0c\u7531\u4e8e\u5176\u6027\u8d28\u5fae\u5999\u5730\u53d8\u5316\uff0c\u6240\u4ee5\u6211\u4eec\u5047\u8bbe\u8fdb\u884c\u4e86\u51e0\u4e2a\u5206\u7c7b\u3002<br \/>\n\u5f53\u4ed4\u7ec6\u7814\u7a76\u540e\uff0c\u4e25\u683c\u6765\u8bf4\uff0c\u8fd9\u4e9b\u5206\u7c7b\u5e76\u4e0d\u9002\u7528\uff0c\u4f46\u4e3a\u4e86\u7b80\u5355\u660e\u4e86\u7684\u76ee\u7684\uff0c\u6211\u4eec\u6309\u7167\u4e0a\u56fe\u8fdb\u884c\u4e86\u5212\u5206\u3002<\/p>\n<h3>\u6309\u7167\u6bcf\u4e2a\u7c7b\u522b\u8fdb\u884c\u5206\u7c7b<\/h3>\n<p>\u6bcf\u4e2a\u7c7b\u522b\u90fd\u5177\u6709\u4ee5\u4e0b\u7684\u5b9a\u4f4d\/\u529f\u80fd\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u30a8\u30f3\u30b8\u30f3<\/ul>\n<\/li>\n<\/ul>\n<p>\u7279\u306b\u4ed6\u306b\u7279\u5225\u306a\u6a5f\u80fd\u3092\u6301\u305f\u306a\u3044\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u30a8\u30f3\u30b8\u30f3<\/p>\n<p>UI\u3067Dataflow\u5b9a\u7fa9<\/p>\n<p>UI\u3067Dataflow\u304c\u5b9a\u7fa9\u53ef\u80fd\u306b\u306a\u3063\u3066\u304a\u308a\u3001\u305d\u306e\u5b9a\u7fa9\u306e\u901a\u308a\u52d5\u4f5c\u3059\u308b\u6a5f\u80fd\u3092\u4fdd\u6301<\/p>\n<p>DSL<\/p>\n<p>\u540c\u4e00\u306e\u8a18\u8ff0\u3067\u8907\u6570\u306e\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u30a8\u30f3\u30b8\u30f3\u4e0a\u3067\u30a2\u30d7\u30ea\u30b1\u30fc\u30b7\u30e7\u30f3\u304c\u5b9f\u884c\u53ef\u80fd<\/p>\n<p>\u30de\u30cd\u30fc\u30b8\u30c9\u30b5\u30fc\u30d3\u30b9<\/p>\n<p>\u30af\u30e9\u30a6\u30c9\u4e0a\u3067\u5b9f\u884c\u57fa\u76e4\u304c\u63d0\u4f9b\u3002\u30a2\u30d7\u30ea\u30b1\u30fc\u30b7\u30e7\u30f3\u3092\u30c7\u30d7\u30ed\u30a4\u3059\u308b\u3068\u305d\u308c\u306b\u5f93\u3063\u3066\u52d5\u4f5c\u3059\u308b\u3002<\/p>\n<h3>\u4ea7\u54c1\u8bf4\u660e\uff08\u90e8\u5206\u5185\u5bb9\uff09<\/h3>\n<p>\u6211\u5c06\u89e3\u91ca\u4e00\u4e9b\u56fe\u4e2d\u4ea7\u54c1\u548c\u670d\u52a1\u7684\u6982\u8981\u3002<br \/>\n\u7531\u4e8e\u57fa\u51c6\u662f\u6211\u81ea\u5df1\u5bf9\u8fd9\u4e9b\u4e8b\u7269\u6709\u4e00\u5b9a\u4e86\u89e3\uff0c\u6240\u4ee5\u53ef\u80fd\u4f1a\u6709\u4e00\u4e9b\u504f\u89c1&#8230;<\/p>\n<h4>\u66b4\u98ce\u96e8<\/h4>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">2011\u5e74\u306bTwitter\u306b\u3088\u3063\u3066\u516c\u958b<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u5b9f\u88c5\u8a00\u8a9e\uff1aClojure<\/ul>\n<\/li>\n<\/ul>\n<p>Clojure\u304c\u8aad\u307f\u66f8\u304d\u51fa\u6765\u306a\u3044\u3068\u6df1\u3044\u554f\u984c\u306f\u8ffd\u3048\u306a\u3044\u3002<\/p>\n<p>\u5b9f\u8cea\u7684\u306b\u30e6\u30fc\u30b6\u306b\u5bfe\u3057\u3066\u521d\u306eOSS\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u30a8\u30f3\u30b8\u30f3<\/p>\n<p>\u30e1\u30c3\u30bb\u30fc\u30b8\u306e\u51e6\u7406\u30bb\u30de\u30f3\u30c6\u30a3\u30af\u30b9\u3068\u3057\u3066\u3001At Least Once\u304c\u53ef\u80fd\u306b\u306a\u3063\u305f\u306e\u304c\u5927\u304d\u3044\u3002<\/p>\n<p>\u305f\u3060\u3057\u3001\u521d\u671f\u306e\u30d7\u30ed\u30c0\u30af\u30c8\u306e\u305f\u3081\u554f\u984c\u3082\u591a\u304b\u3063\u305f\u3002<\/p>\n<p>\u521d\u671f\u306f\u30e1\u30c3\u30bb\u30fc\u30b8\u5358\u4f4d\u306e\u51e6\u7406\u3067\u3042\u308a\u3001\u30ec\u30a4\u30c6\u30f3\u30b7\u306f\u4f4e\u3044\u304c\u30b9\u30eb\u30fc\u30d7\u30c3\u30c8\u304c\u5c0f\u3055\u3044\u3002<br \/>\n\u30c7\u30fc\u30bf\u53d6\u5f97\u5074\u306e\u6027\u80fd\u304c\u9ad8\u3044\u3068\u3042\u3075\u308c\u3066\u6b7b\u306c\u3002<br \/>\n\u30c7\u30d5\u30a9\u30eb\u30c8\u306e\u30b9\u30ec\u30c3\u30c9\u914d\u7f6e\u304c\u975e\u52b9\u7387\u3002<br \/>\nAt Least Once\u306eAck\u306e\u51e6\u7406\u52b9\u7387\u304c\u975e\u52b9\u7387\u3002<\/p>\n<p>\u4ee5\u5f8c\u306e\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u30d7\u30ed\u30c0\u30af\u30c8\u306b\u5927\u304d\u304f\u5f71\u97ff\u3057\u3066\u3044\u308b\u3002<\/p>\n<h4>\u5b9e\u65f6\u6570\u636e\u6d41\u8ba1\u7b97<\/h4>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">2013\u5e74\u306bamplab\u306b\u3088\u3063\u3066\u516c\u958b<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u5b9f\u88c5\u8a00\u8a9e\uff1aScala<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u30d0\u30c3\u30c1\u51e6\u7406\u30d5\u30ec\u30fc\u30e0\u30ef\u30fc\u30afSpark\u4e0a\u3067\u5c0f\u30d0\u30c3\u30c1\u3092\u9023\u7d9a\u5b9f\u884c\u3057\u3001\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u3092\u5b9f\u73fe<\/ul>\n<\/li>\n<\/ul>\n<p>\u30de\u30a4\u30af\u30ed\u30d0\u30c3\u30c1\u3068\u547c\u3070\u308c\u308b\u3002<br \/>\n\u767a\u8868\u6642\u70b9\u3067\u3001\u4e3b\u306bStorm\u3068\u6bd4\u3057\u3066\u30b9\u30eb\u30fc\u30d7\u30c3\u30c8\u306f\u5927\u304d\u3044\u304c\u3001\u30ec\u30b9\u30dd\u30f3\u30b9\u306f\u9045\u3044\u3002<\/p>\n<p>\u73fe\u6642\u70b9\u3067\u306fFlink\u3084Apex\u304c\u51fa\u3066\u304d\u3066\u304a\u308a\u3001\u305d\u308c\u3068\u6bd4\u3079\u308b\u3068\u4e21\u65b9\u5fae\u5999\u306a\u72b6\u6cc1\u3002<br \/>\n\u5168\u4f53\u3068\u3057\u3066\u3001\u5927\u898f\u6a21\u30af\u30e9\u30b9\u30bf\u3092\u524d\u63d0\u3068\u3057\u3066\u3044\u308bSpark\u306e\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u306b\u6cbf\u3063\u3066\u3044\u308b\u305f\u3081\u52b9\u7387\u3082\u60aa\u3044\u3002<\/p>\n<p>Spark\u30a8\u30b3\u30b7\u30b9\u30c6\u30e0\u4e0a\u3067\u5b9f\u884c\u53ef\u80fd\u306a\u306e\u304c\u5927\u304d\u3044\u3002<\/p>\n<p>\u6a5f\u68b0\u5b66\u7fd2\u30e9\u30a4\u30d6\u30e9\u30ea\u306e\u5229\u7528<br \/>\nSQL\u306b\u3088\u308b\u30c7\u30fc\u30bf\u64cd\u4f5c<br \/>\n\u958b\u767a\u624b\u6cd5\u3082\u540c\u69d8\u306e\u3082\u306e\u304c\u5229\u7528\u53ef\u80fd<\/p>\n<h4>\u4f60\u83f2<\/h4>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">2014\u5e74\u306bNSA\u306b\u3088\u3063\u3066\u516c\u958b<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u5b9f\u88c5\u8a00\u8a9e\uff1aJava<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u753b\u9762\u4e0a\u3067\u30c7\u30fc\u30bf\u30d5\u30ed\u30fc\u3092\u5b9a\u7fa9\u3057\u3001\u8907\u6570\u30b5\u30fc\u30d0\u306b\u30c7\u30d7\u30ed\u30a4\u3057\u3066\u5b9f\u884c\u53ef\u80fd<\/ul>\n<\/li>\n<\/ul>\n<p>HTTP\u3067\u30c7\u30fc\u30bf\u53d6\u5f97\uff1e\u5909\u63db\uff1eHDFS\u306b\u6295\u5165 etc\u2026<br \/>\n\u30b3\u30f3\u30dd\u30fc\u30cd\u30f3\u30c8\u9593\u306e\u30ad\u30e5\u30fc\u6bce\u306b\u512a\u5148\u5ea6\u8a2d\u5b9a\u3084QoS\u8a2d\u5b9a\u304c\u5b9a\u7fa9\u53ef\u80fd<\/p>\n<p>\u5404\u30c7\u30fc\u30bf\u306e\u767a\u751f\u5143\u3084\u51e6\u7406\u5c65\u6b74\u306e\u78ba\u8a8d\u6a5f\u80fd\u304c\u3042\u308a\u3001\u500b\u5225\u306e\u30c7\u30fc\u30bf\u5358\u4f4d\u3067\u5b9f\u969b\u306b\u3069\u3046\u51e6\u7406\u304c\u884c\u308f\u308c\u305f\u304b\u3092\u8ffd\u3048\u308b\u3002<\/p>\n<h4>Flink (\u5f17\u6797\u514b)<\/h4>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">2014\u5e74\u306b\u516c\u958b<\/ul>\n<\/li>\n<\/ul>\n<p>\u5b9f\u969b\u306f2011\u5e74\u304b\u3089Stratosphere\u3068\u3057\u3066\u516c\u958b\u3055\u308c\u3066\u306f\u3044\u305f<\/p>\n<p>\u5b9f\u88c5\u8a00\u8a9e\uff1aScala<br \/>\n\u30d0\u30c3\u30c1\u51e6\u7406\u3068\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u4e21\u65b9\u306eAPI\u3092\u63d0\u4f9b\u3059\u308b\u30c7\u30fc\u30bf\u51e6\u7406\u30a8\u30f3\u30b8\u30f3<br \/>\n\u969c\u5bb3\u5bfe\u5fdc\u306e\u305f\u3081\u306e\u72b6\u614b\u306e\u81ea\u52d5\u4fdd\u5b58\u3068\u3057\u3066\u3001\u52b9\u7387\u306e\u3044\u3044\u5206\u6563\u30b9\u30ca\u30c3\u30d7\u30b7\u30e7\u30c3\u30c8\u65b9\u5f0f\u3092\u4f7f\u7528<\/p>\n<p>\u81ea\u52d5\u7684\u306b\u5404\u30b3\u30f3\u30dd\u30fc\u30cd\u30f3\u30c8\u306e\u72b6\u614b\u3092\u4fdd\u5b58<br \/>\n\u8a73\u7d30\u306fApache Flink\u306e\u5206\u6563\u30b9\u30ca\u30c3\u30d7\u30b7\u30e7\u30c3\u30c8\u306e\u65b9\u5f0f\u53c2\u7167<\/p>\n<p>\u9ad8\u30ec\u30d9\u30ebAPI\u3068\u4f4e\u30ec\u30d9\u30ebAPI\u306e\u4e21\u65b9\u3092\u63d0\u4f9b<\/p>\n<p>\u7c21\u6613\u306a\u3082\u306e\u306f\u7c21\u5358\u306b\u7d44\u3081\u308b\u3002<\/p>\n<h4>\u5c16\u9876<\/h4>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">2015\u5e74\u306bDataTorrent\u306b\u3088\u3063\u3066\u516c\u958b<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u5b9f\u88c5\u8a00\u8a9e\uff1aJava<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u5143\u3005\u91d1\u878d\u30a2\u30d7\u30ea\u30b1\u30fc\u30b7\u30e7\u30f3\u7528\u30d7\u30ed\u30c0\u30af\u30c8\u3067\u53ef\u7528\u6027\u3001\u8010\u969c\u5bb3\u6027\u91cd\u8996\u306e\u8a2d\u8a08<\/ul>\n<\/li>\n<\/ul>\n<p>\u904b\u7528\u6642\u306e\u554f\u984c\u5207\u308a\u5206\u3051\u3082\u5bb9\u6613\u306a\u69cb\u6210<br \/>\n\u30e1\u30c3\u30bb\u30fc\u30b8\u30d0\u30c3\u30d5\u30a1\u30ea\u30f3\u30b0\u3067\u969c\u5bb3\u767a\u751f\u6642\u306e\u5f71\u97ff\u3092\u4f4e\u6e1b<\/p>\n<p>\u72b6\u614b\u306e\u4fdd\u5b58\u3001\u5b9f\u884c\u74b0\u5883\u5171\u306b\u30d0\u30e9\u30f3\u30b9\u3088\u304f\u9ad8\u30ec\u30d9\u30eb<\/p>\n<p>HDFS\u3092KVS\u306e\u3088\u3046\u306b\u4f7f\u3063\u3066\u8010\u969c\u5bb3\u6027\u3068\u30ec\u30b9\u30dd\u30f3\u30b9\u3092\u4e21\u7acb<br \/>\nYARN\u4e0a\u3067\u30aa\u30fc\u30d0\u30fc\u30d8\u30c3\u30c9\u304c\u5c11\u306a\u304f\u52d5\u4f5c\u53ef\u80fd<\/p>\n<p>\u30b9\u30b1\u30fc\u30e9\u30d3\u30ea\u30c6\u30a3\u306b\u3082\u512a\u308c\u305f\u69cb\u6210<\/p>\n<p>\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u4e2d\u306bAutoScaling\u304c\u53ef\u80fd<\/p>\n<h4>Gearpump is a term in English and does not have a direct translation in Chinese. However, it can be paraphrased as &#8220;\u9f7f\u8f6e\u6cf5&#8221; (ch\u01d0 in Chinese, which is the common term used to refer to a gear pump.<\/h4>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">2015\u5e74\u306bIntel\u306b\u3088\u3063\u3066\u516c\u958b<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u5b9f\u88c5\u8a00\u8a9e\uff1aScala<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">Google\u306e\u300cMillWheel\u300d\u306b\u5f71\u97ff\u3092\u53d7\u3051\u305f\u30d7\u30ed\u30c0\u30af\u30c8<\/ul>\n<\/li>\n<\/ul>\n<p>\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u30e2\u30c7\u30eb\u306e\u8ad6\u6587<\/p>\n<p>Actor\u30d9\u30fc\u30b9\u306e\u300c\u8584\u3044\u300d\u69cb\u6210\u3067\u62e1\u5f35\u6027\u304c\u9ad8\u3044\u3002<\/p>\n<p>\u305d\u306e\u5206\u3001\u72b6\u614b\u7ba1\u7406\u306a\u3069\u3082\u81ea\u524d\u3067\u6e96\u5099\u3059\u308b\u5fc5\u8981\u6709<br \/>\n\u958b\u767a\u30b3\u30b9\u30c8\u306f\u9ad8\u3081\u3060\u304c\u3001\u6027\u80fd\u7684\u306b\u3082\u512a\u308c\u308b\u3002<\/p>\n<p>Reactive Streams\u306b\u6e96\u62e0\u3057\u3066\u304a\u308a\u3001\u6a19\u6e96\u5316\u3055\u308c\u305fBack Pressure\u6a5f\u69cb\u3092\u4fdd\u6301<br \/>\nAkka Stream\u65b9\u5f0f\u306e\u8a18\u8ff0\u306b\u3088\u308a\u76f4\u89b3\u7684\u306b\u30b0\u30e9\u30d5\u3092\u5b9a\u7fa9\u53ef\u80fd<\/p>\n<h4>\u5149\u675f sh\u00f9)<\/h4>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">2016\u5e74\u306bGoogle\u306b\u3088\u3063\u3066\u516c\u958b<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u5b9f\u88c5\u8a00\u8a9e\uff1aJava<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u30d0\u30c3\u30c1\u51e6\u7406\uff0f\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u3092\u62bd\u8c61\u5316\u3057\u3001\u8907\u6570\u306e\u5b9f\u884c\u30a8\u30f3\u30b8\u30f3\u4e0a\u306b\u30c7\u30d7\u30ed\u30a4\u53ef\u80fd\u306aDSL<\/ul>\n<\/li>\n<\/ul>\n<p>\u4e0b\u8a18\u306e\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u30a8\u30f3\u30b8\u30f3\u306b\u5bfe\u3057\u3066\u30c7\u30d7\u30ed\u30a4\u53ef\u80fd\uff08\u7aef\u7684\u306b\u6700\u8fd1\u306eJVM\u4e0a\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u30a8\u30f3\u30b8\u30f3\u306a\u3089\u5927\u4f53\u5bfe\u5fdc\uff09<\/p>\n<p>\u30ed\u30fc\u30ab\u30eb\u74b0\u5883(\u30c7\u30d5\u30a9\u30eb\u30c8)<br \/>\nGoogle Cloud Dataflow<br \/>\nApache Spark<br \/>\nApache Flink<br \/>\nApache Apex<br \/>\nApache Gearpump<\/p>\n<p>\u30dd\u30fc\u30bf\u30d3\u30ea\u30c6\u30a3\u306f\u9ad8\u3044\u304c\u3001\u5404\u30a8\u30f3\u30b8\u30f3\u306e\u56fa\u6709\u62e1\u5f35\u6a5f\u80fd\u3001\u56fa\u6709\u30e9\u30a4\u30d6\u30e9\u30ea\u306f\u4f7f\u7528\u3067\u304d\u306a\u3044\u3002<\/p>\n<p>\u6a5f\u68b0\u5b66\u7fd2\u306a\u3069\u306e\u6a5f\u80fd\u3092\u4f7f\u7528\u3059\u308b\u5834\u5408\u3001\u5225\u9014TensorFlow\u306a\u3069\u306b\u30c7\u30fc\u30bf\u3092\u6e21\u3057\u3066\u5b9f\u884c\u3059\u308b\u5fc5\u8981\u304c\u3042\u308b\u3002<\/p>\n<h4>\u8c37\u6b4c\u4e91\u6570\u636e\u6d41<\/h4>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">2015\u5e74\u306bGoogle\u306b\u3088\u3063\u3066\u516c\u958b<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">Beam(\u516c\u958b\u5f53\u521d\u306fDataflow API)\u3067\u8a18\u8ff0\u3057\u305f\u51e6\u7406\u3092\u30de\u30cd\u30fc\u30b8\u30c9\u30b5\u30fc\u30d3\u30b9\u4e0a\u3067\u52d5\u4f5c\u53ef\u80fd<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u4ed6\u306e\u30de\u30cd\u30fc\u30b8\u30c9\u30b5\u30fc\u30d3\u30b9\u7cfb\u30d7\u30ed\u30c0\u30af\u30c8\u306b\u6bd4\u3079\u3066\u3001\u53ef\u80fd\u306a\u51e6\u7406\u306e\u5e45\u304c\u5e83\u3044<\/ul>\n<\/li>\n<\/ul>\n<p>\u300c\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u30a2\u30d7\u30ea\u30b1\u30fc\u30b7\u30e7\u30f3\u300d\u3068\u3057\u3066\u8a18\u8ff0\u53ef\u80fd\u3067\u3042\u308b\u305f\u3081<\/p>\n<p>CPU\u306e\u5229\u7528\u7387\u3092\u6975\u529b\u9ad8\u3081\u308b\u3088\u3046\u306b\u81ea\u52d5\u3067\u8abf\u6574\u3055\u308c\u308b\u305f\u3081\u3001\u30ea\u30bd\u30fc\u30b9\u306e\u904a\u3073\u304c\u767a\u751f\u3057\u306b\u304f\u3044\u3002<\/p>\n<h4>\u4e9a\u9a6c\u900a Kinesis \u5206\u6790<\/h4>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">2016\u5e74\u306bAmazon\u306b\u3088\u3063\u3066\u516c\u958b<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">SQL\u3067\u30b9\u30c8\u30ea\u30fc\u30e0\u4e0a\u306b\u6d41\u308c\u3066\u3044\u308b\u30c7\u30fc\u30bf\u306b\u5bfe\u3057\u3066\u7d99\u7d9a\u7684\u30af\u30a8\u30ea\u3092\u5b9f\u884c\u53ef\u80fd<\/ul>\n<\/li>\n<\/ul>\n<p>\u51fa\u6765\u308b\u6a5f\u80fd\u306f\u7d5e\u3089\u308c\u308b\u304c\u3001\u975e\u5e38\u306b\u304a\u624b\u8efd\u306b\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u3092\u5b9a\u7fa9\u53ef\u80fd<br \/>\n\u30c7\u30fc\u30bf\u3092\u84c4\u7a4d\u3059\u308b\u524d\u306b\u5206\u6790\u30fb\u30a2\u30af\u30b7\u30e7\u30f3\u3092\u5b9f\u884c\u3059\u308b\u3068\u3044\u3046Data in Motion\u306e\u8003\u3048\u65b9\u306e\u30d9\u30fc\u30b9<\/p>\n<p>SQL\u65b9\u5f0f\u3067\u306f\u3042\u308b\u3082\u306e\u306e\u3001EventTime\u3068ProcessingTime\u306e\u533a\u5225\u304c\u53ef\u80fd\u306a\u3069\u3001\u6a5f\u80fd\u306f\u305d\u3053\u305d\u3053\u5e83\u3044\u3002<br \/>\n\u5185\u90e8\u3067\u81ea\u52d5\u7684\u306b\u30b9\u30b1\u30fc\u30eb\u3059\u308b\u304c\u3001\u305d\u306e\u5206\u30ea\u30bd\u30fc\u30b9\u4f7f\u7528\u91cf\u304c\u4e8b\u524d\u306b\u8aad\u307f\u306b\u304f\u3044\u3002<\/p>\n<h3>\u7528\u54ea\u4e2a\u6bd4\u8f83\u597d\uff1f ?)<\/h3>\n<p>\u4ece\u73b0\u5728\u6765\u770b\uff0c\u5c31\u4e2a\u4eba\u800c\u8a00\uff0c\u5982\u679c\u5728\u672c\u5730\u73af\u5883\u4e2d\u8fd0\u884c\u7684\u8bdd\uff0c\u6211\u8ba4\u4e3aFlink\u548cApex\u5728\u6027\u80fd\u3001\u529f\u80fd\u548c\u5f00\u53d1\u6548\u7387\u65b9\u9762\u90fd\u80fd\u53d6\u5f97\u5f88\u597d\u7684\u5e73\u8861\u3002Gearpump\u867d\u7136\u5728\u8bf4\u660e\u4e2d\u88ab\u63d0\u5230\u4e86\uff0c\u5177\u6709\u9ad8\u5ea6\u53ef\u5b9a\u5236\u6027\u548c\u6027\u80fd\uff0c\u4f46\u7531\u4e8e\u9700\u8981\u5c06\u5404\u4e2a\u5143\u7d20\u5b9e\u73b0\u4e3aActor\uff0c\u6240\u4ee5\u5728\u5f00\u53d1\u5927\u91cf\u5e94\u7528\u65f6\uff0c\u5f00\u53d1\u6548\u7387\u4e0a\u7ed9\u4eba\u5370\u8c61\u7a0d\u5fae\u6b20\u7f3a\u3002\u7531\u4e8e\u5728\u8fc1\u79fb\u6279\u5904\u7406\u65f6\u6210\u672c\u8f83\u4f4e\uff0c\u6240\u4ee5\u5728\u73b0\u9636\u6bb5\u5982\u679c\u5df2\u7ecf\u5728\u4f7f\u7528Spark\u7684\u8bdd\uff0cSpark Streaming\u662f\u6bd4\u8f83\u5408\u9002\u7684\u9009\u62e9\u3002<\/p>\n<p>\u5982\u679c\u5728\u4e91\u4e0a\u6267\u884c\u7684\u8bdd\uff0c\u6700\u597d\u4f7f\u7528\u60a8\u76ee\u524d\u6b63\u5728\u4f7f\u7528\u7684\u4e91\u670d\u52a1\u63d0\u4f9b\u5546\u7684\u670d\u52a1\u3002<\/p>\n<h1>\u5728\u6784\u5efa\u6d41\u5904\u7406\u7cfb\u7edf\u65f6\u9700\u8981\u6ce8\u610f\u7684\u4e8b\u9879\u662f\u4ec0\u4e48\uff1f<\/h1>\n<p>\u5728\u6784\u5efa\u6d41\u5904\u7406\u7cfb\u7edf\u65f6\u8981\u8003\u8651\u7684\u8981\u70b9\u548c\u5728\u5f00\u53d1\u65f6\u8981\u8bbe\u7f6e\u7684\u8981\u70b9\uff0c\u4ee5\u53ca\u8981\u6ce8\u610f\u7684\u9677\u9631\u548c\u5e38\u89c1\u8bef\u89e3\u7b49\u8fdb\u884c\u603b\u7ed3\u3002<\/p>\n<h2>\u9700\u8981\u8003\u8651\u7684\u8981\u70b9<\/h2>\n<p>\u5728\u6784\u5efa\u6d41\u5904\u7406\u7cfb\u7edf\u65f6\uff0c\u6709\u8bb8\u591a\u9700\u8981\u8003\u8651\u7684\u8981\u70b9\u3002\u6211\u4eec\u5c06\u603b\u7ed3\u5176\u4e2d\u51e0\u4e2a\u4ee3\u8868\u6027\u7684\u8981\u70b9\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u72b6\u614b\u7ba1\u7406\u65b9\u5f0f<\/ul>\n<\/li>\n<\/ul>\n<p>\u300c\u72b6\u614b\u300d\u3092\u30ed\u30fc\u30ab\u30eb\u306b\u7ba1\u7406\u3059\u308b\u304b\uff1f\u305d\u308c\u3068\u3082\u30ea\u30e2\u30fc\u30c8\u306e\u30c7\u30fc\u30bf\u30b9\u30c8\u30a2\u306b\u7ba1\u7406\u3059\u308b\u304b\uff1f<\/p>\n<p>\u8010\u969c\u5bb3\u6027\u3068\u30d1\u30d5\u30a9\u30fc\u30de\u30f3\u30b9\u306e\u30c8\u30ec\u30fc\u30c9\u30aa\u30d5\u304c\u767a\u751f<br \/>\n\u969c\u5bb3\u5fa9\u65e7\u6642\u306b\u3069\u308c\u304f\u3089\u3044\u304b\u304b\u308b\u304b\uff1f<\/p>\n<p>\u305d\u306e\u300c\u72b6\u614b\u300d\u306b\u5bfe\u3057\u3066\u30a2\u30af\u30bb\u30b9\u53ef\u80fd\u306a\u69cb\u6210\u306b\u306a\u3063\u3066\u3044\u308b\u304b\uff1f<\/p>\n<p>\u30c7\u30fc\u30bf\u306e\u62bd\u8c61\u5b9a\u7fa9<\/p>\n<p>\u30c7\u30fc\u30bf\u3092\u3069\u3046\u3044\u3046\u5f62\u5f0f\u3068\u3057\u3066\u3068\u3089\u3048\u3066\u51e6\u7406\u3059\u308b\u304b\u3001\u3068\u3044\u3046\u89b3\u70b9<br \/>\n\u5927\u4f53\u306e\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u30a8\u30f3\u30b8\u30f3\u306f\u30c7\u30fc\u30bf\u30b9\u30c8\u30ea\u30fc\u30e0\u3068\u3057\u3066\u30c7\u30fc\u30bf\u3092\u6271\u3046<br \/>\nKafka Streams\u3001Spark Streaming\u3067\u306f\u4e00\u5b9a\u306e\u9593\u9694\u3054\u3068\u306e\u5b9f\u884c\u7d50\u679c\u3092\u300cTable\u300d\u3068\u3057\u3066\u6271\u3046\u3053\u3068\u3082\u53ef\u80fd<\/p>\n<p>\u6642\u523b\u30e2\u30c7\u30eb<\/p>\n<p>EventTime\u3092\u6271\u3046\u5fc5\u8981\u304c\u3042\u308b\u304b\u3001\u305d\u308c\u3068\u3082ProcessingTime\u3060\u3051\u3067\u3044\u3044\u306e\u304b\uff1f<\/p>\n<p>ProcessingTime\u3060\u3051\u3067\u3044\u3044\u5834\u5408\u3001\u5927\u5e45\u306b\u5358\u7d14\u5316\u3059\u308b\u305f\u3081\u3001\u8af8\u3005\u30b3\u30b9\u30c8\u304c\u4e0b\u304c\u308b\u3002<br \/>\n\u305f\u3060\u3001\u3042\u3068\u304b\u3089\u5207\u308a\u66ff\u3048\u308b\u306e\u306f\u5927\u5909\u306a\u306e\u3067\u3001\u3042\u3089\u304b\u3058\u3081\u6c7a\u3081\u3066\u304a\u304f\u3053\u3068\u3002<\/p>\n<p>\u30a6\u30a3\u30f3\u30c9\u30a6\u51e6\u7406<\/p>\n<p>\u300c\u4f55\u6545\u305d\u308c\u3067\u56f0\u308b\u306e\u304b\uff1f\u300d\u3067\u8a18\u8ff0\u3057\u3066\u3044\u307e\u3059\u304c\u3001\u3069\u306e\u7a2e\u5225\u306e\u30a6\u30a3\u30f3\u30c9\u30a6\u51e6\u7406\u304c\u5fc5\u8981\u306b\u306a\u308b\u304b\uff1f<\/p>\n<p>Out of order\u306a\u30c7\u30fc\u30bf\u3078\u306e\u5bfe\u5fdc<\/p>\n<p>\u300c\u30d0\u30c3\u30c1\u51e6\u7406\u306b\u306f\u306a\u3044\u3069\u3046\u3044\u3046\u65b0\u305f\u306a\u56f0\u3063\u305f\u70b9\u304c\u3042\u308b\u306e\u304b\uff1f\u300d\u300c\u65b0\u305f\u306a\u56f0\u3063\u305f\u70b9\u3092\u3069\u3046\u3084\u3063\u305f\u3089\u89e3\u6d88\u3067\u304d\u308b\u306e\u304b\uff1f\u300d\u53c2\u7167<br \/>\n\u3069\u3053\u307e\u3067\u3092\u8a31\u5bb9\u3059\u308b\u304b\u3001\u3069\u3046\u6271\u3046\u304b\u3092\u6c7a\u3081\u308b\u5fc5\u8981\u304c\u3042\u308b\u3002<\/p>\n<p>\u518d\u51e6\u7406\u304c\u53ef\u80fd\u306a\u69cb\u6210\u306b\u3059\u308b\u304b\uff1f<\/p>\n<p>A\/B\u30c6\u30b9\u30c8\u3092\u884c\u3063\u305f\u308a\u3001\u6a5f\u80fd\u306e\u8ffd\u52a0\u3084\u4e0d\u5177\u5408\u5bfe\u5fdc\u3092\u884c\u3063\u305f\u969b\u306b\u518d\u51e6\u7406\u3092\u53ef\u80fd\u306b\u3059\u308b\u304b\uff1f<br \/>\n\u518d\u51e6\u7406\u3092\u53ef\u80fd\u3068\u3059\u308b\u5834\u5408\u3001\u65e2\u5b58\u306e\u30c7\u30fc\u30bf\u3068\u306e\u4e92\u63db\u6027\u7dad\u6301\u3084\u3069\u3046\u30de\u30fc\u30b8\u3059\u308b\u304b\u3068\u3044\u3046\u3053\u3068\u304c\u4e8b\u524d\u306b\u691c\u8a0e\u304c\u5fc5\u8981\u3002<\/p>\n<p>\u51e6\u7406\u306e\u62e1\u5f35\u30fb\u7e2e\u5c0f<\/p>\n<p>\u5b9f\u884c\u4e2d\u306b\u51e6\u7406\u898f\u6a21\u306e\u7e2e\u5c0f\u30fb\u62e1\u5927\u3092\u53ef\u80fd\u3068\u3059\u308b\u304b\uff1f<br \/>\n\u305d\u308c\u3068\u3082\u30a2\u30d7\u30ea\u30b1\u30fc\u30b7\u30e7\u30f3\u306e\u518d\u8d77\u52d5\u3067\u554f\u984c\u306a\u3044\u306e\u304b\uff1f<br \/>\n\u6700\u5c0f\u5b9f\u884c\u5358\u4f4d\u306f\u3069\u306e\u5358\u4f4d\u306b\u306a\u308b\u304b\uff1f<\/p>\n<p>\u57fa\u672c\u3001\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u30b7\u30b9\u30c6\u30e0\u306f\u5e38\u6642\u5b9f\u884c\u3068\u306a\u308b\u305f\u3081\u3001\u6700\u5c0f\u3067\u3082\u3053\u308c\u306f\u5e38\u306b\u5fc5\u8981\u3068\u660e\u78ba\u306b\u3059\u308b\u3053\u3068\u304c\u5fc5\u8981<\/p>\n<p>\u30e1\u30c3\u30bb\u30fc\u30b8\u51e6\u7406\u30bb\u30de\u30f3\u30c6\u30a3\u30af\u30b9<\/p>\n<p>\u30e1\u30c3\u30bb\u30fc\u30b8\u306e\u51e6\u7406\u30bb\u30de\u30f3\u30c6\u30a3\u30af\u30b9\u3068\u3057\u3066\u3001\u3069\u308c\u3092\u9069\u7528\u3059\u308c\u3070\u3044\u3044\u306e\u304b\uff1f<\/p>\n<p>At most once<br \/>\nAt least once<br \/>\nExactly once<\/p>\n<p>\u5168\u30b1\u30fc\u30b9\u3092\u30ab\u30d0\u30fc\u3059\u308bExactly once\u306f\u5b58\u5728\u3057\u306a\u3044\u305f\u3081\u3001\u305d\u308c\u3092\u5927\u524d\u63d0\u306b\u7f6e\u304f\u5fc5\u8981\u304c\u3042\u308b\u3002<\/p>\n<p>\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u30a8\u30f3\u30b8\u30f3\u3067\u4fdd\u8a3c\u53ef\u80fd\u306a\u306e\u306f\u3001\u300c\u81ea\u8eab\u306e\u5185\u90e8\u72b6\u614b\u304cExactly once\u3067\u51e6\u7406\u3057\u305f\u72b6\u614b\u306b\u306a\u3063\u3066\u3044\u308b\u3053\u3068\u300d<br \/>\n\u5916\u90e8\u3078\u51fa\u529b\u3092\u884c\u3046\u5834\u5408\u306b\u306f\u6d3b\u7528\u3059\u308b\u5074\u3067\u30c7\u30fc\u30bf\u306e\u30ad\u30fc\u3067\u91cd\u8907\u9664\u53bb\u3092\u884c\u3063\u305f\u308a\u3001Accumulation\u65b9\u5f0f\u306b\u3088\u308b\u5bfe\u51e6\u304c\u5fc5\u8981<\/p>\n<p>\u8010\u969c\u5bb3\u6027<\/p>\n<p>\u969c\u5bb3\u304c\u767a\u751f\u3057\u305f\u5834\u5408\u306b\u3069\u308c\u304f\u3089\u3044\u5f71\u97ff\u3084\u9045\u5ef6\u304c\u51fa\u308b\u69cb\u6210\u306b\u306a\u3063\u3066\u3044\u308b\u304b\uff1f<br \/>\n\u969c\u5bb3\u767a\u751f\u6642\u306f\u81ea\u52d5\u3067\u5fa9\u65e7\u3059\u308b\u306e\u304b\uff1f\u305d\u308c\u3068\u3082\u624b\u52d5\u304b\uff1f<br \/>\n\u969c\u5bb3\u5fa9\u65e7\u306b\u304b\u304b\u308b\u6642\u9593\u306f\u3069\u308c\u304f\u3089\u3044\u304b\uff1f<\/p>\n<p>\u30d0\u30c3\u30af\u30d7\u30ec\u30c3\u30b7\u30e3\u30fc\u6a5f\u80fd\u306e\u6709\u7121<\/p>\n<p>\u69cb\u6210\u8981\u7d20\u9593\u3067\u30b9\u30eb\u30fc\u30d7\u30c3\u30c8\u306e\u5dee\u5206\u304c\u767a\u751f\u3057\u305f\u5834\u5408\u306b\u3001\u4e0b\u6d41\u306b\u5408\u308f\u305b\u3066\u6d41\u91cf\u3092\u5236\u5fa1\u3059\u308b\u6a5f\u80fd<br \/>\n\u3053\u306e\u6a5f\u80fd\u304c\u306a\u3044\u5834\u5408\u3001\u4e8b\u524d\u306b\u3069\u308c\u3060\u3051\u306e\u30b9\u30eb\u30fc\u30d7\u30c3\u30c8\u304c\u51fa\u308b\u304b\u3092\u898b\u5207\u3063\u3066\u304a\u304f\u5fc5\u8981\u304c\u51fa\u308b\u3002<\/p>\n<p>\u30bb\u30ad\u30e5\u30ea\u30c6\u30a3<\/p>\n<p>\u4fdd\u5b58\u3059\u308b\u30c7\u30fc\u30bf\u306e\u6697\u53f7\u5316\u304c\u5fc5\u8981\u304b\uff1f<\/p>\n<p>\u5fc5\u8981\u306a\u5834\u5408\u3001\u3069\u306e\u30bf\u30a4\u30df\u30f3\u30b0\u3067\u6697\u53f7\u5316\u3057\u3066\u304a\u304f\u304b\uff1f<\/p>\n<p>\u30a2\u30d7\u30ea\u30b1\u30fc\u30b7\u30e7\u30f3\u6bce\u306b\u3069\u306e\u30c7\u30fc\u30bf\u306b\u30a2\u30af\u30bb\u30b9\u53ef\u80fd\u304b\u306e\u5236\u5fa1\u3092\u3069\u3046\u884c\u3046\u304b\uff1f<\/p>\n<p>\u30c7\u30fc\u30bf\u306e\u51e6\u7406\u30e2\u30c7\u30eb<\/p>\n<p>\u30ec\u30b3\u30fc\u30c9\u6bce\u306e\u51e6\u7406\u304b\uff1f\uff08\u30de\u30a4\u30af\u30ed\uff09\u30d0\u30c3\u30c1\u30e2\u30c7\u30eb\u304b\uff1f<\/p>\n<p>\u30ec\u30b3\u30fc\u30c9\u6bce\u51e6\u7406\u3059\u308b\u30e2\u30c7\u30eb<\/p>\n<p>\u4f4e\u30ec\u30a4\u30c6\u30f3\u30b7\u304c\u7279\u5fb4\u3060\u304c\u3001\u30b9\u30eb\u30fc\u30d7\u30c3\u30c8\u3092\u51fa\u305b\u308b\u304b\u306f\u5b9f\u884c\u30a8\u30f3\u30b8\u30f3\u306b\u4f9d\u5b58\u3002<br \/>\nOut of order\u306a\u30c7\u30fc\u30bf\u306e\u53d6\u308a\u6271\u3044\u3082\u884c\u3044\u3084\u3059\u3044\u3002<\/p>\n<p>\uff08\u30de\u30a4\u30af\u30ed\uff09\u30d0\u30c3\u30c1\u30e2\u30c7\u30eb<\/p>\n<p>\u30d0\u30c3\u30c1\u81ea\u4f53\u306b\u8d77\u52d5\u7d42\u4e86\u30fb\u72b6\u614b\u4fdd\u5b58\u51e6\u7406\u3082\u4f34\u3046\u305f\u3081\u3001\u30de\u30a4\u30af\u30ed\u30d0\u30c3\u30c1\u306f\u975e\u5e38\u306b\u4e2d\u9014\u534a\u7aef\u306a\u72b6\u614b\u306b\u306a\u308a\u304c\u3061\u3002<\/p>\n<p>\u958b\u767aAPI<\/p>\n<p>\u3069\u306e\u3088\u3046\u306a\u62bd\u8c61\u5ea6\u306e\u958b\u767a\u304c\u53ef\u80fd\u304b\uff1f<\/p>\n<p>\u958b\u767a\u30e1\u30f3\u30d0\u306e\u30b9\u30ad\u30eb\u3084\u512a\u5148\u3059\u3079\u304d\u3082\u306e\uff08\u30ab\u30b9\u30bf\u30de\u30a4\u30ba\u6027\uff1f\u958b\u767a\u30b9\u30d4\u30fc\u30c9\uff1f\uff09\u7b49\u306b\u3088\u3063\u3066\u691c\u8a0e\u304c\u5fc5\u8981<\/p>\n<p>\u958b\u767a\u6642\u306eAPI\u306f\u4e3b\u306b\u4e0b\u8a18\u306e3\u3064\u3002<\/p>\n<p>\u5ba3\u8a00\u578b\u5b9a\u7fa9\uff08\u958b\u767a\u30b9\u30d4\u30fc\u30c9\u3007\u3001\u30ab\u30b9\u30bf\u30de\u30a4\u30ba\u6027\u3007\uff09<\/p>\n<p>map()\u3001filter()\u306a\u3069\u306e\u30c7\u30fc\u30bf\u30b9\u30c8\u30ea\u30fc\u30e0\u306b\u5bfe\u3059\u308b\u51e6\u7406\u3092\u5ba3\u8a00<\/p>\n<p>\u547d\u4ee4\u578b\u5b9a\u7fa9\uff08\u958b\u767a\u30b9\u30d4\u30fc\u30c9\u25b3\u3001\u30ab\u30b9\u30bf\u30de\u30a4\u30ba\u6027\u25ce\uff09<\/p>\n<p>process(event)\u3068\u3044\u3046\u5f62\u3067\u5404\u30a4\u30d9\u30f3\u30c8\u30ec\u30d9\u30eb\u3067\u51e6\u7406\u3092\u5b9f\u88c5<\/p>\n<p>Streaming SQL\uff08\u958b\u767a\u30b9\u30d4\u30fc\u30c9\u25ce\u3001\u30ab\u30b9\u30bf\u30de\u30a4\u30ba\u6027\u25b3\uff09<\/p>\n<p>\u30c7\u30fc\u30bf\u30b9\u30c8\u30ea\u30fc\u30e0\u306b\u5bfe\u3057\u3066\u9069\u7528\u3059\u308b\u30af\u30a8\u30ea(STREAM SELECT FROM WHERE&#8230;.)<\/p>\n<p>\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u3068\u30d0\u30c3\u30c1\u51e6\u7406\u3092\u540c\u4e00\u306eAPI\u3067\u958b\u767a\u53ef\u80fd\u306a\u306e\u304b\uff1f<\/p>\n<p>\u3069\u306e\u3088\u3046\u306a\u62e1\u5f35\u30e9\u30a4\u30d6\u30e9\u30ea\u304c\u5fc5\u8981\u304b\uff1f<\/p>\n<p>\u6a5f\u68b0\u5b66\u7fd2<br \/>\n\u30b0\u30e9\u30d5\u51e6\u7406<br \/>\n\u5916\u90e8\u30b3\u30f3\u30dd\u30fc\u30cd\u30f3\u30c8\u3068\u306e\u63a5\u7d9a<\/p>\n<p>UI\u306e\u5145\u5b9f\u6027<\/p>\n<p>\u5927\u62b5\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u30a8\u30f3\u30b8\u30f3\u306b\u306fUI\u3082\u7d44\u307f\u8fbc\u307e\u308c\u3066\u3044\u308b<\/p>\n<p>UI\u304b\u3089\u3069\u306e\u3088\u3046\u306a\u60c5\u5831\u304c\u898b\u308c\u308b\u304b\uff1f<br \/>\nUI\u304b\u3089\u3069\u306e\u3088\u3046\u306a\u30aa\u30da\u30ec\u30fc\u30b7\u30e7\u30f3\u304c\u53ef\u80fd\u304b\uff1f<\/p>\n<p>\u5b9f\u884c\u30b0\u30e9\u30d5\u304c\u8868\u793a\u3055\u308c\u308b\u306e\u3082\u6709\u7528<\/p>\n<p>Shuffle\u306e\u56de\u6570\u306a\u3069\u304c\u308f\u304b\u308c\u3070\u3001\u975e\u52b9\u7387\u306a\u51e6\u7406\u3092\u884c\u3063\u3066\u3044\u308b\u304b\u306b\u3064\u3044\u3066\u3082\u89e3\u6790\u3057\u3084\u3059\u3044\u3002<\/p>\n<p>\u958b\u767a\u30b5\u30a4\u30af\u30eb\u3068\u30aa\u30da\u30ec\u30fc\u30b7\u30e7\u30f3<\/p>\n<p>\u30a2\u30d7\u30ea\u30b1\u30fc\u30b7\u30e7\u30f3\u3068\u3044\u3046\u5f62\u3067\u30ed\u30fc\u30ab\u30eb\u3067\u5b9f\u884c\u3067\u304d\u308b\u304b\uff1f\u305d\u308c\u3068\u3082\u30af\u30e9\u30b9\u30bf\u4e0a\u3067\u306a\u3044\u3068\u5b9f\u884c\u3067\u304d\u306a\u3044\u304b\uff1f<\/p>\n<p>\u30ed\u30fc\u30ab\u30eb\u3067\u5b9f\u884c\u3067\u304d\u308b\u5834\u5408\u3001\u30c6\u30b9\u30c8\u30b3\u30fc\u30c9\uff1e\u30af\u30e9\u30b9\u30bf\u74b0\u5883\u306e\u5dee\u5206\u3092\u57cb\u3081\u308b\u3053\u3068\u304c\u51fa\u6765\u308b\u3002<\/p>\n<p>\u4e8b\u524d\u306b\u4f55\u304b\u3057\u3089\u306e\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u304c\u5fc5\u8981\u306a\u306e\u304b\uff1f\u305d\u308c\u3068\u3082\u30ea\u30bd\u30fc\u30b9\u30de\u30cd\u30fc\u30b8\u30e3\u4e0a\u306b\u30c7\u30d7\u30ed\u30a4\u3059\u308c\u3070\u3044\u3044\u306e\u304b\uff1f<br \/>\n\u30ed\u30b0\u3084\u30a8\u30e9\u30fc\u306e\u96c6\u7d04\u304c\u53ef\u80fd\u306a\u306e\u304b\uff1f<\/p>\n<p>\u5206\u6563\u51e6\u7406\u3067\u500b\u3005\u306e\u30b5\u30fc\u30d0\u304b\u3089\u30ed\u30b0\u3092\u53d6\u5f97\u3057\u3066\u78ba\u8a8d\u3059\u308b\u306e\u306f\u7121\u8b00\u3002<br \/>\n\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u30a8\u30f3\u30b8\u30f3\u306e\u6a5f\u80fd\u304b\u3001\u307e\u305f\u306f\u72ec\u81ea\u3067\u30ed\u30b0\u3084\u30a8\u30e9\u30fc\u3092\u4e00\u304b\u6240\u3067\u898b\u308c\u308b\u3088\u3046\u306b\u3057\u3066\u304a\u304f\u5fc5\u8981\u304c\u3042\u308b\u3002<\/p>\n<p>\u30a2\u30d7\u30ea\u30b1\u30fc\u30b7\u30e7\u30f3\u66f4\u65b0\u6642\u306b\u518d\u8d77\u52d5\u3057\u3066\u3044\u3044\u306e\u304b\uff1f\u305d\u308c\u3068\u3082No downtime\u3067\u30ed\u30fc\u30ea\u30f3\u30b0\u30a2\u30c3\u30d7\u30c7\u30fc\u30c8\u304c\u5fc5\u8981\u306a\u306e\u304b\uff1f<br \/>\n\u65e2\u306b\u4f7f\u7528\u3057\u3066\u3044\u308b\u958b\u767a\u30c4\u30fc\u30eb\u3068\u3069\u308c\u3060\u3051\u89aa\u548c\u6027\u304c\u3042\u308b\u304b\uff1f<br \/>\n\u3044\u3056\u3068\u3044\u3046\u6642\u306b\u89e3\u6790\u304c\u53ef\u80fd\u306a\u3088\u3046\u3001\u958b\u767a\u30c1\u30fc\u30e0\u304c\u7406\u89e3\u3067\u304d\u308b\u8a00\u8a9e\u3067\u958b\u767a\u3055\u308c\u3066\u3044\u308b\u304b\uff1f<\/p>\n<h2>\u5b9e\u9645\u4e0a\uff0c\u5728\u6784\u5efa\u7cfb\u7edf\u65f6\u4f1a\u51fa\u73b0\u54ea\u4e9b\u5730\u65b9\u4f1a\u6210\u4e3a\u74f6\u9888\uff1f<\/h2>\n<p>\u5f53\u6211\u4eec\u771f\u6b63\u6784\u5efa\u6d41\u5904\u7406\u7cfb\u7edf\u7684\u65f6\u5019\uff0c\u4f1a\u9047\u5230\u5404\u79cd\u74f6\u9888\u95ee\u9898\u3002<br \/>\n\u4ee5\u4e0b\u662f\u5176\u4e2d\u7684\u4e00\u4e2a\u4ee3\u8868\u6027\u74f6\u9888\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u30d5\u30a1\u30a4\u30eb\u30a2\u30af\u30bb\u30b9<\/ul>\n<\/li>\n<\/ul>\n<p>\u521d\u671f\u306e\u7406\u89e3\u304c\u6d45\u3044\u6642\u671f\u306b\u4e3b\u306b\u3088\u304f\u306f\u307e\u308b\u3002<br \/>\n\u30c7\u30fc\u30bf\u3092\u51e6\u7406\u3059\u308b\u305f\u3073\u306b\u30d5\u30a1\u30a4\u30eb\u30a2\u30af\u30bb\u30b9\u304c\u767a\u751f\u3059\u308b\u3088\u3046\u306a\u6a5f\u80fd\u306e\u5834\u5408\u3001\u305d\u3053\u304c\u30dc\u30c8\u30eb\u30cd\u30c3\u30af\u306b\u306a\u308b\u3002<br \/>\n\u30ad\u30e3\u30c3\u30b7\u30e5\u3092\u6d3b\u7528\u3059\u308b\u306a\u3069\u3057\u3066\u540c\u671f\u7684\u306a\u30d5\u30a1\u30a4\u30eb\u30a2\u30af\u30bb\u30b9\u3092\u767a\u751f\u3057\u306a\u3044\u3088\u3046\u306b\u3059\u308b\u5fc5\u8981\u304c\u3042\u308b\u3002<\/p>\n<p>\u4e0d\u8981\u306a\u30d7\u30ed\u30bb\u30b9\u9593\u306e\u901a\u4fe1<\/p>\n<p>\u30d0\u30c3\u30c1\u51e6\u7406\u3067\u3082\u540c\u69d8\u3060\u304c\u3001Shuffle\u3092\u5b9f\u884c\u3057\u305f\u6642\u306e\u30d7\u30ed\u30bb\u30b9\u9593\u306e\u901a\u4fe1\u8ca0\u8377\u306f\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u3067\u3082\u5927\u304d\u3044\u3002<br \/>\n\u4e8b\u524d\u306b\u30d7\u30ed\u30bb\u30b9\u5185\u3067\u96c6\u7d04\u3057\u3066\u304a\u304f\u306a\u3069\u3082\u884c\u3044\u3001\u30d7\u30ed\u30bb\u30b9\u9593\u306e\u901a\u4fe1\u983b\u5ea6\u3068\u91cf\u3092\u6291\u3048\u308b\u3053\u3068\u3002<\/p>\n<p>\u30e1\u30c3\u30bb\u30fc\u30b8\u30d0\u30b9\u306e\u51e6\u7406\u6027\u80fd\u304c\u8ffd\u3044\u4ed8\u304b\u306a\u304f\u306a\u308b<\/p>\n<p>Kafka\u3067\u3042\u308c\u3070Storm\u3084Spark Streaming\u306e\u6642\u4ee3\u306f\u30e1\u30c3\u30bb\u30fc\u30b8\u30d0\u30b9\u5074\u304c\u30dc\u30c8\u30eb\u30cd\u30c3\u30af\u306b\u306a\u308b\u3053\u3068\u306f\u5c11\u306a\u304b\u3063\u305f\u3002<br \/>\nFlink\u3084Apex\u306e\u4e16\u4ee3\u306b\u306a\u3063\u3066\u304f\u308b\u3068Kafka\u306e\u6027\u80fd\u3092\u632f\u308a\u5207\u308b\u3053\u3068\u3082\u591a\u304f\u306a\u3063\u3066\u304d\u3066\u3044\u308b\u3002<\/p>\n<p>\u30e1\u30c3\u30bb\u30fc\u30b8\u30d0\u30b9\u5074\u306e\u30af\u30e9\u30b9\u30bf\u30b5\u30a4\u30ba\u8abf\u6574\u3084\u30ec\u30d7\u30ea\u30b1\u30fc\u30b7\u30e7\u30f3\u8a2d\u5b9a\u3092\u898b\u76f4\u3059\u3002<\/p>\n<p>GC<\/p>\n<p>JVM\u4e0a\u3067\u52d5\u4f5c\u3059\u308b\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u30a8\u30f3\u30b8\u30f3\u304c\u591a\u3044\u305f\u3081\u3001\u3069\u3046\u3057\u3066\u3082\u3076\u3064\u304b\u308b\u554f\u984c\u3002<\/p>\n<p>\u307e\u305a\u306fJVM\u306e\u30c1\u30e5\u30fc\u30cb\u30f3\u30b0\u3092\u884c\u3046\u3002<br \/>\n\u305d\u308c\u3067\u3082\u3069\u3046\u3057\u3088\u3046\u3082\u306a\u3044\u5834\u5408<\/p>\n<p>\u30a2\u30d7\u30ea\u30b1\u30fc\u30b7\u30e7\u30f3\u3067\u306e\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u306e\u751f\u6210\u6570\u3092\u6975\u529b\u6291\u3048\u308b\u3002<br \/>\n\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u306b\u500b\u3005\u306e\u30d5\u30a3\u30fc\u30eb\u30c9\u3067\u306f\u306a\u304f\u30d0\u30a4\u30c8\u914d\u5217\u3067\u307e\u3068\u3081\u3066\u6301\u305f\u305b\u308b\u3002<\/p>\n<h2>\u5bf9\u4e8e\u6d41\u5904\u7406\u7684\u8bef\u89e3<\/h2>\n<p>\u6211\u5c06\u89e3\u91ca\u5173\u4e8e\u6d41\u5904\u7406\u5e38\u89c1\u7684\u8bef\u89e3\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u3067\u306f\u8fd1\u4f3c\u5024\u3057\u304b\u51fa\u305b\u305a\u3001\u30d0\u30c3\u30c1\u3068\u306e\u7d44\u307f\u5408\u308f\u305b\u304c\u5fc5\u9808\u3067\u3042\u308b\u3002<\/ul>\n<\/li>\n<\/ul>\n<p>\u521d\u671f\u306eStorm\u3067\u306f\u5b9f\u969b\u305d\u3046\u3067\u3001Lambda Architecture\u3068\u3044\u3046\u5f62\u3067\u5bfe\u51e6\u304c\u3068\u3089\u308c\u3066\u3044\u307e\u3057\u305f\u3002<br \/>\n\u4eca\u3067\u306fWatermark\u3001Trigger\u306a\u3069\u3092\u4f7f\u3063\u3066\u305d\u308c\u3082\u5236\u5fa1\u53ef\u80fd\u3067\u3059\u3002<\/p>\n<p>\u30ec\u30a4\u30c6\u30f3\u30b7\u3068\u30b9\u30eb\u30fc\u30d7\u30c3\u30c8\u306f\u3069\u3061\u3089\u304b\u3092\u9078\u3076\u5fc5\u8981\u304c\u3042\u308b\u3002<\/p>\n<p>\u305d\u308c\u3082\u3001\u521d\u671f\u306eStorm VS Spark Streaming\u306e\u6bd4\u8f03\u304b\u3089\u6765\u3066\u3044\u308b\u8a71\u3067\u3059\u3002<br \/>\n\u5b9f\u969b\u30d7\u30ed\u30b0\u30e9\u30e0\u4e0a\u3067\u306f\u305d\u306e2\u8ef8\u306e\u30c8\u30ec\u30fc\u30c9\u30aa\u30d5\u306f\u3042\u308a\u307e\u305b\u3093\u3002<\/p>\n<p>\u305f\u3060\u3057\u3001\u4eca\u306f\u65b0\u305f\u306a\u30c8\u30ec\u30fc\u30c9\u30aa\u30d5\u304c\u3042\u308b\u305f\u3081\u3001\u300c\u305d\u306e\u5bfe\u51e6\u3067\u5168\u90e8\u306b\u5bfe\u5fdc\u3059\u308b\u306e\u306f\u7121\u7406\u306a\u3093\u3058\u3083\u306a\u3044\u306e\uff1f\u300d\u3092\u53c2\u7167<\/p>\n<p>\u30de\u30a4\u30af\u30ed\u30d0\u30c3\u30c1\u65b9\u5f0f\u306e\u65b9\u304c\u30b9\u30eb\u30fc\u30d7\u30c3\u30c8\u304c\u9ad8\u3044\u3002<\/p>\n<p>\u30ec\u30b3\u30fc\u30c9\u6bce\u51e6\u7406\u3067\u3042\u3063\u3066\u3082\u30c7\u30fc\u30bf\u306f\u30d0\u30c3\u30d5\u30a1\u30ea\u30f3\u30b0\u3055\u308c\u3066\u3044\u308b\u305f\u3081\u3001\u5b9f\u969b\u306b\u306f\u305d\u3046\u3067\u306f\u306a\u3044\u3088\u3046\u3067\u3059\u3002<br \/>\n\u30de\u30a4\u30af\u30ed\u30d0\u30c3\u30c1\u65b9\u5f0f\u306f\u30d0\u30c3\u30c1\u3068\u3057\u3066\u533a\u5207\u308b\u3053\u3068\u306b\u3088\u308b\u7ba1\u7406\u30b3\u30b9\u30c8\u304c\u767a\u751f\u3059\u308b\u305f\u3081\u3001\u4eca\u3068\u306a\u3063\u3066\u306f\u5229\u70b9\u304c\u8584\u3044\u3068\u8a00\u308f\u308c\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>Exactly once\u306f\u4e0d\u53ef\u80fd<\/p>\n<p>\u5b9f\u969b\u306b\u5168\u30b1\u30fc\u30b9\u3078\u306e\u5bfe\u5fdc\u306f\u4e0d\u53ef\u80fd\u3067\u3059\u304c\u3001\u4e0b\u8a18\u306e2\u30b1\u30fc\u30b9\u3092\u5206\u3051\u3066\u8003\u3048\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n<p>\u300c\u6b63\u78ba\u306b1\u56de\u5b9f\u884c\u3055\u308c\u305f\u3068\u3044\u3046\u72b6\u614b\u300d\u3092\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u30a8\u30f3\u30b8\u30f3\u5074\u306b\u7dad\u6301\u3059\u308b\u3053\u3068\u306f\u53ef\u80fd\u3067\u3059\u3002<br \/>\n\u300c\u6b63\u78ba\u306b1\u56de\u5916\u90e8\u306b\u901a\u77e5\u300d\u306f\u4ed6\u306e\u30c7\u30fc\u30bf\u30b9\u30c8\u30a2\u306a\u3069\u3068\u306e\u91cd\u8907\u9664\u53bb\u6a5f\u69cb\u3092\u304b\u307e\u305b\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n<p>\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u306f\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u7cfb\u306e\u51e6\u7406\u306b\u3057\u304b\u9069\u7528\u3067\u304d\u306a\u3044<\/p>\n<p>\u300c\u30d0\u30c3\u30c1\u51e6\u7406\u3068\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u306e\u9055\u3044\u300d\u306b\u3042\u308b\u3068\u304a\u308a\u3001\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u306f\u30d0\u30c3\u30c1\u51e6\u7406\u306e\u6a5f\u80fd\u7684\u306a\u4e0a\u4f4d\u4e92\u63db\u306b\u306a\u308a\u307e\u3059\u3002<br \/>\n\u305d\u306e\u305f\u3081\u3001\u52b9\u7387\u306f\u5225\u9014\u78ba\u8a8d\u304c\u5fc5\u8981\u3067\u3059\u304c\u3001\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u7cfb\u306e\u51e6\u7406\u3067\u306f\u306a\u304f\u3066\u3082\u9069\u7528\u53ef\u80fd\u3002<\/p>\n<p>\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u306f\u96e3\u3057\u3044<\/p>\n<p>\u7121\u9650\u306e\u30c7\u30fc\u30bf\u30bd\u30fc\u30b9\u304c\u3042\u308a\u3001\u30c7\u30fc\u30bf\u304c\u983b\u7e41\u306b\u5909\u308f\u308b\u306a\u3089\u30e2\u30c7\u30eb\u3067\u5bfe\u5fdc\u3059\u308b\u3088\u308a\u3080\u3057\u308d\u5bfe\u5fdc\u3057\u3084\u3059\u3044\u3067\u3059\u3002<br \/>\n\u521d\u671f\u306e\u30b9\u30c8\u30ea\u30fc\u30e0\u51e6\u7406\u306f\u547d\u4ee4\u578b\u5b9a\u7fa9\u306e\u307f\u3067\u958b\u767a\u30b3\u30b9\u30c8\u304c\u9ad8\u304b\u3063\u305f\u3067\u3059\u304c\u3001\u6700\u8fd1\u306f\u5ba3\u8a00\u578b\u5b9a\u7fa9\u3084SQL\u306b\u3088\u308b\u5b9a\u7fa9\u3082\u53ef\u80fd\u306b\u306a\u3063\u3066\u3044\u308b\u305f\u3081\u3001\u30cf\u30fc\u30c9\u30eb\u306f\u4e0b\u304c\u3063\u3066\u3044\u308b\u304b\u3068\u3002<\/p>\n<h1>2016\u5e74\u53d1\u751f\u4e86\u54ea\u4e9b\u4e0e\u6d41\u5904\u7406\u76f8\u5173\u7684\u4e8b\u60c5\uff1f<\/h1>\n<p>\u5728\u8fd9\u4e00\u7ae0\u4e2d\uff0c\u6211\u5c06\u603b\u7ed32016\u5e74\u53d1\u751f\u7684\u6d41\u5904\u7406\u65b9\u9762\u7684\u4e00\u4e9b\u91cd\u8981\u4e8b\u4ef6\u3002<br \/>\n\u4e5f\u5c31\u662f\u8bf4\uff0c\u6211\u4eec\u5c06\u56de\u987e\u201c2016\u5e74\u7684\u91cd\u5927\u4e8b\u4ef6\u201d\u3002<br \/>\n\uff08\u203b\u4e2a\u4eba\u5370\u8c61\u8f83\u4e3a\u6df1\u523b\u7684\u4e8b\u4ef6\uff09\u3002<\/p>\n<h2>Apache Beam\u7684\u63a8\u51fa<\/h2>\n<p>\u8fd9\u4e2a\u5bf9\u6211\u6765\u8bf4\u662f\u6700\u5927\u7684\u3002<\/p>\n<p>\u5728\u89e3\u91ca\u4e86\u6d41\u5904\u7406\u662f\u6279\u5904\u7406\u7684\u529f\u80fd\u4e0a\u4f4d\u4e92\u6362\u4e4b\u540e\uff0c\u4ed6\u4eec\u901a\u8fc7DSL\u5b9a\u4e49\u4e86\u6d41\u5904\u7406\u548c\u6279\u5904\u7406\uff0c\u5f00\u53d1\u51fa\u53ef\u4ee5\u5728\u591a\u4e2a\u73af\u5883\u4e0b\u5171\u4eab\u6267\u884c\u7684\u4ea7\u54c1\uff0c\u6574\u4e2a\u8fc7\u7a0b\u6d41\u7545\u800c\u4ee4\u4eba\u6fc0\u52a8\u3002<\/p>\n<p>\u800c\u4e14\uff0c\u7531\u4e8eDSL\u7684\u5b9a\u4e49\u901a\u5e38\u53ef\u4ee5\u4f7f\u7528Java\u6216Scala\u8fdb\u884c\u5f00\u53d1\uff0c\u56e0\u6b64\u53ef\u4ee5\u76f4\u63a5\u4f7f\u7528IDE\u8fdb\u884c\u5f00\u53d1\uff0c\u8fd9\u4e5f\u662f\u4e00\u4e2a\u91cd\u8981\u7684\u4f18\u70b9\u3002<\/p>\n<h2>\u4e9a\u9a6c\u900aKinesis Analytics\u7684\u53d1\u5e03<\/h2>\n<p>\u5c3d\u7ba1\u4ece\u53bb\u5e74\u5f00\u59cb\u5df2\u7ecf\u6709\u9884\u544a\uff0c\u4f46\u968f\u7740Amazon Kinesis Analytics\u4e0a\u7684\u53d1\u5e03\uff0c\u6211\u4eec\u73b0\u5728\u80fd\u591f\u5728\u6258\u7ba1\u670d\u52a1\u4e0a\u4f7f\u7528SQL\u5904\u7406Kinesis\u6570\u636e\u3002\u8fd9\u4f7f\u5f97\u4e4b\u524d\u9700\u8981\u4f7f\u7528AWS Lambda\u6216\u81ea\u5df1\u5904\u7406\u7684\u4efb\u52a1\u53ef\u4ee5\u5728\u6258\u7ba1\u670d\u52a1\u4e0a\u8fd0\u884c\u3002<\/p>\n<p>\u5f53\u7136\uff0c\u867d\u7136\u6709\u9650\u5236\u7684\u8303\u56f4\uff0c\u4f46\u80fd\u591f\u8f7b\u677e\u5730\u5b9e\u73b0\u662f\u975e\u5e38\u91cd\u8981\u7684\u3002<\/p>\n<h1>\u4f60\u8ba4\u4e3a\u660e\u5e74\u4f1a\u600e\u4e48\u6837\uff1f<\/h1>\n<p>\u8bf7\u4e0d\u8981\u628a\u8fd9\u4e00\u7ae0\u5b8c\u5168\u5f53\u4f5c\u6211\u7684\u5984\u60f3\uff0c\u7edd\u5bf9\u4e0d\u8981\u8ba4\u771f\u5bf9\u5f85\u3002<br \/>\n\u5e76\u4e14\uff0c\u5176\u4e2d\u8fd8\u63ba\u6742\u4e86\u5df2\u7ecf\u5ba3\u5e03\u8981\u505a\u7684\u5185\u5bb9\uff0c\u6240\u4ee5\u4e5f\u8bf7\u6ce8\u610f\u3002<\/p>\n<p>\u6211\u5bf9\u4eca\u5e74Apache Beam\u7684\u63a8\u51fa\u611f\u5230\u975e\u5e38\u6ee1\u8db3\uff0c\u611f\u89c9\u7f3a\u4e86\u4e00\u4e2a\u91cd\u8981\u7684\u7ec4\u6210\u90e8\u5206\u7ec8\u4e8e\u5b8c\u6574\u4e86\u3002<br \/>\n\u56e0\u6b64\uff0c\u5c3d\u7ba1\u6784\u6210\u65b9\u9762\u6ca1\u6709\u592a\u5927\u53d8\u5316\uff0c\u4f46\u6211\u4eec\u9884\u8ba1\u660e\u5e74\u4f1a\u6709\u4e00\u4e2a\u7a33\u5b9a\u7684\u6f14\u8fdb\u3002<\/p>\n<p>\u4f5c\u4e3a\u4e00\u79cd\u786e\u5b9e\u7684\u8fdb\u5316\u65b9\u5411\uff0c\u6211\u4e2a\u4eba\u731c\u6d4b\u53ef\u80fd\u51fa\u73b0\u7684\u662f\u66f4\u5bb9\u6613\u5f00\u53d1\u7684\u65b9\u5411\u4ee5\u53ca\u5bf9\u786c\u4ef6\u8fdb\u5316\u7684\u8ffd\u968f\u65b9\u5411\u3002<\/p>\n<p>\u5f00\u53d1\u53d8\u5f97\u66f4\u5bb9\u6613\u7684\u65b9\u5f0f\u6709API\u7684\u4e30\u5bcc\u548cUI\u578b\u4ea7\u54c1\u7684\u6269\u5927\u3002\u6d41\u5904\u7406\u5f15\u64ce\u7684\u5f00\u53d1\u5c06\u66f4\u5bb9\u6613\uff0c\u6309\u7167\u547d\u4ee4\u5f0f\u5b9a\u4e49&gt;\u58f0\u660e\u5f0f\u5b9a\u4e49&gt;SQL\u5b9a\u4e49\u7684\u65b9\u5f0f\u8fdb\u884c\uff0c\u4f46\u80fd\u63d0\u4f9b\u5b8c\u6574\u8bbe\u7f6e\u7684\u5f15\u64ce\u662f\u6709\u9650\u7684\u3002\u56e0\u6b64\uff0c\u6211\u4eec\u9884\u8ba1\u4f1a\u6709\u591a\u4e2a\u5f15\u64ce\u6765\u4e30\u5bcc\u5176API\u3002<\/p>\n<p>\u5173\u4e8eUI\u578b\u4ea7\u54c1\u7684\u6269\u5c55\uff0c\u5728NiFi\u548cSpring Cloud Dataflow\u4e2d\uff0c\u53ef\u4ee5\u4fdd\u5b58\u4eceUI\u5b9a\u4e49\u7684\u72b6\u6001\uff0c\u4f46\u662f\u4eceUI\u8fdb\u884c\u5b9a\u4e49\u610f\u5473\u7740\u65e0\u6cd5\u901a\u8fc7\u4ee3\u7801\u8fdb\u884c\u7ba1\u7406\uff0c\u4f7f\u7528\u8fc7\u7a0b\u4e2d\u53ef\u80fd\u4f1a\u51fa\u95ee\u9898\u3002\u56e0\u6b64\uff0c\u6211\u8ba4\u4e3a\u5e94\u8be5\u91c7\u7528\u6613\u4e8e\u4e0a\u624b\u5e76\u4e14\u4f7f\u7528\u8fc7\u7a0b\u4e2d\u5bb9\u6613\u7ef4\u62a4\u7684\u7ed3\u6784\u3002<\/p>\n<p>\u8ffd\u8ff0\u786c\u4ef6\u8fdb\u5316\u7684\u610f\u601d\u662f\u8ffd\u968fManyCore\u3001\u5927\u5bb9\u91cf\u5185\u5b58\u7684\u786c\u4ef6\u8fdb\u5316\uff0c\u4f46\u5b9e\u9645\u4e0a\u53ef\u80fd\u4e5f\u662f\u4e3a\u4e86\u5728\u5176\u4e0a\u8fd0\u884c\u65f6\u6d17\u6e05\u6d41\u5904\u7406\u4e2d\u4f1a\u51fa\u73b0\u7684\u95ee\u9898\u3002\u4f8b\u5982\uff0c\u6279\u5904\u7406\u5728Boosting Spark\u6027\u80fd\u4e0a\u53ef\u80fd\u51fa\u73b0\u65b0\u786c\u4ef6\u4e0a\u7684\u95ee\u9898\u3002\u76ee\u524d\u8fd8\u4e0d\u6e05\u695a\u6d41\u5904\u7406\u662f\u5426\u5b58\u5728\u76f8\u540c\u7ea7\u522b\u7684\u95ee\u9898\uff0c\u4e5f\u6ca1\u6709\u5f97\u5230\u592a\u591a\u6709\u5173\u6d41\u5904\u7406\u5f15\u64ce\u65b9\u9762\u7684\u4fe1\u606f\u3002\u8fd9\u53ef\u80fd\u662f\u6211\u53ea\u662f\u65e0\u77e5\u7684\u53ef\u80fd\u6027\u8f83\u5927\u3002<\/p>\n<p>\u5c3d\u7ba1\u6211\u52aa\u529b\u5bfb\u627e\u8bba\u6587\uff0c\u4f46\u76f4\u63a5\u627e\u5230\u6240\u9700\u7684\u8fd8\u672a\u6210\u529f\u3002<br \/>\n\u867d\u7136MES 2016\u7684\u8bba\u6587\u4f3c\u4e4e\u76f8\u8fd1\uff0c\u4f46\u7531\u4e8e\u5c1a\u672a\u9605\u8bfb\uff0c\u6211\u65e0\u6cd5\u786e\u5b9a\u786e\u5207\u60c5\u51b5\u3002<\/p>\n<p>\u65e0\u8bba\u5982\u4f55\uff0c\u5c31\u662f\u8fd9\u4e2a\u611f\u89c9\u3002<\/p>\n<h1>\u603b\u7ed3<\/h1>\n<p>\u611f\u8c22\u60a8\u9605\u8bfb\u8fd9\u7bc7\u957f\u6587\u3002<\/p>\n<p>\u9996\u5148\uff0c\u6211\u89e3\u91ca\u4e86\u6d41\u5904\u7406\u7684\u6982\u5ff5\u3002\u7136\u540e\uff0c\u6211\u603b\u7ed3\u4e86\u5728\u5b9e\u9645\u7cfb\u7edf\u6784\u5efa\u4e2d\u6240\u9047\u5230\u7684\u95ee\u9898\u548c\u9700\u8981\u8003\u8651\u7684\u8981\u70b9\u3002\u6700\u540e\uff0c\u6211\u8fd8\u5199\u4e86\u4e00\u4e9b\u5173\u4e8e\u4e2a\u4eba\u57282016\u5e74\u7684\u7ecf\u5386\u4ee5\u53ca\u660e\u5e74\u53ef\u80fd\u4f1a\u53d1\u751f\u7684\u4e8b\u60c5\u3002<\/p>\n<p>\u7531\u4e8e\u4e00\u6b21\u6027\u5199\u7684\u4e1c\u897f\uff0c\u53ef\u80fd\u4f1a\u6709\u9519\u8bef\u6216\u4e0d\u5bb9\u6613\u7406\u89e3\u7684\u5730\u65b9\uff0c\u5982\u679c\u6709\u7684\u8bdd\uff0c\u8bf7\u5728\u8bc4\u8bba\u4e2d\u5199\u4e0b\uff0c\u4e0d\u80dc\u611f\u6fc0\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u8fd9\u7bc7\u6587\u7ae0\u8bb2\u8ff0\u4e86\u4ec0\u4e48\u5185\u5bb9\uff1f \u6d41\u5904\u7406\u6700\u521d\u5c31\u662f\u4ece\u67d0\u4e2a\u5730\u65b9\u5f00\u59cb\u7684\uff0c\u5b83\u662f\u5173\u4e8e\u5fc5\u8981\u8003\u8651\u4e8b\u9879\u7684\u4fe1\u606f\u7684\u603b\u7ed3\uff0c\u4f9b\u56de\u987e\u4f7f\u7528\u3002 \u6b64\u5916 [&hellip;]<\/p>\n","protected":false},"author":11,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-47439","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.5 (Yoast SEO v21.5) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>\u4ec0\u4e48\u662f\u6d41\u5904\u7406\uff1f\u4ee5\u53ca2016\u5e74\u53d1\u751f\u7684\u4e8b\u4ef6 - Blog - Silicon Cloud<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.silicloud.com\/zh\/blog\/\u4ec0\u4e48\u662f\u6d41\u5904\u7406\uff1f\u4ee5\u53ca2016\u5e74\u53d1\u751f\u7684\u4e8b\u4ef6\u3002\/\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"\u4ec0\u4e48\u662f\u6d41\u5904\u7406\uff1f\u4ee5\u53ca2016\u5e74\u53d1\u751f\u7684\u4e8b\u4ef6\" \/>\n<meta property=\"og:description\" content=\"\u8fd9\u7bc7\u6587\u7ae0\u8bb2\u8ff0\u4e86\u4ec0\u4e48\u5185\u5bb9\uff1f \u6d41\u5904\u7406\u6700\u521d\u5c31\u662f\u4ece\u67d0\u4e2a\u5730\u65b9\u5f00\u59cb\u7684\uff0c\u5b83\u662f\u5173\u4e8e\u5fc5\u8981\u8003\u8651\u4e8b\u9879\u7684\u4fe1\u606f\u7684\u603b\u7ed3\uff0c\u4f9b\u56de\u987e\u4f7f\u7528\u3002 \u6b64\u5916 [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.silicloud.com\/zh\/blog\/\u4ec0\u4e48\u662f\u6d41\u5904\u7406\uff1f\u4ee5\u53ca2016\u5e74\u53d1\u751f\u7684\u4e8b\u4ef6\u3002\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog - Silicon Cloud\" \/>\n<meta property=\"article:published_time\" content=\"2023-04-16T05:36:35+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-04-29T01:17:08+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/30-0.png\" \/>\n<meta name=\"author\" content=\"\u65b0, \u97f5\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"\u65b0, \u97f5\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bb%80%e4%b9%88%e6%98%af%e6%b5%81%e5%a4%84%e7%90%86%ef%bc%9f%e4%bb%a5%e5%8f%8a2016%e5%b9%b4%e5%8f%91%e7%94%9f%e7%9a%84%e4%ba%8b%e4%bb%b6%e3%80%82\/\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bb%80%e4%b9%88%e6%98%af%e6%b5%81%e5%a4%84%e7%90%86%ef%bc%9f%e4%bb%a5%e5%8f%8a2016%e5%b9%b4%e5%8f%91%e7%94%9f%e7%9a%84%e4%ba%8b%e4%bb%b6%e3%80%82\/\",\"name\":\"\u4ec0\u4e48\u662f\u6d41\u5904\u7406\uff1f\u4ee5\u53ca2016\u5e74\u53d1\u751f\u7684\u4e8b\u4ef6 - Blog - Silicon Cloud\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\"},\"datePublished\":\"2023-04-16T05:36:35+00:00\",\"dateModified\":\"2024-04-29T01:17:08+00:00\",\"author\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/4ba4019495123db3038fd0809e6959c9\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bb%80%e4%b9%88%e6%98%af%e6%b5%81%e5%a4%84%e7%90%86%ef%bc%9f%e4%bb%a5%e5%8f%8a2016%e5%b9%b4%e5%8f%91%e7%94%9f%e7%9a%84%e4%ba%8b%e4%bb%b6%e3%80%82\/#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bb%80%e4%b9%88%e6%98%af%e6%b5%81%e5%a4%84%e7%90%86%ef%bc%9f%e4%bb%a5%e5%8f%8a2016%e5%b9%b4%e5%8f%91%e7%94%9f%e7%9a%84%e4%ba%8b%e4%bb%b6%e3%80%82\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bb%80%e4%b9%88%e6%98%af%e6%b5%81%e5%a4%84%e7%90%86%ef%bc%9f%e4%bb%a5%e5%8f%8a2016%e5%b9%b4%e5%8f%91%e7%94%9f%e7%9a%84%e4%ba%8b%e4%bb%b6%e3%80%82\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.silicloud.com\/zh\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"\u4ec0\u4e48\u662f\u6d41\u5904\u7406\uff1f\u4ee5\u53ca2016\u5e74\u53d1\u751f\u7684\u4e8b\u4ef6\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/\",\"name\":\"Blog - Silicon Cloud\",\"description\":\"\",\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/4ba4019495123db3038fd0809e6959c9\",\"name\":\"\u65b0, \u97f5\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d484b6c6e4ae82e8a9efea989e1d2af46d9b6ef128101e63b18f559fca0ae627?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d484b6c6e4ae82e8a9efea989e1d2af46d9b6ef128101e63b18f559fca0ae627?s=96&d=mm&r=g\",\"caption\":\"\u65b0, \u97f5\"},\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/author\/yunxin\/\"},{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bb%80%e4%b9%88%e6%98%af%e6%b5%81%e5%a4%84%e7%90%86%ef%bc%9f%e4%bb%a5%e5%8f%8a2016%e5%b9%b4%e5%8f%91%e7%94%9f%e7%9a%84%e4%ba%8b%e4%bb%b6%e3%80%82\/#local-main-organization-logo\",\"url\":\"\",\"contentUrl\":\"\",\"caption\":\"Blog - Silicon Cloud\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"\u4ec0\u4e48\u662f\u6d41\u5904\u7406\uff1f\u4ee5\u53ca2016\u5e74\u53d1\u751f\u7684\u4e8b\u4ef6 - Blog - Silicon Cloud","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.silicloud.com\/zh\/blog\/\u4ec0\u4e48\u662f\u6d41\u5904\u7406\uff1f\u4ee5\u53ca2016\u5e74\u53d1\u751f\u7684\u4e8b\u4ef6\u3002\/","og_locale":"zh_CN","og_type":"article","og_title":"\u4ec0\u4e48\u662f\u6d41\u5904\u7406\uff1f\u4ee5\u53ca2016\u5e74\u53d1\u751f\u7684\u4e8b\u4ef6","og_description":"\u8fd9\u7bc7\u6587\u7ae0\u8bb2\u8ff0\u4e86\u4ec0\u4e48\u5185\u5bb9\uff1f \u6d41\u5904\u7406\u6700\u521d\u5c31\u662f\u4ece\u67d0\u4e2a\u5730\u65b9\u5f00\u59cb\u7684\uff0c\u5b83\u662f\u5173\u4e8e\u5fc5\u8981\u8003\u8651\u4e8b\u9879\u7684\u4fe1\u606f\u7684\u603b\u7ed3\uff0c\u4f9b\u56de\u987e\u4f7f\u7528\u3002 \u6b64\u5916 [&hellip;]","og_url":"https:\/\/www.silicloud.com\/zh\/blog\/\u4ec0\u4e48\u662f\u6d41\u5904\u7406\uff1f\u4ee5\u53ca2016\u5e74\u53d1\u751f\u7684\u4e8b\u4ef6\u3002\/","og_site_name":"Blog - Silicon Cloud","article_published_time":"2023-04-16T05:36:35+00:00","article_modified_time":"2024-04-29T01:17:08+00:00","og_image":[{"url":"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7e82913a08637a6a31ff\/30-0.png"}],"author":"\u65b0, \u97f5","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"\u65b0, \u97f5","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"3 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bb%80%e4%b9%88%e6%98%af%e6%b5%81%e5%a4%84%e7%90%86%ef%bc%9f%e4%bb%a5%e5%8f%8a2016%e5%b9%b4%e5%8f%91%e7%94%9f%e7%9a%84%e4%ba%8b%e4%bb%b6%e3%80%82\/","url":"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bb%80%e4%b9%88%e6%98%af%e6%b5%81%e5%a4%84%e7%90%86%ef%bc%9f%e4%bb%a5%e5%8f%8a2016%e5%b9%b4%e5%8f%91%e7%94%9f%e7%9a%84%e4%ba%8b%e4%bb%b6%e3%80%82\/","name":"\u4ec0\u4e48\u662f\u6d41\u5904\u7406\uff1f\u4ee5\u53ca2016\u5e74\u53d1\u751f\u7684\u4e8b\u4ef6 - Blog - Silicon Cloud","isPartOf":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website"},"datePublished":"2023-04-16T05:36:35+00:00","dateModified":"2024-04-29T01:17:08+00:00","author":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/4ba4019495123db3038fd0809e6959c9"},"breadcrumb":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bb%80%e4%b9%88%e6%98%af%e6%b5%81%e5%a4%84%e7%90%86%ef%bc%9f%e4%bb%a5%e5%8f%8a2016%e5%b9%b4%e5%8f%91%e7%94%9f%e7%9a%84%e4%ba%8b%e4%bb%b6%e3%80%82\/#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.silicloud.com\/zh\/blog\/%e4%bb%80%e4%b9%88%e6%98%af%e6%b5%81%e5%a4%84%e7%90%86%ef%bc%9f%e4%bb%a5%e5%8f%8a2016%e5%b9%b4%e5%8f%91%e7%94%9f%e7%9a%84%e4%ba%8b%e4%bb%b6%e3%80%82\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bb%80%e4%b9%88%e6%98%af%e6%b5%81%e5%a4%84%e7%90%86%ef%bc%9f%e4%bb%a5%e5%8f%8a2016%e5%b9%b4%e5%8f%91%e7%94%9f%e7%9a%84%e4%ba%8b%e4%bb%b6%e3%80%82\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.silicloud.com\/zh\/blog\/"},{"@type":"ListItem","position":2,"name":"\u4ec0\u4e48\u662f\u6d41\u5904\u7406\uff1f\u4ee5\u53ca2016\u5e74\u53d1\u751f\u7684\u4e8b\u4ef6"}]},{"@type":"WebSite","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website","url":"https:\/\/www.silicloud.com\/zh\/blog\/","name":"Blog - Silicon Cloud","description":"","inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/4ba4019495123db3038fd0809e6959c9","name":"\u65b0, \u97f5","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d484b6c6e4ae82e8a9efea989e1d2af46d9b6ef128101e63b18f559fca0ae627?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d484b6c6e4ae82e8a9efea989e1d2af46d9b6ef128101e63b18f559fca0ae627?s=96&d=mm&r=g","caption":"\u65b0, \u97f5"},"url":"https:\/\/www.silicloud.com\/zh\/blog\/author\/yunxin\/"},{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bb%80%e4%b9%88%e6%98%af%e6%b5%81%e5%a4%84%e7%90%86%ef%bc%9f%e4%bb%a5%e5%8f%8a2016%e5%b9%b4%e5%8f%91%e7%94%9f%e7%9a%84%e4%ba%8b%e4%bb%b6%e3%80%82\/#local-main-organization-logo","url":"","contentUrl":"","caption":"Blog - Silicon Cloud"}]}},"_links":{"self":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/47439","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/users\/11"}],"replies":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/comments?post=47439"}],"version-history":[{"count":2,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/47439\/revisions"}],"predecessor-version":[{"id":84389,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/47439\/revisions\/84389"}],"wp:attachment":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/media?parent=47439"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/categories?post=47439"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/tags?post=47439"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}