Apach Kafka\u306fLinkedIn\u3067\u958b\u767a\u3055\u308c\u305f\u5206\u6563\u30e1\u30c3\u30bb\u30fc\u30b8\u30f3\u30b0\u30b7\u30b9\u30c6\u30e0\u3067\u3059\u3002Arduino\u304b\u3089MQTT\u30d6\u30ed\u30fc\u30ab\u30fc\u306bpublish\u3057\u305f\u30bb\u30f3\u30b7\u30f3\u30b0\u30c7\u30fc\u30bf\u3092Kafka\u3067consume\u3059\u308b\u4e88\u5b9a\u3067\u3059\u3002Riemann\u3001Spark Streaming\u3001Storm\u306a\u3069\u306e\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u51e6\u7406\u304c\u76ee\u7684\u3067\u3059\u3002\u307e\u305a\u306fDocker\u3067Kafka\u30af\u30e9\u30b9\u30bf\u3092\u69cb\u7bc9\u3057\u307e\u3059\u3002<\/p>\n
Kafka\u306eDocker\u30a4\u30e1\u30fc\u30b8\u306fwurstmeister\/kafka-docker\u3092\u4f7f\u3044\u307e\u3059\u3002Kafka\u30af\u30e9\u30b9\u30bf\u306e\u69cb\u6210\u7ba1\u7406\u306bZookeeper\u304c\u5fc5\u8981\u306b\u306a\u308a\u307e\u3059\u3002\u3053\u306e\u30a4\u30e1\u30fc\u30b8\u306fDocker Compose\u3092\u4f7f\u3063\u3066\u3044\u308b\u306e\u3067\u3061\u3087\u3046\u3069\u826f\u3044\u52c9\u5f37\u306b\u306a\u308a\u307e\u3059\u3002\u4e8b\u524d\u306b\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3057\u3066\u304a\u304d\u307e\u3059\u3002<\/p>\n
$ <\/span>curl -L<\/span> https:\/\/github.com\/docker\/compose\/releases\/download\/1.2.0\/docker-compose-`<\/span>uname<\/span> -s<\/span>`<\/span>-`<\/span>uname<\/span> -m<\/span>`<\/span> ><\/span> \/usr\/local\/bin\/docker-compose;<\/span> chmod<\/span> +x \/usr\/local\/bin\/docker-compose\r\n<\/code><\/pre>\n\u30d7\u30ed\u30b8\u30a7\u30af\u30c8\u306e\u4f5c\u6210<\/h2>\n
\u9069\u5f53\u306a\u30c7\u30a3\u30ec\u30af\u30c8\u30ea\u3092\u4f5c\u6210\u3057\u3066\u3001\u30ea\u30dd\u30b8\u30c8\u30ea\u304b\u3089git clone\u3057\u307e\u3059\u3002<\/p>\n
$ <\/span>cd<\/span> ~\/docker_apps\r\n$ <\/span>git clone https:\/\/github.com\/wurstmeister\/kafka-docker.git\r\n$ <\/span>cd <\/span>kafka-docker\r\n<\/code><\/pre>\n\u30af\u30e9\u30b9\u30bf\u3092\u8d77\u52d5\u3059\u308b<\/h2>\n
Kafka Docker\u306e\u624b\u9806\u3092\u8aad\u307f\u306a\u304c\u3089\u30af\u30e9\u30b9\u30bf\u306e\u69cb\u7bc9\u3068\u7c21\u5358\u306a\u30c6\u30b9\u30c8\u307e\u3067\u884c\u3044\u307e\u3059\u3002<\/p>\n
docker-compose.yml<\/h3>\n
\u30ea\u30dd\u30b8\u30c8\u30ea\u306b\u306f\u30af\u30e9\u30b9\u30bf\u7528\u30681\u53f0\u69cb\u6210\u7528\u306edocker-compose.yml\u304c\u7528\u610f\u3055\u308c\u3066\u3044\u307e\u3059\u3002\u4eca\u56de\u306f\u30d6\u30ed\u30fc\u30ab\u30fc\u30922\u53f0\u8d77\u52d5\u3057\u305f\u3044\u306e\u3067\u30af\u30e9\u30b9\u30bf\u7528\u306edocker-compose.yml\u3092\u4f7f\u3044\u307e\u3059\u3002YAML\u306eKAFKA_ADVERTISED_HOST_NAME\u306e\u5024\u3092Docker\u30db\u30b9\u30c8\u306eIP\u30a2\u30c9\u30ec\u30b9\u306b\u66f8\u304d\u63db\u3048\u3066\u5b9f\u884c\u3057\u307e\u3059\u3002<\/p>\n
zookeeper<\/span>:<\/span>\r\n image<\/span>:<\/span> wurstmeister\/zookeeper<\/span>\r\n ports<\/span>:<\/span>\r\n -<\/span> \"<\/span>2181\"<\/span>\r\nkafka<\/span>:<\/span>\r\n build<\/span>:<\/span> .<\/span>\r\n ports<\/span>:<\/span>\r\n -<\/span> \"<\/span>9092\"<\/span>\r\n links<\/span>:<\/span>\r\n -<\/span> zookeeper:zk<\/span>\r\n environment<\/span>:<\/span>\r\n KAFKA_ADVERTISED_HOST_NAME<\/span>:<\/span> 10.3.0.165<\/span>\r\n volumes<\/span>:<\/span>\r\n -<\/span> \/var\/run\/docker.sock:\/var\/run\/docker.sock<\/span>\r\n<\/code><\/pre>\n\u6700\u521d\u306bdocker-compose up\u3059\u308b\u3068kafka\u306eDocker\u30a4\u30e1\u30fc\u30b8\u306e\u30d3\u30eb\u30c9\u304c\u59cb\u307e\u308a\u307e\u3059\u3002\u5fc5\u8981\u306a\u30b3\u30f3\u30c6\u30ca\u304c\u3059\u3079\u3066\u8d77\u52d5\u3059\u308b\u307e\u3067\u3057\u3070\u3089\u304f\u5f85\u3061\u307e\u3059\u3002<\/p>\n
$ <\/span>docker-compose up\r\n<\/code><\/pre>\n\u30af\u30e9\u30b9\u30bf\u304c\u3059\u3079\u3066\u8d77\u52d5\u3057\u305f\u3089\u5225\u30b7\u30a7\u30eb\u304b\u3089Kafka\u30d6\u30ed\u30fc\u30ab\u30fc\u30922\u53f0\u306b\u5909\u66f4\u3057\u307e\u3059\u3002<\/p>\n
$ <\/span>docker-compose scale kafka<\/span>=<\/span>2\r\nCreating kafkadocker_kafka_2...\r\nStarting kafkadocker_kafka_2...\r\n<\/code><\/pre>\nps\u3067\u30b3\u30f3\u30c6\u30ca\u306e\u8d77\u52d5\u72b6\u6cc1\u3092\u78ba\u8a8d\u3057\u307e\u3059\u3002<\/p>\n
$ <\/span>docker-compose ps\r\n Name Command State Ports\r\n-----------------------------------------------------------------------------------------------------<\/span>\r\nkafkadocker_kafka_2 \/bin\/sh -c<\/span> start- Up 0.0.0.0:32789->9092\/tcp\r\n kafka.sh\r\nkafkadocker_kafka_3 \/bin\/sh -c<\/span> start- Up 0.0.0.0:32790->9092\/tcp\r\n kafka.sh\r\nkafkadocker_zookeeper_1 \/bin\/sh -c<\/span> Up 0.0.0.0:32788->2181\/tcp\r\n \/usr\/sbin\/sshd ... , 22\/tcp, 2888\/tcp,\r\n 3888\/tcp\r\n<\/code><\/pre>\nKafka Shell<\/h2>\n
Kafka Shell\u3092\u8d77\u52d5\u3059\u308b\u66f8\u5f0f\u306f\u4ee5\u4e0b\u3067\u3059\u3002<\/p>\n
$ <\/span>start-kafka-shell.sh <DOCKER_HOST_IP> <ZK_HOST:ZK_PORT>\r\n<\/code><\/pre>\nZK_HOST:PORT\u306e\u5024\u306fdocker-compose ps\u3067\u78ba\u8a8d\u3057\u305f\u5024\u3092\u4f7f\u3044\u307e\u3059\u3002<\/p>\n
$ <\/span>cd<\/span> ~\/docker_apps\/kafka-docker\/\r\n$ <\/span>.\/start-kafka-shell.sh 10.3.0.165 10.3.0.165:32788\r\n<\/code><\/pre>\nstart-kafka-shell.sh\u3067\u306f\u4f7f\u3044\u6368\u3066\u306eDocker\u30b3\u30f3\u30c6\u30ca\u3092\u8d77\u52d5\u3057\u3066bash\u3092\u5b9f\u884c\u3057\u307e\u3059\u3002\u3053\u306e\u30b3\u30f3\u30c6\u30ca\u3092\u4f7f\u3044Kafka\u306eproducer\u3068consumer\u306e\u30d7\u30ed\u30bb\u30b9\u3092\u8d77\u52d5\u3057\u307e\u3059\u3002<\/p>\n
Kafka\u30b3\u30f3\u30c6\u30ca\u306e\u74b0\u5883\u5909\u6570\u3092\u78ba\u8a8d\u3057\u3066\u304a\u304d\u307e\u3059\u3002<\/p>\n
$ <\/span>echo<\/span> $KAFKA_HOME<\/span>\r\n\/opt\/kafka_2.10-0.8.2.0\r\n$ <\/span>echo<\/span> $ZK<\/span>\r\n10.3.0.165:32788\r\n<\/code><\/pre>\ntopic\u306e\u4f5c\u6210<\/h3>\n
Kafka Shell\u3092\u4f7f\u3044topic\u3092\u4f5c\u6210\u3057\u307e\u3059\u3002Kafka\u306etopic\u306f\u30e1\u30c3\u30bb\u30fc\u30b8\u306e\u30ab\u30c6\u30b4\u30ea\u306b\u306a\u308a\u307e\u3059\u3002\u3053\u306etopic\u306f\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3\u306f4\u3064\u3001\u30e1\u30c3\u30bb\u30fc\u30b8\u306e\u30ec\u30d7\u30ea\u30ab\u306f2\u500b\u306e\u8a2d\u5b9a\u3067\u3059\u3002<\/p>\n
$ $KAFKA_HOME<\/span>\/bin\/kafka-topics.sh --create<\/span> --topic<\/span> topic \\<\/span>\r\n--partitions<\/span> 4 --zookeeper<\/span> $ZK<\/span> --replication-factor<\/span> 2\r\nCreated topic \"topic\"<\/span>.<\/span>\r\n<\/code><\/pre>\n\u4f5c\u6210\u3057\u305ftopic\u3092\u78ba\u8a8d\u3057\u307e\u3059\u3002<\/p>\n
$ $KAFKA_HOME\/bin\/kafka-topics.sh --describe --topic topic --zookeeper $ZK\r\nTopic:topic PartitionCount:4 ReplicationFactor:2 Configs:\r\n Topic: topic Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2\r\n Topic: topic Partition: 1 Leader: 2 Replicas: 2,1 Isr: 2,1\r\n Topic: topic Partition: 2 Leader: 1 Replicas: 1,2 Isr: 1,2\r\n Topic: topic Partition: 3 Leader: 2 Replicas: 2,1 Isr: 2,1\r\n<\/code><\/pre>\nproducer\u306e\u9001\u4fe1<\/h3>\n
Kafka Shell\u306e\u4e00\u3064\u304b\u3089producer\u3092\u5b9f\u884c\u3057\u307e\u3059\u3002\u30b3\u30f3\u30bd\u30fc\u30eb\u306f\u6587\u5b57\u5217\u306e\u5165\u529b\u5f85\u3061\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n
$ $KAFKA_HOME<\/span>\/bin\/kafka-console-producer.sh --topic<\/span>=<\/span>topic \\<\/span>\r\n--broker-list<\/span>=<\/span>`<\/span>broker-list.sh`<\/span>\r\n<\/code><\/pre>\nconsumer\u306e\u53d7\u4fe1<\/h3>\n
Kafka Shell\u3092\u8d77\u52d5\u3057\u3066consumer\u3092\u5b9f\u884c\u3057\u307e\u3059\u3002<\/p>\n
$ <\/span>.\/start-kafka-shell.sh 10.3.0.165 10.3.0.165:32788\r\n$ $KAFKA_HOME<\/span>\/bin\/kafka-console-consumer.sh --topic<\/span>=<\/span>topic --zookeeper<\/span>=<\/span>$ZK<\/span>\r\n<\/code><\/pre>\n\u5165\u529b\u5f85\u3061\u306eproducer\u306e\u30b3\u30f3\u30bd\u30fc\u30eb\u306b\u6587\u5b57\u5217\u3092\u30bf\u30a4\u30d7\u3059\u308b\u3068\u3001consumer\u306e\u30b3\u30f3\u30bd\u30fc\u30eb\u306b\u8868\u793a\u3055\u308c\u307e\u3059\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"
Apach Kafka\u306fLinkedIn\u3067\u958b\u767a\u3055\u308c\u305f\u5206\u6563\u30e1\u30c3\u30bb\u30fc\u30b8\u30f3\u30b0\u30b7\u30b9\u30c6\u30e0\u3067\u3059\u3002Arduino\u304b\u3089MQTT […]<\/p>\n","protected":false},"author":11,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-46678","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"\n
- Blog - Silicon Cloud<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.silicloud.com\/zh\/blog\/46678-2\/\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:description\" content=\"Apach Kafka\u306fLinkedIn\u3067\u958b\u767a\u3055\u308c\u305f\u5206\u6563\u30e1\u30c3\u30bb\u30fc\u30b8\u30f3\u30b0\u30b7\u30b9\u30c6\u30e0\u3067\u3059\u3002Arduino\u304b\u3089MQTT […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.silicloud.com\/zh\/blog\/46678-2\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog - Silicon Cloud\" \/>\n<meta property=\"article:published_time\" content=\"2023-08-27T23:21:56+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-04-30T13:20:24+00:00\" \/>\n<meta name=\"author\" content=\"\u65b0, \u97f5\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"\u65b0, \u97f5\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/46678-2\/\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/46678-2\/\",\"name\":\"- Blog - Silicon Cloud\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\"},\"datePublished\":\"2023-08-27T23:21:56+00:00\",\"dateModified\":\"2024-04-30T13:20:24+00:00\",\"author\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/4ba4019495123db3038fd0809e6959c9\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.silicloud.com\/zh\/blog\/46678-2\/\"]}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/\",\"name\":\"Blog - Silicon Cloud\",\"description\":\"\",\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/4ba4019495123db3038fd0809e6959c9\",\"name\":\"\u65b0, \u97f5\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d484b6c6e4ae82e8a9efea989e1d2af46d9b6ef128101e63b18f559fca0ae627?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d484b6c6e4ae82e8a9efea989e1d2af46d9b6ef128101e63b18f559fca0ae627?s=96&d=mm&r=g\",\"caption\":\"\u65b0, \u97f5\"},\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/author\/yunxin\/\"},{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/46678-2\/#local-main-organization-logo\",\"url\":\"\",\"contentUrl\":\"\",\"caption\":\"Blog - Silicon Cloud\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"- Blog - Silicon Cloud","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.silicloud.com\/zh\/blog\/46678-2\/","og_locale":"zh_CN","og_type":"article","og_description":"Apach Kafka\u306fLinkedIn\u3067\u958b\u767a\u3055\u308c\u305f\u5206\u6563\u30e1\u30c3\u30bb\u30fc\u30b8\u30f3\u30b0\u30b7\u30b9\u30c6\u30e0\u3067\u3059\u3002Arduino\u304b\u3089MQTT […]","og_url":"https:\/\/www.silicloud.com\/zh\/blog\/46678-2\/","og_site_name":"Blog - Silicon Cloud","article_published_time":"2023-08-27T23:21:56+00:00","article_modified_time":"2024-04-30T13:20:24+00:00","author":"\u65b0, \u97f5","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"\u65b0, \u97f5","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"2 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.silicloud.com\/zh\/blog\/46678-2\/","url":"https:\/\/www.silicloud.com\/zh\/blog\/46678-2\/","name":"- Blog - Silicon Cloud","isPartOf":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website"},"datePublished":"2023-08-27T23:21:56+00:00","dateModified":"2024-04-30T13:20:24+00:00","author":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/4ba4019495123db3038fd0809e6959c9"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.silicloud.com\/zh\/blog\/46678-2\/"]}]},{"@type":"WebSite","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website","url":"https:\/\/www.silicloud.com\/zh\/blog\/","name":"Blog - Silicon Cloud","description":"","inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/4ba4019495123db3038fd0809e6959c9","name":"\u65b0, \u97f5","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d484b6c6e4ae82e8a9efea989e1d2af46d9b6ef128101e63b18f559fca0ae627?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d484b6c6e4ae82e8a9efea989e1d2af46d9b6ef128101e63b18f559fca0ae627?s=96&d=mm&r=g","caption":"\u65b0, \u97f5"},"url":"https:\/\/www.silicloud.com\/zh\/blog\/author\/yunxin\/"},{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/46678-2\/#local-main-organization-logo","url":"","contentUrl":"","caption":"Blog - Silicon Cloud"}]}},"_links":{"self":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/46678","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/users\/11"}],"replies":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/comments?post=46678"}],"version-history":[{"count":2,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/46678\/revisions"}],"predecessor-version":[{"id":94363,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/46678\/revisions\/94363"}],"wp:attachment":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/media?parent=46678"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/categories?post=46678"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/tags?post=46678"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}