{"id":36137,"date":"2023-08-26T20:28:18","date_gmt":"2024-01-20T14:54:15","guid":{"rendered":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8kubernetes%e4%b8%8a%e9%aa%8c%e8%af%81apache-spark\/"},"modified":"2024-04-30T20:13:16","modified_gmt":"2024-04-30T12:13:16","slug":"%e5%9c%a8kubernetes%e4%b8%8a%e9%aa%8c%e8%af%81apache-spark","status":"publish","type":"post","link":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8kubernetes%e4%b8%8a%e9%aa%8c%e8%af%81apache-spark\/","title":{"rendered":"\u5728Kubernetes\u4e0a\u9a8c\u8bc1Apache Spark"},"content":{"rendered":"<h1>Apache Spark on Kubernetes \u662f\u4ec0\u4e48<\/h1>\n<p>Apache Spark\uff08\u4ee5\u4e0b\u79f0\u4e3aSpark\uff09\u4ecev2.3.0\u5f00\u59cb\u652f\u6301\u539f\u751f\u7684 Kubernetes\u3002\u5728\u6b64\u4e4b\u524d\uff0c\u901a\u8fc7\u4e8b\u5148\u51c6\u5907\u597d\u7684 Spark\/Hadoop \u548c\u76f8\u5e94\u7684\u8ba1\u7b97\u8d44\u6e90\uff0c\u7531\u7ba1\u7406\u5458\u8fdb\u884c\u5e76\u884c\u8ba1\u7b97\u3002\u73b0\u5728\u901a\u8fc7\u539f\u751f\u652f\u6301\u7684 Kubernetes\uff0c\u7ba1\u7406\u5458\u53ea\u9700\u51c6\u5907\u597d\u7528\u4e8e Spark \u7684 Pod\uff08\u5bb9\u5668\uff09\u5373\u53ef\uff0c\u65e0\u9700\u62c5\u5fc3\u7528\u4e8e Spark \u7684\u8ba1\u7b97\u8d44\u6e90\u3002<br \/>\n\u5927\u81f4\u6267\u884c\u6d41\u7a0b\u5982\u4e0b\uff1aSpark \u7528\u6237\u6267\u884c Spark \u7684 CLI\uff08spark-submit \u547d\u4ee4\uff09\uff0c\u901a\u8fc7 Kubernetes \u7684 API \u670d\u52a1\u5668\uff0c\u901a\u8fc7\u8c03\u5ea6\u5668\u6307\u5b9a\u8981\u6267\u884c Spark \u7684\u8ba1\u7b97\u8d44\u6e90\uff08\u8282\u70b9\uff09\u3002\u7136\u540e\uff0cSpark Driver \u7684 Pod\uff08\u8d1f\u8d23\u7ba1\u7406 Executor \u7684 Pod\uff09\u548c\u7528\u4e8e\u8ba1\u7b97\u7684 Spark Executor \u7684 Pod \u5c06\u90e8\u7f72\u5e76\u8fdb\u884c\u5e76\u884c\u8ba1\u7b97\u3002\u6b64\u5916\uff0c\u6bcf\u5f53 Spark \u5b8c\u6210\u8ba1\u7b97\u65f6\uff0c\u4f1a\u5220\u9664\/\u505c\u6b62\u76f8\u5173\u7684 Pod\uff0c\u4ee5\u907f\u514d\u6d6a\u8d39\u8ba1\u7b97\u8d44\u6e90\u3002\u6362\u8a00\u4e4b\uff0c\u5728\u4f7f\u7528\u6309\u9700\u8ba1\u8d39\u7684\u516c\u5171\u4e91\uff08\u5982 AWS \u6216 GCP\uff09\u65f6\uff0c\u7531\u4e8e\u4e0d\u4f1a\u4e00\u76f4\u5b58\u5728\u50cf\u5b88\u62a4\u8fdb\u7a0b\u4e00\u6837\u542f\u52a8\u7684 Pod\uff0c\u56e0\u6b64\u8ba1\u7b97\u8d44\u6e90\uff08CPU\u3001\u5185\u5b58\uff09\u7684\u8d39\u7528\u53ea\u4f1a\u6309\u5b9e\u9645\u4f7f\u7528\u91cf\u8ba1\u7b97\u3002<br \/>\n\u901a\u8fc7\u8fd9\u79cd\u65b9\u5f0f\uff0cSpark \u7528\u6237\u53ea\u9700\u77e5\u9053\u8981\u6307\u5b9a\u7684 Kubernetes URL\uff0c\u5373\u53ef\u8f7b\u677e\u83b7\u5f97\u4f7f\u7528\u5927\u91cf\u8ba1\u7b97\u8d44\u6e90\u7684\u5e76\u884c\u8ba1\u7b97\u73af\u5883\u3002\u8fd9\u6837\u4e00\u6765\uff0c\u5f53\u81ea\u8eab\u62e5\u6709\u7684\u8ba1\u7b97\u8d44\u6e90\u4e0d\u8db3\u4ee5\u5b8c\u6210\u4efb\u52a1\u65f6\uff0c\u7528\u6237\u53ef\u4ee5\u65b9\u4fbf\u5730\u901a\u8fc7\u5e76\u884c\u8ba1\u7b97\u73af\u5883\u89e3\u51b3\u95ee\u9898\uff0c\u65e0\u9700\u82b1\u8d39\u989d\u5916\u7684\u65f6\u95f4\u548c\u7cbe\u529b\uff0c\u53ea\u9700\u652f\u4ed8\u76f8\u5e94\u7684\u8d39\u7528\u5373\u53ef\u3002<\/p>\n<h1>\u5904\u7406\u6d41\u7a0b<\/h1>\n<p>\u6211\u4eec\u5c06\u4f7f\u7528\u4e0b\u56fe\u6309\u987a\u5e8f\u8ffd\u8e2aSpark\u5728Kubernetes\u4e0a\u7684\u5904\u7406\u6d41\u7a0b\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d277e37434c4406c3cade\/4-0.png\" alt=\"\u30b9\u30af\u30ea\u30fc\u30f3\u30b7\u30e7\u30c3\u30c8 2018-05-07 10.09.44.png\" \/><\/div>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">(Step1)\u30e6\u30fc\u30b6\u306b\u3066&#8217;spark-submit&#8217;\u30b3\u30de\u30f3\u30c9\u3092\u4f7f\u3044\u3001\u8a08\u7b97\u306e\u5b9f\u884c\u3092\u6307\u793a\u3057\u307e\u3059<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">(Step2) Kubernetes\u306eNode\u4e0a\u306bSpark Driver\u306ePod(\u30b3\u30f3\u30c6\u30ca)\u304c\u30c7\u30d7\u30ed\u30a4\u3055\u308c\u307e\u3059<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">(Step3) SparkDriver\u304cspark-submit\u3067\u6307\u5b9a\u3055\u308c\u305f\u6570\u306eExecutor\u306ePod\u3092\u30c7\u30d7\u30ed\u30a4\u3057\u3001\u5404Executor\u306b\u3066\u8a08\u7b97\u304c\u5b9f\u884c\u3055\u308c\u307e\u3059<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">(Step4) \u8a08\u7b97\u304c\u7d42\u4e86\u3057\u305fExecutor\u306ePod\u306f\u81ea\u52d5\u3067\u524a\u9664\u3055\u308c\u307e\u3059\u3002\u5168\u3066\u306eExecutor\u306ePod\u304c\u524a\u9664\u3055\u308c\u305f\u5f8c\u3001\u6700\u5f8c\u306bSpark Driver\u306ePod\u304c\u505c\u6b62(Terminate)\u3055\u308c\u307e\u3059\u3002\u8a08\u7b97\u7d50\u679c\u306f\u3001Spark Driver\u306ePod\u306e\u30ed\u30b0\u306b\u51fa\u529b\u3055\u308c\u307e\u3059\u3002<\/ul>\n<p>\u7531\u4e8eSpark Driver\u7684Pod\u4e0d\u4f1a\u81ea\u52a8\u5220\u9664\uff0c\u56e0\u6b64\u5728\u786e\u8ba4\u8ba1\u7b97\u7ed3\u679c\u540e\uff0c\u9700\u8981\u4f7f\u7528kubectl\u547d\u4ee4\u624b\u52a8\u5220\u9664\u5b83\u3002<\/p>\n<h1>\u5728 Spark v2.3.0 \u4e2d\u7684\u9650\u5236\u6761\u4ef6<\/h1>\n<p>Spark\u5728Kubernetes\u4e0a\u4ecd\u7136\u662f\u4e00\u4e2a\u5904\u4e8e\u5f00\u53d1\u521d\u671f\u7684\u9879\u76ee\u3002<br \/>\n\u56e0\u6b64\uff0c\u4ee5\u4e0b\u5185\u5bb9\u88ab\u5217\u4e3a\u672a\u6765\u7684\u5de5\u4f5c\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">PySpark<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">R<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">Dynamic Executor Scaling<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">Local File Dependency Management<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">Spark Application Management<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">Job Queues and Resource Management<\/ul>\n<p>\u4f7f\u7528Python\u6216\u8005R\u5728Kubernetes\u4e0a\u5c1d\u8bd5Spark\u53ef\u80fd\u8fd8\u9700\u8981\u4e00\u6bb5\u65f6\u95f4\u3002\u8bf7\u53c2\u8003\u6b64\u5904\u83b7\u53d6\u66f4\u591a\u8be6\u7ec6\u4fe1\u606f\u3002<\/p>\n<h1>\u9a8c\u8bc1\u884c\u52a8<\/h1>\n<p>\u63a5\u4e0b\u6765\uff0c\u6211\u4eec\u5c06\u5b9e\u9645\u8fd0\u884c\u548c\u9a8c\u8bc1Kubernetes\u4e0a\u7684Spark\u3002<br \/>\n\u9a8c\u8bc1\u73af\u5883\u5c06\u91c7\u7528\u4ee5\u4e0b\u73af\u5883\u3002<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">Spark\u306e\u30b3\u30f3\u30d1\u30a4\u30eb\/CLI\u5b9f\u884c\u74b0\u5883: Ubuntu 16.04.4 LTS + Docker 18.03.1-ce<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">Kubernetes\u74b0\u5883: Kubernetes v1.10.2<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">(kubeadm\u3092\u4f7f\u3044Virtual Box\u4e0a\u306b\u69cb\u7bc9\u3057\u305fMaster, WorkerNode x 2VM\u306e\u5408\u8a083VM\u306e\u30af\u30e9\u30b9\u30bf\u74b0\u5883)<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u30b3\u30f3\u30c6\u30ca\u30ec\u30dd\u30b8\u30c8\u30ea: Docker Hub<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\u30b5\u30f3\u30d7\u30eb\u30a2\u30d7\u30ea: Spark\u306b\u4ed8\u5c5e\u306eSparkPi (Java\u3067\u66f8\u304b\u308c\u305f\u03c0\u306e\u8a08\u7b97\u30d7\u30ed\u30b0\u30e9\u30e0)<\/ul>\n<p>\u5173\u4e8e\u64cd\u4f5c\u7cfb\u7edf\u3001Docker\u548cKubernetes\u672c\u8eab\u7684\u8bbe\u7f6e\u4ee5\u53caDocker Hub\u7528\u6237\u7684\u521b\u5efa\uff0c\u5c06\u5176\u7701\u7565\u3002<\/p>\n<h2>\u4e0b\u8f7d\u5e76\u7f16\u8bd1<\/h2>\n<p>\u4eceSpark\u7684\u4e0b\u8f7d\u7f51\u7ad9\u4e0a\u4e0b\u8f7dSpark 2.3.0\u7248\u672c\u3002<\/p>\n<pre class=\"post-pre\"><code>$ wget http:\/\/ftp.jaist.ac.jp\/pub\/apache\/spark\/spark-2.3.0\/spark-2.3.0.tgz \r\n\r\n$ gzip -cd spark-2.3.0.tgz | tar xvf -\r\n$ cd spark-2.3.0\/\r\n<\/code><\/pre>\n<p>\u6211\u5011\u5c07\u5728spark-2.3.0\u7684\u76ee\u9304\u4e0b\u9032\u884c\u64cd\u4f5c\u3002\u7136\u5f8c\uff0c\u6211\u5011\u5c07\u7de8\u8b6f\u4e0b\u8f09\u7684Spark\u6e90\u78bc\uff08\u5927\u7d04\u9700\u898145\u5206\u9418\uff09\u3002<\/p>\n<pre class=\"post-pre\"><code>$ build\/mvn -DskipTests clean package\r\n&lt;snip&gt;\r\n[INFO] ------------------------------------------------------------------------\r\n[INFO] Reactor Summary:\r\n[INFO] \r\n[INFO] Spark Project Parent POM ........................... SUCCESS [09:56 min]\r\n[INFO] Spark Project Tags ................................. SUCCESS [05:12 min]\r\n[INFO] Spark Project Sketch ............................... SUCCESS [  9.913 s]\r\n[INFO] Spark Project Networking ........................... SUCCESS [ 58.391 s]\r\n[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 12.961 s]\r\n[INFO] Spark Project Unsafe ............................... SUCCESS [ 32.245 s]\r\n[INFO] Spark Project Launcher ............................. SUCCESS [02:26 min]\r\n[INFO] Spark Project Core ................................. SUCCESS [05:59 min]\r\n[INFO] Spark Project ML Local Library ..................... SUCCESS [02:44 min]\r\n[INFO] Spark Project GraphX ............................... SUCCESS [ 25.961 s]\r\n[INFO] Spark Project Streaming ............................ SUCCESS [01:10 min]\r\n[INFO] Spark Project Catalyst ............................. SUCCESS [02:45 min]\r\n[INFO] Spark Project SQL .................................. SUCCESS [05:33 min]\r\n[INFO] Spark Project ML Library ........................... SUCCESS [02:18 min]\r\n[INFO] Spark Project Tools ................................ SUCCESS [ 10.025 s]\r\n[INFO] Spark Project Hive ................................. SUCCESS [02:19 min]\r\n[INFO] Spark Project REPL ................................. SUCCESS [  6.327 s]\r\n[INFO] Spark Project Assembly ............................. SUCCESS [  3.664 s]\r\n[INFO] Spark Project External Flume Sink .................. SUCCESS [ 33.437 s]\r\n[INFO] Spark Project External Flume ....................... SUCCESS [ 12.331 s]\r\n[INFO] Spark Project External Flume Assembly .............. SUCCESS [  3.056 s]\r\n[INFO] Spark Integration for Kafka 0.8 .................... SUCCESS [ 39.349 s]\r\n[INFO] Kafka 0.10 Source for Structured Streaming ......... SUCCESS [ 33.933 s]\r\n[INFO] Spark Project Examples ............................. SUCCESS [ 25.154 s]\r\n[INFO] Spark Project External Kafka Assembly .............. SUCCESS [  4.582 s]\r\n[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 12.030 s]\r\n[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [  4.851 s]\r\n[INFO] ------------------------------------------------------------------------\r\n[INFO] BUILD SUCCESS\r\n[INFO] ------------------------------------------------------------------------\r\n[INFO] Total time: 45:57 min\r\n[INFO] Finished at: 2018-05-1T23:21:21+09:00\r\n[INFO] Final Memory: 86M\/861M\r\n[INFO] ------------------------------------------------------------------------\r\n<\/code><\/pre>\n<p>\u5982\u679c\u8f93\u51faBUILD SUCCESS\uff0c\u5219\u8868\u793a\u7f16\u8bd1\u6210\u529f\u3002<\/p>\n<p>\u8fd9\u6b21\uff0c\u6211\u4ece\u6e90\u4ee3\u7801\u6587\u4ef6\u7f16\u8bd1\u4e86\uff0c\u4f46\u4e5f\u53ef\u4ee5\u4eceApache Spark\u7684\u5b98\u65b9\u7f51\u7ad9\u4e0b\u8f7d\u7f16\u8bd1\u597d\u7684\u4e8c\u8fdb\u5236\u6587\u4ef6\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d277e37434c4406c3cade\/22-0.png\" alt=\"\u30b9\u30af\u30ea\u30fc\u30f3\u30b7\u30e7\u30c3\u30c8 2018-05-02 21.19.49.png\" \/><\/div>\n<h2>Spark\u5bb9\u5668\u7684\u521b\u5efa\u548c\u4ed3\u5e93\u6ce8\u518c\u3002<\/h2>\n<p>\u6211\u5011\u5c07\u5275\u5efa\u4e00\u500b\u90e8\u7f72\u5728Kubernetes\u4e0a\u7684Spark\u5bb9\u5668\u6620\u50cf\u3002\u6211\u5011\u6703\u4f7f\u7528docker-image-tool.sh\u9032\u884c\u5275\u5efa\u3002\u5982\u679c\u9700\u8981\uff0c\u8acb\u5728kubernetes\/dockerfiles\/spark\u76ee\u9304\u4e0b\u7684Dockerfile\u4e2d\u6dfb\u52a0\u4f60\u81ea\u5df1\u7684Spark\u7a0b\u5e8f\u7b49\u3002<\/p>\n<pre class=\"post-pre\"><code>$ sudo bin\/docker-image-tool.sh -r ysakashita -t v2.3.0 build\r\n&lt;snap&gt;\r\nSuccessfully built f9cd85baa796\r\nSuccessfully tagged ysakashita\/spark:v2.3.0\r\n\r\n$ sudo docker images\r\nREPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE\r\nysakashita\/spark                           v2.3.0              f9cd85baa796        2 minutes ago       350MB\r\n<\/code><\/pre>\n<p>\u63a5\u4e0b\u6765\uff0c\u5c06\u6784\u5efa\u7684\u5bb9\u5668\u6ce8\u518c\u5230\u4ed3\u5e93\u4e2d\u3002<\/p>\n<pre class=\"post-pre\"><code>$ sudo docker login\r\nLogin with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https:\/\/hub.docker.com to create one.\r\nUsername (ysakashita): ysakashita\r\nPassword: \r\n\r\n$ sudo bin\/docker-image-tool.sh -r ysakashita -t v2.3.0 push\r\nThe push refers to a repository [docker.io\/ysakashita\/spark]\r\n&lt;snip&gt;\r\nv2.3.0: digest: sha256:3252c88b5527a97b9743824ae283c40e9d69b8587155ebd8c4d6f1b451d972f8 size: 2626\r\n<\/code><\/pre>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d277e37434c4406c3cade\/28-0.png\" alt=\"\u30b9\u30af\u30ea\u30fc\u30f3\u30b7\u30e7\u30c3\u30c8 2018-05-02 22.42.42.png\" \/><\/div>\n<p>\u5982\u679c\u5df2\u7ecf\u6ce8\u518c\uff0c\u90a3\u4e48\u5bb9\u5668\u955c\u50cf\u5c31\u5b8c\u6210\u4e86\u3002<br \/>\n\u7136\u540e\uff0c\u5728Kubernetes\u4e0a\u521b\u5efa\u4e00\u4e2a\u7528\u4e8e\u6267\u884cSpark on Kubernetes\u7684\u8d26\u53f7\u3002<\/p>\n<pre class=\"post-pre\"><code>$ kubectl create serviceaccount spark\r\n\r\n$ kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default\r\n<\/code><\/pre>\n<p>\u5728\u672c\u6b21\u9a8c\u8bc1\u4e2d\uff0c\u7531\u4e8e\u4f7f\u7528\u9ed8\u8ba4\u547d\u540d\u7a7a\u95f4\uff0c\u56e0\u6b64\u8bf7\u6388\u4e88\u9ed8\u8ba4\u6743\u9650\u3002<\/p>\n<p>\u4ee5\u4e0a\uff0c\u6e96\u5099\u5de5\u4f5c\u5df2\u7ecf\u5b8c\u6210\u3002<br \/>\n\u63a5\u4e0b\u6765\uff0c\u6211\u4eec\u5c06\u5f00\u59cb\u6267\u884c\u4f7f\u7528Spark on Kubernetes\u8fdb\u884c\u5e76\u884c\u8ba1\u7b97\u7684\u6b65\u9aa4\u3002<\/p>\n<h2>\u4f7f\u7528 Kubernetes \u4e0a\u7684 Spark \u6267\u884c\u5e76\u884c\u8ba1\u7b97\u3002<\/h2>\n<p>\u4f7f\u7528Kubernetes\u4e0a\u7684Spark\u6765\u6267\u884c\u5e76\u884c\u8ba1\u7b97\u3002<\/p>\n<p>\u4f7f\u7528spark-submit\u547d\u4ee4\u6765\u6267\u884c\u3002<\/p>\n<pre class=\"post-pre\"><code>$ kubectl cluster-info\r\nKubernetes master is running at https:\/\/192.168.0.23:6443\r\nKubeDNS is running at https:\/\/192.168.0.23:6443\/api\/v1\/namespaces\/kube-system\/services\/kube-dns:dns\/proxy\r\n\r\n$ bin\/spark-submit \\\r\n    --master k8s:\/\/https:\/\/192.168.0.23:6443 \\\r\n    --deploy-mode cluster \\\r\n    --conf spark.executor.instances=3 \\\r\n    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \\\r\n    --conf spark.kubernetes.container.image=ysakashita\/spark:v2.3.0 \\\r\n    --class org.apache.spark.examples.SparkPi \\\r\n    --name spark-pi \\\r\n    local:\/\/\/opt\/spark\/examples\/jars\/spark-examples_2.11-2.3.0.jar\r\n<\/code><\/pre>\n<div>\n<div class=\"post-table\">OptionDescription&#8211;masterKubernetes master(API Server)\u306e\u30a2\u30c9\u30ec\u30b9&#8211;deploy-modeSpark on Kubernetes\u3067\u306fcluster\u3092\u6307\u5b9a&#8211;conf spark.executor.instancesSpark Executor\u306e\u6570&#8211;conf spark.kubernetes.authenticate.driver.serviceAccountNameService Account\u540d&#8211;conf spark.kubernetes.container.imageSpark \u306e\u30b3\u30f3\u30c6\u30ca\u30a4\u30e1\u30fc\u30b8&#8211;class\u5b9f\u884c\u3059\u308bJava\u30a2\u30d7\u30ea\u306e\u30af\u30e9\u30b9\u540d&#8211;name\u5b9f\u884c\u3059\u308bJava\u30a2\u30d7\u30ea\u306e\u30d7\u30ed\u30b0\u30e9\u30e0\u540d&lt;app jar&gt;\u5b9f\u884c\u3059\u308bJava\u30a2\u30d7\u30ea\u306ejar \u30d5\u30a1\u30a4\u30eb\u540d<\/div>\n<\/div>\n<p>\u5173\u4e8e\u5176\u4ed6\u9009\u9879\uff0c\u8bf7\u53c2\u8003\u8fd9\u91cc\u3002<br \/>\n\u5728spark-submit\u547d\u4ee4\u7684\u6267\u884c\u7ed3\u679c\u4e2d\uff0c\u5982\u679c\u663e\u793aExit code\u4e3a0\uff08\u5373\u6b63\u5e38\u7ed3\u675f\uff09\uff0c\u5219\u8868\u793a\u8ba1\u7b97\u5df2\u6b63\u786e\u7ed3\u675f\u3002<\/p>\n<pre class=\"post-pre\"><code>         Container name: spark-kubernetes-driver\r\n         Container image: ysakashita\/spark:v2.3.0\r\n         Container state: Terminated\r\n         Exit code: 0\r\n<\/code><\/pre>\n<p>\u4f7f\u7528 kubectl logs \u547d\u4ee4\u67e5\u770b Spark Driver Pod \u7684\u65e5\u5fd7\u4ee5\u786e\u8ba4\u8ba1\u7b97\u7ed3\u679c\u3002<\/p>\n<pre class=\"post-pre\"><code>$ kubectl get pods\r\nNAME                                               READY     STATUS      RESTARTS   AGE\r\nspark-pi-9efb8a5e1e3b38f789505ace52dac3fe-driver   0\/1       Completed   0          29s\r\n\r\n$ kubectl logs spark-pi-9efb8a5e1e3b38f789505ace52dac3fe-driver |less\r\n&lt;snip&gt;\r\n2018-05-03 01:04:22 INFO  DAGScheduler:54 - Job 0 finished: reduce at SparkPi.scala:38, took 1.594295 s\r\nPi is roughly 3.144675723378617\r\n&lt;snip&gt;\r\n<\/code><\/pre>\n<p>\u4f60\u53ef\u4ee5\u901a\u8fc7 SparkPi \u7684\u6267\u884c\u7ed3\u679c\u6765\u786e\u8ba4\u8f93\u51fa\u7684\u662f 3.144675723378617\u3002<br \/>\n\u4e3a\u4e86\u8ba9\u7528\u6237\u80fd\u591f\u786e\u8ba4\u6267\u884c\u7ed3\u679c\uff0cSpark Driver \u7684 Pod \u4f1a\u4ee5 Ready 0\/1 \u7684\u72b6\u6001\u505c\u6b62\uff0c\u4f46\u4e0d\u4f1a\u88ab\u5220\u9664\u3002<br \/>\n\u786e\u8ba4\u4e86\u6267\u884c\u7ed3\u679c\u540e\uff0c\u8bf7\u5220\u9664 Pod\u3002<\/p>\n<pre class=\"post-pre\"><code>$ kubectl delete pods spark-pi-9efb8a5e1e3b38f789505ace52dac3fe-driver\r\n<\/code><\/pre>\n<h2>\u6b63\u5728\u8ba1\u7b97\u5904\u7406\u4e2d\u7684Pod\u90e8\u7f72\u7684\u9a8c\u8bc1\u3002<\/h2>\n<p>\u9996\u5148\uff0c\u6211\u4eec\u6765\u770b\u4e00\u4e0b\u5728\u4f7f\u7528spark-submit\u547d\u4ee4\u8fdb\u884c\u8ba1\u7b97\u5904\u7406\u65f6\uff0cKubernetes\u4e0aPod\u7684\u90e8\u7f72\u60c5\u51b5\u3002<\/p>\n<pre class=\"post-pre\"><code>$ kubectl get pods\r\nNAME                                               READY     STATUS    RESTARTS   AGE\r\nspark-pi-9efb8a5e1e3b38f789505ace52dac3fe-driver   1\/1       Running   0          4s\r\n<\/code><\/pre>\n<p>Spark Driver\u7684Pod\u5df2\u7ecf\u90e8\u7f72\u3002<br \/>\n\u8fd9\u662f\u5728\u5904\u7406\u6d41\u7a0b\u7ae0\u8282\u4e2d\u63d0\u5230\u7684(\u6b65\u9aa42)\u7684\u72b6\u6001\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d277e37434c4406c3cade\/48-0.png\" alt=\"\u30b9\u30af\u30ea\u30fc\u30f3\u30b7\u30e7\u30c3\u30c8 2018-05-07 10.10.04.png\" \/><\/div>\n<p>\u8fc7\u51e0\u79d2\u540e\uff0c\u6211\u4f1a\u518d\u6b21\u786e\u8ba4Pod\u7684\u72b6\u6001\u3002<\/p>\n<pre class=\"post-pre\"><code>$ kubectl get pods\r\nNAME                                               READY     STATUS    RESTARTS   AGE\r\nspark-pi-9efb8a5e1e3b38f789505ace52dac3fe-driver   1\/1       Running   0          14s\r\nspark-pi-9efb8a5e1e3b38f789505ace52dac3fe-exec-1   1\/1       Running   0          7s\r\nspark-pi-9efb8a5e1e3b38f789505ace52dac3fe-exec-2   1\/1       Running   0          7s\r\nspark-pi-9efb8a5e1e3b38f789505ace52dac3fe-exec-3   1\/1       Running   0          7s\r\n<\/code><\/pre>\n<p>\u7136\u540e\uff0c\u73b0\u5728\u53ef\u4ee5\u786e\u8ba4\u5df2\u7ecf\u90e8\u7f72\u4e863\u4e2aSpark Executor\u7684Pod\u3002\u8fd9\u662f\u5728\u6d41\u7a0b\u7684\u7ae0\u8282\u4e2d\u63d0\u5230\u7684\uff08\u7b2c3\u6b65\uff09\u72b6\u6001\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d277e37434c4406c3cade\/52-0.png\" alt=\"\u30b9\u30af\u30ea\u30fc\u30f3\u30b7\u30e7\u30c3\u30c8 2018-05-07 10.10.17.png\" \/><\/div>\n<p>\u7531\u4e8e\u6211\u4eec\u5728\u8fd9\u6b21\u4f7f\u7528spark-submit\u547d\u4ee4\u65f6\u5c06Executor\u6570\u76ee\u6307\u5b9a\u4e3a3\uff0c\u6240\u4ee5\u5bf9\u5e94\u7684\u662f3\u4e2aPods\u3002\u5982\u679c\u60a8\u60f3\u589e\u52a0\u5e76\u884c\u8ba1\u7b97\u7684\u6570\u91cf\uff0c\u8bf7\u5c1d\u8bd5\u589e\u52a0Spark Executor\u7684\u6570\u76ee\u3002<\/p>\n<p>\u53e6\u5916\uff0c\u4f1a\u7b49\u5f85\u4e00\u6bb5\u65f6\u95f4\u3002<br \/>\n\u63a5\u7740\uff0c\u6bcf\u4e2aSpark Executor\u4f1a\u5728\u8ba1\u7b97\u5b8c\u6210\u540e\u81ea\u52a8\u5220\u9664\u3002<\/p>\n<pre class=\"post-pre\"><code>$ kubectl get pods\r\nNAME                                               READY     STATUS        RESTARTS   AGE\r\nspark-pi-9efb8a5e1e3b38f789505ace52dac3fe-driver   0\/1       Completed     0          19s\r\nspark-pi-9efb8a5e1e3b38f789505ace52dac3fe-exec-1   0\/1       Terminating   0          12s\r\nspark-pi-9efb8a5e1e3b38f789505ace52dac3fe-exec-2   0\/1       Terminating   0          12s\r\nspark-pi-9efb8a5e1e3b38f789505ace52dac3fe-exec-3   0\/1       Terminating   0          12s\r\n\r\n$ kubectl get pods\r\nNAME                                               READY     STATUS      RESTARTS   AGE\r\nspark-pi-9efb8a5e1e3b38f789505ace52dac3fe-driver   0\/1       Completed   0          29s\r\n<\/code><\/pre>\n<p>\u6700\u540e\uff0c\u53ef\u4ee5\u786e\u8ba4Spark\u9a71\u52a8\u7a0b\u5e8f\u7684Pod\u5904\u4e8eREADY 0\/1\u7684\u72b6\u6001\u3002<br \/>\n\u8fd9\u662f\u5728\u5904\u7406\u6d41\u7a0b\u7ae0\u8282\u4e2d\u6240\u8ff0\u7684\uff08\u7b2c4\u6b65\uff09\u72b6\u6001\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d277e37434c4406c3cade\/57-0.png\" alt=\"\u30b9\u30af\u30ea\u30fc\u30f3\u30b7\u30e7\u30c3\u30c8 2018-05-07 10.10.04.png\" \/><\/div>\n<p>\u6839\u636eSpark\u5e76\u884c\u8ba1\u7b97\u7684\u5904\u7406\u65b9\u5f0f\uff0c\u5728Kubernetes\u4e0a\u90e8\u7f72\u548c\u5220\u9664Pod\u3002<\/p>\n<h2>\u6e05\u7406<\/h2>\n<p>\u767b\u5f55Docker Hub\u540e\uff0c\u5728&#8221;\u8bbe\u7f6e&#8221;\u6807\u7b7e\u4e0b\uff0c\u9009\u62e9&#8221;\u5220\u9664\u4ed3\u5e93&#8221;\u5e76\u70b9\u51fb&#8221;\u5220\u9664&#8221;\u6309\u94ae\uff0c\u4ee5\u4ece\u5b58\u50a8\u5e93\u4e2d\u5220\u9664\u5df2\u521b\u5efa\u7684Spark\u5bb9\u5668\u955c\u50cf\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d277e37434c4406c3cade\/61-0.png\" alt=\"\u30b9\u30af\u30ea\u30fc\u30f3\u30b7\u30e7\u30c3\u30c8 2018-05-03 11.41.27.png\" \/><\/div>\n<p>\u4ece\u672c\u5730\u7684Docker\u4e2d\u5220\u9664\u5bb9\u5668\u955c\u50cf\u3002<\/p>\n<pre class=\"post-pre\"><code>$ sudo docker images\r\nREPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE\r\nysakashita\/spark                           v2.3.0              f9cd85baa796        14 hours ago        350MB\r\n\r\n$ sudo docker rmi f9cd85baa796\r\nUntagged: ysakashita\/spark:v2.3.0\r\nUntagged: ysakashita\/spark@sha256:3252c88b5527a97b9743824ae283c40e9d69b8587155ebd8c4d6f1b451d972f8\r\nDeleted: sha256:f9cd85baa79655ccefe0d7b646b499cbf568acd064c77add7e1773dfe8e14eaa\r\n&lt;snip&gt;\r\n<\/code><\/pre>\n<p>\u6700\u7ec8\uff0c\u5c06\u5220\u9664\u5df2\u4e0b\u8f7d\u548c\u7f16\u8bd1\u7684 Spark\u3002<\/p>\n<pre class=\"post-pre\"><code>$ cd ..\r\n$ rm -rf spark-2.3.0\r\n<\/code><\/pre>\n<h1>\u4e13\u4e1a\u77e5\u8bc6<\/h1>\n<p>\u5728\u672c\u6b21\u9a8c\u8bc1\u4e2d\uff0c\u6211\u4eec\u9047\u5230\u4e86Spark Executor\u7684Pod\u65e0\u6cd5\u88ab\u6267\u884c\u800c\u4e00\u76f4\u5904\u4e8e\u6302\u8d77\u72b6\u6001\u7684\u95ee\u9898\u3002<\/p>\n<pre class=\"post-pre\"><code>$ kubectl get pods\r\nNAME                                                READY     STATUS      RESTARTS   AGE\r\n&lt;snip&gt;\r\nspark-pi-8d89987d72ba32c2a89b99876ca81129-exec-3   0\/1       Pending     0          10s\r\n&lt;snip&gt;\r\n<\/code><\/pre>\n<p>\u56e0\u6b64\uff0c\u6211\u5011\u53ef\u4ee5\u4f7f\u7528kubectl describe\u547d\u4ee4\u4f86\u6aa2\u67e5\u5931\u6557\u7684Pod\u3002<\/p>\n<pre class=\"post-pre\"><code>$ kubectl describe pods spark-pi-8d89987d72ba32c2a89b99876ca81129-exec-3\r\n&lt;snip&gt;\r\nEvents:\r\n  Type     Reason            Age                From               Message\r\n  ----     ------            ----               ----               -------\r\n  Warning  FailedScheduling  12s (x6 over 27s)  default-scheduler  0\/3 nodes are available: 1 Insufficient memory, 3 Insufficient cpu.\r\n<\/code><\/pre>\n<p>\u7531\u4e8eCPU\u548c\u5185\u5b58\u4e0d\u8db3\uff0c\u5bfc\u81f4Kubernetes\u8c03\u5ea6\u7a0b\u5e8f\u65e0\u6cd5\u5206\u914dPod\u8fd0\u884c\u7684\u8282\u70b9\u3002\u6839\u636eSpark 2.3.0\u7684\u5b98\u65b9\u6587\u6863\uff0c\u6211\u4eec\u5efa\u8bae\u4f7f\u75283\u4e2aCPU\u548c4GB\u7684\u5185\u5b58\u6765\u542f\u52a8\u4e00\u4e2a\u7b80\u5355\u7684Spark\u5e94\u7528\u7a0b\u5e8f\uff0c\u53ea\u9700\u8981\u4e00\u4e2a\u6267\u884c\u5668\u3002\u56e0\u6b64\uff0c\u9700\u8981\u81f3\u5c113\u4e2aCPU\u548c4GB\u4ee5\u4e0a\u7684\u5185\u5b58\u3002\u56e0\u6b64\uff0c\u6211\u589e\u52a0\u4e86VirtualBox\u865a\u62df\u673a\u7684CPU\u548c\u5185\u5b58\uff0c\u6210\u529f\u5730\u8fd0\u884c\u4e86Pod\u3002\u5728\u4ee5\u4e0b\u547d\u4ee4\u4e2d\uff0c\u6211\u4fee\u6539\u4e86\u5206\u914d\u7ed9VirtualBox\u865a\u62df\u673a\u7684CPU\u548c\u5185\u5b58\u3002\u8bf7\u786e\u4fdd\u5728\u64cd\u4f5cVirtualBox\u7684\u4e3b\u673a\u4e0a\u6267\u884c\u6b64\u547d\u4ee4\u4e4b\u524d\u5c06\u6240\u6709Kubernetes\u865a\u62df\u673a\u5173\u95ed\u3002<\/p>\n<pre class=\"post-pre\"><code>$ VBoxManage list vms\r\n\"k8s\" {3c2bf51f-3344-431f-8a3e-5925e065389b}\r\n\"k8s-node1\" {82226c81-30ac-4292-9987-e401d9f41356}\r\n\"k8s-node2\" {e82c1953-1d60-4f3f-9a2b-282299249f78}\r\n\r\n$ VBoxManage modifyvm k8s --memory 4096 --cpus 3\r\n$ VBoxManage modifyvm k8s-node1 --memory 4096 --cpus 3\r\n$ VBoxManage modifyvm k8s-node2 --memory 4096 --cpus 3\r\n<\/code><\/pre>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d277e37434c4406c3cade\/73-0.png\" alt=\"\u30b9\u30af\u30ea\u30fc\u30f3\u30b7\u30e7\u30c3\u30c8 2018-05-03 12.29.07.png\" \/><\/div>\n<h1>\u5370\u8c61<\/h1>\n<p>\u7b2c\u4e00\u6b21\u63a5\u89e6Spark on Kubernetes\u7684\u5370\u8c61\u662f\uff0c\u5b83\u5e76\u4e0d\u662f\u4e3a&#8221;Kubernetes&#8221;\u7528\u6237\u800c\u8bbe\u8ba1\u7684Spark\u96c6\u6210\u5de5\u5177\uff0c\u800c\u662f\u4e3a&#8221;Spark&#8221;\u7528\u6237\u800c\u8bbe\u8ba1\u7684Kubernetes\u96c6\u6210\u5de5\u5177\u3002\u5bf9\u4e8e\u5df2\u7ecf\u4f7f\u7528Kubernetes\u7684\u7528\u6237\u6765\u8bf4\uff0c\u53ef\u80fd\u4f1a\u611f\u89c9\u6709\u70b9\u4e0d\u4e60\u60ef\uff0c\u4f8b\u5982\u4e0d\u4f7f\u7528Kubernetes\u7684Replicaset\uff0c\u800c\u662f\u81ea\u5df1\u63a7\u5236Executor\u7684\u6570\u91cf\u3002<\/p>\n<p>\u80fd\u591f\u719f\u7ec3\u4f7f\u7528Spark\u548cKubernetes\u7684\u7528\u6237\u53ef\u80fd\u5728\u5168\u7403\u8303\u56f4\u5185\u90fd\u5f88\u5c11\u89c1\u3002\u6211\u8ba4\u4e3aSpark\u7684\u7528\u6237\u4e3b\u8981\u662f\u90a3\u4e9b\u62e5\u6709\u7c7b\u4f3c\u6570\u636e\u79d1\u5b66\u5bb6\u4e13\u6ce8\u4e8e\u6570\u636e\u5206\u6790\u7b49\u6280\u80fd\u7684\u4eba\u3002\u5bf9\u4e8e\u8fd9\u6837\u7684\u7528\u6237\u6765\u8bf4\uff0c\u53ef\u4ee5\u901a\u8fc7\u4f20\u7edf\u7684Spark\u4e2d\u4f7f\u7528\u7684spark-submit\u547d\u4ee4\u6765\u6307\u5b9a\u5e76\u884c\u8ba1\u7b97\u53ef\u80fd\u4f1a\u8ba9\u4ed6\u4eec\u611f\u5230\u9ad8\u5174\u3002\u9664\u4e86&#8211;conf\u53c2\u6570\u5916\uff0c&#8211;master\u53c2\u6570\u53ef\u4ee5\u6307\u5b9a\u4e3ak8s:\/\/\uff0c\u5176\u4ed6\u9009\u9879\u90fd\u4e0d\u53d8\u3002\u56e0\u6b64\uff0c\u5373\u4f7f\u662f\u4e0d\u4e86\u89e3\u57fa\u7840\u8bbe\u65bd\u6216Kubernetes\u7684Spark\u7528\u6237\uff0c\u4e5f\u89c9\u5f97\u4f7f\u7528\u8d77\u6765\u95e8\u69db\u8f83\u4f4e\u3002<\/p>\n<p>\u6b64\u5916\uff0c\u8fd8\u9700\u8981\u8003\u8651\u53e6\u4e00\u4e2a\u65b9\u9762\u7684\u95ee\u9898\uff0c\u5373Spark\u7528\u6237\u662f\u5426\u53ef\u4ee5\u521b\u5efa\u548c\u6ce8\u518c\u5bb9\u5668\u955c\u50cf\u3002\u5728\u8fd9\u6b21\u9a8c\u8bc1\u4e2d\uff0c\u7531\u4e8e\u4f7f\u7528\u4e86\u793a\u4f8b\u7a0b\u5e8f\uff0c\u6240\u4ee5Spark\u5e94\u7528\u7a0b\u5e8f\u5df2\u7ecf\u5305\u542b\u5728\u5bb9\u5668\u4e2d\uff0c\u4f46\u4e5f\u53ef\u4ee5\u5c06\u5176\u653e\u5728\u5916\u90e8\u800c\u4e0d\u5305\u542b\u5728\u5bb9\u5668\u4e2d\uff08\u53c2\u89c1Using Remote Dependencies\uff09\u3002\u8fd9\u6837\u4e00\u6765\uff0c\u6bcf\u6b21\u66f4\u65b0Spark\u5e94\u7528\u7a0b\u5e8f\u65f6\uff0c\u65e0\u9700\u91cd\u65b0\u521b\u5efa\u548c\u6ce8\u518c\u5bb9\u5668\u955c\u50cf\u3002\u53ea\u9700\u7531\u5177\u5907\u7ba1\u7406Kubernetes\u96c6\u7fa4\u7684\u57fa\u7840\u8bbe\u65bd\u6280\u80fd\u7684\u7ba1\u7406\u5458\u521b\u5efa\u5e76\u6ce8\u518c\u4e00\u6b21Spark on Kubernetes\u7684\u5bb9\u5668\u955c\u50cf\uff0cSpark\u7528\u6237\u53ea\u9700\u6307\u5b9aspark.kubernetes.container.image\u53c2\u6570\u5c31\u53ef\u4ee5\u4f7f\u7528Spark on Kubernetes\u3002<\/p>\n<p>\u5982\u4e0a\u6240\u8ff0\uff0c\u6211\u8ba4\u4e3aSpark on Kubernetes\u5bf9\u4e8e&#8221;Spark&#8221;\u7528\u6237\u6765\u8bf4\u662f\u6613\u4e8e\u4f7f\u7528\u7684\u3002\u8981\u63d0\u5230\u7684\u9057\u61be\u4e4b\u5904\u662f\uff0c\u5728v2.3.0\u7248\u672c\u4e2d\u5c1a\u4e0d\u652f\u6301Job Queues\u3002\u6211\u8ba4\u4e3aSpark\u7528\u6237\u66f4\u591a\u5730\u4f7f\u7528\u6279\u5904\u7406\u7b49\u4f5c\u4e1a\uff0c\u800c\u4e0d\u662f\u4ea4\u4e92\u5f0f\u8ba1\u7b97\u3002\u5173\u4e8e\u8fd9\u4e00\u70b9\uff0c\u5b83\u88ab\u5217\u4e3a\u5c06\u6765\u7684\u5de5\u4f5c\uff0c\u6211\u671f\u5f85\u7740\u5b83\u7684\u5230\u6765\u3002<\/p>\n<h1>\u53c2\u8003\u8d44\u6599<\/h1>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">Running Spark on Kubernetes<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">apache-spark-on-k8s\/spark<\/ul>\n<h1>\u6dfb\u52a0\u5185\u5bb9\uff082018\u5e748\u6708\uff09\u3002<\/h1>\n<p>\u6211\u5011\u5728Kubernetes\u4e0a\u767c\u5e03\u4e86\u4e00\u7bc7\u95dc\u65bc\u9a57\u8b49HDFS\u7684\u6587\u7ae0\u3002<br \/>\n\u9019\u610f\u5473\u8457\u6211\u5011\u53ef\u4ee5\u5728Kubernetes\u4e0a\u7ba1\u7406\u5b58\u5132spark.files\uff08\u6216&#8211;files\uff09\u7684HDFS\u6578\u64da\u4f4d\u7f6e\uff0c\u9019\u5728&#8221;Using Remote Dependencies&#8221;\u4e2d\u6709\u63d0\u5230\u7684spark-submit\u547d\u4ee4\u4e2d\u53ef\u4ee5\u6307\u5b9a\u3002<\/p>\n<div>\u5982\u679c\u9000\u51fa\u4ee3\u7801\u4e0d\u4e3a0\uff0c\u5219\u53ef\u80fd\u662f\u8ba1\u7b97\u8d44\u6e90\u4e0d\u8db3\u3002\u8bf7\u53c2\u8003&#8221;\u77e5\u8bc6&#8221;\u90e8\u5206\uff0c\u5c1d\u8bd5\u589e\u52a0Kubernetes\u7684\u8ba1\u7b97\u8d44\u6e90\u3002<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Apache Spark on Kubernetes \u662f\u4ec0\u4e48 Apache Spark\uff08\u4ee5\u4e0b\u79f0\u4e3aSpark\uff09\u4ece [&hellip;]<\/p>\n","protected":false},"author":5,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-36137","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.5 (Yoast SEO v21.5) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>\u5728Kubernetes\u4e0a\u9a8c\u8bc1Apache Spark - Blog - Silicon Cloud<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.silicloud.com\/zh\/blog\/\u5728kubernetes\u4e0a\u9a8c\u8bc1apache-spark\/\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"\u5728Kubernetes\u4e0a\u9a8c\u8bc1Apache Spark\" \/>\n<meta property=\"og:description\" content=\"Apache Spark on Kubernetes \u662f\u4ec0\u4e48 Apache Spark\uff08\u4ee5\u4e0b\u79f0\u4e3aSpark\uff09\u4ece [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.silicloud.com\/zh\/blog\/\u5728kubernetes\u4e0a\u9a8c\u8bc1apache-spark\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog - Silicon Cloud\" \/>\n<meta property=\"article:published_time\" content=\"2024-01-20T14:54:15+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-04-30T12:13:16+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d277e37434c4406c3cade\/4-0.png\" \/>\n<meta name=\"author\" content=\"\u6e05, \u5b87\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"\u6e05, \u5b87\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8kubernetes%e4%b8%8a%e9%aa%8c%e8%af%81apache-spark\/\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8kubernetes%e4%b8%8a%e9%aa%8c%e8%af%81apache-spark\/\",\"name\":\"\u5728Kubernetes\u4e0a\u9a8c\u8bc1Apache Spark - Blog - Silicon Cloud\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\"},\"datePublished\":\"2024-01-20T14:54:15+00:00\",\"dateModified\":\"2024-04-30T12:13:16+00:00\",\"author\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/1a6ecd3d914d22a5ac32791ffc1fbd8e\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8kubernetes%e4%b8%8a%e9%aa%8c%e8%af%81apache-spark\/#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8kubernetes%e4%b8%8a%e9%aa%8c%e8%af%81apache-spark\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8kubernetes%e4%b8%8a%e9%aa%8c%e8%af%81apache-spark\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.silicloud.com\/zh\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"\u5728Kubernetes\u4e0a\u9a8c\u8bc1Apache Spark\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/\",\"name\":\"Blog - Silicon Cloud\",\"description\":\"\",\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/1a6ecd3d914d22a5ac32791ffc1fbd8e\",\"name\":\"\u6e05, \u5b87\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/4b2016c18459a605fc469c7566608f5686491baa112d0871ee613f61b7210565?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/4b2016c18459a605fc469c7566608f5686491baa112d0871ee613f61b7210565?s=96&d=mm&r=g\",\"caption\":\"\u6e05, \u5b87\"},\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/author\/qingyu\/\"},{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8kubernetes%e4%b8%8a%e9%aa%8c%e8%af%81apache-spark\/#local-main-organization-logo\",\"url\":\"\",\"contentUrl\":\"\",\"caption\":\"Blog - Silicon Cloud\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"\u5728Kubernetes\u4e0a\u9a8c\u8bc1Apache Spark - Blog - Silicon Cloud","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.silicloud.com\/zh\/blog\/\u5728kubernetes\u4e0a\u9a8c\u8bc1apache-spark\/","og_locale":"zh_CN","og_type":"article","og_title":"\u5728Kubernetes\u4e0a\u9a8c\u8bc1Apache Spark","og_description":"Apache Spark on Kubernetes \u662f\u4ec0\u4e48 Apache Spark\uff08\u4ee5\u4e0b\u79f0\u4e3aSpark\uff09\u4ece [&hellip;]","og_url":"https:\/\/www.silicloud.com\/zh\/blog\/\u5728kubernetes\u4e0a\u9a8c\u8bc1apache-spark\/","og_site_name":"Blog - Silicon Cloud","article_published_time":"2024-01-20T14:54:15+00:00","article_modified_time":"2024-04-30T12:13:16+00:00","og_image":[{"url":"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d277e37434c4406c3cade\/4-0.png"}],"author":"\u6e05, \u5b87","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"\u6e05, \u5b87","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"6 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8kubernetes%e4%b8%8a%e9%aa%8c%e8%af%81apache-spark\/","url":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8kubernetes%e4%b8%8a%e9%aa%8c%e8%af%81apache-spark\/","name":"\u5728Kubernetes\u4e0a\u9a8c\u8bc1Apache Spark - Blog - Silicon Cloud","isPartOf":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website"},"datePublished":"2024-01-20T14:54:15+00:00","dateModified":"2024-04-30T12:13:16+00:00","author":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/1a6ecd3d914d22a5ac32791ffc1fbd8e"},"breadcrumb":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8kubernetes%e4%b8%8a%e9%aa%8c%e8%af%81apache-spark\/#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8kubernetes%e4%b8%8a%e9%aa%8c%e8%af%81apache-spark\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8kubernetes%e4%b8%8a%e9%aa%8c%e8%af%81apache-spark\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.silicloud.com\/zh\/blog\/"},{"@type":"ListItem","position":2,"name":"\u5728Kubernetes\u4e0a\u9a8c\u8bc1Apache Spark"}]},{"@type":"WebSite","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website","url":"https:\/\/www.silicloud.com\/zh\/blog\/","name":"Blog - Silicon Cloud","description":"","inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/1a6ecd3d914d22a5ac32791ffc1fbd8e","name":"\u6e05, \u5b87","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/4b2016c18459a605fc469c7566608f5686491baa112d0871ee613f61b7210565?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4b2016c18459a605fc469c7566608f5686491baa112d0871ee613f61b7210565?s=96&d=mm&r=g","caption":"\u6e05, \u5b87"},"url":"https:\/\/www.silicloud.com\/zh\/blog\/author\/qingyu\/"},{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%9c%a8kubernetes%e4%b8%8a%e9%aa%8c%e8%af%81apache-spark\/#local-main-organization-logo","url":"","contentUrl":"","caption":"Blog - Silicon Cloud"}]}},"_links":{"self":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/36137","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/comments?post=36137"}],"version-history":[{"count":2,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/36137\/revisions"}],"predecessor-version":[{"id":94052,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/36137\/revisions\/94052"}],"wp:attachment":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/media?parent=36137"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/categories?post=36137"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/tags?post=36137"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}