{"id":47266,"date":"2023-11-05T00:34:21","date_gmt":"2023-03-19T07:31:53","guid":{"rendered":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%b0%9d%e8%af%95%e4%bd%bf%e7%94%a8spark-x-keras-x-docker%e5%af%b9%e6%b7%b1%e5%ba%a6%e5%ad%a6%e4%b9%a0%e8%bf%9b%e8%a1%8c%e5%8f%af%e6%89%a9%e5%b1%95%e7%9a%84%e5%b0%9d%e8%af%952%ef%bc%9a\/"},"modified":"2024-05-04T07:34:58","modified_gmt":"2024-05-03T23:34:58","slug":"%e5%b0%9d%e8%af%95%e4%bd%bf%e7%94%a8spark-x-keras-x-docker%e5%af%b9%e6%b7%b1%e5%ba%a6%e5%ad%a6%e4%b9%a0%e8%bf%9b%e8%a1%8c%e5%8f%af%e6%89%a9%e5%b1%95%e7%9a%84%e5%b0%9d%e8%af%952%ef%bc%9a","status":"publish","type":"post","link":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%b0%9d%e8%af%95%e4%bd%bf%e7%94%a8spark-x-keras-x-docker%e5%af%b9%e6%b7%b1%e5%ba%a6%e5%ad%a6%e4%b9%a0%e8%bf%9b%e8%a1%8c%e5%8f%af%e6%89%a9%e5%b1%95%e7%9a%84%e5%b0%9d%e8%af%952%ef%bc%9a\/","title":{"rendered":"\u5c1d\u8bd5\u4f7f\u7528Spark \u00d7 Keras \u00d7 Docker\u5bf9\u6df1\u5ea6\u5b66\u4e60\u8fdb\u884c\u53ef\u6269\u5c55\u7684\u5c1d\u8bd52\uff1a\u591a\u8282\u70b9\u7248\u672c"},"content":{"rendered":"<h1>\u901a\u8fc7 Spark \u00d7 Keras \u00d7 Docker \u5c1d\u8bd5\u5b9e\u73b0\u53ef\u6269\u5c55\u7684\u6df1\u5ea6\u5b66\u4e602\u3002<\/h1>\n<p>\u6211\u4ee5\u524d\u5c1d\u8bd5\u5c06Dist-keras\u653e\u5165Docker\u4e2d\uff0c\u4ee5\u521b\u5efa\u53ef\u6269\u5c55\u7684\u6df1\u5ea6\u5b66\u4e60\u6a21\u578b\u3002<br \/>\n\u60a8\u53ef\u4ee5\u5728\u6b64\u94fe\u63a5\u4e2d\u4e86\u89e3\u66f4\u591a\u4fe1\u606f\uff1ahttp:\/\/qiita.com\/cvusk\/items\/3e6c3bade8c0e1c0d9bf<\/p>\n<p>\u5f53\u65f6\u7684\u53cd\u7701\u70b9\u662f\u8868\u73b0\u4e0d\u4f73\uff0c\u4ed4\u7ec6\u56de\u987e\u540e\u53d1\u73b0\u53c2\u6570\u8bbe\u7f6e\u9519\u4e86\u3002<br \/>\n\u4e8e\u662f\u4e3a\u4e86\u53cd\u7701\u800c\u5c1d\u8bd5\u4e86\u5f88\u591a\u4e0d\u540c\u7684\u65b9\u6cd5\u3002<\/p>\n<h2>\u4e4b\u524d\u7684\u6545\u4e8b\u6897\u6982<\/h2>\n<p>\u8bf7\u53c2\u8003\u5148\u524d\u7684\u5e16\u5b50\u83b7\u53d6\u5173\u4e8eDist-Keras\u672c\u8eab\u7684\u89e3\u91ca\uff0c\u57fa\u672c\u4e0a\u5b83\u662f\u5728Spark\u96c6\u7fa4\u4e0a\u8fd0\u884c\u7684Keras\u3002<br \/>\n\u6211\u5c06\u5176\u5236\u4f5c\u6210\u4e86Docker\u6620\u50cf\uff0c\u4ee5\u4fbf\u53ef\u4ee5\u8f7b\u677e\u5730\u6269\u5c55\u3002<\/p>\n<p>\u53e6\u5916\uff0cDockerfile\u5df2\u5728GitHub\u4e0a\u516c\u5f00\u3002<br \/>\nhttps:\/\/github.com\/shibuiwilliam\/distkeras-docker<\/p>\n<h2>\u6211\u8981\u505a\u7684\u4e8b\u60c5<\/h2>\n<p>\u6211\u60f3\u5728\u672c\u6b21\u6d4b\u8bd5\u4e2d\uff0c\u4f7f\u7528Docker\u4e0a\u7684Dist-Keras\u5728\u5355\u4e3b\u673a\u548c\u591a\u4e3b\u673a\u4e0a\u8fdb\u884c\u9a8c\u8bc1\uff0c\u4ee5\u6539\u5584\u6027\u80fd\u3002<br \/>\n\u4e4b\u524d\uff0c\u6211\u5728\u5355\u4e3b\u673a\u4e0a\u542f\u52a8\u4e86\u591a\u4e2a\u5bb9\u5668\uff0c\u521b\u5efa\u4e86Spark\u96c6\u7fa4\u3002<br \/>\n\u8fd9\u6b21\uff0c\u6211\u60f3\u589e\u52a0\u4e00\u4e9b\u6a21\u5f0f\u3002<\/p>\n<p>\u6211\u4eec\u5c06\u8fd0\u884c\u7684\u662fMNIST\u3002<br \/>\nMNIST\u7684\u8bad\u7ec3\u7a0b\u5e8f\u662f\u57fa\u4e8edist-keras\u63d0\u4f9b\u7684\u8fdb\u884c\u81ea\u5b9a\u4e49\u4f7f\u7528\u3002<\/p>\n<h2>\u9a8c\u8bc1\u8fc7\u7684\u7ed3\u6784 le de<\/h2>\n<p>\u6211\u4eec\u5c06\u4f7f\u7528\u5355\u4e3b\u673a\u548c\u591a\u4e3b\u673a\u8fdb\u884c\u9a8c\u8bc1\u3002<br \/>\n\u65e0\u8bba\u662f\u5355\u4e3b\u673a\u8fd8\u662f\u591a\u4e3b\u673a\uff0c\u90fd\u662f\u4ee5Spark\u4e3b\u8282\u70b9+\u5de5\u4f5c\u8282\u70b9\u7684\u914d\u7f6e\uff0c\u5e76\u8c03\u6574\u5de5\u4f5c\u8282\u70b9\u7684\u6570\u91cf\u548c\u89c4\u683c\u3002<br \/>\n\u5728\u591a\u4e3b\u673a\u7684\u60c5\u51b5\u4e0b\uff0c\u6709\u4e24\u53f0\u670d\u52a1\u5668\u3002<br \/>\n\u503c\u5f97\u6ce8\u610f\u7684\u662f\uff0c\u4e3b\u673a\u662f\u4f7f\u7528AWS EC2 CentOS7.3\u7684m4.xlarge\u3002<\/p>\n<div>\n<div class=\"post-table\">nohostsworkersresources1single11 processor, 2GB RAM2single22 processors, 5GB RAM3single31 processor, 3GB RAM4multihost22 processors, 5GB RAM5multihost23 processors, 8GB RAM6multihost42 processors, 5GB RAM<\/div>\n<\/div>\n<h2>\u5355\u4e2a\u4e3b\u673a\u7684\u8bbe\u7f6e\u65b9\u6cd5<\/h2>\n<p>\u5728\u5355\u4e00\u4e3b\u673a\u7684\u60c5\u51b5\u4e0b\uff0c\u56fe\u50cf\u4f1a\u662f\u8fd9\u6837\u7684\u3002<br \/>\nDocker\u5bb9\u5668\u7684\u6570\u91cf\u4f1a\u6839\u636e\u9a8c\u8bc1\u6761\u4ef6\u505a\u51fa\u8c03\u6574\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7dc3913a08637a6a0e96\/14-0.jpeg\" alt=\"14.jpg\" \/><\/div>\n<p>\u5728\u5355\u4e00\u4e3b\u673a\u4e0a\u542f\u52a8\u591a\u4e2aDocker\u5bb9\u5668\uff0c\u8fd9\u662f\u5f53\u53ea\u6709\u4e00\u4e2a\u4e3b\u673a\u65f6\u7684\u60c5\u51b5\u3002<\/p>\n<pre class=\"post-pre\"><code># docker dist-keras for spark master and slave\r\ndocker run -it -p 18080:8080 -p 17077:7077 -p 18888:8888 -p 18081:8081 -p 14040:4040 -p 17001:7001 -p 17002:7002 \\\r\n -p 17003:7003 -p 17004:7004 -p 17005:7005 -p 17006:7006 --name spmaster -h spmaster distkeras \/bin\/bash\r\n\r\n# docker dist-keras for spark slave1\r\ndocker run -it --link spmaster:master -p 28080:8080 -p 27077:7077 -p 28888:8888 -p 28081:8081 -p 24040:4040 -p 27001:7001 \\\r\n-p 27002:7002 -p 27003:7003 -p 27004:7004 -p 27005:7005 -p 27006:7006 --name spslave1 -h spslave1 distkeras \/bin\/bash\r\n\r\n# docker dist-keras for spark slave2\r\ndocker run -it --link spmaster:master -p 38080:8080 -p 37077:7077 -p 38888:8888 -p 38081:8081 -p 34040:4040 -p 37001:7001 \\\r\n-p 37002:7002 -p 37003:7003 -p 37004:7004 -p 37005:7005 -p 37006:7006 --name spslave2 -h spslave2 distkeras \/bin\/bash\r\n\r\n\r\n<\/code><\/pre>\n<p>\u4f7f\u7528Spark Master\u6765\u542f\u52a8Spark\u4e3b\u8282\u70b9\u548c\u5de5\u4f5c\u8282\u70b9\uff0c\u800cSpark Slave\u4ec5\u7528\u4e8e\u542f\u52a8\u5de5\u4f5c\u8282\u70b9\u3002<\/p>\n<pre class=\"post-pre\"><code># for spark master\r\n${SPARK_HOME}\/sbin\/start-master.sh\r\n\r\n# for spark worker\r\n${SPARK_HOME}\/sbin\/start-slave.sh -c 1 -m 3G spark:\/\/spmaster:${SPARK_MASTER_PORT}\r\n\r\n<\/code><\/pre>\n<p>\u63a5\u4e0b\u6765\uff0c\u6211\u4eec\u5c06\u5728Spark\u7684\u4e3b\u8282\u70b9\u4e0a\u5bf9MNIST\u7a0b\u5e8f\u8fdb\u884c\u5b9a\u5236\u3002<br \/>\nMNIST\u7684\u793a\u4f8b\u4ee3\u7801\u662f\u4f7f\u7528Dist-Keras\u63d0\u4f9b\u7684\u3002<br \/>\n\u4ee3\u7801\u5b58\u653e\u5728\u76ee\u5f55\/opt\/dist-keras\/examples\u4e2d\uff0c\u90a3\u91cc\u5305\u542b\u4ee5\u4e0b\u7684\u793a\u4f8b\u6570\u636e\u548c\u7a0b\u5e8f\u3002<\/p>\n<pre class=\"post-pre\"><code>[root@spm examples]# tree\r\n.\r\n|-- cifar-10-preprocessing.ipynb\r\n|-- data\r\n|   |-- atlas_higgs.csv\r\n|   |-- mnist.csv\r\n|   |-- mnist.zip\r\n|   |-- mnist_test.csv\r\n|   `-- mnist_train.csv\r\n|-- example_0_data_preprocessing.ipynb\r\n|-- example_1_analysis.ipynb\r\n|-- kafka_producer.py\r\n|-- kafka_spark_high_throughput_ml_pipeline.ipynb\r\n|-- mnist.ipynb\r\n|-- mnist.py\r\n|-- mnist_analysis.ipynb\r\n|-- mnist_preprocessing.ipynb\r\n|-- spark-warehouse\r\n`-- workflow.ipynb\r\n\r\n<\/code><\/pre>\n<p>\u590d\u5236\u539f\u59cb\u6587\u4ef6\u4ee5\u5907\u4efd\uff0c\u7136\u540e\u5e94\u7528\u4ee5\u4e0b\u66f4\u6539\u3002<\/p>\n<pre class=\"post-pre\"><code>cp mnist.py mnist.py.bk\r\n\r\n<\/code><\/pre>\n<h5>\u4fee\u6539\u98791\uff1a\u5bfc\u5165SparkSession\u3002<\/h5>\n<p>\u6211\u4f1a\u628a\u4ee5\u4e0b\u5185\u5bb9\u6dfb\u52a0\u5230\u5f00\u5934\u3002<\/p>\n<pre class=\"post-pre\"><code>from pyspark.sql import SparkSession\r\n\r\n<\/code><\/pre>\n<h5>\u4fee\u6539\u90e8\u52062\uff1a\u53c2\u6570\u8bbe\u7f6e<\/h5>\n<p>\u5c06Spark\u53c2\u6570\u6839\u636e\u672c\u6b21\u73af\u5883\u8fdb\u884c\u66f4\u6539\u3002<br \/>\n\u66f4\u6539\u7684\u610f\u56fe\u5982\u4e0b\uff1a<br \/>\n&#8211; \u4f7f\u7528Spark2<br \/>\n&#8211; \u4f7f\u7528\u672c\u5730\u73af\u5883<br \/>\n&#8211; \u5728\u672c\u5730\u73af\u5883\u4e2d\u5b9a\u4e49\u4e3b\u8282\u70b9url<br \/>\n&#8211; \u6839\u636e\u9a8c\u8bc1\u6761\u4ef6\u66f4\u6539\u5904\u7406\u5668\u6570\u91cf<br \/>\n&#8211; \u6839\u636e\u9a8c\u8bc1\u6761\u4ef6\u66f4\u6539\u5de5\u4f5c\u8282\u70b9\u6570\u91cf<\/p>\n<pre class=\"post-pre\"><code># Modify these variables according to your needs.\r\napplication_name = \"Distributed Keras MNIST\"\r\nusing_spark_2 = True  # False to True\r\nlocal = True  # False to True\r\npath_train = \"data\/mnist_train.csv\"\r\npath_test = \"data\/mnist_test.csv\"\r\nif local:\r\n    # Tell master to use local resources.\r\n#     master = \"local[*]\"   comment out\r\n    master = \"spark:\/\/spm:7077\"  # add\r\n    num_processes = 1 # change to number of processors per worker\r\n    num_executors = 3  # change to number of workers\r\nelse:\r\n    # Tell master to use YARN.\r\n    master = \"yarn-client\"\r\n    num_executors = 20\r\n    num_processes = 1\r\n<\/code><\/pre>\n<h5>\u53d8\u66f4\u98793\uff1a\u5de5\u4eba\u7684\u5185\u5b58<\/h5>\n<p>\u6839\u636e\u9a8c\u8bc1\u6761\u4ef6\uff0c\u8c03\u6574\u5de5\u4eba\u7684\u5185\u5b58\u3002<\/p>\n<pre class=\"post-pre\"><code>conf = SparkConf()\r\nconf.set(\"spark.app.name\", application_name)\r\nconf.set(\"spark.master\", master)\r\nconf.set(\"spark.executor.cores\", `num_processes`)\r\nconf.set(\"spark.executor.instances\", `num_executors`)\r\nconf.set(\"spark.executor.memory\", \"4g\") # change RAM size\r\nconf.set(\"spark.locality.wait\", \"0\")\r\nconf.set(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\");\r\n\r\n<\/code><\/pre>\n<p>\u5728Spark\u4e3b\u8282\u70b9\u4e0a\uff0c\u60a8\u53ef\u4ee5\u4f7f\u7528python mnist.py\u6765\u8fd0\u884c\u3002<\/p>\n<h2>\u591a\u4e2a\u4e3b\u673a\u7684\u914d\u7f6e\u65b9\u6cd5<\/h2>\n<p>\u591a\u4e3b\u673a\u914d\u7f6e\u5982\u4e0b\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7dc3913a08637a6a0e96\/35-0.jpeg\" alt=\"23.jpg\" \/><\/div>\n<p>\u5728\u591a\u4e3b\u673a\u73af\u5883\u4e2d\uff0c\u9700\u8981\u4f7f\u7528overlay\u7f51\u7edc\u6765\u8fde\u63a5Docker\u5bb9\u5668\u3002<br \/>\n\u6709\u5173\u5728\u591a\u4e3b\u673a\u73af\u5883\u4e0b\u6784\u5efaDocker\u7f51\u7edc\u7684\u8be6\u7ec6\u6b65\u9aa4\uff0c\u8bf7\u67e5\u9605\u4ee5\u4e0b\u94fe\u63a5\uff1a<br \/>\nhttp:\/\/knowledge.sakura.ad.jp\/knowledge\/4786\/<br \/>\n<iframe class=\"wp-embedded-content\" sandbox=\"allow-scripts\" security=\"restricted\" title=\"\u30de\u30eb\u30c1\u30db\u30b9\u30c8\u4e0a\u306boverlay network\u3092\u69cb\u7bc9\u3057\u3066\u30b3\u30f3\u30c6\u30ca\u9593\u3067\u758e\u901a - Carpe Diem\" src=\"https:\/\/hatenablog-parts.com\/embed?url=http%3A%2F%2Fchristina04.hatenablog.com%2Fentry%2F2016%2F05%2F16%2F065853#?secret=V4Cb1AD94M\" data-secret=\"V4Cb1AD94M\" scrolling=\"no\" frameborder=\"0\"><\/iframe><\/p>\n<p>\u5728\u8fd9\u91cc\uff0c\u6211\u53ea\u4f1a\u5199\u51fa\u6211\u7684\u6b65\u9aa4\u3002<br \/>\n\u5728EC2\u4e0a\u51c6\u5907host1\u548chost2\uff0c\u5b89\u88c5etcd\u5e76\u5728host1\u4e0a\u542f\u52a8\u3002<\/p>\n<pre class=\"post-pre\"><code>yum -y install etcd\r\n\r\nvi \/etc\/etcd\/etcd.conf\r\nsystemctl enable etcd\r\nsystemctl start etcd\r\n\r\n<\/code><\/pre>\n<p>\u63a5\u4e0b\u6765\uff0c\u6211\u4eec\u5c06\u5728host1\u548chost2\u4e0a\u90fd\u6dfb\u52a0Docker\u7f51\u7edc\u7684\u914d\u7f6e\u3002<\/p>\n<pre class=\"post-pre\"><code># edit docker-network file\r\nvi \/etc\/sysconfig\/docker-network\r\n\r\n# for host1\r\nDOCKER_NETWORK_OPTIONS='--cluster-store=etcd:\/\/&lt;host1&gt;:2379 --cluster-advertise=&lt;host1&gt;:2376'\r\n\r\n# for host2\r\nDOCKER_NETWORK_OPTIONS='--cluster-store=etcd:\/\/&lt;host1&gt;:2379 --cluster-advertise=&lt;host2&gt;:2376'\r\n\r\n# from host2 to ensure network connection to host1 etcd is available\r\ncurl -L http:\/\/&lt;host1&gt;:2379\/version\r\n{\"etcdserver\":\"3.1.3\",\"etcdcluster\":\"3.1.0\"}\r\n\r\n<\/code><\/pre>\n<p>\u8fd9\u5c06\u4f7f\u5f97Docker\u4e4b\u95f4\u7684\u7f51\u7edc\u8fde\u63a5\u6210\u4e3a\u53ef\u80fd\uff0c\u7136\u540e\u53ea\u9700\u5728host1\u4e0a\u521b\u5efaDocker\u7f51\u7edc\u3002<br \/>\n\u5728\u8fd9\u91cc\uff0c\u6211\u4eec\u5c06\u4f7f\u7528\u5b50\u7f5110.0.1.0\/24\u521b\u5efa\u540d\u4e3atest1\u7684Docker\u7f51\u7edc\u3002<\/p>\n<pre class=\"post-pre\"><code>\r\n# for host1\r\ndocker network create --subnet=10.0.1.0\/24 -d overlay test1\r\n\r\n<\/code><\/pre>\n<p>\u5982\u679c\u5728\u6700\u540e\u8fd0\u884cdocker network ls\u547d\u4ee4\u65f6\uff0c\u51fa\u73b0\u4e86test1\u7f51\u7edc\uff0c\u5219\u8868\u793aOK\u3002<\/p>\n<pre class=\"post-pre\"><code>\r\nNETWORK ID          NAME                DRIVER              SCOPE\r\nfeb90a5a5901        bridge              bridge              local\r\nde3c98c59ba6        docker_gwbridge     bridge              local\r\nd7bd500d1822        host                host                local\r\nd09ac0b6fed4        none                null                local\r\n9d4c66170ea0        test1               overlay             global\r\n\r\n<\/code><\/pre>\n<p>\u63a5\u4e0b\u6765\u5c06\u5411test1\u7f51\u7edc\u4e2d\u6dfb\u52a0Docker\u5bb9\u5668\u3002<br \/>\n\u6211\u4eec\u5c06\u5c1d\u8bd5\u5728host1\u548chost2\u4e0a\u90e8\u7f72Docker\u5bb9\u5668\u3002<\/p>\n<pre class=\"post-pre\"><code># for host1 as spark master\r\ndocker run -it --net=test1 --ip=10.0.1.10 -p 18080:8080 -p 17077:7077 -p 18888:8888 -p 18081:8081 -p 14040:4040 -p 17001:7001 -p 17002:7002 \\\r\n-p 17003:7003 -p 17004:7004 -p 17005:7005 -p 17006:7006 --name spm -h spm distkeras \/bin\/bash\r\n\r\n# for host2 as spark slave\r\ndocker run -it --net=test1 --ip=10.0.1.20 --link=spm:master -p 28080:8080 -p 27077:7077 -p 28888:8888 -p 28081:8081 -p 24040:4040 -p 27001:7001 \\\r\n-p 27002:7002 -p 27003:7003 -p 27004:7004 -p 27005:7005 -p 27006:7006 --name sps1 -h sps1 distkeras \/bin\/bash\r\n\r\n<\/code><\/pre>\n<p>\u73b0\u5728\u5728test1\u7f51\u7edc\u4e2d\u6210\u529f\u90e8\u7f72\u4e86\u4e24\u4e2aDocker\u5bb9\u5668\uff0c\u53ef\u4ee5\u4f7f\u7528\u591a\u4e3b\u673a\u3002<br \/>\n\u63a5\u4e0b\u6765\uff0c\u6309\u7167\u5355\u4e3b\u673a\u7684\u6b65\u9aa4\u542f\u52a8Spark Master\u548cSpark Worker\uff0c\u5e76\u7f16\u8f91MNIST.py\u6587\u4ef6\uff0c\u7136\u540e\u6267\u884cpython mnist.py\u3002<\/p>\n<h2>\u6027\u80fd\u9a8c\u8bc1\u7ed3\u679c<\/h2>\n<p>\u8fd9\u4e2a\u5b9e\u9a8c\u662f\u4e3a\u4e86\u9a8c\u8bc1\u6bcf\u79cd\u6784\u6210\u7684\u6027\u80fd\u800c\u8fdb\u884c\u7684\u3002\u6211\u4eec\u8fd9\u6b21\u6d4b\u91cf\u7684\u662f\u5b8c\u6210\u6240\u9700\u7684\u65f6\u95f4\uff08\u79d2\u6570\uff09\u3002<\/p>\n<div>\n<div class=\"post-table\">no.hostsworkersresourcestime in second1single11 processor, 2GB RAM1615.637572single22 processors, 5GB RAM1418.569353single31 processor, 3GB RAM1475.842124multihost22 processors, 5GB RAM805.3825185multihost23 processors, 8GB RAM734.2903246multihost42 processors, 5GB RAM723.878466<\/div>\n<\/div>\n<p>\u591a\u91cd\u4e3b\u673a\u8bbe\u7f6e\u53ef\u4ee5\u63d0\u9ad8\u6027\u80fd\u3002\u6211\u8ba4\u4e3a\u8fd9\u662f\u56e0\u4e3a\u53ef\u7528\u8d44\u6e90\u7684\u6570\u91cf\u4ea7\u751f\u4e86\u6027\u80fd\u5dee\u5f02\u3002<br \/>\n\u5c3d\u7ba1\u9a8c\u8bc12\u548c\u9a8c\u8bc14\u7684Docker\u5bb9\u5668\u914d\u7f6e\u548c\u5de5\u4f5c\u8d44\u6e90\u4f7f\u7528\u76f8\u540c\uff0c\u4f46\u4ecd\u7136\u5b58\u5728600\u79d2\u7684\u5dee\u8ddd\u3002<br \/>\n\u65e0\u8bba\u662f\u6bd4\u8f83\u9a8c\u8bc11\u548c\u9a8c\u8bc12\uff0c\u8fd8\u662f\u9a8c\u8bc14\u548c\u9a8c\u8bc15\uff0c\u9a8c\u8bc16\uff0cSpark\u5de5\u4f5c\u8005\u6570\u91cf\u548c\u8d44\u6e90\u6570\u91cf\u672c\u8eab\u4f3c\u4e4e\u5e76\u4e0d\u4ea7\u751f\u91cd\u5927\u7684\u6027\u80fd\u5dee\u5f02\u3002<br \/>\n\u5982\u679c\u60f3\u8981\u663e\u8457\u6539\u5584\u6027\u80fd\uff0c\u6700\u597d\u9009\u62e9\u591a\u91cd\u4e3b\u673a\u8bbe\u7f6e\u3002<\/p>\n<p>\u30102017\/05\/26\u3000\u9644\u8a00\u3011<br \/>\n\u6211\u4eec\u4f7f\u7528Kubernetes\u6784\u5efa\u4e86\u4e00\u4e2a\u96c6\u7fa4\u3002<br \/>\nhttp:\/\/qiita.com\/cvusk\/items\/42a5ffd4e3228963234d<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u901a\u8fc7 Spark \u00d7 Keras \u00d7 Docker \u5c1d\u8bd5\u5b9e\u73b0\u53ef\u6269\u5c55\u7684\u6df1\u5ea6\u5b66\u4e602\u3002 \u6211\u4ee5\u524d\u5c1d\u8bd5\u5c06Dist-ker [&hellip;]<\/p>\n","protected":false},"author":5,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-47266","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.5 (Yoast SEO v21.5) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>\u5c1d\u8bd5\u4f7f\u7528Spark \u00d7 Keras \u00d7 Docker\u5bf9\u6df1\u5ea6\u5b66\u4e60\u8fdb\u884c\u53ef\u6269\u5c55\u7684\u5c1d\u8bd52\uff1a\u591a\u8282\u70b9\u7248\u672c - Blog - Silicon Cloud<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.silicloud.com\/zh\/blog\/\u5c1d\u8bd5\u4f7f\u7528spark-x-keras-x-docker\u5bf9\u6df1\u5ea6\u5b66\u4e60\u8fdb\u884c\u53ef\u6269\u5c55\u7684\u5c1d\u8bd52\uff1a\/\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"\u5c1d\u8bd5\u4f7f\u7528Spark \u00d7 Keras \u00d7 Docker\u5bf9\u6df1\u5ea6\u5b66\u4e60\u8fdb\u884c\u53ef\u6269\u5c55\u7684\u5c1d\u8bd52\uff1a\u591a\u8282\u70b9\u7248\u672c\" \/>\n<meta property=\"og:description\" content=\"\u901a\u8fc7 Spark \u00d7 Keras \u00d7 Docker \u5c1d\u8bd5\u5b9e\u73b0\u53ef\u6269\u5c55\u7684\u6df1\u5ea6\u5b66\u4e602\u3002 \u6211\u4ee5\u524d\u5c1d\u8bd5\u5c06Dist-ker [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.silicloud.com\/zh\/blog\/\u5c1d\u8bd5\u4f7f\u7528spark-x-keras-x-docker\u5bf9\u6df1\u5ea6\u5b66\u4e60\u8fdb\u884c\u53ef\u6269\u5c55\u7684\u5c1d\u8bd52\uff1a\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog - Silicon Cloud\" \/>\n<meta property=\"article:published_time\" content=\"2023-03-19T07:31:53+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-05-03T23:34:58+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7dc3913a08637a6a0e96\/14-0.jpeg\" \/>\n<meta name=\"author\" content=\"\u6e05, \u5b87\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"\u6e05, \u5b87\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%b0%9d%e8%af%95%e4%bd%bf%e7%94%a8spark-x-keras-x-docker%e5%af%b9%e6%b7%b1%e5%ba%a6%e5%ad%a6%e4%b9%a0%e8%bf%9b%e8%a1%8c%e5%8f%af%e6%89%a9%e5%b1%95%e7%9a%84%e5%b0%9d%e8%af%952%ef%bc%9a\/\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%b0%9d%e8%af%95%e4%bd%bf%e7%94%a8spark-x-keras-x-docker%e5%af%b9%e6%b7%b1%e5%ba%a6%e5%ad%a6%e4%b9%a0%e8%bf%9b%e8%a1%8c%e5%8f%af%e6%89%a9%e5%b1%95%e7%9a%84%e5%b0%9d%e8%af%952%ef%bc%9a\/\",\"name\":\"\u5c1d\u8bd5\u4f7f\u7528Spark \u00d7 Keras \u00d7 Docker\u5bf9\u6df1\u5ea6\u5b66\u4e60\u8fdb\u884c\u53ef\u6269\u5c55\u7684\u5c1d\u8bd52\uff1a\u591a\u8282\u70b9\u7248\u672c - Blog - Silicon Cloud\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\"},\"datePublished\":\"2023-03-19T07:31:53+00:00\",\"dateModified\":\"2024-05-03T23:34:58+00:00\",\"author\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/1a6ecd3d914d22a5ac32791ffc1fbd8e\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%b0%9d%e8%af%95%e4%bd%bf%e7%94%a8spark-x-keras-x-docker%e5%af%b9%e6%b7%b1%e5%ba%a6%e5%ad%a6%e4%b9%a0%e8%bf%9b%e8%a1%8c%e5%8f%af%e6%89%a9%e5%b1%95%e7%9a%84%e5%b0%9d%e8%af%952%ef%bc%9a\/#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%b0%9d%e8%af%95%e4%bd%bf%e7%94%a8spark-x-keras-x-docker%e5%af%b9%e6%b7%b1%e5%ba%a6%e5%ad%a6%e4%b9%a0%e8%bf%9b%e8%a1%8c%e5%8f%af%e6%89%a9%e5%b1%95%e7%9a%84%e5%b0%9d%e8%af%952%ef%bc%9a\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%b0%9d%e8%af%95%e4%bd%bf%e7%94%a8spark-x-keras-x-docker%e5%af%b9%e6%b7%b1%e5%ba%a6%e5%ad%a6%e4%b9%a0%e8%bf%9b%e8%a1%8c%e5%8f%af%e6%89%a9%e5%b1%95%e7%9a%84%e5%b0%9d%e8%af%952%ef%bc%9a\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.silicloud.com\/zh\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"\u5c1d\u8bd5\u4f7f\u7528Spark \u00d7 Keras \u00d7 Docker\u5bf9\u6df1\u5ea6\u5b66\u4e60\u8fdb\u884c\u53ef\u6269\u5c55\u7684\u5c1d\u8bd52\uff1a\u591a\u8282\u70b9\u7248\u672c\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/\",\"name\":\"Blog - Silicon Cloud\",\"description\":\"\",\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/1a6ecd3d914d22a5ac32791ffc1fbd8e\",\"name\":\"\u6e05, \u5b87\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/4b2016c18459a605fc469c7566608f5686491baa112d0871ee613f61b7210565?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/4b2016c18459a605fc469c7566608f5686491baa112d0871ee613f61b7210565?s=96&d=mm&r=g\",\"caption\":\"\u6e05, \u5b87\"},\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/author\/qingyu\/\"},{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e5%b0%9d%e8%af%95%e4%bd%bf%e7%94%a8spark-x-keras-x-docker%e5%af%b9%e6%b7%b1%e5%ba%a6%e5%ad%a6%e4%b9%a0%e8%bf%9b%e8%a1%8c%e5%8f%af%e6%89%a9%e5%b1%95%e7%9a%84%e5%b0%9d%e8%af%952%ef%bc%9a\/#local-main-organization-logo\",\"url\":\"\",\"contentUrl\":\"\",\"caption\":\"Blog - Silicon Cloud\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"\u5c1d\u8bd5\u4f7f\u7528Spark \u00d7 Keras \u00d7 Docker\u5bf9\u6df1\u5ea6\u5b66\u4e60\u8fdb\u884c\u53ef\u6269\u5c55\u7684\u5c1d\u8bd52\uff1a\u591a\u8282\u70b9\u7248\u672c - Blog - Silicon Cloud","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.silicloud.com\/zh\/blog\/\u5c1d\u8bd5\u4f7f\u7528spark-x-keras-x-docker\u5bf9\u6df1\u5ea6\u5b66\u4e60\u8fdb\u884c\u53ef\u6269\u5c55\u7684\u5c1d\u8bd52\uff1a\/","og_locale":"zh_CN","og_type":"article","og_title":"\u5c1d\u8bd5\u4f7f\u7528Spark \u00d7 Keras \u00d7 Docker\u5bf9\u6df1\u5ea6\u5b66\u4e60\u8fdb\u884c\u53ef\u6269\u5c55\u7684\u5c1d\u8bd52\uff1a\u591a\u8282\u70b9\u7248\u672c","og_description":"\u901a\u8fc7 Spark \u00d7 Keras \u00d7 Docker \u5c1d\u8bd5\u5b9e\u73b0\u53ef\u6269\u5c55\u7684\u6df1\u5ea6\u5b66\u4e602\u3002 \u6211\u4ee5\u524d\u5c1d\u8bd5\u5c06Dist-ker [&hellip;]","og_url":"https:\/\/www.silicloud.com\/zh\/blog\/\u5c1d\u8bd5\u4f7f\u7528spark-x-keras-x-docker\u5bf9\u6df1\u5ea6\u5b66\u4e60\u8fdb\u884c\u53ef\u6269\u5c55\u7684\u5c1d\u8bd52\uff1a\/","og_site_name":"Blog - Silicon Cloud","article_published_time":"2023-03-19T07:31:53+00:00","article_modified_time":"2024-05-03T23:34:58+00:00","og_image":[{"url":"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d7dc3913a08637a6a0e96\/14-0.jpeg"}],"author":"\u6e05, \u5b87","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"\u6e05, \u5b87","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"3 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%b0%9d%e8%af%95%e4%bd%bf%e7%94%a8spark-x-keras-x-docker%e5%af%b9%e6%b7%b1%e5%ba%a6%e5%ad%a6%e4%b9%a0%e8%bf%9b%e8%a1%8c%e5%8f%af%e6%89%a9%e5%b1%95%e7%9a%84%e5%b0%9d%e8%af%952%ef%bc%9a\/","url":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%b0%9d%e8%af%95%e4%bd%bf%e7%94%a8spark-x-keras-x-docker%e5%af%b9%e6%b7%b1%e5%ba%a6%e5%ad%a6%e4%b9%a0%e8%bf%9b%e8%a1%8c%e5%8f%af%e6%89%a9%e5%b1%95%e7%9a%84%e5%b0%9d%e8%af%952%ef%bc%9a\/","name":"\u5c1d\u8bd5\u4f7f\u7528Spark \u00d7 Keras \u00d7 Docker\u5bf9\u6df1\u5ea6\u5b66\u4e60\u8fdb\u884c\u53ef\u6269\u5c55\u7684\u5c1d\u8bd52\uff1a\u591a\u8282\u70b9\u7248\u672c - Blog - Silicon Cloud","isPartOf":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website"},"datePublished":"2023-03-19T07:31:53+00:00","dateModified":"2024-05-03T23:34:58+00:00","author":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/1a6ecd3d914d22a5ac32791ffc1fbd8e"},"breadcrumb":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%b0%9d%e8%af%95%e4%bd%bf%e7%94%a8spark-x-keras-x-docker%e5%af%b9%e6%b7%b1%e5%ba%a6%e5%ad%a6%e4%b9%a0%e8%bf%9b%e8%a1%8c%e5%8f%af%e6%89%a9%e5%b1%95%e7%9a%84%e5%b0%9d%e8%af%952%ef%bc%9a\/#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.silicloud.com\/zh\/blog\/%e5%b0%9d%e8%af%95%e4%bd%bf%e7%94%a8spark-x-keras-x-docker%e5%af%b9%e6%b7%b1%e5%ba%a6%e5%ad%a6%e4%b9%a0%e8%bf%9b%e8%a1%8c%e5%8f%af%e6%89%a9%e5%b1%95%e7%9a%84%e5%b0%9d%e8%af%952%ef%bc%9a\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%b0%9d%e8%af%95%e4%bd%bf%e7%94%a8spark-x-keras-x-docker%e5%af%b9%e6%b7%b1%e5%ba%a6%e5%ad%a6%e4%b9%a0%e8%bf%9b%e8%a1%8c%e5%8f%af%e6%89%a9%e5%b1%95%e7%9a%84%e5%b0%9d%e8%af%952%ef%bc%9a\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.silicloud.com\/zh\/blog\/"},{"@type":"ListItem","position":2,"name":"\u5c1d\u8bd5\u4f7f\u7528Spark \u00d7 Keras \u00d7 Docker\u5bf9\u6df1\u5ea6\u5b66\u4e60\u8fdb\u884c\u53ef\u6269\u5c55\u7684\u5c1d\u8bd52\uff1a\u591a\u8282\u70b9\u7248\u672c"}]},{"@type":"WebSite","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website","url":"https:\/\/www.silicloud.com\/zh\/blog\/","name":"Blog - Silicon Cloud","description":"","inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/1a6ecd3d914d22a5ac32791ffc1fbd8e","name":"\u6e05, \u5b87","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/4b2016c18459a605fc469c7566608f5686491baa112d0871ee613f61b7210565?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4b2016c18459a605fc469c7566608f5686491baa112d0871ee613f61b7210565?s=96&d=mm&r=g","caption":"\u6e05, \u5b87"},"url":"https:\/\/www.silicloud.com\/zh\/blog\/author\/qingyu\/"},{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e5%b0%9d%e8%af%95%e4%bd%bf%e7%94%a8spark-x-keras-x-docker%e5%af%b9%e6%b7%b1%e5%ba%a6%e5%ad%a6%e4%b9%a0%e8%bf%9b%e8%a1%8c%e5%8f%af%e6%89%a9%e5%b1%95%e7%9a%84%e5%b0%9d%e8%af%952%ef%bc%9a\/#local-main-organization-logo","url":"","contentUrl":"","caption":"Blog - Silicon Cloud"}]}},"_links":{"self":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/47266","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/comments?post=47266"}],"version-history":[{"count":2,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/47266\/revisions"}],"predecessor-version":[{"id":96988,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/47266\/revisions\/96988"}],"wp:attachment":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/media?parent=47266"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/categories?post=47266"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/tags?post=47266"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}