{"id":46441,"date":"2023-01-31T23:27:27","date_gmt":"2023-05-27T04:23:15","guid":{"rendered":"https:\/\/www.silicloud.com\/zh\/blog\/46441-2\/"},"modified":"2024-04-29T12:28:21","modified_gmt":"2024-04-29T04:28:21","slug":"46441-2","status":"publish","type":"post","link":"https:\/\/www.silicloud.com\/zh\/blog\/46441-2\/","title":{"rendered":""},"content":{"rendered":"<p>Linux\u30b5\u30fc\u30d0\u30fcJupyter Notebook\u3067Spark\u3092\u52d5\u304b\u305b\u308b\u30ac\u30a4\u30c9<\/p>\n<h1>Spark Download<\/h1>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">Spark Download Link<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\u89e3\u51cd<\/ul>\n<pre class=\"post-pre\"><code><span class=\"nb\">tar<\/span> <span class=\"nt\">-xzvf<\/span> spark-3.0.1-bin-hadoop3.2.tgz\r\n<\/code><\/pre>\n<h1>Java Installation<\/h1>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">JDK Download<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">\u4eca\u56de\u306fUbuntu[Windows Subsystem for Linux]\u3092\u4f7f\u3046\u305f\u3081\u3001Linux\u30d0\u30fc\u30b8\u30e7\u30f3\u306eJDK[jdk-8u261-linux-x64.tar.gz]\u3092\u30c0\u30a6\u30f3\u30ed\u30fc\u30c9\u3059\u308b<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">JDK8 Download Link<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\u89e3\u51cd<\/ul>\n<pre class=\"post-pre\"><code><span class=\"nb\">tar<\/span> <span class=\"nt\">-xzvf<\/span> jdk-8u261-linux-x64.tar.gz\r\n<\/code><\/pre>\n<ul class=\"post-ul\">\u79fb\u52d5<\/ul>\n<pre class=\"post-pre\"><code><span class=\"nb\">mv  <\/span>jdk1.8.0_261  \/usr\/env\/java8\r\n<\/code><\/pre>\n<ul class=\"post-ul\">JAVA_HOME\u8a2d\u5b9a<\/ul>\n<pre class=\"post-pre\"><code>vim \/etc\/profile\r\n<\/code><\/pre>\n<p>\u6700\u5f8c\u306b\u4e0b\u8a18\u306e\u30b3\u30f3\u30c6\u30f3\u30c4\u3092\u8ffd\u52a0<\/p>\n<pre class=\"post-pre\"><code><span class=\"nb\">export <\/span><span class=\"nv\">JAVA_HOME<\/span><span class=\"o\">=<\/span>\/usr\/env\/java8\r\n<span class=\"nb\">export <\/span><span class=\"nv\">PATH<\/span><span class=\"o\">=<\/span><span class=\"nv\">$JAVA_HOME<\/span>\/bin:<span class=\"nv\">$PATH<\/span>\r\n<span class=\"nb\">export <\/span><span class=\"nv\">CLASSPATH<\/span><span class=\"o\">=<\/span>.:<span class=\"nv\">$JAVA_HOME<\/span>\/lib\/dt.jar:<span class=\"nv\">$JAVA_HOME<\/span>\/lib\/tools.jar\r\n<\/code><\/pre>\n<ul class=\"post-ul\">\u5909\u52d5\u3092\u6709\u52b9<\/ul>\n<pre class=\"post-pre\"><code><span class=\"nb\">source<\/span>  \/etc\/profile\r\n<\/code><\/pre>\n<ul class=\"post-ul\">\u78ba\u8a8d<\/ul>\n<pre class=\"post-pre\"><code>java <span class=\"nt\">--version<\/span>\r\n<\/code><\/pre>\n<h1>Spark Configuration<\/h1>\n<p>\u3053\u306e\u6642Spark\u306ebin\/pyspark\u3092\u5b9f\u884c\u3059\u308b\u3068JAVA_HOME is not set\u3068\u3044\u3046\u30a8\u30e9\u30fc\u306f\u3067\u307e\u3059\u3001\u539f\u56e0\u306f\u4e0a\u8a18JAVA_HOME\u3092\u8a2d\u5b9a\u3057\u305f\u3051\u3069Spark\u306f\u8a8d\u8b58\u3067\u304d\u307e\u305b\u3093\u3002<br \/>\nSpark\u306e\u30c9\u30ad\u30e5\u30e1\u30f3\u30c8[environment-variables]\u306b\u3088\u308b\u3068conf\/spark-env.sh\u3092\u3061\u3083\u3093\u3068\u8a2d\u5b9a\u3059\u308c\u3070\u826f\u3044\u3068\u3002<\/p>\n<ul class=\"post-ul\">Template\u304b\u3089\u8a2d\u5b9a\u30d5\u30a1\u30a4\u30eb\u4f5c\u6210<\/ul>\n<pre class=\"post-pre\"><code><span class=\"nb\">cp <\/span>conf\/spark-env.sh.template conf\/spark-env.sh\r\n<\/code><\/pre>\n<ul class=\"post-ul\">\u4e0b\u8a18\u306e\u8a2d\u5b9a\u3092\u8ffd\u52a0<\/ul>\n<pre class=\"post-pre\"><code><span class=\"nv\">JAVA_HOME<\/span><span class=\"o\">=<\/span>\/usr\/env\/java8\/\r\n<span class=\"nv\">PYSPARK_PYTHON<\/span><span class=\"o\">=<\/span>\/usr\/bin\/python3\r\n<span class=\"nv\">SPARK_LOCAL_IP<\/span><span class=\"o\">=<\/span>&lt;YOUR_LOCAL_IP_ADDRESS&gt;\r\n<\/code><\/pre>\n<p>\u4eca\u306fbin\/pyspark\u306f\u52d5\u3051\u308b\u3088\u3046\u306b\u306a\u308a\u307e\u3057\u305f\u3002???<\/p>\n<h1>Jupyter Configuration<\/h1>\n<ul class=\"post-ul\">Jupyter Notebook \u3092\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb<\/ul>\n<pre class=\"post-pre\"><code>apt <span class=\"nb\">install <\/span>jupyter-notebook\r\n<\/code><\/pre>\n<p>conf\/spark-env.sh\u306b\u4e0b\u8a18\u306e\u8a2d\u5b9a\u3092\u8ffd\u52a0<\/p>\n<pre class=\"post-pre\"><code>\r\n<span class=\"nv\">PYSPARK_DRIVER_PYTHON<\/span><span class=\"o\">=<\/span>jupyter-notebook\r\n<span class=\"nv\">PYSPARK_DRIVER_PYTHON_OPTS<\/span><span class=\"o\">=<\/span><span class=\"s2\">\" --ip=0.0.0.0 --port=8888\"<\/span>\r\n<\/code><\/pre>\n<p>\u4eca\u306fbin\/pyspark\u3092\u5b9f\u884c\u3057\u3066Browser\u3067http:\/\/127.0.0.1:8888\/?token=\u3092\u958b\u304f\u3068Jupyter Notebook\u304c\u958b\u3051\u307e\u3059\u3002<br \/>\nspark, sc\u306a\u3069\u306e\u5f15\u6570\u3082\u5229\u7528\u3067\u304d\u308b\u72b6\u614b\u3067\u3059\u3002<\/p>\n<p>\u203bbin\/pyspark\u3092\u5b9f\u884c\u3057\u305f\u3089Jupyter notebook URL\u304c\u8868\u793a\u3055\u308c\u306a\u3044\u5834\u5408\u3001Ctrl+C\u3092\u62bc\u3059\u3068\u8868\u793a\u3057\u307e\u3059<\/p>\n<h6>\u304a\u307e\u3051<\/h6>\n<p>Jupyter Notebook\u30d1\u30b9\u30ef\u30fc\u30c9\u8a2d\u5b9a<br \/>\n\u203b\u8a2d\u5b9a\u3059\u308b\u3068\u30d1\u30b9\u30ef\u30fc\u30c9\u3067\u30ed\u30b0\u30a4\u30f3\u3067\u304d\u308b<\/p>\n<pre class=\"post-pre\"><code>jupyter notebook password\r\n<\/code><\/pre>\n<h6>\u8ffd\u8a18<\/h6>\n<h6>20201223<\/h6>\n<p>df.show()\u3092\u4f7f\u3046\u6642\u306b\u4e0b\u8a18\u306eException\u304c\u51fa\u307e\u3059<\/p>\n<pre class=\"post-pre\"><code>Exception: Python\u00a0in\u00a0worker\u00a0has\u00a0different\u00a0version\u00a03.8\u00a0than\u00a0that\u00a0in\u00a0driver\u00a03.6,\u00a0PySpark\u00a0cannot\u00a0run\u00a0with\u00a0different\u00a0minor\u00a0versions.\u00a0Please\u00a0check\u00a0environment\u00a0variables\u00a0PYSPARK_PYTHON\u00a0and\u00a0PYSPARK_DRIVER_PYTHON\u00a0are\u00a0correctly\u00a0set.\r\n<\/code><\/pre>\n<p>\u89e3\u6c7a\u3059\u308b\u65b9\u6cd5\u306fPYSPARK_PYTHON\u306e\u30d0\u30fc\u30b8\u30e7\u30f3\u3068PYSPARK_DRIVER_PYTHON\u306e\u30d0\u30fc\u30b8\u30e7\u30f3\u3092\u4e00\u81f4\u306b\u3057\u307e\u3059<\/p>\n<p>conf\/spark-env.sh\u306b\u4e0b\u8a18\u306e\u8a2d\u5b9a\u3092\u4fee\u6b63\u3057\u307e\u3059<\/p>\n<pre class=\"post-pre\"><code><span class=\"nv\">PYSPARK_PYTHON<\/span><span class=\"o\">=<\/span>python3.6\r\n<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Linux\u30b5\u30fc\u30d0\u30fcJupyter Notebook\u3067Spark\u3092\u52d5\u304b\u305b\u308b\u30ac\u30a4\u30c9 Spark Download  [&hellip;]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-46441","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.5 (Yoast SEO v21.5) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>- Blog - Silicon Cloud<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.silicloud.com\/zh\/blog\/46441-2\/\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:description\" content=\"Linux\u30b5\u30fc\u30d0\u30fcJupyter Notebook\u3067Spark\u3092\u52d5\u304b\u305b\u308b\u30ac\u30a4\u30c9 Spark Download [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.silicloud.com\/zh\/blog\/46441-2\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog - Silicon Cloud\" \/>\n<meta property=\"article:published_time\" content=\"2023-05-27T04:23:15+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-04-29T04:28:21+00:00\" \/>\n<meta name=\"author\" content=\"\u79d1, \u96c5\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"\u79d1, \u96c5\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/46441-2\/\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/46441-2\/\",\"name\":\"- Blog - Silicon Cloud\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\"},\"datePublished\":\"2023-05-27T04:23:15+00:00\",\"dateModified\":\"2024-04-29T04:28:21+00:00\",\"author\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/41e222757cdd2a3365361328bd79970a\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.silicloud.com\/zh\/blog\/46441-2\/\"]}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/\",\"name\":\"Blog - Silicon Cloud\",\"description\":\"\",\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/41e222757cdd2a3365361328bd79970a\",\"name\":\"\u79d1, \u96c5\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/1b2d3e00a7df03689797ebd4af8c5827ba5af936849a71050ec331f4cf902c5d?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/1b2d3e00a7df03689797ebd4af8c5827ba5af936849a71050ec331f4cf902c5d?s=96&d=mm&r=g\",\"caption\":\"\u79d1, \u96c5\"},\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/author\/keya\/\"},{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/46441-2\/#local-main-organization-logo\",\"url\":\"\",\"contentUrl\":\"\",\"caption\":\"Blog - Silicon Cloud\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"- Blog - Silicon Cloud","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.silicloud.com\/zh\/blog\/46441-2\/","og_locale":"zh_CN","og_type":"article","og_description":"Linux\u30b5\u30fc\u30d0\u30fcJupyter Notebook\u3067Spark\u3092\u52d5\u304b\u305b\u308b\u30ac\u30a4\u30c9 Spark Download [&hellip;]","og_url":"https:\/\/www.silicloud.com\/zh\/blog\/46441-2\/","og_site_name":"Blog - Silicon Cloud","article_published_time":"2023-05-27T04:23:15+00:00","article_modified_time":"2024-04-29T04:28:21+00:00","author":"\u79d1, \u96c5","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"\u79d1, \u96c5","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"1 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.silicloud.com\/zh\/blog\/46441-2\/","url":"https:\/\/www.silicloud.com\/zh\/blog\/46441-2\/","name":"- Blog - Silicon Cloud","isPartOf":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website"},"datePublished":"2023-05-27T04:23:15+00:00","dateModified":"2024-04-29T04:28:21+00:00","author":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/41e222757cdd2a3365361328bd79970a"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.silicloud.com\/zh\/blog\/46441-2\/"]}]},{"@type":"WebSite","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website","url":"https:\/\/www.silicloud.com\/zh\/blog\/","name":"Blog - Silicon Cloud","description":"","inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/41e222757cdd2a3365361328bd79970a","name":"\u79d1, \u96c5","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/1b2d3e00a7df03689797ebd4af8c5827ba5af936849a71050ec331f4cf902c5d?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1b2d3e00a7df03689797ebd4af8c5827ba5af936849a71050ec331f4cf902c5d?s=96&d=mm&r=g","caption":"\u79d1, \u96c5"},"url":"https:\/\/www.silicloud.com\/zh\/blog\/author\/keya\/"},{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/46441-2\/#local-main-organization-logo","url":"","contentUrl":"","caption":"Blog - Silicon Cloud"}]}},"_links":{"self":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/46441","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/comments?post=46441"}],"version-history":[{"count":2,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/46441\/revisions"}],"predecessor-version":[{"id":85277,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/46441\/revisions\/85277"}],"wp:attachment":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/media?parent=46441"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/categories?post=46441"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/tags?post=46441"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}