{"id":36760,"date":"2023-01-08T19:05:49","date_gmt":"2022-10-30T13:37:39","guid":{"rendered":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer%e7%ac%94%e8%ae%b0\/"},"modified":"2024-04-29T09:07:24","modified_gmt":"2024-04-29T01:07:24","slug":"apache-airflow-cloud-composer%e7%ac%94%e8%ae%b0","status":"publish","type":"post","link":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer%e7%ac%94%e8%ae%b0\/","title":{"rendered":"Apache Airflow \/ Cloud Composer\u7b14\u8bb0"},"content":{"rendered":"<h1>DAG\u7684\u7b80\u5355\u6784\u6210<\/h1>\n<pre class=\"post-pre\"><code><\/code><\/pre>\n<h1>\u516c\u5f0f\u6587\u4ef6<\/h1>\n<p>Apache Airflow<\/p>\n<p>http:\/\/airflow.apache.org<br \/>\n\u30d0\u30fc\u30b8\u30e7\u30f3\u6bce\u306e\u4f7f\u3048\u305f\u308a\u4f7f\u3048\u306a\u304b\u3063\u305f\u308a\u304c\u591a\u3044\u306e\u3067\u4f7f\u3063\u3066\u3044\u308b\u30d0\u30fc\u30b8\u30e7\u30f3\u3067\u56fa\u5b9a\u3057\u3066\u8abf\u3079\u308b\u306e\u304c\u826f\u3055\u305d\u3046\u3002<\/p>\n<p>https:\/\/airflow.readthedocs.io\/en\/1.10.4\/<\/p>\n<p>Google Cloud Composer<\/p>\n<p>Airflow\u306e\u30de\u30cd\u30fc\u30b8\u30c9\u30b5\u30fc\u30d3\u30b9<br \/>\nhttps:\/\/cloud.google.com\/composer\/docs\/concepts\/overview<\/p>\n<h1>\u521b\u5efa\u672c\u5730\u5f00\u53d1\u73af\u5883\uff08\u4f7f\u7528Docker\uff09<\/h1>\n<p>\u62c9\u53d6\u5e76\u4f7f\u7528\u8fd9\u4e2a\u56fe\u7247\u5f88\u65b9\u4fbf\u3002<\/p>\n<ul class=\"post-ul\">https:\/\/hub.docker.com\/r\/puckel\/docker-airflow\/<\/ul>\n<h1>\u68c0\u67e5\u5de5\u4f5c\u6d41\u7a0b\u3002<\/h1>\n<p>\u53ef\u4ee5\u4f7f\u7528python dags\/hoge.py\u8fdb\u884c\u68c0\u67e5\u3002<\/p>\n<p>\u4f7f\u7528airflow CLI\u7684`list_dags`\u547d\u4ee4\u53ef\u4ee5\u770b\u5230\u4e0e\u4e0a\u8ff0\u76f8\u540c\u7684\u7ed3\u679c\uff0c\u4f46\u662f\u5b83\u4e0d\u80fd\u94fe\u63a5\u5230\u6e90DAG\uff0c\u6240\u4ee5\u4e0d\u80fd\u68c0\u67e5\u5b50DAG\u6587\u4ef6\u3002\u6240\u4ee5\uff0c\u4e5f\u8bb8\u6700\u597d\u662f\u83b7\u53d6\u6587\u4ef6\u540d\u5e76\u5206\u522b\u8fd0\u884c`python xxx.py`\uff0c\u8fd9\u6837\u53ef\u4ee5\u66f4\u5168\u9762\u5730\u8986\u76d6\u3002<\/p>\n<ul class=\"post-ul\">https:\/\/airflow.apache.org\/tutorial.html#running-the-script<\/ul>\n<h1>\u60f3\u8981\u5728Slack\u4e0a\u901a\u77e5\uff08\u4f7f\u7528SlackWebHookOperator\u3001SlackPostAPIOperator\uff09\u3002<\/h1>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">SlackWebHookOperator<\/ul>\n<\/li>\n<\/ul>\n<p>WebHook URL\u4f5c\u6210\u3001Airflow Conn \u4f5c\u308b\u5fc5\u8981\u3042\u308b\u3002<br \/>\nattachments \u66f8\u3051\u306a\u3044\uff081.10.2\u6642\u70b9\uff09<br \/>\nslack\u7528\u306e\u30d1\u30c3\u30b1\u30fc\u30b8\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u306a\u3069\u306f\u4e0d\u8981\u3002<\/p>\n<p>SlackPostAPIOperator<\/p>\n<p>Airflow Conn\u306f\u4e0d\u8981\u3002slack API token\u3092\u5f15\u6570\u306b\u308f\u305f\u3059\u5fc5\u8981\u3042\u308b\u3002<br \/>\nslackclient(&lt;2.0)\u306e\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u304c\u5fc5\u8981\u3002<br \/>\nattachements\u66f8\u3051\u308b\u3002<\/p>\n<p>\u4e0d\u7ba1\u9009\u62e9\u54ea\u79cd\u65b9\u6cd5\uff0c\u90fd\u5e0c\u671b\u80fd\u591f\u4f7f\u7528\u7c7b\u4f3cTerraform\u7684\u5de5\u5177\u8fdb\u884c\u8fde\u63a5\u6216\u6dfb\u52a0Python\u5305\u8fdb\u884c\u914d\u7f6e\uff0c\u8fd9\u8ba9\u4eba\u611f\u5230\u56f0\u60d1\u3002<\/p>\n<h1>\u6211\u60f3\u8981\u94a9\u5b50\u5e76\u901a\u77e5\u6709\u5173\u5de5\u4f5c\u7684\u6210\u529f\u4e0e\u5426\u4ee5\u53ca\u91cd\u8bd5\u72b6\u6001\u3002<\/h1>\n<p>\u5728 BaseOperator \u4e2d\u6709\u4e00\u4e2a\u540d\u4e3a on_{success|failure|retry}_callback \u7684\u53c2\u6570\uff0c\u4f60\u53ea\u9700\u8981\u5c06\u901a\u77e5\u7684\u65b9\u6cd5\u4f5c\u4e3a\u53c2\u6570\u4f20\u9012\u5373\u53ef\u3002<\/p>\n<ul class=\"post-ul\">https:\/\/airflow.apache.org\/howto\/operator\/dingding.html<\/ul>\n<h1>\u6211\u60f3\u5728JST\u65f6\u533a\u5904\u7406\u65e5\u671f\u6570\u636e\uff08\u7528\u6237\u5b9a\u4e49\u7684\u8fc7\u6ee4\u5668\uff09\u3002<\/h1>\n<p>\u53c2\u8003\u6587\u732e<\/p>\n<ul class=\"post-ul\">https:\/\/stackoverflow.com\/questions\/47112291\/how-to-trigger-daily-dag-run-at-midnight-local-time-instead-of-midnight-utc-time\/47227399#47227399<\/ul>\n<h1>\u6211\u60f3\u4ee5SubDAG\u7684\u7ed3\u679c\u548c\u4f9d\u8d56\u5173\u7cfb\u521b\u5efa\u53e6\u4e00\u4e2aDAG\uff08ExternalTaskSensor\uff09\u3002<\/h1>\n<h1>\u6211\u60f3\u8981\u6267\u884c\u4e0d\u4f9d\u8d56\u4e8eCloudComposer\u7684\u4f5c\u4e1a\uff08KubernetesPodOperator\uff09\u3002<\/h1>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">image_pull_policy \u306e\u521d\u671f\u5024\u306f IfNotPresent \u306a\u306e\u3067 Always \u306b\u3059\u308b\u3002<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">\u672a\u6307\u5b9a\u3060\u3068\u3001image\u3092\u6700\u65b0\u306b\u3057\u3066\u3082\u5143\u306eimage\u3092\u6d88\u3057\u3066\u3082\u6700\u65b0\u306b\u306a\u3089\u306a\u3044\u306e\u3067\u6ce8\u610f\u3002<\/ul>\n<p>\u5f15\u7528\u6765\u6e90<\/p>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">https:\/\/cloud.google.com\/composer\/docs\/how-to\/using\/using-kubernetes-pod-operator<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<ul class=\"post-ul\">https:\/\/kubernetes.io\/docs\/concepts\/configuration\/overview\/#container-images<\/ul>\n<h1>\u5176\u4ed6<\/h1>\n<ul class=\"post-ul\">\n<li style=\"list-style-type: none;\">\n<ul class=\"post-ul\">scheduler\u306b\u3088\u308b\u5b9f\u884c\u6642\u306e execution_date \u3068 Web UI\u306etrigger\u304b\u3089\u5b9f\u884c\u3059\u308b\u3068\u304d\u306e\u305d\u308c\u306b\u5165\u308b\u5024\u306b\u30ba\u30ec\u304c\u3042\u308b\u306e\u304c\u9055\u548c\u611f\u3002<\/ul>\n<\/li>\n<\/ul>\n<p>\u30b9\u30b1\u30b8\u30e5\u30fc\u30ea\u30f3\u30b0\u306e\u52d5\u4f5c\u306b\u3064\u3044\u3066\u308f\u304b\u308a\u3084\u304b\u3063\u305f\u8aac\u660e<br \/>\nhttps:\/\/medium.com\/programming-soda\/590ad9f9fb80<\/p>\n<p>AirflowSkipException \u306b\u3064\u3044\u3066\u3082\u3042\u3068\u3067\u66f8\u304f\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>DAG\u7684\u7b80\u5355\u6784\u6210 \u516c\u5f0f\u6587\u4ef6 Apache Airflow http:\/\/airflow.apache.org  [&hellip;]<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-36760","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.5 (Yoast SEO v21.5) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Apache Airflow \/ Cloud Composer\u7b14\u8bb0 - Blog - Silicon Cloud<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer\u7b14\u8bb0\/\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Airflow \/ Cloud Composer\u7b14\u8bb0\" \/>\n<meta property=\"og:description\" content=\"DAG\u7684\u7b80\u5355\u6784\u6210 \u516c\u5f0f\u6587\u4ef6 Apache Airflow http:\/\/airflow.apache.org [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer\u7b14\u8bb0\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog - Silicon Cloud\" \/>\n<meta property=\"article:published_time\" content=\"2022-10-30T13:37:39+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-04-29T01:07:24+00:00\" \/>\n<meta name=\"author\" content=\"\u96c5, \u609f\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"\u96c5, \u609f\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer%e7%ac%94%e8%ae%b0\/\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer%e7%ac%94%e8%ae%b0\/\",\"name\":\"Apache Airflow \/ Cloud Composer\u7b14\u8bb0 - Blog - Silicon Cloud\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\"},\"datePublished\":\"2022-10-30T13:37:39+00:00\",\"dateModified\":\"2024-04-29T01:07:24+00:00\",\"author\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/f044a4b7fa4ee2701702942002419ca6\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer%e7%ac%94%e8%ae%b0\/#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer%e7%ac%94%e8%ae%b0\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer%e7%ac%94%e8%ae%b0\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.silicloud.com\/zh\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Apache Airflow \/ Cloud Composer\u7b14\u8bb0\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/\",\"name\":\"Blog - Silicon Cloud\",\"description\":\"\",\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/f044a4b7fa4ee2701702942002419ca6\",\"name\":\"\u96c5, \u609f\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/e71a913e914f1aad1efc391f92084294bac54bc782acd289638580134cf667a6?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/e71a913e914f1aad1efc391f92084294bac54bc782acd289638580134cf667a6?s=96&d=mm&r=g\",\"caption\":\"\u96c5, \u609f\"},\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/author\/yawu\/\"},{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer%e7%ac%94%e8%ae%b0\/#local-main-organization-logo\",\"url\":\"\",\"contentUrl\":\"\",\"caption\":\"Blog - Silicon Cloud\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Apache Airflow \/ Cloud Composer\u7b14\u8bb0 - Blog - Silicon Cloud","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer\u7b14\u8bb0\/","og_locale":"zh_CN","og_type":"article","og_title":"Apache Airflow \/ Cloud Composer\u7b14\u8bb0","og_description":"DAG\u7684\u7b80\u5355\u6784\u6210 \u516c\u5f0f\u6587\u4ef6 Apache Airflow http:\/\/airflow.apache.org [&hellip;]","og_url":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer\u7b14\u8bb0\/","og_site_name":"Blog - Silicon Cloud","article_published_time":"2022-10-30T13:37:39+00:00","article_modified_time":"2024-04-29T01:07:24+00:00","author":"\u96c5, \u609f","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"\u96c5, \u609f","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"1 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer%e7%ac%94%e8%ae%b0\/","url":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer%e7%ac%94%e8%ae%b0\/","name":"Apache Airflow \/ Cloud Composer\u7b14\u8bb0 - Blog - Silicon Cloud","isPartOf":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website"},"datePublished":"2022-10-30T13:37:39+00:00","dateModified":"2024-04-29T01:07:24+00:00","author":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/f044a4b7fa4ee2701702942002419ca6"},"breadcrumb":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer%e7%ac%94%e8%ae%b0\/#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer%e7%ac%94%e8%ae%b0\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer%e7%ac%94%e8%ae%b0\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.silicloud.com\/zh\/blog\/"},{"@type":"ListItem","position":2,"name":"Apache Airflow \/ Cloud Composer\u7b14\u8bb0"}]},{"@type":"WebSite","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website","url":"https:\/\/www.silicloud.com\/zh\/blog\/","name":"Blog - Silicon Cloud","description":"","inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/f044a4b7fa4ee2701702942002419ca6","name":"\u96c5, \u609f","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/e71a913e914f1aad1efc391f92084294bac54bc782acd289638580134cf667a6?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e71a913e914f1aad1efc391f92084294bac54bc782acd289638580134cf667a6?s=96&d=mm&r=g","caption":"\u96c5, \u609f"},"url":"https:\/\/www.silicloud.com\/zh\/blog\/author\/yawu\/"},{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-cloud-composer%e7%ac%94%e8%ae%b0\/#local-main-organization-logo","url":"","contentUrl":"","caption":"Blog - Silicon Cloud"}]}},"_links":{"self":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/36760","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/comments?post=36760"}],"version-history":[{"count":2,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/36760\/revisions"}],"predecessor-version":[{"id":84342,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/36760\/revisions\/84342"}],"wp:attachment":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/media?parent=36760"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/categories?post=36760"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/tags?post=36760"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}