{"id":36295,"date":"2023-04-20T08:18:58","date_gmt":"2023-10-14T15:27:46","guid":{"rendered":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-%e5%ae%9e%e8%b7%b5%e6%95%99%e7%a8%8b\/"},"modified":"2024-04-30T00:21:05","modified_gmt":"2024-04-29T16:21:05","slug":"apache-airflow-%e5%ae%9e%e8%b7%b5%e6%95%99%e7%a8%8b","status":"publish","type":"post","link":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-%e5%ae%9e%e8%b7%b5%e6%95%99%e7%a8%8b\/","title":{"rendered":"Apache Airflow \u5b9e\u8df5\u6559\u7a0b"},"content":{"rendered":"<h1>\u9996\u5148<\/h1>\n<p>\u4e3a\u4e86\u6784\u5efa\u673a\u5668\u5b66\u4e60\u6d41\u6c34\u7ebf\uff0c\u6211\u5b66\u4e60\u4e86Apache Airflow\u3002\u6211\u8bd5\u7740\u901a\u8fc7\u5b98\u65b9\u6559\u7a0b\u7b49\u8fdb\u884c\u5b9e\u9645\u64cd\u4f5c\uff0c\u4f46\u7406\u89e3\u8fdb\u5c55\u4e0d\u5927\uff0c\u6240\u4ee5\u6211\u8d2d\u4e70\u4e86\u300aThe Complete Hands-On Introduction to Apache Airflow\u300b\u3002\u867d\u7136\u662f\u82f1\u8bed\u7684\uff0c\u4f46\u5b83\u6709\u5b57\u5e55\uff0c\u662f\u975e\u5e38\u597d\u7684\u6559\u6750\uff0c\u6211\u63a8\u8350\u5b83\u3002\u5728\u672c\u6587\u4e2d\uff0c\u6211\u5c06\u6839\u636e\u8fd9\u95e8\u8bfe\u7a0b\u4ecb\u7ecd\u6d41\u6c34\u7ebf\u6784\u5efa\u7684\u4e00\u7cfb\u5217\u6b65\u9aa4\u3002<\/p>\n<h1>\u73af\u5883\u642d\u5efa<\/h1>\n<p>\u6211\u4f7f\u7528 venv \u6784\u5efa\u4e86\u865a\u62df\u73af\u5883\u3002Apache Airflow \u7684\u7248\u672c\u548c\u4f9d\u8d56\u5173\u7cfb\u90fd\u6839\u636e\u8fd9\u91cc\u7684\u8981\u6c42\u8fdb\u884c\u4e86\u8c03\u6574\uff08\u622a\u81f32022\u5e741\u6708\u4e3a\u6b62\uff0c\u7248\u672c\u4e3a2.1.0\uff09\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"nv\">$ <\/span>python3 <span class=\"nt\">-m<\/span> venv sandbox\r\n<span class=\"nv\">$ <\/span><span class=\"nb\">source <\/span>sandbox\/bin\/activate\r\n<span class=\"o\">(<\/span>sandbox<span class=\"o\">)<\/span> <span class=\"nv\">$ <\/span>sandbox\/bin\/python3 <span class=\"nt\">-m<\/span> pip <span class=\"nb\">install<\/span> <span class=\"nt\">--upgrade<\/span> pip\r\n<span class=\"o\">(<\/span>sandbox<span class=\"o\">)<\/span> <span class=\"nv\">$ <\/span>pip <span class=\"nb\">install <\/span>apache-airflow<span class=\"o\">==<\/span>2.1.0 <span class=\"nt\">--constraint<\/span> https:\/\/gist.githubusercontent.com\/marclamberti\/742efaef5b2d94f44666b0aec020be7c\/raw\/21c88601337250b6fd93f1adceb55282fb07b7ed\/constraint.txt\r\n<span class=\"o\">(<\/span>sandbox<span class=\"o\">)<\/span> <span class=\"nv\">$ <\/span>airflow db init\r\n<span class=\"o\">(<\/span>sandbox<span class=\"o\">)<\/span> <span class=\"nv\">$ <\/span>airflow <span class=\"nb\">users <\/span>create <span class=\"nt\">--username<\/span> admin <span class=\"nt\">--password<\/span> admin <span class=\"nt\">--firstname<\/span> Taro\r\n<span class=\"nt\">--lastname<\/span> Yamada <span class=\"nt\">--role<\/span> Admin <span class=\"nt\">--email<\/span> admin@airflow.com\r\n<\/code><\/pre>\n<p>\u8fd0\u884c airflow webserver \u548c airflow scheduler\uff0c\u8bbf\u95ee http:\/\/localhost:8080\uff0c\u6253\u5f00\u4ee5\u4e0b\u753b\u9762\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d286537434c4406c3ff52\/6-0.png\" alt=\"airflow_home.png\" \/><\/div>\n<p>\u7a0d\u540e\uff0c\u7531\u4e8e\u754c\u9762\u4e0a\u8fd8\u6709\u4e00\u4e9b\u64cd\u4f5c\u9700\u8981\u8fdb\u884c\uff0c\u6240\u4ee5\u8bf7\u4fdd\u6301\u6253\u5f00\u72b6\u6001\u3002<\/p>\n<h1>\u7ba1\u7ebf\u6982\u89c8<\/h1>\n<p>\u7531\u4e8e\u5df2\u521b\u5efa\u4e86\u540d\u4e3a~\/airflow\u7684\u76ee\u5f55\uff0c\u56e0\u6b64\u5728\u5176\u4e2d\u521b\u5efa\u4e00\u4e2a\u540d\u4e3adags\u7684\u76ee\u5f55\u3002DAG\u662fDirected Acyclic Graph\u7684\u7f29\u5199\uff0c\u7ffb\u8bd1\u6210\u4e2d\u6587\u4e3a\u6709\u5411\u975e\u5faa\u73af\u56fe\u3002\u8fd9\u662f\u5728\u56fe\u5f62\u7406\u8bba\u4e2d\u51fa\u73b0\u7684\u672f\u8bed\uff0c\u7b80\u800c\u8a00\u4e4b\uff0c\u8868\u793a\u53ea\u80fd\u5355\u5411\u524d\u8fdb\u4e14\u6ca1\u6709\u5faa\u73af\u7684\u7ed3\u6784\u3002\u901a\u8fc7dag\u76ee\u5f55\u6765\u6784\u5efa\u7ba1\u9053\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"o\">(<\/span>airflow-test<span class=\"o\">)<\/span> <span class=\"nv\">$ <\/span><span class=\"nb\">cd<\/span> ~\/airflow\r\n<span class=\"o\">(<\/span>airflow-test<span class=\"o\">)<\/span> <span class=\"nv\">$ <\/span><span class=\"nb\">mkdir <\/span>dags\r\n<\/code><\/pre>\n<p>\u5728\u672c\u6587\u4e2d\uff0c\u6211\u4eec\u6700\u7ec8\u5c06\u5efa\u7acb\u5982\u4e0b\u6240\u793a\u7684\u6d41\u6c34\u7ebf\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d286537434c4406c3ff52\/12-0.png\" alt=\"user_processing.png\" \/><\/div>\n<p>\u4ee5\u4e0b\u662f\u6bcf\u4e2a\u4efb\u52a1\u7684\u7b80\u8981\u8bf4\u660e\u3002<\/p>\n<ol>\n<li style=\"list-style-type: none;\">\n<ol>\u521b\u5efa\u8868\u683c\uff1a\u521b\u5efa\u4e00\u4e2a\u7528\u4e8e\u4fdd\u5b58\u6570\u636e\u7684\u8868\u683c\u3002<\/ol>\n<\/li>\n<\/ol>\n<p>API\u662f\u5426\u53ef\u7528\uff1a\u7b49\u5f85Random User Generator\u7f51\u7ad9\u7684API\u53d8\u4e3a\u53ef\u7528\u72b6\u6001\u3002<\/p>\n<p>\u63d0\u53d6\u7528\u6237\uff1a\u8c03\u7528API\u83b7\u53d6\u7528\u6237\u4fe1\u606f\u3002<\/p>\n<p>\u5904\u7406\u7528\u6237\uff1a\u4ece\u83b7\u53d6\u7684\u7528\u6237\u4fe1\u606f\u4e2d\u63d0\u53d6\u6240\u9700\u7684\u6570\u636e\u5e76\u4fdd\u5b58\u5230CSV\u6587\u4ef6\u4e2d\u3002<\/p>\n<p>\u5b58\u50a8\u7528\u6237\uff1a\u5c06CSV\u6587\u4ef6\u4e2d\u7684\u6570\u636e\u5b58\u50a8\u5230\u8868\u683c\u4e2d\u3002<\/p>\n<h1>\u5efa\u7acb\u6d41\u6c34\u7ebf<\/h1>\n<p>\u5728 airflow\/dags\/user_processing.py \u6587\u4ef6\u4e2d\u521b\u5efa\u4efb\u52a1\u3002<\/p>\n<h2>0. \u521b\u5efaDAG\u5bf9\u8c61\u7684\u5b9e\u4f8b<\/h2>\n<p>\u5728\u6309\u987a\u5e8f\u521b\u5efa\u4efb\u52a1\u4e4b\u524d\uff0c\u9996\u5148\u8981\u521b\u5efa\u5305\u542b\u4efb\u52a1\u7684DAG\u5b9e\u4f8b\u3002\u5728\u4ee5\u4e0b\u7684\u201cwith\u201d\u8bed\u6cd5\u4e2d\u5bf9\u6bcf\u4e2a\u4efb\u52a1\u8fdb\u884c\u5b9a\u4e49\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"kn\">from<\/span> <span class=\"n\">datetime<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">datetime<\/span>\r\n\r\n<span class=\"kn\">from<\/span> <span class=\"n\">airflow.models<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">DAG<\/span>\r\n\r\n<span class=\"n\">default_args<\/span> <span class=\"o\">=<\/span> <span class=\"p\">{<\/span>\r\n    <span class=\"sh\">\"<\/span><span class=\"s\">start_date<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"nf\">datetime<\/span><span class=\"p\">(<\/span><span class=\"mi\">2020<\/span><span class=\"p\">,<\/span> <span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">1<\/span><span class=\"p\">),<\/span>\r\n<span class=\"p\">}<\/span>\r\n\r\n\r\n<span class=\"k\">with<\/span> <span class=\"nc\">DAG<\/span><span class=\"p\">(<\/span>\r\n    <span class=\"sh\">\"<\/span><span class=\"s\">user_processing<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span>\r\n    <span class=\"n\">schedule_interval<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">@daily<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span>\r\n    <span class=\"n\">default_args<\/span><span class=\"o\">=<\/span><span class=\"n\">default_args<\/span><span class=\"p\">,<\/span>\r\n    <span class=\"n\">catchup<\/span><span class=\"o\">=<\/span><span class=\"bp\">False<\/span><span class=\"p\">,<\/span>\r\n<span class=\"p\">)<\/span> <span class=\"k\">as<\/span> <span class=\"n\">dag<\/span><span class=\"p\">:<\/span>\r\n\r\n    <span class=\"c1\"># define tasks\/operators\r\n<\/span><\/code><\/pre>\n<h2>\u521b\u5efa\u8868\u683c<\/h2>\n<p>\u521b\u5efa\u4e00\u4e2a\u7528\u4e8e\u4fdd\u5b58\u6570\u636e\u7684\u8868\u683c\u3002\u7531\u4e8e\u4f7f\u7528\u4e86SQLite\uff0c\u56e0\u6b64\u9700\u8981\u4f7f\u7528SqliteOperator\uff0c\u53ef\u4ee5\u901a\u8fc7\u4ee5\u4e0b\u547d\u4ee4\u8fdb\u884c\u5b89\u88c5\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"o\">(<\/span>sandbox<span class=\"o\">)<\/span> <span class=\"nv\">$ <\/span>pip <span class=\"nb\">install<\/span> <span class=\"s1\">'apache-airflow-providers-sqlite'<\/span>\r\n<\/code><\/pre>\n<p>Airflow\u6709\u8bb8\u591a\u63d0\u4f9b\u5546\uff0c\u6839\u636e\u9700\u8981\u9700\u8981\u5b89\u88c5\u3002\u6709\u5173\u8be6\u7ec6\u4fe1\u606f\uff0c\u8bf7\u53c2\u8003\u63d0\u4f9b\u5546\u8f6f\u4ef6\u5305\u53c2\u8003\u8d44\u6599\u3002<\/p>\n<p>\u5728\u4e0b\u9762\uff0c\u5b9a\u4e49\u4e00\u4e2a\u4efb\u52a1\u3002\u5982\u679c\u8868\u4e0d\u5b58\u5728\uff0c\u5219\u521b\u5efa\u4e00\u4e2a\u5177\u67096\u4e2a\u5217\u7684\u8868\u683c\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"bp\">...<\/span>\r\n<span class=\"kn\">from<\/span> <span class=\"n\">airflow.providers.sqlite.operators.sqlite<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">SqliteOperator<\/span>\r\n\r\n<span class=\"bp\">...<\/span>\r\n\r\n<span class=\"k\">with<\/span> <span class=\"nc\">DAG<\/span><span class=\"p\">(...)<\/span> <span class=\"k\">as<\/span> <span class=\"n\">dag<\/span><span class=\"p\">:<\/span>\r\n\r\n    <span class=\"n\">creating_table<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">SqliteOperator<\/span><span class=\"p\">(<\/span>\r\n        <span class=\"n\">task_id<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">creating_table<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span>\r\n        <span class=\"n\">sqlite_conn_id<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">db_sqlite<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span>\r\n        <span class=\"n\">sql<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"\"\"<\/span><span class=\"s\">\r\n            CREATE TABLE IF NOT EXISTS users (\r\n                firstname TEXT NOT NULL,\r\n                lastname TEXT NOT NULL,\r\n                country TEXT NOT NULL,\r\n                username TEXT NOT NULL,\r\n                password TEXT NOT NULL,\r\n                email TEXT NOT NULL PRIMARY KEY\r\n            );\r\n            <\/span><span class=\"sh\">\"\"\"<\/span><span class=\"p\">,<\/span>\r\n    <span class=\"p\">)<\/span>\r\n<\/code><\/pre>\n<p>\u9700\u8981\u4e0e\u6570\u636e\u5e93\u5efa\u7acb\u8fde\u63a5\u624d\u80fd\u8fde\u63a5\u5230\u8868\u683c\uff0c\u56e0\u6b64\u9700\u8981\u4ece\u4e4b\u524d\u6253\u5f00\u7684\u753b\u9762\u4e2d\u521b\u5efa\u3002\u9009\u62e9\u9876\u90e8\u7684 [Admin] -&gt; [Connections]\uff0c\u7136\u540e\u70b9\u51fb [+] \u6309\u94ae\u8fdb\u884c\u65b0\u5efa\u3002\u914d\u7f6e\u8bf7\u53c2\u8003\u4ee5\u4e0b\u56fe\u7247\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d286537434c4406c3ff52\/27-0.png\" alt=\"sqlite_conn.png\" \/><\/div>\n<p>\u5c06Conn Id\u8bbe\u7f6e\u4e3asqlite_conn_id\u7684\u540c\u540d\u3002\u5c06Host\u6307\u5b9a\u4e3aairflow.db\u7684\u8def\u5f84\u3002<\/p>\n<p>\u5982\u679c\u53ef\u4ee5\u505a\u5230\u8fd9\u4e00\u70b9\uff0c\u5c31\u8fdb\u884c\u4efb\u52a1\u6d4b\u8bd5\uff0c\u4ee5\u786e\u4fdd\u4efb\u52a1\u80fd\u591f\u8fd0\u884c\u6b63\u5e38\u3002\u4ee5\u4e0b\u662f\u6d4b\u8bd5\u547d\u4ee4\u3002\u5982\u679c\u6ca1\u6709\u51fa\u73b0\u9519\u8bef\uff0c\u5219\u8bf4\u660e\u5df2\u6b63\u786e\u521b\u5efa\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"o\">(<\/span>sandbox<span class=\"o\">)<\/span> <span class=\"nv\">$ <\/span>airflow tasks <span class=\"nb\">test <\/span>user_processing creating_table 2020-01-01\r\n<\/code><\/pre>\n<h1>2. API\u662f\u5426\u53ef\u7528<\/h1>\n<p>\u4e3a\u4e86\u521b\u5efa\u4e00\u4e2a\u7b49\u5f85 API \u53ef\u7528\u7684\u5904\u7406\u8fc7\u7a0b\uff0c\u4f7f\u7528 Sensor\u3002\u4ee5\u4e0b\u662f Sensor \u7684\u5b98\u65b9\u6587\u6863\u4e2d\u7684\u63cf\u8ff0\u3002<\/p>\n<blockquote><p>\u4f20\u611f\u5668\u662f\u4e00\u79cd\u7279\u5b9a\u7c7b\u578b\u7684\u8fd0\u7b97\u7b26\uff0c\u4f1a\u4e00\u76f4\u8fd0\u884c\uff0c\u76f4\u5230\u6ee1\u8db3\u7279\u5b9a\u7684\u6761\u4ef6\u3002\u4f8b\u5982\uff0c\u4e00\u4e2a\u7279\u5b9a\u7684\u6587\u4ef6\u964d\u843d\u5728HDFS\u6216S3\u4e2d\uff0c\u4e00\u4e2a\u5206\u533a\u51fa\u73b0\u5728Hive\u4e2d\uff0c\u6216\u8005\u662f\u67d0\u4e2a\u7279\u5b9a\u7684\u65f6\u95f4\u3002\u4f20\u611f\u5668\u662f\u4eceBaseSensorOperator\u6d3e\u751f\u800c\u6765\u7684\uff0c\u4f1a\u5728\u6307\u5b9a\u7684poke_interval\u95f4\u9694\u5185\u8fd0\u884cpoke\u65b9\u6cd5\uff0c\u76f4\u5230\u5b83\u8fd4\u56deTrue\u4e3a\u6b62\u3002<\/p><\/blockquote>\n<p>\u57fa\u672c\u4e0a\uff0c\u5b83\u662f\u7528\u4e8e\u7b49\u5f85\u67d0\u4e8b\u53d1\u751f\u5e76\u4fdd\u8bc1\u5176\u884c\u52a8\u7684\u5de5\u5177\uff0c\u5c31\u50cf\u8fd9\u6b21\u4e00\u6837\u3002\u7531\u4e8e\u4f7f\u7528HTTP\uff0c\u56e0\u6b64\u8bf7\u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u6765\u5b89\u88c5\u63d0\u4f9b\u7a0b\u5e8f\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"o\">(<\/span>sandbox<span class=\"o\">)<\/span> <span class=\"nv\">$ <\/span>pip <span class=\"nb\">install <\/span>apache-airflow-providers-http<span class=\"o\">==<\/span>2.0.0\r\n<\/code><\/pre>\n<p>\u4f7f\u7528 HttpSensor \u7684\u4ee3\u7801\u5982\u4e0b\u6240\u793a\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"bp\">...<\/span>\r\n<span class=\"kn\">from<\/span> <span class=\"n\">airflow.providers.http.sensors.http<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">HttpSensor<\/span>\r\n\r\n<span class=\"bp\">...<\/span>\r\n\r\n<span class=\"k\">with<\/span> <span class=\"nc\">DAG<\/span><span class=\"p\">(...)<\/span> <span class=\"k\">as<\/span> <span class=\"n\">dag<\/span><span class=\"p\">:<\/span>\r\n\r\n    <span class=\"bp\">...<\/span>\r\n\r\n    <span class=\"n\">is_api_available<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">HttpSensor<\/span><span class=\"p\">(<\/span>\r\n        <span class=\"n\">task_id<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">is_api_available<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">http_conn_id<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">user_api<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">endpoint<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">api\/<\/span><span class=\"sh\">\"<\/span>\r\n    <span class=\"p\">)<\/span>\r\n<\/code><\/pre>\n<p>\u5efa\u7acb\u4e0eAPI\u7684\u8fde\u63a5\u3002\u4e0eSQLite\u76f8\u540c\uff0c\u6309\u7167\u4ee5\u4e0b\u56fe\u7247\u6240\u793a\u8fdb\u884c\u8bbe\u7f6e\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d286537434c4406c3ff52\/39-0.png\" alt=\"api_conn.png\" \/><\/div>\n<p>\u5728\u8fd9\u91cc\u5bf9\u521a\u521a\u521b\u5efa\u7684\u4efb\u52a1\u8fdb\u884c\u6d4b\u8bd5\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"o\">(<\/span>sandbox<span class=\"o\">)<\/span> <span class=\"nv\">$ <\/span>airflow tasks <span class=\"nb\">test <\/span>user_processing is_api_available 2020-01-01\r\n<\/code><\/pre>\n<h1>3. \u63d0\u53d6\u7528\u6237<\/h1>\n<p>\u5b9a\u4e49\u4e00\u4e2a\u4eceAPI\u83b7\u53d6\u6570\u636e\u7684\u4efb\u52a1\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"bp\">...<\/span>\r\n<span class=\"kn\">import<\/span> <span class=\"n\">json<\/span>\r\n<span class=\"kn\">from<\/span> <span class=\"n\">airflow.providers.http.operators.http<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">SimpleHttpOperator<\/span>\r\n\r\n<span class=\"bp\">...<\/span>\r\n\r\n<span class=\"k\">with<\/span> <span class=\"nc\">DAG<\/span><span class=\"p\">(...)<\/span> <span class=\"k\">as<\/span> <span class=\"n\">dag<\/span><span class=\"p\">:<\/span>\r\n\r\n    <span class=\"bp\">...<\/span>\r\n\r\n    <span class=\"n\">extracting_user<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">SimpleHttpOperator<\/span><span class=\"p\">(<\/span>\r\n        <span class=\"n\">task_id<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">extracting_user<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span>\r\n        <span class=\"n\">http_conn_id<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">user_api<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span>\r\n        <span class=\"n\">endpoint<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">api\/<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span>\r\n        <span class=\"n\">method<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">GET<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span>\r\n        <span class=\"n\">response_filter<\/span><span class=\"o\">=<\/span><span class=\"k\">lambda<\/span> <span class=\"n\">response<\/span><span class=\"p\">:<\/span> <span class=\"n\">json<\/span><span class=\"p\">.<\/span><span class=\"nf\">loads<\/span><span class=\"p\">(<\/span><span class=\"n\">response<\/span><span class=\"p\">.<\/span><span class=\"n\">text<\/span><span class=\"p\">),<\/span>\r\n        <span class=\"n\">log_response<\/span><span class=\"o\">=<\/span><span class=\"bp\">True<\/span><span class=\"p\">,<\/span>\r\n    <span class=\"p\">)<\/span>\r\n<\/code><\/pre>\n<p>\u53ea\u9700\u8981\u4e00\u4e2a\u9009\u9879\uff1a\u4e0e\u4e4b\u524d\u4e00\u6837\u8fdb\u884c\u6d4b\u8bd5\u5c06\u83b7\u5f97\u4ee5\u4e0b\u7ed3\u679c\u3002\u53ef\u4ee5\u786e\u8ba4\u83b7\u5f97\u4e0eRandom User Generator API\u76f8\u4f3c\u7684\u7ed3\u679c\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"o\">(<\/span>sandbox<span class=\"o\">)<\/span> <span class=\"nv\">$ <\/span>airflow tasks <span class=\"nb\">test <\/span>user_processing extracting_user 2020-01-01\r\n...\r\n<span class=\"o\">[<\/span>2022-01-21 06:56:21,045] <span class=\"o\">{<\/span>http.py:115<span class=\"o\">}<\/span> INFO - <span class=\"o\">{<\/span><span class=\"s2\">\"results\"<\/span>:[<span class=\"o\">{<\/span><span class=\"s2\">\"gender\"<\/span>:<span class=\"s2\">\"male\"<\/span>,<span class=\"s2\">\"name\"<\/span>:<span class=\"o\">{<\/span><span class=\"s2\">\"title\"<\/span>:<span class=\"s2\">\"Mr\"<\/span>,<span class=\"s2\">\"first\"<\/span>:<span class=\"s2\">\"Darren\"<\/span>,<span class=\"s2\">\"last\"<\/span>:<span class=\"s2\">\"King\"<\/span><span class=\"o\">}<\/span>,<span class=\"s2\">\"location\"<\/span>:<span class=\"o\">{<\/span><span class=\"s2\">\"street\"<\/span>:<span class=\"o\">{<\/span><span class=\"s2\">\"number\"<\/span>:6614,<span class=\"s2\">\"name\"<\/span>:<span class=\"s2\">\"Bridge Road\"<\/span><span class=\"o\">}<\/span>,<span class=\"s2\">\"city\"<\/span>:<span class=\"s2\">\"Exeter\"<\/span>,<span class=\"s2\">\"state\"<\/span>:<span class=\"s2\">\"Clwyd\"<\/span>,<span class=\"s2\">\"country\"<\/span>:<span class=\"s2\">\"United Kingdom\"<\/span>,<span class=\"s2\">\"postcode\"<\/span>:<span class=\"s2\">\"K86 3ZN\"<\/span>,<span class=\"s2\">\"coordinates\"<\/span>:<span class=\"o\">{<\/span><span class=\"s2\">\"latitude\"<\/span>:<span class=\"s2\">\"6.5705\"<\/span>,<span class=\"s2\">\"longitude\"<\/span>:<span class=\"s2\">\"-158.8391\"<\/span><span class=\"o\">}<\/span>,<span class=\"s2\">\"timezone\"<\/span>:<span class=\"o\">{<\/span><span class=\"s2\">\"offset\"<\/span>:<span class=\"s2\">\"-3:30\"<\/span>,<span class=\"s2\">\"description\"<\/span>:<span class=\"s2\">\"Newfoundland\"<\/span><span class=\"o\">}}<\/span>,<span class=\"s2\">\"email\"<\/span>:<span class=\"s2\">\"darren.king@example.com\"<\/span>,<span class=\"s2\">\"login\"<\/span>:<span class=\"o\">{<\/span><span class=\"s2\">\"uuid\"<\/span>:<span class=\"s2\">\"384a152f-66a9-472d-a592-e1f8557e564f\"<\/span>,<span class=\"s2\">\"username\"<\/span>:<span class=\"s2\">\"purplecat905\"<\/span>,<span class=\"s2\">\"password\"<\/span>:<span class=\"s2\">\"flyers\"<\/span>,<span class=\"s2\">\"salt\"<\/span>:<span class=\"s2\">\"yyPLDi3z\"<\/span>,<span class=\"s2\">\"md5\"<\/span>:<span class=\"s2\">\"495955a7b7b4fc0664ac0fc580f2cca4\"<\/span>,<span class=\"s2\">\"sha1\"<\/span>:<span class=\"s2\">\"f85a89f3cfb86675bc42b5c0d787e247bbb0ad5e\"<\/span>,<span class=\"s2\">\"sha256\"<\/span>:<span class=\"s2\">\"b0824af8b0fcc5de0a996b3f6c41e7ea6a916a749b0aeeb6c648264f7f3c4447\"<\/span><span class=\"o\">}<\/span>,<span class=\"s2\">\"dob\"<\/span>:<span class=\"o\">{<\/span><span class=\"s2\">\"date\"<\/span>:<span class=\"s2\">\"1951-03-25T05:24:14.327Z\"<\/span>,<span class=\"s2\">\"age\"<\/span>:71<span class=\"o\">}<\/span>,<span class=\"s2\">\"registered\"<\/span>:<span class=\"o\">{<\/span><span class=\"s2\">\"date\"<\/span>:<span class=\"s2\">\"2019-01-22T19:37:00.577Z\"<\/span>,<span class=\"s2\">\"age\"<\/span>:3<span class=\"o\">}<\/span>,<span class=\"s2\">\"phone\"<\/span>:<span class=\"s2\">\"017683 00410\"<\/span>,<span class=\"s2\">\"cell\"<\/span>:<span class=\"s2\">\"0752-503-650\"<\/span>,<span class=\"s2\">\"id\"<\/span>:<span class=\"o\">{<\/span><span class=\"s2\">\"name\"<\/span>:<span class=\"s2\">\"NINO\"<\/span>,<span class=\"s2\">\"value\"<\/span>:<span class=\"s2\">\"PJ 90 72 17 S\"<\/span><span class=\"o\">}<\/span>,<span class=\"s2\">\"picture\"<\/span>:<span class=\"o\">{<\/span><span class=\"s2\">\"large\"<\/span>:<span class=\"s2\">\"https:\/\/randomuser.me\/api\/portraits\/men\/56.jpg\"<\/span>,<span class=\"s2\">\"medium\"<\/span>:<span class=\"s2\">\"https:\/\/randomuser.me\/api\/portraits\/med\/men\/56.jpg\"<\/span>,<span class=\"s2\">\"thumbnail\"<\/span>:<span class=\"s2\">\"https:\/\/randomuser.me\/api\/portraits\/thumb\/men\/56.jpg\"<\/span><span class=\"o\">}<\/span>,<span class=\"s2\">\"nat\"<\/span>:<span class=\"s2\">\"GB\"<\/span><span class=\"o\">}]<\/span>,<span class=\"s2\">\"info\"<\/span>:<span class=\"o\">{<\/span><span class=\"s2\">\"seed\"<\/span>:<span class=\"s2\">\"53304cf938cd3a25\"<\/span>,<span class=\"s2\">\"results\"<\/span>:1,<span class=\"s2\">\"page\"<\/span>:1,<span class=\"s2\">\"version\"<\/span>:<span class=\"s2\">\"1.3\"<\/span><span class=\"o\">}}<\/span>\r\n...\r\n<\/code><\/pre>\n<h1>4. \u5904\u7406\u7528\u6237<\/h1>\n<p>\u7531\u4e8e\u83b7\u5f97\u4e86\u4e0a\u8ff0\u6570\u636e\uff0c\u6211\u4eec\u5e0c\u671b\u80fd\u591f\u9002\u5f53\u5730\u5904\u7406\u5e76\u63d0\u53d6\u4ec5\u9700\u8981\u6ce8\u518c\u5230\u8868\u4e2d\u7684\u6570\u636e\u3002\u4f7f\u7528PythonOperator\u5728Python\u811a\u672c\u4e2d\u6267\u884c\u5904\u7406\uff0c\u56e0\u6b64\u9700\u8981\u901a\u8fc7python_callable\u5b9a\u4e49\u6267\u884c\u51fd\u6570\u3002<\/p>\n<p>\u5728\u672c\u6b21\u60c5\u51b5\u4e0b\uff0c\u9700\u8981\u901a\u8fc7 extracting_user \u83b7\u53d6\u7684\u6570\u636e\uff0c\u6240\u4ee5\u8981\u4f7f\u7528 Xcoms\u3002Xcoms \u662f Cross Communications \u7684\u7f29\u5199\uff0c\u5b83\u662f\u4e00\u79cd\u5141\u8bb8\u4efb\u52a1\u4e4b\u95f4\u8fdb\u884c\u76f8\u4e92\u901a\u4fe1\u7684\u673a\u5236\u3002\u8be6\u7ec6\u4fe1\u606f\u8bf7\u53c2\u8003\u5b98\u65b9\u6587\u6863\u3002\u7531\u4e8e\u4efb\u52a1\u7684\u8fd4\u56de\u503c\u9ed8\u8ba4\u662f\u88ab\u63a8\u9001\u7684\uff0c\u56e0\u6b64\u53ef\u4ee5\u901a\u8fc7 xcom_pull() \u63a5\u6536\u6570\u636e\u3002\uff08*\u8bf7\u6ce8\u610f\uff0c\u7981\u6b62\u4f20\u9012\u50cf\u6570\u636e\u6846\u8fd9\u6837\u5927\u7684\u6570\u636e\u3002\uff09<\/p>\n<pre class=\"post-pre\"><code><span class=\"bp\">...<\/span>\r\n<span class=\"kn\">from<\/span> <span class=\"n\">airflow.operators.python<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">PythonOperator<\/span>\r\n<span class=\"kn\">from<\/span> <span class=\"n\">pandas<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">json_normalize<\/span>\r\n\r\n<span class=\"bp\">...<\/span>\r\n\r\n\r\n<span class=\"k\">def<\/span> <span class=\"nf\">_processing_user<\/span><span class=\"p\">(<\/span><span class=\"n\">ti<\/span><span class=\"p\">):<\/span>\r\n    <span class=\"n\">users<\/span> <span class=\"o\">=<\/span> <span class=\"n\">ti<\/span><span class=\"p\">.<\/span><span class=\"nf\">xcom_pull<\/span><span class=\"p\">(<\/span><span class=\"n\">task_ids<\/span><span class=\"o\">=<\/span><span class=\"p\">[<\/span><span class=\"sh\">\"<\/span><span class=\"s\">extracting_user<\/span><span class=\"sh\">\"<\/span><span class=\"p\">])<\/span>\r\n    <span class=\"k\">if<\/span> <span class=\"ow\">not<\/span> <span class=\"nf\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">users<\/span><span class=\"p\">)<\/span> <span class=\"ow\">or<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">results<\/span><span class=\"sh\">\"<\/span> <span class=\"ow\">not<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">users<\/span><span class=\"p\">[<\/span><span class=\"mi\">0<\/span><span class=\"p\">]:<\/span>\r\n        <span class=\"k\">raise<\/span> <span class=\"nc\">ValueError<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">User is Empy<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\r\n    <span class=\"n\">user<\/span> <span class=\"o\">=<\/span> <span class=\"n\">users<\/span><span class=\"p\">[<\/span><span class=\"mi\">0<\/span><span class=\"p\">][<\/span><span class=\"sh\">\"<\/span><span class=\"s\">results<\/span><span class=\"sh\">\"<\/span><span class=\"p\">][<\/span><span class=\"mi\">0<\/span><span class=\"p\">]<\/span>\r\n    <span class=\"n\">processed_user<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">json_normalize<\/span><span class=\"p\">(<\/span>\r\n        <span class=\"p\">{<\/span>\r\n            <span class=\"sh\">\"<\/span><span class=\"s\">firstname<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"n\">user<\/span><span class=\"p\">[<\/span><span class=\"sh\">\"<\/span><span class=\"s\">name<\/span><span class=\"sh\">\"<\/span><span class=\"p\">][<\/span><span class=\"sh\">\"<\/span><span class=\"s\">first<\/span><span class=\"sh\">\"<\/span><span class=\"p\">],<\/span>\r\n            <span class=\"sh\">\"<\/span><span class=\"s\">lastname<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"n\">user<\/span><span class=\"p\">[<\/span><span class=\"sh\">\"<\/span><span class=\"s\">name<\/span><span class=\"sh\">\"<\/span><span class=\"p\">][<\/span><span class=\"sh\">\"<\/span><span class=\"s\">last<\/span><span class=\"sh\">\"<\/span><span class=\"p\">],<\/span>\r\n            <span class=\"sh\">\"<\/span><span class=\"s\">country<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"n\">user<\/span><span class=\"p\">[<\/span><span class=\"sh\">\"<\/span><span class=\"s\">location<\/span><span class=\"sh\">\"<\/span><span class=\"p\">][<\/span><span class=\"sh\">\"<\/span><span class=\"s\">country<\/span><span class=\"sh\">\"<\/span><span class=\"p\">],<\/span>\r\n            <span class=\"sh\">\"<\/span><span class=\"s\">username<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"n\">user<\/span><span class=\"p\">[<\/span><span class=\"sh\">\"<\/span><span class=\"s\">login<\/span><span class=\"sh\">\"<\/span><span class=\"p\">][<\/span><span class=\"sh\">\"<\/span><span class=\"s\">username<\/span><span class=\"sh\">\"<\/span><span class=\"p\">],<\/span>\r\n            <span class=\"sh\">\"<\/span><span class=\"s\">password<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"n\">user<\/span><span class=\"p\">[<\/span><span class=\"sh\">\"<\/span><span class=\"s\">login<\/span><span class=\"sh\">\"<\/span><span class=\"p\">][<\/span><span class=\"sh\">\"<\/span><span class=\"s\">password<\/span><span class=\"sh\">\"<\/span><span class=\"p\">],<\/span>\r\n            <span class=\"sh\">\"<\/span><span class=\"s\">email<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"n\">user<\/span><span class=\"p\">[<\/span><span class=\"sh\">\"<\/span><span class=\"s\">email<\/span><span class=\"sh\">\"<\/span><span class=\"p\">],<\/span>\r\n        <span class=\"p\">}<\/span>\r\n    <span class=\"p\">)<\/span>\r\n    <span class=\"n\">processed_user<\/span><span class=\"p\">.<\/span><span class=\"nf\">to_csv<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">\/tmp\/processed_user.csv<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">index<\/span><span class=\"o\">=<\/span><span class=\"bp\">None<\/span><span class=\"p\">,<\/span> <span class=\"n\">header<\/span><span class=\"o\">=<\/span><span class=\"bp\">False<\/span><span class=\"p\">)<\/span>\r\n\r\n<span class=\"k\">with<\/span> <span class=\"nc\">DAG<\/span><span class=\"p\">(...)<\/span> <span class=\"k\">as<\/span> <span class=\"n\">dag<\/span><span class=\"p\">:<\/span>\r\n\r\n    <span class=\"bp\">...<\/span>\r\n\r\n    <span class=\"n\">processing_user<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">PythonOperator<\/span><span class=\"p\">(<\/span>\r\n        <span class=\"n\">task_id<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">processing_user<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">python_callable<\/span><span class=\"o\">=<\/span><span class=\"n\">_processing_user<\/span>\r\n    <span class=\"p\">)<\/span>\r\n<\/code><\/pre>\n<p>\u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u8fdb\u884c\u6d4b\u8bd5\uff0c\u786e\u8ba4\u4ece cat tmp\/processed_user.csv \u63d0\u53d6\u7684\u6570\u636e\u5df2\u5b58\u50a8\u5230\u6587\u4ef6\u4e2d\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"o\">(<\/span>sandbox<span class=\"o\">)<\/span> <span class=\"nv\">$ <\/span>airflow tasks <span class=\"nb\">test <\/span>user_processing processing_user 2020-01-01\r\n...\r\n<span class=\"o\">(<\/span>sandbox<span class=\"o\">)<\/span> <span class=\"nv\">$ <\/span><span class=\"nb\">cat<\/span> \/tmp\/processed_user.csv\r\nDarren,King,United Kingdom,purplecat905,flyers,darren.king@example.com\r\n<\/code><\/pre>\n<h1>5. \u5b58\u50a8\u7528\u6237<\/h1>\n<p>\u6700\u540e\u7528 BashOperator \u5c06 CSV \u6570\u636e\u4fdd\u5b58\u5230\u6570\u636e\u5e93\u4e2d\u3002\u7701\u7565\u4e86\u8be6\u7ec6\u8bf4\u660e\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"bp\">...<\/span>\r\n<span class=\"kn\">from<\/span> <span class=\"n\">airflow.operators.bash<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">BashOperator<\/span>\r\n\r\n<span class=\"bp\">...<\/span>\r\n\r\n\r\n<span class=\"k\">with<\/span> <span class=\"nc\">DAG<\/span><span class=\"p\">(...)<\/span> <span class=\"k\">as<\/span> <span class=\"n\">dag<\/span><span class=\"p\">:<\/span>\r\n\r\n    <span class=\"bp\">...<\/span>\r\n\r\n    <span class=\"n\">storing_user<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">BashOperator<\/span><span class=\"p\">(<\/span>\r\n        <span class=\"n\">ask_id<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">storing_user<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span>\r\n        <span class=\"n\">bash_command<\/span><span class=\"o\">=<\/span><span class=\"sh\">'<\/span><span class=\"s\">echo -e <\/span><span class=\"sh\">\"<\/span><span class=\"s\">.separator <\/span><span class=\"sh\">\"<\/span><span class=\"s\">,<\/span><span class=\"sh\">\"<\/span><span class=\"se\">\\n<\/span><span class=\"s\">.import  \/tmp\/processed_user.csv users<\/span><span class=\"sh\">\"<\/span><span class=\"s\"> | sqlite3 \/home\/airflow\/airflow\/airflow.db<\/span><span class=\"sh\">'<\/span><span class=\"p\">,<\/span>\r\n    <span class=\"p\">)<\/span>\r\n<\/code><\/pre>\n<p>\u53ef\u4ee5\u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u6765\u786e\u8ba4\u6570\u636e\u662f\u5426\u5b58\u50a8\u5728\u6570\u636e\u5e93\u4e2d\u3002<\/p>\n<pre class=\"post-pre\"><code><span class=\"o\">(<\/span>sandbox<span class=\"o\">)<\/span> <span class=\"nv\">$ <\/span>airflow tasks <span class=\"nb\">test <\/span>user_processing storing_user 2020-01-01\r\n...\r\n<span class=\"o\">(<\/span>sandbox<span class=\"o\">)<\/span> <span class=\"nv\">$ <\/span>splite3 airflow.db\r\nsqlite&gt; <span class=\"k\">select<\/span> <span class=\"k\">*<\/span> from <span class=\"nb\">users<\/span><span class=\"p\">;<\/span>\r\nAna|Blanchard|France|sadbutterfly288|bounty|ana.blanchard@example.com\r\n<\/code><\/pre>\n<h2>6. \u4f9d\u8d56\u5173\u7cfb<\/h2>\n<p>\u5b8c\u6210\u4ee5\u4e0a\u6b65\u9aa4\u540e\uff0c\u8fd4\u56de\u6d4f\u89c8\u5668\uff0c\u4ece[DAG]\u5217\u8868\u4e2d\u9009\u62e9&#8217;user_processing&#8217;\uff0c\u7136\u540e\u70b9\u51fb[Graph View]\u3002\u8fd9\u6837\uff0c\u4f1a\u5448\u73b0\u5982\u4e0b\u56fe\u6240\u793a\u7684\u72b6\u6001\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d286537434c4406c3ff52\/60-0.png\" alt=\"only_tasks.png\" \/><\/div>\n<p>\u5728\u8fd9\u4e2a\u9636\u6bb5\uff0c\u4efb\u52a1\u53ea\u662f\u96f6\u6563\u5730\u521b\u5efa\uff0c\u5e76\u4e14\u9700\u8981\u5b9a\u4e49\u5b83\u4eec\u7684\u6267\u884c\u987a\u5e8f\u3002\u5982\u679c\u9700\u8981\u7684\u8bdd\uff0c\u53ef\u4ee5\u8fd9\u6837\u63cf\u8ff0\uff1a<\/p>\n<pre class=\"post-pre\"><code><span class=\"bp\">...<\/span>\r\n\r\n\r\n<span class=\"k\">with<\/span> <span class=\"nc\">DAG<\/span><span class=\"p\">(...)<\/span> <span class=\"k\">as<\/span> <span class=\"n\">dag<\/span><span class=\"p\">:<\/span>\r\n\r\n    <span class=\"bp\">...<\/span>\r\n\r\n    <span class=\"p\">(<\/span>\r\n        <span class=\"n\">creating_table<\/span>\r\n        <span class=\"o\">&gt;&gt;<\/span> <span class=\"n\">is_api_available<\/span>\r\n        <span class=\"o\">&gt;&gt;<\/span> <span class=\"n\">extracting_user<\/span>\r\n        <span class=\"o\">&gt;&gt;<\/span> <span class=\"n\">processing_user<\/span>\r\n        <span class=\"o\">&gt;&gt;<\/span> <span class=\"n\">storing_user<\/span>\r\n    <span class=\"p\">)<\/span>\r\n<\/code><\/pre>\n<p>\u8fd9\u4e2a\u987a\u5e8f\u88ab\u79f0\u4e3a\u4f9d\u8d56\u5173\u7cfb\uff0c\u53ef\u4ee5\u901a\u8fc7\u4f7f\u7528&gt;&gt;\u6765\u5b9a\u4e49\u3002\u5237\u65b0\u9875\u9762\u540e\uff0c\u53ef\u4ee5\u786e\u8ba4\u5df2\u521b\u5efa\u4e86\u50cf\u9644\u4ef6\u4e00\u6837\u7684\u6d41\u6c34\u7ebf\u3002\u6309\u4e0b\u5de6\u4e0a\u89d2\u7684\u5207\u6362\u6309\u94ae\uff0c\u53ef\u4ee5\u6267\u884c\u6574\u4e2a\u6d41\u6c34\u7ebf\u3002<\/p>\n<div><img decoding=\"async\" class=\"post-images\" title=\"\" src=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d286537434c4406c3ff52\/64-0.png\" alt=\"user_processing.png\" \/><\/div>\n<h1>\u6700\u540e<\/h1>\n<p>\u4f7f\u7528Apache Airflow\u8fdb\u884c\u7b80\u5355\u7684\u7ba1\u9053\u6784\u5efa\u3002\u6211\u4eec\u60f3\u8981\u8003\u8651\u4f7f\u7528\u5404\u79cd\u63d0\u4f9b\u5546\u4ee5\u53ca\u5904\u7406\u7528\u4e8e\u673a\u5668\u5b66\u4e60\u7684\u5927\u6570\u636e\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u9996\u5148 \u4e3a\u4e86\u6784\u5efa\u673a\u5668\u5b66\u4e60\u6d41\u6c34\u7ebf\uff0c\u6211\u5b66\u4e60\u4e86Apache Airflow\u3002\u6211\u8bd5\u7740\u901a\u8fc7\u5b98\u65b9\u6559\u7a0b\u7b49\u8fdb\u884c\u5b9e\u9645\u64cd\u4f5c\uff0c\u4f46\u7406\u89e3\u8fdb [&hellip;]<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-36295","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.5 (Yoast SEO v21.5) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Apache Airflow \u5b9e\u8df5\u6559\u7a0b - Blog - Silicon Cloud<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-\u5b9e\u8df5\u6559\u7a0b\/\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Airflow \u5b9e\u8df5\u6559\u7a0b\" \/>\n<meta property=\"og:description\" content=\"\u9996\u5148 \u4e3a\u4e86\u6784\u5efa\u673a\u5668\u5b66\u4e60\u6d41\u6c34\u7ebf\uff0c\u6211\u5b66\u4e60\u4e86Apache Airflow\u3002\u6211\u8bd5\u7740\u901a\u8fc7\u5b98\u65b9\u6559\u7a0b\u7b49\u8fdb\u884c\u5b9e\u9645\u64cd\u4f5c\uff0c\u4f46\u7406\u89e3\u8fdb [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-\u5b9e\u8df5\u6559\u7a0b\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog - Silicon Cloud\" \/>\n<meta property=\"article:published_time\" content=\"2023-10-14T15:27:46+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-04-29T16:21:05+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d286537434c4406c3ff52\/6-0.png\" \/>\n<meta name=\"author\" content=\"\u96c5, \u609f\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"\u96c5, \u609f\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-%e5%ae%9e%e8%b7%b5%e6%95%99%e7%a8%8b\/\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-%e5%ae%9e%e8%b7%b5%e6%95%99%e7%a8%8b\/\",\"name\":\"Apache Airflow \u5b9e\u8df5\u6559\u7a0b - Blog - Silicon Cloud\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\"},\"datePublished\":\"2023-10-14T15:27:46+00:00\",\"dateModified\":\"2024-04-29T16:21:05+00:00\",\"author\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/f044a4b7fa4ee2701702942002419ca6\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-%e5%ae%9e%e8%b7%b5%e6%95%99%e7%a8%8b\/#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-%e5%ae%9e%e8%b7%b5%e6%95%99%e7%a8%8b\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-%e5%ae%9e%e8%b7%b5%e6%95%99%e7%a8%8b\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.silicloud.com\/zh\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Apache Airflow \u5b9e\u8df5\u6559\u7a0b\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/\",\"name\":\"Blog - Silicon Cloud\",\"description\":\"\",\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/f044a4b7fa4ee2701702942002419ca6\",\"name\":\"\u96c5, \u609f\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/e71a913e914f1aad1efc391f92084294bac54bc782acd289638580134cf667a6?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/e71a913e914f1aad1efc391f92084294bac54bc782acd289638580134cf667a6?s=96&d=mm&r=g\",\"caption\":\"\u96c5, \u609f\"},\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/author\/yawu\/\"},{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-%e5%ae%9e%e8%b7%b5%e6%95%99%e7%a8%8b\/#local-main-organization-logo\",\"url\":\"\",\"contentUrl\":\"\",\"caption\":\"Blog - Silicon Cloud\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Apache Airflow \u5b9e\u8df5\u6559\u7a0b - Blog - Silicon Cloud","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-\u5b9e\u8df5\u6559\u7a0b\/","og_locale":"zh_CN","og_type":"article","og_title":"Apache Airflow \u5b9e\u8df5\u6559\u7a0b","og_description":"\u9996\u5148 \u4e3a\u4e86\u6784\u5efa\u673a\u5668\u5b66\u4e60\u6d41\u6c34\u7ebf\uff0c\u6211\u5b66\u4e60\u4e86Apache Airflow\u3002\u6211\u8bd5\u7740\u901a\u8fc7\u5b98\u65b9\u6559\u7a0b\u7b49\u8fdb\u884c\u5b9e\u9645\u64cd\u4f5c\uff0c\u4f46\u7406\u89e3\u8fdb [&hellip;]","og_url":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-\u5b9e\u8df5\u6559\u7a0b\/","og_site_name":"Blog - Silicon Cloud","article_published_time":"2023-10-14T15:27:46+00:00","article_modified_time":"2024-04-29T16:21:05+00:00","og_image":[{"url":"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d286537434c4406c3ff52\/6-0.png"}],"author":"\u96c5, \u609f","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"\u96c5, \u609f","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"4 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-%e5%ae%9e%e8%b7%b5%e6%95%99%e7%a8%8b\/","url":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-%e5%ae%9e%e8%b7%b5%e6%95%99%e7%a8%8b\/","name":"Apache Airflow \u5b9e\u8df5\u6559\u7a0b - Blog - Silicon Cloud","isPartOf":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website"},"datePublished":"2023-10-14T15:27:46+00:00","dateModified":"2024-04-29T16:21:05+00:00","author":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/f044a4b7fa4ee2701702942002419ca6"},"breadcrumb":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-%e5%ae%9e%e8%b7%b5%e6%95%99%e7%a8%8b\/#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-%e5%ae%9e%e8%b7%b5%e6%95%99%e7%a8%8b\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-%e5%ae%9e%e8%b7%b5%e6%95%99%e7%a8%8b\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.silicloud.com\/zh\/blog\/"},{"@type":"ListItem","position":2,"name":"Apache Airflow \u5b9e\u8df5\u6559\u7a0b"}]},{"@type":"WebSite","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website","url":"https:\/\/www.silicloud.com\/zh\/blog\/","name":"Blog - Silicon Cloud","description":"","inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/f044a4b7fa4ee2701702942002419ca6","name":"\u96c5, \u609f","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/e71a913e914f1aad1efc391f92084294bac54bc782acd289638580134cf667a6?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e71a913e914f1aad1efc391f92084294bac54bc782acd289638580134cf667a6?s=96&d=mm&r=g","caption":"\u96c5, \u609f"},"url":"https:\/\/www.silicloud.com\/zh\/blog\/author\/yawu\/"},{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/apache-airflow-%e5%ae%9e%e8%b7%b5%e6%95%99%e7%a8%8b\/#local-main-organization-logo","url":"","contentUrl":"","caption":"Blog - Silicon Cloud"}]}},"_links":{"self":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/36295","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/comments?post=36295"}],"version-history":[{"count":2,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/36295\/revisions"}],"predecessor-version":[{"id":88537,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/36295\/revisions\/88537"}],"wp:attachment":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/media?parent=36295"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/categories?post=36295"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/tags?post=36295"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}