{"id":48849,"date":"2023-03-30T23:40:59","date_gmt":"2023-11-06T15:33:36","guid":{"rendered":"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bd%bf%e7%94%a8terraform-kubespray%e5%9c%a8kvm%e4%b8%8a%e6%9e%84%e5%bb%bakubernetes%e9%9b%86%e7%be%a4%e3%80%82\/"},"modified":"2024-04-29T21:14:31","modified_gmt":"2024-04-29T13:14:31","slug":"%e4%bd%bf%e7%94%a8terraform-kubespray%e5%9c%a8kvm%e4%b8%8a%e6%9e%84%e5%bb%bakubernetes%e9%9b%86%e7%be%a4%e3%80%82","status":"publish","type":"post","link":"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bd%bf%e7%94%a8terraform-kubespray%e5%9c%a8kvm%e4%b8%8a%e6%9e%84%e5%bb%bakubernetes%e9%9b%86%e7%be%a4%e3%80%82\/","title":{"rendered":"\u4f7f\u7528Terraform + Kubespray\u5728KVM\u4e0a\u6784\u5efaKubernetes\u96c6\u7fa4"},"content":{"rendered":"

\u592a\u957f\u4e0d\u8bfb b\u00f9 d\u00fa)<\/h2>\n

\u5728\u5b9a\u4e49\u4e86\u7ec4\u6210\u4e4b\u540e\uff0c\u53ef\u4ee5\u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u521b\u5efa Kubernetes \u96c6\u7fa4\uff1a<\/p>\n

$ terraform init\r\n$ terraform apply -auto-approve\r\n$ docker run --rm -it \\\r\n  --mount type=bind,source=\"$(pwd)\"\/inventory,dst=\/inventory \\\r\n  --mount type=bind,source=\"$(pwd)\"\/generate_inventory.py,dst=\/kubespray\/generate_inventory.py \\\r\n  --mount type=bind,source=\"$(pwd)\"\/terraform.tfstate,dst=\/kubespray\/terraform.tfstate \\\r\n  --mount type=bind,source=\"${HOME}\"\/.ssh\/id_rsa,dst=\/root\/.ssh\/id_rsa \\\r\n  quay.io\/kubespray\/kubespray:v2.23.1 bash\r\n\r\n# Inside a container\r\n$ ansible-playbook -i generate_inventory.py cluster.yml\r\n<\/code><\/pre>\n
\u6837\u672c\u4ee3\u7801\uff1a<\/p>\n
 <\/p>\n
\u7b80\u8981\u6982\u8ff0<\/h2>\n
\u4f7f\u7528Terraform + Kubespray\u5728KVM\u4e0a\u6784\u5efaKubernetes\u96c6\u7fa4\u3002\u5927\u81f4\u6b65\u9aa4\u5982\u4e0b\u3002<\/p>\n
    \n
  1. \n
      \u4f7f\u7528 Terraform \u521b\u5efa\u865a\u62df\u673a<\/ol>\n<\/li>\n<\/ol>\n
       <\/p>\n
        \n
      1. \n
          \u4ece\u751f\u6210\u7684 terraform.tfstate \u6587\u4ef6\u4e2d\u63d0\u53d6\u5b58\u8d27\u4fe1\u606f<\/ol>\n<\/li>\n<\/ol>\n
           <\/p>\n
            \u4f7f\u7528\u5b58\u8d27\u4fe1\u606f\u548c Kuberspray \u7684 Ansible playbooks \u521b\u5efa Kubernetes \u96c6\u7fa4<\/ol>\n
            \"dynamic_inventory.drawio.png\"<\/div>\n
            \u7f51\u7edc\u914d\u7f6e\u5982\u4e0b\u56fe\u6240\u793a\uff1a<\/p>\n
            \"network_architecture.drawio.png\"<\/div>\n
            \u5728\u96c6\u7fa4\u4e2d\uff0c\u5c06\u516c\u5171\u7f51\u7edc\uff08eth0\uff09\u4e0e\u7528\u4e8eAnsible SSH\u767b\u5f55\u7684\u7f51\u7edc\uff08eth1\uff09\u8fdb\u884c\u4e86\u5206\u9694\u3002<\/p>\n
            \u524d\u63d0\u6761\u4ef6 (Qian ti tiao jian)<\/h2>\n
              \n
            • \n
                terraform<\/ul>\n<\/li>\n<\/ul>\n
                 <\/p>\n
                  \n
                • \n
                    Container runtime (docker, podman, nerdctl, etc.)<\/ul>\n<\/li>\n<\/ul>\n
                     <\/p>\n
                      \n
                    • \n
                        KVM packages<\/ul>\n<\/li>\n<\/ul>\n
                        qemu-kvm
                        \nlibvirt<\/p>\n
                        crdtools
                        \nnmcli
                        \n(Host OS: Ubuntu 22.04)<\/p>\n
                        \u6b65\u9aa4<\/h2>\n
                        \u4e3b\u6301\u4eba\u7684\u9884\u5148\u8bbe\u7f6e<\/h3>\n
                        \u7f51\u7edc\u8bbe\u7f6e<\/h4>\n
                        \u521b\u5efa\u4e00\u4e2a\u865a\u62df\u6865\u63a5\u53e3br0\uff0c\u5e76\u5c06\u73b0\u6709\u4e3b\u673a\u7684\u7f51\u7edc\u63a5\u53e3\uff08\u5982enp1s0\uff09\u8fde\u63a5\u5230br0\u3002\u901a\u8fc7\u8fd9\u6837\u505a\uff0c\u53ef\u4ee5\u5c06\u540e\u7eed\u521b\u5efa\u7684\u865a\u62df\u673a\u8fde\u63a5\u5230br0\uff0c\u4f7f\u5f97\u53ef\u4ee5\u76f4\u63a5\u4ece\u5c40\u57df\u7f51\u4e2d\u7684\u4efb\u4f55\u4e3b\u673a\u8bbf\u95ee\u865a\u62df\u673a\u3002<\/p>\n
                        \u56fe\u793a\u8bf4\u660e\u5982\u4e0b\uff0c\u5c06\u8bbe\u7f6e\u4ece “Before” \u66f4\u6539\u4e3a “After”\u3002<\/p>\n
                        \"network-diff.drawio.png\"<\/div>\n
                        $ HOST_IP<\/span>=<\/span>192.168.8.10\r\n$ CIDR<\/span>=<\/span>24\r\n$ GATEWAY<\/span>=<\/span>192.168.8.1\r\n$ DNS<\/span>=<\/span>192.168.8.1\r\n$ NWIF<\/span>=<\/span>enp1s0\r\n\r\n# br0 \u3092\u4f5c\u6210<\/span>\r\n$ <\/span>nmcli connection add type <\/span>bridge ifname br0\r\n$ <\/span>nmcli connection show\r\nNAME                UUID                                  TYPE       DEVICE\r\nbridge-br0          55bef68c-1232-46c3-adac-e40964c24d4d  bridge     br0\r\n...\r\n\r\n# \u65e2\u5b58\u306e enp1s0 \u3068\u540c\u3058\u8a2d\u5b9a\u3092 br0 \u306b\u4e0e\u3048\u308b<\/span>\r\n$ <\/span>nmcli connection modify bridge-br0 \\<\/span>\r\nipv4.method manual \\<\/span>\r\nipv4.addresses \"<\/span>$HOST_IP<\/span>\/<\/span>$CIDR<\/span>\"<\/span> \\<\/span>\r\nipv4.gateway \"<\/span>$GATEWAY<\/span>\"<\/span> \\<\/span>\r\nipv4.dns $DNS<\/span>\r\n\r\n# enp1s0 \u3092 br0 \u306b\u63a5\u7d9a<\/span>\r\n$ <\/span>nmcli connection add type <\/span>bridge-slave ifname $NWIF<\/span> master bridge-br0\r\n# \u65e2\u5b58\u306e enp1s0 \u306e\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u30a4\u30f3\u30bf\u30d5\u30a7\u30fc\u30b9\u3092\u524a\u9664<\/span>\r\n$ <\/span>nmcli connection delete $NWIF<\/span>\r\n\r\n# br0 \u306e\u8a2d\u5b9a\u3092\u53cd\u6620<\/span>\r\n$ <\/span>nmcli connection up bridge-br0\r\n<\/code><\/pre>\n
                        \u68c0\u67e5libvirt\u7684\u9ed8\u8ba4\u5b58\u50a8\u6c60\u3002<\/h4>\n
                        \u9996\u5148\uff0c\u9700\u8981\u786e\u8ba4\u662f\u5426\u5b58\u5728\u9ed8\u8ba4\u5b58\u50a8\u6c60\uff0clibvirt\u4f7f\u7528\u5404\u79cd\u8d44\u6e90\uff0c\u8fd9\u4e9b\u8d44\u6e90\u5728\u79f0\u4e3a”default pool”\u7684\u76ee\u5f55\/var\/lib\/libvirt\/images\u5185\u8fdb\u884c\u7ba1\u7406\u3002<\/p>\n
                        $ virsh pool-list --all\r\n Name      State    Autostart\r\n-------------------------------\r\n default   active   yes\r\n<\/code><\/pre>\n
                        \u5982\u679c\u6ca1\u6709\u9ed8\u8ba4\u6c60\u7684\u8bdd\uff0c\u5728\u8fd9\u91cc\u53ef\u4ee5\u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u521b\u5efa\u4e00\u4e2a\uff1a<\/p>\n
                        $ mkdir \/var\/lib\/libvirt\/images\r\n$ chmod 755 \/var\/lib\/libvirt\/images\r\n\r\n$ virsh pool-define \/dev\/stdin <<EOF\r\n<pool type='dir'>\r\n  <name>default<\/name>\r\n  <target>\r\n    <path>\/var\/lib\/libvirt\/images<\/path>\r\n  <\/target>\r\n<\/pool>\r\nEOF\r\n\r\n$ virsh pool-start default\r\n$ virsh pool-autostart default\r\n$ virsh pool-list --all\r\n<\/code><\/pre>\n
                        \u83b7\u53d6 Linux \u955c\u50cf<\/h4>\n
                        \u5728\u672c\u6587\u4e2d\uff0c\u6211\u4eec\u5c06\u4f7f\u7528Rocky Linux\u4f5c\u4e3aVM\u7684\u64cd\u4f5c\u7cfb\u7edf\uff0c\u53d6\u4ee3\u539f\u5148\u7684CentOS\u3002\u6211\u4eec\u5c06\u4ecedownload.rockylinux.org\u7f51\u7ad9\u4e0b\u8f7d\u6620\u50cf\u6587\u4ef6\uff0c\u5e76\u5c06\u5176\u4fdd\u5b58\u5230libvirt\u7684\u9ed8\u8ba4\u5b58\u50a8\u6c60\/var\/lib\/libvirt\/images\/\u4e2d\u3002<\/p>\n
                        $ sudo curl -L -o \/var\/lib\/libvirt\/images\/Rocky-9-GenericCloud.latest.x86_64.qcow2 https:\/\/download.rockylinux.org\/pub\/rocky\/9.2\/images\/x86_64\/Rocky-9-GenericCloud.latest.x86_64.qcow2\r\n<\/code><\/pre>\n
                        \u4f7f\u7528 Terraform \u8fdb\u884c\u865a\u62df\u673a\u7684\u914d\u7f6e<\/h3>\n
                        \u4e91\u521d\u59cb\u5316\u914d\u7f6e<\/h4>\n
                        cloud-init\u662f\u4e00\u79cd\u7528\u4e8e\u81ea\u52a8\u5316\u865a\u62df\u673a\u7684\u521d\u59cb\u8bbe\u7f6e\u7684\u673a\u5236\uff0c\u5728\u8bb8\u591aLinux\u53d1\u884c\u7248\u4e2d\u88ab\u91c7\u7528\u3002\u53ef\u4ee5\u901a\u8fc7\u51c6\u5907\u4ee5\u4e0b\u4e24\u4e2a\u914d\u7f6e\u6587\u4ef6\u6765\u63d0\u4f9b\u8bbe\u7f6e\u3002<\/p>\n
                        cloud_init.cfg: \u30e6\u30fc\u30b6\u8a2d\u5b9a\u3092\u8a18\u8ff0<\/p>\n
                        network_config.cfg: \u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u8a2d\u5b9a\u3092\u8a18\u8ff0<\/p>\n
                        #cloud-config<\/span>\r\nusers<\/span>:<\/span>\r\n  -<\/span> name<\/span>:<\/span> root<\/span>\r\n    ssh-authorized-keys<\/span>:<\/span>\r\n      -<\/span> \"<\/span><YOUR_SSH_KEY>\"<\/span> # \u81ea\u5206\u306e SSH \u516c\u958b\u9375\u3092\u66f8\u304f<\/span>\r\n\r\nruncmd<\/span>:<\/span>\r\n  -<\/span> dnf install -y python3<\/span>\r\n  -<\/span> dnf install -y libselinux-python<\/span>\r\n<\/code><\/pre>\n
                        \u5f53\u6267\u884c kubespray \u65f6\uff0c\u9700\u8981\u5b89\u88c5python3\u548clibselinux-python\u3002<\/div>\n
                        \u7f51\u7edc\u8bbe\u7f6e\u7531 network_config.cfg \u63d0\u4f9b\u3002<\/p>\n
                        \u7136\u800c\uff0c\u5982\u679c\u6bcf\u4e2a\u865a\u62df\u673a\u6709\u4e0d\u540c\u7684\u503c\uff08\u4f8b\u5982\u5206\u914d\u9759\u6001 IP \u5730\u5740\uff09\uff0c\u53ef\u4ee5\u521b\u5efa\u4e00\u4e2a\u5305\u542b\u53c2\u6570\u90e8\u5206\u7c7b\u4f3c\u4e8e ${foo} \u7684\u5360\u4f4d\u7b26\u7684\u6a21\u677f\uff0c\u5982\u4e0b\u6240\u793a\u3002\u4f7f\u7528 Terraform \u7684\u6a21\u677f\u529f\u80fd\uff0c\u7a0d\u540e\u53ef\u4ee5\u5c06\u503c\u5206\u914d\u7ed9\u5b83\uff0c\u5e76\u751f\u6210\u6bcf\u4e2a\u865a\u62df\u673a\u7684 network_config.cfg\u3002<\/p>\n
                        version<\/span>:<\/span> 2<\/span>\r\nethernets<\/span>:<\/span>\r\n  eth0<\/span>:<\/span>\r\n    dhcp4<\/span>:<\/span> no<\/span>\r\n    addresses<\/span>:<\/span> [<\/span>$<\/span>{<\/span>ip<\/span>}]<\/span>\r\n    gateway4<\/span>:<\/span> ${gateway}<\/span>\r\n    nameservers<\/span>:<\/span>\r\n      addresses<\/span>:<\/span> ${nameservers}<\/span>\r\n<\/code><\/pre>\n
                        \u51c6\u5907 Terraform \u7684 libvirt \u63d0\u4f9b\u7a0b\u5e8f<\/h4>\n
                        \u4f7f\u7528Terraform libvirt provider (dmacvicar\/libvirt)\uff0c\u53ef\u4ee5\u901a\u8fc7Terraform\u63a7\u5236libvirt\u3002<\/p>\n
                        \u7531\u4e8elibvirt\u9ed8\u8ba4\u9650\u5236\u5916\u90e8\u5e94\u7528\u7a0b\u5e8f\u5bf9libvirt\u8d44\u6e90\u7684\u8bbf\u95ee\uff0c\u56e0\u6b64\u4e3a\u4e86\u7ed9libvirt\u63d0\u4f9b\u8005\u6388\u4e88\u8bbf\u95ee\u6743\u9650\uff0c\u9700\u8981\u7f16\u8f91\/etc\/apparmor.d\/libvirt\/TEMPLATE.qemu\u6587\u4ef6\u5982\u4e0b\u3002<\/p>\n
                        # This profile is for the domain whose UUID matches this file.\r\n#\r\n\r\n#include <tunables\/global>\r\n\r\nprofile LIBVIRT_TEMPLATE flags=(attach_disconnected) {\r\n  #include <abstractions\/libvirt-qemu>\r\n   \/var\/lib\/libvirt\/images\/** rwk,\r\n   \/tmp\/** rwk,\r\n}\r\n<\/code><\/pre>\n
                        \u91cd\u65b0\u542f\u52a8libvirt\u3002<\/p>\n
                        $ sudo systemctl restart libvirtd\r\n<\/code><\/pre>\n
                        \u51c6\u5907Terraform\u6587\u4ef6<\/h3>\n
                        \u672c\u8282\u5c06\u89e3\u91ca\u5982\u4f55\u5b9e\u65bd\u7528\u4e8e\u6784\u5efaKubespray\u7684VM\u7684Terraform\u4ee3\u7801\u3002\u8fd9\u4e9b\u4ee3\u7801\u88ab\u6a21\u5757\u5316\u4e86\uff0c\u6240\u4ee5\u5982\u679c\u4f60\u4e0d\u77e5\u9053\u5982\u4f55\u5b9e\u65bd\uff0c\u4f46\u662f\u60f3\u8981\u5bfc\u5165\u5e76\u4f7f\u7528\u5b83\u4eec\uff0c\u53ef\u4ee5\u53c2\u8003\u672c\u6587\u4e2d\u5173\u4e8e\u5c06\u672c\u6587\u7684Terraform\u4ee3\u7801\u4f5c\u4e3a\u6a21\u5757\u4f7f\u7528\u7684\u6280\u5de7\u3002<\/div>\n
                        \u6587\u4ef6\u7531\u4ee5\u4e0b\u56db\u90e8\u5206\u7ec4\u6210\uff1a<\/p>\n
                          \n
                        • \n
                            provider.tf<\/ul>\n<\/li>\n<\/ul>\n
                             <\/p>\n
                              \n
                            • \n
                                main.tf<\/ul>\n<\/li>\n<\/ul>\n
                                 <\/p>\n
                                  \n
                                • \n
                                    variables.tf<\/ul>\n<\/li>\n<\/ul>\n
                                     <\/p>\n
                                      output.tf<\/ul>\n
                                      provider.tf (\u63d0\u4f9b\u8005.tf)<\/h4>\n
                                      \u58f0\u660e\u4f7f\u7528Terraform Libvirt Provider\u3002<\/p>\n
                                      terraform<\/span> {<\/span>\r\n  required_providers<\/span> {<\/span>\r\n    libvirt<\/span> =<\/span> {<\/span>\r\n      source<\/span>  =<\/span> \"dmacvicar\/libvirt\"<\/span>\r\n      version<\/span> =<\/span> \"0.7.1\"<\/span>\r\n    }<\/span>\r\n  }<\/span>\r\n}<\/span>\r\n\r\nprovider<\/span> \"libvirt\"<\/span> {<\/span>\r\n  uri<\/span> =<\/span> var<\/span>.<\/span>libvirt_uri<\/span>\r\n}<\/span>\r\n<\/code><\/pre>\n
                                      \u4e3b\u8981.tf<\/h4>\n
                                      \u4e3b\u8981\u63cf\u8ff0\u4e0e\u521b\u5efa\u865a\u62df\u673a\u76f8\u5173\u7684\u5904\u7406\u8fc7\u7a0b\u3002<\/p>\n
                                      locals<\/span> {<\/span>\r\n  cluster_cidr_splitted<\/span>      =<\/span> split<\/span>(<\/span>\"\/\"<\/span>,<\/span> var<\/span>.<\/span>cidr<\/span>)<\/span>\r\n  cluster_cidr_subnet<\/span>        =<\/span> local<\/span>.<\/span>cluster_cidr_splitted<\/span>[<\/span>0<\/span>]<\/span>\r\n  cluster_cidr_prefix<\/span>        =<\/span> local<\/span>.<\/span>cluster_cidr_splitted<\/span>[<\/span>1<\/span>]<\/span>\r\n  cluster_nameservers_string<\/span> =<\/span> \"[<\/span>\\\"<\/span>${join(\"<\/span>\\<\/span>\", <\/span>\\\"<\/span>\"<\/span>,<\/span> var<\/span>.<\/span>nameservers<\/span>)}<\/span>\\<\/span>\"]\"<\/span>\r\n\r\n  # Auto-calculate mac address from IP <\/span>\r\n  cluster_ips_parts<\/span> =<\/span> [<\/span>for<\/span> vm<\/span> in<\/span> var<\/span>.<\/span>vms<\/span> :<\/span> split<\/span>(<\/span>\".\"<\/span>,<\/span> vm<\/span>.<\/span>public_ip<\/span>)]<\/span>\r\n  cluster_mac_addrs<\/span> =<\/span> [<\/span>\r\n    for<\/span> ip_parts<\/span> in<\/span> local<\/span>.<\/span>cluster_ips_parts<\/span> :<\/span> format<\/span>(<\/span>\r\n      \"52:54:00:%02X:%02X:%02X\"<\/span>,<\/span>\r\n      tonumber<\/span>(<\/span>ip_parts<\/span>[<\/span>1<\/span>]),<\/span>\r\n      tonumber<\/span>(<\/span>ip_parts<\/span>[<\/span>2<\/span>]),<\/span>\r\n      tonumber<\/span>(<\/span>ip_parts<\/span>[<\/span>3<\/span>])<\/span>\r\n    )<\/span>\r\n  ]<\/span>\r\n  private_ips_parts<\/span> =<\/span> [<\/span>for<\/span> vm<\/span> in<\/span> var<\/span>.<\/span>vms<\/span> :<\/span> split<\/span>(<\/span>\".\"<\/span>,<\/span> vm<\/span>.<\/span>private_ip<\/span>)]<\/span>\r\n  private_mac_addrs<\/span> =<\/span> [<\/span>\r\n    for<\/span> ip_parts<\/span> in<\/span> local<\/span>.<\/span>private_ips_parts<\/span> :<\/span> format<\/span>(<\/span>\r\n      \"52:54:00:%02X:%02X:%02X\"<\/span>,<\/span>\r\n      tonumber<\/span>(<\/span>ip_parts<\/span>[<\/span>1<\/span>]),<\/span>\r\n      tonumber<\/span>(<\/span>ip_parts<\/span>[<\/span>2<\/span>]),<\/span>\r\n      tonumber<\/span>(<\/span>ip_parts<\/span>[<\/span>3<\/span>])<\/span>\r\n    )<\/span>\r\n  ]<\/span>\r\n}<\/span>\r\n\r\ndata<\/span> \"template_file\"<\/span> \"user_data\"<\/span> {<\/span>\r\n  count<\/span>    =<\/span> length<\/span>(<\/span>var<\/span>.<\/span>vms<\/span>)<\/span>\r\n  template<\/span> =<\/span> file<\/span>(<\/span>var<\/span>.<\/span>vms<\/span>[<\/span>count<\/span>.<\/span>index<\/span>].<\/span>cloudinit_file<\/span>)<\/span>\r\n}<\/span>\r\n\r\ndata<\/span> \"template_file\"<\/span> \"network_config\"<\/span> {<\/span>\r\n  count<\/span>    =<\/span> length<\/span>(<\/span>var<\/span>.<\/span>vms<\/span>)<\/span>\r\n  template<\/span> =<\/span> file<\/span>(<\/span>\"${path.module}\/network_config.cfg\"<\/span>)<\/span>\r\n  vars<\/span> =<\/span> {<\/span>\r\n    ip<\/span>          =<\/span> var<\/span>.<\/span>vms<\/span>[<\/span>count<\/span>.<\/span>index<\/span>].<\/span>public_ip<\/span>\r\n    cidr_prefix<\/span> =<\/span> local<\/span>.<\/span>cluster_cidr_prefix<\/span>\r\n    gateway<\/span>     =<\/span> var<\/span>.<\/span>gateway<\/span>\r\n    nameservers<\/span> =<\/span> local<\/span>.<\/span>cluster_nameservers_string<\/span>\r\n  }<\/span>\r\n}<\/span>\r\n\r\nresource<\/span> \"libvirt_cloudinit_disk\"<\/span> \"commoninit\"<\/span> {<\/span>\r\n  count<\/span>          =<\/span> length<\/span>(<\/span>var<\/span>.<\/span>vms<\/span>)<\/span>\r\n  name<\/span>           =<\/span> \"commoninit_${var.vms[count.index].name}.iso\"<\/span>\r\n  user_data<\/span>      =<\/span> data<\/span>.<\/span>template_file<\/span>.<\/span>user_data<\/span>[<\/span>count<\/span>.<\/span>index<\/span>].<\/span>rendered<\/span>\r\n  network_config<\/span> =<\/span> data<\/span>.<\/span>template_file<\/span>.<\/span>network_config<\/span>[<\/span>count<\/span>.<\/span>index<\/span>].<\/span>rendered<\/span>\r\n}<\/span>\r\n\r\nlocals<\/span> {<\/span>\r\n  volume_list<\/span>      =<\/span> {<\/span> for<\/span> vm<\/span> in<\/span> var<\/span>.<\/span>vms<\/span> :<\/span> \"${vm.name}\"<\/span> =<\/span>><\/span> flatten<\/span>([<\/span>for<\/span> volume<\/span> in<\/span> vm<\/span>.<\/span>volumes<\/span> :<\/span> volume<\/span>])<\/span> }<\/span>\r\n  volume_name_list<\/span> =<\/span> [<\/span>for<\/span> vm<\/span>,<\/span> volumes<\/span> in<\/span> local<\/span>.<\/span>volume_list<\/span> :<\/span> [<\/span>for<\/span> volume<\/span> in<\/span> volumes<\/span> :<\/span> {<\/span> \"name\"<\/span> :<\/span> \"${vm}_${volume.name}\"<\/span>,<\/span> \"disk\"<\/span> :<\/span> volume<\/span>.<\/span>disk<\/span> }]]<\/span>\r\n  volumes<\/span>          =<\/span> flatten<\/span>(<\/span>local<\/span>.<\/span>volume_name_list<\/span>)<\/span>\r\n  volumes_indexed<\/span>  =<\/span> {<\/span> for<\/span> index<\/span>,<\/span> volume<\/span> in<\/span> local<\/span>.<\/span>volumes<\/span> :<\/span> volume<\/span>.<\/span>name<\/span> =<\/span>><\/span> index<\/span> }<\/span>\r\n}<\/span>\r\n\r\nresource<\/span> \"libvirt_domain\"<\/span> \"vm\"<\/span> {<\/span>\r\n  count<\/span>  =<\/span> length<\/span>(<\/span>var<\/span>.<\/span>vms<\/span>)<\/span>\r\n  name<\/span>   =<\/span> var<\/span>.<\/span>vms<\/span>[<\/span>count<\/span>.<\/span>index<\/span>].<\/span>name<\/span>\r\n  vcpu<\/span>   =<\/span> var<\/span>.<\/span>vms<\/span>[<\/span>count<\/span>.<\/span>index<\/span>].<\/span>vcpu<\/span>\r\n  memory<\/span> =<\/span> var<\/span>.<\/span>vms<\/span>[<\/span>count<\/span>.<\/span>index<\/span>].<\/span>memory<\/span>\r\n\r\n  disk<\/span> {<\/span>\r\n    volume_id<\/span> =<\/span> libvirt_volume<\/span>.<\/span>system<\/span>[<\/span>count<\/span>.<\/span>index<\/span>].<\/span>id<\/span>\r\n  }<\/span>\r\n\r\n  cloudinit<\/span> =<\/span> libvirt_cloudinit_disk<\/span>.<\/span>commoninit<\/span>[<\/span>count<\/span>.<\/span>index<\/span>].<\/span>id<\/span>\r\n  autostart<\/span> =<\/span> true<\/span>\r\n\r\n  # Public network<\/span>\r\n  network_interface<\/span> {<\/span>\r\n    bridge<\/span>    =<\/span> var<\/span>.<\/span>bridge<\/span>\r\n    addresses<\/span> =<\/span> [<\/span>var<\/span>.<\/span>vms<\/span>[<\/span>count<\/span>.<\/span>index<\/span>].<\/span>public_ip<\/span>]<\/span>\r\n    mac<\/span>       =<\/span> local<\/span>.<\/span>cluster_mac_addrs<\/span>[<\/span>count<\/span>.<\/span>index<\/span>]<\/span>\r\n  }<\/span>\r\n\r\n  # Private network<\/span>\r\n  network_interface<\/span> {<\/span>\r\n    network_name<\/span> =<\/span> \"default\"<\/span>\r\n    addresses<\/span>    =<\/span> [<\/span>var<\/span>.<\/span>vms<\/span>[<\/span>count<\/span>.<\/span>index<\/span>].<\/span>private_ip<\/span>]<\/span>\r\n    mac<\/span>          =<\/span> local<\/span>.<\/span>private_mac_addrs<\/span>[<\/span>count<\/span>.<\/span>index<\/span>]<\/span>\r\n  }<\/span>\r\n\r\n  qemu_agent<\/span> =<\/span> true<\/span>\r\n\r\n  cpu<\/span> {<\/span>\r\n    mode<\/span> =<\/span> \"host-passthrough\"<\/span>\r\n  }<\/span>\r\n\r\n  graphics<\/span> {<\/span>\r\n    type<\/span>        =<\/span> \"vnc\"<\/span>\r\n    listen_type<\/span> =<\/span> \"address\"<\/span>\r\n  }<\/span>\r\n\r\n  # Makes the tty0 available via `virsh console`<\/span>\r\n  console<\/span> {<\/span>\r\n    type<\/span>        =<\/span> \"pty\"<\/span>\r\n    target_port<\/span> =<\/span> \"0\"<\/span>\r\n    target_type<\/span> =<\/span> \"serial\"<\/span>\r\n  }<\/span>\r\n}<\/span>\r\n\r\nresource<\/span> \"libvirt_volume\"<\/span> \"system\"<\/span> {<\/span>\r\n  count<\/span>          =<\/span> length<\/span>(<\/span>var<\/span>.<\/span>vms<\/span>)<\/span>\r\n  name<\/span>           =<\/span> \"${var.vms[count.index].name}_system.qcow2\"<\/span>\r\n  pool<\/span>           =<\/span> var<\/span>.<\/span>pool<\/span>\r\n  format<\/span>         =<\/span> \"qcow2\"<\/span>\r\n  base_volume_id<\/span> =<\/span> var<\/span>.<\/span>vm_base_image_uri<\/span>\r\n  size<\/span>           =<\/span> var<\/span>.<\/span>vms<\/span>[<\/span>count<\/span>.<\/span>index<\/span>].<\/span>disk<\/span>\r\n}<\/span>\r\n<\/code><\/pre>\n
                                      variables.tf\u7684\u610f\u601d\u662ftf\u662f\u6307Terraform\uff0cvariables\u662f\u6307\u53d8\u91cf\uff0c.tf\u662f\u6307Terraform\u7684\u6587\u4ef6\u6269\u5c55\u540d\u3002<\/h4>\n
                                      \u58f0\u660e\u53d8\u91cf\u3002<\/p>\n
                                      variable<\/span> \"libvirt_uri\"<\/span> {<\/span>\r\n  type<\/span> =<\/span> string<\/span>\r\n}<\/span>\r\n\r\nvariable<\/span> \"vm_base_image_uri\"<\/span> {<\/span>\r\n  type<\/span> =<\/span> string<\/span>\r\n}<\/span>\r\n\r\nvariable<\/span> \"bridge\"<\/span> {<\/span>\r\n  type<\/span> =<\/span> string<\/span>\r\n}<\/span>\r\n\r\nvariable<\/span> \"gateway\"<\/span> {<\/span>\r\n  type<\/span> =<\/span> string<\/span>\r\n}<\/span>\r\n\r\nvariable<\/span> \"cidr\"<\/span> {<\/span>\r\n  type<\/span> =<\/span> string<\/span>\r\n}<\/span>\r\n\r\nvariable<\/span> \"nameservers\"<\/span> {<\/span>\r\n  type<\/span> =<\/span> list<\/span>(<\/span>string<\/span>)<\/span>\r\n}<\/span>\r\n\r\nvariable<\/span> \"pool\"<\/span> {<\/span>\r\n  type<\/span>    =<\/span> string<\/span>\r\n  default<\/span> =<\/span> \"default\"<\/span>\r\n}<\/span>\r\n\r\nvariable<\/span> \"vms\"<\/span> {<\/span>\r\n  type<\/span> =<\/span> list<\/span>(<\/span>\r\n    object<\/span>({<\/span>\r\n      name<\/span>           =<\/span> string<\/span>\r\n      vcpu<\/span>           =<\/span> number<\/span>\r\n      memory<\/span>         =<\/span> number<\/span>\r\n      disk<\/span>           =<\/span> number<\/span>\r\n      public_ip<\/span>      =<\/span> string<\/span>\r\n      private_ip<\/span>     =<\/span> string<\/span>\r\n      cloudinit_file<\/span> =<\/span> string<\/span>\r\n\r\n      kube_control_plane<\/span> =<\/span> bool<\/span>\r\n      kube_node<\/span>          =<\/span> bool<\/span>\r\n      etcd<\/span>               =<\/span> bool<\/span>\r\n    })<\/span>\r\n  )<\/span>\r\n}<\/span>\r\n\r\n<\/code><\/pre>\n
                                      output.tf \u7684\u6c49\u8bed\u7ffb\u8bd1\u662f\u8f93\u51fa.tf\u3002<\/h4>\n
                                      \u5728\u63a5\u4e0b\u6765\u7684\u8fc7\u7a0b\u4e2d\uff0c\u8f93\u51fa\u8981\u4f20\u9012\u7ed9Ansible\u7684\u4e3b\u673a\u4fe1\u606f\u3002<\/p>\n
                                      locals<\/span> {<\/span>\r\n  kubespray_hosts_keys<\/span> =<\/span> [<\/span>\"name\"<\/span>,<\/span> \"kube_control_plane\"<\/span>,<\/span> \"kube_node\"<\/span>,<\/span> \"etcd\"<\/span>]<\/span>\r\n  kubespray_hosts<\/span> =<\/span> [<\/span>for<\/span> vm<\/span> in<\/span> var<\/span>.<\/span>vms<\/span> :<\/span>\r\n    merge<\/span>(<\/span>\r\n      {<\/span>\r\n        for<\/span> key<\/span>,<\/span> value<\/span> in<\/span> vm<\/span> :<\/span> key<\/span> =<\/span>><\/span> value<\/span> if<\/span> contains<\/span>(<\/span>local<\/span>.<\/span>kubespray_hosts_keys<\/span>,<\/span> key<\/span>)<\/span>\r\n      },<\/span>\r\n      {<\/span>\r\n        ip<\/span>        =<\/span> vm<\/span>.<\/span>public_ip<\/span>\r\n        access_ip<\/span> =<\/span> vm<\/span>.<\/span>private_ip<\/span>\r\n    })<\/span>\r\n  ]<\/span>\r\n}<\/span>\r\n\r\noutput<\/span> \"kubespray_hosts\"<\/span> {<\/span>\r\n  value<\/span> =<\/span> local<\/span>.<\/span>kubespray_hosts<\/span>\r\n}<\/span>\r\n<\/code><\/pre>\n
                                      \u6267\u884c Terraform<\/h3>\n
                                      \u6267\u884cTerraform\u5e76\u8fdb\u884c\u865a\u62df\u673a\u914d\u7f6e\u3002<\/p>\n
                                      $ terraform init\r\n$ terraform apply -auto-approve\r\n<\/code><\/pre>\n
                                      \u786e\u8ba4\u865a\u62df\u673a\u5df2\u7ecf\u88ab\u521b\u5efa\u5e76\u6b63\u5728\u8fd0\u884c\u3002<\/p>\n
                                      $ virsh list --all\r\n Id   Name           State\r\n------------------------------\r\n 1    k8s-master-1   running\r\n 2    k8s-worker-1   running\r\n 3    k8s-worker-2   running\r\n<\/code><\/pre>\n
                                      \u53ef\u4ee5\u901a\u8fc7\u4ee5\u4e0b\u547d\u4ee4\u786e\u8ba4\u521b\u5efa\u7684\u865a\u62df\u673a\u5df2\u8fde\u63a5\u5230br0\u3002<\/p>\n
                                      $ ip link show master br0\r\nbridge name     bridge id               STP enabled     interfaces\r\n2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP mode DEFAULT group default qlen 1000\r\n    link\/ether 48:21:0b:57:b2:52 brd ff:ff:ff:ff:ff:ff\r\n5: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN mode DEFAULT group default qlen 1000\r\n    link\/ether fe:54:00:00:00:04 brd ff:ff:ff:ff:ff:ff\r\n6: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN mode DEFAULT group default qlen 1000\r\n    link\/ether fe:54:00:00:00:02 brd ff:ff:ff:ff:ff:ff\r\n7: vnet6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN mode DEFAULT group default qlen 1000\r\n    link\/ether fe:54:00:00:00:03 brd ff:ff:ff:ff:ff:ff\r\n8: vnet7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN mode DEFAULT group default qlen 1000\r\n    link\/ether fe:54:00:00:00:01 brd ff:ff:ff:ff:ff:ff\r\n<\/code><\/pre>\n
                                      \u4f7f\u7528 Kubespray \u6765\u521b\u5efa Kubernetes \u96c6\u7fa4\u3002<\/h3>\n
                                      Kubespray\u662f\u4e00\u4e2a\u5f00\u6e90\u9879\u76ee\uff0c\u63d0\u4f9b\u4e86\u81ea\u52a8\u5316\u6784\u5efaKubernetes\u96c6\u7fa4\u7684Ansible playbook\u3002\u6309\u7167kubespray\/docs\/setting-up-your-first-cluster.md at master \u00b7 kubernetes-sigs\/kubespray\u4e2d\u7684\u6b65\u9aa4\u8fdb\u884c\u64cd\u4f5c\u3002<\/p>\n
                                      \u9996\u5148\uff0c\u514b\u9686 kubespray \u7684\u4ee3\u7801\u5e93\u3002<\/p>\n
                                      $ git clone git@github.com:kubernetes-sigs\/kubespray.git\r\n$ cd kubespray\r\n<\/code><\/pre>\n
                                      \u63a5\u4e0b\u6765\uff0c\u590d\u5236\u73b0\u6709\u7684\u6837\u672c\u5e76\u521b\u5efa\u914d\u7f6e\u6587\u4ef6\u7684\u57fa\u7840\u3002<\/p>\n
                                      $ git checkout release-2.23\r\n$ cp -rfp inventory\/sample inventory\/mycluster\r\n<\/code><\/pre>\n
                                      \u5728\u672c\u6587\u4e2d\uff0c\u6211\u4eec\u51b3\u5b9a\u4f7f\u7528crio\u4f5c\u4e3a\u5bb9\u5668\u8fd0\u884c\u65f6\uff08\u9ed8\u8ba4\u60c5\u51b5\u4e0b\u662fcontainerd\uff09\u3002\u5c06group_vars\/k8s_cluster\/k8s_cluster.yml\u4e2d\u7684container_manager\u7684\u503c\u66f4\u6539\u4e3acrio\u3002<\/p>\n
                                      ## Container runtime<\/span>\r\n## docker for docker, crio for cri-o and containerd for containerd.<\/span>\r\n## Default: containerd<\/span>\r\ncontainer_manager<\/span>:<\/span> crio<\/span>\r\n<\/code><\/pre>\n
                                      \u5982\u679c\u8981\u8bbe\u7f6e\u66f4\u8be6\u7ec6\u7684\u9009\u9879\uff0c\u8bf7\u4fee\u6539 inventory\/mycluster\/group_vars\/ \u76ee\u5f55\u4e0b\u7684\u6587\u4ef6\uff08\u9644\u6ce8\uff1a\u4ee3\u7406\u8bbe\u7f6e\uff09\u3002<\/p>\n
                                      \u521b\u5efa\u5e93\u5b58\u6e05\u5355<\/h4>\n
                                      \u6211\u5c06\u4ecb\u7ecd\u4ee5\u4e0b\u4e24\u79cd\u65b9\u6cd5\uff0c\u4f46\u4f60\u53ea\u9700\u8981\u9009\u62e9\u5176\u4e2d\u4e00\u79cd\u5c31\u53ef\u4ee5\u4e86\u3002<\/p>\n
                                        \n
                                      1. \n
                                          \u4f7f\u7528\u52a8\u6001\u5e93\u5b58\u7684\u65b9\u6cd5\u3002<\/ol>\n<\/li>\n<\/ol>\n
                                           <\/p>\n
                                            \u624b\u52a8\u7f16\u8f91hosts.yaml\u6587\u4ef6\u7684\u65b9\u6cd5\u3002<\/ol>\n
                                            \u4f7f\u7528\u52a8\u6001\u5e93\u5b58\u65b9\u6cd5<\/h5>\n
                                            \u5728\u6267\u884cterraform\u540e\uff0c\u4ece\u751f\u6210\u7684terraform.tfstate\u6587\u4ef6\u4e2d\u52a8\u6001\u5730\u63d0\u53d6\u6e05\u5355\u4fe1\u606f\u3002<\/p>\n
                                            \u53ea\u9700\u51c6\u5907\u4e00\u4e2a\u4ec5\u8fdb\u884c\u8f93\u51fa\u7684\u811a\u672c\uff08\u793a\u4f8b\u5b9e\u73b0\uff1a.\/generate_inventory.py\uff09\uff0c\u8be5\u811a\u672c\u5c06\u8f93\u51fa\u4ee5 JSON \u63cf\u8ff0\u7684 inventory \u4fe1\u606f\u3002\u4ee5\u4e0b\u662f\u4e00\u4e2a\u8bfb\u53d6 terraform.state \u7684 .outputs \u5e76\u521b\u5efa inventory \u7684\u811a\u672c\u5b9e\u73b0\u793a\u4f8b\uff1a<\/p>\n
                                            #!\/usr\/bin\/env python3\r\n<\/span>\r\nimport<\/span> json<\/span>\r\nimport<\/span> re<\/span>\r\n\r\n\r\ndef<\/span> main<\/span>():<\/span>\r\n    output<\/span> =<\/span> get_outputs<\/span>()<\/span>\r\n    hosts<\/span> =<\/span> output<\/span>[<\/span>'<\/span>kubespray_hosts<\/span>'<\/span>][<\/span>'<\/span>value<\/span>'<\/span>]<\/span>\r\n    libvirt_uri<\/span> =<\/span> output<\/span>[<\/span>'<\/span>libvirt_uri<\/span>'<\/span>][<\/span>'<\/span>value<\/span>'<\/span>]<\/span>\r\n\r\n    hostvars<\/span> =<\/span> {}<\/span>\r\n    kube_control_plane<\/span> =<\/span> []<\/span>\r\n    kube_node<\/span> =<\/span> []<\/span>\r\n    etcd<\/span> =<\/span> []<\/span>\r\n\r\n    for<\/span> host<\/span> in<\/span> hosts<\/span>:<\/span>\r\n        name<\/span> =<\/span> host<\/span>[<\/span>'<\/span>name<\/span>'<\/span>]<\/span>\r\n        ip<\/span> =<\/span> host<\/span>[<\/span>'<\/span>ip<\/span>'<\/span>]<\/span>\r\n        access_ip<\/span> =<\/span> host<\/span>[<\/span>'<\/span>access_ip<\/span>'<\/span>]<\/span>\r\n        hostvars<\/span>.<\/span>update<\/span>({<\/span>\r\n          name<\/span>:<\/span> {<\/span>\r\n              \"<\/span>ansible_host<\/span>\"<\/span>:<\/span> access_ip<\/span>,<\/span>\r\n              \"<\/span>ip<\/span>\"<\/span>:<\/span> ip<\/span>,<\/span>\r\n          }<\/span>\r\n        })<\/span>\r\n\r\n        regex<\/span> =<\/span> r<\/span>\"<\/span>^qemu(\\+ssh)?:\/\/([^\/]*)\/.*<\/span>\"<\/span>\r\n        res<\/span> =<\/span> re<\/span>.<\/span>match<\/span>(<\/span>regex<\/span>,<\/span> libvirt_uri<\/span>)<\/span>\r\n        if<\/span> res<\/span>:<\/span>\r\n          hostname<\/span> =<\/span> res<\/span>[<\/span>2<\/span>]<\/span>\r\n          if<\/span> hostname<\/span> !=<\/span> \"\"<\/span>:<\/span>\r\n            hostvars<\/span>[<\/span>name<\/span>].<\/span>update<\/span>({<\/span>\r\n              \"<\/span>ansible_ssh_common_args<\/span>\"<\/span>:<\/span> f<\/span>\"<\/span>-J <\/span>{<\/span>hostname<\/span>}<\/span>\"<\/span>\r\n          })<\/span>\r\n\r\n        if<\/span> host<\/span>[<\/span>\"<\/span>kube_control_plane<\/span>\"<\/span>]:<\/span>\r\n            kube_control_plane<\/span>.<\/span>append<\/span>(<\/span>name<\/span>)<\/span>\r\n        if<\/span> host<\/span>[<\/span>\"<\/span>kube_node<\/span>\"<\/span>]:<\/span>\r\n            kube_node<\/span>.<\/span>append<\/span>(<\/span>name<\/span>)<\/span>\r\n        if<\/span> host<\/span>[<\/span>\"<\/span>etcd<\/span>\"<\/span>]:<\/span>\r\n            etcd<\/span>.<\/span>append<\/span>(<\/span>name<\/span>)<\/span>\r\n\r\n    inventory<\/span> =<\/span> {<\/span>\r\n        \"<\/span>_meta<\/span>\"<\/span>:<\/span> {<\/span>\r\n            \"<\/span>hostvars<\/span>\"<\/span>:<\/span> hostvars<\/span>,<\/span>\r\n        },<\/span>\r\n        \"<\/span>kube_control_plane<\/span>\"<\/span>:<\/span> kube_control_plane<\/span>,<\/span>\r\n        \"<\/span>kube_node<\/span>\"<\/span>:<\/span> kube_node<\/span>,<\/span>\r\n        \"<\/span>etcd<\/span>\"<\/span>:<\/span> etcd<\/span>,<\/span>\r\n        \"<\/span>k8s_cluster<\/span>\"<\/span>:<\/span> {<\/span>\r\n            \"<\/span>children<\/span>\"<\/span>:<\/span> [<\/span>\r\n                \"<\/span>kube_control_plane<\/span>\"<\/span>,<\/span>\r\n                \"<\/span>kube_node<\/span>\"<\/span>,<\/span>\r\n            ]<\/span>\r\n        }<\/span>\r\n    }<\/span>\r\n\r\n\r\n    print<\/span>(<\/span>json<\/span>.<\/span>dumps<\/span>(<\/span>inventory<\/span>))<\/span>\r\n\r\n\r\ndef<\/span> get_outputs<\/span>():<\/span>\r\n    tfstate_path<\/span> =<\/span> '<\/span>.\/terraform.tfstate<\/span>'<\/span>\r\n    with<\/span> open<\/span>(<\/span>tfstate_path<\/span>)<\/span> as<\/span> f<\/span>:<\/span>\r\n        tfstate<\/span> =<\/span> json<\/span>.<\/span>load<\/span>(<\/span>f<\/span>)<\/span>\r\n    return<\/span> tfstate<\/span>[<\/span>'<\/span>outputs<\/span>'<\/span>]<\/span>\r\n\r\n\r\nmain<\/span>()<\/span>\r\n<\/code><\/pre>\n
                                            \u8fd0\u884c\u6b64generate_inventory.py\u811a\u672c\u4f1a\u8f93\u51fa\u4ee5\u4e0bJSON\uff1a<\/p>\n
                                            {<\/span>\r\n  \"_meta\"<\/span>:<\/span> {<\/span>\r\n    \"hostvars\"<\/span>:<\/span> {<\/span>\r\n      \"storage1\"<\/span>:<\/span> {<\/span>\r\n        \"ansible_host\"<\/span>:<\/span> \"192.168.122.201\"<\/span>,<\/span>\r\n        \"ip\"<\/span>:<\/span> \"192.168.8.201\"<\/span>\r\n      },<\/span>\r\n      \"storage2\"<\/span>:<\/span> {<\/span>\r\n        \"ansible_host\"<\/span>:<\/span> \"192.168.122.202\"<\/span>,<\/span>\r\n        \"ip\"<\/span>:<\/span> \"192.168.8.202\"<\/span>\r\n      },<\/span>\r\n      \"storage3\"<\/span>:<\/span> {<\/span>\r\n        \"ansible_host\"<\/span>:<\/span> \"192.168.122.203\"<\/span>,<\/span>\r\n        \"ip\"<\/span>:<\/span> \"192.168.8.203\"<\/span>\r\n      }<\/span>\r\n    }<\/span>\r\n  },<\/span>\r\n  \"kube_control_plane\"<\/span>:<\/span> [<\/span>\r\n    \"storage1\"<\/span>\r\n  ],<\/span>\r\n  \"kube_node\"<\/span>:<\/span> [<\/span>\r\n    \"storage1\"<\/span>,<\/span>\r\n    \"storage2\"<\/span>,<\/span>\r\n    \"storage3\"<\/span>\r\n  ],<\/span>\r\n  \"etcd\"<\/span>:<\/span> [<\/span>\r\n    \"storage1\"<\/span>\r\n  ],<\/span>\r\n  \"k8s_cluster\"<\/span>:<\/span> {<\/span>\r\n    \"children\"<\/span>:<\/span> [<\/span>\r\n      \"kube_control_plane\"<\/span>,<\/span>\r\n      \"kube_node\"<\/span>\r\n    ]<\/span>\r\n  }<\/span>\r\n}<\/span>\r\n<\/code><\/pre>\n
                                            \u6307\u5b9a\u8fd9\u4e2a\u4f5c\u4e3a\u5e93\u5b58\uff0c\u5e76\u521b\u5efaKubernetes\u96c6\u7fa4\uff1a<\/p>\n
                                            $ docker pull quay.io\/kubespray\/kubespray:v2.23.1\r\n$ docker run --rm -it \\\r\n  --mount type=bind,source=\"$(pwd)\"\/inventory,dst=\/inventory \\\r\n  --mount type=bind,source=\"$(pwd)\"\/generate_inventory.py,dst=\/kubespray\/generate_inventory.py \\\r\n  --mount type=bind,source=\"$(pwd)\"\/terraform.tfstate,dst=\/kubespray\/terraform.tfstate \\\r\n  --mount type=bind,source=\"${HOME}\"\/.ssh\/id_rsa,dst=\/root\/.ssh\/id_rsa \\\r\n  quay.io\/kubespray\/kubespray:v2.23.1 bash\r\n\r\n# Inside a container\r\n$ ansible-playbook -i generate_inventory.py cluster.yml\r\n\u2026\r\nPLAY RECAP *****************************************************************************************************************\r\nlocalhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   \r\nk8s-master-1               : ok=765  changed=138  unreachable=0    failed=0    skipped=1263 rescued=0    ignored=8   \r\nk8s-worker-1               : ok=583  changed=104  unreachable=0    failed=0    skipped=795  rescued=0    ignored=2   \r\nk8s-worker-2               : ok=523  changed=82   unreachable=0    failed=0    skipped=763  rescued=0    ignored=1   \r\n\r\nTuesday 01 August 2023  13:16:52 +0000 (0:00:00.041)       0:59:01.983 ********                                                                                      ===============================================================================      \r\n<\/code><\/pre>\n
                                            \u5982\u679c\u5728\u5b89\u88c5\u8fc7\u7a0b\u4e2d\u51fa\u73b0\u6545\u969c\u6216\u5176\u4ed6\u539f\u56e0\u60f3\u8981\u91cd\u65b0\u5b89\u88c5\u7684\u8bdd\uff0c\u53ef\u4ee5\u901a\u8fc7\u4ee5\u4e0b\u65b9\u6cd5\u8fdb\u884c\u91cd\u7f6e\u3002<\/p>\n
                                            $ ansible-playbook -i .\/generate_inventory.py reset.yml\r\n<\/code><\/pre>\n
                                            \u8bf7\u53c2\u8003Kubespray\u5b98\u65b9\u4ed3\u5e93\u7684 https:\/\/github.com\/kubernetes-sigs\/kubespray\/blob\/master\/docs\/aws.md#dynamic-inventory \uff0c\u83b7\u53d6\u6709\u5173Kubespray\u52a8\u6001\u6e05\u5355\u7684\u8be6\u7ec6\u4fe1\u606f\u3002<\/p>\n
                                            (\u9009\u98792) \u624b\u52a8\u7f16\u8f91 hosts.yaml<\/h5>\n
                                            \u542f\u52a8 kubespray \u5bb9\u5668\u3002<\/p>\n
                                            $ docker pull quay.io\/kubespray\/kubespray:v2.23.1\r\n$ docker run --rm -it \\\r\n  --mount type=bind,source=\"$(pwd)\"\/inventory\/mycluster,dst=\/inventory \\\r\n  --mount type=bind,source=\"${HOME}\"\/.ssh\/id_rsa,dst=\/root\/.ssh\/id_rsa \\\r\n  quay.io\/kubespray\/kubespray:v2.23.1 bash\r\n<\/code><\/pre>\n
                                            \u8fdb\u5165\u5bb9\u5668\u540e\uff0c\u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\u3002<\/p>\n
                                            $ declare -a IPS=(192.168.8.201 192.168.8.202 192.168.8.203)\r\n$ CONFIG_FILE=\/inventory\/hosts.yaml python3 contrib\/inventory_builder\/inventory.py ${IPS[@]}\r\n<\/code><\/pre>\n
                                            \u56e0\u4e3a\u751f\u6210\u4e86\u5b58\u8d27\u6587\u4ef6 \/inventory\/mycluster\/hosts.yaml\uff0c\u56e0\u6b64\u6839\u636e\u9700\u8981\u8fdb\u884c\u4fee\u6b63\u4e3b\u673a\u4fe1\u606f\u3002\u5728\u672c\u6587\u4e2d\uff0c\u6211\u4eec\u4f7f\u7528\u4ee5\u4e0b\u914d\u7f6e\uff1a<\/p>\n
                                            all<\/span>:<\/span>\r\n  hosts<\/span>:<\/span>\r\n    storage1<\/span>:<\/span>\r\n      ansible_host<\/span>:<\/span> 192.168.122.201<\/span>\r\n      ip<\/span>:<\/span> 192.168.8.201<\/span>\r\n    storage2<\/span>:<\/span>\r\n      ansible_host<\/span>:<\/span> 192.168.122.202<\/span>\r\n      ip<\/span>:<\/span> 192.168.8.202<\/span>\r\n    storage3<\/span>:<\/span>\r\n      ansible_host<\/span>:<\/span> 192.168.122.203<\/span>\r\n      ip<\/span>:<\/span> 192.168.8.203<\/span>\r\n  children<\/span>:<\/span>\r\n    kube_control_plane<\/span>:<\/span>\r\n      hosts<\/span>:<\/span>\r\n        storage1<\/span>:<\/span>\r\n    kube_node<\/span>:<\/span>\r\n      hosts<\/span>:<\/span>\r\n        storage1<\/span>:<\/span>\r\n        storage2<\/span>:<\/span>\r\n        storage3<\/span>:<\/span>\r\n    etcd<\/span>:<\/span>\r\n      hosts<\/span>:<\/span>\r\n        storage1<\/span>:<\/span>\r\n    k8s_cluster<\/span>:<\/span>\r\n      children<\/span>:<\/span>\r\n        kube_control_plane<\/span>:<\/span>\r\n        kube_node<\/span>:<\/span>\r\n<\/code><\/pre>\n
                                            $ docker run --rm -it \\\r\n  --mount type=bind,source=\"$(pwd)\"\/inventory,dst=\/inventory \\\r\n  --mount type=bind,source=\"$(pwd)\"\/cluster.yml,dst=\/kubespray\/cluster.yml \\\r\n  --mount type=bind,source=\"${HOME}\"\/.kube,dst=\/root\/.kube \\\r\n  --mount type=bind,source=\"${HOME}\"\/.ssh\/id_rsa,dst=\/root\/.ssh\/id_rsa \\\r\n  quay.io\/kubespray\/kubespray:v2.23.1 bash\r\n\r\n# Inside a container\r\n$ ansible-playbook -i \/inventory\/hosts.yaml cluster.yml\r\n<\/code><\/pre>\n
                                            \u8bbf\u95ee Kubernetes \u96c6\u7fa4<\/h4>\n
                                            \u4ece\u4efb\u4e00\u4e3b\u8282\u70b9\u83b7\u53d6\u8ba4\u8bc1\u4fe1\u606f admin.conf\u3002<\/p>\n
                                            $ mkdir -p ~\/.kube\r\n$ scp root@192.168.8.201:\/etc\/kubernetes\/admin.conf ~\/.kube\r\n<\/code><\/pre>\n
                                            \u8bf7\u5c06 admin.conf \u6587\u4ef6\u4e2d clusters.cluster.server \u7684 IP \u5730\u5740\u4fee\u6539\u4e3a\u516c\u5171IP\u5730\u5740\uff08\u4f8b\u5982\uff1a192.168.8.201\uff09\uff0c\u539f\u5730\u5740\u4e3a127.0.0.1\u3002<\/p>\n
                                            apiVersion: v1\r\nclusters:\r\n- cluster:\r\n    certificate-authority-data: <CERTIFICATE_AUTHORITY_DATA>\r\n    server: https:\/\/192.168.8.201:6443\r\n  name: cluster.local\r\n...\r\n<\/code><\/pre>\n
                                            \u5728.bashrc\u7b49\u6587\u4ef6\u4e2d\u8bbe\u7f6e\u4ee5\u4e0b\u73af\u5883\u53d8\u91cf\u3002<\/p>\n
                                            $ export KUBECONFIG=$HOME\/.kube\/admin.conf \r\n<\/code><\/pre>\n
                                            \u901a\u8fc7\u4ee5\u4e0a\u7684\u6b65\u9aa4\uff0c\u60a8\u53ef\u4ee5\u4f7f\u7528 kubectl \u8fdb\u5165\u96c6\u7fa4\u3002<\/p>\n
                                            $ kubectl cluster-info\r\nKubernetes control plane is running at https:\/\/192.168.8.201:6443\r\n\r\nTo further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.\r\n\r\n$ kubectl get nodes\r\nNAME           STATUS   ROLES           AGE   VERSION\r\nk8s-master-1   Ready    control-plane   70m   v1.26.5\r\nk8s-worker-1   Ready    <none>          68m   v1.26.5\r\nk8s-worker-2   Ready    <none>          68m   v1.26.5\r\n<\/code><\/pre>\n
                                            \n
                                            \u6280\u5de7<\/h2>\n
                                            \u4f7f\u7528Kubespray\u8fdb\u884c\u4ee3\u7406\u914d\u7f6e<\/h3>\n
                                            \u5982\u679c\u8981\u8bbe\u7f6e\u4ee3\u7406\uff0c\u8bf7\u7f16\u8f91inventory\/mycluster\/group_vars\/all\/all.yml\u6587\u4ef6\u4e2d\u7684\u4ee5\u4e0b\u9879\u76ee\u3002<\/p>\n
                                            http_proxy: 'http:\/\/your-proxy-server:8080'\r\nhttps_proxy: 'http:\/\/your-proxy-server:8080'\r\nno_proxy: 'localhost,127.0.0.1,.yourdomain.com'\r\n<\/code><\/pre>\n
                                            \u5c06\u81ea\u5df1\u7684playbook\u96c6\u6210\u5230Kubespray\u4e2d\u3002<\/h3>\n
                                            \u5982\u679c\u5728\u5e94\u7528cluster.yml\u4e4b\u540e\u9700\u8981\u6267\u884c\u81ea\u5b9a\u4e49\u7684\u9644\u52a0\u5904\u7406\uff0c\u53ef\u4ee5\u8fd9\u6837\u64cd\u4f5c\u3002<\/p>\n
                                            \u67e5\u770bKubespray\u7684cluster.yml\u6587\u4ef6\uff0c\u53ef\u4ee5\u770b\u51fa\u53ea\u662f\u7b80\u5355\u5730\u5bfc\u5165\u4e86playbooks\/cluster.yml\u6587\u4ef6\uff0c\u6240\u4ee5\u5982\u679c\u9700\u8981\u6267\u884c\u989d\u5916\u7684\u4efb\u52a1\uff0c\u9700\u8981\u5728cluster.yml\u7684\u672b\u5c3e\u6dfb\u52a0\u5b9a\u5236\u5316\u7684\u5904\u7406\uff0c\u521b\u5efa\u4e00\u4e2a\u540d\u4e3acustomized_cluster.yml\u7684\u6587\u4ef6\u3002<\/p>\n
                                            ---<\/span>\r\n# This role assumes to call Kubespray's `playbooks\/cluster`.<\/span>\r\n-<\/span> name<\/span>:<\/span> Install Kubernetes<\/span>\r\n  ansible.builtin.import_playbook<\/span>:<\/span> playbooks\/cluster.yml<\/span>\r\n\r\n-<\/span> name<\/span>:<\/span> Registernqualified<\/span>\r\n  hosts<\/span>:<\/span> all<\/span>\r\n  tasks<\/span>:<\/span>\r\n    -<\/span> ansible.builtin.file<\/span>:<\/span>\r\n        path<\/span>:<\/span> \/etc\/containers\/registries.conf.d<\/span>\r\n        state<\/span>:<\/span> directory<\/span>\r\n        mode<\/span>:<\/span> '<\/span>0755'<\/span>\r\n    -<\/span> ansible.builtin.copy<\/span>:<\/span>\r\n        dest<\/span>:<\/span> \/etc\/containers\/registries.conf.d\/01-unqualified.conf<\/span>\r\n        content<\/span>:<\/span> |<\/span>\r\n\r\n          unqualified-search-registries = ['docker.io', 'quay.io']<\/span>\r\n\r\n-<\/span> name<\/span>:<\/span> Download amind.conf to localhost<\/span>\r\n  hosts<\/span>:<\/span> kube_control_plane<\/span>\r\n  run_once<\/span>:<\/span> true<\/span>\r\n  tasks<\/span>:<\/span>\r\n    -<\/span> ansible.builtin.fetch<\/span>:<\/span>\r\n        src<\/span>:<\/span> \/etc\/kubernetes\/admin.conf<\/span>\r\n        dest<\/span>:<\/span> ~\/.kube\/admin.conf<\/span>\r\n        flat<\/span>:<\/span> yes<\/span>\r\n    -<\/span> delegate_to<\/span>:<\/span> localhost<\/span>\r\n      ansible.builtin.replace<\/span>:<\/span>\r\n        path<\/span>:<\/span> ~\/.kube\/admin.conf<\/span>\r\n        regexp<\/span>:<\/span> '<\/span>127.0.0.1'<\/span>\r\n        replace<\/span>:<\/span> \"<\/span>{{<\/span> hostvars[inventory_hostname]['ansible_default_ipv4']['address']<\/span> }}\"<\/span>\r\n<\/code><\/pre>\n
                                            \u5728\u4e0a\u9762\u7684\u4f8b\u5b50\u4e2d\uff0c\u4e0b\u4e00\u6b65\u7684\u4efb\u52a1\u662f\u521b\u5efa\u96c6\u7fa4\u3002<\/p>\n
                                              \n
                                            • \n
                                                Unqualified registries (docker.io \u3068 quay.io) \u306e\u8ffd\u52a0<\/ul>\n<\/li>\n<\/ul>\n
                                                 <\/p>\n
                                                  admin.conf \u3092\u30ed\u30fc\u30ab\u30eb\u306b\u30c0\u30a6\u30f3\u30ed\u30fc\u30c9\u3057\u3001API \u30b5\u30fc\u30d0\u306e\u30a2\u30c9\u30ec\u30b9\u3092 Public IP \u306b\u7f6e\u63db<\/ul>\n
                                                  \u5c06\u5176\u81ea\u52a8\u5316\u3002\u53ea\u9700\u5c06Kubespray\u7684cluster.yml\u66ff\u6362\u4e3a\u6b64\u6587\u4ef6\uff0c\u7136\u540e\u6309\u7167\u6b63\u5e38 playbook \u7684\u65b9\u5f0f\u6267\u884c\u5373\u53ef\u3002<\/p>\n
                                                  $ docker run --rm -it \\\r\n  --mount type=bind,source=\"$(pwd)\"\/inventory,dst=\/inventory \\\r\n  --mount type=bind,source=\"$(pwd)\"\/.terraform\/modules\/kubernetes\/kubernetes\/generate_inventory.py,dst=\/kubespray\/generate_inventory.py \\\r\n  --mount type=bind,source=\"$(pwd)\"\/terraform.tfstate,dst=\/kubespray\/terraform.tfstate \\\r\n  --mount type=bind,source=\"$(pwd)\"\/cluster.yml,dst=\/kubespray\/cluster.yml \\\r\n  --mount type=bind,source=\"${HOME}\"\/.kube,dst=\/root\/.kube \\\r\n  --mount type=bind,source=\"${HOME}\"\/.ssh\/id_rsa,dst=\/root\/.ssh\/id_rsa \\\r\n  quay.io\/kubespray\/kubespray:v2.23.1 bash\r\n\r\n# Inside a container\r\nansible-playbook -i generate_inventory.py cluster.yml\r\n<\/code><\/pre>\n
                                                  \u8bf7\u53c2\u8003 kubespray\/docs\/integration.md at master \u00b7 kubernetes-sigs\/kubespray \u00b7 GitHub \u4e2d\u7684\u516c\u5f0f\u6307\u5357\u3002<\/p>\n
                                                  \u5728\u79bb\u7ebf\u73af\u5883\u4e0b\u6267\u884c Terraform<\/h3>\n
                                                  \u901a\u5e38\u60c5\u51b5\u4e0b\uff0cProvider \u662f\u901a\u8fc7\u4e92\u8054\u7f51\u81ea\u52a8\u4e0b\u8f7d\u7684\uff0c\u4f46\u5728\u79bb\u7ebf\u73af\u5883\u4e0b\uff0c\u60a8\u53ef\u4ee5\u5728\u4e3b\u76ee\u5f55\u4e0b\u51c6\u5907\u4e00\u4e2a ~\/terraform.d\/providers \u6587\u4ef6\u5939\uff0c\u5e76\u5c06 Provider \u653e\u5728\u90a3\u91cc\u3002<\/p>\n
                                                  \u4ee5Terraform Libvirt Provider\u4e3a\u4f8b\uff0c\u4ecedmacvicar\/terraform-provider-libvirt\u7684Releases\u9875\u9762\u4e0b\u8f7d\u4e8c\u8fdb\u5236\u6587\u4ef6\uff0c\u5e76\u6309\u7167\u4ee5\u4e0b\u65b9\u5f0f\u8fdb\u884c\u914d\u7f6e\uff1a<\/p>\n
                                                  ~\/terraform.d\/providers\/\r\n\u2514\u2500\u2500 providers\/\r\n    \u2514\u2500\u2500 registry.terraform.io\/\r\n        \u2514\u2500\u2500 dmacvicar\/\r\n            \u2514\u2500\u2500 libvirt\/\r\n                \u2514\u2500\u2500 0.7.1\/\r\n                    \u2514\u2500\u2500 linux_amd64\/\r\n                        \u251c\u2500\u2500 CHANGELOG.md\r\n                        \u251c\u2500\u2500 LICENSE\r\n                        \u251c\u2500\u2500 README.md\r\n                        \u2514\u2500\u2500 terraform-provider-libvirt_v0.7.4\r\n<\/code><\/pre>\n
                                                  \u5728\u9060\u7a0b\u4e3b\u6a5f\u4e0a\u61c9\u7528 Terraform.<\/h3>\n
                                                  \u53ef\u4ee5\u4ece\u672c\u5730\u6267\u884cTerraform\uff0c\u5e76\u5728\u8fdc\u7a0b\u4e3b\u673a\u4e0a\u521b\u5efa\u865a\u62df\u673a\u3002<\/p>\n
                                                  \u9996\u5148\uff0c\u5728\u8fdc\u7a0b\u4e3b\u673a\u4e0a\u5b8c\u6210\u4e3b\u673a\u7684\u9884\u914d\u7f6e\u540e\uff0c\u786e\u8ba4libvirtd\u6b63\u5728\u8fd0\u884c\u3002<\/p>\n
                                                  # Remote side\r\n$ systemctl status libvirtd\r\n<\/code><\/pre>\n
                                                  \u901a\u8fc7\u4f7f\u7528 qemu+ssh:\/\/\u534f\u8bae\uff0c\u4ece\u5ba2\u6237\u7aef\u9a8c\u8bc1\u8fde\u63a5\uff1a<\/p>\n
                                                  # Client side\r\n$ virsh -c qemu+ssh:\/\/<user>@<remote_ip>\/system list\r\n Id   Name   State\r\n--------------------\r\n\r\n<\/code><\/pre>\n
                                                  \u786e\u8ba4\u540e\uff0c\u5728 main.tf \u6587\u4ef6\u4e2d\u6307\u5b9a\u4ee5\u4e0b\u7684\u8fdc\u7a0b\u4e3b\u673a\u4fe1\u606f\uff0c\u5e76\u5728\u5ba2\u6237\u7aef\u4e0a\u6267\u884c terraform apply\u3002<\/p>\n
                                                  locals<\/span> {<\/span>\r\n  user_home_directory<\/span> =<\/span> pathexpand<\/span>(<\/span>\"~\"<\/span>)<\/span>\r\n}<\/span>\r\n\r\nprovider<\/span> \"libvirt\"<\/span> {<\/span>\r\n  uri<\/span> =<\/span> \"qemu+ssh:\/\/<remote-user>@<remote_host>\/system?keyfile=${local.user_home_directory}\/.ssh\/id_rsa&known_hosts_verify=ignore\"<\/span>\r\n}<\/span>\r\n<\/code><\/pre>\n
                                                  \u5982\u679c\u901a\u8fc7\u8df3\u677f\u670d\u52a1\u5668\u7684\u8bdd<\/h4>\n
                                                  \u5f53\u901a\u8fc7\u8df3\u677f\u670d\u52a1\u5668\u8fdc\u7a0b\u8bbf\u95ee\u65f6\uff0c\u9700\u8981\u5c06\u8fdc\u7a0b\u768422\u7aef\u53e3\u8f6c\u53d1\u5230\u672c\u5730\u7684\u4e00\u4e2a\u9002\u5f53\u7aef\u53e3\uff08\u4f8b\u5982\uff1a50000\uff09\u3002<\/p>\n
                                                  # Client side\r\n$ ssh -C -N -f -L 50000:<remote-user>@<remote-host>:22 <bastion-host> -p <bastion-port>\r\n$ virsh -c qemu+ssh:\/\/<remote-user>@localhost:50000\/system list\r\n Id   Name   State\r\n--------------------\r\n\r\n<\/code><\/pre>\n
                                                  locals<\/span> {<\/span>\r\n  user_home_directory<\/span> =<\/span> pathexpand<\/span>(<\/span>\"~\"<\/span>)<\/span>\r\n}<\/span>\r\n\r\nprovider<\/span> \"libvirt\"<\/span> {<\/span>\r\n  uri<\/span> =<\/span> \"qemu+ssh:\/\/<remote-user>@localhost:50000\/system?keyfile=${local.user_home_directory}\/.ssh\/id_rsa&known_hosts_verify=ignore\"<\/span>\r\n}<\/span>\r\n<\/code><\/pre>\n
                                                  \u5c06\u8fd9\u7bc7\u6587\u7ae0\u4e2d\u7684 Terraform \u4ee3\u7801\u4f5c\u4e3a\u6a21\u5757\u4f7f\u7528<\/h3>\n
                                                  \u51c6\u5907\u4e00\u4e2a\u5982\u4e0b\u7684 main.tf \u6587\u4ef6\u3002\u5728\u8fd9\u4e2a\u4f8b\u5b50\u4e2d\uff0c\u5c06\u521b\u5efa\u4e00\u4e2a\u75311\u4e2a\u4e3b\u8282\u70b9\u548c2\u4e2a\u5de5\u4f5c\u8282\u70b9\u7ec4\u6210\u7684\u5171\u8ba13\u53f0\u865a\u62df\u673a\uff08VM\uff09\uff0c\u4f46\u8bf7\u6839\u636e\u6240\u9700\u7684\u914d\u7f6e\u9002\u5f53\u4fee\u6539\u53c2\u6570\u3002<\/p>\n
                                                  output<\/span> \"kubespray_hosts\"<\/span> {<\/span>\r\n  value<\/span> =<\/span> module<\/span>.<\/span>kubernetes<\/span>.<\/span>kubespray_hosts<\/span>\r\n}<\/span>\r\n\r\noutput<\/span> \"libvirt_uri\"<\/span> {<\/span>\r\n  value<\/span> =<\/span> module<\/span>.<\/span>kubernetes<\/span>.<\/span>libvirt_uri<\/span>\r\n}<\/span>\r\n\r\nlocals<\/span> {<\/span>\r\n  user_home_directory<\/span> =<\/span> pathexpand<\/span>(<\/span>\"~\"<\/span>)<\/span>\r\n}<\/span>\r\n\r\nmodule<\/span> \"kubernetes\"<\/span> {<\/span>\r\n  source<\/span> =<\/span> \"github.com\/sawa2d2\/k8s-on-kvm\/\/kubernetes\/\"<\/span>\r\n\r\n  ## Localhost:<\/span>\r\n  # libvirt_uri = \"qemu:\/\/\/system\"<\/span>\r\n  ## Remote:<\/span>\r\n  # libvirt_uri = \"qemu+ssh:\/\/<user>@<remote-host>\/system?keyfile=${local.user_home_directory}\/.ssh\/id_rsa&known_hosts_verify=ignore\"<\/span>\r\n  ## Remote via bastion:<\/span>\r\n  ##   Forward port in advance.<\/span>\r\n  ##   $ ssh -C -N -f -L 50000:<remote-user>@<remote-host>:22 <bastion-host> -p <bastion-port><\/span>\r\n  # libvirt_uri = \"qemu+ssh:\/\/<remote-user>@localhost:50000\/system?keyfile=${local.user_home_directory}\/.ssh\/id_rsa&known_hosts_verify=ignore\"<\/span>\r\n  libvirt_uri<\/span> =<\/span> \"qemu:\/\/\/system\"<\/span>\r\n\r\n  # Download the image by:<\/span>\r\n  #   sudo curl -L -o \/var\/lib\/libvirt\/images\/Rocky-9-GenericCloud.latest.x86_64.qcow2 https:\/\/download.rockylinux.org\/pub\/rocky\/9.2\/images\/x86_64\/Rocky-9-GenericCloud.latest.x86_64.qcow2 <\/span>\r\n  vm_base_image_uri<\/span> =<\/span> \"\/var\/lib\/libvirt\/images\/Rocky-9-GenericCloud.latest.x86_64.qcow2\"<\/span>\r\n  pool<\/span>              =<\/span> \"default\"<\/span>\r\n\r\n  # Cluster network<\/span>\r\n  bridge<\/span>      =<\/span> \"br0\"<\/span>\r\n  cidr<\/span>        =<\/span> \"192.168.8.0\/24\"<\/span>\r\n  gateway<\/span>     =<\/span> \"192.168.8.1\"<\/span>\r\n  nameservers<\/span> =<\/span> [<\/span>\"192.168.8.1\"<\/span>]<\/span>\r\n\r\n  vms<\/span> =<\/span> [<\/span>\r\n    {<\/span>\r\n      name<\/span>           =<\/span> \"k8s-master-1\"<\/span>\r\n      vcpu<\/span>           =<\/span> 4<\/span>\r\n      memory<\/span>         =<\/span> 16000<\/span>                    # in MiB<\/span>\r\n      disk<\/span>           =<\/span> 100<\/span> *<\/span> 1024<\/span> *<\/span> 1024<\/span> *<\/span> 1024<\/span> # 100 GB<\/span>\r\n      public_ip<\/span>      =<\/span> \"192.168.8.101\"<\/span>\r\n      private_ip<\/span>     =<\/span> \"192.168.122.201\"<\/span>\r\n      cloudinit_file<\/span> =<\/span> \"cloud_init.cfg\"<\/span>\r\n      volumes<\/span>        =<\/span> []<\/span>\r\n\r\n      kube_control_plane<\/span> =<\/span> true<\/span>\r\n      kube_node<\/span>          =<\/span> true<\/span>\r\n      etcd<\/span>               =<\/span> true<\/span>\r\n    },<\/span>\r\n    {<\/span>\r\n      name<\/span>           =<\/span> \"k8s-worker-1\"<\/span>\r\n      vcpu<\/span>           =<\/span> 4<\/span>\r\n      memory<\/span>         =<\/span> 16000<\/span>                    # in MiB<\/span>\r\n      disk<\/span>           =<\/span> 100<\/span> *<\/span> 1024<\/span> *<\/span> 1024<\/span> *<\/span> 1024<\/span> # 100 GB<\/span>\r\n      public_ip<\/span>      =<\/span> \"192.168.8.102\"<\/span>\r\n      private_ip<\/span>     =<\/span> \"192.168.122.202\"<\/span>\r\n      cloudinit_file<\/span> =<\/span> \"cloud_init.cfg\"<\/span>\r\n      volumes<\/span>        =<\/span> []<\/span>\r\n\r\n      kube_control_plane<\/span> =<\/span> false<\/span>\r\n      kube_node<\/span>          =<\/span> true<\/span>\r\n      etcd<\/span>               =<\/span> false<\/span>\r\n    },<\/span>\r\n    {<\/span>\r\n      name<\/span>           =<\/span> \"k8s-worker-2\"<\/span>\r\n      vcpu<\/span>           =<\/span> 2<\/span>\r\n      memory<\/span>         =<\/span> 8000<\/span>                     # in MiB<\/span>\r\n      disk<\/span>           =<\/span> 100<\/span> *<\/span> 1024<\/span> *<\/span> 1024<\/span> *<\/span> 1024<\/span> # 100 GB<\/span>\r\n      public_ip<\/span>      =<\/span> \"192.168.8.103\"<\/span>\r\n      private_ip<\/span>     =<\/span> \"192.168.122.203\"<\/span>\r\n      cloudinit_file<\/span> =<\/span> \"cloud_init.cfg\"<\/span>\r\n      volumes<\/span>        =<\/span> []<\/span>\r\n\r\n      kube_control_plane<\/span> =<\/span> false<\/span>\r\n      kube_node<\/span>          =<\/span> true<\/span>\r\n      etcd<\/span>               =<\/span> false<\/span>\r\n    },<\/span>\r\n  ]<\/span>\r\n}<\/span>\r\n<\/code><\/pre>\n
                                                  \u4ee5\u4e0b\u662f\u53c2\u8003\u8d44\u6599<\/h2>\n
                                                  Terraform \u548c Dynamic Inventory \u76f8\u5173<\/h3>\n
                                                    \n
                                                  • \n
                                                      End-to-End Application Provisioning with Ansible and Terraform – IBM Blog<\/ul>\n<\/li>\n<\/ul>\n
                                                       <\/p>\n
                                                        Integrating Ansible and Jenkins with Terraform to make a powerful infrastructure | by Ankush Chavan | Medium<\/ul>\n
                                                        libvirt\u7684\u914d\u7f6e\u76f8\u5173<\/h3>\n
                                                        error when default storage pool is missing \u00b7 Issue #8 \u00b7 simon3z\/virt-deploy … default pool \u306e\u4f5c\u6210\u65b9\u6cd5\u306a\u3069<\/p>\n
                                                        \u5173\u4e8eKuberspray\u7684\u76f8\u5173\u5185\u5bb9<\/h3>\n
                                                          \n
                                                        • \n
                                                            kubespray\u3092\u4f7f\u3063\u305fKubernetes\u306e\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb | Kubernetes<\/ul>\n<\/li>\n<\/ul>\n
                                                             <\/p>\n
                                                              An introduction to Kubespray | Enable Sysadmin<\/ul>\n
                                                              \u6865\u6881\u8bbe\u7f6e\u76f8\u5173<\/h3>\n
                                                                KVM\u3067\u30b2\u30b9\u30c8OS\u3092\u30d6\u30ea\u30c3\u30b8\u63a5\u7d9a\u3059\u308b – Qiita<\/ul>\n
                                                                \u76f8\u5173\u6587\u7ae0<\/h2>\n
                                                                 <\/p>\n","protected":false},"excerpt":{"rendered":"
                                                                \u592a\u957f\u4e0d\u8bfb b\u00f9 d\u00fa) \u5728\u5b9a\u4e49\u4e86\u7ec4\u6210\u4e4b\u540e\uff0c\u53ef\u4ee5\u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u521b\u5efa Kubernetes \u96c6\u7fa4\uff1a $ terraf […]<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-48849","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"\n\u4f7f\u7528Terraform + Kubespray\u5728KVM\u4e0a\u6784\u5efaKubernetes\u96c6\u7fa4 - Blog - Silicon Cloud<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.silicloud.com\/zh\/blog\/\u4f7f\u7528terraform-kubespray\u5728kvm\u4e0a\u6784\u5efakubernetes\u96c6\u7fa4\u3002\/\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"\u4f7f\u7528Terraform + Kubespray\u5728KVM\u4e0a\u6784\u5efaKubernetes\u96c6\u7fa4\" \/>\n<meta property=\"og:description\" content=\"\u592a\u957f\u4e0d\u8bfb b\u00f9 d\u00fa) \u5728\u5b9a\u4e49\u4e86\u7ec4\u6210\u4e4b\u540e\uff0c\u53ef\u4ee5\u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u521b\u5efa Kubernetes \u96c6\u7fa4\uff1a $ terraf […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.silicloud.com\/zh\/blog\/\u4f7f\u7528terraform-kubespray\u5728kvm\u4e0a\u6784\u5efakubernetes\u96c6\u7fa4\u3002\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog - Silicon Cloud\" \/>\n<meta property=\"article:published_time\" content=\"2023-11-06T15:33:36+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-04-29T13:14:31+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d87c2913a08637a6c0c09\/8-0.png\" \/>\n<meta name=\"author\" content=\"\u96c5, \u609f\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"\u96c5, \u609f\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"13 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bd%bf%e7%94%a8terraform-kubespray%e5%9c%a8kvm%e4%b8%8a%e6%9e%84%e5%bb%bakubernetes%e9%9b%86%e7%be%a4%e3%80%82\/\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bd%bf%e7%94%a8terraform-kubespray%e5%9c%a8kvm%e4%b8%8a%e6%9e%84%e5%bb%bakubernetes%e9%9b%86%e7%be%a4%e3%80%82\/\",\"name\":\"\u4f7f\u7528Terraform + Kubespray\u5728KVM\u4e0a\u6784\u5efaKubernetes\u96c6\u7fa4 - Blog - Silicon Cloud\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\"},\"datePublished\":\"2023-11-06T15:33:36+00:00\",\"dateModified\":\"2024-04-29T13:14:31+00:00\",\"author\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/f044a4b7fa4ee2701702942002419ca6\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bd%bf%e7%94%a8terraform-kubespray%e5%9c%a8kvm%e4%b8%8a%e6%9e%84%e5%bb%bakubernetes%e9%9b%86%e7%be%a4%e3%80%82\/#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bd%bf%e7%94%a8terraform-kubespray%e5%9c%a8kvm%e4%b8%8a%e6%9e%84%e5%bb%bakubernetes%e9%9b%86%e7%be%a4%e3%80%82\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bd%bf%e7%94%a8terraform-kubespray%e5%9c%a8kvm%e4%b8%8a%e6%9e%84%e5%bb%bakubernetes%e9%9b%86%e7%be%a4%e3%80%82\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.silicloud.com\/zh\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"\u4f7f\u7528Terraform + Kubespray\u5728KVM\u4e0a\u6784\u5efaKubernetes\u96c6\u7fa4\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#website\",\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/\",\"name\":\"Blog - Silicon Cloud\",\"description\":\"\",\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/f044a4b7fa4ee2701702942002419ca6\",\"name\":\"\u96c5, \u609f\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/e71a913e914f1aad1efc391f92084294bac54bc782acd289638580134cf667a6?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/e71a913e914f1aad1efc391f92084294bac54bc782acd289638580134cf667a6?s=96&d=mm&r=g\",\"caption\":\"\u96c5, \u609f\"},\"url\":\"https:\/\/www.silicloud.com\/zh\/blog\/author\/yawu\/\"},{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bd%bf%e7%94%a8terraform-kubespray%e5%9c%a8kvm%e4%b8%8a%e6%9e%84%e5%bb%bakubernetes%e9%9b%86%e7%be%a4%e3%80%82\/#local-main-organization-logo\",\"url\":\"\",\"contentUrl\":\"\",\"caption\":\"Blog - Silicon Cloud\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"\u4f7f\u7528Terraform + Kubespray\u5728KVM\u4e0a\u6784\u5efaKubernetes\u96c6\u7fa4 - Blog - Silicon Cloud","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.silicloud.com\/zh\/blog\/\u4f7f\u7528terraform-kubespray\u5728kvm\u4e0a\u6784\u5efakubernetes\u96c6\u7fa4\u3002\/","og_locale":"zh_CN","og_type":"article","og_title":"\u4f7f\u7528Terraform + Kubespray\u5728KVM\u4e0a\u6784\u5efaKubernetes\u96c6\u7fa4","og_description":"\u592a\u957f\u4e0d\u8bfb b\u00f9 d\u00fa) \u5728\u5b9a\u4e49\u4e86\u7ec4\u6210\u4e4b\u540e\uff0c\u53ef\u4ee5\u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u521b\u5efa Kubernetes \u96c6\u7fa4\uff1a $ terraf […]","og_url":"https:\/\/www.silicloud.com\/zh\/blog\/\u4f7f\u7528terraform-kubespray\u5728kvm\u4e0a\u6784\u5efakubernetes\u96c6\u7fa4\u3002\/","og_site_name":"Blog - Silicon Cloud","article_published_time":"2023-11-06T15:33:36+00:00","article_modified_time":"2024-04-29T13:14:31+00:00","og_image":[{"url":"https:\/\/cdn.silicloud.com\/blog-img\/blog\/img\/657d87c2913a08637a6c0c09\/8-0.png"}],"author":"\u96c5, \u609f","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"\u96c5, \u609f","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"13 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bd%bf%e7%94%a8terraform-kubespray%e5%9c%a8kvm%e4%b8%8a%e6%9e%84%e5%bb%bakubernetes%e9%9b%86%e7%be%a4%e3%80%82\/","url":"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bd%bf%e7%94%a8terraform-kubespray%e5%9c%a8kvm%e4%b8%8a%e6%9e%84%e5%bb%bakubernetes%e9%9b%86%e7%be%a4%e3%80%82\/","name":"\u4f7f\u7528Terraform + Kubespray\u5728KVM\u4e0a\u6784\u5efaKubernetes\u96c6\u7fa4 - Blog - Silicon Cloud","isPartOf":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website"},"datePublished":"2023-11-06T15:33:36+00:00","dateModified":"2024-04-29T13:14:31+00:00","author":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/f044a4b7fa4ee2701702942002419ca6"},"breadcrumb":{"@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bd%bf%e7%94%a8terraform-kubespray%e5%9c%a8kvm%e4%b8%8a%e6%9e%84%e5%bb%bakubernetes%e9%9b%86%e7%be%a4%e3%80%82\/#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.silicloud.com\/zh\/blog\/%e4%bd%bf%e7%94%a8terraform-kubespray%e5%9c%a8kvm%e4%b8%8a%e6%9e%84%e5%bb%bakubernetes%e9%9b%86%e7%be%a4%e3%80%82\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bd%bf%e7%94%a8terraform-kubespray%e5%9c%a8kvm%e4%b8%8a%e6%9e%84%e5%bb%bakubernetes%e9%9b%86%e7%be%a4%e3%80%82\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.silicloud.com\/zh\/blog\/"},{"@type":"ListItem","position":2,"name":"\u4f7f\u7528Terraform + Kubespray\u5728KVM\u4e0a\u6784\u5efaKubernetes\u96c6\u7fa4"}]},{"@type":"WebSite","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#website","url":"https:\/\/www.silicloud.com\/zh\/blog\/","name":"Blog - Silicon Cloud","description":"","inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/f044a4b7fa4ee2701702942002419ca6","name":"\u96c5, \u609f","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/e71a913e914f1aad1efc391f92084294bac54bc782acd289638580134cf667a6?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e71a913e914f1aad1efc391f92084294bac54bc782acd289638580134cf667a6?s=96&d=mm&r=g","caption":"\u96c5, \u609f"},"url":"https:\/\/www.silicloud.com\/zh\/blog\/author\/yawu\/"},{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.silicloud.com\/zh\/blog\/%e4%bd%bf%e7%94%a8terraform-kubespray%e5%9c%a8kvm%e4%b8%8a%e6%9e%84%e5%bb%bakubernetes%e9%9b%86%e7%be%a4%e3%80%82\/#local-main-organization-logo","url":"","contentUrl":"","caption":"Blog - Silicon Cloud"}]}},"_links":{"self":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/48849","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/comments?post=48849"}],"version-history":[{"count":2,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/48849\/revisions"}],"predecessor-version":[{"id":87678,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/posts\/48849\/revisions\/87678"}],"wp:attachment":[{"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/media?parent=48849"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/categories?post=48849"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.silicloud.com\/zh\/blog\/wp-json\/wp\/v2\/tags?post=48849"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}