我尝试使用 Pulumi 在 Kubernetes 上构建应用程序
首先
最近，我开始尝试使用 Pulumi 作为 IaC 工具。虽然有类似的 Terraform 工具，但我喜欢 Pulumi 的 GPL 许可使得表达能力更强，学习成本较低，可以利用使用语言的生态系统。尽管提供的 Provider 数量还不如 Terraform 多，但主要的功能基本都覆盖了。
因此，这次我们将尝试使用 Pulumi 和 Kubernetes Provider 在 K8s 上（使用 OKE）构建一个简单的应用程序。
整体的形象
我会尝试创建以下的构成。
这是一个通过Ingress公开Java应用程序的简单示例。它将相同的应用程序部署到不同的Namespace（dev/prod），但通过环境变量来改变应用程序返回的消息。
步骤
首先，我们要创建一个项目。
pulumi new kubernetes-typescript \
--name k8s-helidon-app \
--stack dev \
--description "A TypeScript Pulumi program for simple kubernetes application."
运行后，将会创建以下文件群。
.
├── Pulumi.yaml
├── index.ts
├── node_modules
├── package-lock.json
├── package.json
└── tsconfig.json
生成された index.ts を確認してみると以下のようになっています。
import * as k8s from "@pulumi/kubernetes";
import * as kx from "@pulumi/kubernetesx";
const appLabels = { app: "nginx" };
const deployment = new k8s.apps.v1.Deployment("nginx", {
spec: {
selector: { matchLabels: appLabels },
replicas: 1,
template: {
metadata: { labels: appLabels },
spec: { containers: [{ name: "nginx", image: "nginx" }] },
},
},
});
export const name = deployment.metadata.name;
如果您有编写过Kubernetes清单文件的经验，我相信您可以毫不犹豫地表达想部署的资源。
ということで、先の構成を実現するべく以下のように実装してみました。以降で簡単に解説します。
import * as pulumi from "@pulumi/pulumi";
import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
import * as kx from "@pulumi/kubernetesx";
interface Data {
namespace: string;
deployment: {
replicas: number;
};
ingress: {
host: string;
tls: {
hosts: string[];
secretName: string;
};
};
cowweb: {
message: string;
};
}
let config = new pulumi.Config();
const data = config.requireObject<Data>("data");
const appName = "k8s-helidon-app";
const appLabels = { app: appName };
function defaultProbeGenerator(path: string) {
return {
httpGet: {
path: path,
port: "api",
},
initialDelaySeconds: 30,
periodSeconds: 5,
};
}
const deployment = new k8s.apps.v1.Deployment(appName, {
kind: "Deployment",
apiVersion: "apps/v1",
metadata: {
name: appName,
namespace: data.namespace,
},
spec: {
replicas: data.deployment.replicas,
selector: { matchLabels: appLabels },
template: {
metadata: { labels: appLabels },
spec: {
containers: [
{
name: appName,
image: `ghcr.io/shukawam/${appName}:latest`,
imagePullPolicy: "IfNotPresent",
ports: [{ name: "api", containerPort: 8080 }],
readinessProbe: defaultProbeGenerator("/health/ready"),
livenessProbe: defaultProbeGenerator("/health/live"),
env: [{ name: "cowweb.message", value: data.cowweb.message }],
},
],
imagePullSecrets: [{ name: "ghcr-secret" }],
},
},
},
});
const service = new k8s.core.v1.Service(appName, {
kind: "Service",
apiVersion: "v1",
metadata: {
name: appName,
namespace: data.namespace,
labels: {
app: appName,
"prometheus.io/scrape": "true",
},
},
spec: {
type: "ClusterIP",
selector: appLabels,
ports: [{ port: 8080, targetPort: 8080, name: "http" }],
},
});
const ingress = new k8s.networking.v1.Ingress(appName, {
kind: "Ingress",
apiVersion: "networking.k8s.io/v1",
metadata: {
name: appName,
namespace: data.namespace,
annotations: {
"kubernetes.io/ingress.class": "nginx",
"cert-manager.io/cluster-issuer": "letsencrypt-prod",
},
},
spec: {
tls: [
{
hosts: data.ingress.tls.hosts,
secretName: data.ingress.tls.secretName,
},
],
rules: [
{
host: data.ingress.tls.hosts[0],
http: {
paths: [
{
backend: {
service: {
name: appName,
port: {
number: 8080,
},
},
},
pathType: "Prefix",
path: "/",
},
],
},
},
],
},
});
export const deploymentName = deployment.metadata.name;
export const serviceName = service.metadata.name;
export const ingressName = ingress.metadata.name;
首先，是这个部分。
interface Data {
namespace: string;
deployment: {
replicas: number;
};
ingress: {
host: string;
tls: {
hosts: string[];
secretName: string;
};
};
cowweb: {
message: string;
};
}
const config = new pulumi.Config();
const data = config.requireObject<Data>("data");
将可能在每个环境中更改的部分作为 Pulumi 的配置进行处理。配置文件被放置在项目的根目录下，命名为Pulumi..yaml。在这个例子中，内容如下：
config:
k8s-helidon-app:data:
namespace: dev
deployment:
replicas: 1
ingress:
host: helidon.dev.shukawam.me
tls:
hosts:
- helidon.dev.shukawam.me
secretName: shukawam-tls-secret
cowweb:
message: Dev!
config:
k8s-helidon-app:data:
namespace: prod
deployment:
replicas: 3
ingress:
host: helidon.prod.shukawam.me
tls:
hosts:
- helidon.prod.shukawam.me
secretName: shukawam-tls-secret
cowweb:
message: Prod!
後は、これを TypeScript として扱うための interface 定義と専用の API (e.g. new pulumi.Config()requireObject(“data”))を通して扱います。
ちなみに、Pulumi ではこの独立した単位を Stack として扱っています。今回は、最初に dev の Stack を作成しましたが、新しく prod の Stack を追加する際には以下のように実行します。
pulumi stack init prod
確認してみると、確かに Stack が 2 つ存在することが分かります。
pulumi stack ls
NAME LAST UPDATE RESOURCE COUNT URL
dev 1 hour ago 5 https://app.pulumi.com/shukawam/k8s-helidon-app/dev
prod* 1 hour ago 5 https://app.pulumi.com/shukawam/k8s-helidon-app/prod
接下来，就是这一部分了。
function defaultProbeGenerator(path: string) {
return {
httpGet: {
path: path,
port: "api",
},
initialDelaySeconds: 30,
periodSeconds: 5,
};
}
const deployment = new k8s.apps.v1.Deployment(appName, {
kind: "Deployment",
apiVersion: "apps/v1",
metadata: {
name: appName,
namespace: data.namespace,
},
spec: {
replicas: data.deployment.replicas,
selector: { matchLabels: appLabels },
template: {
metadata: { labels: appLabels },
spec: {
containers: [
{
name: appName,
image: `ghcr.io/shukawam/${appName}:latest`,
imagePullPolicy: "IfNotPresent",
ports: [{ name: "api", containerPort: 8080 }],
readinessProbe: defaultProbeGenerator("/health/ready"),
livenessProbe: defaultProbeGenerator("/health/live"),
env: [{ name: "cowweb.message", value: data.cowweb.message }],
},
],
imagePullSecrets: [{ name: "ghcr-secret" }],
},
},
},
});
Kubernetes的部署定义。基本上与YAML表达式没有太大差别，但共享了readinessProbe和livenessProbe等周边工具，并使用了模板文字（ghcr.io/shukawam/${appName}:latest）来实现高效且简单的实施，这正是GPL的优势所在。根据环境的不同，相关差异通过前面提到的配置来处理，Deployment定义方面涉及.metadata.namespace、.spec.replicas和.spec.template.spec.containers[0].env[0]等内容。
由于上述描述与Kubernetes – Service/Ingress部分重复较多，故略去该部分的介绍。
实际上，我们将进行配置。首先，我们会预览所创建的基础架构配置。（类似于Terraform中的terraform plan。）
pulumi preview
Previewing update (dev)
View in Browser (Ctrl+O): https://app.pulumi.com/shukawam/k8s-helidon-app/dev/previews/67c4080b-ee7f-46c1-b681-c895875f9bcc
Type Name Plan
+ pulumi:pulumi:Stack k8s-helidon-app-dev create
+ ├─ kubernetes:networking.k8s.io/v1:Ingress k8s-helidon-app create
+ ├─ kubernetes:apps/v1:Deployment k8s-helidon-app create
+ └─ kubernetes:core/v1:Service k8s-helidon-app create
Outputs:
deploymentName: "k8s-helidon-app"
ingressName : "k8s-helidon-app"
serviceName : "k8s-helidon-app"
Resources:
+ 4 to create
devの Stack に紐づき Ingress, Deployment, Service が新たに追加されることが確認できましたので、実際に作成してみます。（Terraform でいうところの terraform apply 相当）
pulumi up
Previewing update (dev)
View in Browser (Ctrl+O): https://app.pulumi.com/shukawam/k8s-helidon-app/dev/previews/667c0f36-78dc-4d51-8d15-17edf4ff7607
Type Name Plan
+ pulumi:pulumi:Stack k8s-helidon-app-dev create
+ ├─ kubernetes:networking.k8s.io/v1:Ingress k8s-helidon-app create
+ ├─ kubernetes:apps/v1:Deployment k8s-helidon-app create
+ └─ kubernetes:core/v1:Service k8s-helidon-app create
Outputs:
deploymentName: "k8s-helidon-app"
ingressName : "k8s-helidon-app"
serviceName : "k8s-helidon-app"
Resources:
+ 4 to create
Do you want to perform this update? yes
Updating (dev)
View in Browser (Ctrl+O): https://app.pulumi.com/shukawam/k8s-helidon-app/dev/updates/3
Type Name Status
+ pulumi:pulumi:Stack k8s-helidon-app-dev created (45s)
+ ├─ kubernetes:networking.k8s.io/v1:Ingress k8s-helidon-app created (42s)
+ ├─ kubernetes:apps/v1:Deployment k8s-helidon-app created (36s)
+ └─ kubernetes:core/v1:Service k8s-helidon-app created (11s)
Outputs:
deploymentName: "k8s-helidon-app"
ingressName : "k8s-helidon-app"
serviceName : "k8s-helidon-app"
Resources:
+ 4 created
Duration: 47s
我会试着确认一下。
kubectl --namespace dev get deployment,service,ingress
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/k8s-helidon-app 1/1 1 1 6m52s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/k8s-helidon-app ClusterIP 10.96.244.218 <none> 8080/TCP 6m52s
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/k8s-helidon-app <none> helidon.dev.shukawam.me 129.153.126.126 80, 443 6m53s
对于prod的Stack，同样进行操作。首先，切换Stack。
pulumi stack select prod
预览即将创建的环境。
pulumi preview
Previewing update (prod)
View in Browser (Ctrl+O): https://app.pulumi.com/shukawam/k8s-helidon-app/prod/previews/bd788219-4142-4e78-b316-a77ec0ae0a6f
Type Name Plan
+ pulumi:pulumi:Stack k8s-helidon-app-prod create
+ ├─ kubernetes:apps/v1:Deployment k8s-helidon-app create
+ ├─ kubernetes:networking.k8s.io/v1:Ingress k8s-helidon-app create
+ └─ kubernetes:core/v1:Service k8s-helidon-app create
Outputs:
deploymentName: "k8s-helidon-app"
ingressName : "k8s-helidon-app"
serviceName : "k8s-helidon-app"
Resources:
+ 4 to create
作成します。
pulumi up
Previewing update (prod)
View in Browser (Ctrl+O): https://app.pulumi.com/shukawam/k8s-helidon-app/prod/previews/9a3b89ab-e49d-42a5-bd16-9d3d5224c410
Type Name Plan
+ pulumi:pulumi:Stack k8s-helidon-app-prod create
+ ├─ kubernetes:apps/v1:Deployment k8s-helidon-app create
+ ├─ kubernetes:networking.k8s.io/v1:Ingress k8s-helidon-app create
+ └─ kubernetes:core/v1:Service k8s-helidon-app create
Outputs:
deploymentName: "k8s-helidon-app"
ingressName : "k8s-helidon-app"
serviceName : "k8s-helidon-app"
Resources:
+ 4 to create
Do you want to perform this update? yes
Updating (prod)
View in Browser (Ctrl+O): https://app.pulumi.com/shukawam/k8s-helidon-app/prod/updates/14
Type Name Status
+ pulumi:pulumi:Stack k8s-helidon-app-prod created (39s)
+ ├─ kubernetes:apps/v1:Deployment k8s-helidon-app created (36s)
+ ├─ kubernetes:networking.k8s.io/v1:Ingress k8s-helidon-app created (7s)
+ └─ kubernetes:core/v1:Service k8s-helidon-app created (11s)
Outputs:
deploymentName: "k8s-helidon-app"
ingressName : "k8s-helidon-app"
serviceName : "k8s-helidon-app"
Resources:
+ 4 created
Duration: 41s
我会尝试确认一下。
kubectl --namespace prod get deployment,service,ingress
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/k8s-helidon-app 3/3 3 3 82s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/k8s-helidon-app ClusterIP 10.96.26.215 <none> 8080/TCP 83s
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/k8s-helidon-app <none> helidon.prod.shukawam.me 129.153.126.126 80, 443 83s
当比较dev和prod的创建结果时，我们可以观察到副本数量的差异（dev:1，prod:3）以及Ingress的主机名不同。
当我实际发送请求并进行确认后，我也注意到了响应结果的不同。
# Stack: dev
curl https://helidon.dev.shukawam.me/cowsay/say
______
< Dev! >
------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
# Stack: prod
curl https://helidon.prod.shukawam.me/cowsay/say
_______
< Prod! >
-------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
最后
这次我使用 Pulumi 和 Kubernetes Provider，在 K8s 上（使用了 OKE）构建了一个简单的应用程序，但是我在实现的过程中一直在思考，感觉有点像 Kustomize，还有我对于如何处理自定义资源的方式仍然有些困惑，也没有充分利用 GPL 生态系统（测试库、IDE 等），所以我还需要进一步研究。
请参阅相关信息
请参考我使用的代码。
