尝试在Openshift上添加NGINX Ingress Controller

2 年 ago
文, 翔
7 minutes

为了什么

除了OpenShift提供的默认路由器之外，我还想使用NGINX Ingress Controller。

环境

OpenShift 4.6：开放变革4.6版

    • Openshift自体はInternet 接続なし

 

    • bastion nodeのみInternet接続可能

 

    • IBM CloudのVirtual Private Cloudを利用

 

    コンテナImageを管理するプライベートレジストリにIBM Cloud Container Registryを利用

参考资料

    • IBM Cloud > Virtual Private Cloud について

 

    • NGINX Ingress Controller

 

    • OpenShift > Configuring Ingress Controller sharding by using route labels

 

    DNS サブドメインへの VPC ロード・バランサーのホスト名の登録

将NGINX Ingress Controller部署到Openshift

使用Helm v3将NGINX Ingress Controller部署到Openshift中。

准备一个用于helm命令的bastion节点。

[user01@bastion-node ~]$ curl -o helm.tar.gz -k https://get.helm.sh/helm-v3.4.2-linux-amd64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12.7M  100 12.7M    0     0  7776k      0  0:00:01  0:00:01 --:--:-- 7778k
[user01@bastion-node ~]$ tar xf helm.tar.gz
[user01@bastion-node ~]$ mv linux-amd64/helm ~/.local/bin/.
[user01@bastion-node ~]$ PATH=$PATH:~/.local/bin
[user01@bastion-node ~]$ helm version
version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"clean", GoVersion:"go1.14.13"}
[user01@bastion-node ~]$

下载 ingress-nginx chart

[user01@bastion-node ~]$ helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
"ingress-nginx" has been added to your repositories
[user01@bastion-node ~]$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ingress-nginx" chart repository
...Successfully got an update from the "ibm-helm" chart repository
Update Complete. ⎈Happy Helming!⎈
[user01@bastion-node ~]$
[user01@bastion-node ~]$ helm repo list
NAME            URL
ibm-helm        https://raw.githubusercontent.com/IBM/charts/master/repo/ibm-helm
ingress-nginx   https://kubernetes.github.io/ingress-nginx
[user01@bastion-node ~]$
[user01@bastion-node ~]$ helm search repo ingress-nginx/ingress-nginx
WARNING: Repo "ibm-helm" is corrupt or missing. Try 'helm repo update'.
NAME                            CHART VERSION   APP VERSION     DESCRIPTION
ingress-nginx/ingress-nginx     3.15.2          0.41.2          Ingress controller for Kubernetes using NGINX a...
[user01@bastion-node ~]$
[user01@bastion-node ~]$ mkdir ~/work/ingress-nginx
[user01@bastion-node ~]$ cd ~/work/ingress-nginx
[user01@bastion-node ingress-nginx]$ helm pull ingress-nginx/ingress-nginx
[user01@bastion-node ingress-nginx]$ tar xf ingress-nginx-3.15.2.tgz
[user01@bastion-node ingress-nginx]$ ls
ingress-nginx  ingress-nginx-3.15.2.tgz
[user01@bastion-node ingress-nginx]$

3. 在私有仓库中准备必要的镜像

    helmチャート内のファイルからデプロイに使うimageを確認。
[user01@bastion-node ingress-nginx]$ grep image: ingress-nginx/values.yaml -A2
  image:
    repository: k8s.gcr.io/ingress-nginx/controller
    tag: "v0.41.2"
--
  #    image: nginx:latest
  #  - name: lemonldap-ng-controller
  #    image: lemonldapng/lemonldap-ng-controller:0.2.0
  #    args:
  #      - /lemonldap-ng-controller
--
  #   image: busybox
  #   command: ['sh', '-c', 'until nslookup myservice; do echo waiting for myservice; sleep 2; done;']

--
      image:
        repository: docker.io/jettech/kube-webhook-certgen
        tag: v1.5.0
--
  image:
    repository: k8s.gcr.io/defaultbackend-amd64
    tag: "1.5"
[user01@bastion-node ingress-nginx]$

以下是目标镜像：
– k8s.gcr.io/ingress-nginx/controller:v0.41.2
– docker.io/jettech/kube-webhook-certgen:v1.5.0
– k8s.gcr.io/defaultbackend-amd64:1.5

    bastion nodeでimageをpullして、tagを付与し、プライベートレジストリにpush
docker pull k8s.gcr.io/defaultbackend-amd64:1.5
docker tag k8s.gcr.io/defaultbackend-amd64:1.5 jp.icr.io/test/defaultbackend-amd64:1.5
docker push jp.icr.io/test/defaultbackend-amd64:1.5

照着上述的步骤，将其他图片推送到私有镜像仓库中。

    • クラスターがプライベートレジストリにアクセスできるようにPullシークレットをOpenshiftに追加

 

    ※pull secretの中身（user id, password）を用意する手順は省略
[user01@bastion-node ingress-nginx]$ oc -n user01-test create secret docker-registry containerregistry-test --docker-server=jp.icr.io --docker-username=iamapikey --docker-password=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX --docker-email=e34559@jp.ibm.com
secret/containerregistry-test created
[user01@bastion-node ingress-nginx]$

更新helm chart文件。

因为没有互联网连接，所以需要更改部分图片等内容。

ingress-nginx/values.yaml 的释义

以下是对注释部分进行的本地化汉语解释，仅需一个选项：
这些说明是为了在IBM Cloud上创建NGINX Ingress Controller前端负载均衡器。

[user01@bastion-node ingress-nginx]$ cp ingress-nginx/values.yaml ingress-nginx/values.yaml.org
[user01@bastion-node ingress-nginx]$ vi ingress-nginx/values.yaml
[user01@bastion-node ingress-nginx]$ diff ingress-nginx/values.yaml ingress-nginx/values.yaml.org
13c13
<     repository: jp.icr.io/test/ingress-nginx/controller
---
>     repository: k8s.gcr.io/ingress-nginx/controller
15c15
<     #digest: sha256:1f4f402b9c14f3ae92b11ada1dfe9893a88f0faeb0b2f4b903e2c67a0c3bf0de
---
>     digest: sha256:1f4f402b9c14f3ae92b11ada1dfe9893a88f0faeb0b2f4b903e2c67a0c3bf0de
269a270
>
352,353c353
<     annotations:
<       service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type: private
---
>     annotations: {}
481c481
<         repository: jp.icr.io/test/jettech/kube-webhook-certgen
---
>         repository: docker.io/jettech/kube-webhook-certgen
490,491c490
<       #runAsUser: 2000
<       runAsUser: 1000622000
---
>       runAsUser: 2000
597c596
<     repository: jp.icr.io/test/defaultbackend-amd64
---
>     repository: k8s.gcr.io/defaultbackend-amd64
714,715c713
< imagePullSecrets:
< - name: containerregistry-test
---
> imagePullSecrets: []
[user01@bastion-node ingress-nginx]$

ingress-nginx/templates/controller-role.yaml –> 策划控制器角色.yaml

以下继续补充信息。

[user01@bastion-node ingress-nginx]$ vi ingress-nginx/templates/controller-role.yaml
[user01@bastion-node ingress-nginx]$ tail -n 10 ingress-nginx/templates/controller-role.yaml
{{- end }}
  - apiGroups:
    - security.openshift.io
    resourceNames:
    - privileged
    resources:
    - securitycontextconstraints
    verbs:
    - use

[user01@bastion-node ingress-nginx]$

请勿将原文件复制并放置在同一目录中，因为所有以下模板的文件将会被加载。

5. 执行 NGINX Ingress Controller 的部署

    ocコマンドでopenshiftにログインし、NGINX Ingress Controllerを導入するproject(namespace)に切り替える
oc login -u user01
oc project user01-test
    デプロイ実行
[user01@bastion-node ingress-nginx]$ helm install ingress-nginx-test ./ingress-nginx
NAME: ingress-nginx-test
LAST DEPLOYED: Mon Dec 21 16:53:25 2020
NAMESPACE: user01-test
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace user01-test get services -o wide -w ingress-nginx-test-controller'

An example Ingress that makes use of the controller:

  apiVersion: networking.k8s.io/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls
[user01@bastion-node ingress-nginx]$
[user01@bastion-node ingress-nginx]$ oc get all
NAME                                                READY   STATUS    RESTARTS   AGE
pod/ingress-nginx-test-controller-cfc699764-lpm5t   1/1     Running   0          19m

NAME                                              TYPE           CLUSTER-IP      EXTERNAL-IP                          PORT(S)                      AGE
service/ingress-nginx-test-controller             LoadBalancer   172.21.22.87    d1b930bf-jp-tok.lb.appdomain.cloud   80:31140/TCP,443:32343/TCP   19m
service/ingress-nginx-test-controller-admission   ClusterIP      172.21.199.70   <none>                               443/TCP                      19m

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-test-controller   1/1     1            1           19m

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-test-controller-cfc699764   1         1         1       19m
[user01@bastion-node ingress-nginx]$

注册VPC负载均衡器主机名到DNS子域名

[user01@bastion-node ~]$ ibmcloud ks nlb-dns create vpc-gen2 --cluster tmpcluster --lb-host d1b930bf-jp-tok.lb.appdomain.cloud --type private
Creating NLB DNS...
OK
NLB hostname was created as test-cluster-xx1.jp-tok.containers.appdomain.cloud
[user01@bastion-node ~]$
[user01@bastion-node ~]$ ibmcloud ks nlb-dns ls --cluster tmpcluster
OK
Subdomain                                                                            Load Balancer Hostname               SSL Cert Status   SSL Cert Secret Name                               Secret Namespace
test-cluster-xx0.jp-tok.containers.appdomain.cloud   818aee49-jp-tok.lb.appdomain.cloud   created           test-cluster-xx0   openshift-ingress
test-cluster-xx1.jp-tok.containers.appdomain.cloud   d1b930bf-jp-tok.lb.appdomain.cloud   created           test-cluster-xx1   default
[user01@bastion-node ~]$

设置分片

如果继续这样操作，使用NGINX Ingress Controller来公开服务时，将会对默认的路由器进行更改。（虽然问题不太大。）为了不对默认的路由器进行更改，需要进行附加设置。

    ingress controllerに設定を追加
oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"routeSelector":{"matchExpressions":[{"key":"nginxingress", "operator":"DoesNotExist"}]}}}' --type=merge

测试 NGINX Ingress 控制器

    テスト用イメージ用意
docker pull openshift/hello-openshift:latest
docker tag openshift/hello-openshift:latest jp.icr.io/test/test/openshift/hello-openshift:latest
docker push jp.icr.io/test/test/openshift/hello-openshift:latest
    テスト用podをデプロイ
oc new-app --docker-image=jp.icr.io/test/test/openshift/hello-openshift:latest --name user01-test -o yaml > test_deploy.yaml
vi test_deploy.yaml
# 以下の様にpull secretや envを追記
---
      spec:
        imagePullSecrets:
        - name: containerregistry-test
        containers:
        - image: ' '
          name: user01-test
          ports:
          - containerPort: 8080
            protocol: TCP
          - containerPort: 8888
            protocol: TCP
          resources: {}
          env:
          - name: RESPONSE
            value: Hello World!
~~
---

oc create -f test_deploy.yaml
    サービスをingressで公開
[user01@bastion-node ingress-nginx]$ cat test_ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-myservicea
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: user01-test.test-cluster-xx1.jp-tok.containers.appdomain.cloud
    http:
      paths:
      - path: /
        backend:
          serviceName: user01-test
          servicePort: 8080
[user01@bastion-node ingress-nginx]$
[user01@bastion-node ingress-nginx]$ oc create -f test_ingress.yaml
ingress.networking.k8s.io/ingress-user01-test created
[user01@bastion-node ingress-nginx]$
[user01@bastion-node ingress-nginx]$ oc get ingress
NAME                  CLASS    HOSTS                                                                                                        ADDRESS                              PORTS   AGE
ingress-user01-test   <none>   user01-test.test-cluster-xx1.jp-tok.containers.appdomain.cloud   d1b930bf-jp-tok.lb.appdomain.cloud   80      46s
[user01@bastion-node ingress-nginx]$
    接続確認
[user01@bastion-node ingress-nginx]$ curl http://user01-test.test-cluster-xx1.jp-tok.containers.appdomain.cloud
Hello World!
[user01@bastion-node ingress-nginx]$

以上

bannerAds