将GitLab Helm Chart安装到Azure AKS上
自 GitLab v11.0.0 起，Cloud Native GitLab Helm Chart 已改名为 GitLab Helm Chart，并发布了 Beta 版本。
- https://docs.gitlab.com/ee/install/kubernetes/gitlab_chart.html
由于AKS没有官方的安装流程，所以我尽力自己创建了一个。
预先准备
我在使用Mac OS进行工作。您需要事先准备以下内容。
-
- kubectl
-
- Helm
- Azure CLI
构建AKS集群
由于只使用CLI无法成功，因此我会在Azure Web Portal上创建。
如果你使用命令行界面来创建，以下这些命令肯定会很有帮助：
-
- az account list-locations
-
- az group create –name gitlab-chart-group –location eastus
-
- az aks create -g gitlab-chart-group -n bakeneco -l eastus –enable-rbac –node-count 3 –generate-ssh-keys -k 1.9.6
- az aks delete -g gitlab-chart-group -n bakeneco
接下来，使用 Azure CLI 进行登录，并连接到集群。
> az login
Note, we have launched a browser for you to login. For old experience with device code, use "az login --use-device-code"
You have logged in. Now let us find all subscriptions you have access to...
[
{
"cloudName": "AzureCloud",
"id": "nyan",
"isDefault": true,
"name": "Microsoft Azure \u30a8\u30f3\u30bf\u30fc\u30d7\u30e9\u30a4\u30ba",
"state": "Enabled",
"tenantId": "nyan",
"user": {
"name": "nyan@nyan.com",
"type": "user"
}
}
]
> az aks get-credentials --resource-group gitlab-chart-group --name bakeneco
Merged "bakeneco" as current context in /Users/jb/.kube/config
> kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-agentpool-15572265-0 Ready agent 13m v1.9.6
aks-agentpool-15572265-1 Ready agent 13m v1.9.6
> kubectl get services --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 13m
kube-system addon-http-application-routing-default-http-backend ClusterIP 10.0.169.174 <none> 80/TCP 12m
kube-system addon-http-application-routing-nginx-ingress LoadBalancer 10.0.156.43 168.62.41.195 80:31997/TCP,443:31037/TCP 12m
kube-system heapster ClusterIP 10.0.76.188 <none> 80/TCP 12m
kube-system kube-dns ClusterIP 10.0.0.10 <none> 53/UDP,53/TCP 12m
kube-system kubernetes-dashboard ClusterIP 10.0.31.110 <none> 80/TCP 12m
> kubectl get deployments --all-namespaces
NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kube-system addon-http-application-routing-default-http-backend 1 1 1 1 13m
kube-system addon-http-application-routing-external-dns 1 1 1 1 13m
kube-system addon-http-application-routing-nginx-ingress-controller 1 1 1 1 13m
kube-system azureproxy 1 1 1 1 13m
kube-system heapster 1 1 1 1 13m
kube-system kube-dns-v20 2 2 2 2 13m
kube-system kubernetes-dashboard 1 1 1 1 13m
kube-system tunnelfront 1 1 1 1 13m
> kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system addon-http-application-routing-default-http-backend-66c97fvmg46 1/1 Running 0 13m
kube-system addon-http-application-routing-external-dns-6c65ddc45d-nlf4k 1/1 Running 0 13m
kube-system addon-http-application-routing-nginx-ingress-controller-64hzwq8 1/1 Running 0 13m
kube-system azureproxy-7c677567f6-hlgdl 1/1 Running 0 13m
kube-system heapster-55f855b47-pqf57 2/2 Running 0 5m
kube-system kube-dns-v20-7c556f89c5-mgm9w 3/3 Running 0 13m
kube-system kube-dns-v20-7c556f89c5-xkwrm 3/3 Running 0 13m
kube-system kube-proxy-6pjzz 1/1 Running 0 13m
kube-system kube-proxy-fzhc6 1/1 Running 0 13m
kube-system kube-svc-redirect-4r7jm 1/1 Running 2 13m
kube-system kube-svc-redirect-wdcbs 1/1 Running 1 13m
kube-system kubernetes-dashboard-b85c46fc-6sznr 1/1 Running 5 13m
kube-system tunnelfront-7b685bd4c5-fkgwh 1/1 Running 0 13m
集群环境的配置
网络设置
由于一开始就有外部IP，因此需要注册DNS记录。
安装时，链接附加的HTTP应用程序路由-nginx入口提供的外部IP与另外一个内容被准备好，所以在安装执行之后需要确认并进行DNS注册。
存储设置
在Azure中，提供了默认的StorageClass选项。这次我们将直接使用它。
耕耘机
有些集群需要在使用kubectl创建Tiller角色之前进行身份验证。
创建一个适用于Tiller的ClusterRole和ServiceAccount，并授予cluster-admin权限。
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: null
name: cluster-admin
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'
> kubectl create -f cluster-admin-role.yaml
clusterrole.rbac.authorization.k8s.io "cluster-admin" created
apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system
> kubectl create -f rbac-config.yaml
serviceaccount "tiller" created
clusterrolebinding.rbac.authorization.k8s.io "tiller" created
然后安装和初始化Tiller。
> helm init --service-account tiller
$HELM_HOME has been configured at /Users/jb/.helm.
Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
安装GitLab Chart
添加GitLab图表仓库。
> helm repo add gitlab https://charts.gitlab.io/
"gitlab" has been added to your repositories
使用Helm安装GitLab Chart。
> helm upgrade --install gitlab gitlab/gitlab \
--timeout 600 \
--set global.hosts.domain=bakeneco.io \
--set gitlab.migrations.initialRootPassword="nyan" \
--set gitlab.gitaly.persistence.storageClass=default \
--set postgresql.persistence.storageClass=default \
--set gitlab.redis.persistence.storageClass=default \
--set gitlab.minio.persistence.storageClass=default \
--set certmanager-issuer.email=nyan@nyan.com
Release "gitlab" does not exist. Installing it now.
NAME: gitlab
LAST DEPLOYED: Sat Jul 21 18:38:43 2018
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1beta1/PodDisruptionBudget
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
gitlab-gitaly N/A 1 0 8s
gitlab-gitlab-shell N/A 1 0 8s
gitlab-sidekiq N/A 1 0 7s
gitlab-unicorn N/A 1 0 7s
gitlab-minio-v1 N/A 1 0 7s
gitlab-nginx-ingress-controller 2 N/A 0 7s
gitlab-nginx-ingress-default-backend 1 N/A 0 7s
gitlab-redis-v1 N/A 1 0 7s
gitlab-registry-v1 N/A 1 0 7s
==> v1/ServiceAccount
NAME SECRETS AGE
gitlab-certmanager-issuer 1 10s
certmanager-gitlab 1 10s
gitlab-gitlab-runner 1 10s
gitlab-nginx-ingress 1 10s
gitlab-prometheus-alertmanager 1 10s
gitlab-prometheus-kube-state-metrics 1 10s
gitlab-prometheus-node-exporter 1 10s
gitlab-prometheus-server 1 10s
==> v1/Role
NAME AGE
gitlab-certmanager-issuer 9s
gitlab-nginx-ingress 9s
==> v1/Job
NAME DESIRED SUCCESSFUL AGE
gitlab-issuer.1 1 0 8s
gitlab-migrations.1 1 0 8s
gitlab-minio-create-buckets.1 1 0 8s
==> v1/RoleBinding
NAME AGE
gitlab-certmanager-issuer 9s
gitlab-nginx-ingress 9s
==> v1beta2/StatefulSet
NAME DESIRED CURRENT AGE
gitlab-gitaly 1 1 8s
==> v1/PersistentVolumeClaim
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
gitlab-minio Pending default 10s
gitlab-postgresql Pending default 10s
gitlab-prometheus-server Pending default 10s
gitlab-redis Pending default 10s
==> v1beta1/CustomResourceDefinition
NAME AGE
certificates.certmanager.k8s.io 10s
clusterissuers.certmanager.k8s.io 10s
issuers.certmanager.k8s.io 9s
==> v1beta1/Role
gitlab-gitlab-runner 9s
==> v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
certmanager-gitlab 1 1 1 0 9s
gitlab-gitlab-runner 1 1 1 0 9s
gitlab-postgresql 1 1 1 0 8s
gitlab-prometheus-server 1 1 1 0 8s
==> v1beta1/Ingress
NAME HOSTS ADDRESS PORTS AGE
gitlab-unicorn gitlab.bakeneco.io 80, 443 8s
gitlab-minio minio.bakeneco.io 80, 443 8s
gitlab-registry registry.bakeneco.io 80, 443 8s
==> v2beta1/HorizontalPodAutoscaler
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
gitlab-gitlab-shell Deployment/gitlab-gitlab-shell <unknown>/75% 2 10 0 8s
gitlab-sidekiq-all-in-1 Deployment/gitlab-sidekiq-all-in-1 <unknown>/75% 1 10 0 8s
gitlab-unicorn Deployment/gitlab-unicorn <unknown>/75% 2 10 0 8s
gitlab-registry Deployment/gitlab-registry <unknown>/75% 2 10 0 8s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
certmanager-gitlab-56f5486fdb-f6dpb 0/2 ContainerCreating 0 9s
gitlab-gitlab-runner-84dff6795b-ffjg6 0/1 Init:0/1 0 9s
gitlab-gitlab-shell-59c74669b-8fjwg 0/1 Init:0/1 0 9s
gitlab-sidekiq-all-in-1-f9c56885f-wct2v 0/1 Init:0/2 0 9s
gitlab-task-runner-69f766f6dc-l4tq4 0/1 Pending 0 9s
gitlab-unicorn-59ccf4d574-m56z4 0/1 Pending 0 8s
gitlab-minio-567c4bd69d-zrp8m 0/1 Pending 0 8s
gitlab-nginx-ingress-controller-754587444-7lxn7 0/1 Pending 0 8s
gitlab-nginx-ingress-controller-754587444-hgjwk 0/1 Error 0 8s
gitlab-nginx-ingress-controller-754587444-qx86s 0/1 Pending 0 8s
gitlab-nginx-ingress-default-backend-c955db9c6-plvvw 1/1 Running 0 8s
gitlab-nginx-ingress-default-backend-c955db9c6-qx5jh 0/1 Pending 0 8s
gitlab-postgresql-57c66b8d6b-2sxb6 0/2 Pending 0 8s
gitlab-prometheus-server-8cf4fdd8-2m4dk 0/2 Pending 0 8s
gitlab-redis-75764d5485-9bcpg 0/2 Pending 0 8s
gitlab-registry-5d797444d7-fnpb9 0/1 Pending 0 7s
gitlab-gitaly-0 0/1 Pending 0 8s
gitlab-issuer.1-tvnpc 0/1 ContainerCreating 0 8s
gitlab-migrations.1-25sv6 0/1 Pending 0 8s
gitlab-minio-create-buckets.1-5lz7p 0/1 Pending 0 8s
==> v1/ConfigMap
NAME DATA AGE
gitlab-certmanager-issuer-certmanager 2 10s
gitlab-gitlab-runner 3 10s
gitlab-gitaly 3 10s
gitlab-gitlab-shell 2 10s
gitlab-nginx-ingress-tcp 1 10s
gitlab-migrations 4 10s
gitlab-sidekiq-all-in-1 1 10s
gitlab-sidekiq 5 10s
gitlab-task-runner 4 10s
gitlab-unicorn 8 10s
gitlab-unicorn-tests 1 10s
gitlab-minio-config-cm 3 10s
gitlab-nginx-ingress-controller 7 10s
gitlab-postgresql 0 10s
gitlab-prometheus-server 3 10s
gitlab-redis 2 10s
gitlab-registry 2 10s
==> v1beta1/ClusterRole
NAME AGE
certmanager-gitlab 9s
gitlab-prometheus-kube-state-metrics 9s
gitlab-prometheus-server 9s
==> v1beta1/RoleBinding
NAME AGE
gitlab-gitlab-runner 9s
==> v1beta1/ClusterRoleBinding
NAME AGE
certmanager-gitlab 9s
gitlab-prometheus-alertmanager 9s
gitlab-prometheus-kube-state-metrics 9s
gitlab-prometheus-node-exporter 9s
gitlab-prometheus-server 9s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
gitlab-gitaly ClusterIP None <none> 8075/TCP,9236/TCP 9s
gitlab-gitlab-shell ClusterIP 10.0.216.145 <none> 22/TCP 9s
gitlab-unicorn ClusterIP 10.0.226.44 <none> 8080/TCP,8181/TCP 9s
gitlab-minio-svc ClusterIP 10.0.113.89 <none> 9000/TCP 9s
gitlab-nginx-ingress-controller LoadBalancer 10.0.65.171 <pending> 80:32670/TCP,443:31479/TCP,22:31190/TCP 9s
gitlab-nginx-ingress-default-backend ClusterIP 10.0.104.154 <none> 80/TCP 9s
gitlab-postgresql ClusterIP 10.0.238.193 <none> 5432/TCP 9s
gitlab-prometheus-server ClusterIP 10.0.222.23 <none> 80/TCP 9s
gitlab-redis ClusterIP 10.0.148.79 <none> 6379/TCP,9121/TCP 9s
gitlab-registry ClusterIP 10.0.49.168 <none> 5000/TCP 9s
==> v1beta2/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
gitlab-gitlab-shell 1 1 1 0 9s
gitlab-sidekiq-all-in-1 1 1 1 0 9s
gitlab-task-runner 1 1 1 0 9s
gitlab-unicorn 1 1 1 0 9s
gitlab-minio 1 1 1 0 8s
gitlab-nginx-ingress-controller 3 3 3 0 8s
gitlab-nginx-ingress-default-backend 2 2 2 1 8s
gitlab-redis 1 1 1 0 8s
gitlab-registry 1 1 1 0 8s
将确认Ingress的外部IP并将其注册到DNS记录中。
> kubectl describe service gitlab-nginx-ingress-controller | grep Ingress
LoadBalancer Ingress: 23.101.134.115
过一段时间，所有的Pods都会启动起来，然后您就可以看到登录界面了。
获取密码并登录。
> kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode
nyan
整理步骤
az login
az aks get-credentials --resource-group gitlab-chart-group --name bakeneco
kubectl get nodes
kubectl get pods --all-namespaces
kubectl create -f cluster-admin-role.yaml
kubectl create -f rbac-config.yaml
helm init --service-account tiller
helm repo add gitlab https://charts.gitlab.io/
helm upgrade --install gitlab gitlab/gitlab \
--timeout 600 \
--set global.hosts.domain=bakeneco.io \
--set gitlab.migrations.initialRootPassword="nyan" \
--set gitlab.gitaly.persistence.storageClass=default \
--set postgresql.persistence.storageClass=default \
--set gitlab.redis.persistence.storageClass=default \
--set gitlab.minio.persistence.storageClass=default \
--set certmanager-issuer.email=nyan@nyan.com
kubectl describe service gitlab-nginx-ingress-controller | grep Ingress
kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode
附加：采用了失败的Azure文件存储类的显眼。
啊～ 存储真的很有趣啊。
在Azure中有提供默认的存储类型，但是我不太喜欢Azure-Disk。
我看到了一篇关于Azure存储类的好文章，所以参考了一下。
首先，在与集群节点相同的资源组（不是集群服务组！）中创建存储账户。
> az storage account create --resource-group MC_gitlab-chart-group_bakeneco_eastus --name gitlabchartstorage
{
"accessTier": null,
"creationTime": "2018-07-21T07:18:12.543726+00:00",
"customDomain": null,
"enableHttpsTrafficOnly": false,
"encryption": {
"keySource": "Microsoft.Storage",
"keyVaultProperties": null,
"services": {
"blob": {
"enabled": true,
"lastEnabledTime": "2018-07-21T07:18:12.762473+00:00"
},
"file": {
"enabled": true,
"lastEnabledTime": "2018-07-21T07:18:12.762473+00:00"
},
"queue": null,
"table": null
}
},
"id": "/subscriptions/cc07cdf5-785f-45be-9718-2283b78ab03a/resourceGroups/gitlab-chart-group/providers/Microsoft.Storage/storageAccounts/gitlabchartstorage",
"identity": null,
"kind": "Storage",
"lastGeoFailoverTime": null,
"location": "eastus",
"name": "gitlabchartstorage",
"networkRuleSet": {
"bypass": "AzureServices",
"defaultAction": "Allow",
"ipRules": [],
"virtualNetworkRules": []
},
"primaryEndpoints": {
"blob": "https://gitlabchartstorage.blob.core.windows.net/",
"file": "https://gitlabchartstorage.file.core.windows.net/",
"queue": "https://gitlabchartstorage.queue.core.windows.net/",
"table": "https://gitlabchartstorage.table.core.windows.net/"
},
"primaryLocation": "eastus",
"provisioningState": "Succeeded",
"resourceGroup": "gitlab-chart-group",
"secondaryEndpoints": {
"blob": "https://gitlabchartstorage-secondary.blob.core.windows.net/",
"file": null,
"queue": "https://gitlabchartstorage-secondary.queue.core.windows.net/",
"table": "https://gitlabchartstorage-secondary.table.core.windows.net/"
},
"secondaryLocation": "westus",
"sku": {
"capabilities": null,
"kind": null,
"locations": null,
"name": "Standard_RAGRS",
"resourceType": null,
"restrictions": null,
"tier": "Standard"
},
"statusOfPrimary": "available",
"statusOfSecondary": "available",
"tags": {},
"type": "Microsoft.Storage/storageAccounts"
}
然后创建StorageClass并将其设为默认值。
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: azurefilestorage
provisioner: kubernetes.io/azure-file
parameters:
storageAccount: gitlabchartstorage
reclaimPolicy: Retain
> kubectl create -f aks-storage-class.yaml
storageclass.storage.k8s.io "azurefilestorage" created
> kubectl patch storageclass azurefilestorage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io "azurefilestorage" patched
> kubectl delete sc default
storageclass.storage.k8s.io "default" deleted
> kubectl get storageclass
NAME PROVISIONER AGE
azurefilestorage (default) kubernetes.io/azure-file 43s
managed-premium kubernetes.io/azure-disk 40m
由于存在两个StorageClass会带来困扰，所以我删除了默认的StorageClass。然而，它却不知怎么地自动恢复了。
如果你在安装GitLab Chart之前匆忙删除了默认的StorageClass，那么安装将在它恢复之前开始。
> kubectl delete sc default
storageclass.storage.k8s.io "default" deleted
最终没有成功，我们决定放弃。虽然持久化卷声明成功了，但是似乎minio和postgresql在卷上没有正确的权限。
> kubectl logs gitlab-minio-567c4bd69d-6cf5s
time="2018-07-21T09:01:18Z" level=error msg="Initializing object layer failed" cause="Unable to initialize '.minio.sys' meta volume, mkdir /export/.minio.sys: permission denied" source="[server-main.go:214:serverMain()]"
> kubectl logs gitlab-postgresql-57c66b8d6b-b7tsh gitlab-postgresql
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
initdb: could not change permissions of directory "/var/lib/postgresql/data/pgdata": Operation not permitted
fixing permissions on existing directory /var/lib/postgresql/data/pgdata ...
应该可以找到办法，但这次我放弃了。
