可以方便地体验OpenShift 4环境（Code Ready Container）的笔记
我试用了Code Ready Container。
如何运行 Code Ready Container
我参考了下面这个人的总结，非常易懂。
这个总结详细地介绍了如何免费使用Redhat的订阅。
https://qiita.com/zaki-lknr/items/ac2223152661886438da#インストール
主要步骤大约包括以下四个：
• 获取Redhat免费订阅
• 获取crc（下载一个大约2GB的文件）
• 下载crc所需的秘密信息（pull-secret），以使crc运行起来。
• 可以构建Openshift环境。
请登录RedHat Openshift Cluster Manager，选择“Download pull secret”，并将pull-secret文件放置在适当的目录中。
同样也要下载自己操作系统适用的crc文件。
以下内容假设在Linux操作系统上进行。
crc start -p pull-secret
每次使用相同的秘密，创建别名以提高效率。
echo 'alias crcs="crc start -p pull-secret"' >> ~/.bashrc
. ~/.bashrc
启用由CRC创建的OC命令及其配置。
通过设置以下内容，您可以使用oc命令。
eval $(crc oc-env)
export KUBECONFIG=$HOME/.crc/machines/crc/kubeconfig
由于在crc start的日志中会显示如下内容，您可以打开oc login命令，或在crc console中进行openshift控制台操作。
INFO To access the cluster, first set up your environment by following 'crc oc-env' instructions
INFO Then you can access it by running 'oc login -u developer -p developer https://api.crc.testing:6443'
INFO To login as an admin, run 'oc login -u kubeadmin -p 7z6T5-qmTth-oxaoD-p3xQF https://api.crc.testing:6443'
INFO
INFO You can now run 'crc console' and use these credentials to access the OpenShift web console
下面所列，某些Openshift的操作程序被暂停以减少资源使用量。
WARN The cluster might report a degraded or error state. This is expected since several operators have been disabled to lower the resource usage. For more information, please consult the documentation
可以使用以下命令确认哪个操作者正在运行。
只有监控操作员的”Available”列为False，看起来已经停止运行了。
[openshift@base ~]$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.3.0 True False False 22d
cloud-credential 4.3.0 True False False 22d
cluster-autoscaler 4.3.0 True False False 22d
console 4.3.0 True False False 22d
dns 4.3.0 True False False 21d
image-registry 4.3.0 True False False 22d
ingress 4.3.0 True False False 22d
insights 4.3.0 True False False 22d
kube-apiserver 4.3.0 True False False 22d
kube-controller-manager 4.3.0 True False False 22d
kube-scheduler 4.3.0 True False False 22d
machine-api 4.3.0 True False False 22d
machine-config 4.3.0 True False False 22d
marketplace 4.3.0 True False False 11m
monitoring 4.3.0 False True True 22d
network 4.3.0 True False False 22d
node-tuning 4.3.0 True False False 11m
openshift-apiserver 4.3.0 True False False 22d
openshift-controller-manager 4.3.0 True False False 21d
openshift-samples 4.3.0 True False False 22d
operator-lifecycle-manager 4.3.0 True False False 22d
operator-lifecycle-manager-catalog 4.3.0 True False False 22d
operator-lifecycle-manager-packageserver 4.3.0 True False False 11m
service-ca 4.3.0 True False False 22d
service-catalog-apiserver 4.3.0 True False False 22d
service-catalog-controller-manager 4.3.0 True False False 22d
storage 4.3.0 True False False 22d
当在给定的命名空间内进行检查时，可以看到如下定义，副本数被定义为0。
此外，CRC的手册中也写着类似的内容。
[openshift@base ~]$ oc get all -n openshift-monitoring
NAME READY STATUS RESTARTS AGE
pod/node-exporter-hffz9 2/2 Running 0 22d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/alertmanager-main ClusterIP 172.30.206.91 <none> 9094/TCP 22d
service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 22d
service/cluster-monitoring-operator ClusterIP None <none> 8080/TCP 22d
service/grafana ClusterIP 172.30.191.225 <none> 3000/TCP 22d
service/kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 22d
service/node-exporter ClusterIP None <none> 9100/TCP 22d
service/openshift-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 22d
service/prometheus-adapter ClusterIP 172.30.20.184 <none> 443/TCP 22d
service/prometheus-k8s ClusterIP 172.30.22.83 <none> 9091/TCP,9092/TCP 22d
service/prometheus-operated ClusterIP None <none> 9090/TCP,10901/TCP 22d
service/prometheus-operator ClusterIP None <none> 8080/TCP 22d
service/telemeter-client ClusterIP None <none> 8443/TCP 22d
service/thanos-querier ClusterIP 172.30.169.150 <none> 9091/TCP,9092/TCP 22d
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/node-exporter 1 1 1 1 1 kubernetes.io/os=linux 22d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cluster-monitoring-operator 0/0 0 0 22d
deployment.apps/grafana 0/0 0 0 22d
deployment.apps/kube-state-metrics 0/0 0 0 22d
deployment.apps/openshift-state-metrics 0/0 0 0 22d
deployment.apps/prometheus-adapter 0/0 0 0 22d
deployment.apps/prometheus-operator 0/0 0 0 22d
deployment.apps/telemeter-client 0/0 0 0 22d
deployment.apps/thanos-querier 0/0 0 0 22d
NAME DESIRED CURRENT READY AGE
replicaset.apps/cluster-monitoring-operator-7bbc9f9895 0 0 0 22d
replicaset.apps/grafana-687f7dfcf4 0 0 0 22d
replicaset.apps/grafana-7847db887 0 0 0 22d
replicaset.apps/kube-state-metrics-777f6bf798 0 0 0 22d
replicaset.apps/openshift-state-metrics-b6755756 0 0 0 22d
replicaset.apps/prometheus-adapter-79f9c99d67 0 0 0 22d
replicaset.apps/prometheus-adapter-7f9c5d699 0 0 0 22d
replicaset.apps/prometheus-operator-985bf8dd5 0 0 0 22d
replicaset.apps/telemeter-client-54dfc4d54c 0 0 0 22d
replicaset.apps/telemeter-client-7c87f56869 0 0 0 22d
replicaset.apps/thanos-querier-5856664597 0 0 0 22d
replicaset.apps/thanos-querier-7f9657d4f7 0 0 0 22d
NAME READY AGE
statefulset.apps/alertmanager-main 0/0 22d
statefulset.apps/prometheus-k8s 0/0 22d
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
route.route.openshift.io/alertmanager-main alertmanager-main-openshift-monitoring.apps-crc.testing alertmanager-main web reencrypt/Redirect None
route.route.openshift.io/grafana grafana-openshift-monitoring.apps-crc.testing grafana https reencrypt/Redirect None
route.route.openshift.io/prometheus-k8s prometheus-k8s-openshift-monitoring.apps-crc.testing prometheus-k8s web reencrypt/Redirect None
route.route.openshift.io/thanos-querier thanos-querier-openshift-monitoring.apps-crc.testing thanos-querier web reencrypt/Redirect None
在初始状态下运行的Pod大约有70个左右。
[openshift@base ~]$ oc get po --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
openshift-apiserver-operator openshift-apiserver-operator-7cc77d965f-4mcgm 1/1 Running 0 22d
openshift-apiserver apiserver-6jtsm 1/1 Running 0 10m
openshift-authentication-operator authentication-operator-57d4974d5d-mwdkl 1/1 Running 1 22d
openshift-authentication oauth-openshift-67585659c6-g8lxt 1/1 Running 0 4m45s
openshift-authentication oauth-openshift-67585659c6-shk64 1/1 Running 0 3m43s
openshift-cluster-machine-approver machine-approver-57dd49d7c5-mvdz2 2/2 Running 0 22d
openshift-cluster-node-tuning-operator cluster-node-tuning-operator-6986d4dff4-cn54n 1/1 Running 0 22d
openshift-cluster-node-tuning-operator tuned-tbljv 1/1 Running 0 9m41s
openshift-cluster-samples-operator cluster-samples-operator-889fb7599-zjblq 2/2 Running 0 22d
openshift-cluster-storage-operator cluster-storage-operator-5dc75b588c-mh9w6 1/1 Running 0 22d
openshift-console-operator console-operator-57f5bcc578-b59hx 1/1 Running 0 22d
openshift-console console-8c7b46fb4-68x4w 1/1 Running 0 22d
openshift-controller-manager-operator openshift-controller-manager-operator-68dcf95c47-bxbln 1/1 Running 0 22d
openshift-controller-manager controller-manager-c26bm 1/1 Running 0 21d
openshift-dns-operator dns-operator-7785d9f869-nqmh8 2/2 Running 0 22d
openshift-dns dns-default-s4r76 2/2 Running 0 22d
openshift-etcd etcd-member-crc-w6th5-master-0 2/2 Running 0 22d
openshift-image-registry cluster-image-registry-operator-f9697f69d-44484 2/2 Running 0 22d
openshift-image-registry image-registry-864894cbd5-8n5ff 1/1 Running 0 22d
openshift-image-registry node-ca-kp85n 1/1 Running 0 22d
openshift-ingress-operator ingress-operator-556dd68cb9-gfbwf 2/2 Running 0 22d
openshift-ingress router-default-77c77568f4-npdrs 1/1 Running 0 22d
openshift-kube-apiserver-operator kube-apiserver-operator-566b9798-fzvtd 1/1 Running 0 22d
openshift-kube-apiserver installer-10-crc-w6th5-master-0 0/1 Completed 0 21d
openshift-kube-apiserver installer-11-crc-w6th5-master-0 0/1 Completed 0 8m21s
openshift-kube-apiserver installer-12-crc-w6th5-master-0 0/1 OOMKilled 0 6m6s
openshift-kube-apiserver installer-9-crc-w6th5-master-0 0/1 Completed 0 22d
openshift-kube-apiserver kube-apiserver-crc-w6th5-master-0 3/3 Running 0 5m36s
openshift-kube-apiserver revision-pruner-10-crc-w6th5-master-0 0/1 Completed 0 21d
openshift-kube-apiserver revision-pruner-11-crc-w6th5-master-0 0/1 OOMKilled 0 6m12s
openshift-kube-apiserver revision-pruner-12-crc-w6th5-master-0 0/1 Completed 0 3m37s
openshift-kube-apiserver revision-pruner-8-crc-w6th5-master-0 0/1 Completed 0 22d
openshift-kube-apiserver revision-pruner-9-crc-w6th5-master-0 0/1 Completed 0 22d
openshift-kube-controller-manager-operator kube-controller-manager-operator-7c8b7465b-4mbkc 1/1 Running 0 22d
openshift-kube-controller-manager installer-7-crc-w6th5-master-0 0/1 Completed 0 8m29s
openshift-kube-controller-manager kube-controller-manager-crc-w6th5-master-0 3/3 Running 1 8m11s
openshift-kube-controller-manager revision-pruner-6-crc-w6th5-master-0 0/1 Completed 0 22d
openshift-kube-controller-manager revision-pruner-7-crc-w6th5-master-0 0/1 OOMKilled 0 6m14s
openshift-kube-scheduler-operator openshift-kube-scheduler-operator-557777c86b-zxqx7 1/1 Running 0 22d
openshift-kube-scheduler installer-7-crc-w6th5-master-0 0/1 Completed 0 8m19s
openshift-kube-scheduler openshift-kube-scheduler-crc-w6th5-master-0 1/1 Running 1 8m3s
openshift-kube-scheduler revision-pruner-6-crc-w6th5-master-0 0/1 Completed 0 22d
openshift-kube-scheduler revision-pruner-7-crc-w6th5-master-0 0/1 Completed 0 5m53s
openshift-machine-config-operator machine-config-daemon-xtdmj 2/2 Running 0 22d
openshift-machine-config-operator machine-config-server-pv6nm 1/1 Running 0 22d
openshift-marketplace certified-operators-5d6f745457-qkm8w 1/1 Running 0 9m47s
openshift-marketplace community-operators-55b7cc57bf-rcqwl 1/1 Running 0 9m43s
openshift-marketplace marketplace-operator-7fbcb88798-wxcdc 1/1 Running 0 22d
openshift-marketplace redhat-operators-65ffcdcd6-rjmzn 1/1 Running 0 9m39s
openshift-monitoring node-exporter-hffz9 2/2 Running 0 22d
openshift-multus multus-admission-controller-z6sx4 1/1 Running 0 22d
openshift-multus multus-vbjms 1/1 Running 0 22d
openshift-network-operator network-operator-5c7c7dc988-dt8qx 1/1 Running 0 22d
openshift-operator-lifecycle-manager catalog-operator-5d644f7b4b-zfhb6 1/1 Running 0 22d
openshift-operator-lifecycle-manager olm-operator-6d454db9dd-4sz4q 1/1 Running 0 22d
openshift-operator-lifecycle-manager packageserver-55b886b6db-fc64w 1/1 Running 0 9m34s
openshift-operator-lifecycle-manager packageserver-55b886b6db-klpq2 1/1 Running 0 10m
openshift-sdn ovs-m586r 1/1 Running 0 22d
openshift-sdn sdn-controller-7s5hg 1/1 Running 0 22d
openshift-sdn sdn-twbd8 1/1 Running 0 22d
openshift-service-ca-operator service-ca-operator-595657f77-rbmjs 1/1 Running 0 22d
openshift-service-ca apiservice-cabundle-injector-d84c98485-v787m 1/1 Running 0 22d
openshift-service-ca configmap-cabundle-injector-6cc5ccdd7f-tcl4m 1/1 Running 0 22d
openshift-service-ca service-serving-cert-signer-d59b877-thvch 1/1 Running 0 22d
openshift-service-catalog-apiserver-operator openshift-service-catalog-apiserver-operator-6cddfd76cc-pmzmw 1/1 Running 1 22d
openshift-service-catalog-controller-manager-operator openshift-service-catalog-controller-manager-operator-5886hlmm2 1/1 Running 1 22d
使用CRC构建的节点登录SSH的方法
在 .crc 文件夹下存在 SSH 秘钥，可以通过 crc ip 获取节点的 IP。
ssh -i ~/.crc/machines/crc/id_rsa core@`crc ip`
除此之外，貌似还有另一种方法可以在节点上实现，但在获取Docker镜像时失败了。
oc debug nodes/`oc get node -ojsonpath='{.items[0].metadata.name}'`
似乎是通过docker login从Redhat仓库获取Debug pod的镜像，但未能获取以下内容。
sudo docker pull registry.redhat.io/rhel7/support-tools
遇到的問題
无法解决的问题
在 CRC 开始时遇到以下类似的错误。
ERRO Failed to query DNS from host: lookup foo.apps-crc.testing on [240d:1a:4a3:1b00:e67e:66ff:fe43:9a43]:53: no such host
可以通过下面列出的方法来避免这个问题：
https://medium.com/@trlogic/how-to-setup-local-openshift-4-cluster-with-red-hat-codeready-containers-6c5aefba72ad
将以下内容添加到/etc/hosts文件中。
192.168.130.11 api.crc.testing
192.168.130.11 oauth-openshift.apps-crc.testing
192.168.130.11 console-openshift-console.apps-crc.testing
在执行crc的计算机上运行Dnsmasq
如果单独设置了bind等等，53号端口会产生冲突而导致困扰。
因为我为Openshift 3.11创建了绑定，所以遇到了麻烦。
通过使用systemctl命令进行确认，似乎无法看到dnsmasq正在运行，所以它似乎是在其他地方单独运行。
其他
etcd 备份
可以登录到Code Ready的主节点上进行确认。
[root@crc-w6th5-master-0 ~]# sh /usr/local/bin/etcd-snapshot-backup.sh .
Creating asset directory ./assets
Downloading etcdctl binary..
etcdctl version: 3.3.17
API version: 3.3
Trying to backup etcd client certs..
etcd client certs found in /etc/kubernetes/static-pod-resources/kube-apiserver-pod-3 backing up to ./assets/backup/
Backing up /etc/kubernetes/manifests/etcd-member.yaml to ./assets/backup/
Trying to backup latest static pod resources..
{"level":"warn","ts":"2020-03-07T10:05:45.648Z","caller":"clientv3/retry_interceptor.go:116","msg":"retry stream intercept"}
Snapshot saved at ./assets/tmp/snapshot.db
snapshot db and kube resources are successfully saved to ./snapshot_db_kuberesources_2020-03-07_100542.tar.gz!
[root@crc-w6th5-master-0 ~]# ls
assets snapshot_db_kuberesources_2020-03-07_100542.tar.gz
[root@crc-w6th5-master-0 ~]# tar xzvf snapshot_db_kuberesources_2020-03-07_100542.tar.gz
static-pod-resources/kube-apiserver-pod-10/
static-pod-resources/kube-apiserver-pod-10/secrets/
static-pod-resources/kube-apiserver-pod-10/secrets/etcd-client/
static-pod-resources/kube-apiserver-pod-10/secrets/etcd-client/tls.crt
static-pod-resources/kube-apiserver-pod-10/secrets/etcd-client/tls.key
static-pod-resources/kube-apiserver-pod-10/secrets/kube-apiserver-cert-syncer-client-cert-key/
static-pod-resources/kube-apiserver-pod-10/secrets/kube-apiserver-cert-syncer-client-cert-key/tls.key
static-pod-resources/kube-apiserver-pod-10/secrets/kube-apiserver-cert-syncer-client-cert-key/tls.crt
static-pod-resources/kube-apiserver-pod-10/secrets/kubelet-client/
static-pod-resources/kube-apiserver-pod-10/secrets/kubelet-client/tls.crt
static-pod-resources/kube-apiserver-pod-10/secrets/kubelet-client/tls.key
static-pod-resources/kube-apiserver-pod-10/configmaps/
static-pod-resources/kube-apiserver-pod-10/configmaps/config/
static-pod-resources/kube-apiserver-pod-10/configmaps/config/config.yaml
static-pod-resources/kube-apiserver-pod-10/configmaps/etcd-serving-ca/
static-pod-resources/kube-apiserver-pod-10/configmaps/etcd-serving-ca/ca-bundle.crt
static-pod-resources/kube-apiserver-pod-10/configmaps/kube-apiserver-cert-syncer-kubeconfig/
static-pod-resources/kube-apiserver-pod-10/configmaps/kube-apiserver-cert-syncer-kubeconfig/kubeconfig
static-pod-resources/kube-apiserver-pod-10/configmaps/kube-apiserver-pod/
static-pod-resources/kube-apiserver-pod-10/configmaps/kube-apiserver-pod/forceRedeploymentReason
static-pod-resources/kube-apiserver-pod-10/configmaps/kube-apiserver-pod/pod.yaml
static-pod-resources/kube-apiserver-pod-10/configmaps/kube-apiserver-pod/version
static-pod-resources/kube-apiserver-pod-10/configmaps/kubelet-serving-ca/
static-pod-resources/kube-apiserver-pod-10/configmaps/kubelet-serving-ca/ca-bundle.crt
static-pod-resources/kube-apiserver-pod-10/configmaps/sa-token-signing-certs/
static-pod-resources/kube-apiserver-pod-10/configmaps/sa-token-signing-certs/service-account-001.pub
static-pod-resources/kube-apiserver-pod-10/configmaps/sa-token-signing-certs/service-account-002.pub
static-pod-resources/kube-apiserver-pod-10/configmaps/kube-apiserver-server-ca/
static-pod-resources/kube-apiserver-pod-10/configmaps/kube-apiserver-server-ca/ca-bundle.crt
static-pod-resources/kube-apiserver-pod-10/configmaps/oauth-metadata/
static-pod-resources/kube-apiserver-pod-10/configmaps/oauth-metadata/oauthMetadata
static-pod-resources/kube-apiserver-pod-10/kube-apiserver-pod.yaml
snapshot.db
[root@crc-w6th5-master-0 ~]#
启用CRC的Cluster Monitoring
在crc的默认设置中，集群监控似乎没有启用。
然后，所有复制品的数量都为0。
oc scale --replicas=1 statefulset --all -n openshift-monitoring; oc scale --replicas=1 deployment --all -n openshift-monitoring
当尝试使用上述方法启动时，由于请求的内存值超过节点的限制，导致内存不足，Pod无法被调度。
以下是相关问题。
解决方案之一是增加crc虚拟机的内存分配方法如下：
$ crc config set memory 16398
Changes to configuration property 'memory' are only applied when a new CRC instance is created.
If you already have a CRC instance, then for this configuration change to take effect, delete the CRC instance with 'crc delete' and start a new one with 'crc start'.
$ crc delete && crc create
监测支持的情况
OCP4的监控堆栈文档可以在下方找到，但关于引入crc的thanos等内容没有说明。
在Openshift的博客中提到，thanos和prometheus实例在OCP中不受支持。
使用Thanos和对象存储，实现从多个Openshift平台永久化指标数据。
这篇文章介绍了使用Prometheus、Thanos和S3进行指标持久化的方法。
Thanos Receiver将数据持久化到S3，Thanos Gateway对S3进行查询。
