【图解】什么是缓存策略?什么是基于缓存的查询调优?
基础知识编 – Paraphrase in Chinese:
入门知识编辑
在Web系统开发中,理解和设计缓存是非常重要的。
缓存可以减轻服务器的负担并大幅缩短响应时间。通过暂时存储经常访问的数据和计算结果,避免重新计算或重新获取相同的请求。
在前半的”基礎知識編”中,主要总结了缓存策略和AWS提供的内存缓存服务,并在文章后半部分的”实践指南”中,我们将实践使用”MemCached”进行查询优化。
缓存策略类型
缓存策略选择非常重要,必须根据系统的目标、架构和负载特性进行选择。主要通过“本地或远程”、“读取或写入”、“内联或旁路”这三个维度来分类策略。在每个维度上的选择都会对性能和一致性产生重大影响。
本地或远程

领导或者主席

内嵌或侧边

MemCached
内建缓存
MemCache是AWS Elastic Cache支持的缓存引擎之一,作为简单的键值存储运行。它还支持多线程,可以充分利用多核处理器的能力。它适用于具有简单数据结构或小规模数据集,具有非常高的内存效率。
Redis是一种开源的内存数据结构存储系统。
Redis是AWS Elastic Cache的支持引擎,支持更高级的数据结构和事务,并且还支持丰富的数据类型,如列表/哈希/集合/有序集合等。然而,由于Redis采用单线程设计,所以有一次只能执行一个命令的限制。适用于具有大规模复杂数据结构或对性能要求较高的应用。
实践学习篇
我們接下來將使用實際的代碼進行使用快取進行性能調整的演示。
本文将介绍如何在本地环境中使用Docker进行操作验证。我们将使用memcached作为缓存服务器。
另外,这次假设使用了引导和旁观策略的情况。也就是说,在第一次阅读时(如果缓存未命中),从数据库获取数据,而在之后的请求中,从缓存获取。图示如下所示。

样例API概览
在这里,我们将使用用Go编写的API(使用Gin框架)作为样例。同时,这个应用程序提供了两个端点。
(1)慢查询API(路径:/db/1)
第一个选项是,当请求发生时,向数据库(MySQL)发送查询。然而,根据以下查询语句:SELECT id, value, SLEEP(10) FROM customers WHERE id = ?,这个查询非常缓慢,需要等待10秒钟才能完成。
// 例如:http://localhost:8080/db/1
r.GET(“/db/:id”, func(c *gin.Context) {
paramId := c.Param(“id”)
var result Customer
// 执行一个延迟10秒的查询
err := db.QueryRow(“SELECT id, value, SLEEP(10) FROM customers WHERE id = ?”, paramId).Scan(&result.ID, &result.Value, &result.SleepResult)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{“error”: err.Error()})
return
}
c.JSON(http.StatusOK, result)
})
(2)利用缓存的API(路径:/cache/1)
第二个选项是,当有请求时,首先检查缓存服务器中的值,如果不存在,则从数据库中获取值并保存到缓存服务器上,然后返回结果。对于之后的第二次请求,将从缓存服务器中获取值。
// 例如:http://localhost:8080/cache/1
r.GET(“/cache/:id”, func(c *gin.Context) {
paramId := c.Param(“id”)
// 从缓存中获取
item, err := mc.Get(paramId)
// 如果缓存不存在
if err == memcache.ErrCacheMiss {
var result Customer
// 执行延迟10秒的查询
err := db.QueryRow(“SELECT id, value, SLEEP(10) FROM customers WHERE id = ?”, paramId).Scan(&result.ID, &result.Value, &result.SleepResult)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{“error”: err.Error()})
return
}
// 注册缓存
resultBytes, err := json.Marshal(result)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{“error”: err.Error()})
return
}
item = &memcache.Item{
Key: paramId,
Value: resultBytes,
}
if err := mc.Set(item); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{“error”: err.Error()})
return
}
c.JSON(http.StatusOK, result)
return
}
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{“error”: err.Error()})
return
}
// 将缓存结果处理为响应格式
result := Customer{}
if err = json.Unmarshal(item.Value, &result); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{“error”: err.Error()})
return
}
c.JSON(http.StatusOK, result)
})
本地版
以下是基础设施构成图。

让我们从本地机器上实际体验使用缓存进行查询优化。首先,请克隆以下存储库。然后,请参考README.md来进行环境设置。
# クローン
$ git clone git@github.com:WebEngrChild/go-rds-memcached.git
# Docker起動
$ docker compose up -d
# API起動
$ docker compose exec app go run main.go
测量慢查询API
即使在使用缓存服务器进行性能调优时,”现状分析”也是必不可少的。通过定量测量在实施前后可以实现哪些改进,可以成为一个很好的衡量政策成败的标准来进行回顾。
那么,我们尝试向上述的第一个慢查询API发送请求。在收到响应之前,预计会有大约10秒钟的延迟。
> which ab
/usr/sbin/ab
> curl http://localhost:8080/db/1
{"id":1,"value":"Initial Value","sleepResult":0}
使用Apache Bench进行负载测试
下一步,我们将使用Apache Bench来测量API的性能。需要注意的是,如果使用Mac,此工具已经默认安装。您可以参考下面的文章来了解该工具的详细信息。
> ab -n 30 -c 30 http://localhost:8080/db/1
Benchmarking localhost (be patient)...
# 一部割愛
# レイテンシの分布(パーセンタイル情報)
Percentage of the requests served within a certain time (ms)
50% 10079
66% 10083
75% 10085
80% 10087
90% 10088
95% 10089
98% 10089
99% 10089
100% 10089 (longest request)
在这里,-n选项指定了总请求数,-c选项指定了并行请求数。在这个例子中,我们使用30个并行请求执行了30次请求。从执行结果来看,我们可以看出延迟在分布的每个百分位点上都有大约10秒的延迟。
确认慢查询
在本地环境中使用的MySQL镜像默认会输出慢查询日志。此外,在docker-compose.yml中将慢查询映射到本地机器上。
mysql:
container_name: mysql
build: .docker/mysql/
volumes:
– .docker/mysql/init:/docker-entrypoint-initdb.d
– .docker/mysql/my.cnf:/etc/mysql/conf.d/my.cnf
– .docker/mysql/log:/var/log/mysql # 在這裡將日誌映射
environment:
– MYSQL_ROOT_PASSWORD=${DB_PASS}
ports:
– “3306:3306”
networks:
sample_go_network:
查看日志后,我们可以确认有一些输出。通过Query_time,我们可以知道查询大约需要10秒钟。
# Time: 2023-07-22T02:32:36.702832Z
# User@Host: root[root] @ [192.168.192.4] Id: 24
# Query_time: 10.007020 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1
SET timestamp=1689993146;
SELECT id, value, SLEEP(10) FROM customers WHERE id = '1';
使用缓存进行调优后的测量
# 1回目は同じく10秒程度かかる
> curl http://localhost:8080/cache/1
{"id":1,"value":"Initial Value","sleepResult":0}
# 2回目以降は遅延の発生はしない
> curl http://localhost:8080/cache/1
{"id":1,"value":"Initial Value","sleepResult":0}
第一次请求由于在存储在MemCached中之前,也会出现大约10秒的延迟。但是,从第二次请求开始,由于使用了已保存的数据,所以不会有响应延迟。
> ab -n 30 -c 30 http://localhost:8080/cache/1
Benchmarking localhost (be patient)...
# 一部割愛
# レイテンシの分布(パーセンタイル情報)
Percentage of the requests served within a certain time (ms)
50% 22
66% 22
75% 24
80% 25
90% 26
95% 26
98% 10047
99% 10047
100% 10047 (longest request)
在Apache Bench中可以确认到响应延迟显著改善。另外,可以看出除了首次查询之外,并没有输出慢查询。
亚马逊云服务编程
基础设施架构图

使用AWS的各种资源都会产生费用,其中包括很多需要付费的项目。个人用户尤其需要注意。对于产生的费用我们无法承担任何责任。
预先设定
在本文中,如果您已经完成了设置,则无需操作。但是,本文需要准备与IAM用户、接近根目录的AWS CLI安装相关的内容。由于Terraform在Docker上运行,因此无需单独安装。
-
- AWS CLIのインストール
- rootに近い権限を持つIAMユーザーの作成
请从以下存储库克隆,请参考README.md文件进行环境设置。
# クローン
$ git clone git@github.com:WebEngrChild/go-rds-memcached.git
# Docker起動
$ docker compose up -d
# API起動
$ docker compose exec app go run main.go
引用下面的中文释义,只需要一种选项:
介绍Terraform代码。
本文使用Terraform来构建AWS资源。我将省略详细说明,但为参考,将提供代码。
provider “aws” {
region = “ap-northeast-1”
}
variable “project” {
type = string
default = “go-api”
}
variable “environment” {
type = string
default = “dev”
}
variable “cidr_blocks” {
description = “CIDR块列表”
type = list(string)
default = [“<请设置您的全局IP地址>/32”]
}
# ————————————————————#
# 现有的ECR
# ————————————————————#
data “aws_ecr_repository” “existing” {
name = “go-dev-repo”
}
# ————————————————————#
# 现有的SSM参数存储
# ————————————————————#
data “aws_ssm_parameter” “existing” {
name = “/env”
}
# ————————————————————#
# 现有的ECR
# ————————————————————#
data “aws_ecr_repository” “existing” {
name = “go-dev-repo”
}
# ————————————————————#
# 现有的SSM参数存储
# ————————————————————#
data “aws_ssm_parameter” “existing” {
name = “/env”
}
# ————————————————————#
# 最新的EC2 AMI
# ————————————————————#
data “aws_ssm_parameter” “ami” {
name = “/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-6.1-x86_64”
}
# 最新のAMIイメージを取得
data "aws_ssm_parameter" "ami" {
name = "/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-6.1-x86_64"
}
# ————————————————————#
# 本地变量
# ————————————————————#
locals {
zones = [“1a”, “1c”, “1d”]
public_cidrs = [“10.0.1.0/24”, “10.0.2.0/24”, “10.0.3.0/24”]
private_cidrs = [“10.0.10.0/24”, “10.0.20.0/24”, “10.0.30.0/24”]
}
# ————————————————————#
# VPC
# ————————————————————#
resource “aws_vpc” “main” {
cidr_block = “10.0.0.0/16”
tags = {
Name = format(“%s-%s-aws_vpc”, var.environment, var.project)
}
}
# ————————————————————#
# 公共子网
# ————————————————————#
resource “aws_subnet” “public” {
for_each = toset(local.zones)
vpc_id = aws_vpc.main.id
availability_zone = “ap-northeast-${each.value}”
cidr_block = local.public_cidrs[index(local.zones, each.value)]
tags = {
Name = format(“%s-%s-public-%s”, var.environment, var.project, each.value)
}
}
# ————————————————————#
# 私有子网
# ————————————————————#
resource “aws_subnet” “private” {
for_each = toset(local.zones)
vpc_id = aws_vpc.main.id
availability_zone = “ap-northeast-${each.value}”
cidr_block = local.private_cidrs[index(local.zones, each.value)]
tags = {
Name = format(“%s-%s-private-%s”, var.environment, var.project, each.value)
}
}
# ————————————————————#
# 互联网网关
# ————————————————————#
resource “aws_internet_gateway” “main” {
vpc_id = aws_vpc.main.id
tags = {
Name = format(“%s-%s-aws_internet_gateway”, var.environment, var.project)
}
}
# ————————————————————#
# 弹性IP
# ————————————————————#
resource “aws_eip” “nat” {
for_each = toset(local.zones)
domain = “vpc”
tags = {
Name = format(“%s-%s-aws_eip-nat_%s”, var.environment, var.project, each.value)
}
}
# ————————————————————#
# NAT网关
# ————————————————————#
resource “aws_nat_gateway” “nat” {
for_each = toset(local.zones)
subnet_id = aws_subnet.public[each.value].id
allocation_id = aws_eip.nat[each.value].id
tags = {
Name = format(“%s-%s-nat_%s”, var.environment, var.project, each.value)
}
}
# ————————————————————#
# 公共路由表
# ————————————————————#
resource “aws_route_table” “public” {
vpc_id = aws_vpc.main.id
tags = {
Name = format(“%s-%s-aws_route_table-public”, var.environment, var.project)
}
}
# ————————————————————#
# 公共路由
# ————————————————————#
resource “aws_route” “public” {
destination_cidr_block = “0.0.0.0/0”
route_table_id = aws_route_table.public.id
gateway_id = aws_internet_gateway.main.id
}
# ————————————————————#
# 关联公共路由表
# ————————————————————#
resource “aws_route_table_association” “public” {
for_each = toset(local.zones)
subnet_id = aws_subnet.public[each.value].id
route_table_id = aws_route_table.public.id
}
# ————————————————————#
# 私有路由表
# ————————————————————#
resource “aws_route_table” “private” {
for_each = toset(local.zones)
vpc_id = aws_vpc.main.id
tags = {
Name = format(“%s-%s-private_%s”, var.environment, var.project, each.value)
}
}
# ————————————————————#
# 私有路由
# ————————————————————#
resource “aws_route” “private” {
for_each = toset(local.zones)
destination_cidr_block = “0.0.0.0/0”
route_table_id = aws_route_table.private[each.value].id
nat_gateway_id = aws_nat_gateway.nat[each.value].id
}
# ————————————————————#
# 关联私有路由表
# ————————————————————#
resource “aws_route_table_association” “private” {
for_each = toset(local.zones)
subnet_id = aws_subnet.private[each.value].id
route_table_id = aws_route_table.private[each.value].id
}
# ————————————————————#
# 安全组ALB
# ————————————————————#
resource “aws_security_group” “alb” {
name = var.project
description = var.project
vpc_id = aws_vpc.main.id
tags = {
Name = format(“%s-%s-aws_security_group-alb”, var.environment, var.project)
}
}
# ————————————————————#
# 安全组规则
# ————————————————————#
resource “aws_security_group_rule” “alb_egress” {
security_group_id = aws_security_group.alb.id
type = “egress”
from_port = 0
to_port = 0
protocol = “-1”
cidr_blocks = [“0.0.0.0/0”]
}
resource “aws_security_group_rule” “alb_ingress” {
security_group_id = aws_security_group.alb.id
type = “ingress”
from_port = 80
to_port = 80
protocol = “tcp”
cidr_blocks = [“0.0.0.0/0”]
}
你可以在本地定义文件中可用的变量。在这里,我们使用 `for_each` 循环定义了可用区和子网的创建。
同时,使用format()函数以任意的格式输出每个变量。
# ------------------------------------------------------------#
# local variables
# ------------------------------------------------------------#
locals {
zones = ["1a", "1c", "1d"]
public_cidrs = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
private_cidrs = ["10.0.10.0/24", "10.0.20.0/24", "10.0.30.0/24"]
}
## 省略...
# ------------------------------------------------------------#
# Subnet Public
# ------------------------------------------------------------#
resource "aws_subnet" "public" {
for_each = toset(local.zones)
vpc_id = aws_vpc.main.id
availability_zone = "ap-northeast-${each.value}"
cidr_block = local.public_cidrs[index(local.zones, each.value)]
tags = {
Name = format("%s-%s-public-%s", var.environment, var.project, each.value)
}
}
## 省略...
ecs.tf
# ————————————————————#
# 負載均衡器
# ————————————————————#
resource “aws_lb” “main” {
load_balancer_type = “application”
name = var.projectsecurity_groups = [aws_security_group.alb.id]
subnets = values(aws_subnet.public)[*].id
}
# ————————————————————#
# 負載均衡器監聽器
# ————————————————————#
resource “aws_lb_listener” “main” {
port = 80
protocol = “HTTP”
load_balancer_arn = aws_lb.main.arn
default_action {
type = “fixed-response”
fixed_response {
content_type = “text/plain”
status_code = “200”
message_body = “ok”
}
}
}
# ————————————————————#
# 任務定義
# ————————————————————#
resource “aws_ecs_task_definition” “main” {
family = var.project
requires_compatibilities = [“FARGATE”]
cpu = “256”
memory = “512”
network_mode = “awsvpc”
container_definitions = < .env && ./main”],
“logConfiguration”: {
“logDriver”: “awslogs”,
“options”: {
“awslogs-group” : “${aws_cloudwatch_log_group.ecs_logs.name}”,
“awslogs-region”: “ap-northeast-1”,
“awslogs-stream-prefix”: “ecs”
}
},
“healthCheck”: {
“command”: [“CMD-SHELL”, “curl -f http://localhost:8080/ || exit 1”],
“interval”: 30,
“timeout”: 5,
“retries”: 3,
“startPeriod”: 10
}
}
]
EOL
execution_role_arn = aws_iam_role.ecs_task_execution_role.arn
task_role_arn = aws_iam_role.task_role.arn
}
# ————————————————————#
# ECS 集群
# ————————————————————#
resource “aws_ecs_cluster” “main” {
name = var.project
}
# ————————————————————#
# ELB 目標組
# ————————————————————#
resource “aws_lb_target_group” “main” {
name = var.project
vpc_id = aws_vpc.main.id
port = 8080
protocol = “HTTP”
target_type = “ip”
health_check {
port = 8080
path = “/”
protocol = “HTTP”
}
}
# ————————————————————#
# ALB 監聽器規則
# ————————————————————#
resource “aws_lb_listener_rule” “main” {
listener_arn = aws_lb_listener.main.arn
action {
type = “forward”
target_group_arn = aws_lb_target_group.main.arn
}
condition {
path_pattern {
values = [“*”]
}
}
}
# ————————————————————#
# ECS 安全群組
# ————————————————————#
resource “aws_security_group” “ecs” {
name = format(“%s-ecs”, var.project)
description = format(“%s-ecs”, var.project)
vpc_id = aws_vpc.main.id
tags = {
Name = format(“%s-ecs”, var.project)
}
}
# ————————————————————#
# ECS 安全群組出站規則
# ————————————————————#
resource “aws_security_group_rule” “ecs_egress” {
security_group_id = aws_security_group.ecs.id
type = “egress”
from_port = 0
to_port = 0
protocol = “-1”
cidr_blocks = [“0.0.0.0/0”]
}
# ————————————————————#
# ECS 安全群組入站規則
# ————————————————————#
resource “aws_security_group_rule” “ecs_ingress” {
security_group_id = aws_security_group.ecs.id
type = “ingress”
from_port = 8080
to_port = 8080
protocol = “tcp”
cidr_blocks = [“10.0.0.0/16”]
}
# ————————————————————#
# ECS 服務
# ————————————————————#
resource “aws_ecs_service” “main” {
name = var.project
depends_on = [aws_lb_listener_rule.main]
cluster = aws_ecs_cluster.main.name
launch_type = “FARGATE”
desired_count = 1
task_definition = aws_ecs_task_definition.main.arn
network_configuration {
subnets = values(aws_subnet.private)[*].id
security_groups = [aws_security_group.ecs.id]
}
load_balancer {
target_group_arn = aws_lb_target_group.main.arn
container_name = “go”
container_port = 8080
}
}
# ————————————————————#
# 任務執行角色
# ————————————————————#
resource “aws_iam_role” “ecs_task_execution_role” {
name = format(“%s-%s-ecs_task_execution_role”, var.environment, var.project)
assume_role_policy = <<EOF
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Action”: “sts:AssumeRole”,
“Principal”: {
“Service”: “ecs-tasks.amazonaws.com”
},
“Effect”: “Allow”,
“Sid”: “”
}
]
}
EOF
}
# ————————————————————#
# 聯繫Private
# ————————————————————#
resource “aws_iam_policy” “ecs_task_execution_policy” {
name = format(“%s-%s-ecs_task_execution_policy”, var.environment, var.project)
path = “/”
description = format(“%s-ecs-task-execution-policy”, var.project)
policy = <<EOF
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: [
“ecs:StartTask”,
“ecs:StopTask”,
“ecs:DescribeTasks”,
“ecs:ListTasks”,
“ecr:GetAuthorizationToken”,
“ecr:BatchCheckLayerAvailability”,
“ecr:GetDownloadUrlForLayer”,
“ecr:BatchGetImage”,
“logs:CreateLogStream”,
“logs:PutLogEvents”,
“ssm:GetParameters”
],
“Resource”: “*”
}
]
}
EOF
}
# ————————————————————#
# 任務角色策略關聯
# ————————————————————#
resource “aws_iam_role_policy_attachment” “ecs_task_execution_role_policy_attach” {
role = aws_iam_role.ecs_task_execution_role.name
policy_arn = aws_iam_policy.ecs_task_execution_policy.arn
}
# ————————————————————#
# 任務角色
# ————————————————————#
resource “aws_iam_role” “task_role” {
name = var.project
assume_role_policy = <<EOF
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Action”: “sts:AssumeRole”,
“Principal”: {
“Service”: “ecs-tasks.amazonaws.com”
},
“Effect”: “Allow”,
“Sid”: “”
}
]
}
EOF
}
# ————————————————————#
# SSM參數存儲策略
# ————————————————————#
resource “aws_iam_policy” “ssm_parameter_store_policy” {
name = var.project
description = “允許訪問SSM參數存儲”
policy = <<EOF
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: “ssm:GetParameters”,
“Resource”: “*”
}
]
}
EOF
}
# ————————————————————#
# SSM參數存儲策略關聯
# ————————————————————#
resource “aws_iam_role_policy_attachment” “ssm_policy_attach” {
role = aws_iam_role.task_role.name
policy_arn = aws_iam_policy.ssm_parameter_store_policy.arn
}
# ————————————————————#
# CloudWatch日誌組
# ————————————————————#
resource “aws_cloudwatch_log_group” “ecs_logs” {
name = “/ecs/handson”
retention_in_days = 14
}
# ————————————————————#
# 輸出
# ————————————————————#
output “alb_dns” {
value = aws_lb.main.dns_name
description = “DNS名稱”
}
可以通过在任务定义中按以下方式描述,除了ALB的健康检查之外,还可以描述容器的启动检查。
# ------------------------------------------------------------#
# Task Definition
# ------------------------------------------------------------#
resource "aws_ecs_task_definition" "main" {
## 省略...
"healthCheck": {
"command": ["CMD-SHELL", "curl -f http://localhost:8080/ || exit 1"],
"interval": 30,
"timeout": 5,
"retries": 3,
"startPeriod": 10
}
# ————————————————————#
# RDS构建
# ————————————————————#
resource “aws_db_parameter_group” “main” {
name = var.project
family = “mysql8.0″parameter {
name = “character_set_database”
value = “utf8mb4”
}
parameter {
name = “character_set_server”
value = “utf8mb4”
}
tags = {
Name = format(“%s-%s-aws-db-parameter-group”, var.environment, var.project)
}
}
# ————————————————————#
# RDS选项组
# ————————————————————#
resource “aws_db_option_group” “main” {
name = var.project
option_group_description = var.project
engine_name = “mysql”
major_engine_version = “8.0”
tags = {
Name = format(“%s-%s-aws-db-option-group”, var.environment, var.project)
}
}
# ————————————————————#
# RDS子网组
# ————————————————————#
resource “aws_db_subnet_group” “main” {
name = var.project
subnet_ids = [
aws_subnet.private[“1c”].id,
aws_subnet.private[“1d”].id,
]
tags = {
Name = format(“%s-%s-aws-db-subnet-group”, var.environment, var.project)
}
}
# ————————————————————#
# RDS安全组
# ————————————————————#
resource “aws_security_group” “rds” {
name = format(“%s-%s-aws-security-group-rds”, var.environment, var.project)
description = “允许3306端口的入站流量”
vpc_id = aws_vpc.main.id
tags = {
Name = format(“%s-%s-aws-security-group”, var.environment, var.project)
}
}
resource “aws_security_group_rule” “allow_ecs_mysql” {
security_group_id = aws_security_group.rds.id
type = “ingress”
from_port = 3306
to_port = 3306
protocol = “tcp”
source_security_group_id = aws_security_group.ecs.id
}
resource “aws_security_group_rule” “allow_ec2_mysql” {
security_group_id = aws_security_group.rds.id
type = “ingress”
from_port = 3306
to_port = 3306
protocol = “tcp”
source_security_group_id = aws_security_group.ssm.id
}
# ————————————————————#
# RDS实例
# ————————————————————#
resource “random_string” “db_password” {
length = 16
special = false
}
resource “aws_db_instance” “main” {
engine = “mysql”
engine_version = “8.0”
identifier = var.project
username = “admin”
password = random_string.db_password.result
skip_final_snapshot = true
instance_class = “db.t3.medium”
storage_type = “gp2”
allocated_storage = 20
storage_encrypted = false
multi_az = false
availability_zone = “ap-northeast-1d”
db_subnet_group_name = aws_db_subnet_group.main.name
vpc_security_group_ids = [aws_security_group.rds.id]
publicly_accessible = false
port = 3306
parameter_group_name = aws_db_parameter_group.main.name
option_group_name = aws_db_option_group.main.name
apply_immediately = true
performance_insights_enabled = true
tags = {
Name = format(“%s-%s-aws-db-instance”, var.environment, var.project)
}
}
# ————————————————————#
# 堡垒机EC2
# ————————————————————#
resource “aws_security_group” “ssm” {
name = format(“%s-%s-aws-security-group-ssm”, var.environment, var.project)
description = “用于SSM EC2实例的安全组”
vpc_id = aws_vpc.main.id
tags = {
Name = format(“%s-%s-aws-security-group-ssm”, var.environment, var.project)
}
}
resource “aws_security_group_rule” “ssm_egress” {
type = “egress”
from_port = 0
to_port = 0
protocol = “-1”
cidr_blocks = [“0.0.0.0/0”]
security_group_id = aws_security_group.ssm.id
}
resource “aws_security_group_rule” “ssm_ingress_ssh” {
type = “ingress”
from_port = 22
to_port = 22
protocol = “tcp”
cidr_blocks = var.cidr_blocks
security_group_id = aws_security_group.ssm.id
}
resource “aws_security_group_rule” “ssm_ingress_mysql” {
type = “ingress”
from_port = 3306
to_port = 3306
protocol = “tcp”
cidr_blocks = var.cidr_blocks
security_group_id = aws_security_group.ssm.id
}
resource “aws_iam_role” “rds_access” {
name = format(“%s-%s-aws_iam_role-rds_access”, var.environment, var.project)
assume_role_policy = jsonencode({
Version = “2012-10-17”
Statement = [
{
Effect = “Allow”
Principal = {
Service = “ec2.amazonaws.com”
}
Action = “sts:AssumeRole”
}
]
})
}
resource “aws_iam_role_policy” “rds_access” {
name = “RDSPolicy”
role = aws_iam_role.rds_access.id
policy = jsonencode({
Version = “2012-10-17”
Statement = [
{
Effect = “Allow”
Action = [
“rds:*”,
]
Resource = [“*”]
}
]
})
}
resource “aws_iam_role_policy_attachment” “ssm_managed_policy_attach” {
role = aws_iam_role.rds_access.name
policy_arn = “arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore”
}
resource “aws_iam_instance_profile” “rds_access” {
name = “RDSAccessProfile”
role = aws_iam_role.rds_access.name
}
resource “aws_instance” “ssm” {
ami = “ami-0947c48ae0aaf6781”
instance_type = “t2.micro”
iam_instance_profile = aws_iam_instance_profile.rds_access.name
subnet_id = aws_subnet.public[“1d”].id
vpc_security_group_ids = [aws_security_group.ssm.id]
associate_public_ip_address = true
key_name = “access_db”
user_data = <<-EOF
#!/bin/bash
sudo systemctl enable amazon-ssm-agent
sudo systemctl start amazon-ssm-agent
EOF
tags = {
Name = format(“%s-%s-aws-ssm-ec2-instance”, var.environment, var.project)
}
}
# ————————————————————#
# 输出
# ————————————————————#
output “DB_USER” {
value = “admin”
description = “数据库用户名”
}
output “DB_PASS” {
value = random_string.db_password.result
description = “数据库密码”
}
output “DB_HOST” {
value = aws_db_instance.main.address
description = “数据库端点”
}
output “DB_NAME” {
value = “golang”
description = “数据库名称”
}
output “DB_PORT” {
value = aws_db_instance.main.port
description = “数据库端口”
}
output “bastion_ec2_id” {
value = aws_instance.ssm.id
description = “堡垒机EC2实例ID”
}
最新的EC2的AMI已经预装了ssm agent,但尚未启动。因此,您可以通过在user_data中记录命令来在EC2启动时执行它。
resource "aws_instance" "ssm" {
ami = data.aws_ssm_parameter.ami.value
instance_type = "t2.micro"
iam_instance_profile = aws_iam_instance_profile.rds_access.name
subnet_id = aws_subnet.public["1d"].id
vpc_security_group_ids = [aws_security_group.ssm.id]
associate_public_ip_address = true
key_name = "access_db"
user_data = <<-EOF
#!/bin/bash
sudo systemctl enable amazon-ssm-agent
sudo systemctl start amazon-ssm-agent
EOF
tags = {
Name = format("%s-%s-aws-ssm-ec2-instance", var.environment, var.project)
}
}
# ————————————————————#
# Memcached安全组
# ————————————————————#
resource “aws_security_group” “memcached” {
name = format(“%s-%s-aws-security-group”, var.environment, var.project)
description = “允许11211端口的入站流量”
vpc_id = aws_vpc.main.id
tags = {
Name = format(“%s-%s-aws-security-group”, var.environment, var.project)
}
}
resource “aws_security_group_rule” “memcached” {
security_group_id = aws_security_group.memcached.id
type = “ingress”
from_port = 11211
to_port = 11211
protocol = “tcp”
source_security_group_id = aws_security_group.ecs.id
}
# ————————————————————#
# Memcached子网组
# ————————————————————#
resource “aws_elasticache_subnet_group” “memcached” {
name = format(“%s-%s-memcached-subnet-group”, var.environment, var.project)
subnet_ids = [
aws_subnet.private[“1c”].id,
aws_subnet.private[“1d”].id,
]
}
# ————————————————————#
# Memcached集群
# ————————————————————#
resource “aws_elasticache_cluster” “memcached” {
cluster_id = format(“%s-%s-memcached-cluster”, var.environment, var.project)
engine = “memcached”
node_type = “cache.t3.micro”
num_cache_nodes = 2
parameter_group_name = “default.memcached1.6”
subnet_group_name = aws_elasticache_subnet_group.memcached.name
security_group_ids = [aws_security_group.memcached.id]
}
# ————————————————————#
# 输出
# ————————————————————#
output “CACHE_HOST1” {
value = “${aws_elasticache_cluster.memcached.cache_nodes.0.address}:11211”
description = “缓存集群节点1的端点”
}
output “CACHE_HOST2” {
value = “${aws_elasticache_cluster.memcached.cache_nodes.1.address}:11211”
description = “缓存集群节点2的端点”
}
建设 ECR
首先是创建 ECR。ECR 是使用 AWS CLI 来构建资源。请在 中输入您自己的 AWS 账户 ID。此外,除了 #构建镜像 命令之外,您还可以在以下找到其他命令的确认。
- マネージメントコンソール > Amazon ECR > リポジトリ > go-dev-repo > プッシュコマンドの表示
# ECR作成
aws ecr create-repository --repository-name go-dev-repo
# ログイン
aws ecr get-login-password --region ap-northeast-1 | docker login --username AWS --password-stdin <account_id>.dkr.ecr.ap-northeast-1.amazonaws.com
# イメージビルド
docker build --no-cache --target runner -t go-dev-repo --platform linux/amd64 -f ./.docker/go/Dockerfile .
# タグ付け
docker tag go-dev-repo:latest <account_id>.dkr.ecr.ap-northeast-1.amazonaws.com/go-dev-repo:latest
# プッシュ
docker push <account_id>.dkr.ecr.ap-northeast-1.amazonaws.com/go-dev-repo:latest
系统管理器参数的初始值设定
接下来,我们将创建一个参数存储来存储在ECS任务上运行的应用程序所使用的环境变量。具体的值将在后续的工作中进行设置,所以在这里我们只暂时设置一个初始值。
# /envというパスに初期値を仮置き
aws ssm put-parameter \
--name "/env" \
--value "init" \
--type SecureString
执行Terraform
为了确保最低限度的安全性,本示例应用将对其可以访问的资源进行IP限制。请参考以下网站等获取全球IP地址,然后将其转录到variables.tf文件中。
variable "cidr_blocks" {
description = "List of CIDR blocks"
type = list(string)
default = ["<ご自身のグローバルIPを設定してください>/32"]
}
本文介绍了一种方法,即使用Docker来启动HashiCorp官方提供的Terraform镜像(版本1.5),而不是直接在本地计算机上安装Terraform。
# ローカルマシンのAWS認証キー格納先ディレクトリと設定ファイルをマウントしてコンテナ起動
docker run \
-v ~/.aws:/root/.aws \
-v $(pwd)/.infrastructure:/terraform \
-w /terraform \
-it \
--entrypoint=ash \
hashicorp/terraform:1.5
# 初期化
terraform init
# 差分検出
terraform plan
# コードを適用する
terraform apply -auto-approve
> aws_elasticache_cluster.memcached: Creation complete after 7m15s [id=dev-go-api-memcached-cluster]
> Apply complete! Resources: 69 added, 0 changed, 0 destroyed.
> Outputs:
# ここにParameter Storeに登録する環境変数の値が表示される
请在执行后将显示的输出结果进行备忘,或者保持终端标签页的打开状态。
使用系统管理器端口转发功能连接到RDS数据库。
使用Session Manager后,不再需要为跳板EC2实例分配公共IP地址或允许入站SSH连接。此外,无需生成和管理私钥,可以通过IAM策略进行访问管理,因此更加安全。
以前,使用SSM的方法只能转发EC2实例内正在监听的端口,
但是通过此次更新,现在也可以转发远程主机的端口。
# セッション開始
aws ssm start-session \
--target "<terraformコマンドで実行後に出力されるbastion_ec2_idを転記>" \
--document-name AWS-StartPortForwardingSessionToRemoteHost \
--parameters \
'{
"host": ["<terraformコマンドで実行後に出力されるDB_HOSTを転記>"],
"portNumber": ["3306"],
"localPortNumber":["3306"]
}'
# 以下が表示されたらタブは開いたままにする
> Waiting for connections...
# 別タブでDocker起動Docker起動
docker run --name mysql-client --rm -it mysql:8.0 /bin/bash
# MySQLクライアントで接続
mysql -h host.docker.internal -P 3306 -u admin -p
# パスワード入力
Enter password: <terraformコマンドで実行後に出力されるDB_PASSを転記>
# 初期クエリ
mysql> <.docker/mysql/init/1_create.sqlの内容をそのまま転記して実行>
将环境变量存储在系统管理器参数中
请将以下值直接存储在键名为“env”的环境变量中,而不是分别将其以键和值的形式存储为键值对。
在ecs.tf的任务定义文件中,将从Parameter中存储的值传递到.env文件中,方式如下所示。
# 省略
"secrets": [
{
"name": "ENV_FILE",
"valueFrom": "${data.aws_ssm_parameter.existing.arn}"
}
],
"command": ["/bin/sh", "-c", "printenv ENV_FILE > .env && ./main"],
# 省略
# Terraformでリソース作成が完了すると出力される
Apply complete! Resources: 69 added, 0 changed, 0 destroyed.
Outputs:
CACHE_HOST1 = "dev-go-api-memcached-cluster.xxxx.0001.apne1.cache.amazonaws.com:11211"
CACHE_HOST2 = "dev-go-api-memcached-cluster.xxxx.0002.apne1.cache.amazonaws.com:11211"
DB_HOST = "go-api.xxxxx.ap-northeast-1.rds.amazonaws.com"
DB_NAME = "golang"
DB_PASS = "xxxxxx"
DB_PORT = 3306
DB_USER = "admin"
# terraformコマンドで実行後に出力される内容を以下の形に修正して`env`にそのままの形で転記
# ダブルコロン""は不要なので注意すること
DB_USER=admin
DB_PASS=xxxxxxx
DB_HOST=go-api.xxxxxxx.ap-northeast-1.rds.amazonaws.com
DB_NAME=golang
DB_PORT=3306
CACHE_HOST1=go-api-dev-memcached-cluster.xxxxxxx.0001.apne1.cache.amazonaws.com:11211
CACHE_HOST2=go-api-dev-memcached-cluster.xxxxxxx.0002.apne1.cache.amazonaws.com:11211
部署
使用AWS CLI的update-service命令进行部署。执行命令后,若#在状态检查中显示为ACTIVE,则表示部署完成。大约需要3分钟的时间。
# デプロイ
aws ecs update-service --cluster go-api --service go-api --task-definition go-api --force-new-deployment
# ステータス確認
aws ecs describe-services --cluster go-api --services go-api --query 'services[*].status' --output text
> ACTIVE
确认动作
使用Terraform创建完成后输出的alb_dns。
# スロークエリ
curl http://go-api-xxxxxx.ap-northeast-1.elb.amazonaws.com/db/1
> {"id":1,"value":"Initial Value","sleepResult":0}%
# キャッシュ
curl http://go-api-xxxxxx.ap-northeast-1.elb.amazonaws.com/cache/1
> {"id":1,"value":"Initial Value","sleepResult":0}%
使用RDS Performance Insight进行性能检查。
RDS性能洞察是AWS提供的一项功能,用于可视化和诊断数据库的性能问题。它会分析SQL查询的负载分布,以便确定性能瓶颈。
在这里,我们将确认db.SQL.Queries.avg和db.SQL.Com_select.avg。
# スロークエリ
ab -n 30 -c 30 http://go-api-xxxxxx.ap-northeast-1.elb.amazonaws.com/db/1
> Benchmarking go-api-xxxxxx.ap-northeast-1.elb.amazonaws.com (be patient).....done
# キャッシュ
ab -n 30 -c 30 http://go-api-xxxxxx.ap-northeast-1.elb.amazonaws.com/cache/1
> Benchmarking go-api-xxxxxx.ap-northeast-1.elb.amazonaws.com (be patient).....done
管理控制台 > RDS > 数据库 > go-api > 监控 > 切换到新的监控视图 > 指标

当执行慢查询API时可以确认指标都在上升。然后执行缓存API时指标减少了。
删除资源
请确保在完成本篇文章的实践之后,务必进行资源清理。如果需要的话,请删除 ECR 和 SSM Parameter。
# コンテナを立ち上げる
docker run \
-v ~/.aws:/root/.aws \
-v $(pwd)/.infrastructure:/terraform \
-w /terraform \
-it \
--entrypoint=ash \
hashicorp/terraform:1.5
# 削除
terraform destroy