请查阅使用AWS FireLens时，针对Fluent Bit（Fluentd）标签的命名规则
什么？ ?)
当使用Fluent Bit或Fluentd时，您需要在配置文件中编写使用标签进行路由的设置。
我想确认一下使用AWS FireLens后标签会发生什么变化。
AWS FireLens是一个用于容器日志的开放式规范，支持Fluent Bit和Fluentd。
AWS FireLens是一个在Amazon ECS上运行的日志驱动程序，用于将容器日志发送到Fluent Bit或Fluentd。
在这个机制中使用的Fluent Bit和Fluentd的设置将自动生成于任务定义（的容器定义）。然而，除此之外，您还可以包含自定义的配置文件。
创建使用FireLens配置的任务定义/指定自定义配置文件。
然而，在文件中并没有关于编写路由标签的描述。
在GitHub上查看了使用AWS FireLens的示例后，发现匹配规则通常以*或**为主，但是从中无法理解规则。
Amazon ECS FireLens 示例
虽然唯一、标签重写设置中有少量具体数值，但规则仍不清楚。
<match app**>
@type rewrite_tag_filter
当我调查原因时，发现在下一个博客中有关于标签信息的内容：“在生成的配置文件中进行日志标记”。
引擎内部：Amazon ECS 任务的 FireLens
以下内容如下所写。
容器的标准输出日志带有 <容器名称>-firelens-<任务ID> 的标签。因此，如果您的容器名称是app，任务ID为dcef9dee-d960-4af8-a206-46c31a7f1e67，标签将为app-firelens-dcef9dee-d960-4af8-a206-46c31a7f1e67。
换句话说，标签将变为[容器名称]-firelens-[任务ID]。
因此，请记住这个结论。
我来确认一下
所以，仅仅在这里结束还不够有趣，我想要实际确认一下标签。
由于准备自定义配置文件很麻烦，所以采用其他方法吧…
我们在这里使用Fluent Bit Plugin for CloudWatch Logs。
云监控日志的 Fluent Bit 插件
如果使用插件配置中的log_stream_prefix，日志流中可能会包含标签。
日志流前缀：（已弃用）日志流名称的前缀。将其设置为前缀- 相当于设置 log_stream_name = prefix-$(tag)。
目前AWS for Fluent Bit的版本是2.10.0，其中嵌入的Fluent Bit Plugin for CloudWatch Logs的版本是v1.6.0。
看到这段源代码，确定有使用标签的可能性。
e.stream = output.logStreamPrefix + e.Tag
那么，让我们使用log_stream_prefix进行确认。
顺便提一下，虽然写着log_stream_prefix已被弃用，但如果不指定的话，启动将会无法成功，所以我们还是使用log_stream_prefix。
2021-02-22T14:08:55.594000+00:00 fluentbit-log-stream/log_router/634a24ad44334f9aa9799f5b4e3257b1 time="2021-02-22T14:08:55Z" level=error msg="log_stream_name or log_stream_prefix is required"
环境
这次的环境是这个。
$ terraform version
Terraform v0.14.7
+ provider registry.terraform.io/hashicorp/aws v3.29.0
AWS的凭证将在环境变量中进行设置。
$ export AWS_ACCESS_KEY_ID=...
$ export AWS_SECRET_ACCESS_KEY=...
$ export AWS_DEFAULT_REGION=ap-northeast-1
“お題”的中文本地化翻譯如下：
题目
我将建立一个AWS Fargate集群，并在其中运行nginx和Fluent Bit容器。我将把Fluent Bit作为AWS FireLens日志驱动程序使用，将nginx的日志发送到Amazon CloudWatch Logs中。
我們要確認為Nginx準備的Amazon CloudWatch Logs的日誌流名稱。
我们将使用Terraform构建环境。
确认
首先，只摘取必要的最基本部分。
※最后将整体内容附上。
任务定义。
resource "aws_ecs_task_definition" "nginx" {
family = "nginx-task-definition"
cpu = "512"
memory = "1024"
network_mode = "awsvpc"
requires_compatibilities = ["FARGATE"]
execution_role_arn = aws_iam_role.ecs_task_execution_role.arn
task_role_arn = aws_iam_role.ecs_task_role.arn
container_definitions = <<JSON
[
{
"name": "log_router",
"image": "906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.10.1",
"essential": true,
"firelensConfiguration": {
"type": "fluentbit"
},
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "${aws_cloudwatch_log_group.fluentbit.name}",
"awslogs-region": "ap-northeast-1",
"awslogs-stream-prefix": "fluentbit-log-stream"
}
}
},
{
"name": "nginx",
"image": "nginx:1.19.7",
"essential": true,
"portMappings": [
{
"protocol": "tcp",
"containerPort": 80
}
],
"logConfiguration": {
"logDriver": "awsfirelens",
"options": {
"Name": "cloudwatch",
"region": "ap-northeast-1",
"log_group_name": "${aws_cloudwatch_log_group.nginx.name}",
"log_stream_prefix": "nginx-log-stream-",
"auto_create_group": "false"
}
}
}
]
JSON
}
nginx的日志将通过AWS FireLens传输到Amazon CloudWatch Logs中。
{
"name": "nginx",
"image": "nginx:1.19.7",
"essential": true,
"portMappings": [
{
"protocol": "tcp",
"containerPort": 80
}
],
"logConfiguration": {
"logDriver": "awsfirelens",
"options": {
"Name": "cloudwatch",
"region": "ap-northeast-1",
"log_group_name": "${aws_cloudwatch_log_group.nginx.name}",
"log_stream_prefix": "nginx-log-stream-",
"auto_create_group": "false"
}
}
}
由于设置了log_stream_prefix为nginx-log-stream-，所以流名称应该是nginx-log-stream-nginx-firelens-[任务ID]。
*容器名称为nginx。
以下是定义nginx日志输出目标日志组的位置。
resource "aws_cloudwatch_log_group" "nginx" {
name = "/fargate/containers/nginx"
}
我申请。
$ terraform apply
资源搭建完成后，同时监视日志组
$ aws logs tail --follow /fargate/containers/nginx
使用curl命令通过ALB访问nginx。
$ curl [ALBのDNS名]
这是生成的日志。
2021-02-22T15:38:15.159000+00:00 nginx-log-stream-nginx-firelens-6c820b75bc1648c1a7ca4b429c0e2711 {"container_id":"6c820b75bc1648c1a7ca4b429c0e2711-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/6c820b75bc1648c1a7ca4b429c0e2711","ecs_task_definition":"nginx-task-definition:15","log":"10.0.10.193 - - [22/Feb/2021:15:38:15 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T15:38:17.204000+00:00 nginx-log-stream-nginx-firelens-0beffa5df3b445f19fe658e9fdb2663c {"container_id":"0beffa5df3b445f19fe658e9fdb2663c-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/0beffa5df3b445f19fe658e9fdb2663c","ecs_task_definition":"nginx-task-definition:15","log":"10.0.10.193 - - [22/Feb/2021:15:38:17 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T15:38:19.249000+00:00 nginx-log-stream-nginx-firelens-6c820b75bc1648c1a7ca4b429c0e2711 {"container_id":"6c820b75bc1648c1a7ca4b429c0e2711-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/6c820b75bc1648c1a7ca4b429c0e2711","ecs_task_definition":"nginx-task-definition:15","log":"10.0.10.193 - - [22/Feb/2021:15:38:19 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T15:38:21.294000+00:00 nginx-log-stream-nginx-firelens-116c37fd39b84657874faf2eb3a96af2 {"container_id":"116c37fd39b84657874faf2eb3a96af2-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/116c37fd39b84657874faf2eb3a96af2","ecs_task_definition":"nginx-task-definition:15","log":"10.0.10.193 - - [22/Feb/2021:15:38:21 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T15:38:23.345000+00:00 nginx-log-stream-nginx-firelens-0beffa5df3b445f19fe658e9fdb2663c {"container_id":"0beffa5df3b445f19fe658e9fdb2663c-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/0beffa5df3b445f19fe658e9fdb2663c","ecs_task_definition":"nginx-task-definition:15","log":"10.0.10.193 - - [22/Feb/2021:15:38:23 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T15:38:25.390000+00:00 nginx-log-stream-nginx-firelens-116c37fd39b84657874faf2eb3a96af2 {"container_id":"116c37fd39b84657874faf2eb3a96af2-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/116c37fd39b84657874faf2eb3a96af2","ecs_task_definition":"nginx-task-definition:15","log":"10.0.10.193 - - [22/Feb/2021:15:38:25 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T15:38:27.785000+00:00 nginx-log-stream-nginx-firelens-116c37fd39b84657874faf2eb3a96af2 {"container_id":"116c37fd39b84657874faf2eb3a96af2-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/116c37fd39b84657874faf2eb3a96af2","ecs_task_definition":"nginx-task-definition:15","log":"10.0.10.193 - - [22/Feb/2021:15:38:27 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"ELB-HealthChecker/2.0\" \"-\"","source":"stdout"}
2021-02-22T15:38:27.785000+00:00 nginx-log-stream-nginx-firelens-0beffa5df3b445f19fe658e9fdb2663c {"container_id":"0beffa5df3b445f19fe658e9fdb2663c-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/0beffa5df3b445f19fe658e9fdb2663c","ecs_task_definition":"nginx-task-definition:15","log":"10.0.10.193 - - [22/Feb/2021:15:38:27 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"ELB-HealthChecker/2.0\" \"-\"","source":"stdout"}
2021-02-22T15:38:27.793000+00:00 nginx-log-stream-nginx-firelens-6c820b75bc1648c1a7ca4b429c0e2711 {"container_id":"6c820b75bc1648c1a7ca4b429c0e2711-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/6c820b75bc1648c1a7ca4b429c0e2711","ecs_task_definition":"nginx-task-definition:15","log":"10.0.10.193 - - [22/Feb/2021:15:38:27 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"ELB-HealthChecker/2.0\" \"-\"","source":"stdout"}
2021-02-22T15:38:29.475000+00:00 nginx-log-stream-nginx-firelens-0beffa5df3b445f19fe658e9fdb2663c {"container_id":"0beffa5df3b445f19fe658e9fdb2663c-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/0beffa5df3b445f19fe658e9fdb2663c","ecs_task_definition":"nginx-task-definition:15","log":"10.0.10.193 - - [22/Feb/2021:15:38:29 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T15:38:31.520000+00:00 nginx-log-stream-nginx-firelens-116c37fd39b84657874faf2eb3a96af2 {"container_id":"116c37fd39b84657874faf2eb3a96af2-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/116c37fd39b84657874faf2eb3a96af2","ecs_task_definition":"nginx-task-definition:15","log":"10.0.10.193 - - [22/Feb/2021:15:38:31 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T15:38:33.567000+00:00 nginx-log-stream-nginx-firelens-6c820b75bc1648c1a7ca4b429c0e2711 {"container_id":"6c820b75bc1648c1a7ca4b429c0e2711-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/6c820b75bc1648c1a7ca4b429c0e2711","ecs_task_definition":"nginx-task-definition:15","log":"10.0.10.193 - - [22/Feb/2021:15:38:33 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T15:38:35.606000+00:00 nginx-log-stream-nginx-firelens-116c37fd39b84657874faf2eb3a96af2 {"container_id":"116c37fd39b84657874faf2eb3a96af2-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/116c37fd39b84657874faf2eb3a96af2","ecs_task_definition":"nginx-task-definition:15","log":"10.0.10.193 - - [22/Feb/2021:15:38:35 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
2021-02-22T15:38:37.646000+00:00 nginx-log-stream-nginx-firelens-0beffa5df3b445f19fe658e9fdb2663c {"container_id":"0beffa5df3b445f19fe658e9fdb2663c-2531612879","container_name":"nginx","ecs_cluster":"nginx-cluster","ecs_task_arn":"arn:aws:ecs:ap-northeast-1:[AWSアカウントID]:task/nginx-cluster/0beffa5df3b445f19fe658e9fdb2663c","ecs_task_definition":"nginx-task-definition:15","log":"10.0.10.193 - - [22/Feb/2021:15:38:37 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.68.0\" \"aaa.bbb.ccc.ddd\"","source":"stdout"}
让我们专注于此时的日志流名称。
nginx-log-stream-nginx-firelens-6c820b75bc1648c1a7ca4b429c0e2711
nginx-log-stream-nginx-firelens-0beffa5df3b445f19fe658e9fdb2663c
nginx-log-stream-nginx-firelens-116c37fd39b84657874faf2eb3a96af2
6c820b75bc1648c1a7ca4b429c0e2711、0beffa5df3b445f19fe658e9fdb2663c、116c37fd39b84657874faf2eb3a96af2是任务ID，所以（desired_count设为3），格式为nginx-log-stream-nginx-firelens-[任务ID]（日志流前缀+标签）。
现在，我们确认完毕了。
赠品
最后，我会附上本次使用的资源定义的全部内容。
terraform {
required_version = "0.14.7"
required_providers {
aws = {
source = "hashicorp/aws"
version = "3.29.0"
}
}
}
provider "aws" {
}
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "2.71.0"
name = "my-vpc"
cidr = "10.0.0.0/16"
enable_dns_hostnames = true
enable_dns_support = true
azs = ["ap-northeast-1a", "ap-northeast-1c"]
public_subnets = ["10.0.10.0/24", "10.0.20.0/24"]
private_subnets = ["10.0.30.0/24", "10.0.40.0/24"]
map_public_ip_on_launch = false
enable_nat_gateway = true
single_nat_gateway = false
one_nat_gateway_per_az = false
}
module "load_balancer_sg" {
source = "terraform-aws-modules/security-group/aws//modules/http-80"
version = "3.18.0"
name = "load-balancer-sg"
vpc_id = module.vpc.vpc_id
ingress_cidr_blocks = ["0.0.0.0/0"]
}
module "nginx_service_sg" {
source = "terraform-aws-modules/security-group/aws"
version = "3.18.0"
name = "nginx-service-sg"
vpc_id = module.vpc.vpc_id
ingress_with_cidr_blocks = [
{
from_port = 80
to_port = 80
protocol = "tcp"
description = "nginx-service inbound ports"
cidr_blocks = "10.0.10.0/24"
},
{
from_port = 80
to_port = 80
protocol = "tcp"
description = "nginx-service inbound ports"
cidr_blocks = "10.0.20.0/24"
}
]
egress_with_cidr_blocks = [
{
from_port = 0
to_port = 0
protocol = "-1"
description = "nginx-service outbound ports"
cidr_blocks = "0.0.0.0/0"
}
]
}
module "load_balancer" {
source = "terraform-aws-modules/alb/aws"
version = "5.11.0"
name = "nginx-alb"
vpc_id = module.vpc.vpc_id
load_balancer_type = "application"
internal = false
subnets = module.vpc.public_subnets
security_groups = [module.load_balancer_sg.this_security_group_id]
target_groups = [
{
backend_protocol = "HTTP"
backend_port = 80
target_type = "ip"
health_check = {
interval = 20
}
}
]
http_tcp_listeners = [
{
port = 80
protocol = "HTTP"
}
]
}
locals {
vpc_id = module.vpc.vpc_id
private_subnets = module.vpc.private_subnets
nginx_service_security_groups = [module.nginx_service_sg.this_security_group_id]
load_balancer_target_group_arn = module.load_balancer.target_group_arns[0]
nginx_simple_container_definition = <<JSON
[
{
"name": "nginx",
"image": "nginx:1.19.7",
"essential": true,
"portMappings": [
{
"protocol": "tcp",
"containerPort": 80
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "${aws_cloudwatch_log_group.nginx.name}",
"awslogs-region": "ap-northeast-1",
"awslogs-stream-prefix": "nginx-log-stream"
}
}
}
]
JSON
nginx_with_fluentbit_container_definitions = <<JSON
[
{
"name": "log_router",
"image": "906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.10.1",
"essential": true,
"firelensConfiguration": {
"type": "fluentbit",
"options":{
"enable-ecs-log-metadata": "true"
}
},
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "${aws_cloudwatch_log_group.fluentbit.name}",
"awslogs-region": "ap-northeast-1",
"awslogs-stream-prefix": "fluentbit-log-stream"
}
}
},
{
"name": "nginx",
"image": "nginx:1.19.7",
"essential": true,
"portMappings": [
{
"protocol": "tcp",
"containerPort": 80
}
],
"logConfiguration": {
"logDriver": "awsfirelens",
"options": {
"Name": "cloudwatch",
"region": "ap-northeast-1",
"log_group_name": "${aws_cloudwatch_log_group.nginx.name}",
"log_stream_prefix": "nginx-log-stream-",
"auto_create_group": "false"
}
}
}
]
JSON
}
data "aws_iam_policy_document" "ecs_assume_role" {
statement {
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["ecs-tasks.amazonaws.com"]
}
}
}
data "aws_iam_policy" "ecs_task_execution_role_policy" {
arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}
resource "aws_iam_role" "ecs_task_execution_role" {
name = "MyEcsTaskExecutionRole"
assume_role_policy = data.aws_iam_policy_document.ecs_assume_role.json
}
resource "aws_iam_role_policy_attachment" "ecs_task_execution_role_policy_attachment" {
role = aws_iam_role.ecs_task_execution_role.name
policy_arn = data.aws_iam_policy.ecs_task_execution_role_policy.arn
}
data "aws_iam_policy_document" "ecs_task_role_policy_document" {
statement {
effect = "Allow"
actions = [
"logs:DescribeLogStreams",
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
]
resources = ["*"]
}
}
resource "aws_iam_policy" "ecs_task_role_policy" {
name = "MyEcsTaskPolicy"
policy = data.aws_iam_policy_document.ecs_task_role_policy_document.json
}
resource "aws_iam_role" "ecs_task_role" {
name = "MyEcsTaskRole"
assume_role_policy = data.aws_iam_policy_document.ecs_assume_role.json
}
resource "aws_iam_role_policy_attachment" "ecs_task_role_policy_attachment" {
role = aws_iam_role.ecs_task_role.name
policy_arn = aws_iam_policy.ecs_task_role_policy.arn
}
resource "aws_cloudwatch_log_group" "nginx" {
name = "/fargate/containers/nginx"
}
resource "aws_cloudwatch_log_group" "fluentbit" {
name = "/fargate/containers/fluentbit"
}
resource "aws_ecs_cluster" "nginx" {
name = "nginx-cluster"
}
resource "aws_ecs_task_definition" "nginx" {
family = "nginx-task-definition"
cpu = "512"
memory = "1024"
network_mode = "awsvpc"
requires_compatibilities = [
"FARGATE"]
execution_role_arn = aws_iam_role.ecs_task_execution_role.arn
task_role_arn = aws_iam_role.ecs_task_role.arn
# container_definitions = local.nginx_simple_container_definition
container_definitions = local.nginx_with_fluentbit_container_definitions
}
resource "aws_ecs_service" "nginx" {
name = "nginx-service"
cluster = aws_ecs_cluster.nginx.arn
task_definition = aws_ecs_task_definition.nginx.arn
desired_count = 3
launch_type = "FARGATE"
platform_version = "1.4.0"
deployment_minimum_healthy_percent = 50
network_configuration {
assign_public_ip = false
security_groups = local.nginx_service_security_groups
subnets = local.private_subnets
}
load_balancer {
target_group_arn = local.load_balancer_target_group_arn
container_name = "nginx"
container_port = 80
}
}
output "alb_arn" {
value = module.load_balancer.this_lb_arn
}
output "alb_dns_name" {
value = module.load_balancer.this_lb_dns_name
}
