使用curl将数据注册到Elasticsearch的安装和Kibana及Elasticsearch的安装
首先
因为工作中引起了话题，所以我想要尝试一下Elasticsearch和Kibana，从安装到简单的数据输入都要试试看。
请参考
-
- Elasticsearchリファレンス [5.4] » 始めてみよう » インストール
-
- JDK Installation for Linux Platforms
-
- [Linux]PATHの設定方法
-
- ElasticSearch6.4.1とkibanaをインストールしてみた
-
- Elasticsearchのインストールからデータ投入まで
-
- Elasticsearch 6 を利用する前に把握しておいた方がよさそうなこと
-
- Elasticsearch 超入門
-
- Elasticsearch マッピング
- Elasticsearchのcurlを使ったindexの削除
环境
-
- Amazon Linux 2 AMI (HVM), SSD Volume Type, 64ビット (x86)
-
- t2.medium(メモリ4GiB, vCPU 2 core)
- SSD 10GiB
操作步骤
安装Java8
由於這次是基於rpm的系統，所以我們將使用rpm進行安裝。您可以使用以下指令來檢查rpm是否可用。
# rpm --version
RPM version 4.11.3
创建目录/usr/java，并将下载的jdk-8u191-linux-x64.rpm放置在此处。
使用rpm -ivh jdk-8u191-linux-x64.rpm进行安装。
安装路径将为/usr/java/jdk1.8.0_191-amd64。
我会检查版本，包括确认密码是否正确。
# java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
将JAVA_HOME设置为/usr/java/jdk1.8.0_191-amd64来配置路径。
可以使用echo $JAVA_HOME来确认路径是否已经配置成功。
安装Elasticsearch
我将添加Yum软件仓库。
# vi /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
可以通过yum -y install elasticsearch进行安装。
安装完成后会显示以下内容。
Installing : elasticsearch-6.5.4-1.noarch 1/1
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
sudo systemctl start elasticsearch.service
Created elasticsearch keystore in /etc/elasticsearch
Verifying : elasticsearch-6.5.4-1.noarch 1/1
Installed:
elasticsearch.noarch 0:6.5.4-1
Complete!
Elasticsearch的配置和启动
設定可以訪問Elasticsearch的IP。
#vi /etc/elasticsearch/elasticsearch.yml
# network.host: 192.168.0.1 # コメントアウト
network.host: 0.0.0.0 # 追加
设定内存使用量。因为内存较少，所以设定为512MB。
$ vi /etc/elasticsearch/jvm.options
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
#-Xms1g # コメントアウト
#-Xmx1g # コメントアウト
-Xms512m # 追加
-Xmx512m # 追加
可以使用systemctl start elasticsearch来启动。
可以用curl来确认是否已经启动。
# curl localhost:9200
{
"name" : "j9pM6nO",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "nu9-DxInSaufjNu-JHCsIg",
"version" : {
"number" : "6.5.4",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "d2ef93d",
"build_date" : "2018-12-17T21:17:40.758843Z",
"build_snapshot" : false,
"lucene_version" : "7.5.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
Kibana的安装
添加 yum 软件仓库。
#vi /etc/yum.repos.d/kibana.repo
[kibana-6.x]
name=Kibana repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
使用yum -y install kibana命令来进行安装。
# yum -y install kibana
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Resolving Dependencies
--> Running transaction check
---> Package kibana.x86_64 0:6.5.4-1 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
kibana x86_64 6.5.4-1 elasticsearch-6.x 200 M
Transaction Summary
================================================================================
Install 1 Package
Total download size: 200 M
Installed size: 480 M
Downloading packages:
kibana-6.5.4-x86_64.rpm | 200 MB 00:28
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : kibana-6.5.4-1.x86_64 1/1
Verifying : kibana-6.5.4-1.x86_64 1/1
Installed:
kibana.x86_64 0:6.5.4-1
Complete!
Kibana的设置和启动
进行IP连接设置。
vi /etc/kibana/kibana.yml
# Specifies the address to which the Kibana server will bind. IP addresses and h
ost names are both valid values.
# The default is 'localhost', which usually means remote machines will not be ab
le to connect.
# To allow connections from remote users, set this parameter to a non-loopback a
ddress.
#server.host: "localhost"
server.host: "0.0.0.0" #追加
# The URL of the Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://localhost:9200" # コメントアウト解除
使用systemctl start kibana命令启动kibana。
检查是否处于活动状态。
# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-01-02 17:58:34 JST; 1h 25min ago
Docs: http://www.elastic.co
Main PID: 3283 (java)
CGroup: /system.slice/elasticsearch.service
tq3283 /bin/java -Xms512m -Xmx512m -XX:+UseConcMarkSweepGC -XX:CMS...
mq3411 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x...
Jan 02 17:58:34 ip-172-31-25-200.ap-northeast-1.compute.internal systemd[1]: ...
Jan 02 17:58:34 ip-172-31-25-200.ap-northeast-1.compute.internal systemd[1]: ...
Hint: Some lines were ellipsized, use -l to show in full.
从浏览器中访问http://ipaddress:5601。
安装kuromoji
安装插件以便能够使用日语。
# /usr/share/elasticsearch/bin/elasticsearch-plugin install analysis-kuromoji
-> Downloading analysis-kuromoji from elastic
[=================================================] 100%??
-> Installed analysis-kuromoji
我将确认是否已安装。
# /usr/share/elasticsearch/bin/elasticsearch-plugin list | grep analysis-kuromoji
analysis-kuromoji
数据注册
根据Elasticsearch的安装步骤和简单使用方法，尝试投入数据。
创建索引
在RDB的数据库中，相当于Database的东西被称为索引。创建索引。
# curl -X PUT "http://127.0.0.1:9200/sample_index"
{"acknowledged":true,"shards_acknowledged":true,"index":"sample_index"}
确认指数是否正确。
检查是否存在创建的索引。如果添加“pretty”选项，将对JSON进行格式化并输出。
通过查看输出，可以确认之前创建的“sample_index”（以及Kibana的样本数据“.kibana_1”）已经注册。
另外，还显示了别名。别名是可以附加到索引上的替代名称，用于在搜索数据时以别名方式批量作为搜索目标等用途。
# curl http://127.0.0.1:9200/_aliases?pretty
{
".kibana_1" : {
"aliases" : {
".kibana" : { }
}
},
"sample_index" : {
"aliases" : { }
}
}
查看样本索引的详细信息
将显示索引的每个设置。
# curl http://127.0.0.1:9200/sample_index/_settings?pretty
{
"sample_index" : {
"settings" : {
"index" : {
"creation_date" : "1546437654399",
"number_of_shards" : "5",
"number_of_replicas" : "1",
"uuid" : "HeBhtAVkSlWPaQQhcNAzCw",
"version" : {
"created" : "6050499"
},
"provided_name" : "sample_index"
}
}
}
}
数据注册
在RDB中，需要预先定义表的结构，如列名和数据类型。
而在Elasticsearch中，会自动根据输入的数据定义结构。在Elasticsearch中，称这个定义的结构为映射。当然，也可以预先自己定义映射。
在RDB数据库的索引”sample_index”下创建一个对应RDB表的”类型”为”doc01″的文档，并将下面的数据插入到RDB记录中。/sample_index/doc01/1中的末尾的”1″是文档的ID。如果没有指定，将自动分配。
自从5系版本之后，Content-Type头部变成了必需项。
# curl -H "Content-Type: application/json" -X PUT "http://127.0.0.1:9200/sample_index/doc01/1" -d '{
"title" : "Sample No.1",
"description" : "This is a sample data",
"tags" : ["Elasticsearch", "Search-Engine"],
"date" : "2018-07-20T06:30:00",
"no" : 82,
"ratio": 0.73,
"enabled": true
}'
{"_index":"sample_index","_type":"doc01","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}
确认映射
我来确认一下地图。
# curl "http://127.0.0.1:9200/sample_index/_mapping/doc01?pretty"
{
"sample_index" : {
"mappings" : {
"doc01" : {
"properties" : {
"date" : {
"type" : "date"
},
"description" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"enabled" : {
"type" : "boolean"
},
"no" : {
"type" : "long"
},
"ratio" : {
"type" : "float"
},
"tags" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"title" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}
查询数据
我会确认已注册的数据。
# curl "http://127.0.0.1:9200/sample_index/doc01/1?pretty"
{
"_index" : "sample_index",
"_type" : "doc01",
"_id" : "1",
"_version" : 1,
"found" : true,
"_source" : {
"title" : "Sample No.1",
"description" : "This is a sample data",
"tags" : [
"Elasticsearch",
"Search-Engine"
],
"date" : "2018-07-20T06:30:00",
"no" : 82,
"ratio" : 0.73,
"enabled" : true
}
}
第二轮数据登记
我将尝试登记下方的数据。
curl -H "Content-Type: application/json" -X PUT "http://127.0.0.1:9200/sample_index/doc01/2" -d '{
"title" : "Sample No.2",
"description" : "This is a sample data",
"tags" : ["Elasticsearch", "Search-Engine","Database"],
"date" : "2018-07-21T06:40:00",
"no" : 89,
"ratio": 0.61,
"enabled": true,
"name": "4/2"
}'
{"_index":"sample_index","_type":"doc01","_id":"2","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}
我来确认一下映射。已经添加了“name”。
# curl "http://127.0.0.1:9200/sample_index/_mapping/do01?pretty"
{
"sample_index" : {
"mappings" : {
"doc01" : {
"properties" : {
"date" : {
"type" : "date"
},
"description" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"enabled" : {
"type" : "boolean"
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"no" : {
"type" : "long"
},
"ratio" : {
"type" : "float"
},
"tags" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"title" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}
我尝试将不同于映射类型的数据进行注册。让我们将”no”设为字符串类型。
结果将出现以下错误。
curl -H "Content-Type: application/json" -X PUT "http://127.0.0.1:9200/sample_index/doc01/3" -d '{
"title" : "Sample No.3",
"description" : "This is a sample data",
"tags" : ["Elasticsearch", "Search-Engine","Database"],
"date" : "2018-07-21T06:40:00",
"no" : "ABCDE",
"ratio": 0.61,
"enabled": true,
"name": "4/2"
}'
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse field [no] of type [long]"}],"type":"mapper_parsing_exception","reason":"failed to parse field [no] of type [long]","caused_by":{"type":"illegal_argument_exception","reason":"For input string: \"ABCDE\""}},"status":400}
从以上内容可以得出，可以对项目进行添加和修改，但一旦注册数据类型则不能更改。
用手动方式创建地图定义。
在创建新的索引的同时定义映射。
curl -H "Content-Type: application/json" -X PUT "http://127.0.0.1:9200/sample_index2" -d '{
"mappings" : {
"doc02" : {
"properties" : {
"date" : {
"type" : "date"
},
"description" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"enabled" : {
"type" : "boolean"
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"no" : {
"type" : "long"
},
"ratio" : {
"type" : "float"
},
"tags" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"title" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}'
{"acknowledged":true,"shards_acknowledged":true,"index":"sample_index2"}
我会确认地图的准确性。
# curl "http://127.0.0.1:9200/sample_index2/_mapping/doc02?pretty"
{
"sample_index2" : {
"mappings" : {
"doc02" : {
"properties" : {
"date" : {
"type" : "date"
},
"description" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"enabled" : {
"type" : "boolean"
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"no" : {
"type" : "long"
},
"ratio" : {
"type" : "float"
},
"tags" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"title" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}
数据搜索 de
您可以通过“search?q=标签名称:值”来发送搜索查询。
# curl "http://127.0.0.1:9200/sample_index/doc01/_search?q=tags:Database&pretty=true"
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 0.2876821,
"hits" : [
{
"_index" : "sample_index",
"_type" : "doc01",
"_id" : "2",
"_score" : 0.2876821,
"_source" : {
"title" : "Sample No.2",
"description" : "This is a sample data",
"tags" : [
"Elasticsearch",
"Search-Engine",
"Database"
],
"date" : "2018-07-21T06:40:00",
"no" : 89,
"ratio" : 0.61,
"enabled" : true,
"name" : "4/2"
}
}
]
}
}
搜索API可以参考Elasticsearch超级入门。
删除索引
删除索引表。
# curl -XDELETE localhost:9200/sample_index?pretty=true
最后
我希望能在另一篇文章中详细介绍如何进行准确数据的录入，并在Kibana上进行可视化。
