Filebeat + Elasticsearch + Kibana 轻量日志收集与展示系统
https://wzyboy.im/post/1111.html?utm_source=tuicool&utm_medium=referral
提到
beat -> logstash -> elk
可以
beat -> elk ingest plugs ( Elasticsearch Ingest Node )
与 /_template 类似,Ingest API 位于 /_ingest 下面。用户将 pipeline 定义提交之后,在 Beats 中即可指定某 pipeline 为数据预处理器。
已經內建了
https://www.elastic.co/guide/en/elasticsearch/plugins/6.5/ingest-geoip.html
https://www.elastic.co/guide/en/elasticsearch/plugins/6.5/ingest-user-agent.html
===============
.filebeat
filebeat.yml
補上 like example
output.elasticsearch:
hosts: ["http://localhost:9200/"]
pipelines:
- pipeline: nginx.access
when.equals:
fields.type: nginx.access
- pipeline: nginx.error
when.equals:
fields.type: nginx.error
OK, use bottom way to make pipeline.
.pipeline
https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html
https://qbox.io/blog/indexing-elastic-stack-5-0-ingest-apis
https://dev.classmethod.jp/server-side/elasticsearch/elasticsearch-ingest-node/
https://qbox.io/blog/how-to-index-geographical-location-of-ip-addresses-to-elasticsearch-5-0-1
Get a pipeline
GET _ingest/pipeline/geoippipeline
write a pipeline
PUT _ingest/pipeline/geoippipeline
{
"description" : "Add geoip information to the given IP address",
"processors": [
{
"geoip" : {
"field" : "ip",
"ignore_missing": true
}
},
{
"geoip" : {
"field" : "src_ip",
"ignore_missing": true
}
},
{
"geoip" : {
"field" : "clientip",
"ignore_missing": true
}
},
{
"set" : {
"field" : "location",
"value" : "{{geoip.location.lon}}, {{geoip.location.lat}}"
}
}
]
}
real use pipeline with test data, check is ok.
POST _ingest/pipeline/geoippipeline/_simulate
{
"docs":[
{
"_source": {
"ip": "8.8.0.0",
"src_ip": "8.8.0.0",
"clientip": "8.8.0.0"
}
}
]
}
Developer test
POST _ingest/pipeline/_simulate
{
"pipeline": {
"description" : "parse multiple patterns",
"processors": [
{
"geoip" : {
"field" : "ip",
"ignore_missing": true
}
},
{
"geoip" : {
"field" : "src_ip",
"ignore_missing": true
}
},
{
"geoip" : {
"field" : "clientip",
"ignore_missing": true
}
},
{
"set" : {
"field" : "location",
"value" : "{{geoip.location.lon}}, {{geoip.location.lat}}"
}
}
]
},
"docs":[
{
"_source": {
"ip": "8.8.0.0",
"src_ip": "8.8.0.0",
"clientip": "8.8.0.0"
}
}
]
}