sueboy: elk ingest plugs pipeline

Filebeat + Elasticsearch + Kibana 轻量日志收集与展示系统

https://wzyboy.im/post/1111.html?utm_source=tuicool&utm_medium=referral

提到

beat -> logstash -> elk

可以

beat -> elk ingest plugs ( Elasticsearch Ingest Node )

Elasticsearch Ingest Node 是 Elasticsearch 5.0 起新增的功能。在 Ingest Node 出现之前，人们通常会在 ES 前置一个 Logstash Indexer，用于对数据进行预处理。有了 Ingest Node 之后，Logstash Indexer 的大部分功能就可以被它替代了，grok, geoip 等 Logstash 用户所熟悉的处理器，在 Ingest Node 里也有。对于数据量较小的 ES 用户来说，省掉一台 Logstash 的开销自然是令人开心的，对于数据量较大的 ES 用户来说，Ingest Node 和 Master Node, Data Node 一样也是可以分配独立节点并横向扩展的，也不用担心性能瓶颈。

目前 Ingest Node 已支持数十种处理器，其中的 script 处理器具有最大的灵活性。

与 /_template 类似，Ingest API 位于 /_ingest 下面。用户将 pipeline 定义提交之后，在 Beats 中即可指定某 pipeline 为数据预处理器。

FROM docker.elastic.co/elasticsearch/elasticsearch-oss:6.4.2

已經內建了

https://www.elastic.co/guide/en/elasticsearch/plugins/6.5/ingest-geoip.html

https://www.elastic.co/guide/en/elasticsearch/plugins/6.5/ingest-user-agent.html

===============

．filebeat

filebeat.yml

補上 like example


output.elasticsearch:

  hosts: ["http://localhost:9200/"]

  pipelines:
    - pipeline: nginx.access
      when.equals:
        fields.type: nginx.access
    - pipeline: nginx.error
      when.equals:
        fields.type: nginx.error

OK, use bottom way to make pipeline.

．pipeline

https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html
https://qbox.io/blog/indexing-elastic-stack-5-0-ingest-apis
https://dev.classmethod.jp/server-side/elasticsearch/elasticsearch-ingest-node/
https://qbox.io/blog/how-to-index-geographical-location-of-ip-addresses-to-elasticsearch-5-0-1

Get a pipeline


GET _ingest/pipeline/geoippipeline

write a pipeline


PUT _ingest/pipeline/geoippipeline
{
  "description" : "Add geoip information to the given IP address",
  "processors": [
    {
      "geoip" :  {
        "field" : "ip",
        "ignore_missing": true
      }
    },
    {
      "geoip" :  {
        "field" : "src_ip",
        "ignore_missing": true
      }
    },
    {
      "geoip" :  {
        "field" : "clientip",
        "ignore_missing": true
      }
    },
    {
      "set" : {
        "field" : "location",
        "value" : "{{geoip.location.lon}}, {{geoip.location.lat}}"
      }
    }
  ]
}

real use pipeline with test data, check is ok.


POST _ingest/pipeline/geoippipeline/_simulate
{
  "docs":[
    {
      "_source": {
        "ip": "8.8.0.0",
        "src_ip": "8.8.0.0",
        "clientip": "8.8.0.0"
      }
    }
  ]
}

Developer test


POST _ingest/pipeline/_simulate
{
  "pipeline": {
  "description" : "parse multiple patterns",
  "processors": [
    {
      "geoip" :  {
        "field" : "ip",
        "ignore_missing": true
      }
    },
    {
      "geoip" :  {
        "field" : "src_ip",
        "ignore_missing": true
      }
    },
    {
      "geoip" :  {
        "field" : "clientip",
        "ignore_missing": true
      }
    },
    {
      "set" : {
        "field" : "location",
        "value" : "{{geoip.location.lon}}, {{geoip.location.lat}}"
      }
    }
  ]
},
"docs":[
  {
    "_source": {
      "ip": "8.8.0.0",
      "src_ip": "8.8.0.0",
      "clientip": "8.8.0.0"
    }
  }
  ]
}

EExcel 丞燕快速查詢2

elk ingest plugs pipeline