logstash kibana geth log ethereum Grok Constructor

filter


json {
  source => "message"
}

This mean is Try to use json format transfer log, then put some data to message filed. So some filed just be setting, and some data set to message.


．Use this to check mach and log
https://grokconstructor.appspot.com/do/match
https://blog.johnwu.cc/article/elk-logstash-grok-filter.html
https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns


This is geth log for example

A：
INFO [11-14|09:58:17.730] Generating DAG in progress epoch=1 percentage=99 elapsed=4m8.643s
INFO [11-15|01:41:33.455] Generating DAG in progress               epoch=1 percentage=9  elapsed=27.614s

B：
INFO [11-15|01:19:44.590] Loaded most recent local fast block      number=0 hash=656134…58fded td=1 age=49y7mo1h, Loaded most recent local fast block

C：
INFO [11-15|02:09:27.980] 🔨 mined potential block number=119 hash=ebaa58…5d8fa1, 🔨 mined potential block

A：

INFO [11-14|09:58:17.730] Generating DAG in progress epoch=1 percentage=99 elapsed=4m8.643s
INFO [11-15|01:41:33.455] Generating DAG in progress               epoch=1 percentage=9  elapsed=27.614s

%{DATA:logType} %{DATA:MONTHDAY} %{GREEDYDATA:message}\s+epoch=(?<epoch>\b\w+\b) percentage=(?<percentage>\b\w+\b)\s+elapsed=(?<elapsed>\b\w+\b)

B：

INFO [11-15|01:19:44.590] Loaded most recent local fast block      number=0 hash=656134…58fded td=1 age=49y7mo1h, Loaded most recent local fast block  

%{DATA:logType} %{DATA:MONTHDAY} %{DATA:message} number=(?<minedNumber>\b\w+\b) hash=(?<minedHashr>\b\w+...\w+\b) td=(?<minedtd>\b\w+\b) age=(?<minedtd>\b\w+\b)%{DATA:message2}

C：

INFO [11-15|02:09:27.980] 🔨 mined potential block number=119 hash=ebaa58…5d8fa1, 🔨 mined potential block

OK~ C is best easy. No any other special. Only need to check is Space. log have Space, rule must have Space. And Space must same count. Have one Space in log, rule must Have Space.

B is Data same and tail how to do.

%{DATA:message}
%{DATA:message2}

let two "Loaded most recent local fast block" to DATA & message、message2


B have one thing is C Space must same. Here firest %{DATA:message} Data：Loaded most recent local fast block.....   This is all Space in message. So %{DATA:message} & number= have space or not, just to test check. Don't think too much.


A \s+ is different. This is for some data have space, but log look just same. So use this \s+ for have more space. Remeber \s+epoch=  no space rule, NOT \s+ epoch= .  Only little different. Just to test check. Don't think too much.




All Architecture is like this

docker-compose  & elk
https://sueboy.blogspot.com/2018/11/docker-compose-ethereum-geth-private.html

Change logstach pipline -> logstash.log

Now very clear.

So some different is add_field that is for check grok is work ok or not. If kibana have value = grok is work.

add_field marked filed just for test, can open then get double smae filed and value.

Grok can multiple.