EFK架构收集JAVA日志的综合实例问题

来源：12-4 filter插件之grok简介（上）

苦瓜苦也

2020-02-28

单位JAVA日志格式:

2020-02-28 16:25:40.855 INFO  (OrderContorller.java:136) queryOrderDetail() - 调用订单查询接口接口入参返回：{"coupon":{"coupon_num":"JSHBXCGADF272053","status":"1","coupon_eff_date":"2020-03-28","coupon_exp_date":"2020-04-27"},"resultCode":"0000","resultDesc":"成功"}在这里输入代码

我设置的filebeat pipeline的流水线配置文件

{
  "description": "h5-jar-pipeline",
  "processors": [
    {
      "grok":{
        "field" : "message",
        "patterns" : ["(?<jartime>[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3})"]
      }
    },
    {
      "rename": {
        "field": "@timestamp",
        "target_field": "@read_timestamp"
      }
    },
    {
      "date": {
        "field": "jartime",
        "target_field": "@timestamp",
        "formats": ["YYYY-MM-dd HH:mm:ss.SSS"]
      }
    }
  ]
}

filebeat的配置文件

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/sys_total.log
  exclude_lines: ['qemu-ga','filebeat']
  tags: ["system"]

- type: log
  enabled: true
  paths:
    - /home/ncar/service/webapps/logs/ShengDaRedeemCode/ShengDaRedeemCode.log
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  tags: ["redeemcode"]

- type: log
  enabled: true
  paths:
    - /home/ncar/service/logs/start_info.log
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  tags: ["start"]

output.elasticsearch:
  hosts: ["http://192.168.167.234:9200","http://192.168.167.175:9200","http://192.168.167.204:9200"]
  pipeline: "h5-jar-pipeline"
  indices:
    - index: "h5-redeemcode-system-%{+YYYY.MM}"
      when.contains:
        tags: "system"
    - index: "h5-redeemcode-%{+YYYY.MM}"
      when.contains:
        tags: "redeemcode"
    - index: "h5-redeemcode-start-%{+YYYY.MM}"
      when.contains:
        tags: "start"

setup.ilm.enabled: false
setup.template.name: h5
setup.template.pattern: h5-*

问题是kibana显示不正确,这么设置索引不显示数据
图片描述

显示没有数据，但是其实是有数据的
GET /h5-redeemcode-2020.02/_search

  {
        "_index" : "h5-redeemcode-2020.02",
        "_type" : "_doc",
        "_id" : "7-PdinAB1F_ZvchlBYVU",
        "_score" : 1.0,
        "_source" : {
          "agent" : {
            "hostname" : "host-192-168-162-197",
            "id" : "33122428-91a4-4bf0-af52-30529373cb8b",
            "type" : "filebeat",
            "ephemeral_id" : "61bdfee3-6940-46b6-be98-d91781e19a97",
            "version" : "7.5.2"
          },
          "log" : {
            "file" : {
              "path" : "/home/ncar/service/webapps/logs/ShengDaRedeemCode/ShengDaRedeemCode.log"
            },
            "offset" : 137817
          },
          "message" : """2020-02-28 16:14:59.346 INFO  (OrderContorller.java:136) queryOrderDetail() - 调用订单查询接口接口入参返回：{"coupon":{"coupon_num":"JSHBXCFGFG877016","status":"1","coupon_eff_date":"2020-06-28","coupon_exp_date":"2020-07-27"},"resultCode":"0000","resultDesc":"成功"}""",
          "tags" : [
            "redeemcode"
          ],
          "input" : {
            "type" : "log"
          },
          "@read_timestamp" : "2020-02-28T08:15:04.900Z",
          "@timestamp" : "2020-02-28T16:14:59.346Z",
          "ecs" : {
            "version" : "1.1.0"
          },
          "host" : {
            "name" : "host-192-168-162-197"
          },
          "jartime" : "2020-02-28 16:14:59.346"
        }
      },

怎么解决这个问题老师？

写回答

1回答

rockybean

2020-02-28

已采纳

参考这个

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            """%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:level}%{SPACE}\(%{DATA:java_file}:%{INT:line_no}\)%{SPACE}%{DATA:method} -%{SPACE}%{DATA:description}：%{GREEDYDATA:response}"""
          ]
        }
      },
      {
        "date": {
          "field": "timestamp",
          "formats": [
            "yyyy-MM-dd HH:mm:ss.SSS"
          ],
          "timezone": "Asia/Shanghai"
        }
      },
      {
        "remove":{
          "field":"timestamp"
        }
      }
    ],
    "on_failure": [
      {
        "set": {
          "field": "pipeline_error",
          "value": "{{ _ingest.on_failure_message }}"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": """2020-02-28 16:25:40.855 INFO  (OrderContorller.java:136) queryOrderDetail() - 调用订单查询接口接口入参返回：{"coupon":{"coupon_num":"JSHBXCGADF272053","status":"1","coupon_eff_date":"2020-03-28","coupon_exp_date":"2020-04-27"},"resultCode":"0000","resultDesc":"成功"}在这里输入代码"""
      }
    }
  ]
}

苦瓜苦也

非常感谢！

2020-02-28

共1条回复

Elastic Stack从入门到实践，动手搭建数据分析系统

有了Elastic Stack，不用写一行代码，你也可以玩转大数据分析！

1361 学习 · 397 问题

查看课程

相似问题

日志文件数量庞大，使用elk框架收集有没有可行性

回答 1

【急】请老师帮忙EFK架构收入进Kibana时间显示乱序