利用logstash6.4.2监控access访问日志并切割,使用geoip插件分析ip地址在kibana控制台形成用户热点地图

作者: admin 分类: ELK 发布时间: 2019-05-09 12:48  阅读: 485 views

程序猿除了要会记录日志,还要会分析日志,不仅有利于异常的解决与分析,还能从其他层次对业务或性能做调整。

线上服务器的运行日志一大堆,业务日志已经被filebeat监控并用sentinl发送告警邮件,打点日志已用logstash监控并grok切割统计用户操作行为,access访问日志每天产生几百兆用来干什么呢?

tomcat服务器配置的access日志格式如下:

100.117.117.xx|175.19.41.41|0|2019-05-07 18:23:56|GET|200|56|/busines/crowd/type?client=ios&clientversion=6.4.3&deviceid=528980&t=1557224636.271672/6.4.3 (iPhone; iOS 12.1.4; Scale/3.00)|7D70B434FAA00E6DE050190AFD014926 BAB716F7FA2CF5B8E2B967B6ADAE6DD6

服务器IP|客户端IP|时间|请求类型|状态码|xxx|xxx|xxx|xxx

重要的分两部分,1是ip地址,2是访问入口。由于访问入口有单独的记录。所以这里利用ip地址分析时间段内的区域访问热点。

参考地址:

https://www.elastic.co/guide/en/kibana/current/maps.html
https://www.jianshu.com/p/07b82092d4af

 

一、先下载ip解析查询库到logstash服务器上(主要用来解析ip的国家、区域、城市、坐标等)

下载命令(也可以手动下载,上传至服务器)

wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz

出现以下信息表示下载完成

--2019-05-07 15:02:55--  http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
Resolving geolite.maxmind.com (geolite.maxmind.com)... 104.17.200.89, 104.17.201.89, 2606:4700::6811:c959, ...
Connecting to geolite.maxmind.com (geolite.maxmind.com)|104.17.200.89|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 29201876 (28M) [application/octet-stream]
Saving to: ‘GeoLite2-City.mmdb.gz’

100%[========================================================================================================================================>] 29,201,876  6.79MB/s   in 5.0s

2019-05-07 15:03:00 (5.56 MB/s) - ‘GeoLite2-City.mmdb.gz’ saved [29201876/29201876]

 

二、修改logstash的配置文件(主要监控、切割、解析日志)

配置如下:

input {
  beats {
    port => 5044
  }
  file { //监控accesslog日志
    path => ["/home/admin/logs/accesslog/access_log.2018-10-01"]
  }
}

filter {
  grok {
    match => {//解析日志中的服务器地址和客户端地址并起名
      "message" => "%{IPORHOST:innerip}\|%{IPV4:clientip}"
    }
  }

  geoip {//解析ip的坐标、省市、国家
    source => "clientip"
    target => "geoip"//从指定的database地址将clientip进行解析
    database => "/home/admin/logstash/config/GeoLite2-City.mmdb"
    add_field => ["[geoip][coordinates]","%{[geoip][longitude]}"]
    add_field => ["[geoip][coordinates]","%{[geoip][latitude]}"]
  }
}

output {//输出到es
  elasticsearch {
    hosts => ["http://xxx.xx.xxx.xx:9200"]
    index => "access-map-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "changeme"
  }
}

修改之后保存, 并运行logstash安装目录/bin/logstash -f  logstash安装目录/config/logstash-sample.conf,出现以下错误

[admin@service2-c bin]$ tailf -300 nohup.out
could not find java; set JAVA_HOME or ensure java is in PATH
^C 

原因:服务器有jdk但是没有配置环境变量。

处理:单独给logstash启动文件增加环境变量,如下:

这时需要在 logstash/bin/logstash文件头部添加jdk的指定版本,文件编辑如下:

#!/bin/bash
# Run logstash from source
# This is most useful when done from a git checkout.
#
# Usage:
#   bin/logstash <command> [arguments]
#
# See 'bin/logstash --help' for a list of commands.
#
# Supported environment variables:
#   LS_JAVA_OPTS="xxx" to append extra options to the JVM options provided by logstash
#
# Development environment variables:
#   DEBUG=1 to output debugging information
export JAVA_CMD="/opt/java/jdk1.8.0_152/bin"
export JAVA_HOME="/opt/java/jdk1.8.0_152/"

unset CDPATH
# This unwieldy bit of scripting is to try to catch instances where Logstash
# was launched from a symlink, rather than a full path to the Logstash binary 

这个配置好之后,可以正常启动logstash。关于logstash的jvm.options配置根据系统环境进行调整。我这里配置了512m,运行正常。当启动之后,且有正常的accesslog日志追加进来,就可以在elasticsearch的运行日志中看到如下信息:

[2019-05-07T16:08:52,513][INFO ][o.e.c.m.MetaDataMappingService] [wqbVDEq] [access-map-2019.05.07/s8sdiValSG29g46_8M__wQ] create_mapping [doc]
[2019-05-07T16:08:52,516][INFO ][o.e.c.m.MetaDataMappingService] [wqbVDEq] [access-map-2019.05.07/s8sdiValSG29g46_8M__wQ] update_mapping [doc]
[2019-05-07T16:08:52,547][INFO ][o.e.c.m.MetaDataMappingService] [wqbVDEq] [access-map-2019.05.07/s8sdiValSG29g46_8M__wQ] update_mapping [doc]
[2019-05-07T16:08:52,974][INFO ][o.e.c.m.MetaDataMappingService] [wqbVDEq] [access-map-2019.05.07/s8sdiValSG29g46_8M__wQ] update_mapping [doc]

索引文件被正常的创建、修改。针对于不同环境的日志文件,可能生成规则、格式不一样,需要对logstash的input配置做调整。如下:是某一目录下每天根据时间生成带当天时间的日志监控方式。【也可以每天在access.log文件记录日志,对历史日志进行归档。这里的配置目录就是  /home/admin/logs/access.log】

input {
  beats {
    port => 5044
  }
  file {
    path => ["/home/admin/logs/access_log.2019*"]
  }
} 

一切配置成功后,就可以在kibana中看到 access-map-日期的索引文件了。制作对应的Index Patterns

 

三、开始绘制ip地图【可视化-坐标地图】

可以看到在即将选择聚合函数时,提示错误”No Compatible Fields: The “access-map*” index pattern does not contain any of the following field types: geo_point”, 说明是在access-map*”索引中没有所需要的geo_point类型数据。geo_point是种坐标类型的数据,是在创建access-map*索引的时候系统自动将一些字典设置为了float或者text类型。所以,我们这里要创建索引模板,如下:

PUT _template/access-map*
{
  "index_patterns": "access-map*",
  "index.mapping.ignore_malformed": true,
  "order": 0,
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 0
  },
  "mappings": {
    "_doc": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "@version": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "clientip": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "geoip": {
          "properties": {
            "city_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "continent_code": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "coordinates": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "country_code2": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "country_code3": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "country_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "dma_code": {
              "type": "long"
            },
            "ip": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "latitude": {
              "type": "geo_point"
            },
            "location": {
              "properties": {
                "lat": {
                  "type": "geo_point"
                },
                "lon": {
                  "type": "geo_point"
                }
              }
            },
            "longitude": {
              "type": "geo_point"
            },
            "postal_code": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "region_code": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "region_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "timezone": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "host": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "innerip": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "message": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "path": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "tags": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
}

停止logstash,删除access-map.2019-05-07索引文件,新做access-map*索引模板。重新启动logstash,新日志写入时报错如下:

[2019-05-07T16:55:38,315][WARN ][logstash.outputs.elasticsearch] 
Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"access-map-2019.05.07", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0xaaab67b>], :response=>{"index"=>{"_index"=>"access-map-2019.05.07", "_type"=>"doc", "_id"=>"LAOBkWoBHXu_Glm3aII1", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"parse_exception", "reason"=>"geo_point expected"}}}}}

原来 geo_point是个坐标类的数据格式,系统默认映射将x轴、y轴的数据设置为了locations的两个字段。我取的默认配置如下:

"location": {
   "properties": {
     "lat": {
        "type": "float"
      },
      "lon": {
        "type": "float"
      }
   }
},

第一次修改如下:【这个是错误的,因为geo_point的数据格式不匹配

"location": {
    "properties": {
       "lat": {
           "type": "geo_point"
        },
       "lon": {
           "type": "geo_point"
        }
    }
},

第二次修改如下:

"location": {
    "type": "geo_point"
 },

重新运行,出错如下:

Rejecting mapping update to [access-map-2019.05.07] as the final mapping would have more than 1 type: [_doc, doc]"}}}}

原来在创建映射模板时,index的type修改了下,导致出现了_doc和doc两种。所以出现了上面的错误。设置模板为初次创建时的类型,可以正常建立索引。最后通过logstash的geoip插件处理后,每条日志记录到es中的最终格式如下:

{
  "_index": "access-map-2019.05.07",
  "_type": "doc",
  "_id": "RgXSkWoBHXu_Glm3R1DX",
  "_version": 1,
  "_score": null,
  "_source": {
    "innerip": "100.117.xxx.xx",
    "host": "service2-c",
    "@version": "1",
    "geoip": {
      "coordinates": [
        125.3228,
        43.88
      ],
      "ip": "175.19.41.41",
      "region_name": "Jilin",
      "timezone": "Asia/Shanghai",
      "continent_code": "AS",
      "city_name": "Changchun",
      "country_name": "China",
      "region_code": "JL",
      "latitude": 43.88,
      "country_code3": "CN",
      "longitude": 125.3228,
      "location": {
        "lat": 43.88,
        "lon": 125.3228
      },
      "country_code2": "CN"
    },
    "message": "100.117.xxx.xx|175.19.41.41|0|2019-05-07 18:23:56|GET|200|56|/building/crowd/type?client=ios&clientversion=6.4.3&deviceid=528980&t=1557224636.271672|kaiStart/6.4.3 (iPhone; iOS 12.1.4; Scale/3.00)|7D70B434FAA00E6DE050190AFD014926 BAB716F7FA2CF5B8E2B967B6ADAE6DD6",
    "clientip": "175.19.41.41",
    "@timestamp": "2019-05-07T10:23:58.314Z",
    "path": "/home/admin/logs/access_log.2019-05-07"
  },
  "fields": {
    "@timestamp": [
      "2019-05-07T10:23:58.314Z"
    ]
  },
  "sort": [
    1557224638314
  ]
}

最后在建立coordinate map(坐标地图)时,最好先将系统管理 -> index patterns中的 access-maps*先删除后重新创建。因为数据被正确索引之后,index patterns中的索引依然是错误的。所以重建之后才能正确绘制地图。效果如下:功能还是很强大的。


 

 

–补充下 ruby切割日志的配置文件20190510

input {
  beats {
    port => 5044
  }
  file {
    path => ["/home/admin/logs/access_log.2019*"]
  }
}

filter {
  ruby {
    code =>'
    arr = event.get("message").split("|")
    event.set("server_ip",arr[0])
    event.set("client_ip",arr[1].split(",")[0])
    event.set("mills",arr[2])
    event.set("date",arr[3])
    event.set("http_type",arr[4])
    event.set("http_status",arr[5])
    event.set("bytes",arr[6])
    event.set("request_url",arr[7])
    event.set("user_agent",arr[8])
    event.set("sign","")
    event.set("auth",arr[9])
    event.set("program",arr[7].split("/")[0])
    event.set("api",arr[7].split("?")[0])
    '
  }

  geoip {
    source => "client_ip"
    target => "geoip"
    database => "/home/admin/logstash/config/GeoLite2-City.mmdb"
    add_field => ["[geoip][coordinates]","%{[geoip][longitude]}"]
    add_field => ["[geoip][coordinates]","%{[geoip][latitude]}"]
  }
  mutate {
    convert => [ "[geoip][coordinates]", "float"]
  }
}

output {
  stdout {  //这个是在控制台输出切割后的数据,可测试通过后传到es
    codec => rubydebug
  }

#  elasticsearch {
#    hosts => ["http://xxx.xx.xxx.xx:9200"]
#    index => "access-map-%{+YYYY.MM.dd}"
#    user => "elastic"
#    password => "123456"
#  }
}          

 


   原创文章,转载请标明本文链接: 利用logstash6.4.2监控access访问日志并切割,使用geoip插件分析ip地址在kibana控制台形成用户热点地图

如果觉得我的文章对您有用,请随意打赏。您的支持将鼓励我继续创作!

发表评论

电子邮件地址不会被公开。 必填项已用*标注

更多阅读