EFK收集nginx日志并展示来源IP地图

nginx日志详细的记录了客户端访问的各类细节参数，如IP、URL、请求类型、返回值等，EFK通过分析IP字段，可以地图插件的支持下显示客户端地理分布。

§格式化nginx日志

首先，为了方便收集nginx日志，先格式化nginx日志为json，打开nginx.conf配置文件，在http{}中添加如下日志格式化参数

1 log_format logstash_json '{ "@fields": { '
2 '"@timestamp": "$time_iso8601", '
3 '"remote_addr": "$remote_addr", '
4 '"remote_user": "$remote_user", '
5 '"body_bytes_sent": "$body_bytes_sent", '
6 '"request_time": "$request_time", '
7 '"http_device_id": "$http_device_id", '
8 '"http_client_type": "$http_client_type",'
9 '"http_device_name":"$http_device_name",'
10 '"status": "$status", '
11 '"request": "$request", '
12 '"request_method": "$request_method", '
13 '"host": "$host", '
14 '"server_port": "$server_port", '
15 '"http_referrer": "$http_referer", '
16 '"body_bytes_sent":"$body_bytes_sent", '
17 '"http_x_forwarded_for": "$http_x_forwarded_for", '
18 '"http_user_agent": "$http_user_agent" } }';

其次，在站点配置中引用，如下示例

1 access_log logs/yiqihao_staging_access.log logstash_json;

§Fluentd添加地图库插件fluent-plugin-geoip

插件地址，以下为CentOS7的安装

1 yum groupinstall "Development Tools"
2 yum install geoip-devel --enablerepo=epel
3 td-agent-gem install fluent-plugin-geoip

在GeoIP官网下载免费地图数据库包(GeoLite City即可)，解压并放在路径/etc/td-agent/下，假设文件名为GeoLiteCity.dat

§Fluentd添加geoip配置信息

1 # Parse IP to Geo
2 <filter staging.nginx**>
3 @type geoip
4 
5 # Specify one or more geoip lookup field which has ip address (default: host)
6 # in the case of accessing nested value, delimit keys by dot like 'host.ip'.
7 geoip_lookup_keys @fields.remote_addr
8
9 # Specify optional geoip database (using bundled GeoLiteCity databse by default)
10 geoip_database "/etc/td-agent/GeoLiteCity.dat"
11 # Specify optional geoip2 database
12 # geoip2_database "/path/to/your/GeoLite2-City.mmdb" (using bundled GeoLite2-City.mmdb by default)
13 # Specify backend library (geoip2_c, geoip, geoip2_compat)
14 backend_library geoip2_c
15 
16 # Set adding field with placeholder (more than one settings are required.)
17 <record>
18 city ${city.names.en["@fields.remote_addr"]}
19 #latitude ${location.latitude["@fields.remote_addr"]}
20 #longitude ${location.longitude["@fields.remote_addr"]}
21 location '[${location.longitude["@fields.remote_addr"]},${location.latitude["@fields.remote_addr"]}]'
22 country ${country.iso_code["@fields.remote_addr"]}
23 country_name ${country.names.en["@fields.remote_addr"]}
24 #postal_code ${postal.code["@fields.remote_addr"]}
25 </record>
26
27 # To avoid get stacktrace error with `[null, null]` array for elasticsearch.
28 skip_adding_null_record true
29 
30 # Set @log_level (default: warn)
31 @log_level info
32 </filter>

§修改location字段默认类型为geo_point

先不要启动fluentd，需要在elasticsearch未生成nginx的索引日志时，将location字段的默认类型个性为geo_point(若不修改，默认会是float，无法生成地图数据)；直接在kibana的Dev Tools中执行如下命令(索引根据实际情况)

1 PUT _template/logstash-nginx-prod*
2 {
3 "template": "logstash-nginx-prod*",
4 "mappings": {
5 "_default_": {
6 "properties" : {
7 "location": { "type": "geo_point"}
8 }
9 }
10 }
11 }

然后开启nginx的日志收集

1 service td-agent start/restart

§然后在kibana中直接创建Tile Map即可

默认地图全是英文名称，要添加高德中文名称地图支持，在kibana.yaml文件最后添加如下一行配置，并重启kibana即可

1 tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={