Logstash 收集 Nginx 错误日志

在 k8s 集群中，微服务日志的查看存在一定难度，尤其是对于开发者而言，如果没有运维权限，将无法直接进入 pod 查看日志。即使对于运维人员，面对大量 pod，逐个查看也是一项繁琐的工作。

通过将日志采集到 Elasticsearch，并使用 Kibana 进行查询，可以使维护工作变得更加方便。

配置文件

配置 Logstash 以收集不同类型的日志，并通过 type 进行区分。

input {
     file {
        path => ["/data/nginx/log/*error*.log","/data/nginx/log/nginx.log"] 
        type => "nginx-error"
        add_field => {"cluster" => "test-k8s"}
        add_field => {"env" => "test"}
        add_field => {"app" => "k8s-nginx-error"}
        sincedb_path => "/data/nginx/log/logstash_nginx_error_sincedb_path"
        codec => multiline {
            pattern => "(\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}.*)"
            negate => true
            what => "previous"
        }
    }
}

filter {
    if [type] ==  "nginx-error"{
        grok {
                match => [
                "message", ".*\d{2}:\d{2}:\d{2} \[(?<loglevel>(?:\w+))\] .*, client: (?<client>(?:\w+.\w+.\w+.\w+)),.*request: \"(?<method>(?:\w+)) (?<url>(?:.*))\?(?<params>(?:.*)) HTTP.*host: \"(?<domain>(?:\w+.\w+.\w+)|(?:\w+.\w+.\w+.\w+))\".*",
                "message", ".*\d{2}:\d{2}:\d{2} \[(?<loglevel>(?:\w+))\] .*request: \"(?<method>(?:\w+)) (?<url>(?:.*))\?(?<params>(?:.*)) HTTP.*host: \"(?<domain>(?:\w+.\w+.\w+)|(?:\w+.\w+.\w+.\w+))\".*",
                "message", ".*\d{2}:\d{2}:\d{2} \[(?<loglevel>(?:\w+))\] .*request: \"(?<method>(\w+)) (?<url>(?:.*)) HTTP.*host: \"(?<domain>(?:\w+.\w+.\w+)|(?:\w+.\w+.\w+.\w+))\".*",
                "message", ".*\d{2}:\d{2}:\d{2} \[(?<loglevel>(?:\w+))\] .*"
                ]
        }
    }
}

output {
    if [type] ==  "nginx-error"{
        elasticsearch {
            hosts => ["elsticsearch-http:9200"]
            index => "nginx_error-%{+YYYY.MM.dd}"
            user => "xxxxx"
            password => "xxxxx"
            codec => json
        }
    }
}

确保配置文件中的 hosts, user, 和 password 根据你的 Elasticsearch 集群进行相应的替换。

可以创建 ConfigMap 保存配置信息，然后在 Pod 启动时将 ConfigMap 挂载到配置路径。

kubectl create configmap logstash-conf --from-file=nginx-logs.conf

Nginx 错误日志可以通过 hostPath 打印到主机的目录进行持久化，然后 Logstash 通过 DaemonSet 的方式部署到各个 Node 节点。

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: logstash-nginx-error
  labels:
    app: logstash-nginx-error
spec:
  selector:
    matchLabels:
      app: logstash-nginx-error
  template:
    metadata:
      labels:
        app: logstash-nginx-error
    spec:
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      containers:
        - name: logstash-nginx-error
          image: "logstash:6.8.23"
          command:
            - "/usr/share/logstash/bin/logstash"
            - "-f"
            - "/etc/logstash/config.d/nginx-logs.conf"
          resources:
            limits:
              memory: 1024Mi
          volumeMounts:
            - name: nginx-error-dir
              mountPath: /data/nginx/log
              readOnly: false
            - name: logstash-conf
              subPath: nginx-logs.conf
              mountPath: /etc/logstash/config.d/nginx-logs.conf
      terminationGracePeriodSeconds: 10
      volumes:
        - name: nginx-error-dir
          hostPath:
            path: /data/nginx-proxy/log
            type: DirectoryOrCreate
        - name: logstash-conf
          configMap:
            name: logstash-conf
            items:
              - key: nginx-logs.conf
                path: nginx-logs.conf

根据 Elasticsearch 的版本选择合适的 Logstash 版本，以确保兼容性。

kubectl apply -f daemonset.yaml

Kibana 展示

在 Kibana 中创建对应的 Index Patterns 后，就可以进行日志的查看和分析了。