doina

一个小菜鸟运维工程师.

EFK收集Kubernetes集群收集event事件

简介

Event是什么?

Event作为kubernetes的一个对象资源,记录了集群运行所遇到的各种大事件,有助于排错,但大量的事件如果都存储在etcd中,会带来较大的性能与容量压力,所以etcd中默认只保存最近1小时的。

查看Event

[root@T01 elasticsearch]# kubectl get event
LAST SEEN   TYPE     REASON    OBJECT                          MESSAGE
5m16s       Normal   Pulled    pod/nginxtest-bbccd685f-gtf9x   Container image "nginx:1.10" already present on machine
5m15s       Normal   Created   pod/nginxtest-bbccd685f-gtf9x   Created container nginxtest
5m15s       Normal   Started   pod/nginxtest-bbccd685f-gtf9x   Started container nginxtest

[root@T01 elasticsearch]# kubectl get event -o wide
LAST SEEN   TYPE     REASON    OBJECT                          SUBOBJECT                    SOURCE         MESSAGE                                                   FIRST SEEN   COUNT   NAME
5m22s       Normal   Pulled    pod/nginxtest-bbccd685f-gtf9x   spec.containers{nginxtest}   kubelet, t01   Container image "nginx:1.10" already present on machine   5h40m        5       nginxtest-bbccd685f-gtf9x.15c919914460c103
5m21s       Normal   Created   pod/nginxtest-bbccd685f-gtf9x   spec.containers{nginxtest}   kubelet, t01   Created container nginxtest                               5h40m        5       nginxtest-bbccd685f-gtf9x.15c9199145e21995
5m21s       Normal   Started   pod/nginxtest-bbccd685f-gtf9x   spec.containers{nginxtest}   kubelet, t01   Started container nginxtest                               5h40m        5       nginxtest-bbccd685f-gtf9x.15c919914bd75bfe

收集event的方案

  • 使用开源项目eventrouter进行收集
  • 项目地址: https://github.com/heptiolabs/eventrouter

使用持久化把eventroute的数据落到宿主机,然后采用filebeat采集,竟然一系列的redis,logstash的处理后发送到elasticsearch,最终在kibana上展示.

部署eventrouter

官网提供的示例文件 https://raw.githubusercontent.com/heptiolabs/eventrouter/master/yaml/eventrouter.yaml

把日志挂在到主机进行收集

  • 使用hostpath挂载目录
  • 使用init初始化目录权限
  • 使用filebeat收集挂载目录的日志
  • filebeat将日志发送到redis
  • logstash从redis读取数据
  • logstash将数据发送到elasticsearch
  • kibana展示elasticsearch的日志
$ cat eventrouter-outfilebeat.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eventrouter 
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: eventrouter 
rules:
- apiGroups: [""]
  resources: ["events"]
  verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: eventrouter 
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: eventrouter
subjects:
- kind: ServiceAccount
  name: eventrouter
  namespace: kube-system
---
apiVersion: v1
data:
  config.json: |- 
    {
      "sink": "glog"
    }
kind: ConfigMap
metadata:
  name: eventrouter-cm
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: eventrouter
  namespace: kube-system
  labels:
    app: eventrouter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: eventrouter
  template:
    metadata:
      labels:
        app: eventrouter
        tier: control-plane-addons
    spec:
      initContainers:
        - name: init-log-dir
          image: busybox
          command:
            - "/bin/sh"
          args:
            - "-c"
            - "chmod 777 -R /data/log/eventrouter"
          volumeMounts:
          - name: log-path
            mountPath: /data/log/eventrouter
      containers:
        - name: kube-eventrouter
          image: baiyongjie/eventrouter:v0.2
          imagePullPolicy: IfNotPresent
          command:
            - "/bin/sh"
          args:
            - "-c"
            - "/eventrouter -v 3 -log_dir /data/log/eventrouter"
          volumeMounts:
          - name: config-volume
            mountPath: /etc/eventrouter
          - name: log-path
            mountPath: /data/log/eventrouter
      serviceAccount: eventrouter
      volumes:
        - name: config-volume
          configMap:
            name: eventrouter-cm
        - name: log-path
          hostPath:
            path: /data/logs/kube-system/eventrouter

再容器内收集,直接然后发送到es

  • 启动eventrouter容器,挂载/data/log/eventrouter目录
  • 启动filebeat容器,挂载/data/log/eventrouter
  • filebeat数据发送到elasticsearch
  • kibana添加索引,并展示数据
IP 角色
192.168.109.128 Kubernetes
192.168.109.128 kibana
192.168.109.128 elasticsearch

es,kibana准备

$ rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
$ vim /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md


## elasticsearch
$ yum -y install java
$ yum -y install elasticsearch-6.3.2

$ systemctl start elasticsearch 
$ systemctl enable elasticsearch 

## kibana
$ yum -y install kibana-6.3.2
$ chown kibana. /var/log/kibana/

$ vim /etc/kibana/kibana.yml 
server.port: 5601
server.host: "192.168.109.128"
elasticsearch.url: "http://192.168.109.128:9200"
kibana.defaultAppId: "discover"
elasticsearch.pingTimeout: 3000
elasticsearch.shardTimeout: 0
elasticsearch.startupTimeout: 9000
pid.file: /tmp/kibana.pid
logging.dest: /var/log/kibana/kibana.log
logging.verbose: false
ops.interval: 5000

$ systemctl start kibana    
$ systemctl enable kibana
$ systemctl status kibana

yaml文件

$ cat eventrouter-infilebeat.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eventrouter 
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: eventrouter 
rules:
- apiGroups: [""]
  resources: ["events"]
  verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: eventrouter 
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: eventrouter
subjects:
- kind: ServiceAccount
  name: eventrouter
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: eventrouter-cm
  namespace: kube-system
data:
  config.json: |- 
    {
      "sink": "glog"
    }
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
data:
  filebeat.yml: |-
    filebeat.prospectors:
    - input_type: log
      paths:
        - "/data/log/eventrouter/*"
    output.elasticsearch:
      #hosts: ["11.0.16.210:9200","11.0.16.213:9200","11.0.16.217:9200"]
      hosts: ["192.168.109.128:9200"]
      index: "filebeat-k8s-pre-event-%{+yyyy.MM.dd}"
    setup.template.name: "filebeat-k8s-pre-event"
    setup.template.pattern: "filebeat-k8s-pre-event-"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: eventrouter
  namespace: kube-system
  labels:
    app: eventrouter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: eventrouter
  template:
    metadata:
      labels:
        app: eventrouter
        tier: control-plane-addons
    spec:
      containers:
        - name: kube-eventrouter
          image: baiyongjie/eventrouter:v0.2
          command:
            - "/bin/sh"
          args:
            - "-c"
            - "/eventrouter -v 3 -log_dir /data/log/eventrouter"
          volumeMounts:
          - name: eventrouter-cm
            mountPath: /etc/eventrouter
          - name: log-path
            mountPath: /data/log/eventrouter
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:6.3.2
          command:
            - "/bin/sh"
          args:
            - "-c"
            - "filebeat -c /etc/filebeat/filebeat.yml"
          volumeMounts:
          - name: filebeat-config
            mountPath: /etc/filebeat/
          - name: log-path
            mountPath: /data/log/eventrouter
      serviceAccount: eventrouter
      volumes:
        - name: eventrouter-cm
          configMap:
            name: eventrouter-cm
        - name: filebeat-config
          configMap:
            name: filebeat-config
        - name: log-path
          emptyDir: {}

$ kubectl apply -f eventrouter-infilebeat.yaml
serviceaccount/eventrouter created
clusterrole.rbac.authorization.k8s.io/eventrouter created
clusterrolebinding.rbac.authorization.k8s.io/eventrouter created
configmap/eventrouter-cm created
configmap/filebeat-config created
deployment.apps/eventrouter created

$ kubectl get pods -n kube-system |grep event
eventrouter-7bb898ff4b-2jp4r   2/2     Running   0          29s

查看es索引

$ curl http://192.168.109.128:9200/_cat/indices
yellow open filebeat-k8s-pre-event-2019.09.30 GL1lIT6VRp-qvI-reyjiNA 5 1 134 0 32kb 32kb

在kibana添加索引并查看

《EFK收集Kubernetes集群收集event事件》

《EFK收集Kubernetes集群收集event事件》

模拟nginx pod重启

$ kubectl exec -it nginxtest-bbccd685f-gtf9x  -- /bin/bash
root@nginxtest-bbccd685f-gtf9x:/# nginx -s stop
2019/09/30 09:02:46 [notice] 18#18: signal process started
root@nginxtest-bbccd685f-gtf9x:/# command terminated with exit code 137


$ kubectl describe pods nginxtest-bbccd685f-gtf9x  | grep -A 20 Events: 
Events:
  Type     Reason   Age                  From          Message
  ----     ------   ----                 ----          -------
  Normal   Pulled   83s (x5 over 5h36m)  kubelet, t01  Container image "nginx:1.10" already present on machine
  Normal   Created  82s (x5 over 5h36m)  kubelet, t01  Created container nginxtest
  Normal   Started  82s (x5 over 5h36m)  kubelet, t01  Started container nginxtest

《EFK收集Kubernetes集群收集event事件》

点赞

发表评论

电子邮件地址不会被公开。

此站点使用Akismet来减少垃圾评论。了解我们如何处理您的评论数据