描述背景：

注：文章配置文件示例都是拿阳明大佬博客https://www.qikqiak.com/post/elastic-cloud-on-k8s/ 还有elastic官方文档，阳明大佬文档是旧的新版本有所改变尽量参考官方文档，fluent-bit也有必要看下官方文档 —

集群配置：初始集群环境kubeadm 1.16.1

ip	自定义域名	主机名
192.168.3.8	master.k8s.io	k8s-vip
192.168.3.10	master01.k8s.io	k8s-master-01
192.168.3.5	master02.k8s.io	k8s-master-02
192.168.3.12	master03.k8s.io	k8s-master-03
192.168.3.6	node01.k8s.io	k8s-node-01
192.168.3.2	node02.k8s.io	k8s-node-02
192.168.3.4	node03.k8s.io	k8s-node-03

关于Elastic Cloud on Kubernetes

Elastic Cloud on Kubernetes(ECK)是一个 Elasticsearch Operator，elastic推出的一个便于部署管理的项目，而且内部集成了核心安全功能（TLS 加密、基于角色的访问控制，以及文件和原生身份验证）免费提供。

安装Elastic Cloud on Kubernetes

当前最新版本为1.0，参考：https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.html。

kubectl apply -f https://download.elastic.co/downloads/eck/1.0.0-beta1/all-in-one.yaml
kubectl get pods -n elastic-system
kubectl -n elastic-system logs -f statefulset.apps/elastic-operator

安装elasticsearch on cluster

官方示例 0.8 示例中apiversion还是elasticsearch.k8s.elastic.co/v1alpha1，1.0更新为elasticsearch.k8s.elastic.co/v1beta1。所以最好能看下最新的官方文档。

cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: elastic
  namespace: elastic-system
spec:
  version: 7.4.0
  nodeSets:
  - name: default
    count: 3
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: rook-ceph-block
EOF
注：采用默认官方的了,个人安装了rook ceph 就用了rook-ceph-block，由于是测试storage就设置为5G。另外如果指定运行node节点可以设置lable根据个人需求设置

kubectl get pods -n elastic-system -o wide
kubectl -n elastic-system get pods --selector='elasticsearch.k8s.elastic.co/cluster-name=elastic'（官方文档的查看方式，看个人使用习惯了呢）

kubectl get elasticsearch -n elastic-system

kubectl get service -n elastic-system

获取elastic链接用户密码，默认用户为elastic
kubectl -n elastic-system get secret elastic-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode; echo

安装kibana on cluaster

依然官方示例

cat <<EOF | kubectl apply -f -
apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
  name: kibana
  namespace: elastic-system
spec:
  version: 7.4.0
  count: 1
  elasticsearchRef:
    name: elastic
  http:
    tls:
      selfSignedCertificate:
        disabled: true
EOF
注意：属性spec.elasticsearchRef.name的值为上面我们创建的 Elasticsearch 对象的 name：elastic。直接添加这个资源对象即可，配置文件下方关于http的配置参照https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-kibana.html#k8s-kibana-http-configuration disable tls为了方便外部traefik负载。

kubectl get kibana -n elastic-system

kubectl -n elastic-system get pod --selector='kibana.k8s.elastic.co/name=kibana'


kuberctl get svc -n elastic-system

kibana traefik对外暴露

cat <<EOF >  ingress.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: kibana-kb-http
  namespace: elastic-system
spec:
  entryPoints:
    - websecure
  tls:
    secretName: all-sainaihe-com
  routes:
    - match: Host(`kibana123.saynaihe.com`)
      kind: Rule
      services:
        - name: kibana-kb-http
          port: 5601
EOF
kubectl apply -f ingress.yaml
kubectl get ingressroute -n elastic-system

接下来fluent-bit 安装

kubernetes 采集日志比较常用的有很多比如elasitc自己的logstash filebeat ，还有fluent，另外fluent还有专门针对kubernetes的fluent-bit，各种优劣可以找文档对比，最终我选择了fluent-bit.先按照https://github.com/fluent/fluent-bit-kubernetes-logging。 README.md修改下配置文件。

git clone https://github.com/fluent/fluent-bit-kubernetes-logging
cd /fluent/fluent-bit-kubernetes-logging
注： 修改下配置文件将默认的namespace修改下与eck在同一命名空间下
修改后文件如下：
------
cat <<EOF >  fluent-bit-service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: elastic-system
EOF
------
cat <<EOF >  fluent-bit-role.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluent-bit-read
  namespace: elastic-system
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs: ["get", "list", "watch"]
EOF
------
cat <<EOF >  fluent-bit-role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: elastic-system
EOF
------

kubectl apply -f fluent-bit-service-account.yaml
kubectl apply -f fluent-bit-role.yaml
kubectl apply -f fluent-bit-role-binding.yaml
------
cd output/elasticsearch/
cat <<EOF > fluent-bit-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: elastic-system
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-elasticsearch.conf

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            elastic-es-http
        Port            9200
        HTTP_User       elastic
        HTTP_Passwd     *********
        tls   on
        tls.verify   off
        Logstash_Format On
        Replace_Dots    On
        Retry_Limit     False

  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S
EOF
------
注：基本还是默认的就修改了namespace 还有output-elasticsearch.conf中 Host  Port   HTTP_User  HTTP_Passwd  tls   tls.verify具体参考https://docs.fluentbit.io/manual/configuration 官方文档
------
cat <<EOF > fluent-bit-ds.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: elastic-system
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:1.2.1
        imagePullPolicy: Always
        ports:
          - containerPort: 2020
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "elastic-es-http"
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "9200"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"
EOF
------
注: 修改了下apiversion  1.16取消了extensions/v1beta1  更改为apps/v1
配置文件增加  selector: 相关配置。FLUENT_ELASTICSEARCH_HOST  FLUENT_ELASTICSEARCH_PORT  对应value输入
------
kubectl apply -f fluent-bit-configmap.yaml
kubectl apply -f fluent-bit-ds.yaml
kubectl get pods -n elastic-system

登录kibana 进行相关配置

注：发现rook的日志单行了后续将完善下各种日志的格式还有其他相关问题

kuberntes1.16安装efk