ElasticSearch實戰:Linux日誌對接Kibana
本文由雲+社群發表
ElasticSearch是一個基於Lucene的搜尋伺服器。它提供了一個分散式多使用者能力的全文搜尋引擎,基於RESTFul web介面。ElasticSearch是用Java開發的,並作為Apache許可條款下的開放原始碼釋出,是當前流行的企業級搜尋引擎。ElasticSearch常用於全文檢索,結構化檢索,資料分析等。
下面,我們以ElasticSearch接管Linux日誌(/var/log/xxx.log)為例,詳細介紹如何進行配置與部署。
總體架構圖
一,準備工作
1,CVM及ElasticSearch
在騰訊雲帳號下,申請一臺CVM(Linux作業系統)、一個ElasticSearch叢集(後面簡稱ES),使用最簡配置即可;申請的CVM和ES,必須在同一個VPC的同一個子網下。
CVM詳情資訊
ElasticSearch詳情資訊
2,Filebeat工具
為了將Linux日誌提取到ES中,我們需要使用Filebeat工具。Filebeat是一個日誌檔案託運工具,在你的伺服器上安裝客戶端後,Filebeat會監控日誌目錄或者指定的日誌檔案,追蹤讀取這些檔案(追蹤檔案的變化,不停的讀),並且轉發這些資訊到ElasticSearch或者logstarsh中存放。當你開啟Filebeat程式的時候,它會啟動一個或多個探測器(prospectors)去檢測你指定的日誌目錄或檔案,對於探測器找出的每一個日誌檔案,Filebeat啟動收割程序(harvester),每一個收割程序讀取一個日誌檔案的新內容,併發送這些新的日誌資料到處理程式(spooler),處理程式會集合這些事件,最後Filebeat會發送集合的資料到你指定的地點。
官網簡介:https://www.elastic.co/products/beats/filebeat
二,操作步驟
1,Filebeat下載與安裝
首先,登入待接管日誌的CVM,在CVM上下載Filebeat工具:
[root@VM_3_7_centos ~]# cd /opt/ [root@VM_3_7_centos opt]# ll total 4 drwxr-xr-x. 2 root root 4096 Sep72017 rh [root@VM_3_7_centos opt]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.2-x86_64.rpm --2018-12-10 20:24:26--https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.2-x86_64.rpm Resolving artifacts.elastic.co (artifacts.elastic.co)... 107.21.202.15, 107.21.127.184, 54.225.214.74, ... Connecting to artifacts.elastic.co (artifacts.elastic.co)|107.21.202.15|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 12697788 (12M) [binary/octet-stream] Saving to: ‘filebeat-6.2.2-x86_64.rpm’ 100%[=================================================================================================>] 12,697,788160KB/sin 1m 41s 2018-12-10 20:26:08 (123 KB/s) - ‘filebeat-6.2.2-x86_64.rpm’ saved [12697788/12697788]
然後,進行安裝filebeat:
[root@VM_3_7_centos opt]# rpm -vi filebeat-6.2.2-x86_64.rpm warning: filebeat-6.2.2-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY Preparing packages... filebeat-6.2.2-1.x86_64 [root@VM_3_7_centos opt]#
至此,Filebeat安裝完成。
2,Filebeat配置
進入Filebeat配置檔案目錄:/etc/filebeat/
[root@VM_3_7_centos opt]# cd /etc/filebeat/ [root@VM_3_7_centos filebeat]# ll total 108 -rw-r--r-- 1 root root 44384 Feb 172018 fields.yml -rw-r----- 1 root root 52193 Feb 172018 filebeat.reference.yml -rw------- 1 root root7264 Feb 172018 filebeat.yml drwxr-xr-x 2 root root4096 Dec 10 20:35 modules.d [root@VM_3_7_centos filebeat]#
其中,filebeat.yml就是我們需要修改的配置檔案。建議修改配置前,先備份此檔案。
然後,確認需要對接ElasticSearch的Linux的日誌目錄,我們以下圖( /var/log/secure )為例。
/var/log/secure日誌檔案
使用vim開啟/etc/filebeat/filebeat.yml檔案,修改其中的:
1)Filebeat prospectors類目中,enable預設為false,我們要改為 true
2)paths,預設為/var/log/*.log,我們要改為待接管的日誌路徑: /var/log/secure
3)Outputs類目中,有ElasticSearchoutput配置,其中hosts預設為"localhost:9200",需要我們手工修改為上面申請的ES子網地址和埠,即 "10.0.3.8:9200" 。
修改好上述內容後,儲存退出。
修改好的配置檔案全文如下:
[root@VM_3_7_centos /]# vim /etc/filebeat/filebeat.yml [root@VM_3_7_centos /]# cat /etc/filebeat/filebeat.yml ###################### Filebeat Configuration Example ######################### # This file is an example configuration file highlighting only the most common # options. The filebeat.reference.yml file from the same directory contains all the # supported options with more comments. You can use it as a reference. # # You can find the full configuration reference here: # https://www.elastic.co/guide/en/beats/filebeat/index.html # For more available modules and options, please see the filebeat.reference.yml sample # configuration file. #=========================== Filebeat prospectors ============================= filebeat.prospectors: # Each - is a prospector. Most options can be set at the prospector level, so # you can use different prospectors for various configurations. # Below are the prospector specific configurations. - type: log # Change to true to enable this prospector configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /var/log/secure #- c:\programdata\elasticsearch\logs\* # Exclude lines. A list of regular expressions to match. It drops the lines that are # matching any regular expression from the list. #exclude_lines: ['^DBG'] # Include lines. A list of regular expressions to match. It exports the lines that are # matching any regular expression from the list. #include_lines: ['^ERR', '^WARN'] # Exclude files. A list of regular expressions to match. Filebeat drops the files that # are matching any regular expression from the list. By default, no files are dropped. #exclude_files: ['.gz$'] # Optional additional fields. These fields can be freely picked # to add additional information to the crawled log files for filtering #fields: #level: debug #review: 1 ### Multiline options # Mutiline can be used for log messages spanning multiple lines. This is common # for Java Stack Traces or C-Line Continuation # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [ #multiline.pattern: ^\[ # Defines if the pattern set under pattern should be negated or not. Default is false. #multiline.negate: false # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern # that was (not) matched before or after or as long as a pattern is not matched based on negate. # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash #multiline.match: after #============================= Filebeat modules =============================== filebeat.config.modules: # Glob pattern for configuration loading path: ${path.config}/modules.d/*.yml # Set to true to enable config reloading reload.enabled: false # Period on which files under path should be checked for changes #reload.period: 10s #==================== Elasticsearch template setting ========================== setup.template.settings: index.number_of_shards: 3 #index.codec: best_compression #_source.enabled: false #================================ General ===================================== # The name of the shipper that publishes the network data. It can be used to group # all the transactions sent by a single shipper in the web interface. #name: # The tags of the shipper are included in their own field with each # transaction published. #tags: ["service-X", "web-tier"] # Optional fields that you can specify to add additional information to the # output. #fields: #env: staging #============================== Dashboards ===================================== # These settings control loading the sample dashboards to the Kibana index. Loading # the dashboards is disabled by default and can be enabled either by setting the # options here, or by using the `-setup` CLI flag or the `setup` command. #setup.dashboards.enabled: false # The URL from where to download the dashboards archive. By default this URL # has a value which is computed based on the Beat name and version. For released # versions, this URL points to the dashboard archive on the artifacts.elastic.co # website. #setup.dashboards.url: #============================== Kibana ===================================== # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API. # This requires a Kibana endpoint configuration. setup.kibana: # Kibana Host # Scheme and port can be left out and will be set to the default (http and 5601) # In case you specify and additional path, the scheme is required: http://localhost:5601/path # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601 #host: "localhost:5601" #============================= Elastic Cloud ================================== # These settings simplify using filebeat with the Elastic Cloud (https://cloud.elastic.co/). # The cloud.id setting overwrites the `output.elasticsearch.hosts` and # `setup.kibana.host` options. # You can find the `cloud.id` in the Elastic Cloud web UI. #cloud.id: # The cloud.auth setting overwrites the `output.elasticsearch.username` and # `output.elasticsearch.password` settings. The format is `<user>:<pass>`. #cloud.auth: #================================ Outputs ===================================== # Configure what output to use when sending the data collected by the beat. #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: # Array of hosts to connect to. hosts: ["10.0.3.8:9200"] # Optional protocol and basic auth credentials. #protocol: "https" #username: "elastic" #password: "changeme" #----------------------------- Logstash output -------------------------------- #output.logstash: # The Logstash hosts #hosts: ["localhost:5044"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" #================================ Logging ===================================== # Sets log level. The default log level is info. # Available log levels are: error, warning, info, debug #logging.level: debug # At debug level, you can selectively enable logging only for some components. # To enable all selectors use ["*"]. Examples of other selectors are "beat", # "publish", "service". #logging.selectors: ["*"] #============================== Xpack Monitoring =============================== # filebeat can export internal metrics to a central Elasticsearch monitoring # cluster.This requires xpack monitoring to be enabled in Elasticsearch.The # reporting is disabled by default. # Set to true to enable the monitoring reporter. #xpack.monitoring.enabled: false # Uncomment to send the metrics to Elasticsearch. Most settings from the # Elasticsearch output are accepted here as well. Any setting that is not set is # automatically inherited from the Elasticsearch output configuration, so if you # have the Elasticsearch output configured, you can simply uncomment the # following line. #xpack.monitoring.elasticsearch: [root@VM_3_7_centos /]#
執行下列命令啟動filebeat
[root@VM_3_7_centos /]# sudo /etc/init.d/filebeat start Starting filebeat (via systemctl):[OK] [root@VM_3_7_centos /]#
3,Kibana配置
進入ElasticSearch對應的Kibana管理頁,如下圖。
首次訪問Kibana預設會顯示管理頁
首次登陸,會預設進入Management頁面,我們需要將Index pattern內容修改為:filebeat-*,然後頁面會自動填充 Time Filter field name, 不需手動設定,直接點選Create即可。點選Create後,頁面需要一定時間來載入配置和資料,請稍等。如下圖:
將Index pattern內容修改為:filebeat-*,然後點選Create
至此,CVM上,/var/log/secure日誌檔案,已對接到ElasticSearch中,歷史日誌可以通過Kibana進行查詢,最新產生的日誌也會實時同步到Kibana中。
三,實戰效果
日誌接管已完成配置,如何使用呢?
如下圖:
在Index Patterns中可以看到我們配置過的filebeat-*
點選Discover,即可看到secure中的所有日誌,頁面上方的搜尋框中輸入關鍵字,即可完成日誌的檢索。如下圖(點選圖片,可檢視高清大圖):
使用Kibana進行日誌檢索
實際上,檢索只是Kibana提供的諸多功能之一,還有其他功能如視覺化、分詞檢索等,還有待後續研究。