環境說明

1. windows server 2012 R2 64位

2. log4net.ElasticSearch

3. kibana-5.5.0-windows-x86.zip

架構說明

數據採集有兩種方案Beats或Logstash，Logstash支持很多種數據源，有log4j但沒有log4net，本想使用log4net.redis+redis+logstash的方式實現日誌收集。無意間發現了 log4net.ElasticSearch可以直接寫日誌到ES，經過測試非常好用、方便。

log4net.ElasticSearch is a module for the log4netlibrary to log messages to theElasticSearchdocument database. ElasticSearch offers robust full-text searching an analyzation so that errors and messages can be indexed quickly and searched easily.

Features:

• Supports .NET 4.0+
• Easy installation and setup via Nuget
• Full support for the Elasticsearch _bulk API for buffered logging

安裝kibana

1.下載kibana

首先下載windows版本的kibana文件，地址:https://artifacts.elastic.co/downloads/kibana/kibana-5.5.0-windows-x86.zip,解壓到E:elkkibana目錄

2.配置Kibana

打開配置文件E:elkkibanaconfigkibana.yml，修改server.host和server.host，如下：

server.host: "192.168.5.175"

elasticsearch.url: "http://192.168.5.175:9200"

3.運行Kibana

命令行下執行E:elkkibanainkibana.bat文件

使用log4net.ElasticSearch

1.安裝log4net.ElasticSearch

log4net.ElasticSearch可以通過Nuget安裝，要注意保持和正在使用的log4net版本一致

Install-Package log4net.ElasticSearch -Version 2.3.4

log4net.ES.Example是一個測試Demo，修改配置後就能使用。

2.配置log4net.config文件

Index=log：索引的名字
rolling=true:每天生成一個索引，比如log-2017.07.13
bufferSize :批量提交到ES,這裡為了演示設置成1

2.寫入日誌

運行測試程序，即可寫入幾條日誌到ES中。

--查詢索引列表
http://192.168.5.175:9200/_cat/indices
返回值：
yellow open log-2017.07.13 UBT3YG2qQpWkfJKRNw-XCQ 5 1 1 0 16.5kb 16.5kb
yellow open log-2017.07.07 pMXjGKkiRRSsrBdl5qNlNw 5 1 4 0 62.6kb 62.6kb

--查詢索引內容
http://192.168.5.175:9200/log-2017.07.13/_search?q=*
返回值：

{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "log-2017.07.13",
"_type": "logEvent",
"_id": "AV05k9nbXp-PycfFP2XN",
"_score": 1,
"_source": {
"timeStamp": "2017-07-13T01:32:53.9948595Z",
"message": "System Error",
"messageObject": {},
"exception": {
"Type": "System.Exception",
"Message": "Something terrible happened.",
"HelpLink": null,
"Source": null,
"HResult": -2146233088,
"StackTrace": null,
"Data": {},
"InnerException": null
},
"loggerName": "log4net.ES.Example.Program",
"domain": "log4net.ES.Example.exe",
"identity": "",
"level": "ERROR",
"className": "log4net.ES.Example.Program",
"fileName": "F:\soft\elk\log4net.ES.Example-master\log4net.ES.Example\Program.cs",
"lineNumber": "18",
"fullInfo": "log4net.ES.Example.Program.Main(F:\soft\elk\log4net.ES.Example-master\log4net.ES.Example\Program.cs:18)",
"methodName": "Main",
"fix": "LocationInfo, UserName, Identity, Partial",
"properties": {
"log4net:Identity": "",
"log4net:UserName": "BJ-BL\Administrator",
"log4net:HostName": "BJ-BL",
"@timestamp": "2017-07-13T01:32:53.9948595Z"
},
"userName": "BJ-BL\Administrator",
"threadName": "1",
"hostName": "BJ-BL"
}
}
]
}
}

日誌欄位非常全。。。。很贊

配置kibana

1.創建索引模式

打開kibana，http://192.168.5.175:5601。

點擊Management=>Index Patterns=>Create Index Pattern,輸入log-*就可以匹配上log-yyyy.mm.dd的索引了。

2.查詢日誌

點擊Discover，選中log-*索引，就可以查詢日誌了。

總結

ELK日誌框架已經搭建完成，這裡的L變成了Log4net，ES和Kibana的熟練使用還需要時間去積累。Logstash的插件功能真是強大，也寫了幾個簡單的配置文件實現輸入輸出的轉化，如下：

input {
redis {
host => "192.168.5.175"
port => 6379
data_type => "list"
key => "logstash"
codec=> json
}

}
output {
elasticsearch {
hosts => ["192.168.5.175:9200"]
}
stdout { codec => rubydebug }
}

logstash-redis2es.conf

input { stdin { } }
output {
elasticsearch { hosts => ["192.168.5.175:9200"] }
stdout { codec => rubydebug }
}

logstash-simple.conf

input { stdin { } }
output {
elasticsearch { hosts => ["192.168.5.175:9200"] }
file{path=>[
"stdin-log.txt"
]}
stdout { codec => rubydebug }
}

logstash-stdin2file.conf

input { stdin { } }
output {
redis {
host => "192.168.5.175"
port => 6379
data_type => "channel"
key => "logstash_list_0"
}

stdout { codec => rubydebug }
}

logstash-stdin2redis.conf

用到的參考資料如下:

4.ELK部署詳解，CentOS7環境

喜歡這篇文章嗎？立刻分享出去讓更多人知道吧！