Logstash单机入门实验

参考

安装

本次实验的安装环境为：ubuntu 15

jre安装

运行 Logstash 需要Java环境，因此需要安装 jre ：

1
sudo apt-get install openjdk-7-jre

Logstash 安装

从 参考 下载 Logstash ，官方提供的 Installation Steps 说明，只有三步：

  1. 解压
  2. 添加logstash配置文件
  3. 运行

解压 Logstash：

1
sudo tar -zxvf logstash-1.5.2.tar.gz -C /opt

添加 HelloWorld.conf 文件：

1
2
3
4
5
6
7
input {
    stdin { }
}

output {
    stdout { codec=>rubydebug }
}

进入 Logstash 目录并运行

1
2
cd /opt/logstash-1.5.2
bin/logstash -f HelloWorld.conf

在终端中输入测试数据并查看输出:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
hello
{
       "message" => "hello",
      "@version" => "1",
    "@timestamp" => "2015-07-04T00:16:35.595Z",
          "host" => "ubuntu"
}
test
{
       "message" => "test",
      "@version" => "1",
    "@timestamp" => "2015-07-04T00:17:18.090Z",
          "host" => "ubuntu"
}

从文件中读取日志

添加 fileInput.conf 文件：

1
2
3
4
5
6
7
8
9
10
11
12
input{
    file {
        path => ["/var/log/apache2/access.log"]
        type => "system"
        start_position => "beginning"
    }
}

output {
    stdout { codec=>rubydebug 
    }  
}

运行 Logstash：

1
bin/logstash -f fileInput.conf

输出结果:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{
       "message" => "192.168.3.190 - - [03/Jul/2015:17:23:26 -0700] \"GET / HTTP/1.1\" 200 3469 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36\"",
      "@version" => "1",
    "@timestamp" => "2015-07-04T00:27:45.814Z",
          "host" => "ubuntu",
          "path" => "/var/log/apache2/access.log",
          "type" => "system"
}
{
       "message" => "192.168.3.190 - - [03/Jul/2015:17:23:26 -0700] \"GET /icons/ubuntu-logo.png HTTP/1.1\" 200 3634 \"http://192.168.3.99/\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36\"",
      "@version" => "1",
    "@timestamp" => "2015-07-04T00:27:45.815Z",
          "host" => "ubuntu",
          "path" => "/var/log/apache2/access.log",
          "type" => "system"
}
{
       "message" => "192.168.3.190 - - [03/Jul/2015:17:23:26 -0700] \"GET /favicon.ico HTTP/1.1\" 404 448 \"http://192.168.3.99/\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36\"",
      "@version" => "1",
    "@timestamp" => "2015-07-04T00:27:45.816Z",
          "host" => "ubuntu",
          "path" => "/var/log/apache2/access.log",
          "type" => "system"
}

从TCP中读取日志

添加 tcpInput.conf 文件：

1
2
3
4
5
6
7
8
9
10
11
12
input {
    tcp {
        port => 5000
        mode => "server"
        ssl_enable => false
    }
}

output {
    stdout { codec=>rubydebug 
    }  
}

运行 Logstash：

1
bin/logstash -f tcpInput.conf

使用 TPC/IP 调试工具连接5000端口并输入任意内容，Logstash 输出以下内容:

1
2
3
4
5
6
7
8
9
10
11
12
{
       "message" => "send first log",
      "@version" => "1",
    "@timestamp" => "2015-07-04T00:36:24.878Z",
          "host" => "192.168.3.15"
}
{
       "message" => "send second log",
      "@version" => "1",
    "@timestamp" => "2015-07-04T00:36:34.598Z",
          "host" => "192.168.3.15"
}

从Redis中读取日志

添加 redisInput.conf 文件：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
input {
    redis {
	batch_count => 1
        data_type => "list"
        key => "logstash-list"
        host => "127.0.0.1"
        port => 6379
        threads => 1
    }
}

output {
    stdout {
        codec => rubydebug {
        }
    }
}

运行 Logstash：

1
bin/logstash -f redisInput.conf

使用Redis客户端工具，并向 logstash-list 键中添加任意值：

1
2
3
4
5
6
127.0.0.1:6379> RPUSH logstash-list "log from redis NO.1"
(integer) 1
127.0.0.1:6379> RPUSH logstash-list "log from redis NO.2"
(integer) 1
127.0.0.1:6379> RPUSH logstash-list "log from redis NO.3"
(integer) 1

Logstash 输出以下内容:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{
       "message" => "log from redis NO.1",
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
    "@timestamp" => "2015-07-04T00:43:04.628Z"
}
{
       "message" => "log from redis NO.2",
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
    "@timestamp" => "2015-07-04T00:43:06.955Z"
}
{
       "message" => "log from redis NO.3",
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
    "@timestamp" => "2015-07-04T00:43:09.147Z"
}

将日志输出到文件中

添加 fileOutput.conf 文件：

1
2
3
4
5
6
7
8
9
10
input {
    stdin { }
}

output {
    file {
        path => "/var/log/test.log"
        message_format => "%{message}"
    }
}

运行 Logstash：

1
sudo bin/logstash -f fileOutput.conf

输入测试内容:

1
2
hello world
hello world again

查看 /var/log/test.log 文件内容：

1
2
hello world
hello world again

将日志输出到TCP中

添加 tcpOutput.conf 文件：

1
2
3
4
5
6
7
8
9
10
11
input {
    stdin { }
}

output {
    tcp {
        host  => "YourServerIP"
        port  => 5000
        codec=>rubydebug 
    }
}

首先在服务器上运行 TCP Server ，并监听5000端口，然后运行 Logstash：

1
bin/logstash -f tcpOutput.conf

输入测试内容:

1
2
tcp out 
tcp out again

查看TCP Server，收到以下数据：

1
2
3
{.       "message".[0;37m => .[0m.[0;33m"tcp out ".[0m,.      "@version".[0;37m => .[0m.[0;33m"1".[0m,.    "@timestamp".[0;37m => .[0m"2015-07-04T01:27:34.469Z",.          "host".[0;37m => .[0m.[0;33m"ubuntu".[0m.}.

{.       "message".[0;37m => .[0m.[0;33m"tcp out again".[0m,.      "@version".[0;37m => .[0m.[0;33m"1".[0m,.    "@timestamp".[0;37m => .[0m"2015-07-04T01:27:39.147Z",.          "host".[0;37m => .[0m.[0;33m"ubuntu".[0m.}.

将日志输出到Redis中

添加 redisOutput.conf 文件：

1
2
3
4
5
6
7
8
9
10
input {
    stdin { }
}

output {
    redis { 
	host => "127.0.0.1" 
	data_type => "list" 
	key => "logstash-list"}
}

运行 Logstash：

1
bin/logstash -f redisOutput.conf

输入测试内容:

1
2
redis out testt 
redis out test again

使用Redis客户端工具查看logstash-list的值：

1
2
3
127.0.0.1:6379> lrange logstash-list 0 -1
1) "{\"message\":\"redis out test\",\"@version\":\"1\",\"@timestamp\":\"2015-07-04T01:33:03.502Z\",\"host\":\"ubuntu\"}"
2) "{\"message\":\"redis out test again\",\"@version\":\"1\",\"@timestamp\":\"2015-07-04T01:33:12.592Z\",\"host\":\"ubuntu\"}"

结论

本篇只是 Logstash 最基础的安装和测试，若要应用到生产中，还有更多内容要深入掌握。

tag:

    缺失模块。
    1、请确保node版本大于6.2
    2、在博客根目录（注意不是yilia根目录）执行以下命令：
    npm i hexo-generator-json-content --save

    3、在根目录_config.yml里添加配置： 

      jsonContent:
    meta: false
    pages: false
    posts:
      title: true
      date: true
      path: true
      text: false
      raw: false
      content: false
      slug: false
      updated: false
      comments: false
      link: false
      permalink: false
      excerpt: false
      categories: false
      tags: true

那年二十七