flow indexer

流索引器

 flow-indexer indexes flows

Usage: 
  flow-indexer [command]

Available Commands: 
  compact     Compact the database
  daemon      Start daemon
  expandcidr  Expand a CIDR range from those seen in the database
  index       Index flows
  search      Search flows
  help        Help about any command

Flags:
      --dbpath="flows.db": Database path
  -h, --help[=false]: help for flow-indexer


Use "flow-indexer [command] --help" for more information about a command.

Quickstart

安裝

 $ export GOPATH=~/go
$ go get github.com/JustinAzoff/flow-indexer

創建配置

 $ cp ~/go/src/github.com/JustinAzoff/flow-indexer/example_config.json config.json
$ vi config.json # Adjust log paths and database paths.

索引配置如下：

• 名稱 - 索引器的名稱。保持此簡短和小寫，因為您將其用作HTTP查詢參數。
• 後端 - 要使用的後端日誌IP提取器。選擇：Bro，Bro_json，Nfdump，Syslog，PCAP和Argus。
• file_glob-應匹配所有日誌文件的外殼球形圖案。
• 最近的_file_glob-應與今天的日誌文件匹配的Strftime+Shell球形模式。
• filename_to_database_regex-應用於用於提取用於命名數據​​庫的信息的每個文件名應用的正則表達式。
• Database_root-將數據庫寫入的位置。應該是特定於索引的。
• datapath_path-單個數據庫的名稱。這可以包含filename_to_database_regex中設置的$變量。

如何分區數據庫的決定因素是您每天看到多少個獨特的IP。我建議從每月索引開始。如果到本月底的索引性能會大大打擊，請切換到每日索引。

運行初始索引

indexall命令將展開file_glob並索引任何匹配的日誌文件。

 $ ~/go/bin/flow-indexer indexall

開始守護程序

初始索引完成後，開始守護程序。啟動守護程序將展開最近的_file_glob，並索引任何匹配的最近創建的日誌文件。

 $ ~/go/bin/flow-indexer daemon

它將在60秒的循環中執行此操作，以保持自己的最新狀態。

查詢API

 $ curl -s 'localhost:8080/search?i=conn&q=1.2.3.0/24'
$ curl -s 'localhost:8080/dump?i=conn&q=1.2.3.0/24'
$ curl -s 'localhost:8080/stats?i=conn&q=1.2.3.0/24'

服務配置

運行流索引作為服務

Systemd

要使用SystemD在系統上運行Flow-Indexer作為服務，您可以使用提供的Flow-indexer.service文件。

暴發戶

如果您打算在消耗貴族的系統上運行流索引作為服務，則可能需要考慮以下內容，以便從Flow-Indexer中正確使用Syslog STDOUT和STDERR，並作為非root用戶運行。

 # flow-indexer - Flow Indexer
#
# flow-indexer is a service that indexes and allows retrieval of flows using bro logs

description     "Flow Indexer Daemon"

start on runlevel [345]
stop on runlevel [!345]

setuid flowindexer
setgid flowindexer

exec /path/to/bin/flow-indexer daemon --config /path/to/flow-indexer/config.json 2>&1 | logger -t flow-indexer

常見問題

為了避免too many open files錯誤，您可能需要增加允許流量indexer運行的用戶可以訪問的打開文件的數量。這可以通過在/etc/security/limits.conf中更改nofile設置來完成，如下所示。

 flowindexer soft nofile 65535
flowindexer hard nofile 65535

下層命令示例

在實踐中不再真正使用過，守護程序是使用流索引器的推薦方法。但是這些命令對於測試和開發可能很有用。

索引流

 ./flow-indexer --dbpath /tmp/f/flows.db index /tmp/f/conn*
2016/02/06 23:36:51 /tmp/f/conn.00:00:00-01:00:00.log.gz: Read 4260 lines in 24.392765ms
2016/02/06 23:36:51 /tmp/f/conn.00:00:00-01:00:00.log.gz: Wrote 281 unique ips in 2.215219ms
2016/02/06 23:36:51 /tmp/f/conn.01:00:00-02:00:00.log.gz: Read 4376 lines in 24.186168ms
2016/02/06 23:36:51 /tmp/f/conn.01:00:00-02:00:00.log.gz: Wrote 310 unique ips in 1.495277ms
[...]
2016/02/06 23:36:51 /tmp/f/conn.22:00:00-23:00:00.log.gz: Read 7799 lines in 18.350788ms
2016/02/06 23:36:51 /tmp/f/conn.22:00:00-23:00:00.log.gz: Wrote 775 unique ips in 5.155262ms
2016/02/06 23:36:51 /tmp/f/conn.23:00:00-00:00:00.log.gz: Read 5255 lines in 15.296847ms
2016/02/06 23:36:51 /tmp/f/conn.23:00:00-00:00:00.log.gz: Wrote 400 unique ips in 2.910344ms

重新索引流

 ./flow-indexer --dbpath /tmp/f/flows.db index /tmp/f/conn*
2016/02/06 23:37:36 /tmp/f/conn.00:00:00-01:00:00.log.gz Already indexed
2016/02/06 23:37:36 /tmp/f/conn.01:00:00-02:00:00.log.gz Already indexed
2016/02/06 23:37:36 /tmp/f/conn.02:00:00-03:00:00.log.gz Already indexed
2016/02/06 23:37:36 /tmp/f/conn.03:00:00-04:00:00.log.gz Already indexed
[...]
2016/02/06 23:37:36 /tmp/f/conn.20:00:00-21:00:00.log.gz Already indexed
2016/02/06 23:37:36 /tmp/f/conn.21:00:00-22:00:00.log.gz Already indexed
2016/02/06 23:37:36 /tmp/f/conn.22:00:00-23:00:00.log.gz Already indexed
2016/02/06 23:37:36 /tmp/f/conn.23:00:00-00:00:00.log.gz Already indexed

擴展CIDR範圍

 ./flow-indexer --dbpath /tmp/f/flows.db expandcidr 192.30.252.0/24
192.30.252.86
192.30.252.87
192.30.252.92
192.30.252.124
192.30.252.125
192.30.252.126
192.30.252.127
192.30.252.128
192.30.252.129
192.30.252.130
192.30.252.131
192.30.252.141

搜尋

 ./flow-indexer --dbpath /tmp/f/flows.db search 192.30.252.0/24
/tmp/f/conn.03:00:00-04:00:00.log.gz
/tmp/f/conn.04:00:00-05:00:00.log.gz
/tmp/f/conn.06:00:00-07:00:00.log.gz
/tmp/f/conn.14:00:00-15:00:00.log.gz
/tmp/f/conn.18:00:00-19:00:00.log.gz
/tmp/f/conn.20:00:00-21:00:00.log.gz
/tmp/f/conn.22:00:00-23:00:00.log.gz