关于ELK是什么、做什么用，我们不在此讨论。本文重点在如何实现快速方便地安装logstash和filebeat组件，特别是在近千台DB Server的环境下（为了安全保守，公司DB Server 目前尚未部署saltstack一类的管控软件）。在尽可能标准化的条件下，希望可以实现一键化安装。下面是我们功能实现的一些尝试，我们把手动一步步操作打包提炼到一个sh文档中，安装部署时只要执行sh文件即可。部署安装logstash和filebeat组件由原来的10分钟缩减到目前的1分钟左右，并且减少了因手动部署带来的误操作。
1.logstash和filebeat安装包所在指定路径下
logstash的安装包logstash-7.6.0.zip所在路径
/data/logstash/logstash-7.6.0.zip
filebeat的安装包filebeat-7.4.2-linux-x86_64.tar.gz所在路径
2.上传经过标准化的程序配置文件
上传程序的配置文件filebeat.service、filebeat.yml、logstash.conf、startup.options到指定位置，这些文件是格式化后的，不是解压的默认文件，目的是方便替换安装。
如何想直接使用disposelogcollectot.sh文件，上传的路径一定要是/tmp/
3.编写一键安装的可执行文件disposelogcollectot.sh
#!/bin/bash# The version is defined V.001# Version ModifyTime ModifyBy Desc# Ver001 2018-03-25 Carson.Xu Create the Scripts File# Desc: This file is used to despose filebeat \ logstash in order to collect slow log and error log from mysqld.#### step 1 判断 需要上传的文件是否已上传cd /tmp/if [ -f "filebeat.service" -a -f "filebeat.yml" -a -f "logstash.conf" -a -f "startup.options" ]then echo 'step 1 安装过程需要的文件已到位,上传文件项检查通过....'else echo "step 1 安装过程中需要的filebeat.service、 filebeat.yml、 logstash.conf、 startup.options，没有到位，不能继续安装，安装进程退出!!!" exitfi#### step 2 解压指定文件cd /data/logstash/unzip logstash-7.6.0.zipecho 'step 2 解压logstash项工作完成....'sleep 3#### step 3 删除解压后的指定文件cd logstash-7.6.0/config/rm -rf startup.optionsecho 'step 3 删除解压后的指定文件startup.options工作完成....'sleep 3#### step 4 转移上传的文件mv /tmp/logstash.conf /tmp/startup.options -t /data/logstash/logstash-7.6.0/config/echo 'step 4 转移文件logstash.conf的工作完成....'sleep 2#### step 5 修改log上传的ES 索引[必做 建议用业务名称替换，例如qq/weixin/rewu]read -p "请输入业务名称:" productecho -e "\n"echo "用户名为:$product"sed -i "s/qqweixinface/$product/" /data/logstash/logstash-7.6.0/config/logstash.confecho 'step 5 删除解压后的指定文件startup.options工作完成....'sleep 2##### step 6 安装logstash 服务/data/logstash/logstash-7.6.0/bin/system-installecho 'step 6 安装logstash 服务工作完成....'sleep 3##### step 7 解压缩filebeat文件cd /data/filebeat/tar -zxvf filebeat-7.4.2-linux-x86_64.tar.gzecho 'step 7 解压缩filebeat文件工作完成....'sleep 3#### step 8 转移上传的filebeat.yml，允许覆盖掉生成默认配置文件rm -rf /data/filebeat/filebeat-7.4.2-linux-x86_64/filebeat.ymlmv /tmp/filebeat.yml /data/filebeat/filebeat-7.4.2-linux-x86_64/echo 'step 8 转移上传的filebeat.yml，允许覆盖掉生成默认配置文件工作完成....'sleep 2#### step 9 权限调整cd /data/filebeat/filebeat-7.4.2-linux-x86_64chown -R root:root filebeat.ymlchmod 600 filebeat.ymlecho 'step 9 调整filebeat文件权限的工作完成....'sleep 2##### step 10 获取 Server IPip=$(ip a|awk -F "inet|/" '/inet.*brd/ {print $2}'|head -n 1)serverid=$(echo $ip) #去除左右空格echo $serveridecho 'step 10 获取Server IP的工作完成....'echo '获取Server IP的为：' $serveridsleep 1###### step 11 调整host配置[必做 IP替换]sed -i "s/119.119.119.119/$serverid/" /data/filebeat/filebeat-7.4.2-linux-x86_64/filebeat.ymlecho 'step 11 替换配置文件中的Server IP工作完成....'sleep 1###### step 12 转移服务文件mv /tmp/filebeat.service /etc/systemd/system/echo 'step 12 将filebeat服务的文件移动到指定位置的工作完成....'sleep 1###### step 13 服务设置及启动systemctl enable logstash.servicesystemctl enable filebeat.serviceecho 'step 13 将服务设置为自启动的工作完成....'systemctl start logstash.servicesleep 20systemctl start filebeat.servicesleep 10##### step 14 检查服务是否已正常启动logstashservice_check_result=`systemctl status logstash.service | grep "active (running)"| wc -l`if [ "$logstashservice_check_result" == "1" ]then echo 'step 14 检查logstash.service已启动....'else echo "step 14 检查logstash.service未正常启动....，安装进程退出!!!" exitfisleep 3filebeatservice_check_result=`systemctl status filebeat.service | grep "active (running)"| wc -l`if [ "$filebeatservice_check_result" == "1" ]then echo 'step 14 检查filebeat.service已启动....'else echo "step 14 检查filebeat.service未正常启动....，安装进程退出!!!" exitfi###### step 15 安装过程结束echo 'step 15 安装过程结束'4.执行
chmod 755 disposelogcollectot.sh
sh -x disposelogcollectot.sh
5.附录
在附录这一部分，介绍刚刚上传的文件--filebeat.service、filebeat.yml、logstash.conf、startup.options
5.1 文件startup.options
这一个文件主要描述了logstash程序的启动配置
################################################################################# These settings are ONLY used by $LS_HOME/bin/system-install to create a custom# startup script for Logstash and is not used by Logstash itself. It should# automagically use the init system (systemd, upstart, sysv, etc.) that your# Linux distribution uses.## After changing anything here, you need to re-run $LS_HOME/bin/system-install# as root to push the changes to the init script.################################################################################# Override Java location#JAVACMD=/usr/bin/java# Set a home directoryLS_HOME=/data/logstash/logstash-7.6.0# logstash settings directory, the path which contains logstash.ymlLS_SETTINGS_DIR=/data/logstash/logstash-7.6.0/config# Arguments to pass to logstashLS_OPTS="--path.settings ${LS_SETTINGS_DIR} -f /data/logstash/logstash-7.6.0/config/logstash.conf"# Arguments to pass to javaLS_JAVA_OPTS=""# pidfiles aren't used the same way for upstart and systemd; this is for sysv users.LS_PIDFILE=/var/run/logstash.pid# user and group id to be invoked asLS_USER=rootLS_GROUP=root# Enable GC logging by uncommenting the appropriate lines in the GC logging# section in jvm.optionsLS_GC_LOG_FILE=/var/log/logstash/gc.log# Open file limitLS_OPEN_FILES=16384# Nice levelLS_NICE=19# Change these to have the init script named and described differently# This is useful when running multiple instances of Logstash on the same# physical box or vmSERVICE_NAME="logstash"SERVICE_DESCRIPTION="logstash"# If you need to run a command or script before launching Logstash, put it# between the lines beginning with `read` and `EOM`, and uncomment those lines.##### read -r -d '' PRESTART << EOM## EOM5.2 附件logstash.conf
这个文件主要说明的是格式化读取的数据 以及 如何保存到elasticsearch中
# Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { beats { port => 5044 }}filter { if [fields][log_type] == "mysql-slow" { grok { match => ["message", "(?m)^#\s+Time:\s+%{TIMESTAMP_ISO8601}\s*#\s+User@Host:\s+(?<user>.*)\[%{USERNAME:user}?\]\s*@\s*%{IPORHOST:client}?\s*\[%{IPORHOST:client}?\]\s+Id:\s+%{BASE10NUM}\s*#\s+Query_time:\s+%{BASE10NUM:query_time}\s+Lock_time:\s+%{BASE10NUM:lock_time}\s+Rows_sent:\s+%{BASE10NUM:rows_sent}\s+Rows_examined:\s+%{BASE10NUM:rows_examined}\s*(use\s+%{DATA:database};\s*)?SET\s+timestamp=%{BASE10NUM:timestamp};\s*%{GREEDYDATA:sql_stmt}$"] keep_empty_captures => true } date { match => ["timestamp", "UNIX"] remove_field => ["timestamp"] } mutate { convert => { "query_time" => "float" "lock_time" => "float" "rows_sent" => "integer" "rows_examined" => "integer" } remove_field => ["@version", "beat", "host", "input", "log", "offset", "prospector", "source", "tags"] } } if [fields][log_type] == "mysql-error" { grok { match => ["message", "(?m)^%{TIMESTAMP_ISO8601:timestamp} %{BASE10NUM} \[%{WORD:error_level}\] %{GREEDYDATA:error_msg}$"] } date { match=> ["timestamp", "ISO8601"] remove_field => ["timestamp"] } mutate { remove_field => ["@version", "beat", "host", "input", "log", "offset", "prospector", "source", "tags"] } }}output { elasticsearch { hosts => [""] #index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" index => "%{[fields][log_type]}-qqweixinface-%{+YYYY.MM.dd}" user => "qquid_es" password => "xiang_ni_123+yidiandian" }}5.3 附件filebeat.yml
这个文件主要说明了filebeat读取什么log,已经对读取的数据如何处理
###################### Filebeat Configuration Example ########################## This file is an example configuration file highlighting only the most common# options. The filebeat.reference.yml file from the same directory contains all the# supported options with more comments. You can use it as a reference.## You can find the full configuration reference here:# https://www.elastic.co/guide/en/beats/filebeat/index.html# For more available modules and options, please see the filebeat.reference.yml sample# configuration file.#=========================== Filebeat inputs =============================filebeat.inputs:# Each - is an input. Most options can be set at the input level, so# you can use different inputs for various configurations.# Below are the input specific configurations.- type: log # Change to true to enable this input configuration. #enabled: false # Paths that should be crawled and fetched. Glob based paths. #paths: #- /var/log/*.log #- c:\programdata\elasticsearch\logs\* paths: - /data/mysql/data/slow.log fields: log_type: mysql-slow db_host: 119.119.119.119 db_port: 3306 multiline.pattern: "^# Time:" multiline.negate: true multiline.match: after- type: log paths: - /data/mysql/data/error.log fields: log_type: mysql-error db_host: 119.119.119.119 db_port: 3306 multiline.pattern: ^20\d{2}-\d{2}-\d{2}T multiline.negate: true multiline.match: after # Exclude lines. A list of regular expressions to match. It drops the lines that are # matching any regular expression from the list. #exclude_lines: ['^DBG'] # Include lines. A list of regular expressions to match. It exports the lines that are # matching any regular expression from the list. #include_lines: ['^ERR', '^WARN'] # Exclude files. A list of regular expressions to match. Filebeat drops the files that # are matching any regular expression from the list. By default, no files are dropped. #exclude_files: ['.gz$'] # Optional additional fields. These fields can be freely picked # to add additional information to the crawled log files for filtering #fields: # level: debug # review: 1 ### Multiline options # Multiline can be used for log messages spanning multiple lines. This is common # for Java Stack Traces or C-Line Continuation # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [ #multiline.pattern: ^\[ # Defines if the pattern set under pattern should be negated or not. Default is false. #multiline.negate: false # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern # that was (not) matched before or after or as long as a pattern is not matched based on negate. # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash #multiline.match: after#============================= Filebeat modules ===============================filebeat.config.modules: # Glob pattern for configuration loading path: ${path.config}/modules.d/*.yml # Set to true to enable config reloading reload.enabled: false # Period on which files under path should be checked for changes #reload.period: 10s#==================== Elasticsearch template setting ==========================setup.template.settings: index.number_of_shards: 1 #index.codec: best_compression #_source.enabled: false#================================ General =====================================# The name of the shipper that publishes the network data. It can be used to group# all the transactions sent by a single shipper in the web interface.#name:# The tags of the shipper are included in their own field with each# transaction published.#tags: ["service-X", "web-tier"]# Optional fields that you can specify to add additional information to the# output.#fields:# env: staging#============================== Dashboards =====================================# These settings control loading the sample dashboards to the Kibana index. Loading# the dashboards is disabled by default and can be enabled either by setting the# options here or by using the `setup` command.#setup.dashboards.enabled: false# The URL from where to download the dashboards archive. By default this URL# has a value which is computed based on the Beat name and version. For released# versions, this URL points to the dashboard archive on the artifacts.elastic.co# website.#setup.dashboards.url:#============================== Kibana =====================================# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.# This requires a Kibana endpoint configuration.setup.kibana: # Kibana Host # Scheme and port can be left out and will be set to the default (http and 5601) # In case you specify and additional path, the scheme is required: :5601/path # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601 #host: "localhost:5601" # Kibana Space ID # ID of the Kibana Space into which the dashboards should be loaded. By default, # the Default Space will be used. #space.id:#============================= Elastic Cloud ==================================# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).# The cloud.id setting overwrites the `output.elasticsearch.hosts` and# `setup.kibana.host` options.# You can find the `cloud.id` in the Elastic Cloud web UI.#cloud.id:# The cloud.auth setting overwrites the `output.elasticsearch.username` and# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.#cloud.auth:#================================ Outputs =====================================# Configure what output to use when sending the data collected by the beat.#-------------------------- Elasticsearch output ------------------------------#output.elasticsearch: # Array of hosts to connect to. #hosts: ["localhost:9200"] # Optional protocol and basic auth credentials. #protocol: "https" #username: "elastic" #password: "changeme"#----------------------------- Logstash output --------------------------------output.logstash: # The Logstash hosts hosts: ["localhost:5044"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key"#================================ Processors =====================================# Configure processors to enhance or manipulate events generated by the beat.processors: - add_host_metadata: ~ - add_cloud_metadata: ~#================================ Logging =====================================# Sets log level. The default log level is info.# Available log levels are: error, warning, info, debug#logging.level: debug# At debug level, you can selectively enable logging only for some components.# To enable all selectors use ["*"]. Examples of other selectors are "beat",# "publish", "service".#logging.selectors: ["*"]#============================== X-Pack Monitoring ===============================# filebeat can export internal metrics to a central Elasticsearch monitoring# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The# reporting is disabled by default.# Set to true to enable the monitoring reporter.#monitoring.enabled: false# Sets the UUID of the Elasticsearch cluster under which monitoring data for this# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.#monitoring.cluster_uuid:# Uncomment to send the metrics to Elasticsearch. Most settings from the# Elasticsearch output are accepted here as well.# Note that the settings should point to your Elasticsearch *monitoring* cluster.# Any setting that is not set is automatically inherited from the Elasticsearch# output configuration, so if you have the Elasticsearch output configured such# that it is pointing to your Elasticsearch monitoring cluster, you can simply# uncomment the following line.#monitoring.elasticsearch:#================================= Migration ==================================# This allows to enable 6.7 migration aliases#migration.6_to_7.enabled: true5.4.附件filebeat.service
这个文件是关于filebeat.service的定义
[Unit]Description=filebeat.service[Service]User=rootExecStart=/data/filebeat/filebeat-7.4.2-linux-x86_64/filebeat -e -c /data/filebeat/filebeat-7.4.2-linux-x86_64/filebeat.yml[Install]WantedBy=multi-user.target
原文转载：http://www.shaoqun.com/a/608597.html
巴士物流：https://www.ikjzd.com/w/2329
燕文物流：https://www.ikjzd.com/w/2229
关于ELK是什么、做什么用，我们不在此讨论。本文重点在如何实现快速方便地安装logstash和filebeat组件，特别是在近千台DBServer的环境下（为了安全保守，公司DBServer目前尚未部署saltstack一类的管控软件）。在尽可能标准化的条件下，希望可以实现一键化安装。下面是我们功能实现的一些尝试，我们把手动一步步操作打包提炼到一个sh文档中，安装部署时只要执行sh文件即可。部署安
prime：https://www.ikjzd.com/w/129
taofenba：https://www.ikjzd.com/w/1725
新单：https://www.ikjzd.com/w/79
如何提高亚马逊产品转化率？4个提高产品转化率的方法：https://www.ikjzd.com/home/108173
加拿大罢工宣告结束，邮政已经恢复工作！：https://www.ikjzd.com/home/11860
eBay通知：美国有34个州将要求征收互联网销售税：https://www.ikjzd.com/home/109555
没有评论:
发表评论