elasticsearch+kibana搭建配置初体验

最近分析一些日志，搭建elasticsearch+kibana玩一玩。

0x00 环境准备

ubuntu 18.04(不用docker推荐使用centos装)

docker --version
Docker version 18.09.7, build 2d0083d

0x00 环境搭建

1. Docker

apt install docker.io
sudo systemctl start docker
sudo systemctl enable docker

docker代理

sudo mkdir -p /etc/systemd/system/docker.service.d

在服务目录下新建代理配置文件并添加内容
NO_PROXY是不需要代理的地址，比如本地及本地私有仓库等

vi /etc/systemd/system/docker.service.d/http-proxy.conf

[Service]  Environment="HTTP_PROXY=http://ip:port/"

2. portainer

docker可视化工具portainer
https://www.portainer.io/installation/

$ docker volume create portainer_data
$ docker run -d -p 8000:8000 -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer

3. elasticsearch

docker pull elasticsearch:7.2.0

注：7.2没有type概念，以索引为主，开发者认为不应该把elasticsearch当作单纯数据库看待

访问可视化工具->volumns模块
创建本地卷永久存放elasticsearch日志，数据，配置目录：

3.1 设置卷映射

/usr/share/elasticsearch/data -> es_data
/usr/share/elasticsearch/config -> es_config

3.2 设置端口映射

3.3 设置单机模式（参考hub）

https://hub.docker.com/_/elasticsearch

discovery.type=single-node

3.4 修改elasticsearch配置文件

修改完后启动

/var/lib/docker/volumes/es_config/_data# 
cat elasticsearch.yml
cluster.name: "docker-cluster"

# 设置局域网可外连
network.host: 0.0.0.0

# 设置写入缓存清理和限制
indices.fielddata.cache.size: 75%
indices.breaker.fielddata.limit: 85%

# 设置外连否则es会拒绝跨域和一些不允许方法
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-methods: OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-headers: X-Requested-With, Content-Type, Content-Length, X-User

3.5 测试访问

3.6 安装中文ik分词插件

cd /usr/share/elasticsearch/plugins
mkdir ik
curl -O https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.2.0/elasticsearch-analysis-ik-7.2.0.zip
docker restart [docker-id]

4. kibana搭建

docker pull kibana:7.2.0

port map :5601
/usr/share/kibana/config -> kibana_config(本地卷)

4.1 修改配置

/var/lib/docker/volumes/kibana_config/_data# cat kibana.yml
#
# ** THIS IS AN AUTO-GENERATED FILE **
#

# Default Kibana configuration for docker target
server.name: kibana
server.host: "0"
# 配置host
elasticsearch.hosts: [ "http://192.168.123.135:9200" ]
xpack.monitoring.ui.container.elasticsearch.enabled: true

# 设置日志存放
logging.dest: /usr/share/kibana/config/kibana-log.txt

4.2 测试启动

0X04 es常用语句

以下语句均为7.2环境

1.1 新增索引

PUT test1
{
    "mappings" : {
        "properties" : {
            "field1" : { "type" : "text" }
        }
    }
}

1.2 新增分词索引

PUT data1
{
  "settings":{
      "analysis":{
         "analyzer":{
            "email_analyzer":{ 
               "tokenizer":"standard",
               "filter":[
                  "lowercase"
               ]
            }
          }
        }
    },
    "mappings" : {
        "properties" : {
            "username" : {                 
              "type": "text",
              "analyzer": "ik_max_word",
              "search_analyzer": "ik_smart"
            },
            "email":{
              "type": "text",
              "analyzer": "email_analyzer",
              "search_analyzer": "email_analyzer"
            },            
            "sex":{
              "type": "keyword"
            },
            "address" : {                 
              "type": "text",
              "analyzer": "ik_max_word",
              "search_analyzer": "ik_smart"
            },
        }
    }
}

1.3 查看索引

http://10.10.10.10:9200/_cat/indices

1.4 查看数据

查看test1索引下序号为1的数据

GET test1/_doc/1

1.5 搜索数据

http://10.10.10.10:9200/hello/_search?pretty&size=50&from=50

1.6 范围删除

删除data1索引下_seq_no序号大于等于50的数据

POST data1/_delete_by_query
{
  "query": {
    "range" : {
        "_seq_no" : {
           "gte" : 50
        }
    }
  }
}

1.7 group by查询

检索处所有source字段

GET data1/_search
{
  "aggs":{
    "models":{
      "terms":{
        "field":"source"
      }
    }
  }
}

0X05 es常用语句

Bulk 批量插入

Mysql To Elasticsearch

https://blog.csdn.net/weixin_39198406/article/details/82983256

0X06 异常报错

ELASTICSEARCH CIRCUITBREAKINGEXCEPTION FIELDDATA DATA TOO LARGE

尝试添加文章中配置文件限制cache的配置
其次top看一下内存占用，应该是不够用了。