elasticsearch + elasticsearch-analysis-ik中文分词优化记录
库中数据有一批老旧的数据，还有最近刚录入的新数据，查询是需要匹配后优先显示最新录入的，最开始i只是用的简单的分词和匹配，但是查询结果一直不如意最后改造了一下！首先建立索引后进行设置mapping，其中name是我要用ik进行分词的字段，直接建立在建立mapping时增加解析 ik_smart建立的mapping如下{ "properties":{ "createTime": { "type": "date" }, "fileSize": { "type": "text" }, "id": { "type": "text" }, "name": { "type": "text", "analyzer": "ik_smart" }, "suffix": { "type": "keyword" }, "updateTime": { "type": "date" }, "url": { "type": "text" } } }在SpringBoot中查询逻辑如下，注意在搜索时需要使用解析器ik_smart进行分词NativeSearchQueryBuilder nativeSearchQueryBuilder = new NativeSearchQueryBuilder().withPageable(pageable); BoolQueryBuilder queryBuilders = QueryBuilders.boolQuery(); // 使用IK分词 // 2<70% 如果分词数量小于等于2则全部必须 否则需要配70%左右 // operator(Operator.AND) 分词条件是并且 // boost(10) 分值乘以10 // withMinScore(5f) 最小分值为5分 // 最后按照时间排序 queryBuilders.must( QueryBuilders.matchQuery("name",eslz.getName()) .analyzer("ik_smart") .minimumShouldMatch("2<70%") .operator(Operator.AND) .boost(10) ); nativeSearchQueryBuilder .withMinScore(5f) .withSort(SortBuilders.fieldSort("createTime").order(SortOrder.DESC)); // 高亮 HighlightBuilder highlightBuilder = new HighlightBuilder(); highlightBuilder.field("name"); highlightBuilder.preTags(""); highlightBuilder.postTags(""); nativeSearchQueryBuilder.withHighlightBuilder(highlightBuilder); NativeSearchQuery searchQuery = nativeSearchQueryBuilder.withQuery(queryBuilders).build(); SearchHits<EsLz> searchHits = elasticsearchRestTemplate.search(searchQuery,EsLz.class); 最终搜索结果还不错
1. 安装插件1.1 安装插件拼音分词器：https://github.com/medcl/elasticsearch-analysis-pinyin 中文分词器：https://github.com/medcl/elasticsearch-analysis-ik找到自己对应的自己的Elasticsearch版本的插件进行安装Elasticsearch 7.5.1elasticsearch-analysis-ik 7.5.1elasticsearch-analysis-pinyin 7.5.1直接进入Elasticsearch安装目录下，依次进行在线安装./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.5.1/elasticsearch-analysis-ik-7.5.1.zip ./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-pinyin/releases/download/v7.5.1/elasticsearch-analysis-pinyin-7.5.1.zip 安装完成后需要重启 elasticsearch,然后测试分词器是否OK,正常情况下会出现一堆分词结果1.2 测试中文分词器POST http://data:9200/_analyze { "analyzer":"ik_smart", "text":"新型冠状病毒" }分词结果{ "tokens": [ { "token": "新型", "start_offset": 0, "end_offset": 2, "type": "CN_WORD", "position": 0 }, { "token": "冠状病毒", "start_offset": 2, "end_offset": 6, "type": "CN_WORD", "position": 1 } ] }1.3 测试拼音分词器POST http://data:9200/_analyze { "analyzer":"pinyin", "text":"新型冠状病毒" }分词结果{ "tokens": [ { "token": "xin", "start_offset": 0, "end_offset": 0, "type": "word", "position": 0 }, { "token": "xxgzbd", "start_offset": 0, "end_offset": 0, "type": "word", "position": 0 }, { "token": "xing", "start_offset": 0, "end_offset": 0, "type": "word", "position": 1 }, { "token": "guan", "start_offset": 0, "end_offset": 0, "type": "word", "position": 2 }, { "token": "zhuang", "start_offset": 0, "end_offset": 0, "type": "word", "position": 3 }, { "token": "bing", "start_offset": 0, "end_offset": 0, "type": "word", "position": 4 }, { "token": "du", "start_offset": 0, "end_offset": 0, "type": "word", "position": 5 } ] }2. 修改解析器修改分词器，以下所有操作均是对song 索引库进行的操作2.1 关闭索引首先关闭索引，否则会报错的POST http://data:9200/song/_close { }2.2 配置IK+拼音分词然后自定义分词器,我这里使用的IK_SMART+拼音PUT http://data:9200/song/_settings { "index": { "analysis": { "analyzer": { "ik_pinyin_analyzer": { "type": "custom", "tokenizer": "ik_smart", "filter": "pinyin_filter" } }, "filter": { "pinyin_filter": { "type": "pinyin", "keep_first_letter": false } } } } }你也可以使用IK_MAX_WORD + 拼音分词PUT http://data:9200/song/_settings { "index": { "analysis": { "analyzer": { "ik_pinyin_analyzer": { "type": "custom", "tokenizer": "ik_max_word", "filter": "pinyin_filter" } }, "filter": { "pinyin_filter": { "type": "pinyin", "keep_first_letter": false } } } } }2.3 开启索引POST http://data:9200/song/_open { }3. 配置mapping(字段)编辑中