POST _analyze
{"text":"
Where is my cheese?
","tokenizer": "standard", "char_filter": ["html_strip"]
}
//返回：
{"tokens": [{"token": "Where","start_offset": 4,"end_offset": 9,"type": "","position": 0},{"token": "is","start_offset": 10,"end_offset": 12,"type": "","position": 1},{"token": "my","start_offset": 13,"end_offset": 15,"type": "","position": 2},{"token": "cheese","start_offset": 16,"end_offset": 22,"type": "","position": 3}]
}

字符过滤器只是从输入字段中剥离

标签，但是有时候一些标签需要保留，例如：业务需求可能是从句子中去除

标签，但保留预格式化 (

) 标签；这时候就需要使用所需的自定义分析器去创建索引。
 
PUT 
{"settings": {"analysis": {"analyzer": {									# 这这里设置分析器"ANALYZER_NAME":{							# 分析器名"tokenizer":"TOKENIZER", 					# 设置分词器"char_filter":["CHAR_FILTER_NAME"]		# 设置过滤器}},"char_filter": {								# 这里设置过滤器"CHAR_FILTER_NAME":{						# 过滤器名"type":"FILTER",							# 设置过滤器类型"escaped_tags":["html标签","pre","br", "...."] # 需要保留的标签}}}}
}
 
2、mapping 字符过滤器
 
mapping 字符过滤器用指定的替换替换任何出现的指定字符串。
 
POST _analyze
{"text": "I am from CN","char_filter": [{"type": "mapping","mappings": ["CN => 中国"]}]
}
//返回：
{"tokens": [{"token": "I am from 中国","start_offset": 0,"end_offset": 12,"type": "word","position": 0}]
}
 
自定义分析器创建索引：
 
PUT 
{"settings": {"analysis": {"analyzer": {"ANALYZER_NAME": { 						# 分析器名"tokenizer": "TOKENIZER",				# 选择分词器"char_filter": [						# 选择过滤器"CHAR_FILTER_NAME" 			]}},"char_filter": {"CHAR_FILTER_NAME": { 					# 过滤器名"type": "FILTER",						# 设置过滤器类型"mappings": [ 						# 替换配置...]}}}}
}
 
3、patter replace 字符过滤器
 
pattern_replace 字符过滤器用指定的字符替换与正则表达式匹配的任何字符。
 
PUT 
{"settings": {"analysis": {"analyzer": {"ANALYZER_NAME":{ "tokenizer":"keyword","char_filter":["CHAR_FILTER_NAME"] }},"char_filter": { "CHAR_FILTER_NAME":{ "type":"FILTER", "pattern":"被替换字符", "replacement":"替换后的字符" }}}}
}
 
三、分词器（tokenizers）
 
分词器（tokenizers）是对文本进行分析处理的一种手段，基本处理逻辑为按照预先制定的分词规则，把原始文档分割成若干更小粒度的词项（term），粒度大小取决于分词器规则。
 
设置分词器：
 
PUT /
{"settings": {}, # 可在 analysis 下设置自定义分词器"mappings": {"properties": {"FIELD_NAME":{"type": "text","analyzer": "standard|letter|whitespace|lowercase|..." 	# 指定分词器}}}
}
 
下面简单介绍几个常用的系统默认分词器：
 
1、standard tokenizer（标准分词器）
 
标准分词器类型是 standard，用于大多数欧洲语言，使用 unicode文本分割算法 对文档进行分词。
 
//测试：
GET /_analyze
{"tokenizer" : "standard","text" : "I am from CN"
}
//分词结果：
{"tokens": [{"token": "I","start_offset": 0,"end_offset": 1,"type": "","position": 0},{"token": "am","start_offset": 2,"end_offset": 4,"type": "","position": 1},{"token": "from","start_offset": 5,"end_offset": 9,"type": "","position": 2},{"token": "CN","start_offset": 10,"end_offset": 12,"type": "","position": 3}]
}
 
2、letter tokenizer（字母分词器）
 
字符分词器类型是 letter，在非字母位置上分割文本，这就是说，根据相邻的词之间是否存在非字母（例如空格，逗号等）的字符，对文本进行分词，对大多数欧洲语言非常有用。
 
//测试：
GET /_analyze
{"tokenizer" : "letter","text" : "I'm from CN"
}
//分词结果：
{"tokens": [{"token": "I","start_offset": 0,"end_offset": 1,"type": "word","position": 0},{"token": "m","start_offset": 2,"end_offset": 3,"type": "word","position": 1},{"token": "from","start_offset": 4,"end_offset": 8,"type": "word","position": 2},{"token": "CN","start_offset": 9,"end_offset": 11,"type": "word","position": 3}]
}
 
3、whitespace tokenizer（空格分词器）
 
空格分词类型是 whitespace，在空格处分割文本。
 
GET /_analyze
{"tokenizer" : "whitespace","text" : "I'm from CN"
}
//分词结果：
{"tokens": [{"token": "I'm","start_offset": 0,"end_offset": 3,"type": "word","position": 0},{"token": "from","start_offset": 4,"end_offset": 8,"type": "word","position": 1},{"token": "CN","start_offset": 9,"end_offset": 11,"type": "word","position": 2}]
}
 
4、lowercase tokenizer（小写分词器）
 
小写分词器类型是 lowercase，在非字母位置上分割文本，并把分词转换为小写形式，功能上是 letter tokenizer 和 lower case token filter 的结合，但是性能更高，一次性完成两个任务。
 
GET /_analyze
{"tokenizer" : "lowercase","text" : "I'm from CN"
}
//分词结果：
{"tokens": [{"token": "i","start_offset": 0,"end_offset": 1,"type": "word","position": 0},{"token": "m","start_offset": 2,"end_offset": 3,"type": "word","position": 1},{"token": "from","start_offset": 4,"end_offset": 8,"type": "word","position": 2},{"token": "cn","start_offset": 9,"end_offset": 11,"type": "word","position": 3}]
}
 
5、classic tokenizer（经典分词器）
 
经典分词器类型是 classic ，基于语法规则对文本进行分词，对英语文档分词非常有用，在处理首字母缩写，公司名称，邮件地址和Internet主机名上效果非常好。
 
GET /_analyze
{"tokenizer" : "classic","text" : "I'm from CN"
}
// 分词结果：
{"tokens": [{"token": "I'm","start_offset": 0,"end_offset": 3,"type": "","position": 0},{"token": "from","start_offset": 4,"end_offset": 8,"type": "","position": 1},{"token": "CN","start_offset": 9,"end_offset": 11,"type": "","position": 2}]
}
 
6、其他分词器
 
等其它多种分词器，这里不再一一列出，因为不是经常用的。
 
其它分词器参考官网：Tokenizer。
 
四、词项过滤器（token filter）
 
1、ASCII folding token filter（ASCII折叠分词过滤器）
 
将不在 Basic Latin Unicode 块中的字母、数字和符号字符（前 127 个 ASCII 字符）转换为它们的 ASCII 等效字符（如果存在）。 例如，过滤器将 à 更改为 a。
 
GET /_analyze
{"tokenizer" : "standard","filter" : ["asciifolding"],"text" : "açaí à la carte"
}
//返回：
{"tokens": [{"token": "acai","start_offset": 0,"end_offset": 4,"type": "","position": 0},{"token": "a","start_offset": 5,"end_offset": 6,"type": "","position": 1},{"token": "la","start_offset": 7,"end_offset": 9,"type": "","position": 2},{"token": "carte","start_offset": 10,"end_offset": 15,"type": "","position": 3}]
}
 
2、length token filter（长度标记过滤器）
 
删除比指定字符长度更短或更长的标记，它是可配置的，我们可以在settings中设置。 例如，你可以使用长度过滤器来排除短于 2 个字符的标记和长于 5 个字符的标记。
 
GET _analyze
{"tokenizer": "whitespace","filter": [{"type": "length","min": 0,"max": 4}],"text": "the quick brown fox jumps over the lazy dog"
}
//返回：
{"tokens": [{"token": "the","start_offset": 0,"end_offset": 3,"type": "word","position": 0},{"token": "fox","start_offset": 16,"end_offset": 19,"type": "word","position": 3},{"token": "over","start_offset": 26,"end_offset": 30,"type": "word","position": 5},{"token": "the","start_offset": 31,"end_offset": 34,"type": "word","position": 6},{"token": "lazy","start_offset": 35,"end_offset": 39,"type": "word","position": 7},{"token": "dog","start_offset": 40,"end_offset": 43,"type": "word","position": 8}]
}
 
3、lowercase token filter（小写分词过滤器）
 
将分词规范化为小写，它通过language参数支持希腊语、爱尔兰语和土耳其语小写标记过滤器。
 
GET _analyze
{"tokenizer" : "standard","filter" : ["lowercase"],"text" : "THE Quick FoX JUMPs"
}
//返回：
{"tokens": [{"token": "the","start_offset": 0,"end_offset": 3,"type": "","position": 0},{"token": "quick","start_offset": 4,"end_offset": 9,"type": "","position": 1},{"token": "fox","start_offset": 10,"end_offset": 13,"type": "","position": 2},{"token": "jumps","start_offset": 14,"end_offset": 19,"type": "","position": 3}]
}
 
4、uppercase token filter（大写分词过滤器）
 
将分词规范为大写。
 
GET _analyze
{"tokenizer" : "standard","filter" : ["uppercase"],"text" : "the Quick FoX JUMPs"
}
//返回：
{"tokens": [{"token": "THE","start_offset": 0,"end_offset": 3,"type": "","position": 0},{"token": "QUICK","start_offset": 4,"end_offset": 9,"type": "","position": 1},{"token": "FOX","start_offset": 10,"end_offset": 13,"type": "","position": 2},{"token": "JUMPS","start_offset": 14,"end_offset": 19,"type": "","position": 3}]
}
 
5、其他词项过滤器
 
其余分词过滤器不一一列举。详情参见 官网。
 
五、中文分词器
 
中文分词器主要介绍 IK 分词器 和 拼音分词器。
 
1、ik分词器 - 常用
 
安装参考：Elasticsearch：IK 中文分词器。
 
IK Analyzer是一个开源的，基于java语言开发的轻量级的中文分词工具包。
 
采用了特有的“正向迭代最细粒度切分算法“，支持细粒度和最大词长两种切分模式；
具有83万字/秒（1600KB/S）的高速处理能力。
采用了多子处理器分析模式，支持：英文字母、数字、中文词汇等分词处理，兼容韩文、日文字符优化的词典存储，更小的内存占用。
支持用户词典扩展定义，针对Lucene全文检索优化的查询分析器 IKQueryParser；
引入简单搜索表达式，采用歧义分析算法优化查询关键字的搜索排列组合，能极大的提高Lucene检索的命中率。
 

 ik 的分词粒度：
 
ik_max_word：会将文本做较细粒度的拆分，例如「我爱北京天安门」会被拆分为「我、爱、北京、天安门、天安、门」，会穷尽各种可能的组合。
ik_smart：会将文本做简单粗粒度的拆分，例如「我爱北京天安门」会被拆分为「我、爱、北京、天安门」。
 
GET /_analyze
{"analyzer" : "ik_max_word|ik_smart","text" : "我爱北京天安门"
}
 
2、拼音分词器
 
安装使用参考：Elasticsearch：Pinyin 分词器。
 
拼音分词器可以让用户输入拼音，就能查找到相关的关键词。比如在某个商城搜索中，输入yonghui，就能匹配到永辉。
 
3、简体繁体转换分词器
 
安装使用参考：Elasticsearch：简体繁体转换分词器 - STConvert analysis。