| 本文已被：浏览 5219次 下载 10534次 |
码上扫一扫！ |
|
|
| 基于条件随机场方法的开放领域新词发现 |
|
陈飞1,2,3, 刘奕群1,2,3, 魏超1,2,3, 张云亮3, 张敏1,2,3, 马少平1,2,3
|
|
1.智能技术与系统国家重点实验室(清华大学), 北京 100084;2.清华大学 清华信息科学与技术国家实验室(清华大学)(筹), 北京 100084;3.清华大学 计算机科学与技术系, 北京 100084
|
|
| 摘要: |
| 开放领域新词发现研究对于中文自然语言处理的性能提升有着重要的意义.利用条件随机场(condition random field,简称CRF)可对序列输入标注的特点,将新词发现问题转化为预测已分词词语边界是否为新词边界的问题.在对海量规模中文互联网语料进行分析挖掘的基础上,提出了一系列区分新词边界的统计特征,并采用CRF方法综合这些特征实现了开放领域新词发现的算法,同时比较了K-Means 聚类、等频率、基于信息增益这3 种离散化方法对新词发现结果的影响.通过在SogouT 大规模中文语料库上的新词发现实验,验证了所提出的方法有较好的效果. |
| 关键词: 新词发现 condition random field(CRF) 中文分词 |
| DOI：10.3724/SP.J.1001.2013.04254 |
| 分类号: |
| 基金项目:国家自然科学基金(60903107, 61073071); 国家高技术研究发展计划(863)(2011AA01A205) |
|
| Open Domain New Word Detection Using Condition Random Field Method |
|
CHEN Fei1,2,3, LIU Yi-Qun1,2,3, WEI Chao1,2,3, ZHANG Yun-Liang3, ZHANG Min1,2,3, MA Shao-Ping1,2,3
|
|
1.State Key Laboratory of Intelligent Technology and Systems (Tsinghua University), Beijing 100084, China;2.Tsinghua National Laboratory for Information Science and Technology (Tsinghua University), Beijing 100084, China;3.Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China
|
| Abstract: |
| Open domain new word detection is vital for Chinese natural language processing research. This paper proposes a novel detection algorithm based condition random field (CRF), which treats the new word detection problem as a classification problem. In this algorithm, the study tries to separate boundaries of new words from existing words with both the CRF method and a serial of statistical features extracted from large scale corpus. The effectiveness of three different discretization strategies are also compared including K-means, equal-frequency, and information gain. Experimental results on a large-scale Web corpus named SogouT show the effectiveness of the proposed algorithms. |
| Key words: new word detection conditional random field Chinese word segmentation |
