Izvorni znanstveni članak
https://doi.org/10.20532/cit.2021.1005393
Natural Language Processing Using Neighbour Entropy-based Segmentation
Jianfeng Qiao
orcid.org/0000-0002-4379-0810
; Capital University of Economics and Business, Beijing, China
Xingzhi Yan
; University of Birmingham, UK
Shuran Lv
; Capital University of Economics and Business, Beijing, China
Sažetak
In natural language processing (NLP) of Chinese hazard text collected in the process of hazard identification, Chinese word segmentation (CWS) is the first step to extracting meaningful information from such semi-structured Chinese texts. This paper proposes a new neighbor entropy-based segmentation (NES) model for CWS. The model considers the segmentation benefits of neighbor entropies, adopting the concept of "neighbor" in optimization research. It is defined by the benefit ratio of text segmentation, including benefits and losses of combining the segmentation unit with more information than other popular statistical models. In the experiments performed, together with the maximum-based segmentation algorithm, the NES model achieves a 99.3% precision, 98.7% recall, and 99.0% f-measure for text segmentation; these performances are higher than those of existing tools based on other seven popular statistical models. Results show that the NES model is a valid CWS, especially for text segmentation requirements necessitating longer-sized characters. The text corpus used comes from the Beijing Municipal Administration of Work Safety, which was recorded in the fourth quarter of 2018.
Ključne riječi
Text Mining; Text Segmentation; Chinese Word Segmentation; Safety Management; Hazard Analysis
Hrčak ID:
280125
URI
Datum izdavanja:
4.7.2022.
Posjeta: 345 *