44 lines
1.2 KiB
Markdown
44 lines
1.2 KiB
Markdown
# 如何训练LM
|
||
|
||
训练脚本详见（[点击此处](../tools/train_compile_ngram.sh)）
|
||
|
||
## 数据准备
|
||
```shell
|
||
# 下载: 示例训练语料text、lexicon 和 am建模单元units.txt
|
||
wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/requirements/lm.tar.gz
|
||
# 如果是匹配8k的am模型，使用 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/requirements/lm_8358.tar.gz
|
||
tar -zxvf lm.tar.gz
|
||
```
|
||
|
||
## 训练arpa
|
||
```shell
|
||
# make sure that srilm is installed
|
||
# the format of the text should be:
|
||
# BAC009S0002W0122 而 对 楼市 成交 抑制 作用 最 大 的 限 购
|
||
# BAC009S0002W0123 也 成为 地方 政府 的 眼中 钉
|
||
|
||
bash fst/train_lms.sh
|
||
```
|
||
|
||
## 生成lexicon
|
||
```shell
|
||
python3 fst/generate_lexicon.py lm/corpus.dict lm/lexicon.txt lm/lexicon.out
|
||
```
|
||
|
||
## 编译TLG.fst
|
||
编译TLG需要依赖fst的环境，请参考文档安装fts相关环境（[点击此处](../onnxruntime/readme.md)）
|
||
```shell
|
||
|
||
# Compile the lexicon and token FSTs
|
||
fst/compile_dict_token.sh lm lm/tmp lm/lang
|
||
|
||
# Compile the language-model FST and the final decoding graph TLG.fst
|
||
fst/make_decode_graph.sh lm lm/lang || exit 1;
|
||
|
||
# Collect resource files required for decoding
|
||
fst/collect_resource_file.sh lm lm/resource
|
||
|
||
#编译后的模型资源位于 lm/resource
|
||
```
|
||
|