Exploration of Semantic Information of Previous Sentences for Automatic Speech Recognition

Abstract

In a recent study, semantic information of the current sentence helps improve automatic speech recognition (ASR) performance in noisy environments. This work aims to improve the ASR system in noisy conditions by exploiting semantic information from previously recognized sentences to re-evaluate the N-best hypotheses list. The semantic probability score, used to reevaluate the N-best hypotheses list, is obtained by two approaches. The first approach is to use a deep neural network (DNN) semantic model with bidirectional encoder representations from transformers (BERT), namely P-BERT, to compare sentence hypotheses pairwise and choose the hypothesis with better semantic consistency. In the second approach, we exploit Universal Sentence Encoder, a pre-trained sentence encoding model based on transformer architecture. We represent previously recognized sentence and current sentence hypotheses as high dimensional vectors and compute the semantic distance between sentence vectors of previously recognized sentence and current sentence hypotheses. We perform experiments on the publicly available TED-LIUM corpus with different noise levels. We evaluate these two approaches using different context lengths. The proposed methods show the improvement of the ASR system over the baseline method, which only uses semantic information from the current sentence. Our experiment results show that most of the best results are obtained from the P-BERT rescoring method.