Punctuation generation for speech transcripts using lexical, syntactic and prosodic features.

Modification on forked repository (by reducing training to one stage and addition of more word-level prosodic features). This version lets use any combination of word-aligned features.

Prosodically annotated files are in proscript format (https://github.com/alpoktem/proscript). For example data and extraction scripts see: https://github.com/alpoktem/ted_preprocess

English punctuation model was trained from a prosodically annotated TED corpus consisting of 1038 talks (155174 sentences). Link to dataset: http://hdl.handle.net/10230/33981

Punctuation generation accuracy with respect to human transcription:

These scores are obtained with a model trained with leveled pause duration and mean f0 features together with word and POS tags.

• Requirements:
  • Python 3.x
  • Numpy
  • Theano
  • yaml

Data directory (path $datadir) should look like the output folder (data/corpus) in https://github.com/alpoktem/ted_preprocess. Vocabularies and sampled training/testing/development sets are stored here.

Sample run explained here is provided in run.sh.

Training is done on sequenced data stored in train_samples under $datadir.

Dataset features to train with are given with the flag -f. Other training parameters are specified through the parameters.yaml file. To train with word, pause, POS and mean f0:

modelId="mod_word-pause-pos-mf0"

python main.py -m $modelId -f word -f pause_before -f pos -f f0_mean -p parameters.yaml

Testing is done on proscript data using punctuator.py. Either single <input-file> or <input-directory> is given as input using -i or -d respectively. Even if there's punctuation information on this data, it is ignored. Predictions for each file in the $test_samples directory are put into $out_preditions directory. Input files should contain the parameters that the model was trained with.

model_name="Model_single-stage_""$modelId""_h100_lr0.05.pcl"

python punctuator.py -m Model_single-stage_mod_word-pause-pos-mf0_h100_lr0.05.pcl -d $test_samples -o $out_predictions

Predictions are compared with groundtruth data using error_calculator.py. It either takes two files to compare or two directories containing groundtruth/prediction files. Use -r for reducing punctuation marks.

python error_calculator.py -g $groundtruthData -p $out_predictions -r

More details can be found in the publication: https://link.springer.com/chapter/10.1007/978-3-319-68456-7_11

This work can be cited as:

@inproceedings{punkProse,
	author = {Alp Oktem and Mireia Farrus and Leo Wanner},
	title = {Attentional Parallel RNNs for Generating Punctuation in Transcribed Speech},
	booktitle = {5th International Conference on Statistical Language and Speech Processing SLSP 2017},
	year = {2017},
	address = {Le Mans, France}
}