Article

Free access

Dynamic nonlocal language modeling via hierarchical topic-based adaptation

Authors:

David YarowskyAuthors Info & Claims

ACL '99: Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics

Pages 167 - 174

https://doi.org/10.3115/1034678.1034711

Published: 20 June 1999 Publication History

Abstract

This paper presents a novel method of generating and applying hierarchical, dynamic topic-based language models. It proposes and evaluates new cluster generation, hierarchical smoothing and adaptive topic-probability estimation techniques. These combined models help capture long-distance lexical dependencies. Experiments on the Broadcast News corpus show significant improvement in perplexity (10.5% overall and 33.5% on target vocabulary).

References

[1]

P. Brown, J. Cocke, S. Della Pietra, V. Della Pietra, F. Jelinek, J. Lafferty, R. Mercer, and P. Roossin'. 1990. A statistical approach to machine translation. Computational Linguistics, 16(2).]]

Digital Library

[2]

Ciprian Chelba and Fred Jelinek. 1998. Exploiting syntactic structure for language modeling. In Proceedings COLING-ACL, volume 1, pages 225--231, August.]]

Digital Library

[3]

Stanley F. Chen and Joshua Goodman. 1998. An empirical study of smoothing techinques for language modeling. Technical Report TR-10-98, Center for Research in Computing Technology, Harvard University, Cambridge, Massachusettes, August.]]

[4]

Richard O. Duda and Peter E. Hart. 1973. Patern Classification and Scene Analysis. John Wiley & Sons.]]

[5]

Radu Florian. 1998. Exploiting nonlocal word relationships in language models. Technical report, Computer Science Department, Johns Hopkins University. http://nlp.cs.jhu.edu/~rflorian/papers/topiclm-tech-rep.ps.]]

[6]

J. Good. 1953. The population of species and the estimation of population parameters. Biometrica, 40, parts 3, 4: 237--264.]]

[7]

Rukmini Iyer and Mari Ostendorf. 1996. Modeling long distance dependence in language: Topic mixtures vs. dynamic cache models. In Proceedings of the International Conferrence on Spoken Language Processing, volume 1, pages 236--239.]]

[8]

Rukmini Iyer, Mari Ostendorf, and J. Robin Rohlicek. 1994. Language modeling with sentence-level mixtures. In Proceedings ARPA Workshop on Human Language Technology, pages 82--87.]]

Digital Library

[9]

Slava Katz. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. In IEEE Transactions on Acoustics, Speech, and Signal Processing, 1987, volume ASSP-35 no 3, pages 400--401, March 1987.]]

[10]

Sanjeev Khudanpur and Jun Wu. 1999. A maximum entropy language model integrating n-gram and topic dependencies for conversational speech recognition. In Proceedings on ICASSP.]]

Digital Library

[11]

R. Kuhn and R. de Mori. 1992. A cache based natural language model for speech recognition. IEEE Transaction PAMI, 13: 570--583.]]

Digital Library

[12]

R. Lau, Ronald Rosenfeld, and Salim Roukos. 1993. Trigger based language models: a maximum entropy approach. In Proceedings ICASSP, pages 45--48, April.]]

[13]

S. Lowe. 1995. An attempt at improving recognition accuracy on switchboard by using topic identification. In 1995 Johns Hopkins Speech Workshop, Language Modeling Group, Final Report.]]

[14]

Lidia Mangu. 1997. Hierarchical topic-sensitive language models for automatic speech recognition. Technical report, Computer Science Department, Johns Hopkins University. http://nlp.cs.jhu.edu/~lidia/papers/tech-rep1.ps.]]

[15]

Ronald Rosenfeld. 1994. A hybrid approach to adaptive statistical language modeling. In Proceedings ARPA Workshop on Human Language Technology, pages 76--87.]]

Digital Library

[16]

G. Salton and M. McGill. 1983. An Introduction to Modern Information Retrieval. New York, McGram-Hill.]]

Digital Library

[17]

Kristie Seymore and Ronald Rosenfeld. 1997. Using stopy topics for language model adaptation. In EuroSpeech97, volume 4, pages 1987--1990.]]

[18]

Kristie Seymore, Stanley Chen, and Ronald Rosenfeld. 1998. Nonlinear interpolation of topic models for language model adaptation. In Proceedings of ICSLP98.]]

[19]

J. H. Wright, G. J. F. Jones, and H. Lloyd-Thomas. 1993. A consolidated language model for speech recognition. In Proceedings EuroSpeech, volume 2, pages 977--980.]]

Cited By

Trnka KArisoy EMaier WInoue K(2008)Adaptive language modeling for word predictionProceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Student Research Workshop10.5555/1564154.1564167(61-66)Online publication date: 16-Jun-2008
https://dl.acm.org/doi/10.5555/1564154.1564167
Foster GKuhn RCallison-Burch CKoehn PMonz CFordyce C(2007)Mixture-model adaptation for SMTProceedings of the Second Workshop on Statistical Machine Translation10.5555/1626355.1626372(128-135)Online publication date: 23-Jun-2007
https://dl.acm.org/doi/10.5555/1626355.1626372
Trnka KMcCoy KPontelli ETrewin S(2007)Corpus studies in word predictionProceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility10.1145/1296843.1296877(195-202)Online publication date: 15-Oct-2007
https://dl.acm.org/doi/10.1145/1296843.1296877
Show More Cited By

Dynamic nonlocal language modeling via hierarchical topic-based adaptation
1. Computing methodologies
  1. Artificial intelligence
2. Hardware
  1. Power and energy
    1. Power estimation and optimization

Recommendations

Hierarchical topic integration through semi-supervised hierarchical topic modeling
CIKM '12: Proceedings of the 21st ACM international conference on Information and knowledge management

Lots of document collections are well organized in hierarchical structure, and such structure can help users browse and understand these collections. Meanwhile, there are a large number of plain document collections loosely organized, and it is difficult ...
Jointly Discovering Fine-grained and Coarse-grained Sentiments via Topic Modeling
MM '14: Proceedings of the 22nd ACM international conference on Multimedia

The ever-increasing user-generated contents in social media and other web services make it highly desirable to discover opinions of users on all kinds of topics. Motivated by the assumption that individual word and paragraph in documents will deliver ...
Lifelong Hierarchical Topic Modeling via Non-negative Matrix Factorization
Web and Big Data
Abstract
Hierarchical topic modeling has been widely used in mining the latent topic hierarchy of documents. However, most of such models are limited to a one-shot scenario since they do not use the identified topic information to guide the subsequent ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image DL Hosted proceedings

ACL '99: Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics

June 1999

642 pages

ISBN:1558606093

Program Chairs:
Robert Dale,
Ken Church

Publisher

Association for Computational Linguistics

United States

Publication History

Published: 20 June 1999

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 85 of 443 submissions, 19%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
412
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)3

Reflects downloads up to 13 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Trnka KArisoy EMaier WInoue K(2008)Adaptive language modeling for word predictionProceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Student Research Workshop10.5555/1564154.1564167(61-66)Online publication date: 16-Jun-2008
https://dl.acm.org/doi/10.5555/1564154.1564167
Foster GKuhn RCallison-Burch CKoehn PMonz CFordyce C(2007)Mixture-model adaptation for SMTProceedings of the Second Workshop on Statistical Machine Translation10.5555/1626355.1626372(128-135)Online publication date: 23-Jun-2007
https://dl.acm.org/doi/10.5555/1626355.1626372
Trnka KMcCoy KPontelli ETrewin S(2007)Corpus studies in word predictionProceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility10.1145/1296843.1296877(195-202)Online publication date: 15-Oct-2007
https://dl.acm.org/doi/10.1145/1296843.1296877
Trnka KYarrington DMcCoy KPennington CEdmonds ERiecken DParis CSidner C(2006)Topic modeling in fringe word prediction for AACProceedings of the 11th international conference on Intelligent user interfaces10.1145/1111449.1111509(276-278)Online publication date: 29-Jan-2006
https://dl.acm.org/doi/10.1145/1111449.1111509
Kaji NKurohashi S(2005)Lexical choice via topic adaptation for paraphrasing written language to spoken languageProceedings of the Second international joint conference on Natural Language Processing10.1007/11562214_85(981-992)Online publication date: 11-Oct-2005
https://dl.acm.org/doi/10.1007/11562214_85
Church KKay M(2000)Empirical estimates of adaptationProceedings of the 18th conference on Computational linguistics - Volume 110.3115/990820.990847(180-186)Online publication date: 31-Jul-2000
https://dl.acm.org/doi/10.3115/990820.990847
Kurohashi SOri MSchiitze HSu K(2000)Nonlocal language modeling based on context co-occurrence vectorsProceedings of the 2000 Joint SIGDAT conference on Empirical methods in natural language processing and very large corpora: held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics - Volume 1310.3115/1117794.1117804(80-86)Online publication date: 7-Oct-2000
https://dl.acm.org/doi/10.3115/1117794.1117804

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten