Abstract
Forums (or discussion boards) represent a huge information collection structured under different boards, threads and posts. The actual information entity of a forum is a post, which has the information about authors, date and time of post, actual content etc. This information is significant for a number of applications like gathering market intelligence, analyzing customer perceptions etc. However automatically extracting this information from a forum is an extremely challenging task. There are several customized parsers designed for extracting information from a particular forum platform with a specific template (e.g. SMF or phpBB), however the problem with this approach is that these parsers are dependent upon the forum platform and the template used, which makes it unrealistic to use in practical situations. Hence, in this paper we propose a semi-automatic rule based solution for extracting forum post information and inserting the extracted information to a database for the purpose of analysis. The key challenge with this solution is identifying extraction rules, which are normally forum platform and forum template specific. As a result we analyzed 72 forums to derive these rules and test the performance of the algorithm. The results indicate that we were able to extract all the required information from SMF and phpBB forum platforms, which represent the majority of forums on the web.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Scardamalia, M.: In: Education and technology: An encyclopedia, pp. 183–192 (2004)
PHPBB3, http://www.tesl-ej.org/wordpress/past-issues/volume12/ej47/ej47m2/
Welcome to the new Audiworld!, http://forums.audiworld.com/vbguide.pdf
Potdar, V., Hayati, P.: Spammer and Hacker, Two Old Friends. In: 3rd IEEE International Conference on Digital Ecosystems and Technologies (DEST 2009), Istanbul, Turkey (2009)
Weld, D.S., Wu, F., Adar, E., Amershi, S., Fogarty, J., Hoffmann, R., Patel, K., Skinner, M.: Intelligence in Wikipedia. In: 23rd AAAI Conference on Artificial Intelligence (2008)
McCallum, A.: Information Extraction (2005)
Iria, J., Ciravegna, F.: Relation Extraction for Mining the Semantic Web. In: Dagstuhl Seminar on Machine Learning for the Semantic Web, Dagstuhl (2005)
Kristjansson, T., Culotta, A., Viola, P., McCallum, A.: Interactive Information Extraction with Constrained Conditional Random Fields. In: 19th National Conference on Artifical Intelligence, California, pp. 412–418 (2004)
Potdar, V., Hayati, P.: Toward Spam 2.0: An Evaluation of Web 2.0 Anti-Spam Methods. In: 7th IEEE International Conference on Industrial Informatics (INDIN 2009), Cardiff, Wales (2009)
Hammer, J., McHugh, J., Garcia-Molina: Semi structured data: the TSIMMIS experience. In: Proceedings of the 1st East-European Symposium on Advances in Databases and Information Systems,
Sahuguet, A., Azavant, F.: Building intelligent Web applications using lightweight wrappers. Data & Knowledge Engineering 36(3), 283–316 (2001)
Kushmerick, N., Weld, D.S., Doorenbos, R.: Wrapper induction for information extraction. In: Proceedings of the International Joint Conference on Artificial Intelligence, IJCAI 1997 (1997)
Chang, C.-H., Lui, S.-C.: IEPAD: Information Extraction Based on Pattern Discovery. In: 10th international conference on World Wide Web, Hong Kong, pp. 681–688 (2001)
Chang, C.-H., Kuo, S.-C.: OLERA: On-Line Extraction Rule Analysis for Semi-structured Documents. In: 22nd IASTED International Multi-Conference on Applied Informatics (2004)
Chang, C.-H., Hsu, C.-N., Lui, S.C.: Automatic information extraction from semi-structured Web pages by pattern discovery, vol. 35, pp. 129–147. Elsevier Science Publishers B. V, The Netherlands (2003)
Liu, B.: Web data mining: exploring hyperlinks, contents, and usage data, pp. 323–344. Springer, Heidelberg (2006)
Zhang, Q., Yang, S., Huang, X., Wu, L.: Template-independent Wrapper for Web Forums. In: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pp. 794–795. ACM, New York (2009)
Cai, R., Yang, J., Lai, W., Wang, Y., Zhang, L.: iRobot: An Intelligent Crawler for Web Forums. In: 17th international conference on World Wide Web, China, pp. 447–456 (2008)
Crescenzi, M., Mecca, V.: Grammars have exceptions. Information Systems 23(8), 539–565 (1998)
Hayati, P., Potdar, V.: Evaluation of Spam Detection and Prevention Frameworks for Email and Image Spam - A State of Art. In: Proceedings of the 10th International Conference on Information Integration and Web-based Applications & Services, Linz, Austria (2008)
Hayati, P., Potdar, V., Talevski, A., Chai, K.: Web Spambot Detection Based on Navigation Behavior. In: 24th IEEE AINA Conference, Perth, Australia (April 2010) (accepted)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Sarencheh, S., Potdar, V., Yeganeh, E.A., Firoozeh, N. (2010). Semi-automatic Information Extraction from Discussion Boards with Applications for Anti-Spam Technology. In: Taniar, D., Gervasi, O., Murgante, B., Pardede, E., Apduhan, B.O. (eds) Computational Science and Its Applications – ICCSA 2010. ICCSA 2010. Lecture Notes in Computer Science, vol 6017. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-12165-4_30
Download citation
DOI: https://doi.org/10.1007/978-3-642-12165-4_30
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-12164-7
Online ISBN: 978-3-642-12165-4
eBook Packages: Computer ScienceComputer Science (R0)