Cooperative Web Agents by Combining Semantic Technologies with Reinforcement Learning

An increasing number of Web pages are enriched with semantic annotations using, for instance, RDFa and Schema.org in order to ease data interpretation for Web browsers and search engines. Lifted data representations might enable a new class of semantic agents for complex decision-making on the Web, reaching from filling out Web forms and booking flights to mastering any Web-based task. However, exclusively using semantic models has several drawbacks, such as extensive manual efforts for enabling semantic agents to solve Web tasks or limited capabilities to personalize task solutions for end users. While using semantics is a crucial component for guiding prospective semantic agents, we have to find a balance between modelling rich background knowledge and learning optimal agent behaviour through advanced Machine Learning (ML) methods. In this work, we propose a semantic agent framework for (i) modelling agent-related semantic annotations for Web tasks, (ii) using the latter to train statistical agents using Reinforcement Learning (RL) in an offline simulation as well as in real online use and (iii) feeding back the learned agent behaviour and its provenance information in terms of a semantic model that can be directly used by purely semantics-based agents. We evaluate our approach based on the MiniWob++ benchmark for automatically solving Web tasks. We show that our proposed semantic agent framework enables to (i) warm start agents with semantic background knowledge to faster learn task-dependent optimal behaviour in terms of the expected cumulative reward and (ii) directly reuse the learned behaviour by semantic-based agents that act by automatically derived Notation3 (N3) implication rules.