| 摘要: |
| 随着Web2.0时代的到来,越来越多的网站采用了动态脚本的方式与用户进行交互.页面的转换不再仅仅通过点击“”标签进行,URL也不再是页面的唯一标识.传统网络爬虫无法应对含动态脚本的网页,如Google等搜索引擎即对这些网页采取回避的态度.对这些网页的抓取方法的研究仍处在起步阶段,提出了一种高效的动态脚本网站有效页面的获取方法,首先通过训练获得哪些页面元素触发的哪些事件将引向我们所需的页面,并总结出这些页面元素的XPath特征及触发的事件类型.在以后的抓取中,只触发这些页面元素上的特定事件,从而提升抓取效率.此外,通过实验证明了我们方法的效率和性能. |
| 关键词: 动态脚本 AJAX 页面相似性 XPath 网络爬虫 |
| DOI： |
| 分类号: |
| 基金项目:Supported by the National Natural Science Foundation of China under Grant No.60873062(国家自然科学基金); the National High-Tech Research and Development Plan of China under Grant Nos.2009AA01Z150, 2007AA01Z191, 2006AA01Z230 (国家高技术研究发展计划(863)); the Peking Universi |
|
| An Efficient Valid Page Crawling Approach for Websites with Dynamic Scripts |
|
XIA Bing,GAO Jun,WANG Teng-Jiao,YANG Dong-Qing
|
| Abstract: |
| In times of Web 2.0, more and more websites adopt dynamic scripts for user interaction, and the switches between pages are no longer all based on the “” tags and the URL is no longer the unique identification of a Web page. Traditional Web crawlers can’t deal with Web pages containing dynamic scripts, as a result, search engines, such as Google, give up these Web pages. The research on crawling website with dynamic scripts is still in the early stage. This paper proposes an efficient valid page crawling approach for websites with dynamic scripts. Firstly, by training the paper can get the events and the Web elements that triggered the events, which would lead the people to desired Web pages. Then, the paper generates the XPath patterns of these elements and record the events the people need to trigger. During crawling, the paper only considers these event and element combinations for accelerating the crawling. Additionally, the paper demonstrates the efficiency and the effectiveness of the approach by extensive experimental evaluation. |
| Key words: dynamic scripts AJAX page similarity XPath Web crawler |
