Welcome to xpaw 0.12.0

Key Features

• A web scraping framework used to crawl web pages
• Data extraction tools used to extract structured data from web pages

Installation

安装xpaw需要Python 3.5.3或更高版本的环境支持。

可以使用如下命令安装xpaw:

$ pip install xpaw

如果安装过程中遇到lxml安装失败的情况，可参考 lxml installation 。

Requirements

• Python >= 3.5.3
• tornado
• lxml
• cssselect
• selenium

Table of Contents

Getting Started

• Make Requests
• Tutorial I: First Spider
  • Spider Overview
  • Start Requests
  • Data & Links
  • Storing Data
  • Running Spider

Basic References

• Spider
  • Spider API
  • Parsing Data to Callback Functions
  • Request Error Handling in Errback Functions
• Request & Response
  • Request API
  • Response API
• Selector
  • Selector Usage
  • Selector API
• Settings
  • Running
  • Logging
  • Downloading
  • Retry
  • Spider Behaviour
  • Components
• Errors
  • Built-in Errors

Advanced Usage

• Architecture Overview
  • Data Flow
  • Components
• Core API
  • Crawler API
  • Config API
• Specific Problems
  • Cron Job
  • Dynamic Webpages

All the Rest

• Changelog