Python Django & Scrapy solution...

Here is the code of a simple spider that would use Crawls for Scraping AJAX Pages. #------------------------------------------------------------------------------- # Name: module1 # Purpose: # # Author: Ranvijay.Sachan # # Created: 31/10/2014 # Copyright: (c) Ranvijay.Sachan 2014 # Licence: <your licence> #------------------------------------------------------------------------------- from scrapy.http import Request from scrapy.spider import BaseSpider import urllib import json from scraping.DoveItem import DoveItem class DoveAjaxspider(BaseSpider): name = "dove" allowed_domains = ["dove.in"] start_urls = ["http://www.mydove.com.au/en/"] def parse(self, response): # This receives the response from the start url. But we don't do anything with it. allProductType = ['Bar/Body Wash','Lotion','Deodorant','Face','Hair','Men+Care'] url

[Python] Get links of product to every page of a retailor Here is the code of a simple spider that would use Crawling scraped links & next pagination. You have two options to solve your problem. The general one is to use yield to generate new requests instead of return. That way you can issue more than one new request from a single callback. Check the second example at http://doc.scrapy.org/en/latest/topics/spiders.html#basespider-example. #------------------------------------------------------------------------------- # Name: module1 # Purpose: # # Author: Ranvijay.Sachan # # Created: 31/10/2014 # Copyright: (c) Ranvijay.Sachan 2014 # Licence: <your licence> #------------------------------------------------------------------------------- from scrapy.spider import BaseSpider from scrapy.selector import HtmlXPathSelector from scrapy.http.request import Request from scraping.articles import ArticleItem from scrapy.contrib.linkextractors.sgml import

We often have to write spiders that need to login to sites, in order to scrape data from them. Our customers provide us with the site, username and password, and we do the rest. The classic way to approach this problem is: 1.      launch a browser, go to site and search for the login page 2.      inspect the source code of the page to find out:                       I.         which one is the login form (a page can have many forms, but usually one of them is the login form)                      II.         which are the field names used for username and password (these could vary a lot)                     III.         if there are other fields that must be submitted (like an authentication token) 3.      write the Scrapy spider to replicate the form submission using   FormRequest Being fans of automation, we figured we could write some code to automate point 2 (which is actually the most time-consuming) and the result is   login form, a library to automatically fi