Python Django & Scrapy solution...

Here is the code of a simple spider that would use Crawls for Scraping AJAX Pages. #------------------------------------------------------------------------------- # Name: module1 # Purpose: # # Author: Ranvijay.Sachan # # Created: 31/10/2014 # Copyright: (c) Ranvijay.Sachan 2014 # Licence: #------------------------------------------------------------------------------- from scrapy.http import Request from scrapy.spider import BaseSpider import urllib import json from scraping.DoveItem import DoveItem class DoveAjaxspider(BaseSpider): name = "dove" allowed_domains = ["dove.in"] start_urls = ["http://www.mydove.com.au/en/"] def parse(self, response): # This receives the response from the start url. But we don't do anything with it. allProductType = ['Bar/Body Wash','Lotion','Deodorant','Face','Hair','Men+Care'] url = 'http://www.m...

[Python] Get links of product to every page of a retailor Here is the code of a simple spider that would use Crawling scraped links & next pagination. You have two options to solve your problem. The general one is to use yield to generate new requests instead of return. That way you can issue more than one new request from a single callback. Check the second example at http://doc.scrapy.org/en/latest/topics/spiders.html#basespider-example. #------------------------------------------------------------------------------- # Name: module1 # Purpose: # # Author: Ranvijay.Sachan # # Created: 31/10/2014 # Copyright: (c) Ranvijay.Sachan 2014 # Licence: #------------------------------------------------------------------------------- from scrapy.spider import BaseSpider from scrapy.selector import HtmlXPathSelector from scrapy.http.request import Request from scraping.articles import ArticleItem from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor fr...

We often have to write spiders that need to login to sites, in order to scrape data from them. Our customers provide us with the site, username and password, and we do the rest. The classic way to approach this problem is: 1.      launch a browser, go to site and search for the login page 2.      inspect the source code of the page to find out:                       I.         which one is the login form (a page can have many forms, but usually one of them is the login form)                      II.         which are the field names used for username and password (these could vary a lot)             ...