Scrapy - Item Pipeline

Last Updated : 08 Sep, 2021

Scrapy is a web scraping library that is used to scrape, parse and collect web data. For all these functions we are having a pipelines.py file which is used to handle scraped data through various components (known as class) which are executed sequentially.

In this article, we will be learning through the methods defined for this pipeline's file and will show different examples of it.

Setting Up Project

Let's, first, create a scrapy project. For that make sure that Python and PIP are installed in the system. Then run the given commands below one by one to create a scrapy project similar to the one which we will be using in this article.

Step 1: Let's first create a virtual environment in a folder named GFGScrapy and activate that virtual environment there.

# To create a folder named GFGScrapy mkdir GFGScrapy cd GFGScrapy  # making virtual env there. virtualenv cd scripts  # activating it activate cd..

Hence, after running all these commands, we will get the output as shown:

Step 2: Now it's time to create a scrapy project. For that Make sure that scrapy is installed in the system or not. If not installed, install it using the given command below.

pip install scrapy

Now to create a scrapy project use the, given command below and also create a spider.

# projEct name is scrapytutorial scrapy startproject scrapytutorial cd scrapytutorial scrapy genspider spider_to_crawl https://quotes.toscrape.com

Then the output of the project directory looks like the one given in the image. (Please refer this if you want to know more about a scrapy project and get familiar with it).

Let's have a look at our spider_to_crawl.py file present inside our spiders folder. This is the file where we are writing the URL where our spider has to crawl and also a method named parse() which is used to describe what should be done with the data scraped by the spider.

This file is automatically generated by "scrapy genspider" command used above. The file is named after the spider's name. Below given is the default file generated.

Item pipeline is a pipeline method that is written inside pipelines.py file and is used to perform the below-given operations on the scraped data sequentially. The various operations we can perform on the scraped items are listed below:

Parse the scraped files or data.
Store the scraped data in databases.
Validating and checking the data obtained.
Converting files from one format to another. eg to JSON.

We will be performing some of these operations in the examples below.

Operations are performed sequentially since we are using settings.py file to describe the order in which the operations should be done. i.e. we can mention which operation to be performed first and which to be performed next. This is usually done when we are performing several operations on the items.

Let's first see the inner structure of a default pipeline file. Below is the default class mentioned in that file.

For performing different operations on items we have to declare a separated component( classes in the file) which consists of various methods, used for performing operations. The pipelines file in default has a class named after the project name. We can also create our own classes to write what operations they have to perform. If any pipelines file consists of more than one class then we should mention their execution order explicitly. The structure of components are defined below:

Each component (class) must have one default function named process_item(). This is the default method which is always called inside the class or component of the pipelines file.

Syntax: process_item( self, item, spider )

Parameters:

self : This is reference to the self object calling the method.
item : These are the items list scraped by the spider
spider : mentions the spider used to scrape.

The return type of this method is the modified or unmodified item object or an error will be raised if any fault is found in item.

This method is also used to call other method in this class which can be used to modify or store data.

Additional methods: These methods are used along with the above-mentionedself-object method to gain extra control over the items.

Method

Description

open_spider(self,spider)

Spider object which is opened and a reference to self object are the parameters. ( These are default cases of python language).

Returns nothing except the fact that it is used to either make some changes or open a file or close a file.

close_spider(self,spider)

Spider object which is closed and a reference to self-object.

It also either is used to modify the file or open or close it.

from_crawler(cls, crawler)

Crawler object that is specified.

This method is used to give pipeline accessibility to all the core components of the scrapy settings so that pipelines can enhance their functionality,

Apart from all these methods, we can also create our own method to perform more operations like if we want to store some data then we can have the component that initializes the database and create tables in it, Another component may be there which will add the data to the database.

Before we move ahead and refer to examples, an important point to note is that we should have to register all the components (classes) of the pipelines.py file in the settings.py of the folder structure. This is done to maintain an ordering of the components to be executed and hence produce accurate results.

Creating Items to be passed over files.

One more thing to note is that we will require a description of what our item will contain in items.py file. Hence our items.py file contains the below-given code:

Python3

# Define here the models for your scraped items import scrapy  class ScrapytutorialItem(scrapy.Item):        # define the fields for your item here like:     # name = scrapy.Field()     Quote = scrapy.Field()     #only one field that it of Quote.

We will require this file to be imported in our spider_to_crawl.py file. Hence in this way we can create items to be passed to pipeline. We will mainly be using the Wisdom quotes web page where we can get several quotes based on their authors and respective tags and then we will modify and use items pipelines on the scraped data throughout the example.

Example 1: Converting scraped data to JSON format

To convert the data in JSON format we will be using JSON library of python along with its dumps() method.

The idea is that we will get the scraped data in pipelines.py file and then we will open a file named result.json (if not already present then it will be created automatically) and write all the JSON data in it.

open_spider() will be called to open the file (result.json) when spider starts crawling.
close_spider() will be called to close the file when spider is closed and scraping is over.
process_item() will always be called (since it is default) and will be mainly responsible for converting the data to JSON format and print the data to the file. We will be using the concept of python web frameworks, i.e. how they convert backend retrieved data to JSON and other formats.

Hence the code in our pipelines.py looks like this:

Python3

from itemadapter import ItemAdapter import json   class ScrapytutorialPipeline:     def process_item(self, item, spider):                          # calling dumps to create json data.         line = json.dumps(dict(item)) + "\n"            self.file.write(line)                         return item      def open_spider(self, spider):         self.file = open('result.json', 'w')      def close_spider(self, spider):         self.file.close()

Our spider_to_crawl.py:

Python3

import scrapy from ..items import ScrapytutorialItem      class SpiderToCrawlSpider(scrapy.Spider):     name = 'spider_to_crawl'      start_urls = ['https://quotes.toscrape.com/']      def parse(self, response):                  # creating items dictionary          items = ScrapytutorialItem()             Quotes_all = response.xpath('//div/div/div/span[1]')          # These paths are based on the selectors                  for quote in Quotes_all:    #extracting data             items['Quote'] = quote.css('p::text').extract()             yield items

Output:

Explanation:

After using the command "scrapy crawl spider_to_crawl", The below-given steps are going to take place.

The spider is crawled due to which result.json file is created. Now the spider scrapes the web page and collects the data in Quotes_all Variable. Then we will send each data from this variable one by one to our pipelines.py file.
We receive item variables from spider in pipelines.py file which is, then converted to JSON using the dumps() method, and then the output is written in the opened file.

This is the JSON file which got created:

Example 2: Pipeline to upload data to database in SQLite3

Now we are going to present an items pipeline that will scrape the content of the web and store it on the database table defined by us. For simplicity, we will be using the SQLite3 database.

So we will use the idea of how to implement SQLite3 in python to create a pipeline that will receive data from spider scraping and will insert that data to the table in the database created.

spider_to_crawl.py:

Python3

import scrapy from ..items import ScrapytutorialItem      class SpiderToCrawlSpider(scrapy.Spider):     name = 'spider_to_crawl'      start_urls = ['https://quotes.toscrape.com/']      def parse(self, response):                  # creating items dictionary          items = ScrapytutorialItem()             Quotes_all = response.xpath('//div/div/div/span[1]')                   # These paths are based on the selectors                  # extracting data         for quote in Quotes_all:                 items['Quote'] = quote.css('::text').extract()             yield items

We are mentioning the pipeline methods below which are to be written in the pipelines.py File so that the database will be created:

pipelines.py file

Python3

from itemadapter import ItemAdapter import sqlite3  class ScrapytutorialPipeline(object):  # init method to initialize the database # and create connection and tables     def __init__(self):         self.create_conn()         self.create_table()          # create connection method to create     # database or use database to store scraped data     def create_conn(self):         self.conn = sqlite3.connect("mydata.db")         self.curr = self.conn.cursor()      # Create table method     # using SQL commands to create table     def create_table(self):         self.curr.execute("""DROP TABLE IF EXISTS firsttable""")         self.curr.execute("""create table firsttable(                         Quote text                         )""")          # store items to databases.     def process_item(self, item, spider):         self.putitemsintable(item)         return item          def putitemsintable(self,item):         self.curr.execute("""insert into firsttable values (?)""",(             item['Quote'][0],  # extracting item.         ))         self.conn.commit()

Output:

Explanation:

After using the command "scrapy crawl spider_to_crawl", the below-given steps are going to take place

In spider.py we had mentioned the code that our spider should go to that site and extract all data mentioned in the URL format and then will create items list of it and pass that list to the pipelines.py file for further processing.
We are also creating an items object to contain data to be passed and item it at items.py file in the directory.
Then when the spider crawls it collects data in items object and transfers it to the pipelines and what happens next is already clear from the above code with hints in comments. pipelines.py file creates a database and stores all the incoming items.
Here the init() method is called which is called a default method always in any python file. It then calls all other methods which are used to create a table and initialize the database.
Then process_item() method is called which is used to call a method named putitemintable() where data is added to the database. Then, after executing this method the reference is returned to the spider to pass another item to be operated.

Scrapy - Selectors

harshkumarchoudhary144

Improve

Article Tags :

Practice Tags :

python

Scrapy - Item Pipeline

Setting Up Project

Creating Items to be passed over files.

Example 1: Converting scraped data to JSON format

Example 2: Pipeline to upload data to database in SQLite3

Similar Reads

Getting Started With Scrapy

Scrapy Basics

Data Collection and Management

Data Extraction and Export

Appliaction And Projects