pickle — Python object serialization
Last Updated : 21 Nov, 2023
Python is a widely used general-purpose, high-level programming language. In this article, we will learn about pickling and unpickling in Python using the pickle module.
The Python Pickle Module
The pickle module is used for implementing binary protocols for serializing and de-serializing a Python object structure.
- Pickling: It is a process where a Python object hierarchy is converted into a byte stream.
- Unpickling: It is the inverse of the Pickling process where a byte stream is converted into an object hierarchy.
Module Interface
- dumps() – This function is called to serialize an object hierarchy.
- loads() – This function is called to de-serialize a data stream.
Constants provided by the pickle module
- pickle.HIGHEST_PROTOCOL
This is an integer value representing the highest protocol version available. This is considered as the protocol value which is passed to the functions dump(), dumps(). - pickle.DEFAULT_PROTOCOL
This is an integer value representing the default protocol used for pickling whose value may be less than the value of the highest protocol.
Functions provided by the pickle module
pickle.dump(obj, file, protocol = None, *, fix_imports = True)
This function is equivalent to Pickler(file, protocol).dump(obj). This is used to write a pickled representation of obj to the open file object file.
The optional protocol argument is an integer that tells the pickler to use the given protocol. The supported protocols are 0 to HIGHEST_PROTOCOL. If not specified, the default is DEFAULT_PROTOCOL. If a negative number is specified, HIGHEST_PROTOCOL is selected.
If fix_imports is true and protocol is less than 3, the pickle will try to map the new Python 3 names to the old module names used in Python 2, so that the pickle data stream is readable with Python
Example :The code defines a SimpleObject class and creates instances stored in a list. It uses the pickle module to serialize these instances and writes them to an io.StringIO object.
Python3
import pickle
import io
class SimpleObject(object):
def __init__(self, name):
self.name = name
l = list(name)
l.reverse()
self.name_backwards = ''.join(l)
return
data = []
data.append(SimpleObject('pickle'))
data.append(SimpleObject('cPickle'))
data.append(SimpleObject('last'))
out_s = io.StringIO()
for o in data:
print ('WRITING: %s (%s)' % (o.name, o.name_backwards))
pickle.dump(o, out_s)
out_s.flush()
|
Output :
WRITING: pickle (elkcip)
WRITING: cPickle (elkciPc)
WRITING: last (tsal)
pickle.dumps(obj, protocol = None, *, fix_imports = True)
This function returns the pickled representation of the object as a bytes object.
Example : This code uses the ‘pickle' module to serialize a list containing a dictionary with various data types (string, integer, and float). It converts the data into a binary data string and prints it. This binary string represents the serialized data and can be deserialized to reconstruct the original data structure using pickle.loads().
Python3
import pickle
data = [ { 'a':'A', 'b':2, 'c':3.0 } ]
data_string = pickle.dumps(data)
print ('PICKLE:', data_string )
|
Output :
PICKLE: (lp0
(dp1
S'a'
p2
S'A'
p3
sS'c'
p4
F3.0
sS'b'
p5
I2
sa.
pickle.load(file, *, fix_imports = True, encoding = “ASCII”, errors = “strict”)
This function is equivalent to Unpickler(file).load(). This function is used to read a pickled object representation from the open file object file and return the reconstituted object hierarchy specified.
Example : This code defines a ‘SimpleObject' class and creates instances stored in a list. It uses the pickle module to serialize these instances and writes them to an io.StringIO object. Then, it reads and deserializes the pickled data from in_s using pickle.load(). The code demonstrates the complete cycle of pickling and unpickling, successfully printing the information about the unpickled objects.
Python3
import pickle
import io
class SimpleObject(object):
def __init__(self, name):
self.name = name
l = list(name)
l.reverse()
self.name_backwards = ''.join(l)
return
data = []
data.append(SimpleObject('pickle'))
data.append(SimpleObject('cPickle'))
data.append(SimpleObject('last'))
out_s = io.StringIO()
for o in data:
print ('WRITING: %s (%s)' % (o.name, o.name_backwards))
pickle.dump(o, out_s)
out_s.flush()
in_s = io.StringIO(out_s.getvalue())
while True:
try:
o = pickle.load(in_s)
except EOFError:
break
else:
print ('READ: %s (%s)' % (o.name, o.name_backwards))
|
Output
WRITING: pickle (elkcip)
WRITING: cPickle (elkciPc)
WRITING: last (tsal)
READ: pickle (elkcip)
READ: cPickle (elkciPc)
READ: last (tsal)
pickle.loads(bytes_object, *, fix_imports = True, encoding = “ASCII”, errors = “strict”)
This function is used to read a pickled object representation from a bytes object and return the reconstituted object hierarchy specified.
Example : This code demonstrates pickling and unpickling data using the pickle module. It first serializes a list of dictionaries and then loads the pickled data back into another variable. It compares the original and unpickled data, showing that they are equal in content (data1 == data2) but not the same object in memory (data1 is not data2). This highlights the process of serialization and deserialization while preserving the data’s integrity.
Python3
import pickle
import pprint
data1 = [ { 'a':'A', 'b':2, 'c':3.0 } ]
print ('BEFORE:',)
pprint.pprint(data1)
data1_string = pickle.dumps(data1)
data2 = pickle.loads(data1_string)
print ('AFTER:',)
pprint.pprint(data2)
print ('SAME?:', (data1 is data2))
print ('EQUAL?:', (data1 == data2))
|
Output :
BEFORE:[{'a': 'A', 'b': 2, 'c': 3.0}]
AFTER:[{'a': 'A', 'b': 2, 'c': 3.0}]
SAME?: False
EQUAL?: True
Exceptions provided by the pickle module
exception pickle.PickleError
This exception inherits Exception. It is the base class for all other exceptions raised in pickling.
exception pickle.PicklingError
This exception inherits PickleError. This exception is raised when an unpicklable object is encountered by Pickler.
exception pickle.UnpicklingError
This exception inherits PickleError. This exception is raised when there is a problem like data corruption or a security violation while unpickling an object.
Classes exported by the pickle module:
class pickle.Pickler(file, protocol = None, *, fix_imports = True)
This class takes a binary file for writing a pickle data stream.
dump(obj) – This function is used to write a pickled representation of obj to the open file object given in the constructor.
persistent_id(obj) – If persistent_id() returns None, obj is pickled as usual. This does nothing by default and exists so that any subclass can override it.
Dispatch_table – A pickler object’s dispatch table is a mapping whose keys are classes and whose values are reduction functions.
By default, a pickler object will not have a dispatch_table attribute, and it will instead use the global dispatch table managed by the copyreg module.
Example : The below code creates an instance of pickle.Pickler with a private dispatch table that handles the SomeClass class especially.
Python3
f = io.BytesIO()
p = pickle.Pickler(f)
p.dispatch_table = copyreg.dispatch_table.copy()
p.dispatch_table[SomeClass] = reduce_SomeClass
|
class pickle.Unpickler(file, *, fix_imports = True, encoding = “ASCII”, errors = “strict”)
This class takes a binary file for reading a pickle data stream.
load() – This function is used to read a pickled object representation from the open file object file and return the reconstituted object hierarchy specified.
persistent_load(pid) – This raises an UnpicklingError by default.
find_class(module, name) – This function imports module if required and returns the object called name from it, where the module and name arguments are str objects.
What can be pickled and unpickled?
The following types can be pickled :
- None, True, and False
- integers, floating point numbers, complex numbers
- strings, bytes, bytearrays
- tuples, lists, sets, and dictionaries containing only picklable objects
- functions defined at the top level of a module (using def, not lambda)
- built-in functions defined at the top level of a module
- classes that are defined at the top level of a module
- instances of such classes whose __dict__ or the result of calling __getstate__() is picklable
Pickling Class Instances
This section explains the general mechanisms available to define, customize, and control how class instances are pickled and unpickled.
No additional code is needed to make instances pickable. By default, pickle will retrieve the class and the attributes of an instance via introspection.
Classes can alter the default behaviour by providing one or several special methods :
- object.__getnewargs_ex__()
This method dictates the values passed to the __new__() method upon unpickling. The method must return a pair (args, kwargs) where args is a tuple of positional arguments and kwargs a dictionary of named arguments for constructing the object. - object.__getnewargs__()
This method supports only positive arguments. It must return a tuple of arguments args which will be passed to the __new__() method upon unpickling. - object.__getstate__()
If this method is defined by classes, it is called and the returned object is pickled as the contents for the instance, instead of the contents of the instance’s dictionary. - object.__setstate__(state)
If this method is defined by classes, it is called with the unpickled state. The pickled state must be a dictionary and its items are assigned to the new instance’s dictionary. - object.__reduce__()
The __reduce__() method takes no argument and shall return either a string or preferably a tuple. - object.__reduce_ex__(protocol)
This method is similar to __reduce__ method. It takes a single integer argument. The main use for this method is to provide backwards-compatible reduce values for older Python releases.
Example : Handling Stateful Objects
This example shows how to modify pickling behavior for a class. The TextReader class opens a text file, and returns the line number and line contents each time its readline() method is called. If a TextReader instance is pickled, all attributes except the file object member are saved.When the instance is unpickled, the file is reopened, and reading resumes from the last location
This code defines a TextReader class for reading and numbering lines in a text file, with custom serialization and deserialization using the pickle module. It removes the file object during pickling and restores it during unpickling to maintain line numbering. The code demonstrates creating a TextReader, reading lines, and successfully preserving its state through pickling and unpickling, allowing for continued reading from the last line.
Python3
import pickle
class TextReader:
def __init__(self, filename):
self.filename = filename
self.file = open(filename)
self.lineno = 0
def readline(self):
self.lineno + 1
line = self.file.readline()
if not line:
return None
if line.endswith('\n'):
line = line[:-1]
return "%i: %s" % (self.lineno, line)
def __getstate__(self):
state = self.__dict__.copy()
del state['file']
return state
def __setstate__(self, state):
self.__dict__.update(state)
file = open(self.filename)
for _ in range(self.lineno):
file.readline()
self.file = file
reader = TextReader("Geeksforgeeks.txt")
print(reader.readline())
print(reader.readline())
new_reader = pickle.loads(pickle.dumps(reader))
print(new_reader.readline())
|
Output
0: hi geeks!, this is line 1.
0: This is line 2.
0: hi geeks!, this is line 1.
And here is a editor for PickleViewer:
Click here for the Github Repo
Click here for the download of version 0.7.5
Or Click here for the latest release
Similar Reads
marshal — Internal Python object serialization
Serializing a data means converting it into a string of bytes and later reconstructing it from such a string. If the data is composed entirely of fundamental Python objects, the fastest way to serialize the data is by using marshal module (For user defined classes, Pickle should be preferred). Marsh
2 min read
Serialize Python dictionary to XML
XML is a markup language that is designed to transport data. It was made while keeping it self descriptive in mind. Syntax of XML is similar to HTML other than the fact that the tags in XML aren't pre-defined. This allows for data to be stored between custom tags where the tag contains details about
5 min read
Collections.UserList in Python
Python Lists are array-like data structure but unlike it can be homogeneous. A single list may contain DataTypes like Integers, Strings, as well as Objects. List in Python are ordered and have a definite count. The elements in a list are indexed according to a definite sequence and the indexing of a
2 min read
Tips to reduce Python object size
We all know a very common drawback of Python when compared to programming languages such as C or C++. It is significantly slower and isn't quite suitable to perform memory-intensive tasks as Python objects consume a lot of memory. This can result in memory problems when dealing with certain tasks. W
4 min read
How to search a pickle file in Python?
Prerequisites: pickle file Python pickle module is used for serializing and de-serializing a Python object structure. Any object in Python can be pickled so that it can be saved on disk. What pickle does is that it “serializes†the object first before writing it to file. Pickling is a way to conver
3 min read
Collections.UserDict in Python
An unordered collection of data values that are used to store data values like a map is known as Dictionary in Python. Unlike other Data Types that hold only a single value as an element, Dictionary holds key:value pair. Key-value is provided in the dictionary to make it more optimized. Note: For mo
2 min read
Convert Set to String in Python
Converting a set to a string in Python means changing a group of unique items into a text format that can be easily read and used. Since sets do not have a fixed order, the output may look different each time. For example, a set {1, 2, 3} can be turned into the string "{1, 2, 3}" or into "{3, 1, 2}"
3 min read
Modules available for Serialization and Deserialization in Python
Python provides three different modules which allow us to serialize and deserialize objects : Â Marshal ModulePickle ModuleJSON Module 1. Marshal Module: It is the oldest module among these three. It is mainly used to read and write the compiled byte code of Python modules. Even we can use marshal t
3 min read
Convert Object to String in Python
Python provides built-in type conversion functions to easily transform one data type into another. This article explores the process of converting objects into strings which is a basic aspect of Python programming. Since every element in Python is an object, we can use the built-in str() and repr()
2 min read
Flask Serialization and Deserialization
Serialization and deserialization are fundamental concepts in web development, especially when working with REST APIs in Flask. Serialization refers to converting complex data types (such as Python objects) into a format that can be easily stored or transmitted, like JSON or XML. Deserialization, on
4 min read