Skip to content

How To Read .Data Files In Python?

How to read large text files in Python?

Last Updated : 13 Sep, 2022

In this article, we will try to understand how to read a large text file using the fastest way, with less memory usage using Python.

To read large text files in Python, we can use the file object as an iterator to iterate over the file and perform the required task. Since the iterator just iterates over the entire file and does not require any additional data structure for data storage, the memory consumed is less comparatively. Also, the iterator does not perform expensive operations like appending hence it is time-efficient as well. Files are iterable in Python hence it is advisable to use iterators.

Problem with readline() method to read large text files

In Python, files are read by using the readlines() method. The readlines() method returns a list where each item of the list is a complete sentence in the file. This method is useful when the file size is small. Since readlines() method appends each line to the list and then returns the entire list it will be time-consuming if the file size is extremely large say in GB. Also, the list will consume a large chunk of the memory which can cause memory leakage if sufficient memory is unavailable.

Read large text files in Python using iterate

In this method, we will import fileinput module. The input() method of fileinput module can be used to read large files. This method takes a list of filenames and if no parameter is passed it accepts input from the stdin, and returns an iterator that returns individual lines from the text file being scanned.

Note: We will also use it to calculate the time taken to read the file using Python time.

Python3

   # import module
import fileinput
import time
 
#time at the start of program is noted
start = time.time()
 
#keeps a track of number of lines in the file
count = 0
for lines in fileinput.input(['sample.txt']):
    print(lines)
    count = count + 1
     
#time at the end of program execution is noted
end = time.time()
 
#total time taken to print the file
print("Execution time in seconds: ",(end - start))
print("No. of lines printed: ",count)
 
  

Output:

How to read large text files in Python?

The fastest way to read a large text file using the iterator of a file object

Here, the only difference is that we will use the iterator of a file object. The open() function wraps the entire file into a file object. After that, we use an iterator to get the lines in the file object. We open the file in a ‘with’ block as it automatically closes the file as soon as the entire block executes.

Python3

   import time
 
start = time.time()
count = 0
with open("sample.txt") as file:
    for line in file:
       print(line)
       count = count + 1
end =  time.time()
print("Execution time in seconds: ",(end-start))
print("No of lines printed: ",count)
 
  

Output:

The time required in the second approach is comparatively less than the first method.

How to read large text files in Python?

How To Read .Data Files In Python?

S

Shreyasi_Chakraborty

Improve

Article Tags :

Practice Tags :

python

Similar Reads

How to Read Text File Into List in Python?

In this article, we are going to see how to read text files into lists in Python. File for demonstration: Example 1: Converting a text file into a list by splitting the text on the occurrence of '.'. We open the file in reading mode, then read all the text using the read() and store it into a variab

How To Read .Data Files In Python?

Unlocking the secrets of reading .data files in Python involves navigating through diverse structures. In this article, we will unravel the mysteries of reading .data files in Python through four distinct approaches. Understanding the structure of .data files is essential, as their format may vary w

How to read multiple text files from folder in Python?

Prerequisite: File Handlingos Python is a strong language which is extremely capable even when it comes to file handling. In this article, we will learn how to read multiple text files from a folder using python. Approach: Import modulesAdd path of the folderChange directoryGet the list of a file fr

How to open two files together in Python?

Prerequisites: Reading and Writing text files in Python Python provides the ability to open as well as work with multiple files at the same time. Different files can be opened in different modes, to simulate simultaneous writing or reading from these files. An arbitrary number of files can be opened

How to Read from a File in Python

Reading from a file in Python means accessing and retrieving the contents of a file, whether it be text, binary data or a specific data format like CSV or JSON. Python provides built-in functions and methods for reading a file in python efficiently. Example File: geeks.txt Hello World Hello Geeksfor

How To Download Large File In Python With Requests

Downloading large files in Python can sometimes be a tricky task, especially when efficiency and reliability are crucial. Python's requests library provides a convenient way to handle HTTP requests, including downloading files. In this article, we will explore how to download large files in Python w

How to read specific lines from a File in Python?

Text files are composed of plain text content. Text files are also known as flat files or plain files. Python provides easy support to read and access the content within the file. Text files are first opened and then the content is accessed from it in the order of lines. By default, the line numbers

How to search a pickle file in Python?

Prerequisites: pickle fileÂ Python pickle module is used for serializing and de-serializing a Python object structure. Any object in Python can be pickled so that it can be saved on disk. What pickle does is that it â€œserializesâ€ the object first before writing it to file. Pickling is a way to conver

How to compare two text files in python?

Python has provided the methods to manipulate files that too in a very concise manner. In this article we are going to discuss one of the applications of the Python's file handling features i.e. the comparison of files. Files in use: Text File 1Text File 2Method 1: Comparing complete file at once Py

How to search and replace text in a file in Python ?

In this article, we will learn how we can replace text in a file using python. Method 1: Searching and replacing text without using any external module Let see how we can search and replace text in a text file. First, we create a text file in which we want to search and replace text. Let this file b