Eliminating repeated lines from a file using Python Last Updated : 06 Aug, 2023 Comments Improve Suggest changes Like Article Like Report Let us see how to delete several repeated lines from a file using Python's File Handling power. If the file is small with a few lines, then the task of deleting/eliminating repeated lines from it could be done manually, but when it comes to large files, this is where Python comes to your rescue. Eliminating repeated lines from a file in PythonBelow are the methods that we will cover in this article: Using a ListUsing a SetUsing Pandas libraryInput File: For the sake of this example let's create a file (Lorem_input.txt) with some Ipsum text in it. All the repeated lines are marked in bold. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus est neque, mollis vel massa vel, condimentum facilisis ipsum. Mauris vitae mollis magna. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam laoreet vitae nisi quis rutrum. Sed ut ligula nec enim consequat egestas vel a sapien. Pellentesque sit amet euismod felis. Pellentesque in nibh ultricies, convallis sapien id, sagittis odio. Vivamus placerat ex sed ligula porttitor dignissim. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. Morbi posuere eget odio ut venenatis. Nam lobortis bibendum maximus. Donec venenatis sapien sed varius accumsan. Eliminating Repeated Lines from a File using a ListOpen the input file using the open() function and pass in the flag -r to open in reading mode then open an output file, using the -w flag, where we would store the contents of the file after deleting all repeated lines from it. Using the list() method keep track of all the lines seen so far so that we can compare it with the current reading line. Now, iterate over each line of the input file and compare it with the lines seen so far. If the current line is also present in lines seen so far, then skip that line else write that line to the output file, and don't forget to add the current line to the lines seen so far. Now let's create an empty output file (Lorem_output.txt), where we will store the modified input file. Python3 def remove_duplicates(input_file, output_file): lines_seen = set() with open(output_file, 'w') as out_file: with open(input_file, 'r') as in_file: for line in in_file: if line not in lines_seen: out_file.write(line) lines_seen.add(line) # Usage input_file = open('C:/Users/user/Desktop/Lorem_input.txt', "r") output_file = open('C:/Users/user/Desktop/Lorem_output.txt', "w") remove_duplicates(input_file, output_file) Output file: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus est neque, mollis vel massa vel, condimentum facilisis ipsum. Mauris vitae mollis magna. Aliquam laoreet vitae nisi quis rutrum. Sed ut ligula nec enim consequat egestas vel a sapien. Pellentesque sit amet euismod felis. Pellentesque in nibh ultricies, convallis sapien id, sagittis odio. Vivamus placerat ex sed ligula porttitor dignissim. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. Morbi posuere eget odio ut venenatis. Nam lobortis bibendum maximus. Donec venenatis sapien sed varius accumsan. Eliminating Repeated Lines from a File using a SetOpen the input file using the open() function and pass in the flag -r to open in reading mode. Open an output file, using the -w flag, where we would store the contents of the file after deleting all repeated lines from it. Using the set() method keep track of all the lines seen so far so that we can compare it with the current reading line. Now, iterate over each line of the input file and compare it with the lines seen so far. If the current line is also present in lines seen so far, then skip that line else write that line to the output file, and don't forget to add the current line to the lines seen so far and close the files. Now let's create an empty output file (Lorem_output.txt), where we will store the modified input file. Python3 # creating the output file outputFile = open('C:/Users/user/Desktop/Lorem_output.txt', "w") # reading the input file inputFile = open('C:/Users/user/Desktop/Lorem_input.txt', "r") # holds lines already seen lines_seen_so_far = set() # iterating each line in the file for line in inputFile: # checking if line is unique if line not in lines_seen_so_far: # write unique lines in output file outputFile.write(line) # adds unique lines to lines_seen_so_far lines_seen_so_far.add(line) # closing the file inputFile.close() outputFile.close() Running the above Python script will remove all the repeated lines from the input file and write the modified file to the output file. After running this script the output file(Lorem_output.txt) will look something like this Output file: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus est neque, mollis vel massa vel, condimentum facilisis ipsum. Mauris vitae mollis magna. Aliquam laoreet vitae nisi quis rutrum. Sed ut ligula nec enim consequat egestas vel a sapien. Pellentesque sit amet euismod felis. Pellentesque in nibh ultricies, convallis sapien id, sagittis odio. Vivamus placerat ex sed ligula porttitor dignissim. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. Morbi posuere eget odio ut venenatis. Nam lobortis bibendum maximus. Donec venenatis sapien sed varius accumsan. Eliminating Repeated Lines from a File using PandasThe function remove_duplicates reads the data from the input file, represented by the input_file variable, into a pandas DataFrame.Then it uses the drop_duplicates method to remove duplicate rows in place. The resulting unique data frame is saved back to a new file specified by the output_file variable. The correct file paths are defined, and the function is called with the input and output file paths to eliminate repeated lines from the input file and save the unique content to the output file. Now let's create an empty output file (Lorem_output.txt), where we will store the modified input file. Python3 import pandas as pd def remove_duplicates(input_file, output_file): df = pd.read_csv(input_file, header=None) df.drop_duplicates(inplace=True) df.to_csv(output_file, header=False, index=False) # creating the output file outputFile = open('C:/Users/user/Desktop/Lorem_output.txt', "w") # reading the input file inputFile = open('C:/Users/user/Desktop/Lorem_input.txt', "r") remove_duplicates(input_file, output_file) Output file: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus est neque, mollis vel massa vel, condimentum facilisis ipsum. Mauris vitae mollis magna. Aliquam laoreet vitae nisi quis rutrum. Sed ut ligula nec enim consequat egestas vel a sapien. Pellentesque sit amet euismod felis. Pellentesque in nibh ultricies, convallis sapien id, sagittis odio. Vivamus placerat ex sed ligula porttitor dignissim. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. Morbi posuere eget odio ut venenatis. Nam lobortis bibendum maximus. Donec venenatis sapien sed varius accumsan. Comment More infoAdvertise with us Next Article Eliminating repeated lines from a file using Python E equbalzeeshan Follow Improve Article Tags : Python python-file-handling Python file-handling-programs Practice Tags : python Similar Reads Python Programs Practice with Python program examples is always a good choice to scale up your logical understanding and programming skills and this article will provide you with the best sets of Python code examples.The below Python section contains a wide collection of Python programming examples. These Python co 11 min read Basic ProgramsHow to Add Two Numbers in PythonThe task of adding two numbers in Python involves taking two input values and computing their sum using various techniques . For example, if a = 5 and b = 7 then after addition, the result will be 12.Using the "+" Operator+ operator is the simplest and most direct way to add two numbers . It perform 4 min read Find Maximum of two numbers in PythonFinding the maximum of two numbers in Python helps determine the larger of the two values. For example, given two numbers a = 7 and b = 3, you may want to extract the larger number, which in this case is 7. Let's explore different ways to do this efficiently.Using max()max() function is the most opt 2 min read Factorial of a Number - PythonThe factorial of a number is the product of all positive integers less than or equal to that number. For example, the factorial of 5 (denoted as 5!) is 5 × 4 × 3 × 2 × 1 = 120. In Python, we can calculate the factorial of a number using various methods, such as loops, recursion, built-in functions, 4 min read Python Program for Simple InterestThe task of calculating Simple Interest in Python involves taking inputs for principal amount, time period in years, and rate of interest per annum, applying the Simple Interest formula and displaying the result. For example, if p = 1000, t = 2 (years), and r = 5%, the Simple Interest is calculated 3 min read Python Program for Compound InterestLet us discuss the formula for compound interest. The formula to calculate compound interest annually is given by: A = P(1 + R/100) t Compound Interest = A - P Where, A is amount P is the principal amount R is the rate and T is the time spanFind Compound Interest with Python Python3 # Python3 progra 3 min read Python Program to Check Armstrong NumberGiven a number x, determine whether the given number is Armstrong number or not. A positive integer of n digits is called an Armstrong number of order n (order is number of digits) if. abcd... = pow(a,n) + pow(b,n) + pow(c,n) + pow(d,n) + .... Example: Input : 153 Output : Yes 153 is an Armstrong nu 6 min read Array ProgramsPython Program to Find Sum of ArrayGiven an array of integers, find the sum of its elements. Examples: Input : arr[] = {1, 2, 3}Output : 6Explanation: 1 + 2 + 3 = 6This Python program calculates the sum of an array by iterating through each element and adding it to a running total. The sum is then returned. An example usage is provid 4 min read Python Program to Find Largest Element in an ArrayTo find the largest element in an array, iterate over each element and compare it with the current largest element. If an element is greater, update the largest element. At the end of the iteration, the largest element will be found. Given an array, find the largest element in it. Input : arr[] = {1 4 min read Python Program for Array RotationHere we are going to see how we can rotate array with Python code. Array Rotation:  Python Program for Array Rotation ExamplePartitioning the sub arrays and reversing them Approach: Input arr[] = [1, 2, 3, 4, 5, 6, 7, 8], d = 1, size = 8 1) Reverse the entire list by swapping first and last numbers 7 min read Python Program for Reversal algorithm for array rotationWrite a function rotate(arr[], d, n) that rotates arr[] of size n by d elements. In this article, we will explore the Reversal Algorithm for array rotation and implement it in Python. Example Input: arr[] = [1, 2, 3, 4, 5, 6, 7] d = 2 Output: arr[] = [3, 4, 5, 6, 7, 1, 2] Rotation of the above array 2 min read Python Program to Split the array and add the first part to the endThere is a given array and split it from a specified position, and move the first part of the array add to the end. Examples: Input : arr[] = {12, 10, 5, 6, 52, 36} k = 2 Output : arr[] = {5, 6, 52, 36, 12, 10} Explanation : Split from index 2 and first part {12, 10} add to the end .Input : arr[] = 5 min read Python Program for Find remainder of array multiplication divided by nWrite a Python program for a given multiple numbers and a number n, the task is to print the remainder after multiplying all the numbers divided by n. Examples: Input: arr[] = {100, 10, 5, 25, 35, 14}, n = 11Output: 9Explanation: 100 x 10 x 5 x 25 x 35 x 14 = 61250000 % 11 = 9Input : arr[] = {100, 1 3 min read Python Program to check if given array is MonotonicGiven an array A containing n integers. The task is to check whether the array is Monotonic or not. An array is monotonic if it is either monotone increasing or monotone decreasing. An array A is monotone increasing if for all i <= j, A[i] <= A[j]. An array A is monotone decreasing if for all 7 min read List ProgramsPython program to interchange first and last elements in a listGiven a list, write a Python program to swap the first and last element of the list using Python.Examples: The last element of the list can be referred to as a list[-1]. Therefore, we can simply swap list[0] with list[-1].Python# Initialize a list my_list = [1, 2, 3, 4, 5] # Interchange first and la 5 min read Python Program to Swap Two Elements in a ListIn this article, we will explore various methods to swap two elements in a list in Python. The simplest way to do is by using multiple assignment.Example:Pythona = [10, 20, 30, 40, 50] # Swapping elements at index 0 and 4 # using multiple assignment a[0], a[4] = a[4], a[0] print(a)Output[50, 20, 30, 1 min read How To Find the Length of a List in PythonThe length of a list refers to the number of elements in the list. There are several methods to determine the length of a list in Python. For example, consider a list l = [1, 2, 3, 4, 5], length of this list is 5 as it contains 5 elements in it. Let's explore different methods to find the length of 2 min read Check if element exists in list in PythonIn this article, we will explore various methods to check if element exists in list in Python. The simplest way to check for the presence of an element in a list is using the in Keyword. Example:Pythona = [10, 20, 30, 40, 50] # Check if 30 exists in the list if 30 in a: print("Element exists in the 3 min read Different ways to clear a list in PythonIn this article, we will explore various methods to clear a list in Python. The simplest way to do is by using a loop.Using clear() MethodUsing clear() method we can remove all elements from a list. This method modifies the list in place and removes all its elements.Pythona = [1, 2, 3, 4, 5] a.clear 2 min read Reversing a List in PythonIn this article, we are going to explore multiple ways to reverse a list. Python provides several methods to reverse a list using built-in functions and manual approaches. The simplest way to reverse a list is by using the reverse() method. Let's take an example to reverse a list using reverse() met 4 min read Matrix ProgramsAdding and Subtracting Matrices in PythonIn this article, we will discuss how to add and subtract elements of the matrix in Python. Example: Suppose we have two matrices A and B. A = [[1,2],[3,4]] B = [[4,5],[6,7]] then we get A+B = [[5,7],[9,11]] A-B = [[-3,-3],[-3,-3]] Now let us try to implement this using Python 1. Adding elements of 4 min read Add Two Matrices - PythonThe task of adding two matrices in Python involves combining corresponding elements from two given matrices to produce a new matrix. Each element in the resulting matrix is obtained by adding the values at the same position in the input matrices. For example, if two 2x2 matrices are given as:Two 2x2 3 min read Python Program to Multiply Two MatricesGiven two matrices, we will have to create a program to multiply two matrices in Python. Example: Python Matrix Multiplication of Two-DimensionPythonmatrix_a = [[1, 2], [3, 4]] matrix_b = [[5, 6], [7, 8]] result = [[0, 0], [0, 0]] for i in range(2): for j in range(2): result[i][j] = (matrix_a[i][0] 5 min read Matrix Product - PythonThe task of calculating the matrix product in Python involves finding the product of all elements within a matrix or 2D list . This operation requires iterating through each element in the matrix and multiplying them together to obtain a single cumulative product. For example, given a matrix a = [[1 3 min read Transpose a matrix in Single line in PythonTranspose of a matrix is a task we all can perform very easily in Python (Using a nested loop). But there are some interesting ways to do the same in a single line. In Python, we can implement a matrix as a nested list (a list inside a list). Each element is treated as a row of the matrix. For examp 4 min read Python - Matrix creation of n*nMatrices are fundamental structures in programming and are widely used in various domains including mathematics, machine learning, image processing, and simulations. Creating an n×n matrix efficiently is an important step for these applications. This article will explore multiple ways to create such 3 min read Python | Get Kth Column of MatrixSometimes, while working with Python Matrix, one can have a problem in which one needs to find the Kth column of Matrix. This is a very popular problem in Machine Learning Domain and having solution to this is useful. Let's discuss certain ways in which this problem can be solved. Method #1 : Using 6 min read Python - Vertical Concatenation in MatrixGiven a String Matrix, perform column-wise concatenation of strings, handling variable lists lengths. Input : [["Gfg", "good"], ["is", "for"]] Output : ['Gfgis', 'goodfor'] Explanation : Column wise concatenated Strings, "Gfg" concatenated with "is", and so on. Input : [["Gfg", "good", "geeks"], ["i 3 min read String ProgramsPython Program to Check if a String is Palindrome or NotThe task of checking if a string is a palindrome in Python involves determining whether a string reads the same forward as it does backward. For example, the string "madam" is a palindrome because it is identical when reversed, whereas "hello" is not. Using two pointer techniqueThis approach involve 3 min read Python program to check whether the string is Symmetrical or PalindromeThe task of checking whether a string is symmetrical or palindrome in Python involves two main operations . A string is symmetrical if its first half matches the second half, considering the middle character for odd-length strings. A string is palindrome if it reads the same forward and backward.For 4 min read Reverse Words in a Given String in PythonIn this article, we explore various ways to reverse the words in a string using Python. From simple built-in methods to advanced techniques like recursion and stacks. We are going to see various techniques to reverse a string.Using split() and join()Using split() and join() is the most common method 2 min read How to Remove Letters From a String in PythonRemoving letters or specific characters from a string in Python can be done in several ways. But Python strings are immutable, so removal operations cannot be performed in-place. Instead, they require creating a new string, which uses additional memory. Let’s start with a simple method to remove a s 3 min read Check if String Contains Substring in PythonThis article will cover how to check if a Python string contains another string or a substring in Python. Given two strings, check whether a substring is in the given string. Input: Substring = "geeks" String="geeks for geeks"Output: yesInput: Substring = "geek" String="geeks for geeks"Output: yesEx 8 min read Python - Words Frequency in String ShorthandsWord frequency in String Shirthands in Python typically refers to calculating how often words appear in a given string using various concise and efficient methods. Using collections.CounterUsing collections.Counter allows you to count the frequency of each character in a string. By leveraging this, 3 min read Dictionary ProgramsPython - Ways to remove a key from dictionaryWe are given a dictionary and our task is to remove a specific key from it. For example, if we have the dictionary d = {"a": 1, "b": 2, "c": 3}, then after removing the key "b", the output will be {'a': 1, 'c': 3}.Using pop()pop() method removes a specific key from the dictionary and returns its cor 3 min read Merging or Concatenating two Dictionaries in PythonCombining two dictionaries is a common task when working with Python, especially when we need to consolidate data from multiple sources or update existing records. For example, we may have one dictionary containing user information and another with additional details and we'd like to merge them into 2 min read Python - Convert key-values list to flat dictionaryWe are given a list that contains tuples with the pairs of key and values we need to convert that list into a flat dictionary. For example a = [("name", "Ak"), ("age", 25), ("city", "NYC")] is a list we need to convert it to dictionary so that output should be a flat dictionary {'name': 'Ak', 'age': 3 min read Python - Insertion at the beginning in OrderedDictGiven an ordered dict, write a program to insert items in the beginning of the ordered dict. Examples: Input: original_dict = {'a':1, 'b':2} item to be inserted ('c', 3) Output: {'c':3, 'a':1, 'b':2}Input: original_dict = {'akshat':1, 'manjeet':2} item to be inserted ('nikhil', 3) Output: {'nikhil': 2 min read Python | Check order of character in string using OrderedDict( )Given an input string and a pattern, check if characters in the input string follows the same order as determined by characters present in the pattern. Assume there won’t be any duplicate characters in the pattern. Examples: Input: string = "engineers rock"pattern = "er";Output: trueExplanation: All 3 min read Python dictionary with keys having multiple inputsPrerequisite: Python-Dictionary. How to create a dictionary where a key is formed using inputs? Let us consider an example where have an equation for three input variables, x, y, and z. We want to store values of equation for different input triplets. Example 1: Python3 # Python code to demonstrate 4 min read Tuple ProgramsFind the Size of a Tuple in PythonThere are several ways to find the "size" of a tuple, depending on whether we are interested in the number of elements or the memory size it occupies. For Example: if we have a tuple like tup = (10, 20, 30, 40, 50), calling len(tup) will return 5, since there are five elements in the tuple.Using len 3 min read Python - Maximum and Minimum K elements in TupleSometimes, while dealing with tuples, we can have problem in which we need to extract only extreme K elements, i.e maximum and minimum K elements in Tuple. This problem can have applications across domains such as web development and Data Science. Let's discuss certain ways in which this problem can 8 min read Create a List of Tuples with Numbers and Their Cubes - PythonWe are given a list of numbers and our task is to create a list of tuples where each tuple contains a number and its cube. For example, if the input is [1, 2, 3], the output should be [(1, 1), (2, 8), (3, 27)].Using List ComprehensionList comprehension is one of the most efficient ways to achieve th 3 min read Python - Adding Tuple to List and Vice - VersaAdding a tuple to a list in Python can involve appending the entire tuple as a single element or merging its elements into the list. Conversely, adding a list to a tuple requires converting the list to a tuple and concatenating it with the existing tuple, as tuples are immutable. For example, adding 4 min read Python - Closest Pair to Kth index element in TupleIn the given problem, condition K specifies the maximum allowable difference between corresponding elements of tuples, i.e., it represents the proximity threshold for finding the nearest tuple. The goal is to find the tuple in the list whose elements have the smallest maximum difference compared to 7 min read Python - Join Tuples if similar initial elementSometimes, while working with Python tuples, we can have a problem in which we need to perform concatenation of records from the similarity of initial element. This problem can have applications in data domains such as Data Science. Let's discuss certain ways in which this task can be performed. Inp 8 min read Searching and Sorting ProgramsLinear Search - PythonGiven an array, arr of n elements, and an element x, find whether element x is present in the array. Return the index of the first occurrence of x in the array, or -1 if it doesn’t exist.Examples:Input: arr[] = [10, 50, 30, 70, 80, 20, 90, 40], x = 30Output : 2Explanation: For array [10, 50, 30, 70, 4 min read Bubble Sort - PythonBubble Sort is the simplest sorting algorithm that works by repeatedly swapping the adjacent elements if they are in the wrong order. Bubble Sort algorithm, sorts an array by repeatedly comparing adjacent elements and swapping them if they are in the wrong order. The algorithm iterates through the a 2 min read Selection Sort - PythonSelection Sort is a comparison-based sorting algorithm. It sorts an array by repeatedly selecting the smallest (or largest) element from the unsorted portion and swapping it with the first unsorted element. This process continues until the entire array is sorted.First we find the smallest element an 2 min read Insertion Sort - PythonInsertion sort is a simple sorting algorithm that works by iteratively inserting each element of an unsorted list into its correct position in a sorted portion of the list.Insertion SortThe insertionSort function takes an array arr as input. It first calculates the length of the array (n). If the le 3 min read Python Program for Recursive Insertion SortInsertion sort is a simple sorting algorithm that works the way we sort playing cards in our hands. Python Program for Recursive Insertion Sort for Iterative algorithm for insertion sortAlgorithm // Sort an arr[] of size ninsertionSort(arr, n) Loop from i = 1 to n-1. a) Pick element arr[i] and inser 4 min read Binary Search (Recursive and Iterative) - PythonBinary Search Algorithm is a searching algorithm used in a sorted array by repeatedly dividing the search interval in half. The idea of binary search is to use the information that the array is sorted and reduce the time complexity to O(log N). Below is the step-by-step algorithm for Binary Search:D 6 min read Pattern Printing ProgramsProgram to print the pattern 'G'In this article, we will learn how to print the pattern G using stars and white-spaces. Given a number n, we will write a program to print the pattern G over n lines or rows.Examples: Input : 7 Output : *** * * * *** * * * * *** Input : 9 Output : ***** * * * * *** * * * * * * ***** In this program, 6 min read Python | Print an Inverted Star PatternAn inverted star pattern involves printing a series of lines, each consisting of stars (*) that are arranged in a decreasing order. Here we are going to print inverted star patterns of desired sizes in Python Examples: 1) Below is the inverted star pattern of size n=5 (Because there are 5 horizontal 2 min read Python 3 | Program to print double sided stair-case patternBelow mentioned is the Python 3 program to print the double-sided staircase pattern. Examples: Input: 10Output: * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * Note: This code only works for even values of n. Example1 The given 3 min read Print with your own font using Python !!Programming's core function is printing text, but have you ever wished to give it a unique look by utilizing your own custom fonts? Python enables you to use imagination and overcome the standard fonts in your text outputs. In this article, we will do some cool Python tricks. For the user input, the 4 min read Like