Skip to content

Courses
Tutorials
Practice
Contests

NLP
Data Analysis Tutorial
Python - Data visualization tutorial
NumPy
Pandas
OpenCV
R
Machine Learning Tutorial
Machine Learning Projects
Machine Learning Interview Questions
Machine Learning Mathematics
Deep Learning Tutorial
Deep Learning Project
Deep Learning Interview Questions
Computer Vision Tutorial
Computer Vision Projects
NLP
NLP Project
NLP Interview Questions
Statistics with Python
100 Days of Machine Learning

Python Escape Characters

Python | Character Encoding

Last Updated : 29 May, 2021

Comments

Improve

Suggest changes

Like Article

Like

Report

Finding the text which is having nonstandard character encoding is a very common step to perform in text processing.
All the text would have been from utf-8 or ASCII encoding ideally but this might not be the case always. So, in such cases when the encoding is not known, such non-encoded text has to be detected and the be converted to a standard encoding. So, this step is important before processing the text further.
Charade Installation :
For performing the detection and conversion of encoding, charade – a Python library is required. This module can be simply installed using sudo easy_install charade or pip install charade.
Let’s see the wrapper function around the charade module.
Code : encoding.detect(string), to detect the encoding

Python3

   # -*- coding: utf-8 -*-
 
import charade
def detect(s):
     
    try:
        # check it in the charade list
        if isinstance(s, str):
            return charade.detect(s.encode())
        # detecting the string
          else:
            return charade.detect(s)
     
    # in case of error
    # encode with 'utf -8' encoding
    except UnicodeDecodeError:
        return charade.detect(s.encode('utf-8'))
 
  

The detect functions will return 2 attributes :

Confidence : the probability of charade being correct. Encoding   : which encoding it is.

Code : encoding.convert(string) to convert the encoding.

Python3

   # -*- coding: utf-8 -*-
import charade
 
def convert(s):
     
    # if in the charade instance
    if isinstance(s, str):
        s = s.encode()
     
    # retrieving the encoding information 
    # from the detect() output
    encode = detect(s)['encoding']
     
    if encode == 'utf-8':
        return s.decode()
    else:
        return s.decode(encoding) 
 
  

Code : Example

Python3

   # importing library
import encoding 
 
d1  = encoding.detect('geek')
print ("d1 is encoded as  : ", d1)
 
d2  = encoding.detect('ascii')
print ("d2 is encoded as  : ", d2)
 
  

Output :

d1 is encoded as : (confidence': 0.505, 'encoding': 'utf-8') d2 is encoded as : ('confidence': 1.0, 'encoding': 'ascii')

detect() : It is a charade.detect() wrapper. It encodes the strings and handles the UnicodeDecodeError exceptions. It expects a bytes object so therefore the string is encoded before trying to detect the encoding.
convert() : It is a charade.convert() wrapper. It calls detect() first to get the encoding. Then, it returns a decoded string.

Python Escape Characters

M

mathemagic

Improve

Article Tags :

Python
Natural-language-processing

Practice Tags :

python

Similar Reads

Python Escape Characters

In Python, escape characters are used when we need to include special characters in a string that are otherwise hard (or illegal) to type directly. These are preceded by a backslash (\), which tells Python that the next character is going to be a special character. Theyâ€™re especially helpful for: Fo

numpy.defchararray.encode() in Python

numpy.core.defchararray.encode(arr, encoding): This numpy function encodes the string(object) based on the specified codec. Parameters: arr : array-like or string. encoding : [str] Name of encoding being followed. error : Specifying how to handle error. Returns : Encoded string Code: C/C++ Code # Py

chr() Function in Python

chr() function returns a string representing a character whose Unicode code point is the integer specified. chr() Example: C/C++ Code num = 97 print("ASCII Value of 97 is: ", chr(num)) OutputASCII Value of 97 is: a Python chr() Function Syntaxchr(num) Parametersnum: an Unicode code integer

codecs.decode() in Python

With the help of codecs.decode() method, we can decode the binary string into normal form by using codecs.decode() method. Syntax : codecs.decode(b_string) Return : Return the decoded string. Example #1 : In this example we can see that by using codecs.decode() method, we are able to get the decoded

Elias Delta Encoding in Python

In this article, we are going to implement Elias Delta encoding using python. Syntax: Elias Delta Encoding(X)= Elias Gamma encoding (1+floor(log2(X))) + Binary representation of X without MSB. ImplementationFirst, we are going to implement Elias delta Encoding, Before writing code for Elias Delta En

base64.encodestring(s) in Python

With the help of base64.encodestring(s) method, we can encode the string using base64 encoded data into the binary form. Syntax : base64.encodestring(string) Return : Return the encoded string. Example #1 : In this example we can see that by using base64.encodestring(s) method, we are able to get th

Ways to Print Escape Characters in Python

In Python, escape characters like \n (newline) and \t (tab) are used for formatting, with \n moving text to a new line and \t adding a tab space. By default, Python interprets these sequences, so I\nLove\tPython will display "Love" on a new line and a tab before "Python." However, if you want to dis

Python | os.device_encoding() method

OS module in Python provides functions for interacting with the operating system. OS comes under Pythonâ€™s standard utility modules. This module provides a portable way of using operating system dependent functionality. os.device_encoding() method in Python is used to get the encoding of the device a

How To Print Unicode Character In Python?

Unicode characters play a crucial role in handling diverse text and symbols in Python programming. This article will guide you through the process of printing Unicode characters in Python, showcasing five simple and effective methods to enhance your ability to work with a wide range of characters Pr

base64.encodebytes(s) in Python

With the help of base64.encodebytes(s) method, we can encode the string using base64 encoded data into the binary form. Syntax : base64.encodebytes(string) Return : Return the encoded string. Example #1 : In this example we can see that by using base64.encodebytes(s) method, we are able to get the e

Corporate & Communications Address:

A-143, 7th Floor, Sovereign Corporate Tower, Sector- 136, Noida, Uttar Pradesh (201305)

Registered Address:

K 061, Tower K, Gulshan Vivante Apartment, Sector 137, Noida, Gautam Buddh Nagar, Uttar Pradesh, 201305

Advertise with us

Company
About Us
Legal
Privacy Policy
In Media
Contact Us
Advertise with us
GFG Corporate Solution
Placement Training Program

Languages
Python
Java
C++
PHP
GoLang
SQL
R Language
Android Tutorial
Tutorials Archive

DSA
Data Structures
Algorithms
DSA for Beginners
Basic DSA Problems
DSA Roadmap
Top 100 DSA Interview Problems
DSA Roadmap by Sandeep Jain
All Cheat Sheets

Data Science & ML
Data Science With Python
Data Science For Beginner
Machine Learning
ML Maths
Data Visualisation
Pandas
NumPy
NLP
Deep Learning

Web Technologies
HTML
CSS
JavaScript
TypeScript
ReactJS
NextJS
Bootstrap
Web Design

Python Tutorial
Python Programming Examples
Python Projects
Python Tkinter
Python Web Scraping
OpenCV Tutorial
Python Interview Question
Django

Computer Science
Operating Systems
Computer Network
Database Management System
Software Engineering
Digital Logic Design
Engineering Maths
Software Development
Software Testing

DevOps
Git
Linux
AWS
Docker
Kubernetes
Azure
GCP
DevOps Roadmap

System Design
High Level Design
Low Level Design
UML Diagrams
Interview Guide
Design Patterns
OOAD
System Design Bootcamp
Interview Questions

Inteview Preparation
Competitive Programming
Top DS or Algo for CP
Company-Wise Recruitment Process
Company-Wise Preparation
Aptitude Preparation
Puzzles

School Subjects
Mathematics
Physics
Chemistry
Biology
Social Science
English Grammar
Commerce
World GK

GeeksforGeeks Videos
DSA
Python
Java
C++
Web Development
Data Science
CS Subjects

@GeeksforGeeks, Sanchhaya Education Private Limited, All rights reserved

We use cookies to ensure you have the best browsing experience on our website. By using our site, you acknowledge that you have read and understood our Cookie Policy & Privacy Policy

Improvement

Suggest Changes

Help us improve. Share your suggestions to enhance the article. Contribute your expertise and make a difference in the GeeksforGeeks portal.

geeksforgeeks-suggest-icon

Create Improvement

Enhance the article with your expertise. Contribute to the GeeksforGeeks community and help create better learning resources for all.

geeksforgeeks-improvement-icon

Suggest Changes

min 4 words, max Words Limit:1000

Thank You!

Your suggestions are valuable to us.

What kind of Experience do you want to share?

Interview Experiences

Admission Experiences

Career Journeys

Work Experiences

Campus Experiences

Competitive Exam Experiences