'utf-8' codec can't decode byte 0x80 in position 3131: invalid start byte': while reading xml files
I want to define a function that can be implemented on each xml file in the directory in order to parse it and get the content from the tags in a dataframe.
from xml.etree import ElementTree
def func(path, filename):
for filename in os.listdir(path):
with open(os.path.join(path, filename)) as file:
# Read each line in the file, readlines() returns a list of lines
content = file.readlines()
# Combine the lines in the list into a string
content = "".join(content)
bs_content = bs(content, "lxml")
headline = bs_content.find_all("headline")
eventtitle = bs_content.find_all("eventtitle")
city = bs_content.find_all("city")
companyname = bs_content.find_all("companyname")
companyticker = bs_content.find_all("companyticker")
startdate = bs_content.find_all("startdate")
eventstory = bs_content.find_all("eventstory")
data = []
for i in range(0,len(companyname)):
rows = [companyname[i].get_text(),headline[i].get_text(),
city[i].get_text(),eventtitle[i].get_text(),
companyticker[i].get_text(),startdate[i].get_text(),
eventstory[i].get_text()]
data.append(rows)
df = pd.DataFrame(data,columns = ['companyname','headline',
'city','eventtitle','companyticker',
'startdate','eventstory'], dtype = float)
When I call a function I receive this error. Unfortunately, every existing solutions didn't not work for me.
func('./Calls/', '1000015_T.xml')
---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
Input In [58], in <module>
----> 1 func('./Calls/', '1000015_T.xml')
Input In [57], in func(path, filename)
7 for filename in os.listdir(path):
8 with open(os.path.join(path, filename)) as file:
9 # Read each line in the file, readlines() returns a list of lines
---> 10 content = file.readlines()
11 # Combine the lines in the list into a string
12 content = "".join(content)
File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/codecs.py:322, in BufferedIncrementalDecoder.decode(self, input, final)
319 def decode(self, input, final=False):
320 # decode input (taking the buffer into account)
321 data = self.buffer + input
--> 322 (result, consumed) = self._buffer_decode(data, self.errors, final)
323 # keep undecoded input until the next call
324 self.buffer = data[consumed:]
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 3131: invalid start byte
Maybe you can also help me with code optimization. My task is to get the content of 2k xml files and so far I decided to define a function and then to use pandarallel: parallel_apply(func)
Solution 1:
The input file is not UTF-8, it is likely some other code page.
Determine what the correct encoding is and alter your program accordingly.