I have a script that scrapes a data table from the web, so that each node represents a row, and inside the tag is information that I want to obtain. The linear sequence code works fine, but I have decided to try to execute it asynchronously with multiprocessing. This is the code
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import multiprocessing
def ResPartido(node):
ft=node.find_element_by_css_selector('.status').text
if ft.strip()!='FT': return
hora=node.find_element_by_css_selector('.time').text
names=list()
for nam in node.find_elements_by_xpath(
'.//td[contains(@style,"text-align")]/a[contains(@id,"team")]'):
name=nam.text
if '(N)' in name:
name=name.split('(N)')[0]
names.append(name)
score=node.find_element_by_css_selector('.red')
return [hora,name,score.text]
if __name__ == "__main__":
browser=webdriver.Chrome()
SOME CODE
nodes=browser.find_elements_by_xpath(
'//tr[contains(@align,"center")]/following-sibling::tr[.//div[contains(@class,"toolimg")]]')
p = multiprocessing.Pool()
p.map(ResPartido,nodes) <---Here is the error
.......
>>AttributeError: Can't pickle local object '_createenviron.<locals>.encodekey'
As you can see, nodes is a list of all the tr scrapear. If I have not misunderstood it, multiprocessing generates a pool with the nodes inside nodes, and passes them simultaneously to the Resparted function so that its code is executed.
The lists are supposed to be pickable objects, however, I get an error in p.map(ResPartido,nodes)
Attached captures with the complete error in case it would help