Object not pickleable running multiprocessing

1

I have a script that scrapes a data table from the web, so that each node represents a row, and inside the tag is information that I want to obtain. The linear sequence code works fine, but I have decided to try to execute it asynchronously with multiprocessing. This is the code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import multiprocessing

def ResPartido(node):

   ft=node.find_element_by_css_selector('.status').text
   if ft.strip()!='FT': return
   hora=node.find_element_by_css_selector('.time').text
   names=list()
   for nam in node.find_elements_by_xpath(
            './/td[contains(@style,"text-align")]/a[contains(@id,"team")]'):
     name=nam.text
     if '(N)' in name:
        name=name.split('(N)')[0]
     names.append(name)
   score=node.find_element_by_css_selector('.red')

   return [hora,name,score.text]

if __name__ == "__main__":

   browser=webdriver.Chrome()
   SOME CODE
   nodes=browser.find_elements_by_xpath(
        '//tr[contains(@align,"center")]/following-sibling::tr[.//div[contains(@class,"toolimg")]]')
   p = multiprocessing.Pool()

   p.map(ResPartido,nodes)   <---Here is the error
   .......

   >>AttributeError: Can't pickle local object '_createenviron.<locals>.encodekey'

As you can see, nodes is a list of all the tr scrapear. If I have not misunderstood it, multiprocessing generates a pool with the nodes inside nodes, and passes them simultaneously to the Resparted function so that its code is executed.

The lists are supposed to be pickable objects, however, I get an error in p.map(ResPartido,nodes)

Attached captures with the complete error in case it would help

    
asked by puppet 14.11.2017 в 09:56
source

0 answers