5

I am using a Pool to benefit of multiple cores. Each worker in the pool needs its own Calculator object. The initialization of calculator is quite time consuming, so I would like to have it generated only once per worker in the pool and not every time, a new task arrives. The only way, I got this working was by using the “ugly“ keyword global. Is there a “cleaner” way to implement this?

I would like to avoid queues (parent thread is often sigkill’d and leaves child processes when using queues) and managers (performance too slow).

#!/usr/bin/python
# -*- coding: utf-8 -*-

import multiprocessing

def init_pool():
    global calculator
    calculator = Calculator()   # should only executed ones per worker

def run_pool(args):
    return calculator.calculate(*args)  # time consuming calculation

class Organiser():
    def __init__(self):
        self.__pool = multiprocessing.Pool(initializer=init_pool)

    def process(self, tasks):
        results = self.__pool.map(run_pool, tasks)
        return results
4

1 に答える 1

2

あなたが望むものを達成する方法がわかりません（ワーカーごとに正確に1回初期化します）。

ただし、ワーカーのグループ全体に対して「Calculator」を一度だけ初期化する場合は、次のように機能するようです。

def run_pool(args):
    calculator,arg = args
    return calculator.calculate(arg)  # time consuming calculation

class Organiser():
    def __init__(self):
        self.calculator = Calculator()
        self.__pool = multiprocessing.Pool(processes=4)

    def process(self, tasks):
        results = self.__pool.map(run_pool, [(self.calculator,data) for data in tasks])
        return results

ワーカーごとに正確に 1 回初期化するには、グローバル変数またはシングルトン (同等のもの) を使用する必要があるようです。あなたの質問に対する他の回答もお待ちしています:)

よろしく、 シッダールス

于 2012-10-31T06:45:43.823 に答える