I would like to ask: why does it take some time for the Multi-processing module that comes with Python to start?

I built a concurrent Async system using the Multi-processing module that comes with Python. There are only 2 tasks, one fast and one slow, concurrent execution, fast and slow.

import sys, os, time, datetime
from timeit import timeit
import threading
import multiprocessing
from multiprocessing import Pool

def multiprocessor_hack(instance, name, args=(), kwargs=None):
    if kwargs is None:
        kwargs = {}
    return getattr(instance, name)(*args, **kwargs)

class Controller:
    request = "Whatever"
    def task1(self, request):
        print ("Now starting task1 at {}.\n".format(datetime.datetime.now()))
        time.sleep(3)
        print ("Now finished task1 at {}.\n".format(datetime.datetime.now()))
        return "111111"
    def task2(self, request):
        print ("Now starting task2 at {}.\n".format(datetime.datetime.now()))
        time.sleep(10)
        print ("Now finished task2 at {}.\n".format(datetime.datetime.now()))
        return "222222"

    def moderate(self, request):
        pool = multiprocessing.Pool(processes = 2)
        results = [pool.apply_async(multiprocessor_hack, args = (self, task1", (request,))), pool.apply_async(multiprocessor_hack, args = (self, "task2", (request,)))]
        pool.close()
        map (multiprocessing.pool.ApplyResult.wait, results)
        response = [r.get() for r in results]
        return response


if "__main__" == __name__:
    ctrl = Controller()
    print ("\nThe pool starts at {}.\n".format(datetime.datetime.now()))
    response = ctrl.moderate("Whatever")
    print ("\nResponse emerges at {}.".format(datetime.datetime.now()))
    print ("\nThe pool ends at {}.".format(datetime.datetime.now()))

the result is:

The pool starts at 2018-03-23 15:03:51.187000.
Now starting task1 at 2018-03-23 15:03:51.522000.  -sharp 
Now starting task2 at 2018-03-23 15:03:51.522000.  -sharp 
Now finished the task1 at 2018-03-23 15:03:54.524000.
Now finished the task2 at 2018-03-23 15:04:01.524000.
Response emerges at 2018-03-23 15:04:01.526000.
The pool ends at 2018-03-23 15:04:01.528000.

here, the pool starts running at 3: 15 p.m., but the two tasks don"t start until 15: 15, 03, and 51.522! May I ask why there is a delay 0.4 seconds here? Is there any way to reduce the delay ?

Thank you first!

Feb.27,2021

guess that it is possible to import when compiling class, so it is slow. You can try compiling py to see if it has been promoted


I searched it. The Multi-processing module needs a certain amount of time to establish communication between main process and work process, and resources are allocated by the system, so it takes a certain amount of time to start.


opening multiple processes requires fork , which is very expensive, which is equivalent to reopening a python. Can not open multi-process without multi-process, you can use threads instead of processes, CPython also has GIL such things, sometimes open multi-process or multi-thread CPU utilization will be reduced (scheduling and planning is not done).

Menu