1 year ago

#354215

test-img

GvPStack

Parallelize independent class attributes setting

The part of my code that I need to parallelize is something like this:

for ClassInstance in ClassInstancesList:
    ClassInstance.set_attributes(arguments)

With the method "set_attributes" having no return and just setting the attributes of the class instance.

I tried using multiprocessing and concurrent.futures, but both of those do a copy of the class instance which is not what I want.

The fixes that I saw (returning self, returning all the attributes and using another method to set the attributes, or using multiprocessing.Value) would either make copies of a big number of lists of lists or make me change the methods in my class in such a way as to make it very difficult to read. (set_attributes actually calls various methods set_attribute_A, set_attribute_B etc..)

In my case the threads can be completely independent.

EDIT: Here is my attempt at a minimal reproducible example:

class Object:
def _init_(self, initial_attributes):
    self.attributes1 = initial_attributes

def update(self, attributes):
    self.attributes1.append(attributes)

def set_attributes2(self, args):
    # Computations based on attributes 1 and args, in the real code many other
    # similar private methods are called
    self._set_attribute(args)
     

def detect_and_fill_Objects(args):
    ObjectList = detect(args) # other function which initializes instances and updates them 
    # at this point, the objects instances only have attributes 1 set
    # This following loop is the one that I want to parallelize, the one that sets
    # the attributes 2
    for Object in ObjectList:
        Object.set_attributes2(args)

When I ran the code using multiprocessing there was a great speed-up but all the computations were lost because they were done one copies of the instances and not the instances themselves, therefore I believe that a decent speedup could be obtained ?

python

class

multiprocessing

concurrent.futures

0 Answers

Your Answer

Accepted video resources