1 year ago
#354215
GvPStack
Parallelize independent class attributes setting
The part of my code that I need to parallelize is something like this:
for ClassInstance in ClassInstancesList:
ClassInstance.set_attributes(arguments)
With the method "set_attributes" having no return and just setting the attributes of the class instance.
I tried using multiprocessing and concurrent.futures, but both of those do a copy of the class instance which is not what I want.
The fixes that I saw (returning self, returning all the attributes and using another method to set the attributes, or using multiprocessing.Value) would either make copies of a big number of lists of lists or make me change the methods in my class in such a way as to make it very difficult to read. (set_attributes actually calls various methods set_attribute_A, set_attribute_B etc..)
In my case the threads can be completely independent.
EDIT: Here is my attempt at a minimal reproducible example:
class Object:
def _init_(self, initial_attributes):
self.attributes1 = initial_attributes
def update(self, attributes):
self.attributes1.append(attributes)
def set_attributes2(self, args):
# Computations based on attributes 1 and args, in the real code many other
# similar private methods are called
self._set_attribute(args)
def detect_and_fill_Objects(args):
ObjectList = detect(args) # other function which initializes instances and updates them
# at this point, the objects instances only have attributes 1 set
# This following loop is the one that I want to parallelize, the one that sets
# the attributes 2
for Object in ObjectList:
Object.set_attributes2(args)
When I ran the code using multiprocessing there was a great speed-up but all the computations were lost because they were done one copies of the instances and not the instances themselves, therefore I believe that a decent speedup could be obtained ?
python
class
multiprocessing
concurrent.futures
0 Answers
Your Answer