I have a problem with an idea that I have in mind but to achieve it I need a lot of machine resources. The idea is this:
Create a web system, with django, in which there are several users (suppose there are many); each user can create one or several processes (around 200 processes) that execute code indefinitely on the server (it can take up to days to give a result), for the process creation management I'm using celery (which I do not know if it's the Better option). I know this is very expensive, so my question is: Is it possible to achieve this task without spending so many machine resources? As I understand celery works with threads, which are supposed to be lighter than creating new processes, but in the same way it is too expensive.