Dynamic number of threads in celery

I guess this a followup to the previous post, where i talked about celery and some pref and examples about it, i think this could be a series where i show the problems i faced during the project and how i overcome them.

The problem

The project is running in multiple servers with different amount of resources, the mistake i did is running celery with fixed number of threads in all servers this cause smaller servers to hang since the receive huge number of tasks larger thant their resources.

The solution

Solution is very simple i came up with an equation to calculate threads number based on amount of CPUs and Ram the server has.

1
#worker.sh
2
cpus=$(nproc --all)
3
ram=$(free -g | awk '/^Mem:/{print $2}')
4
threads=$((cpus*15 + ram*15))
5

6
celery -A celery_app.tasks worker --loglevel=info --pool=threads -c $threads

the 15 here is arbitrary number, of course this should be determined carefully based on tasks type and wight.