Python - Django - Sum too slow from a list

Question

Python - Django - Sum too slow from a list

Navigation

#1 by (1 votes)

1

I have a Django project related to energy consumption, through which I visualize the consumption of a household during a month. To obtain that information, I use the following code:

consumos = Measurement.objects.filter(idhogar=hogar)
filtrados = consumos.values_list("coste",flat=True).filter(timestampfinal__gt=iniciomes,timestampfinal__lte=finalmes)
suma = sum(filtrados)

Update 1 - The Measurement model:

class Measurement(models.Model):
     measurementid = models.AutoField(db_column='MeasurementID', primary_key=True)  # Field name made lowercase.
     valor = models.FloatField(db_column='Valor', blank=True, null=True)  # Field name made lowercase.
     coste = models.FloatField(db_column='Coste', blank=True, null=True)  # Field name made lowercase.
     timestampinicio = models.BigIntegerField(db_column='timestampinicio', blank=True, null=True)  # Field name made lowercase.
     timestampfinal = models.BigIntegerField(db_column='timestampfinal', blank=True, null=True)  # Field name made lowercase.
     devicesetupid = models.ForeignKey(Devicesetup, models.DO_NOTHING, db_column='DeviceSetupID', blank=True, null=True)  # Field name made lowercase.

The fact is that I get a list with a length of 9000 values. But when adding all the values of said list, it takes between 3 and 4 seconds. I have tried to create a python script that adds a list of 9000 values, and it takes a lot less (0.000176 seconds)

array = range(9000)
inicio = datetime.datetime.now()
suma = sum(array)
print "Tiempo: \t{}".format(datetime.datetime.now()-inicio)

I do not understand the reason for this big time difference, since in the Django code I get a list from the database and I do not do more queries, so I should calculate the sum much faster.

I have tried to put Debug = False because I have read that in this way the time is reduced, but it has not worked.

What can be the cause of this problem, if the treated lists have the same length?

python django python-2.7

asked by ikermdagirre 02.03.2017 в 12:02

source

1 answer

Mock call to external web service with Mockito in Java [closed] Compare a file with an Argument

score 1 · Answer 1

Have you tried using a Aggregation ? So the calculation would only your database engine.

from django.db.models import F, FloatField, Sum

consumos = Measurement.objects.filter(idhogar=hogar)
filtrados = consumos.filter(timestampfinal__range=[iniciomes, finalmes]).aggregate(Sum(F("coste"), output_field=FloatField()))

Also, if you are going to use the expression __lt and __gt with the same field, you can use range , which is much more readable:).

Even so, in the face of when the size of your table grows, you should add a db_index=True in the timestampfinal column.

Greetings!