Django memcache低级api的性能低于cache mid

2024-10-01 04:52:53 发布

您现在位置：Python中文网/ 问答频道 /正文
7957 3

这是我第一次在这里发布问题，所以请忽略与标准格式的偏差。。。在

这是我的设置。 Nginx-->Uwsgi-->Django-->（Memcached/Mysql）

我在Django中有一组api，它们返回json响应。 之前我使用memcached的两部分中间件来进行自动缓存。现在为了获得更多的控制权，我已经转移到低级api。我使用celeri创建pickled数据集，每天在memcached中插入一次数据，我的api从memcached获取数据。在

我在我的生产环境中使用NGINX和Uwsgi（16个工作人员），通过两部分中间件方法获得了大约8K个请求的速度。只有我的第一个请求以前很慢。 但在较低级别的缓存中，我的速度是十分之一。在

最初我认为这是由于额外的中间件和正在执行的自定义代码，但我也删减了这一点。我从pickle转到cPickle，但我不认为这是瓶颈。我使用apache基准测试来进行测试。在

我为中间件方法和低级缓存api附加了Memcached（-vv）日志。 如果有人能给我指出正确的方向。。。。。。。。谢谢

我看到了Django的catche中间件代码。他们似乎在缓存中保存django httpresponse对象。中间件也在内部调用底层API，我也尝试过在缓存中保存httpresponse，但没有改变。。。在

日志

<pre>

MIDDLEWARE approach memcached LOGS
<28 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 END
<28 connection closed.
<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 END
<29 new auto-negotiating client connection
29: Client using the ascii protocol
<29 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 END
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 END
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 END
<31 new auto-negotiating client connection
31: Client using the ascii protocol
<31 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>31 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>31 END
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 END
<34 new auto-negotiating client connection
34: Client using the ascii protocol
<34 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>34 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>34 END
<35 new auto-negotiating client connection
35: Client using the ascii protocol
<35 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>35 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>35 END
<33 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 END
<28 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 END
<37 new auto-negotiating client connection
37: Client using the ascii protocol
<37 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>37 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
<36 new auto-negotiating client connection
36: Client using the ascii protocol
<36 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>36 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>36 END
<40 new auto-negotiating client connection
40: Client using the ascii protocol
<40 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>40 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>40 END
<30 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 END
<34 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>34 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>34 END
<38 new auto-negotiating client connection
<42 new auto-negotiating client connection
38: Client using the ascii protocol
<38 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>38 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>38 END
42: Client using the ascii protocol
<42 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>42 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>42 END
>37 END
<41 new auto-negotiating client connection
<33 connection closed.
<36 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
<31 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>36 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>36 END
41: Client using the ascii protocol
<41 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>41 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>41 END
>31 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>31 END
<39 new auto-negotiating client connection
39: Client using the ascii protocol
<39 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>39 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>39 END
<38 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>38 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>38 END
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 END
<31 connection closed.
<36 connection closed.
<29 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 END
<33 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 END
<39 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>39 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>39 END
<32 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 END
<35 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>35 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>35 END
<33 connection closed.
<30 connection closed.
<42 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>42 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>42 END
<37 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>37 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>37 END
<35 connection closed.
<39 connection closed.
<32 connection closed.
<31 new auto-negotiating client connection
31: Client using the ascii protocol
<31 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
<30 new auto-negotiating client connection
>31 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>31 END
>30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 END
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 END
<32 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 END
<30 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 END
<31 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>31 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>31 END
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 END
<33 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 END
<30 connection closed.
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 END
<38 connection closed.
<32 connection closed.
<30 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 END
<33 connection closed.
<37 connection closed.
<31 connection closed.
<29 connection closed.
<29 new auto-negotiating client connection
29: Client using the ascii protocol
<29 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 END
<34 connection closed.
<42 connection closed.
<41 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>41 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>41 END
<40 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>40 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>40 END
<28 connection closed.
<30 connection closed.
<29 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 END
<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 END
<28 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 END
<30 new auto-negotiating client connection
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 END
<40 connection closed.

</pre>


----------


----------


LOW level logs
<pre>
<43 connection closed.
<43 new auto-negotiating client connection
43: Client using the ascii protocol
<43 get :1:ls|1|0|10|||
>43 sending key :1:ls|1|0|10|||
>43 END
<33 connection closed.
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:ls|1|0|10|||
>33 sending key :1:ls|1|0|10|||
>33 END
<32 connection closed.
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:ls|1|0|10|||
>32 sending key :1:ls|1|0|10|||
>32 END
<36 connection closed.
<36 new auto-negotiating client connection
36: Client using the ascii protocol
<36 get :1:ls|1|0|10|||
>36 sending key :1:ls|1|0|10|||
>36 END
<40 connection closed.
<40 new auto-negotiating client connection
40: Client using the ascii protocol
<40 get :1:ls|1|0|10|||
>40 sending key :1:ls|1|0|10|||
>40 END
<34 connection closed.
<34 new auto-negotiating client connection
34: Client using the ascii protocol
<34 get :1:ls|1|0|10|||
>34 sending key :1:ls|1|0|10|||
>34 END
<29 connection closed.
<29 new auto-negotiating client connection
29: Client using the ascii protocol
<29 get :1:ls|1|0|10|||
>29 sending key :1:ls|1|0|10|||
>29 END
<35 connection closed.
<35 new auto-negotiating client connection
35: Client using the ascii protocol
<35 get :1:ls|1|0|10|||
>35 sending key :1:ls|1|0|10|||
>35 END
<37 connection closed.
<37 new auto-negotiating client connection
37: Client using the ascii protocol
<37 get :1:ls|1|0|10|||
>37 sending key :1:ls|1|0|10|||
>37 END
<41 connection closed.
<41 new auto-negotiating client connection
41: Client using the ascii protocol
<41 get :1:ls|1|0|10|||
>41 sending key :1:ls|1|0|10|||
>41 END
<39 connection closed.
<39 new auto-negotiating client connection
39: Client using the ascii protocol
<39 get :1:ls|1|0|10|||
>39 sending key :1:ls|1|0|10|||
>39 END
<38 connection closed.
<38 new auto-negotiating client connection
38: Client using the ascii protocol
<38 get :1:ls|1|0|10|||
>38 sending key :1:ls|1|0|10|||
>38 END
<30 connection closed.
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:ls|1|0|10|||
>30 sending key :1:ls|1|0|10|||
>30 END
<31 connection closed.
<31 new auto-negotiating client connection
31: Client using the ascii protocol
<31 get :1:ls|1|0|10|||
>31 sending key :1:ls|1|0|10|||
>31 END
<42 connection closed.
<42 new auto-negotiating client connection
42: Client using the ascii protocol
<42 get :1:ls|1|0|10|||
>42 sending key :1:ls|1|0|10|||
>42 END
<28 connection closed.
<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 get :1:ls|1|0|10|||
>28 sending key :1:ls|1|0|10|||
>28 END
<43 connection closed.
<43 new auto-negotiating client connection
43: Client using the ascii protocol
<43 get :1:ls|1|0|10|||
>43 sending key :1:ls|1|0|10|||
>43 END
<33 connection closed.
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:ls|1|0|10|||
>33 sending key :1:ls|1|0|10|||
>33 END
<32 connection closed.
<34 connection closed.
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:ls|1|0|10|||
>32 sending key :1:ls|1|0|10|||

</pre>

Tags： keydecoratorscachegetpageconnectionviewsen
1条回答
网友
1楼 · 发布于 2024-10-01 04:52:53

经过一周的尝试和测试（多亏了我的团队），我意识到Django低级别api有助于获得优于数据库获取的优势，但对于严重的缓存性能却没有好处。 Django中间件类在中间件堆栈吞吐量服务器进程中只实例化一次。如Django official docs和django book所述。在

在使用低级API时，可能启动到Memcache的连接花费了太多时间。 最后，我继承了FetchfromCache middleware并创建了自己的定制中间件，它的性能与Django的两部分中间件相同。剥离不需要的中间件也确实有助于提高性能。在

相关问题 更多 >

    编程相关推荐