0

ここに質問を投稿するのはこれが初めてなので、標準形式からの逸脱は無視してください...

これが私のセットアップです。Nginx --> Uwsgi --> Django --> (Memcached/Mysql)

Django には、json 応答を返す一連の API があります。以前は、自動キャッシュを行うために memcached 2 パート ミドルウェアを使用していました。より詳細に制御するために、低レベルの API に移行しました。私はセロリを使用して、ピクルス化されたデータ セットを作成し、1 日 1 回 memcached にデータを挿入します。My Apis は memcached からデータをフェッチします。

私は実稼働環境で NGINX と Uwsgi (16 ワーカー) を使用しており、2 パートのミドルウェア アプローチで約 8K のリクエストの速度を得ていました。私の最初のリクエストだけが遅かったです。しかし、低レベルのキャッシュを使用すると、1/10 の速度が得られます。

最初は、追加のミドルウェアとカスタム コードが実行されたためだと思っていましたが、それも削除しました。pickle から cPickle に移行しましたが、それがボトルネックになっているとは思いません。Apache ベンチマークを使用してテストを実施しています。

ミドルウェア アプローチと低レベル キャッシュ API の両方の Memcached ( -vv) ログを添付しています。誰かが私を正しい方向に向けることができれば........ありがとう

Django の catche-middleware コードを見ました。彼らはdjango httpresponseオブジェクトをキャッシュに保存しているようです。ミドルウェアも内部で低レベルのAPIを呼び出しており、httpresponseもキャッシュに保存してみましたが変化なし...

ログ

<pre>

MIDDLEWARE approach memcached LOGS
<28 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 END
<28 connection closed.
<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 END
<29 new auto-negotiating client connection
29: Client using the ascii protocol
<29 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 END
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 END
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 END
<31 new auto-negotiating client connection
31: Client using the ascii protocol
<31 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>31 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>31 END
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 END
<34 new auto-negotiating client connection
34: Client using the ascii protocol
<34 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>34 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>34 END
<35 new auto-negotiating client connection
35: Client using the ascii protocol
<35 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>35 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>35 END
<33 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 END
<28 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 END
<37 new auto-negotiating client connection
37: Client using the ascii protocol
<37 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>37 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
<36 new auto-negotiating client connection
36: Client using the ascii protocol
<36 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>36 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>36 END
<40 new auto-negotiating client connection
40: Client using the ascii protocol
<40 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>40 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>40 END
<30 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 END
<34 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>34 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>34 END
<38 new auto-negotiating client connection
<42 new auto-negotiating client connection
38: Client using the ascii protocol
<38 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>38 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>38 END
42: Client using the ascii protocol
<42 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>42 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>42 END
>37 END
<41 new auto-negotiating client connection
<33 connection closed.
<36 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
<31 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>36 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>36 END
41: Client using the ascii protocol
<41 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>41 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>41 END
>31 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>31 END
<39 new auto-negotiating client connection
39: Client using the ascii protocol
<39 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>39 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>39 END
<38 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>38 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>38 END
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 END
<31 connection closed.
<36 connection closed.
<29 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 END
<33 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 END
<39 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>39 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>39 END
<32 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 END
<35 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>35 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>35 END
<33 connection closed.
<30 connection closed.
<42 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>42 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>42 END
<37 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>37 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>37 END
<35 connection closed.
<39 connection closed.
<32 connection closed.
<31 new auto-negotiating client connection
31: Client using the ascii protocol
<31 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
<30 new auto-negotiating client connection
>31 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>31 END
>30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 END
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 END
<32 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 END
<30 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 END
<31 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>31 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>31 END
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 END
<33 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 END
<30 connection closed.
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 END
<38 connection closed.
<32 connection closed.
<30 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 END
<33 connection closed.
<37 connection closed.
<31 connection closed.
<29 connection closed.
<29 new auto-negotiating client connection
29: Client using the ascii protocol
<29 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 END
<34 connection closed.
<42 connection closed.
<41 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>41 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>41 END
<40 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>40 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>40 END
<28 connection closed.
<30 connection closed.
<29 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 END
<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 END
<28 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 END
<30 new auto-negotiating client connection
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 END
<40 connection closed.

</pre>


----------


----------


LOW level logs
<pre>
<43 connection closed.
<43 new auto-negotiating client connection
43: Client using the ascii protocol
<43 get :1:ls|1|0|10|||
>43 sending key :1:ls|1|0|10|||
>43 END
<33 connection closed.
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:ls|1|0|10|||
>33 sending key :1:ls|1|0|10|||
>33 END
<32 connection closed.
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:ls|1|0|10|||
>32 sending key :1:ls|1|0|10|||
>32 END
<36 connection closed.
<36 new auto-negotiating client connection
36: Client using the ascii protocol
<36 get :1:ls|1|0|10|||
>36 sending key :1:ls|1|0|10|||
>36 END
<40 connection closed.
<40 new auto-negotiating client connection
40: Client using the ascii protocol
<40 get :1:ls|1|0|10|||
>40 sending key :1:ls|1|0|10|||
>40 END
<34 connection closed.
<34 new auto-negotiating client connection
34: Client using the ascii protocol
<34 get :1:ls|1|0|10|||
>34 sending key :1:ls|1|0|10|||
>34 END
<29 connection closed.
<29 new auto-negotiating client connection
29: Client using the ascii protocol
<29 get :1:ls|1|0|10|||
>29 sending key :1:ls|1|0|10|||
>29 END
<35 connection closed.
<35 new auto-negotiating client connection
35: Client using the ascii protocol
<35 get :1:ls|1|0|10|||
>35 sending key :1:ls|1|0|10|||
>35 END
<37 connection closed.
<37 new auto-negotiating client connection
37: Client using the ascii protocol
<37 get :1:ls|1|0|10|||
>37 sending key :1:ls|1|0|10|||
>37 END
<41 connection closed.
<41 new auto-negotiating client connection
41: Client using the ascii protocol
<41 get :1:ls|1|0|10|||
>41 sending key :1:ls|1|0|10|||
>41 END
<39 connection closed.
<39 new auto-negotiating client connection
39: Client using the ascii protocol
<39 get :1:ls|1|0|10|||
>39 sending key :1:ls|1|0|10|||
>39 END
<38 connection closed.
<38 new auto-negotiating client connection
38: Client using the ascii protocol
<38 get :1:ls|1|0|10|||
>38 sending key :1:ls|1|0|10|||
>38 END
<30 connection closed.
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:ls|1|0|10|||
>30 sending key :1:ls|1|0|10|||
>30 END
<31 connection closed.
<31 new auto-negotiating client connection
31: Client using the ascii protocol
<31 get :1:ls|1|0|10|||
>31 sending key :1:ls|1|0|10|||
>31 END
<42 connection closed.
<42 new auto-negotiating client connection
42: Client using the ascii protocol
<42 get :1:ls|1|0|10|||
>42 sending key :1:ls|1|0|10|||
>42 END
<28 connection closed.
<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 get :1:ls|1|0|10|||
>28 sending key :1:ls|1|0|10|||
>28 END
<43 connection closed.
<43 new auto-negotiating client connection
43: Client using the ascii protocol
<43 get :1:ls|1|0|10|||
>43 sending key :1:ls|1|0|10|||
>43 END
<33 connection closed.
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:ls|1|0|10|||
>33 sending key :1:ls|1|0|10|||
>33 END
<32 connection closed.
<34 connection closed.
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:ls|1|0|10|||
>32 sending key :1:ls|1|0|10|||

</pre>
4

1 に答える 1

1

(私のチームのおかげで) 1 週間試してテストした後、Django の低レベル Api はデータベースのフェッチよりも優位に立つことができますが、深刻なキャッシングのパフォーマンスには適していないことに気付きました。Django ミドルウェア クラスは、ミドルウェア スタック スループット サーバー プロセスで 1 回だけインスタンス化されます。Django official docsおよびdjango bookで述べたように。

低レベルの API を使用すると、おそらく Memcache への接続の開始に時間がかかりすぎていました。最後に、 FetchfromCache ミドルウェアを継承し、Django の 2 つの部分からなるミドルウェアと同じパフォーマンスを提供する独自のカスタム ミドルウェアを作成しました。不要なミドルウェアを削除することも、パフォーマンスの向上に役立ちます。

于 2013-07-16T14:02:05.410 に答える