送信するリクエストが最大 50 個しかない場合にうまく機能するコードを試していますが、現在はさらに多くのリクエスト (約 200 個) を試しています。サーバーが 60 個のリクエストの後に接続を閉じているように感じます。 「なし」の回答。
私が使用しているコードは次のとおりです。
urls = []
for kw in keywords:
urls.append("https://play.google.com/store/search?q=" + kw + "&c=apps&hl=fr")
from lxml import html
rs = (grequests.get(u) for u in urls)
responses = grequests.map(rs)
for idx, r in enumerate(responses): # r = results page
time.sleep(2)
responses_tree = html.fromstring(r.content) # Readable HTML
反応:
Out[893]:
[<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
None,
<Response [200]>,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None]
とトレースバック:
AttributeError Traceback (most recent call last)
C:\Users\Nathan\googlestore.py in <module>()
249 for idx, r in enumerate(responses): # r = results page
250 time.sleep(2)
--> 251 responses_tree = html.fromstring(r.content) # Readable HTML
252 apps_links = responses_tree.xpath('//a[@class="title"]/@href') # Get all links
253 if len(apps_links) > 10:
AttributeError: 'NoneType' object has no attribute 'content'
どうすればこれを修正できますか? 50 のリクエストが送信されてから 30 秒ほど待ってから、さらに 50 を送信する必要がありますか? サーバーへのリクエストに関する Google ドキュメントへのリンクを持っている人はいますか?
ありがとう!