0

Link Extractor がリンクを取得しないのはなぜでしょうか。リンクエクストラクタでand属性を拡張CrawlSpiderおよび設定しています。Web ページ (http://www.realclearpolitics.com/epolls/latest_polls/president/) の選択ボックスからリンクを削除しようとしているので、値属性を持つオプション タグを探しています。tagsattrs

ここでコードを確認してください: https://github.com/jtfairbank/RCP-Poll-Scraper/blob/master/polldata/spiders/pres.py

ルート ディレクトリは次のとおりです: https://github.com/jtfairbank/RCP-Poll-Scraper

何か案は？ありがとう！

4

1 に答える 1

0

<option>でタグを検索しており、このページには選択タグまたはオプションタグtags=['option']がないため、リンク エクストラクタはリンクを取得しません。

試してくださいrestrict_xpaths='//td[@class="lp-race"]'：

>>> SgmlLinkExtractor(
...     allow=(r"epolls/2012/president/[a-z]{2}/[a-z]+_romney_vs_obama-[0-9]{4}\.html"),
...     allow_domains=('realclearpolitics.com',),
...     restrict_xpaths='//td[@class="lp-race"]',
... ).extract_links(response)

[Link(url='http://www.realclearpolitics.com/epolls/2012/president/fl/florida_romney_vs_ob...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/oh/ohio_romney_vs_obama...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/va/virginia_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ia/iowa_romney_vs_obama...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/wi/wisconsin_romney_vs_...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/or/oregon_romney_vs_oba...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/mo/missouri_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ca/california_romney_vs...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/md/maryland_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/mi/michigan_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/co/colorado_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/mn/minnesota_romney_vs_...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/wa/washington_romney_vs...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ct/connecticut_romney_v...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/nv/nevada_romney_vs_oba...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/mt/montana_romney_vs_ob...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/pa/pennsylvania_romney_...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/in/indiana_romney_vs_ob...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ma/massachusetts_romney...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ga/georgia_romney_vs_ob...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/id/idaho_romney_vs_obam...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/az/arizona_romney_vs_ob...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/me/maine_romney_vs_obam...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/la/louisiana_romney_vs_...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/hi/hawaii_romney_vs_oba...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/tx/texas_romney_vs_obam...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ar/arkansas_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ne/nebraska_romney_vs_o...',
于 2012-10-20T18:45:33.190 に答える