1

「サポートされていない URL スキーム」を報告する Scrapy Spider に問題があります。検索結果のあるページをスクレイプしたい。この長い動的 URL が原因で、Spider は常に失敗します。

class RadioSpider(CrawlSpider):
    name = 'radio'
    allowed_domains = ['dashitradio.de']
    start_urls = ["[http://www.dashitradio.de/nc/search-in-playlist.html?tx_wfqbe_pi1%5BSTART%5D=2013-06-17%2006:00&tx_wfqbe_pi1%5BEND%5D=2013-06-21%2018:00&tx_wfqbe_pi1%5Bsubmit%5D=Suchen&tx_wfqbe_pi1%5Bshowpage%5D%5B3%5D=1][1]"]
    rules = (
        Rule(SgmlLinkExtractor(allow=r'Items/'), callback='parse_item', follow=True),
    )

    def parse_item(self, response):
        hxs = HtmlXPathSelector(response)
        i = RadioItem()

        i['title'] = hxs.select("//*[@id='playlist-results']/table//tr[1]/td[1]/text()").extract()
        i['interpret'] = hxs.select("//*[@id='playlist-results']/table[1]//tr/td[2]/text()").extract()
        i['date'] = hxs.select("//*[@id='playlist-results']/table//tr[1]/td[3]/text()").extract()

        return i

Scrapy Shell コンソールで実行すると、URL 以外の逆コンマでのみ正常に動作します"URL"。

Spider 内でこの文字列を単一の URL として Scrapy に受け入れさせるにはどうすればよいですか?

4

1 に答える 1

0

start_urls設定が正しくありません。[最初と][1]最後で無効な URL にします。

あなたのコメントに従って、スパイダーのコードを更新しました。

from scrapy.item import Item, Field
from scrapy.selector import HtmlXPathSelector
from scrapy.spider import BaseSpider


class RadioItem(Item):
    title = Field()
    interpret = Field()
    date = Field()


class RadioSpider(BaseSpider):
    name = 'radio'
    allowed_domains = ['dashitradio.de']
    start_urls = ["http://www.dashitradio.de/nc/search-in-playlist.html?tx_wfqbe_pi1%5BSTART%5D=2013-06-17%2006:00&tx_wfqbe_pi1%5BEND%5D=2013-06-21%2018:00&tx_wfqbe_pi1%5Bsubmit%5D=Suchen&tx_wfqbe_pi1%5Bshowpage%5D%5B3%5D=1"]

    def parse(self, response):
        hxs = HtmlXPathSelector(response)

        rows = hxs.select("//div[@id='playlist-results']/table/tbody/tr")
        for row in rows:
            item = RadioItem()

            item['title'] = row.select(".//td[1]/text()").extract()[0]
            item['interpret'] = row.select(".//td[2]/text()").extract()[0]
            item['date'] = row.select(".//td[3]/text()").extract()[0]

            yield item

に保存してmy_spider.py実行しrunspiderます：

scrapy runspider my_spider.py -o output.json

そして、次のように表示されますoutput.json。

{"date": "2013-06-21 17:48:00", "interpret": "MUMFORD & SONS", "title": "I WILL WAIT"}
{"date": "2013-06-21 17:44:00", "interpret": "TASMIN ARCHER", "title": "SLEEPING SATELLITE"}
{"date": "2013-06-21 17:40:03", "interpret": "ROBIN THICKE", "title": "BLURRED LINES (feat. T.I. & PHARRELL)"}
{"date": "2013-06-21 17:35:02", "interpret": "TINA TURNER", "title": "TWO PEOPLE"}
{"date": "2013-06-21 17:31:02", "interpret": "BON JOVI", "title": "WHAT ABOUT NOW"}
{"date": "2013-06-21 17:28:03", "interpret": "ROXETTE", "title": "SHE'S GOT NOTHING ON (BUT THE RADIO)"}
{"date": "2013-06-21 17:18:01", "interpret": "GNARLS BARKLEY", "title": "CRAZY"}
{"date": "2013-06-21 17:08:01", "interpret": "FLO RIDA", "title": "WHISTLE"}
{"date": "2013-06-21 17:05:03", "interpret": "WHAM", "title": "WAKE ME UP BEFORE YOU GO GO"}
{"date": "2013-06-21 17:00:03", "interpret": "P!NK FEAT. NATE RUESS", "title": "JUST GIVE ME A REASON"}
{"date": "2013-06-21 16:48:01", "interpret": "SHAKIRA", "title": "WHENEVER, WHEREVER"}
{"date": "2013-06-21 16:44:00", "interpret": "ALPHAVILLE", "title": "BIG IN JAPAN"}
{"date": "2013-06-21 16:40:01", "interpret": "XAVIER NAIDOO", "title": "BEI MEINER SEELE"}
{"date": "2013-06-21 16:36:02", "interpret": "SANTANA", "title": "SMOOTH"}
{"date": "2013-06-21 16:32:01", "interpret": "OLLY MURS", "title": "ARMY OF TWO"}

それが役立つことを願っています。

于 2013-07-08T08:57:04.737 に答える