2

Python (2.7)とBeautiful Soup (3.2.0)の両方を学習して試しています。最初の問題については、すでにここでいくつかの助けを得ています ( Beautiful Soup は `IndexError` をスローします) 。

これまでの Python コードは次のとおりです。

# Import the classes that are needed
import urllib2
from BeautifulSoup import BeautifulSoup

# URL to scrape and open it with the urllib2
url = 'http://www.wiziwig.tv/competition.php?competitionid=92&part=sports&discipline=football'
source = urllib2.urlopen(url)

# Turn the saced source into a BeautifulSoup object
soup = BeautifulSoup(source)

# From the source HTML page, search and store all <div class="date">...</div> and it's content
datesDiv = soup.findAll('div', { "class" : "date" })
# Loop through the tag and store only the needed information, being the actual date
dates = [tag.contents[0] for tag in datesDiv]

# From the source HTML page, search and store all <span class="time">...</span> and it's content
timesSpan = soup.findAll('span', { "class" : "time" })
# Loop through the tag and store only the needed information, being the actual times
times = [tag.contents[0] for tag in timesSpan]

# From the source HTML page, search and store all <td class="home">..</td> and it's content
hometeamsTd = soup.findAll('td', { "class" : "home" })
# Loop through the tag and store only the needed information, being the home team
# if tag.contents[1] != 'Dutch KNVB Beker' - Do a direct test if output is needed or not
hometeams = [tag.contents[1] for tag in hometeamsTd if tag.contents[1] != 'Dutch KNVB Beker']

# From the source HTML page, search and store all <td class="away">..</td> and it's content
# [1:] at the end meand slice the first one found
awayteamsTd = soup.findAll('td', { "class" : "away" })[1:]
# Loop through the tag and store only the needed information, being the away team
awayteams = [tag.contents[1] for tag in awayteamsTd]

# From the source HTML page, search and store all <a class="broadcast" href="...">..</a> and it's content
broadcastsA = soup.findAll('a', { "class" : "broadcast" })
# Loop through the tag and store only the needed information, being the the broadcast URL, where we can find the streams
broadcasts = [tag['href'] for tag in broadcastsA]

私が得た問題は、配列が互いに等しくないことです:

len(dates)      #9, should be 6
len(times)      #18, should be 12
len(hometeams)  #6, is correct
len(awayteams)  #6, is correct
len(broadcasts) #9, should be 6

私が抱えている問題は、dates配列を取得するために次の検索を行うことです: soup.findAll('div', { "class" : "date" }). これにより、明らかにすべての<div>要素がclass="date". <td>しかし、問題は、 の要素もある場合にのみ日付が必要だということですclass="away"。

私がスクレイピングしている HTML の次の部分を参照してください。

<tr class="odd">
    <td class="logo">
        <img src="/gfx/disciplines/football.gif" alt="football"/>
    </td>
    <td>
        <a href="/competition.php?part=sports&amp;competitionid=92&amp;discipline=football">Dutch Cup</a>
        <img src="/gfx/favourite_off.gif" class="fav off" alt="fav icon" id="comp-92"/>
    </td>
    <td>
        <div class="date" rel="1380054900">Tuesday, September 24</div> <!-- This date is not needed, because within this <tr> there is no <td class="away"> -->
        <span class="time" rel="1380054900">22:35</span> - <!-- This time is not needed, because within this <tr> there is no <td class="away"> -->
    <span class="time" rel="1380058500">23:35</span> <!-- This time is not needed, because within this <tr> there is no <td class="away"> -->
    </td>
    <td class="home" colspan="3">
        <img class="flag" src="/gfx/flags/nl.gif" alt="nl"/>Dutch KNVB Beker<img src="/gfx/favourite_off.gif" alt="fav icon" class="fav off" id="team-6758"/>
    </td>
    <td class="broadcast">
        <a class="broadcast" href="/broadcast.php?matchid=221554&amp;part=sports">Live</a> <!-- This href is not needed, because within this <tr> there is no <td class="away"> -->
    </td>
</tr>
<tr class="even">
    <td class="logo">
        <img src="/gfx/disciplines/football.gif" alt="football"/>
    </td>
    <td>
        <a href="/competition.php?part=sports&amp;competitionid=92&amp;discipline=football">Dutch Cup</a>
        <img src="/gfx/favourite_off.gif" class="fav off" alt="fav icon" id="comp-92"/>
    </td>
    <td>
        <div class="date" rel="1380127500">Wednesday, September 25</div> <!-- This date we would like to have, because now all records are complete, there is a <td class="away"> in this <tr> -->
        <span class="time" rel="1380127500">18:45</span> - <!-- This time we would like to have, because now all records are complete, there is a <td class="away"> in this <tr> -->
    <span class="time" rel="1380134700">20:45</span> <!-- This date we would like to have, because now all records are complete, there is a <td class="away"> in this <tr> -->
    </td>
    <td class="home">
        <img class="flag" src="/gfx/flags/nl.gif" alt="nl"/>PSV<img src="/gfx/favourite_off.gif" alt="fav icon" class="fav off" id="team-3"/>
    </td>
    <td>vs.</td>
    <td class="away">
        <img src="/gfx/favourite_off.gif" class="fav off" alt="fav icon" id="team-428"/>Stormvogels Telstar<img class="flag" src="/gfx/flags/nl.gif" alt="nl"/>
    </td>
    <td class="broadcast">
        <a class="broadcast" href="/broadcast.php?matchid=221555&amp;part=sports">Live</a> <!-- This href we would like to have, because now all records are complete, there is a <td class="away"> in this <tr> -->
    </td>
</tr>
4

1 に答える 1

3

データをスクレイピングする方法を考え直してはどうでしょうか。一致するテーブルがあります-次に、行を反復処理します。

for tr in soup.findAll('tr', {'class': ['odd', 'even']}):
    home_team = tr.find('td', {'class': 'home'}).text
    if home_team == 'Dutch KNVB Beker':
        continue

    away_team = tr.find('td', {'class': 'away'}).text
    date = ' - '.join([span.text for span in tr.findAll('span', {'class': 'time'})])
    broadcast = tr.find('a', {'class': 'broadcast'})['href']

    print home_team, away_team, date, broadcast

5行を出力します：

RKC Waalwijk Heracles 20:45 - 22:45 /broadcast.php?matchid=221553&part=sports
PSV Stormvogels Telstar 18:45 - 20:45 /broadcast.php?matchid=221555&part=sports
Ajax FC Volendam 20:45 - 22:45 /broadcast.php?matchid=221556&part=sports
SC Heerenveen FC Twente 18:45 - 20:45 /broadcast.php?matchid=221558&part=sports
Feyenoord FC Dordrecht 20:45 - 22:45 /broadcast.php?matchid=221559&part=sports

次に、結果を辞書のリストに収集できます。

于 2013-09-24T20:07:51.237 に答える