java - HTML用のWebサイトをスクレイプできませんか？

Question

そのため、アプリケーションをWebサイトに移動させ、そのWebサイトからHTMLを取得し、HTMLから不要な要素を削除してから、makeshiftアプリに「コンテンツ」をロードしようとしています。APIがないためです。またはフィード。私はJsoupを使用しており、Android以外でWebスクレイピングを実行している場合は機能しますが、Androidはそれを好みません。

public class SimpleDiggActivity extends Activity {

private WebView browser;
final Activity activity = this;

@Override
public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    this.getWindow().requestFeature(Window.FEATURE_PROGRESS);

    setContentView(R.layout.main);

    getWindow().setFeatureInt(Window.FEATURE_PROGRESS, Window.PROGRESS_VISIBILITY_ON);

    String url = "http://www.digg.com";
    Document digg;
    browser = (WebView) findViewById(R.id.mybrowser);
    final Button homeDigg = (Button) findViewById(R.id.button1);

    browser.setWebViewClient(new SimpleWebViewClient());

    browser.getSettings().setJavaScriptEnabled(true);
    browser.getSettings().setUseWideViewPort(true);
    browser.getSettings().setLoadWithOverviewMode(true);
    browser.getSettings().setDisplayZoomControls(false);
    browser.getSettings().setEnableSmoothTransition(true);
    browser.getSettings().setBuiltInZoomControls(true);
    browser.getSettings().setUserAgentString("Android");

    // progressCircle = ProgressDialog.show(SimpleDiggActivity.this, "", "Loading...");
    final ProgressDialog progressCircle = new ProgressDialog(activity);
    progressCircle.setProgressStyle(ProgressDialog.STYLE_SPINNER);
    progressCircle.setMessage("Loading...");
    progressCircle.setCancelable(false);

    try{
        Toast.makeText(getApplicationContext(), "No Steps down", Toast.LENGTH_SHORT).show();
        Document diggTest = Jsoup.connect("http://digg.com/enable/mobile").get();
        Toast.makeText(getApplicationContext(), "1 Steps down", Toast.LENGTH_SHORT).show();
        String diggTitle = diggTest.title();
        Toast.makeText(getApplicationContext(), "2 Steps down"    , Toast.LENGTH_SHORT).show();
        Document compressed = Jsoup.parseBodyFragment(diggTitle);
        Toast.makeText(getApplicationContext(), "3 Steps down", Toast.LENGTH_SHORT).show();
        org.jsoup.select.Elements div = diggTest.select("div");
        Toast.makeText(getApplicationContext(), "4 Steps down", Toast.LENGTH_SHORT).show();
        String divBrow = div.toString();
        Toast.makeText(getApplicationContext(), "5 Steps down", Toast.LENGTH_SHORT).show();
        browser.loadUrl(divBrow);
    }catch (Exception e){
        e.printStackTrace();

        Toast.makeText(getApplicationContext(), "Gave up", Toast.LENGTH_SHORT).show();
        String diggBrow = url;
        browser.loadUrl("http://www.google.com");
    }

散らかっていたらごめんなさい、いじりまわっているだけで初めてです。Toastsは、コードが試行に失敗し、キャッチに頼っているときに通知するためのものです。私がそれを実行するとき、それはそれを通り過ぎません

 Document diggTest = Jsoup.connect("http://digg.com/enable/mobile").get();

score 0 · Accepted Answer

JSOUPバージョン1.7.1でコードを試しましたが、最後は正常に機能しています。動作するコードは次のとおりです。

public class SimpleDiggActivity extends Activity {

    final Activity activity = this;

    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        this.getWindow().requestFeature(Window.FEATURE_PROGRESS);

        setContentView(R.layout.activity_simple_digg);

        getWindow().setFeatureInt(Window.FEATURE_PROGRESS,
                Window.PROGRESS_VISIBILITY_ON);

        String url = "http://www.digg.com";
        Document digg;

        // progressCircle = ProgressDialog.show(SimpleDiggActivity.this, "",
        // "Loading...");
        final ProgressDialog progressCircle = new ProgressDialog(activity);
        progressCircle.setProgressStyle(ProgressDialog.STYLE_SPINNER);
        progressCircle.setMessage("Loading...");
        progressCircle.setCancelable(false);

        try {
            Toast.makeText(getApplicationContext(), "No Steps down",
                    Toast.LENGTH_SHORT).show();
            Document diggTest = Jsoup.connect("http://digg.com/enable/mobile")
                    .get();
            Toast.makeText(getApplicationContext(), "1 Steps down",
                    Toast.LENGTH_SHORT).show();
            String diggTitle = diggTest.title();
            Toast.makeText(getApplicationContext(), "2 Steps down",
                    Toast.LENGTH_SHORT).show();
            Document compressed = Jsoup.parseBodyFragment(diggTitle);
            Toast.makeText(getApplicationContext(), "3 Steps down",
                    Toast.LENGTH_SHORT).show();
            org.jsoup.select.Elements div = diggTest.select("div");
            Toast.makeText(getApplicationContext(), "4 Steps down",
                    Toast.LENGTH_SHORT).show();
            String divBrow = div.toString();
            Toast.makeText(getApplicationContext(), "5 Steps down",
                    Toast.LENGTH_SHORT).show();
            Log.d(this.getClass().getSimpleName(), "data is " + divBrow);
        } catch (Exception e) {
            e.printStackTrace();

            Toast.makeText(getApplicationContext(), "Gave up",
                    Toast.LENGTH_SHORT).show();
            String diggBrow = url;
        }
    }
}

divBrowの値は次のとおりです。

10-10 11:58:45.631: D/SimpleDiggActivity(350): data is <div class="site-header-container page-container"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):  <header class="site-header"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):   <h1 class="site-header-logo-container"><a href="/" id="site-header-logo" class="image-replace">Digg</a></h1> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):  </header> 
10-10 11:58:45.631: D/SimpleDiggActivity(350): </div>
10-10 11:58:45.631: D/SimpleDiggActivity(350): <div id="container" class="page-container"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):  <ul id="top-stories"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):   <li class="story-container story-1" data-content-id="Racz8K" id="story-Racz8K"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):    <div class="story-details"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     <div class="story-kicker">
10-10 11:58:45.631: D/SimpleDiggActivity(350):       NO FILTER 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     </div> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     <div class="story-headline"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):      <a data-position="0" class="story-link" href="http://www.fastcompany.com/3001994/no-filter-inside-hipstamatics-lost-year-searching-next-killer-social-app" data-content-id="Racz8K"> Inside Hipstamatic’s Lost Year Searching For The Next Killer Social&nbsp;App </a> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     </div> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     <div class="story-domain"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):      <div class="story-link-wrapper"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):       <a data-position="0" class="story-link" href="http://www.fastcompany.com/3001994/no-filter-inside-hipstamatics-lost-year-searching-next-killer-social-app" data-content-id="Racz8K">fastcompany.com</a> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):      </div> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):      <div class="story-actions"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):       <span class="story-action-item story-score"> <span class="story-score-details"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):         <ul class="story-score-details-list"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):          <li class="story-score-thumb-Racz8K story-score-thumb">20</li> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):          <li class="story-score-tweets-Racz8K story-score-twitter">402</li> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):          <li class="story-score-fb_shares-Racz8K story-score-facebook">72</li> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):         </ul> </span> <span class="story-score-Racz8K">494</span> </span> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):      </div> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     </div> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     <div class="story-image"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):      <a data-position="0" class="story-link" href="http://www.fastcompany.com/3001994/no-filter-inside-hipstamatics-lost-year-searching-next-killer-social-app" data-content-id="Racz8K"><img src="http://static.digg.com/images/Racz8K_1_www_large_thumb.jpeg" alt="" width="312" height="170" /></a> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     </div> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     <div class="story-preview">
10-10 11:58:45.631: D/SimpleDiggActivity(350):      From rooftop bashes and acquisition talks to staff clashes and layoffs, Hipstamatic’s founders and ex-employees describe the startup’s losing struggle to keep pace with Instagram, Facebook, and others in the white-hot photo-sharing space.
10-10 11:58:45.631: D/SimpleDiggActivity(350):     </div> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):    </div> </li> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):   <li class="story-container story-1" data-content-id="Qa2sP3" id="story-Qa2sP3"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):    <div class="story-details"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     <div class="story-kicker">
10-10 11:58:45.631: D/SimpleDiggActivity(350):       PHOTOGRAPHY 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     </div> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     <div class="story-headline"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):      <a data-position="1" class="story-link" href="http://lens.blogs.nytimes.com/2012/10/09/looking-into-the-eyes-of-made-in-china/" data-content-id="Qa2sP3"> Looking Into The Eyes Of 'Made In&nbsp;China' </a> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     </div> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     <div class="story-domain"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):      <div class="story-link-wrapper"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):       <a data-position="1" class="story-link" href="http://lens.blogs.nytimes.com/2012/10/09/looking-into-the-eyes-of-made-in-china/" data-content-id="Qa2sP3">lens.blogs.nytimes.com</a> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):      </div> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):      <div class="story-actions"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):       <span class="story-action-item story-score"> <span class="story-score-details"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):         <ul class="story-score-details-list"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):          <li class="story-score-thumb-Qa2sP3 story-score-thumb">0</li> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):          <li class="story-score-tweets-Qa2sP3 story-score-twitter">252</li> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):          <li class="story-score-fb_shares-Qa2sP3 story-score-facebook">411</li> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):         </ul> </span> <span class="story-score-Qa2sP3">663</span> </span> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):      </div> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     </div> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     <div class="story-image"> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):      <a data-position="1" class="story-link" href="http://lens.blogs.nytimes.com/2012/10/09/looking-into-the-eyes-of-made-in-china/" data-content-id="Qa2sP3"><img src="http://static.digg.com/images/Qa2sP3_1_www_large_thumb.jpeg" alt="" width="312" height="170" /></a> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     </div> 
10-10 11:58:45.631: D/SimpleDiggActivity(350):     <div class="story-preview">
10-10 11:58:45.631: D/SimpleDiggActivity(350):      In “Faces of Made in China,” a series of typological portraits looking at workers inside six Chinese factories, the photographer Lucas Schifres seeks to consider the otherwise anonymous people who produce our essential possessions by looking directly into their eyes.
10-10 11:58:45.631: D/SimpleDiggActivity(350):     </div>

最後に試してみて、どうだったか教えてください。

java - HTML用のWebサイトをスクレイプできませんか？

1 に答える 1

Related

Reference