Use of java.net.URLConnection is asked about pretty often here, and the Oracle tutorial is too concise about it.
That tutorial basically only shows how to fire a GET request and read the response. It doesn't explain anywhere how to use it to, among others, perform a POST request, set request headers, read response headers, deal with cookies, submit a HTML form, upload a file, etc.
So, how can I use java.net.URLConnection to fire and handle "advanced" HTTP requests?
First a disclaimer beforehand: the posted code snippets are all basic examples. You'll need to handle trivial IOExceptions and RuntimeExceptions like NullPointerException, ArrayIndexOutOfBoundsException and consorts yourself.
In case you're developing for Android instead of Java, note also that since introduction of API level 28, cleartext HTTP requests are disabled by default. You are encouraged to use HttpsURLConnection, but if it is really necessary, cleartext can be enabled in the Application Manifest.
We first need to know at least the URL and the charset. The parameters are optional and depend on the functional requirements.
String url = "http://example.com";
String charset = "UTF-8"; // Or in Java 7 and later, use the constant: java.nio.charset.StandardCharsets.UTF_8.name()
String param1 = "value1";
String param2 = "value2";
// ...
String query = String.format("param1=%s¶m2=%s",
URLEncoder.encode(param1, charset),
URLEncoder.encode(param2, charset));
The query parameters must be in name=value format and be concatenated by &. You would normally also URL-encode the query parameters with the specified charset using URLEncoder#encode().
The String#format() is just for convenience. I prefer it when I would need the String concatenation operator + more than twice.
It's a trivial task. It's the default request method.
URLConnection connection = new URL(url + "?" + query).openConnection();
connection.setRequestProperty("Accept-Charset", charset);
InputStream response = connection.getInputStream();
// ...
Any query string should be concatenated to the URL using ?. The Accept-Charset header may hint the server what encoding the parameters are in. If you don't send any query string, then you can leave the Accept-Charset header away. If you don't need to set any headers, then you can even use the URL#openStream() shortcut method.
InputStream response = new URL(url).openStream();
// ...
Either way, if the other side is an HttpServlet, then its doGet() method will be called and the parameters will be available by HttpServletRequest#getParameter().
For testing purposes, you can print the response body to standard output as below:
try (Scanner scanner = new Scanner(response)) {
String responseBody = scanner.useDelimiter("\\A").next();
System.out.println(responseBody);
}
Setting the URLConnection#setDoOutput() to true implicitly sets the request method to POST. The standard HTTP POST as web forms do is of type application/x-www-form-urlencoded wherein the query string is written to the request body.
URLConnection connection = new URL(url).openConnection();
connection.setDoOutput(true); // Triggers POST.
connection.setRequestProperty("Accept-Charset", charset);
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=" + charset);
try (OutputStream output = connection.getOutputStream()) {
output.write(query.getBytes(charset));
}
InputStream response = connection.getInputStream();
// ...
Note: whenever you'd like to submit a HTML form programmatically, don't forget to take the name=value pairs of any <input type="hidden"> elements into the query string and of course also the name=value pair of the <input type="submit"> element which you'd like to "press" programmatically (because that's usually been used in the server side to distinguish if a button was pressed and if so, which one).
You can also cast the obtained URLConnection to HttpURLConnection and use its HttpURLConnection#setRequestMethod() instead. But if you're trying to use the connection for output you still need to set URLConnection#setDoOutput() to true.
HttpURLConnection httpConnection = (HttpURLConnection) new URL(url).openConnection();
httpConnection.setRequestMethod("POST");
// ...
Either way, if the other side is an HttpServlet, then its doPost() method will be called and the parameters will be available by HttpServletRequest#getParameter().
You can fire the HTTP request explicitly with URLConnection#connect(), but the request will automatically be fired on demand when you want to get any information about the HTTP response, such as the response body using URLConnection#getInputStream() and so on. The above examples does exactly that, so the connect() call is in fact superfluous.
You need an HttpURLConnection here. Cast it first if necessary.
int status = httpConnection.getResponseCode();
for (Entry<String, List<String>> header : connection.getHeaderFields().entrySet()) {
System.out.println(header.getKey() + "=" + header.getValue());
}
When the Content-Type contains a charset parameter, then the response body is likely text based and we'd like to process the response body with the server-side specified character encoding then.
String contentType = connection.getHeaderField("Content-Type");
String charset = null;
for (String param : contentType.replace(" ", "").split(";")) {
if (param.startsWith("charset=")) {
charset = param.split("=", 2)[1];
break;
}
}
if (charset != null) {
try (BufferedReader reader = new BufferedReader(new InputStreamReader(response, charset))) {
for (String line; (line = reader.readLine()) != null;) {
// ... System.out.println(line)?
}
}
} else {
// It's likely binary content, use InputStream/OutputStream.
}
The server side session is usually backed by a cookie. Some web forms require that you're logged in and/or are tracked by a session. You can use the CookieHandler API to maintain cookies. You need to prepare a CookieManager with a CookiePolicy of ACCEPT_ALL before sending all HTTP requests.
// First set the default cookie manager.
CookieHandler.setDefault(new CookieManager(null, CookiePolicy.ACCEPT_ALL));
// All the following subsequent URLConnections will use the same cookie manager.
URLConnection connection = new URL(url).openConnection();
// ...
connection = new URL(url).openConnection();
// ...
connection = new URL(url).openConnection();
// ...
Note that this is known to not always work properly in all circumstances. If it fails for you, then best is to manually gather and set the cookie headers. You basically need to grab all Set-Cookie headers from the response of the login or the first GET request and then pass this through the subsequent requests.
// Gather all cookies on the first request.
URLConnection connection = new URL(url).openConnection();
List<String> cookies = connection.getHeaderFields().get("Set-Cookie");
// ...
// Then use the same cookies on all subsequent requests.
connection = new URL(url).openConnection();
for (String cookie : cookies) {
connection.addRequestProperty("Cookie", cookie.split(";", 2)[0]);
}
// ...
The split(";", 2)[0] is there to get rid of cookie attributes which are irrelevant for the server side like expires, path, etc. Alternatively, you could also use cookie.substring(0, cookie.indexOf(';')) instead of split().
The HttpURLConnection will by default buffer the entire request body before actually sending it, regardless of whether you've set a fixed content length yourself using connection.setRequestProperty("Content-Length", contentLength);. This may cause OutOfMemoryExceptions whenever you concurrently send large POST requests (e.g. uploading files). To avoid this, you would like to set the HttpURLConnection#setFixedLengthStreamingMode().
httpConnection.setFixedLengthStreamingMode(contentLength);
But if the content length is really not known beforehand, then you can make use of chunked streaming mode by setting the HttpURLConnection#setChunkedStreamingMode() accordingly. This will set the HTTP Transfer-Encoding header to chunked which will force the request body being sent in chunks. The below example will send the body in chunks of 1 KB.
httpConnection.setChunkedStreamingMode(1024);
It can happen that a request returns an unexpected response, while it works fine with a real web browser. The server side is probably blocking requests based on the User-Agent request header. The URLConnection will by default set it to Java/1.6.0_19 where the last part is obviously the JRE version. You can override this as follows:
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"); // Do as if you're using Chrome 41 on Windows 7.
Use the User-Agent string from a recent browser.
If the HTTP response code is 4nn (Client Error) or 5nn (Server Error), then you may want to read the HttpURLConnection#getErrorStream() to see if the server has sent any useful error information.
InputStream error = ((HttpURLConnection) connection).getErrorStream();
If the HTTP response code is -1, then something went wrong with connection and response handling. The HttpURLConnection implementation is in older JREs somewhat buggy with keeping connections alive. You may want to turn it off by setting the http.keepAlive system property to false. You can do this programmatically in the beginning of your application by:
System.setProperty("http.keepAlive", "false");
You'd normally use multipart/form-data encoding for mixed POST content (binary and character data). The encoding is in more detail described in RFC2388.
String param = "value";
File textFile = new File("/path/to/file.txt");
File binaryFile = new File("/path/to/file.bin");
String boundary = Long.toHexString(System.currentTimeMillis()); // Just generate some unique random value.
String CRLF = "\r\n"; // Line separator required by multipart/form-data.
URLConnection connection = new URL(url).openConnection();
connection.setDoOutput(true);
connection.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
try (
OutputStream output = connection.getOutputStream();
PrintWriter writer = new PrintWriter(new OutputStreamWriter(output, charset), true);
) {
// Send normal param.
writer.append("--" + boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"param\"").append(CRLF);
writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF);
writer.append(CRLF).append(param).append(CRLF).flush();
// Send text file.
writer.append("--" + boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"textFile\"; filename=\"" + textFile.getName() + "\"").append(CRLF);
writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF); // Text file itself must be saved in this charset!
writer.append(CRLF).flush();
Files.copy(textFile.toPath(), output);
output.flush(); // Important before continuing with writer!
writer.append(CRLF).flush(); // CRLF is important! It indicates end of boundary.
// Send binary file.
writer.append("--" + boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"binaryFile\"; filename=\"" + binaryFile.getName() + "\"").append(CRLF);
writer.append("Content-Type: " + URLConnection.guessContentTypeFromName(binaryFile.getName())).append(CRLF);
writer.append("Content-Transfer-Encoding: binary").append(CRLF);
writer.append(CRLF).flush();
Files.copy(binaryFile.toPath(), output);
output.flush(); // Important before continuing with writer!
writer.append(CRLF).flush(); // CRLF is important! It indicates end of boundary.
// End of multipart/form-data.
writer.append("--" + boundary + "--").append(CRLF).flush();
}
If the other side is an HttpServlet, then its doPost() method will be called and the parts will be available by HttpServletRequest#getPart() (note, thus not getParameter() and so on!). The getPart() method is however relatively new, it's introduced in Servlet 3.0 (Glassfish 3, Tomcat 7, etc.). Prior to Servlet 3.0, your best choice is using Apache Commons FileUpload to parse a multipart/form-data request. Also see this answer for examples of both the FileUpload and the Servelt 3.0 approaches.
In case you're developing for Android instead of Java, be careful: the workaround below may save your day if you don't have correct certificates deployed during development. But you should not use it for production. These days (April 2021) Google will not allow your app be distributed on Play Store if they detect insecure hostname verifier, see https://support.google.com/faqs/answer/7188426.
Sometimes you need to connect an HTTPS URL, perhaps because you're writing a web scraper. In that case, you may likely face a javax.net.ssl.SSLException: Not trusted server certificate on some HTTPS sites who doesn't keep their SSL certificates up to date, or a java.security.cert.CertificateException: No subject alternative DNS name matching [hostname] found or javax.net.ssl.SSLProtocolException: handshake alert: unrecognized_name on some misconfigured HTTPS sites.
The following one-time-run static initializer in your web scraper class should make HttpsURLConnection more lenient as to those HTTPS sites and thus not throw those exceptions anymore.
static {
TrustManager[] trustAllCertificates = new TrustManager[] {
new X509TrustManager() {
@Override
public X509Certificate[] getAcceptedIssuers() {
return null; // Not relevant.
}
@Override
public void checkClientTrusted(X509Certificate[] certs, String authType) {
// Do nothing. Just allow them all.
}
@Override
public void checkServerTrusted(X509Certificate[] certs, String authType) {
// Do nothing. Just allow them all.
}
}
};
HostnameVerifier trustAllHostnames = new HostnameVerifier() {
@Override
public boolean verify(String hostname, SSLSession session) {
return true; // Just allow them all.
}
};
try {
System.setProperty("jsse.enableSNIExtension", "false");
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCertificates, new SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
HttpsURLConnection.setDefaultHostnameVerifier(trustAllHostnames);
}
catch (GeneralSecurityException e) {
throw new ExceptionInInitializerError(e);
}
}
The Apache HttpComponents HttpClient is much more convenient in this all :)
If all you want is parsing and extracting data from HTML, then better use a HTML parser like Jsoup.
HTTP を扱う場合、ほとんどのHttpURLConnection場合、基本クラスよりも参照する方が便利ですURLConnection(なぜなら、HTTP URL でURLConnection要求すると返される抽象クラスだからです)。URLConnection.openConnection()
URLConnection#setDoOutput(true)次に、暗黙的にリクエストメソッドをPOSTに設定することに頼る代わりに、httpURLConnection.setRequestMethod("POST")より自然に感じるかもしれないものを行うことができます (また、 PUT 、 DELETE などの他のリクエストメソッドを指定することもできます)。
また、次のことができるように、便利な HTTP 定数も提供します。
int responseCode = httpURLConnection.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
Stack Overflowに関するこの質問やその他の質問に触発されて、ここにあるほとんどの手法を具体化した最小限のオープンソースの基本的なhttpクライアントを作成しました。
google-http-java-clientも優れたオープンソースリソースです。
kevinsawicki/http-requestのコードを確認することをお勧めします。基本的には、その上にあるラッパーであり、HttpUrlConnection今すぐリクエストを作成したい場合や、ソースを確認できる場合に備えて、はるかに単純な API を提供します (接続がどのように処理されるかを見てみましょう。
例:コンテンツ タイプといくつかのクエリ パラメータを使用してGETリクエストを作成します。application/json
// GET http://google.com?q=baseball%20gloves&size=100
String response = HttpRequest.get("http://google.com", true, "q", "baseball gloves", "size", 100)
.accept("application/json")
.body();
System.out.println("Response was: " + response);
当初、私はを支持するこの記事に惑わされましたHttpClient。
後で、この記事HttpURLConnectionから離れようとしていることに気付きました。
Googleブログによると：
Apache HTTP クライアントには、Eclair と Froyo のバグがほとんどありません。これは、これらのリリースに最適な選択です。Gingerbread の場合、HttpURLConnection が最適です。そのシンプルな API と小さなサイズにより、Android に最適です。
透過的な圧縮と応答キャッシュにより、ネットワークの使用量が削減され、速度が向上し、バッテリーが節約されます。新しいアプリケーションは HttpURLConnection を使用する必要があります。それは私たちが今後エネルギーを費やす場所です。
この記事と他のいくつかのスタック オーバー フローに関する質問を読んだ後、私はそれHttpURLConnectionがより長く続くと確信しています。
好意的な SE の質問の一部HttpURLConnections:
Android で、UrlEncodedFormEntity を使用せずに URL エンコードされたフォーム データを使用して POST 要求を行う
デフォルトで効率的な HTTP クライアントであるOkHttpもあります。
- HTTP/2 サポートにより、同じホストへのすべてのリクエストでソケットを共有できます。
- 接続プールにより、リクエストのレイテンシが短縮されます (HTTP/2 が使用できない場合)。
- 透過的な GZIP は、ダウンロード サイズを縮小します。
- 応答キャッシュは、繰り返し要求に対してネットワークを完全に回避します。
最初に のインスタンスを作成しますOkHttpClient:
OkHttpClient client = new OkHttpClient();
GET次に、リクエストを準備します。
Request request = new Request.Builder()
.url(url)
.build();
最後に、使用OkHttpClientして準備済みを送信しますRequest。
Response response = client.newCall(request).execute();
詳細については、OkHttp のドキュメントを参照してください。
jcabi-http (私は開発者です)JdkRequestから使用することもできます。これは、HttpURLConnection の装飾、HTTP 要求の起動、応答の解析など、すべての作業を自動的に行います。次に例を示します。
String html = new JdkRequest("http://www.google.com").fetch().body();
詳細については、このブログ投稿を確認してください: http://www.yegor256.com/2014/04/11/jcabi-http-intro.html
HTTP GETを使用している場合は、次の行を削除してください。
urlConnection.setDoOutput(true);
Java 11 ( Android を除く)を使用している場合は、従来のHttpUrlConnectionクラスの代わりに、Java 11 の新しいHTTP クライアント APIを使用できます。
var uri = URI.create("https://httpbin.org/get?age=26&isHappy=true");
var client = HttpClient.newHttpClient();
var request = HttpRequest
.newBuilder()
.uri(uri)
.header("accept", "application/json")
.GET()
.build();
var response = client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println(response.statusCode());
System.out.println(response.body());
非同期で実行された同じリクエスト:
var responseAsync = client
.sendAsync(request, HttpResponse.BodyHandlers.ofString())
.thenApply(HttpResponse::body)
.thenAccept(System.out::println);
// responseAsync.join(); // Wait for completion
var request = HttpRequest
.newBuilder()
.uri(uri)
.version(HttpClient.Version.HTTP_2)
.timeout(Duration.ofMinutes(1))
.header("Content-Type", "application/json")
.header("Authorization", "Bearer fake")
.POST(BodyPublishers.ofString("{ title: 'This is cool' }"))
.build();
var response = client.send(request, HttpResponse.BodyHandlers.ofString());
フォーム データを multipart ( multipart/form-data) または url-encoded ( ) 形式で送信する場合は、この解決策application/x-www-form-urlencodedを参照してください。
HTTP クライアント API の例と詳細については、この記事を参照してください。