I recently used HttpClient and looked at the official document: HttpClient implementations are expected to be thread safe. It is recommended that the same instance of this class is reused for multiple request executions. The translation means: the implementation of HttpClient is thread-safe and can reuse the same instance to execute multiple requests. When we encounter this description, we should think that HttpClient needs to be encapsulated. Since it is used spring boot, we will combine spring boot to encapsulate HttpClient.
1. Request retry handler (request retry processing)
In order to make the custom exception mechanism take effect, the HttpRequestRetryHandler interface needs to be implemented, the code is as follows:
import java.io.IOException; import java.io.InterruptedIOException; import java.net.UnknownHostException; import javax.net.ssl.SSLException; import javax.net.ssl.SSLHandshakeException; import org.apache.http.HttpEnclosingRequest; import org.apache.http.HttpRequest; import org.apache.http.NoHttpResponseException; import org.apache.http.client.HttpRequestRetryHandler; import org.apache.http.client.protocol.HttpClientContext; import org.apache.http.conn.ConnectTimeoutException; import org.apache.http.protocol.HttpContext; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; /** * Description: HttpClient's retry processing mechanism*/ @Configuration public class MyhttpRequestRetryHandler { @Value("${httpclient.config.retryTime}")// Here it is recommended to use @ConfigurationProperties(prefix="httpclient.config") method to facilitate reuse of private int retryTime; @Bean public HttpRequestRetryHandler httpRequestRetryHandler() { // Request retry final int retryTime = this.retryTime; return new HttpRequestRetryHandler() { public boolean retryRequest(IOException exception, int executionCount, HttpContext context) { // Do not retry if over max retry count, if the number of retrys exceeds retryTime, the request will no longer be retryed if (executionCount >= retryTime) { return false; } // The server breaks the client's connection exception if (exception instanceof NoHttpResponseException) { return true; } // time out timeout retry if (exception instanceof InterruptedIOException) { return true; } // Unknown host if (exception instanceof UnknownHostException) { return false; } // Connection refused if (exception instanceof ConnectTimeoutException) { return false; } // SSL handshake exception if (exception instanceof SSLException) { return false; } HttpClientContext clientContext = HttpClientContext.adapt(context); HttpRequest request = clientContext.getRequest(); if (!(request instanceof HttpEntityEnclosingRequest)) { return true; } return false; } }; } } 2. Pooling connection manager (connection pool management)
PoolingHttpClientConnectionManager is used to manage the client's connection pool and can provide services for requests from multiple threads. The code is as follows:
import org.apache.http.config.Registry; import org.apache.http.config.RegistryBuilder; import org.apache.http.conn.socket.ConnectionSocketFactory; import org.apache.http.conn.socket.LayeredConnectionSocketFactory; import org.apache.http.conn.socket.PlainConnectionSocketFactory; import org.apache.http.conn.ssl.SSLConnectionSocketFactory; import org.apache.http.impl.conn.PoolingHttpClientConnectionManager; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; @Configuration public class MyPoolingHttpClientConnectionManager { /** * Maximum number of connections in the connection pool*/ @Value("${httpclient.config.connMaxTotal}") private int connMaxTotal = 20; /** * */ @Value("${httpclient.config.maxPerRoute}") private int maxPerRoute = 20; /** * Connection survival time, unit is s */ @Value("${httpclient.config.timeToLive}") private int timeToLive = 60; @Bean public PoolingHttpClientConnectionManager poolingClientConnectionManager(){ PoolingHttpClientConnectionManager poolHttpcConnManager = new PoolingHttpClientConnectionManager(60, TimeUnit.SECONDS); // Maximum number of connections poolHttpcConnManager.setMaxTotal(this.connMaxTotal); // Route radix poolHttpcConnManager.setDefaultMaxPerRoute(this.maxPerRoute); return poolHttpcConnManager; } } Note: When the HttpClient instance is no longer needed and is about to be out of scope, it is important to close its connection manager to ensure that all connections that the manager keeps active are closed and free up the system resources allocated by those connections.
The constructor of the above PoolingHttpClientConnectionManager class is as follows:
public PoolingHttpClientConnectionManager(final long timeToLive, final TimeUnit tunit) { this(getDefaultRegistry(), null, null ,null, timeToLive, tunit); } private static Registry<ConnectionSocketFactory> getDefaultRegistry() { return RegistryBuilder.<ConnectionSocketFactory>create() .register("http", PlainConnectionSocketFactory.getSocketFactory()) .register("https", SSLConnectionSocketFactory.getSocketFactory()) .build(); } There are two maximum connections in the PoolingHttpClientConnectionManager configuration, which control the total maximum connection number and the maximum connection number per route respectively. If there is no explicit setting, by default, only up to 2 connections per route are allowed, and the total number of connections does not exceed 20. This value is not enough for many applications with high concurrency. The appropriate value must be set according to the actual situation. The idea is similar to the size of the thread pool. If all connection requests are to the same url, the value of MaxPerRoute can be set to be consistent with MaxTotal, so that the connection can be reused more efficiently
Special note: If you want to reuse a connection, you must make the system resources it occupies correctly released. The release method is as follows:
If you are using outputStream, you must ensure that the entire entity is written out. If it is an inputStream, you must remember to call inputStream.close() in the end. Or use EntityUtils.consume(entity) or EntityUtils.consumeQuietly(entity) to make the entity completely exhausted (the latter does not throw exceptions) to do this. There is a toString method in EntityUtils, which is also very convenient (the inputStream will be automatically closed in the end when calling this method, but in the actual test process, the connection will not be released), but it can only be used if it can be determined that the received entity is not particularly large. If the entire entity is not fully consumed, the connection cannot be reused, and soon the available connection timeout or blocks here will not be obtained in the connection pool (because the state of the connection will always be spared, that is, the state being used). Therefore, if you want to reuse the connection, you must remember to consume the entity fully. As long as the eof of the stream is detected, the releaseConnection method of the ConnectionHolder will be automatically called for processing.
3. Connection keep alive strategy
The HTTP specification does not specify how long a persistent connection may and should remain alive. Some HTTP servers use non-standard Keep-Alive headers to communicate to clients the period in seconds when they intend to keep the connection on the server side. HttpClient can use this information. If the Keep-Alive header does not exist in the response, HttpClient assumes that the connection can remain active indefinitely. However, many HTTP servers generally used are configured to delete persistent connections after a period of inactive status in order to save system resources without notifying the client. If the default policy is too optimistic, you may need to provide a custom keep-alive policy, the code is as follows:
import org.apache.http.HeaderElement; import org.apache.http.HeaderElementIterator; import org.apache.http.HttpResponse; import org.apache.http.conn.ConnectionKeepAliveStrategy; import org.apache.http.message.BasicHeaderElementIterator; import org.apache.http.protocol.HTTP; import org.apache.http.protocol.HttpContext; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; /** * Description: Connection Keeping Policy* @author chhliu */ @Configuration public class MyconnectionKeepAliveStrategy { @Value("${httpclient.config.keepAliveTime}") private int keepAliveTime = 30; @Bean("connectionKeepAliveStrategy") public ConnectionKeepAliveStrategy connectionKeepAliveStrategy() { return new ConnectionKeepAliveStrategy() { public long getKeepAliveDuration(HttpResponse response, HttpContext context) { // Honor 'keep-alive' header HeaderElementIterator it = new BasicHeaderElementIterator( response.headerIterator(HTTP.CONN_KEEP_ALIVE)); while (it.hasNext()) { HeaderElement he = it.nextElement(); String param = he.getName(); String value = he.getValue(); if (value != null && param.equalsIgnoreCase("timeout")) { try { return Long.parseLong(value) * 1000; } catch (NumberFormatException ignore) { } } } return 30 * 1000; } }; } } Note: Long connections are not used in all situations, especially nowadays, most of the systems are deployed on multiple servers and have load balancing functions. If we keep a long connection when accessing, once that server is suspended, it will affect the client. At the same time, we cannot fully utilize the load balancing characteristics of the server. Instead, short connections are more beneficial. These need to be determined according to specific needs, rather than summarizing them in one sentence.
4. HttpClient proxy configuration (proxy configuration)
Used to configure the proxy, the code is as follows:
import org.apache.http.HttpHost; import org.apache.http.impl.conn.DefaultProxyRoutePlanner; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; /** * Description: HttpClient proxy* @author chhliu */ @Configuration public class MyDefaultProxyRoutePlanner { // The proxy's host address @Value("${httpclient.config.proxyhost}") private String proxyHost; // The proxy's port number @Value("${httpclient.config.proxyPort}") private int proxyPort = 8080; @Bean public DefaultProxyRoutePlanner defaultProxyRoutePlanner(){ HttpHost proxy = new HttpHost(this.proxyHost, this.proxyPort); return new DefaultProxyRoutePlanner(proxy); } } HttpClient not only supports simple direct connections, complex routing policies and proxying. HttpRoutePlanner is based on the http context, the client-to-server routing computing policy. Generally, if there is no proxy, you don't need to set this thing. There is a very critical concept here - Route: In HttpClient, a Route refers to a line of the running environment machine-> target machine host, that is, if the host of the target url is the same, then their route is the same.
5. RequestConfig
The various configurations used to set the request, the code is as follows:
import org.apache.http.client.config.RequestConfig; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; @Configuration public class MyRequestConfig { @Value("${httpclient.config.connectTimeout}") private int connectTimeout = 2000; @Value("${httpclient.config.connectRequestTimeout}") private int connectRequestTimeout = 2000; @Value("${httpclient.config.socketTimeout}") private int socketTimeout = 2000; @Bean public RequestConfig config(){ return RequestConfig.custom() .setConnectionRequestTimeout(this.connectRequestTimeout) .setConnectTimeout(this.connectTimeout) .setSocketTimeout(this.socketTimeout) .build(); } } RequestConfig is some configuration of request. There are three timeout times that are more important. By default, all three timeout times are 0 (if the request's Config is not set, the default parameters will be set in HttpClientParamConfig's getRequestConfig during execution). This means infinite waiting, which can easily cause all requests to block and wait indefinitely in this place. These three timeouts are:
a. connectionRequestTimeout—Timeout for getting connection from the connection pool
This time defines the timeout time for retrieving the connection from the connection pool managed by ConnectionManager. If there is no available connection in the connection pool, the request will be blocked, and the maximum time to wait for connectionRequestTimeout. If it has not been served, a ConnectionPoolTimeoutException exception is thrown and no further waiting is allowed.
b. connectTimeout—connection timeout time
This time defines the timeout for establishing a connection with the server through the network, that is, the connection waiting time to connect to the target URL after obtaining a connection in the connection pool. A timeout occurs, a ConnectionTimeoutException exception will be thrown.
c. socketTimeout—request timeout time
This time defines the timeout time for socket reading data, that is, the time it takes to wait after connecting to the server to obtain response data from the server, or the time it takes to get the response after connecting to the previous URL. A timeout occurs and a SocketTimeoutException exception will be thrown.
6. Instantiate HttpClient
The HttpClient is instantiated by implementing FactoryBean, the code is as follows:
import org.apache.http.client.HttpRequestRetryHandler; import org.apache.http.client.config.RequestConfig; import org.apache.http.impl.conn.ConnectionKeepAliveStrategy; import org.apache.http.impl.client.CloseableHttpClient; import org.apache.http.impl.client.HttpClients; import org.apache.http.impl.conn.DefaultProxyRoutePlanner; import org.apache.http.impl.conn.PoolingHttpClientConnectionManager; import org.springframework.beans.factory.DisposableBean; import org.springframework.beans.factory.FactoryBean; import org.springframework.beans.factory.InitializingBean; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Service; /** * Description: HttpClient client encapsulation*/ @Service("httpClientManagerFactoryBen") public class HttpClientManagerFactoryBen implements FactoryBean<CloseableHttpClient>, InitializingBean, DisposableBean { /** * Target object generated by FactoryBean*/ private CloseableHttpClient client; @Autowired private ConnectionKeepAliveStrategy connectionKeepAliveStrategy; @Autowired private HttpRequestRetryHandler httpRequestRetryHandler; @Autowired private DefaultProxyRoutePlanner proxyRoutePlanner; @Autowired private PoolingHttpClientConnectionManager poolHttpcConnManager; @Autowired private RequestConfig config; // When destroying the context, destroying the HttpClient instance @Override public void destroy() throws Exception { /* * Calling httpClient.close() will first shut down connection manager, and then release all resources occupied by the HttpClient, * Close all connections that are in use or idle, including the underlying socket. Since the connection manager it uses is closed here, * So when you have to make an http request next time, you have to renew a connection manager to build an HttpClient, * That is, when you need to close and create a new client, the connection manager cannot be a singleton. */ if(null != this.client){ this.client.close(); } } @Override// Initialize the instance public void afterPropertiesSet() throws Exception { /* * It is recommended to use HttpClients.custom to create HttpClientBuilder here, instead of using HttpClientBuilder.create() method to create HttpClientBuilder * From the official documentation, HttpClientBuilder is non-thread-safe, but HttpClients is indeed Imutable. The immutable object can not only ensure that the state of the object is not changed, * but also can be shared by other threads without using the lock mechanism*/ this.client = HttpClients.custom().setConnectionManager(poolHttpcConnManager) .setRetryHandler(httpRequestRetryHandler) .setKeepAliveStrategy(connectionKeepAliveStrategy) .setRoutePlanner(proxyRoutePlanner) .setDefaultRequestConfig(config) .build(); } // Return the type of the instance @Override public CloseableHttpClient getObject() throws Exception { return this.client; } @Override public Class<?> getObjectType() { return (this.client == null ? CloseableHttpClient.class : this.client.getClass()); } // The built instance is a singleton@Override public boolean isSingleton() { return true; } }7. Add configuration files
# proxy's host httpclient.config.proxyhost=xxx.xx.xx.xx # proxy port httpclient.config.proxyPort=8080 # connection timeout or exception retry times httpclient.config.retryTime=3 # long connection hold time, unit is s httpclient.config.keepAliveTime=30 # maximum number of connections in the connection pool httpclient.config.connMaxTotal=20 httpclient.config.maxPerRoute=20 # connection timeout, unit ms httpclient.config.connectTimeout=20 # connection timeout, unit is ms httpclient.config.connectTimeout=20 # connection pool Request timeout httpclient.config.connectRequestTimeout=2000 # sock timeout httpclient.config.socketTimeout=2000 # Connection survival time, unit s httpclient.config.timeToLive=60
8. Test
The test code is as follows:
import java.io.IOException; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import javax.annotation.Resource; import org.apache.http.Consts; import org.apache.http.ParseException; import org.apache.http.client.ClientProtocolException; import org.apache.http.client.methods.CloseableHttpResponse; import org.apache.http.client.methods.HttpGet; import org.apache.http.impl.client.CloseableHttpClient; import org.apache.http.util.EntityUtils; import org.junit.Test; import org.junit.runner.RunWith; import org.springframework.boot.test.context.SpringBootTest; import org.springframework.test.context.junit4.SpringRunner; @RunWith(SpringRunner.class) @SpringBootTest public class HttpClientManagerFactoryBenTest { // Inject HttpClient instance @Resource(name = "httpClientManagerFactoryBen") private CloseableHttpClient client; @Test public void test() throws ClientProtocolException, IOException, InterruptedException{ ExecutorService service = Executors.newFixedThreadPool(2); for(int i=0; i<10; i++){ service.submit(new Runnable() { @Override public void run() { System.out.println("the current thread is:"+Thread.currentThread().getName()); HttpEntity entity = null; try { HttpGet get = new HttpGet("https://localhost:8080/testjson"); // Submit the request through httppclient's execution, and use CloseableHttpResponse to accept the return information CloseableHttpResponse response = client.execute(get); System.out.println("client object:"+client); entity = response.getEntity(); System.out.println("======================================================================================; EntityUtils.consumeQuietly(entity);// Release the connection} catch (ClientProtocolException e) { e.printStackTrace(); } catch (ParseException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } finally{ if(null != entity){// Release the connection EntityUtils.consumeQuietly(entity); } } } } }); } Thread.sleep(60000); } } Through the above steps, the encapsulation of HttpClient is basically completed. If you need to be more detailed, you can gradually improve the HttpClient as HttpClientTemplate according to the above ideas, because CloseableHttpClient uses a callback mechanism internally, which is similar to JdbcTemplate or RedisTemplate until the service can be provided in the form of spring boot starter.
The above is all the content of this article. I hope it will be helpful to everyone's learning and I hope everyone will support Wulin.com more.