A series of pits between @Scheduled and HttpClient in Spring

Preface

This article mainly introduces the pitfalls of @Scheduled and HttpClient in Spring. We will share them for your reference and learning. I won’t say much below, let’s take a look at the detailed introduction together.

I once stepped on a big pit:

Due to the particularity of the business, many timed tasks will be run regularly and compensated for business data.

During Spring usage, we can use the @Scheduled annotation to easily implement timing tasks.

One morning I suddenly realized that from a certain moment the night before, all the timed tasks were stuck and stopped running.

@ScheduledDefault single thread

After investigation, it was found that if we use @Scheduled annotation to explain the default configuration, all tasks will be run by a single thread. After writing a test task to sleep, it is easy to find that all other tasks are not triggered when the time comes.

If you need to enable multi-threading, you need to perform the following configuration and set the number of threads:

 @Configurationpublic class ScheduleConfig implements SchedulingConfigurer { @Override public void configureTasks(ScheduledTaskRegistrar taskRegistrar) { taskRegistrar.setScheduler(Executors.newScheduledThreadPool(5)); }}

This solves the problem that if a task is stuck, it will cause all tasks to be stuck.

But why are there tasks stuck?

HttpClient default parameter configuration

It turns out that some tasks will request the restful interface of external services regularly, and the configuration of HttpClient is as follows:

 PoolingHttpClientConnectionManager connManager = new PoolingHttpClientConnectionManager(); connManager.setMaxTotal(maxConnection); httpClient = HttpClients.custom() .setConnectionManager(connManager) .build();

When I first used HttpClient, I didn’t think so much about it, and I basically used the default configuration.

Tracking the source code can find that when configuring using the above method, the timeout time of HttpClient is actually -1, which means that if there is a problem with the other party's service, the request of HttpClient will never time out and will wait. The source code is as follows:

 Builder() { super(); this.staleConnectionCheckEnabled = false; this.redirectsEnabled = true; this.maxRedirects = 50; this.relativeRedirectsAllowed = true; this.authenticationEnabled = true; this.connectionRequestTimeout = -1; this.connectTimeout = -1; this.socketTimeout = -1; this.contentCompressionEnabled = true;}

So we must manually specify the timeout time at this time, and the problem is solved. For example:

 PoolingHttpClientConnectionManager connManager = new PoolingHttpClientConnectionManager(); connManager.setMaxTotal(maxConnection); RequestConfig defaultRequestConfig = RequestConfig.custom() .setSocketTimeout(3000) .setConnectTimeout(3000) .setConnectionRequestTimeout(3000) .build(); httpClient = HttpClients.custom() .setDefaultRequestConfig(defaultRequestConfig) .setConnectionManager(connManager) .build();

Remind of another problem

In fact, another configuration problem encountered during the use of HttpClient, which is the defaultMaxPerRoute parameter.

I didn't pay attention to this parameter when I first used it, but I just set the maximum number of connections in the connection pool maxTotal.

The defaultMaxPerRoute parameter actually represents the maximum number of connections per route. For example, your system needs to access two other services: google.com and bing.com. If your maxTotal is set to 100 and defaultMaxPerRoute is set to 50, then the maximum number of requests for each service can be 50.

So if defaultMaxPerRoute is not set, track the source code:

 public PoolingHttpClientConnectionManager( final HttpClientConnectionOperator httpClientConnectionOperator, final HttpConnectionFactory<HttpRoute, ManagedHttpClientConnection> connFactory, final long timeToLive, final TimeUnit tunit) { super(); this.configData = new ConfigData(); //The CPool constructor method used here, the second parameter is defaultMaxPerRoute, which is the default is 2. this.pool = new CPool(new InternalConnectionFactory( this.configData, connFactory), 2, 20, timeToLive, tunit); this.pool.setValidateAfterInactivity(2000); this.connectionOperator = Args.notNull(httpClientConnectionOperator, "HttpClientConnectionOperator"); this.isShutDown = new AtomicBoolean(false);}

It was found here that the default value was only 2. No wonder there was always a timeout in high concurrency situations at that time, maxTotal was clearly set very high.

So if your service accesses many different external services and has a large concurrency, you must configure the two parameters maxTotal and defaultMaxPerRoute.

So when you use any new things later, you will have a good look at what configurations you have. If you have any questions, you must check it first. Don’t copy a piece of code online and use it directly. It might be fine at that time, but maybe I'll be cheated in the future.

Summarize

The above is the entire content of this article. I hope that the content of this article has certain reference value for everyone's study or work. If you have any questions, you can leave a message to communicate. Thank you for your support to Wulin.com.