Recently, Elasticsearch 5.4 (ES) is a relatively new version. There are many problems during use, which is a headache, but the problem was finally solved.
Question 1: ESClient is retrieved slowly and cannot be retrieved: failed to create a child event loop
Because the business needs to upload a batch of files without uploading an ES index, you have to get a connection and then operate each time you add an index. Especially in large batches, the number of acquisitions is obviously very large. The main reason for this problem is that we frequently operate ES in loops. For example, we need to get 100 files in a batch. In order to reduce the time of obtaining ES Client, we finally adopted a solution, that is, initialize the connection when the service starts, obtain it at one time, and then call it directly later. After the entire batch file is uploaded, the ES index is finally added, instead of adding one file at a time. This method obviously does not require each batch to obtain connections, which greatly improves execution efficiency.
First, when the service is started, we initialize the static ES Client in the startup class:
private static ElasticSearchUtil ElasticSearchUtil=new ElasticSearchUtil(); public static TransportClient client=ElasticSearchUtil.getClient();
Then call it directly when used:
Client client=Main.client;
This can greatly reduce the number of connections to the ES Client, thereby improving efficiency.
The ES code is as follows:
public TransportClient getClient() {String[] ipArr = configUtil.getValue("ESIP").split(",");Settings settings = Settings.builder().put("thread_pool.generic.core",5) .put("thread_pool.generic.max", 10) .put("processors", 5) .put(Constants.ESCLUSTERNAME,configUtil.getValue("clusterName")).build();TransportClient client = new PreBuiltTransportClient(settings); for (String ip : ipArr) {TransportAddress address = new InetSocketTransportAddress (InetAddresses.forString(ip),9300);client.addTransportAddresses(address);} return client;}Question 2: Memory overflow: java.lang.OutOfMemory:unable to create new native thread
During the project development process, memory overflow is a very troublesome thing. I encountered it during the use of ES, and it was very frequent, especially during large-scale stress tests. I thought of many ways to optimize the jvm memory, but there was no effect, and the problem was still not solved. Finally, when looking at the source code, I found a reason. Combined with error reporting exceptions, this is because a large number of threads were automatically created during use with ES, which exceeded the system's capacity, which led to memory overflow. When studying the source code, I found that the number of threads created by ES can be controlled through settings. Here is the default number of ES creation threads:
thread_pool.generic.core=default value---4thread_pool.generic.max=default value--min(512,max(4*processor number, 128))processor number=CPU processor number
Our CPU is 10 cores and 40 threads
Judging from the calculation results, if the default value is used, the number of threads that ES can create is a large value, which is far beyond the capacity of the system itself. It mainly adjusts the setting value. After adjustment, we change the default value of ES as follows:
Settings settings = Settings.builder().put("thread_pool.generic.core",5).put("thread_pool.generic.max", 10).put("processors", 5) .put(Constants.ESCLUSTERNAME,configUtil.getValue("clusterName")).build(); This is the previous Settings settings = Settings.builder().put("thread_pool.generic.core",5).put(Constants.ESCLUSTERNAME,configUtil.getValue("clusterName")).build();After testing, ES created a very small number of threads and met our development needs, and there was no memory overflow problem again.
The above summary of common questions based on Elasticsearch 5.4 is all the content I share with you. I hope you can give you a reference and I hope you can support Wulin.com more.