Detailed explanation of the implementation of Https protocol and HttpClient

1. Background

HTTP is a public protocol that allows for readability of transferring content, and data from the client and server are completely transmitted through plaintext. Under this background, the entire Internet data that relies on the Http protocol is transparent, which brings great data security risks. There are two ideas to solve this problem:

• The C/S ends are responsible for each, that is, the client and the server communicate on Http using negotiated encrypted content.
• The C/S side is not responsible for encryption and decryption, and the encryption and decryption are left to the communication protocol itself to resolve.

The first type has a wider range of applications in reality than imagined. The two parties exchange keys offline, and the client uses a ciphertext when sending data, which is transmitted on the Internet through the transparent Http protocol. After receiving the request, the server decrypts and obtains the plain text in the agreed manner. Even if this kind of content is hijacked, it doesn't matter, because third parties don't know their encryption and decryption methods. However, this approach is too special, and both the client and the server need to care about this special encryption and decryption logic.

The second type of C/S side may not care about the special logic above. They believe that the sending and receiving are plaintext because the encryption and decryption part has been processed by the protocol itself.

Judging from the results, there seems to be no difference between the two solutions, but from the perspective of software engineers, the difference is very huge. Because the first type requires the business system to develop the encryption and decryption function for the response, and the offline interactive key is required, the second type does not have the development volume.

HTTPS is the most popular form of security for HTTP, first created by NetScape. In HTTPS, URLs start with https:// instead of http://. When HTTPS is used, all HTTP requests and responses are encrypted before being sent to the network, which is implemented at the SSL layer.

2. Encryption method

The plain text data is encrypted through the SSL layer and then sent to the Internet to transmit, which solves the original data security problem of the HTTP protocol. Generally speaking, the methods of encrypting data are divided into symmetric encryption and asymmetric encryption.

2.1 Symmetric encryption

Symmetric encryption refers to the use of the same key as encryption and decryption. Common algorithms include DES and AES, etc., and the algorithm time is related to the key length.

The biggest disadvantage of symmetric keys is that they need to maintain a large number of symmetric keys and require offline exchange. To join a network with n entities, n(n-1) keys are required.

2.2 Asymmetric encryption

Asymmetric encryption refers to encryption methods based on public/private keys. Common algorithms include RSA, which generally slower encryption than symmetric encryption.

Symmetric encryption has one more step than asymmetric encryption, that is, to obtain the server public key, rather than the key maintained by each.

The entire encryption algorithm is based on a certain number theory, and the effect is that the encryption result is irreversible. That is, only through the private key can the ciphertext encrypted by the public key be decrypted.

Under this algorithm, the number of keys in the entire network is greatly reduced, and each person only needs to maintain a pair of company keys. That is, in the network of n entities, the number of keys is 2n.

The disadvantage is that it runs slowly.

2.3 Hybrid encryption

There is a scene in Stephen Chow's movie "God of Cookery" where the underworld is in a hot spot, arguing about the issue of chassis division between pee shrimp and beef balls. The God of Food said, "It's really troublesome. It's mixed together to make a peeing beef ball, stupid!"

The advantage of symmetric encryption is its fast speed, and the disadvantage is that it requires key exchange. The advantage of asymmetric encryption is that it does not require an interactive key, and the disadvantage is that it is slow speed. Just mix it together and use it well.

Hybrid encryption is exactly the encryption method used by the HTTPS protocol. The symmetric key is exchanged first through asymmetric encryption, and then the data is transmitted through the symmetric key.

Since the amount of data transmission is much larger than the amount of data used for exchanging keys in the early stage of establishing a connection, the performance impact of asymmetric encryption can be basically negligible, and at the same time improves efficiency.

3. HTTPS handshake

It can be seen that based on the original HTTP protocol, HTTPS has added security layer processing:

• The client exchanges certificates and verifies the identity. In reality, the server rarely verifies the client's certificates.
• Negotiate the version and algorithm of the encryption protocol. There may be a version mismatch here, resulting in failure.
• Negotiate symmetric keys, a process that uses asymmetric encryption
• Use the encryption algorithm in 2 to encrypt the plaintext sent by HTTP to obtain the ciphertext
• TCP layer transmits normally, and has no sense of HTTPS

4. HttpClient's support for HTTPS protocol

4.1 Obtain SSL connection factory and domain name verification device

As a software engineer, what we care about is how the "HTTPS protocol" is implemented in code? To explore the mystery of HttpClient source code, everything starts with HttpClientBuilder.

 public CloseableHttpClient build() { //Omit some code HttpClientConnectionManager connManagerCopy = this.connManager; //If the connection pool manager is specified, use the specified one, otherwise create a new default if (connManagerCopy == null) { LayeredConnectionSocketFactory sslSocketFactoryCopy = this.sslSocketFactory; if (sslSocketFactoryCopy == null) { //If the use environment variable is enabled, the https version and password control read final String[] supportedProtocols = systemProperties ? split( System.getProperty("https.protocols")) : null; final String[] supportedCipherSuites = systemProperties ? split( System.getProperty("https.cipherSuites")) : null; //If not specified, use the default domain name validator, and it will verify whether the domain name matches the certificate returned by the server in the ssl session HostnameVerifier hostnameVerifierCopy = this.hostnameVerifier; if (hostnameVerifierCopy == null) { hostnameVerifierCopy = new DefaultHostnameVerifier(publicSuffixMatcherCopy); } //If SslContext is formulated, a customized SSL connection factory is generated, otherwise the default connection factory is used if (sslContext != null) { sslSocketFactoryCopy = new SSLConnectionSocketFactory(sslContext, supportedProtocols, supportedCipherSuites, hostnameVerifierCopy); } else { if (systemProperties) { sslSocketFactoryCopy = new SSLConnectionSocketFactory( (SSLSocketFactory) SSLSocketFactory.getDefault(), supportedProtocols, supportedCipherSuites, hostnameVerifierCopy); } else { sslSocketFactoryCopy = new SSLConnectionSocketFactory( SSLContexts.createDefault(), hostnameVerifierCopy); } } } //Register the Ssl connection factory in the connection pool manager. When an Https connection is needed, the SSL connection will be produced based on the above SSL connection factory @SuppressWarnings("resource") final PoolingHttpClientConnectionManager poolingmgr = new PoolingHttpClientConnectionManager( RegistryBuilder.<ConnectionSocketFactory>create() .register("http", PlainConnectionSocketFactory.getSocketFactory()) .register("https", sslSocketFactoryCopy) .build(), null, null, dnsResolver, connTimeToLive, connTimeToLiveTimeUnit != null ? connTimeToLiveTimeUnit: TimeUnit.MILLISECONDS); //Omit some codes}}

The above code creates an SSLConnectionSocketFactory SSLConnectionSocketFactory and registers it in the connection pool manager for later production of SSL connections. Reference for connection pooling: http://www.VeVB.COM/article/141015.htm

Here, several key components are used when configuring SSLConnectionSocketFactory, the domain name validator HostnameVerifier and the context SSLContext.

Among them, HostnameVerifier is used to verify whether the server certificate matches the domain name. There are many implementations. DefaultHostnameVerifier uses the default verification rules, replacing the BrowserCompatHostnameVerifier and StrictHostnameVerifier in the previous version. NoopHostnameVerifier replaces AllowAllHostnameVerifier, using a strategy of not verifying the domain name.

Note that there are some differences here, BrowserCompatHostnameVerifier can match multi-level subdomain names, and "*.foo.com" can match "abfoo.com". StrictHostnameVerifier cannot match multi-level subdomain names, only to "a.foo.com".

After 4.4, HttpClient used the new DefaultHostnameVerifier to replace the above two strategies, and only retained one strict strategy and StrictHostnameVerifier. Because strict strategies are the strategies of IE6 and JDK itself, non-strict strategies are the strategies of curl and firefox. That is, the default HttpClient implementation does not support multi-level subdomain name matching strategy.

SSLContext stores key information related to the key, which is directly related to the business and is very important. This is to be analyzed separately later.

4.2 How to get an SSL connection

How to obtain a connection from a connection pool? This process has been analyzed in previous articles. I will not analyze it here. Please refer to the connection://www.VeVB.COM/article/141015.htm.

After obtaining a connection from the connection pool, if the connection is not in the established state, you need to establish the connection first.

The code of the DefaultHttpClientConnectionOperator part is:

 public void connect( final ManagedHttpClientConnection conn, final HttpHost host, final InetSocketAddress localAddress, final int connectTimeout, final SocketConfig socketConfig, final HttpContext context) throws IOException { // Previously, https's connection pool implementation was registered in HttpClientBuilder. Here, lookup obtains the implementation of Https, that is, SSLConnectionSocketFactory final Lookup<ConnectionSocketFactory> registry = getSocketFactoryRegistry(context); final ConnectionSocketFactory sf = registry.lookup(host.getSchemeName()); if (sf == null) { throw new UnsupportedSchemeException(host.getSchemeName() + " protocol is not supported"); } //If the address is in the form of IP, it can be used directly, otherwise use the dns parser to parse the ip corresponding to the domain name to obtain the final InetAddress[] addresses = host.getAddress() != null ? new InetAddress[] { host.getAddress() } : this.dnsResolver.resolve(host.getHostName()); final int port = this.schemePortResolver.resolve(host); //A domain name may correspond to multiple Ip, and try to connect in order for (int i = 0; i < addresses.length; i++) { final InetAddress address = addresses[i]; final boolean last = i == addresses.length - 1; //This is just a socket generated, and there is no connection to Socket sock = sf.createSocket(context); //Set some tcp layer parameters sock.setSoTimeout(socketConfig.getSoTimeout()); sock.setReuseAddress(socketConfig.isSoReuseAddress()); sock.setTcpNoDelay(socketConfig.isTcpNoDelay()); sock.setKeepAlive(socketConfig.isSoKeepAlive()); if (socketConfig.getRcvBufSize() > 0) { sock.setReceiveBufferSize(socketConfig.getRcvBufSize()); } if (socketConfig.getSndBufSize() > 0) { sock.setSendBufferSize(socketConfig.getSndBufSize()); } final int linger = socketConfig.getSoLinger(); if (linger >= 0) { sock.setSoLinger(true, linger); } conn.bind(sock); final InetSocketAddress remoteAddress = new InetSocketAddress(address, port); if (this.log.isDebugEnabled()) { this.log.debug("Connecting to " + remoteAddress); } try { // Establish a connection through SSLConnectionSocketFactory and bind to conn sock = sf.connectSocket( connectTimeout, sock, host, remoteAddress, localAddress, context); conn.bind(sock); if (this.log.isDebugEnabled()) { this.log.debug("Connection established " + conn); } return; } //Omit some code} }

In the above code, we see that the preparation work is before establishing an SSL connection. This is a general process, and the same is true for ordinary HTTP connections. Where does the special process of SSL connection reflect?

The source code of SSLConnectionSocketFactory is as follows:

 @Override public Socket connectSocket( final int connectTimeout, final Socket socket, final HttpHost host, final InetSocketAddress remoteAddress, final InetSocketAddress localAddress, final HttpContext context) throws IOException { Args.notNull(host, "HTTP host"); Args.notNull(remoteAddress, "Remote address"); final Socket sock = socket != null ? socket : createSocket(context); if (localAddress != null) { sock.bind(localAddress); } try { if (connectTimeout > 0 && sock.getSoTimeout() == 0) { sock.setSoTimeout(connectTimeout); } if (this.log.isDebugEnabled()) { this.log.debug("Connecting socket to " + remoteAddress + " with timeout " + connectTimeout); } //Create connection sock.connect(remoteAddress, connectTimeout); } catch (final IOException ex) { try { sock.close(); } catch (final IOException ignore) { } throw ex; } // If it is currently SslSocket, perform SSL handshake and domain name verification if (sock instanceof SSLSocket) { final SSLSocket sslsock = (SSLSocket) sock; this.log.debug("Starting handshake"); sslsock.startHandshake(); verifyHostname(sslsock, host.getHostName()); return sock; } else { //If it is not SslSocket, wrap it as SslSocket return createLayeredSocket(sock, host.getHostName(), remoteAddress.getPort(), context); } } @Override public Socket createLayeredSocket( final Socket socket, final String target, final int port, final HttpContext context) throws IOException { //Wrap a normal socket as SslSocket, socketfactory is generated based on SSLContext in HttpClientBuilder, which contains the key information final SSLSocket sslsock = (SSLSocket) this.socketfactory.createSocket( socket, target, port, true); //If the SSL layer protocol version and encryption algorithm are formulated, use the specified one, otherwise the default if (supportedProtocols != null) { sslsock.setEnabledProtocols(supportedProtocols); } else { // If supported protocols are not explicitly set, remove all SSL protocol versions final String[] allProtocols = sslsock.getEnabledProtocols(); final List<String> enabledProtocols = new ArrayList<String>(allProtocols.length); for (final String protocol: allProtocols) { if (!protocol.startsWith("SSL")) { enabledProtocols.add(protocol); } } if (!enabledProtocols.isEmpty()) { sslsock.setEnabledProtocols(enabledProtocols.toArray(new String[enabledProtocols.size()])); } } if (supportedCipherSuites != null) { sslsock.setEnabledCipherSuites(supportedCipherSuites); } if (this.log.isDebugEnabled()) { this.log.debug("Enabled protocols: " + Arrays.asList(sslsock.getEnabledProtocols())); this.log.debug("Enabled cipher suites:" + Arrays.asList(sslsock.getEnabledCipherSuites())); } prepareSocket(sslsock); this.log.debug("Starting handshake"); //Ssl connection handshake sslsock.startHandshake(); //After the handshake is successful, check whether the returned certificate is consistent with the domain name verifyHostname(sslsock, target); return sslsock; }

As can be seen, for an SSL communication. First, establish a normal socket connection, then perform an SSL handshake, and then verify the consistency of the certificate and the domain name. The subsequent operation is to communicate through SSLSocketImpl. The protocol details are reflected in the SSLSocketImpl class, but this part of the code jdk is not open source. Those who are interested can download the corresponding openJdk source code to continue analysis.

5. Summary of this article

1. The https protocol is a secure version of http, which ensures the security of the transmission layer data, but it consumes additional consumption for the server CPU.
2. https protocol uses asymmetric encryption when negotiating keys, and uses symmetric encryption after the key negotiation is completed
3. In some scenarios, even if the encryption and decryption is performed through https, the service system will encrypt and sign the packets twice.
4. When HttpClient is built, the connection pool manager registers two SslSocketFactory to match http or https strings
5. The principle of socket establishment corresponding to https is to first establish it and then verify the consistency of the domain name and certificate
6. SSL layer encryption and decryption is done by jdk itself, and no additional operations are required for httpClient.

Okay, the above is the entire content of this article. I hope that the content of this article has certain reference value for everyone's study or work. If you have any questions, you can leave a message to communicate. Thank you for your support to Wulin.com.