OkHttp source code reading guide

OkHttp source code reading guide

The content of this article:

  1. OkHttp process guide
  2. The role of each interceptor
  3. Implementation of Https Dns
  4. Implementation of Http1.1 and HTTP2
  5. The role and realization of the connection pool

OkHttp process

Class Diagram

The class diagram here only draws the class diagram in the chain of responsibility pattern process, and the more important network UML diagram will be in the ConnectInterceptor below

flow chart

This flow chart only involves a simple process, not specific details. What is highlighted is the responsibility chain model. This model is suitable and hierarchical, but there is a dependency relationship between top and bottom. This model can be found in many places. Similar examples, such as: object initialization (C++ is recommended here), input event transmission and processing, five-layer network protocol, basically all are passed from the outside to the inside and processed from the inside to the outside

Initialization of OkHttp

Let's first look at the use of OkHttp

URL url = new URL( "http://www.baidu.com/" ); OkHttpClient client = new OkHttpClient.Builder() .build(); Request request = new Request.Builder() .url(url.toString()) .build(); okhttp3.Call call = client.newCall(request); okhttp3.Response response = call.execute(); Copy code
  1. First of all, in the construction method of OkHttpClient.Builder, many default values are initialized. If the default values are not found, most of them can be obtained from here
  2. In OkHttpClient, another SSLSocketFactory is created to support TLS or SSL

RealCall initialization

use

OkHttpClient.newCall()
RealCall is created and another important class Transmitter will be created. According to the official comment, this class can connect okhttp and the network layer (in fact, the http layer)

RealCall.execute()

  1. transfer
    Dispatcher.executed()
    Add the call to the executed Call queue. The main purpose of this step is to control the start, end and cancel of the Call, but because it is a synchronous request, it is already started at this time.
  2. transfer
    getResponseWithInterceptorChain()
    Start sending a request to the server and get a response

RealCall.getResponseWithInterceptorChain()

  1. Add a custom interceptor, the so-called interceptor, in fact, can do some extra things before sending and after receiving, at most it is Logging, but I think it is also a good choice to use EventListener, in fact, it can also be put into each request Do it inside, but the code will repeat a lot
  2. Add RetryAndFollowUpInterceptor, BridgeInterceptor, CacheInterceptor, ConnectInterceptor, custom networkInterceptor (for the time being, I did not expect the corresponding application scenario),
  3. Create a RealInterceptorChain object, and then call
    RealInterceptorChain.proceed()
    Method, call each interceptor processing logic in turn

RealInterceptorChain.proceed()

  1. Check whether the custom interceptor is legal
  2. Create a new RealInterceptorChain, pay attention to index+1
  3. Get the current interceptor of the index and call
    interceptor.intercept()
    The method is handed over to the Interceptor to process, get the return value, and think about why you can t use a loop
  4. Require all custom interceptors to be called
    RealInterceptorChain.proceed()
    , Otherwise the request will not be sent
  5. Determine whether response or response.body() is empty

The role of interceptors

First review the five-layer network model, http is at the application layer, which means that the http protocol is implemented by the application, and finally transmitted to the TCP transport layer through the socket. Let s take a look at how to send http requests if the okhttp framework is not used.

String path = "http://www.baidu.com/" ; String host = "www.baidu.com" ; Socket socket = null ; OutputStreamWriter streamWriter = null ; BufferedWriter bufferedWriter = null ; try { socket = new Socket(host, 80 ); streamWriter = new OutputStreamWriter(socket.getOutputStream()); bufferedWriter = new BufferedWriter(streamWriter); bufferedWriter.write( "GET " + path + " HTTP/1.1\r\n" ); bufferedWriter.write( "Host: www.baidu.com\r\n" ); bufferedWriter.write( "\r\n" ); bufferedWriter.flush(); BufferedReader in = new BufferedReader( new InputStreamReader(socket.getInputStream(), "UTF-8" )); int d = -1 ; while ((d = in.read()) !=- 1 ){ System.out.print(( char )d); } } catch (IOException e) { e.printStackTrace(); } Copy code
  1. The first is to establish a socket connection, this step is equivalent to establishing a three-way handshake connection
  2. Send http request
  3. Get http protocol return result

And okhttp implements the http protocol in the interceptor, and each part implements the corresponding part of http, so as to achieve the purpose of fully decoupling

RetryAndFollowUpInterceptor

From the name, it is easy to see that this is a retry and continue interceptor, why do you want to resend the request?

  1. Routing failure, here is a little explanation. Routing is that a web service may have a proxy or multiple IP addresses. These can form the so-called routing. As long as we can connect to any one of the routes, we can communicate with the server, so after the routing fails You can choose other routes to connect
  2. Obtaining the 3xx redirection requires a redirection request, and the server or proxy requires authentication information

Let's take a look

intercept()
The process:

  1. Set up an endless loop to continuously resend requests for the above two possible situations
  2. transfer
    Transmitter.prepareToConnect()
    Create prepared objects, such as: ExchangeFinder, Address, RouteSelector
  3. Next call
    RealInterceptorChain.proceed()
    , Continue processing by the next Interceptor and get the response returned by it
  4. Handle routing exceptions and IO exceptions, here are the exceptions that cannot be retried, see for details
    recover()
    method
  5. transfer
    followUpRequest()
    Follow-up processing is carried out. Corresponding processing is carried out according to the return code of the server. For details, please refer to the meaning of the return code of the http protocol.
  6. According to the result of followUp, determine whether you need to resend the request or return the response directly

BridgeInterceptor

This interceptor is relatively simple. It repackages the request set by the user and turns it into a real Http request. To put it bluntly, it is to supplement the header in the Http request. If the user has set it, there is no need to set it again. To deal with the response, such as: save the cookie, decompress the ResponseBody

  1. Supplement the header in the Http request
  2. transfer
    RealInterceptorChain.proceed()
    , Continue processing by the next Interceptor and get the returned response
  3. Parse the returned Response, save the cookie, and decompress the ResponseBody

CacheInterceptor

This Interceptor mainly implements If-Modified-Since this type of request. For details, please refer to the http protocol. The function of this field is

Determine whether the data cached by the client has expired, if it expires, get new data

This field is usually used when the server needs to return a large file, which can speed up the http transmission speed, let s take a look

intercept()
The process

  1. Get the current caching strategy, there are still some things in it that I haven't figured out yet, it should be the interpretation of the Cache-Control field
  2. Get the cached Request and Response, the Request inside is generally modified from userRequest
  3. Determine whether the networkRequest is null. If yes, return the cached Response directly. When will the Request become null. When the Cache-Control field is only-if-cached, or when the userRequest is null, whether it can work without a network When to cache server information in this way?
  4. transfer
    RealInterceptorChain.proceed()
    , Continue processing by the next Interceptor and get the returned response
  5. If the returned code is HTTP_NOT_MODIFIED, the cached Response is returned and the cached Response is updated. The main expiration time is updated by the update corresponding to the date in the If-Modified-Since field
  6. Otherwise, the cache is invalid. Use the Response returned by the server and update the cache. Before updating the cache, determine whether the corresponding cache strategy supports caching.

ConnectInterceptor

This can be regarded as the core code of okhttp, some concepts in it are not too clear, and its logic is more complicated. But three cores are constant:

  1. DNS request, convert hostname to ip address
  2. Establish socket and perform three-way handshake
  3. TLS handshake

But there are still two theoretical issues,

  1. How to confirm whether to use http1.1 or http2
  2. How to confirm whether to use http or https

Let me explain it first:

  1. Okhttp currently supports http1.1 and http2, which are two protocols. HTTP2 requires the server to support TLS1.2. If the scheme part of the url is not an https string, it does not support http2. The default is http1.1. If you want to use other protocols To specify, see Protocol.java for details. In addition, if TLS1.2 is supported, determine whether the server and client support HTTP2 through TLS1.2
  2. It depends on whether the string in the scheme is http or https, if it is https, it will try to perform a TLS handshake with the server

Let's take a look

intercept()
The process

  1. Create a new Exchange object, this Exchange object is equivalent to a bridge between I/O and socket connection interaction
  2. transfer
    RealInterceptorChain.proceed()
    , Continue processing by the next Interceptor and return the corresponding Response directly

From the above point of view, the final step is to create an Exchange object

UML diagram

Explain the meaning and function of the classes a little bit

  1. Transmitter: Bridging the okhttp application layer and network layer, in my personal opinion, it is more like an interface that controls socket activation and cancellation
  2. Exchange: Storage class of objects interacting with the I/O layer
  3. ExchangeFinder: Used to find and create RealConnection objects
  4. ExchangeCodec: The interface used to send I/O requests
  5. RouteSelctor: used to create Rout and Proxy, and is responsible for DNS resolution class
  6. Selection: A subclass of RouteSelector, used to store the packaging class of Route
  7. Http1ExchangeCodec: Http1.1 is used to interact with I/O class, codec is the abbreviation of Code and Decode
  8. Http2ExchangeCodec: The class used by Http2 to interact with I/O
  9. Connection: socket and TLS connection interface
  10. Address: Some attributes needed for the connection are stored inside
  11. Route: routing, which is a wrapper class for proxy and Address
  12. RealConnection: used for socket connection and TLS handshake

Timing diagram

Transmitter.newExchange()

The logic of this method is very simple, just call

ExchangeFinder.find()
Create an ExchangeCodec object, and then use it as a parameter to create an Exchange object

ExchangeFinder.find()

Did two things:

  1. transfer
    findHealthyConnection()
    Get RealConnection
  2. transfer
    RealConnection.newCodec()
    Create ExchangeCodec object Http2ExchangeCodec or Http1ExchangeCodec

ExchangeFinder.findHealthyConnection()

  1. Establish an infinite loop. The infinite loop here is established for health. Why doesn't an infinite loop appear?
  2. transfer
    findConnection()
    , Find or create a RealConnection
  3. If it is a new RealConnection, return directly
  4. Determine whether it is a HealthyConnection, the definition of Healthy is whether the socket has been closed, and if it is not a healthy Connection, it will be released

Explain why there is no infinite loop. This is because if it is not a HealthConnection, the next Connection will be found. If all Connections are not Healthy, a new Connection will be created, and this Connection must be Healthy.

ExchangeFinder.findConnection()

  1. Try to obtain Transmitter.connection directly, the failure cases include Transmitter.connection is null or Transmitter.connection cannot create Exchange
  2. transfer
    RealConnectionPool.transmitterAcquirePooledConnection()
    Try to get the matching connection from RealConnectionPool, the matching conditions are mainly
    isEligible()
    :
    1. Transmitter has been charged or the connection cannot create Exchange, it does not match
    2. Whether it is the same Address, if yes, it matches
  3. Select the corresponding Route according to the conditions and prepare for the creation of a new Connection. These conditions have not been fully understood.
  4. 2.call
    RealConnectionPool.transmitterAcquirePooledConnection()
    Try to get the matching connection from RealConnectionPool. This time the matching condition is the Routes obtained after DNS
  5. If it is still not obtained, create a new RealConnection
  6. transfer
    RealConnection.connect()
    Method to make socket and TLS connection
  7. the third time
    RealConnectionPool.transmitterAcquirePooledConnection()
    Try to obtain a matching Connection from RealConnectionPool, and the matching condition is requireMultiplexed = true
  8. If it is still not obtained, call to update connectionPool

The first time I get the connection of http1.1 or http2, the second and third times are to get the multiplexed Connection of http2, and the second time to get the multiplexed Connection is for the speed of http2 (multiple domain names are shared IP?). In the third acquisition, there may be multiple identical Connections created at the same time, because the previous one is still connected, which makes the current one unable to be reused. This acquisition can only be used to acquire the http2 Connection

RealConnection.connect()

  1. Check the legitimacy of Connection
    1. If https is not supported, the type of ConnectionSpec must include CLEARTEXT (plain text request, such as HTTP, FTP, etc.), which is included by default
    2. Determine whether to support CLEARTEXT type request according to the platform,
  2. Determine whether there is an HTTP proxy, if so, call
    connectTunnel()
    Connection, tunnel principle has not been understood yet, do not understand first
  3. Otherwise call
    connectSocket()
    Create rawSocket and perform a three-way handshake
  4. transfer
    establishProtocol()
    Identification protocol

RealConnection.connectSocket()

  1. If the proxy is of HTTP type or there is no proxy, create a Socket from the SocketFactory and call the platform's
    connectSocket()
    , AndroidPlatform has no special treatment, directly call
    socket.connect()
    Make a connection
  2. Create Source and Sink, corresponding to InputStream (to obtain HTTP response) and OutputStream (to send request)

RealConnection.establishProtocol()

  1. Check if SslSocketFactory is null. If it is null, it means that HTTP2 is not supported. By default, the protocol will be declared as HTTP1.1. Of course, if H2_PRIOR_KNOWLEDGE is specified, you can still use http2 to call
    startHttp2()
  2. transfer
    connectTls()
    TLS handshake
  3. If the HTTP2 protocol is supported, call
    startHttp2()
    Establish an HTTP2 PREFACE connection. What is the purpose of establishing this connection? Synchronize streamId? do not know much

RealConnection.connectTls()

  1. Create SSLSocket, which is implemented in the JDK used here. It is not particularly clear. You may use the openssl library or other libraries.
  2. Get ConnectionSpec, here seems to be mainly encryption algorithm support, the default is ConnectionSpec.MODERN_TLS
  3. If TLS extension is supported, the configuration corresponds to TLS extension information. My understanding here is TLS
  4. Get the SSLSession and verify it. In my understanding, it should be the third step of the TLS handshake, the process of verifying the certificate by the client
  5. After the handshake is successful, try to obtain the corresponding HTTP protocol from TLS. The ALPN protocol is based on TLS to obtain whether the client and server support HTTP2.

CallServerInterceptor

The function of this Interceptor is to output fields to the IO stream

  1. For Http1.1, input to the string stream
  2. For Http2, it is input into the binary frame, HEADER frame and DATA frame

Let's take a look

interceptor()
Logic of

  1. This step is mainly to create the request header into the corresponding stream, HTTP1.1 and HTTP2 have different implementations
  2. Judging whether it is a GET request or a POST request (requests such as PUT and DELETE are not involved yet), here is mainly judged by whether there is a RequestBody, if there is, the RequestBody should be written into the sink, and the sink has RealConnection created
  3. Send HTTP request, call for HTTP1.1
    sink.flush()
    , For HTTP2, it is the final call
    FramingSink.close()
    Determine whether there is a DATA frame in close, and send DATA and HEADER frames
  4. Read request header
    exchange.readResponseHeaders()
    And request body
    exchange.readResponseHeaders

Okhttp connection pool

The core function of the connection pool is to keep-alive in the http1.1 protocol field in the , in order to reduce the creation of Sockets to reduce the delay of HTTP. Analyze above

ExchangeFinder.findConnection()
I have already explained the storage and reuse of the connection pool at the time. Next, I will mainly add

  1. Creation of connection pool
  2. The relationship between Call and Connection
  3. Connection pool expiration cleanup

Creation of connection pool

The default is to create a connection pool with a maximum of 5 connections and a single maximum of 5 minutes. The delegation mode is used here, and the RealConnectionPool object is finally created

The relationship between Call and Connection

A Request corresponds to a Call, and a Call corresponds to a Connection in a Transmitter, so if the Connection in the Transmitter can be reused, but a Connection can correspond to multiple Transmitters and Calls, it means that the same request is called

Call.execute()
repeatedly

Connection pool expiration cleanup

First look at the connection pool update

RealConnection.put()
Logic of

  1. First check if it is being cleaned up, if not, start a clean up thread
  2. Add Connection to ArrayDeque

Next look at the logic of cleaning up Runnable

Runnable.run()

  1. Create an endless loop
  2. transfer
    cleanup()
    Clean up the Connection and get the return value, only clean up the one with the longest obsolete time
  3. There are three return values, which represent different meanings
    1. Return -1 to indicate that the ConnectionPool is empty and end the cleanup thread
    2. Return 0 means that a Connection is cleaned up, and then continue to clean up the thread
    3. Returning greater than 0 means that there is a suspended (Idle) Connection, but the corresponding time has not been reached yet, and then try to clean up again after the time of temporary sleep and return

RealConnectionPool.cleanup()

  1. Traverse all Connections in the queue and call them in turn
    pruneAndGetAllocationCount()
    , Get Idle Connection
  2. Pass for
    longestIdleDurationNs
    To find the one with the longest idle time
  3. If the longest Idle time is greater than the survival time of Connection
    keepAliveDurationNs
    Or the Idle Connection in the queue is greater than the maximum value, then the Connection with the largest Idle time is released and 0 is returned.
  4. If there is an Idle Connection, the time to sleep is returned, because during this period, it is impossible for an Idle Connection to exceed the survival time of the Connection
  5. If there is no Idle Connection and there is a Connection in use, then the survival time of the Connection is returned
  6. If the ConnectionPool is empty, it returns -1 and ends the cleanup thread

RealConnectionPool.pruneAndGetAllocationCount()

  1. There is a hidden judgment here, which is when
    Connection.transmitters
    When the number is 0, it will directly return 0, so for a normal Idle Connection, it will directly return 0
  2. The main function of the loop is actually to find out the TransmitterReference of the memory leak. The logic in the loop is as follows:
    1. If there is a Transmitter in use, skip
    2. If no Transmitter is Null, but there is still TransmitterReference, that is, no call
      response.body.close()
      , Print Log information and remove Reference
    3. If you end up in the loop
      Connection.transmitters
      If it is empty, the expiration time is set to the maximum survival time, and 0 is returned

Expired cleanup of the connection pool is still a bit interesting, especially

pruneAndGetAllocationCount()
Toss me for a long time

Problems to be solved

  1. It seems that okhttp will always reuse Connection, even if the Connection field of the Header in the specified Request is not keep-alive
  2. Principle of tunnel
  3. The implementation of Http2, especially okhttp combines multiple requests and did not see how to implement or use