# Class Http_client.pipeline

    class pipeline : object .. end

A pipeline is a queue of HTTP calls to perform

## A

`pipeline` object is a FIFO queue of HTTP calls. It is called "pipeline" because it is processed asynchronously: Requests may be sent to the HTTP server independently of whether responses of the previous requests already arrived or not.

Furthermore, a `pipeline` object may keep connections to several servers at once. (More exactly, it has a FIFO queue for every server it is connected with.)

The `pipeline` object keeps track what is happening, so you need not to care about the details of communications. The API is simple: Create a `pipeline` object, do some setup (add authentication methods; configure the proxy to use), add the requests, and `run` the pipeline. The rest is done automatically. To get the results, you can either memorize the requests you wanted to know yourself and ask every request object about the reply of the server; or you can specify that a callback function should be called once the request is processed (with positive or negative result). It is possible to add further requests to the pipeline from within these callback functions.

If you want to have several pipelines, or some cooperation with other network services, you may specify a `Unixqueue.event_system`. For example, to have two pipelines working concurrently:

     let ues = Unixqueue.create_unix_event_system() in
     let p1 = new pipeline in
     let p2 = new pipeline in
     p1 # set_event_system ues;
     p2 # set_event_system ues;
     Unixqueue.run ues             (* run p1 and p2 in parallel *)

This works not only with pipelines, but with every network client or server which is compatible with the `Unixqueue` design.

By default, the pipeline only supports "http" URLs. You can configure "https" support via `Https_client`.

    method event_system : Unixqueue.event_system

Returns the event system

    method set_event_system : Unixqueue.event_system -> unit

Sets the event system. Must be called before the first call is added

    method connection_cache : connection_cache

The current connection cache. By default, a private restrictive cache is used.

    method set_connection_cache : connection_cache -> unit

Set the connection cache. This must happen before the first call is added.

    method add_authentication_method : basic_auth_method -> unit

adds an old-style authentication method

    method add_auth_handler : auth_handler -> unit

adds a new-style authentication handler

    method set_proxy : string -> int -> unit

`set_proxy name port`: sets that an HTTP proxy `name` listening on `port` is to be used

    method set_proxy_auth : string -> string -> unit

sets user and password for the proxy. Works for both "digest" and "basic" mechanisms. Any realm is acceptable.

    method avoid_proxy_for : string list -> unit

sets a list of host names or domain suffixes for which no proxy should be used. e.g. `"localhost"; ".our.net"`

    method set_proxy_from_environment : unit -> unit

Inspect the environment variables `http_proxy` and `no_proxy` and set the proxy options from them.

    method set_socks5_proxy : string -> int -> unit

Sets that a SOCKS version 5 proxy is used at this host and port. There is no authentication. The `avoid_proxy_for` setting is honoured.

    method configure_transport : channel_binding_id -> transport_channel_type -> unit

`configure_transport id transport`: Configures that messages with channel binding ID `id` are exchanged on `transport`.

By default, there is only a configuration for `Http_client.http_cb_id`, i.e. for normal unencrypted channels.

    method set_transport_proxy : channel_binding_id ->
           string -> int -> (string * string) option -> proxy_type -> unit

`set_transport_proxy id host port auth ptype`: Sets a special proxy for the transport identified by `id`. This overrides `set_proxy`, `set_proxy_auth`, and `set_socks5_proxy` for the given transport.

    method set_transport_proxy_from_environment : (string * channel_binding_id) list -> unit

Like `set_proxy_from_environment`, this method inspects environment variables and configures the proxy settings. This function, however, is more flexible, and can use different environment variables for different transports.

The argument list has pairs `(var_name, id)` meaning that the environment variable `var_name` configures the proxy for `id`. For instance,

     [("http_proxy", http_cb_id); ("https_proxy", https_cb_id)]

means that these two variables are used for the respective transports.

The variable `"no_proxy"` is interpreted anyway.

    method reset : unit -> unit

Empties the pipeline and inactivates any open connection. The currently active operation is interrupted, and every request with response is set to `No_reply` (i.e. you get the exception `No_reply` if you try to access the response). If there are callbacks for these requests, the callback functions are invoked. The queues of open requests and replies are cleared. All connections to all servers are inactivated.

Inactivation means that open connections are given back to the connection cache for further reuse if the state of the connection allows this; otherwise the connections are closed.

    method add : http_call -> unit

Adds the call to the end of the pipeline. One must not add calls that have already been served.

    method add_with_callback : http_call -> (http_call -> unit) -> unit

Adds the call to the end of the pipeline.

After the call has been processed, the callback function is called. This function is called for every call that leaves the pipeline, it does not matter whether processing was successful or not. Invoke `status` on the message to get what happened; either some status information from the server is available (perhaps OK status), or an exception is indicated.

    method add_e : http_call -> http_call Uq_engines.engine

The same as engine: The call `c` is added to the pipeline, and when it is processed, the returned engine transitions to the state ``Done c`.

    method proxy_type : string -> proxy_type option

`proxy_type url` returns `Some pt` if a proxy would be used for this `url`, and `None` if a direct connection would be made.

    method proxy_type_of_call : http_call -> proxy_type option

Same for an already created call object

    method channel_binding : http_call -> channel_binding_id

Reports the current channel binding of this call

    method run : unit -> unit

Runs through the requests in the pipeline. If a request can be fulfilled, i.e. the server sends a response, the state of the request is set and the request is removed from the pipeline. If a request cannot be fulfilled (no response, bad response, network error), the exception is stored in the `http_call` object and will be raised once the state of the object is queried.

Under certain conditions (serious network errors) `run` does not catch the exception; it simply cleans its own state up (aborting the errorneous network connection). In this case, simply invoke `run` again to continue. `run` terminates normally if the pipeline becomes empty.

The engine handles the following HTTP return codes itself:

- 100: This is an intermediate return code
- 301: The redirection is followed if configured
- 302: The redirection is followed if configured
- 401: Content server authentication
- 407: Proxy server authentication

All other return codes remain uninterpreted, it is up to the caller of this function to react on them.

    method get_options : http_options

    method set_options : http_options -> unit

Get/Set the available options for the HTTP engine. The new options will take into effect immediately.

    method number_of_open_messages : int

Returns the number of messages which are still in the pipeline.

    method number_of_open_connections : int

Returns the number of connections which are open at the same time and currently being used by this object (i.e. connections returned to the cache do not count)

    method connections : (string * int * int) list

Reports which connections exist: `(host, port, queue_length)`

    method cnt_new_connections : int

Counts new connections (or better: attempts to establish connections)

    method cnt_timed_out_connections : int

Counts connections given up because of timeouts

    method cnt_crashed_connections : int

Counts connections with network or protocol errors

    method cnt_server_eof_connections : int

Counts connections the server terminated with EOF

    method cnt_successful_connections : int

Counts connections closed because pipelines become empty

    method cnt_failed_connections : int

Counts totally failed connections (no more reconnects allowed)

    method reset_counters : unit -> unit

## Notes on counters:

- `cnt_new_connections`: Is increased when a new connection attempt is started (that may fail or timeout in the future). Reconnects do not count.
- `cnt_timed_out_connections`: Is increased whenever an established connection times out. Usually, it is immediately reconnected.
- `cnt_crashed_connections`: Is increased whenever an established connection crashes. Usually, it is immediately reconnected.
- `cnt_failed_connections`: Is increased when a timed out or crashed connection exceeds the maximum number of errors, and it is not tried to reconnect.
- `cnt_successful_connections`: Is increased when all HTTP calls have been replied.

When the client has done all of its jobs, we have

     cnt_new_connections = cnt_failed_connections + cnt_successful_connections