# Ocamlnet 3 Reference Manual

Ocamlnet is an enhanced system platform library for Ocaml. As the name suggests, large parts of it have to do with network programming, but it is actually not restricted to this. Other parts deal with the management of multiple worker processes, and the interaction with other programs running on the same machine. You can also view Ocamlnet as an extension of the system interface as provided by the `Unix` module of the standard library.

So, which kind of applications can profit from Ocamlnet?

- Web applications can use the `netcgi2` library which provides all core functionality like connecting to web servers, decomposing web requests, and emitting web data. With the `nethttpd` library Ocamlnet even includes a little web server of its own, so stand-alone Ocaml programs can respond to HTTP requests. This is especially interesting for providing high-speed HTTP-based APIs like REST interfaces.
- Client/server architectures can be built with the included SunRPC support. This is a very robust and mature implementation of this binary RPC protocol, both on the client and the server side. It is wire-compatible with C implementations of SunRPC. There is a generator for transforming SunRPC IDL files into Ocaml modules. Authentication and encryption are also supported (using GSS-API, or the non-standard SSL extension).
- Compute jobs profit from Ocamlnet because the `netmulticore` library allows it to run compute tasks on as many cores of the machine as needed. The tasks can communicate with message passing and exchange data via shared memory. Of course, `netmulticore` is strictly typed.
- Any kind of application can cherry-pick the parts of Ocamlnet that are considered useful. Especially, Ocamlnet includes a number of network clients (for HTTP, FTP, Telnet, SMTP, POP), and a large number of network-related string processing functions (e.g. for URL's, Base64, UTF-8, mail headers). For interacting with the system there are functions to invoke subcommands (`shell` library), and also for globbing, sending email, and logging. Local and remote filesystems can be accessed with the `Netfs` framework. The `netsys` library wraps lots of additional system calls that are missing in the Ocaml standard library.

A number of ideas and concepts are used throughout the libraries:

- `Netchannels` are a way to abstract sequential I/O channels
- The event queues provided by `Equeue` and `Unixqueue` make asynchronous protocol interpreters possible
- The `netplex` library manages worker subprocesses, e.g. for accepting network connections or for doing computations.

## Introductory chapters

### Guides

- `Foreword`: How Ocamlnet is organized
- `Platform`: Which functions are available on which platform (POSIX/Win32). Also hints for portable programming.
- `Ipv6`: The state of IPv6 support
- `Regexp`: Regular expression backends

Related external articles:

- Ocamlnet 3 finally released
- Stranger in a strange land, by Gerd Stolpmann. The Win32 port in Ocamlnet 3.

## Contents by library

- Base Libraries
  - The netstring library: string processing functions
  - The netzip library: compression for object channels
  - The equeue library: concurrent execution flows via event queues
  - The netplex library: generic server framework
  - The netshm library: manage shared memory
  - The netsys library: system interfaces
- Web Programming
  - The netcgi2 library: web application framework
  - The netcgi2-apache library: run web apps inside Apache
  - The netcgi2-plex library: run web apps with Netplex
  - The nethttpd library: standalone web apps
- Client/Server Programming
  - The rpc library: SunRPC
  - The rpc-auth-local extension: authentication for local sockets
  - The rpc-auth-dh extension: Diffie-Hellman authentication
  - The rpc-xti extension: System V helpers
  - The rpc_ssl extension: SSL encryption
  - The netgssapi library with the GSS-API framework
  - The netmech-scram library for SCRAM
- Parallel Computing
  - The netcamlbox library: message passing
  - The netmulticore library: compute jobs
- Network Protocol Libraries
  - The netclient library: HTTP, FTP, Telnet
  - The pop library: POP protocol
  - The smtp library: SMTP protocol

# Base Libraries

### Guides

- `Netchannels_tut`: The `Netchannels` tutorial - strongly recommended read!
- `Netmime_tut`: The `Netmime` tutorial
- `Netsendmail_tut`: The `Netsendmail` tutorial
- `Netulex_tut`: The `Netulex` tutorial
- `Neturl_tut`: The `Neturl` tutorial
- `Equeue_howto`: The Equeue, Unixqueue, and Engines HOWTO
- `Equeue_intro`: Introduction to programming with `equeue`
- `Netplex_intro`: Introduction to `Netplex`
- `Netplex_advanced`: Advanced features of `Netplex`
- `Netplex_admin`: Netplex administration guide
- `Netshm_intro`: Shared Memory for IPC
- `Shell_intro`: Motivation for `shell`

### External articles

- Equeue compared to Lwt, by Jake Donham.
- Parallelizing with Ocamlnet, by Gerd Stolpmann. Demonstrates how to parallelize matrix multiplications with Netplex-controlled worker processes.
- Mastering Multi-processing, by Gerd Stolpmann. About the new synchronization primitives in Ocamlnet 3

## The `netstring` library: string processing functions

`netstring` focuses on string processing functions, and base definitions for other libraries.

Netconversion | Conversion between character encodings
Netunichar | Unicode character information
Netchannels | Object-oriented I/O: Basic types and classes
Netstream | A netstream is an input channel that is read block by block.
Mimestring | Low-level functions to parse and print mail and MIME messages
Netmime | Netmime contains high-level classes and functions to process mail and MIME messages.
Netsendmail | Functions to compose and send electronic mails
Neturl | Uniform Resource Locators (URLs)
Netsockaddr | Parsing of socket addresses
Netaddress | Parsing of mail addresses
Netbuffer | A Netbuffer.t is a buffer that can grow and shrink dynamically.
Netpagebuffer | Buffer for page-aligned I/O
Netdate | Support for common date/time parsing and formatting.
Netencoding | Base64, Quoted Printable, URL encoding, HTML escaping
Netulex | Support module for Alain Frisch's `ulex` lexer generator
Netaccel | Accelerators for bytecode
Netaccel_link | Enables accelerator module `Netaccel`
Nethtml | Parsing of HTML
Nethttp | Basic definitions for the HTTP protocol
Netfs | Class type `stream_fs` for filesystems with stream access to files
Netglob | Globbing
Netauth | Some primitives for authentication
Netcompression | Registry for compression algorithms
Netstring_str | Wrapper for regexps with `Str` syntax
Netmappings | Internal access to the character conversion database
Netaux | Internal auxiliary functions
Netnumber | Binary encodings of numbers
Rtypes | Binary encodings of numbers (Legacy)
Xdr | External Data Representation
Xdr_mstring | Managed Strings

## The `netstring-pcre` library: additions for PCRE

Netstring_pcre | Wrapper for regexps with PCRE syntax

## The `netzip` library: compression for object channels

Support for (un)compressing data on the fly with object channels. Requires ocamlzip.

Netgzip | Gzip object channels

## The `equeue` library: concurrent execution flows via event queues

`equeue` is a fundamental library for event queues. It is mainly used by a number of other libraries of Ocamlnet to parallelize network code (so-called multiplexing).

Equeue | `Equeue` implements generic event queues.
Unixqueue | Unixqueues are one of the two forms of system event loops provided by Ocamlnet.
Unixqueue_pollset | Unixqueue implementation on top of `Netsys_pollset`
Unixqueue_select | This the old `Unix.select`-based imeplementation of event systems which was the default one until Ocamlnet-2.2.
Uq_engines | An engine performs a certain task in an autonomous way.
Uq_socks5 | This module implements a SOCKS version 5 client (see RFC 1928) for use with the `Uq_engines.connector`, `Uq_engines.listener`, and `Uq_engines.datagram_provider` engine factories.
Uq_resolver | Support for pluggable resolvers
Uq_io | Unified engines for stream I/O
Uq_lwt | Compatibility with `Lwt`
Uq_libevent | Use Libevent as event loop
Uq_mt | Using engines in multi-threaded programs

### The `equeue-gtk1` and `equeue-gtk2` extensions

Extensions for `equeue` to integrate the event queue into user interfaces made with lablgtk and lablgtk2

Uq_gtk | Integration with lablgtk/lablgtk2 event systems

### The `equeue-ssl` extension

Extensions for `equeue` to support SSL

Uq_ssl | Asynchronous SSL connections
Https_client | HTTPS extension to `Http_client`

### The `equeue-tcl` extension

Extension for `equeue` to integrate the event queue into user interfaces made with labltk

Uq_tcl | Integration with the labltk event system

## The `netplex` library: generic server framework

The `netplex` library is a protocol-independent server framework. Especially, it can be used in conjunction with `nethttpd` to build web servers, and with `rpc` to build RPC servers.

Netplex_types | Types for `Netplex`
Netplex_config | Read the configuration file
Netplex_controller | Controller
Netplex_container | Containers
Netplex_sockserv | Socket service creation
Netplex_workload | Workload management
Netplex_kit | Netplex toolkit
Netplex_cenv | Container environment
Netplex_log | Loggers
Netplex_main | Main program for Netplex servers
Netplex_mp | Multi-processing provider
Netplex_mt | Multi-threading provider
Netplex_mutex | Netplex-wide mutexes
Netplex_semaphore | Netplex-wide semaphores
Netplex_sharedvar | Netplex-wide variables
Netplex_mbox | Netplex message boxes
Netplex_encap | Type-safe marshalling between processes of the same executable
Rpc_netplex | Netplex support for RPC servers (TCP only)

## The `shell` library: start external commands

The `shell` library allows you to start external commands. It is integrated into `equeue`.

Shell | Calls external programs, creates pipelines, etc.
Shell_sys | Calls external programs, creates pipelines, etc.
Shell_uq | Run shell commands within Unixqueues
Shell_fs | Shell filesystem

## The `netshm` library: manage shared memory

The `netshm` library manages a shared memory object either as hash table or array. It is designed to be used in multi-processing program architectures. Note that there is now also the much better The netmulticore library: compute jobs library.

Netshm | Shared memory for O'Caml programs using multi-processing
Netshm_data | Data representation for shared memory
Netshm_hashtbl | Hash tables in shared memory
Netshm_array | Arrays in shared memory

## The `netsys` library: system interfaces

The `netsys` library contains a number of low-level functions used by other Ocamlnet libraries.

Platform |
Netexn | Exception registry
Netlog | Basic logging facility
Netsys | System calls missing in the `Unix` module
Netsys_posix | POSIX-specific system calls missing in the `Unix` module, and further API's from POSIX-style operating systems.
Netsys_oothr | Object-oriented thread API
Netsys_signal | Signal handler framework
Netsys_tmp | Temporary files
Netsys_mem | Bigarrays as memory buffers
Netsys_sem | Generic anonymous semaphores
Netsys_pollset | Sets of file descriptors for polling
Netsys_pollset_generic | Returns a good standard implementation of pollset for this platform.
Netsys_pollset_posix | Pollsets for POSIX operating systems
Netsys_pollset_win32 | Pollsets for Win32
Netsys_win32 | Primitives for Win32
Netsys_rng | Random-number generator

# Web Programming

### Guides

- `Netcgi_porting`: Porting `netcgi1` programs to `netcgi2`
- `Nethttpd_intro`: Overview over the HTTP daemon

### The `netcgi1` library

The `netcgi1` library is no longer supported in Ocamlnet 3. Please switch to `netcgi2`.

## The `netcgi2` library: web application framework

This is the revised library for Web applications, now called `netcgi2`.

Netcgi | Common data-structures for CGI-like connectors.
Netcgi_common | Functions to develop new connectors.
Netcgi_cgi | Classical CGI connector.
Netcgi_fcgi | FastCGI connector.
Netcgi_ajp | Apache JServ Protocol (AJP) 1.3 connector.
Netcgi_scgi | SCGI connector.
Netcgi_test | Connector for testing your code.
Netcgi_dbi | Pools of connections for the ocamldbi generic database interface.
Netcgi1_compat | Compatibility module with the previous version of Netcgi.

## The `netcgi2-apache` library: run web apps inside Apache

This is an Apache connector for the `netcgi2` library.

Netcgi_apache | Netcgi Apache "mod" connector.
Netcgi_modtpl |

## The `netcgi2-plex` library: run web apps with Netplex

Netplex support for running Netcgi2 connectors:

Netcgi_plex | Netplex support for FastCGI, SCGI and AJP connectors

## The `nethttpd` library: standalone web apps

The `nethttpd` library is a Web server component written in O'Caml. For a full web server, you'll also need `netplex`.

Nethttpd_types | Type definitions for the HTTP daemon
Nethttpd_kernel | The protocol kernel of the HTTP daemon
Nethttpd_reactor | The reactive encapsulation of the HTTP daemon
Nethttpd_engine | The event-based encapsulation of the HTTP daemon
Nethttpd_services | Service Providers for HTTP daemon
Nethttpd_plex | Netplex support
Nethttpd_util | Utility functions

# Client/Server Programming

### Guides

- `Rpc_intro`: Introduction
- `Rpc_mapping_ref`: RPC Language Mapping Reference
- `Rpc_intro_gss`: Securing RPC with the GSS-API

### External articles

- The next server, please!, by Gerd Stolpmann. About highly-available RPC.

## The `rpc` library: SunRPC

This library implements OncRPC (alias SunRPC).

Rpc | Common types and exceptions
Rpc_program | RPC programs
Rpc_client | RPC clients
Rpc_simple_client | Synchronous API for RPC clients
Rpc_proxy | RPC proxies
Rpc_server | RPC servers
Rpc_auth_sys | Authentication module AUTH_SYS
Rpc_portmapper | Portmapper version 2
Rpc_portmapper_aux |
Rpc_portmapper_clnt |
Rpc_transport | Low-level RPC transporters
Rpc_packer | The first n bytes of the packed value
Rpc_auth_gssapi | GSS-API for RPC authentication

## The `rpc-auth-local` extension: authentication for local sockets

Authentication for local socket connections.

Rpc_auth_local | Return the authentication method `AUTH_LOCAL`.

## The `rpc-auth-dh` extension: Diffie-Hellman authentication

This is Diffie-Hellman authentication for OncRPC (also known as "Secure RPC").

Rpc_auth_dh | Diffie-Hellman authentication (AUTH_DH alias AUTH_DES)
Rpc_key_service | Contact the keyserv daemon to encrypt/decrypt data with the common key.
Rpc_time | Get the time of the server (using the RFC 868 netdate protocol)

## The `rpc-xti` extension: System V helpers

System V provides a network API called XTI in addition to the socket API. This library allows it to connect to RPC services that can only be reached over a local XTI connection ("cots" connection).

Rpc_xti_client | Minimal support for TI-RPC over the XTI API

## The `rpc_ssl` extension: SSL encryption

This library allows you to tunnel RPC through SSL/TLS.

Rpc_ssl | Securing RPC by SSL

## The `netgssapi` library with the GSS-API framework

Netgssapi | GSS-API Definition

## The `netmech-scram` library for SCRAM

Implements SCRAM (RFC 5802). Right now only for GSS-API.

Netmech_scram | SCRAM mechanism for authentication (RFC 5802)
Netmech_scram_gssapi | The SCRAM security mechanism for GSS-API

# Parallel Computing

### Guides

- `Netmcore_basics`: Basics
- `Netmcore_tut`: Tutorial

Important change since OCaml-4.01: This OCaml version changed the semantics of the built-in primitives `caml_modify` and `caml_initialize`. Essentially, it is no longer possible to modify OCaml values residing outside the regular OCaml heap. As we do this inside Netcamlbox and Netmulticore, this change affects these libraries. Fortunately, there is a workaround on systems supporting weak symbols (all ELF systems and OS X): Here, `caml_modify` and `caml_initialize` are overridden by Ocamlnet so that they are again compatible. Note that this is a global modification of the runtime system!

## The `netcamlbox` library: message passing

Camlbox is a fast message-passing system between processes running on the same machine. It uses shared memory for communication. A message sent to a Camlbox is only copied once, not twice.

Netcamlbox | Camlboxes are a fast IPC mechanism to send Ocaml values from one process to another.

## The `netmulticore` library: compute jobs

Netmulticore is an experimental framework for managing multiple processes, and sending messages between them.

Netmcore | Multi-processing for compute jobs
Netmcore_process | Statically typed processes
Netmcore_mempool | Memory pools
Netmcore_heap | Shared heaps of structured values
Netmcore_array | Shared arrays
Netmcore_buffer | Shared buffer
Netmcore_hashtbl | Shared hashtables
Netmcore_matrix | Shared 2-dimensional arrays (matrices)
Netmcore_queue | Shared queues
Netmcore_ref | Shared mutable variables
Netmcore_mutex | Mutexes
Netmcore_sem | Semaphores
Netmcore_condition | Condition variables
Netmcore_camlbox | Camlboxes for use in netmulticore programs

# Network Protocol Libraries

### Guides

- `Netclient_tut`: The Netclient tutorial (HTTP, FTP)

## The `netclient` library: HTTP, FTP, Telnet

This library includes clients for HTTP, Telnet, and FTP.

Http_client | HTTP 1.1 client
Http_client_conncache | Connection cache
Http_fs | HTTP filesystem
Ftp_client | FTP client
Ftp_data_endpoint | Senders and receivers for the FTP data connection
Ftp_fs | FTP filesystem
Telnet_client | Telnet client

## The `pop` library: POP protocol

This is a simple POP 3 client.

Netpop | This is an interface for the Post Office Protocol - Version 3 (POP3) as specifed by RFC 1939.

## The `smtp` library: SMTP protocol

This is a simple SMTP client.

Netsmtp | This is an interface for the Simple Mail Tranfer Protocol (SMTP) as specified by RFC 2821.

# Trailer

## Index

- Index of types
- Index of exceptions
- Index of values
- Index of class attributes
- Index of class methods
- Index of classes
- Index of class types
- Index of modules
- Index of module types

## Authors

The Ocamlnet libraries have been initially written by Gerd Stolpmann and Patrick Doane.

Contributions by

- Nicolas George: Netdate
- Eric Stokes: FastCGI support
- Pierre Habouzit: SMTP client
- Christophe Troestler: Revised version of `netcgi`
- Deokhwan Kim: Support for EUC-KR

Sponsorships by

- Baretta s.r.l: Nethttpd
- California State University: SSL support for RPC, Netplex