CHUNGA - Portable chunked streams for Common Lisp

Abstract
Chunga implements streams capable of chunked encoding on demand as defined in RFC 2616. For an example of how these streams can be used see Drakma.
The library needs a Common Lisp implementation that supports Gray streams and relies on David Lichteblau's trivial-gray-streams to offer portability between different Lisps.
Chunga is currently not optimized towards performance - it is rather intended to be easy to use and (if possible) to behave correctly.
The code comes with a BSD-style license so you can basically do with it whatever you want.
Download shortcut: http://weitz.de/files/chunga.tar.gz.

Download and installation
Support and mailing lists
The Chunga dictionary
Acknowledgements

Download and installation

Chunga together with this documentation can be downloaded from http://weitz.de/files/chunga.tar.gz. The current version is 1.1.0. (This version is not compatible with pre-2009 releases of Hunchentoot or Drakma.) Chunga will only work with Lisps where the character codes of all Latin-1 characters coincide with their Unicode code points (which is the case for all current implementations I know).

Chunga depends on the trivial-gray-streams library. You can download and install Chunga and its dependencies automatically with ASDF-Install, and there's a port for Gentoo Linux thanks to Matthew Kennedy.

A Mercurial repository of older versions is available at http://arcanes.fr.eu.org/~pierre/2007/02/weitz/ thanks to Pierre Thierry.

Luís Oliveira maintains a darcs repository of Chunga at http://common-lisp.net/~loliveira/ediware/.

Support and mailing lists

For questions, bug reports, feature requests, improvements, or patches please use the drakma-devel mailing list. If you want to be notified about future releases subscribe to the drakma-announce mailing list. These mailing lists were made available thanks to the services of common-lisp.net.

If you want to send patches, please read this first.

The Chunga dictionary

Chunked streams

Chunked streams are the core of the Chunga library. You create them using the function MAKE-CHUNKED-STREAM which takes an open binary stream (called the underlying stream) as its single argument. A binary stream in this context means that if it's an input stream, you can apply READ-SEQUENCE to it where the sequence is an array of element type OCTET, and similarly for WRITE-SEQUENCE and output streams. (Note that this specifically holds for bivalent streams like socket streams.)

A chunked stream behaves like an ordinary Lisp stream of element type OCTET with the addition that you can turn chunking on and off for input as well as for output. With chunking turned on, data is read or written according to the definition in RFC 2616.

[Standard class]
chunked-stream

Every chunked stream returned by MAKE-CHUNKED-STREAM is of this type which is a subtype of STREAM.

[Standard class]
chunked-input-stream

A chunked stream is of this type if its underlying stream is an input stream. This is a subtype of CHUNKED-STREAM.

[Standard class]
chunked-output-stream

A chunked stream is of this type if its underlying stream is an output stream. This is a subtype of CHUNKED-STREAM.

[Standard class]
chunked-io-stream

A chunked stream is of this type if it is both a CHUNKED-INPUT-STREAM as well as a CHUNKED-OUTPUT-STREAM.

[Function]
make-chunked-stream stream => chunked-stream

Creates and returns a chunked stream (a stream of type CHUNKED-STREAM) which wraps stream. stream must be an open binary stream.

[Specialized reader]
chunked-stream-stream (stream chunked-stream) => underlying-stream

Returns the underlying stream of the chunked stream stream.

[Generic reader]
chunked-stream-input-chunking-p object => generalized-boolean

Returns a true value if object is of type CHUNKED-INPUT-STREAM and if input chunking is currently enabled.

[Specialized writer]
(setf (chunked-stream-input-chunking-p (stream chunked-input-stream)) new-value)

This function is used to switch input chunking on stream on or off. Note that input chunking will usally be turned off automatically when the last chunk is read.

[Generic reader]
chunked-stream-output-chunking-p object => generalized-boolean

Returns a true value if object is of type CHUNKED-OUTPUT-STREAM and if output chunking is currently enabled.

[Specialized writer]
(setf (chunked-stream-output-chunking-p (stream chunked-output-stream)) new-value)

This function is used to switch output chunking on stream on or off.

[Specialized reader]
chunked-input-stream-extensions (stream chunked-input-stream) => extensions

Returns an alist of attribute/value pairs corresponding to the optional "chunk extensions" which might have been encountered when reading from stream.

[Specialized reader]
chunked-input-stream-trailers (stream chunked-input-stream) => trailers

Returns the optional "trailer" HTTP headers which might have been sent after the last chunk, i.e. directly before input chunking ended on stream. The format of trailers is identical to that returned by READ-HTTP-HEADERS.

Conditions

Here are conditions which might be signalled if something bad happens with a chunked stream.

[Condition]
chunga-condition

All conditions signalled by Chunga are of this type. This is a subtype of CONDITION.

[Error]
chunga-error

All errors signalled by Chunga are of this type. This is a subtype of CHUNGA-CONDITION and of STREAM-ERROR, so STREAM-ERROR-STREAM can be used to access the offending stream.

[Warning]
chunga-warning

All warnings signalled by Chunga are of this type. This is a subtype of CHUNGA-CONDITION and of WARNING.

[Error]
syntax-error

An error of this type is signalled if Chunga encounters wrong or unknown syntax when reading data. This is a subtype of CHUNGA-ERROR.

[Error]
parameter-error

An error of this type is signalled if a function was called with inconsistent or illegal parameters. This is a subtype of CHUNGA-ERROR.

[Condition type]
input-chunking-body-corrupted

A condition of this type is signaled if an unexpected character (octet) is read while reading from a chunked stream with input chunking enabled. This is a subtype of CHUNGA-ERROR.

[Condition type]
input-chunking-unexpected-end-of-file

A condition of this type is signaled if we reach an unexpected EOF on a chunked stream with input chunking enabled. This is a subtype of CHUNGA-ERROR.

RFC 2616 parsing

Chunga needs to know a bit about RFC 2616 syntax in order to cope with extensions and trailers. As these functions are in there anyway, they're exported, so they can be used by other code like for example Drakma.

Note that all of these functions are designed to work on binary streams, specifically on streams with element type (UNSIGNED-BYTE 8). They will not work with character streams. (But the "bivalent" streams offered by many Lisp implementations will do.) They must be called within the context of WITH-CHARACTER-STREAM-SEMANTICS.

[Macro]
with-character-stream-semantics statement* => result*

Executes the statement* forms in such a way that functions within this section can read characters from binary streams (treating octets as the Latin-1 characters with the corresponding code points). All the functions below must be wrapped with this macro. If your code uses several of these functions which interact on the same stream, all of them must be wrapped with the same macro. See the source code of Drakma or Hunchentoot for examples of how to use this macro.

[Function]
read-line* stream &optional log-stream => line

Reads and assembles characters from the binary stream stream until a carriage return is read. Makes sure that the following character is a linefeed. If *ACCEPT-BOGUS-EOLS* is not NIL, then the function will also accept a lone carriage return or linefeed as a line break. Returns the string of characters read excluding the line break. Additionally logs this string to log-stream if it is not NIL.
See WITH-CHARACTER-STREAM-SEMANTICS.

[Function]
read-http-headers stream &optional log-stream => headers

Reads HTTP header lines from the binary stream stream (except for the initial status line which is supposed to be read already) and returns a corresponding alist of names and values where the names are keywords and the values are strings. Multiple lines with the same name are combined into one value, the individual values separated by commas. Header lines which are spread across multiple lines are recognized and treated correctly. (But see *TREAT-SEMICOLON-AS-CONTINUATION*.) Additonally logs the header lines to log-stream if it is not NIL.
See WITH-CHARACTER-STREAM-SEMANTICS.

[Function]
read-token stream => token

Read characters from the binary stream stream while they are token constituents (according to RFC 2616). It is assumed that there's a token character at the current position. The token read is returned as a string. Doesn't signal an error (but simply stops reading) if END-OF-FILE is encountered after the first character.
See WITH-CHARACTER-STREAM-SEMANTICS.

[Function]
token-char-p char => generalized-boolean

Returns a true value if the Lisp character CHAR is a token constituent according to RFC 2616.

[Function]
read-name-value-pair stream &key value-required-p cookie-syntax => pair

Reads a typical (in RFC 2616) name/value or attribute/value combination from the binary stream stream - a token followed by a #\= character and another token or a quoted string. Returns a cons of the name and the value, both as strings. If value-required-p is NIL (the default is T), the #\= sign and the value are optional. If cookie-syntax is true (the default is NIL), the value is read like the value of a cookie header.
See WITH-CHARACTER-STREAM-SEMANTICS.

[Function]
read-name-value-pairs stream &key value-required-p cookie-syntax => pairs

Uses READ-NAME-VALUE-PAIR to read and return an alist of name/value pairs from the binary stream stream. It is assumed that the pairs are separated by semicolons and that the first char read (except for whitespace) will be a semicolon. The parameters are used as in READ-NAME-VALUE-PAIR. Stops reading in case of END-OF-FILE (instead of signaling an error).
See WITH-CHARACTER-STREAM-SEMANTICS.

[Function]
assert-char stream expected-char => char

Reads the next character from the binary stream stream and checks if it is the character expected-char. Signals an error otherwise.
See WITH-CHARACTER-STREAM-SEMANTICS.

[Function]
skip-whitespace stream => char-or-nil

Consume characters from the binary stream stream until an END-OF-FILE is encountered or a non-whitespace (according to RFC 2616) characters is seen. This character is returned (or NIL in case of END-OF-FILE).
See WITH-CHARACTER-STREAM-SEMANTICS.

[Function]
read-char* stream => char

Reads and returns the next character from the binary stream stream.
See WITH-CHARACTER-STREAM-SEMANTICS.

[Function]
peek-char* stream &optional eof-error-p eof-value => boolean

Returns a true value if a character can be read from the binary stream stream. If eof-error-p has a true value, an error is signalled if no character remains to be read. eof-value specifies the value to return if eof-error-p is false and the end of the file has been reached.
See WITH-CHARACTER-STREAM-SEMANTICS.

[Function]
trim-whitespace string &key start end => string'

Returns a version of the string string (between start and end) where spaces and tab characters are trimmed from the start and the end.

[Special variable]
*current-error-message*

Used by the parsing functions in this section as an introduction to a standardized error message. Should be bound to a string or NIL if one of these functions is called.

[Special variable]
*accept-bogus-eols*

Some web servers do not respond with a correct CRLF line ending for HTTP headers but with a lone linefeed or carriage return instead. If this variable is bound to a true value, READ-LINE* will treat a lone LF or CR character as an acceptable end of line. The initial value is NIL.

[Special variable]
*treat-semicolon-as-continuation*

According to John Foderaro, Netscape v3 web servers bogusly split Set-Cookie headers over multiple lines which means that we'd have to treat Set-Cookie headers ending with a semicolon as incomplete and combine them with the next header. This will only be done if this variable has a true value, though. Its default value is NIL.

[Function]
as-keyword string &key destructivep => keyword

Converts the string string to a keyword where all characters are uppercase or lowercase, taking into account the current readtable case. Might destructively modify string if destructivep is true which is the default. "Knows" several HTTP header names and methods and is optimized to not call INTERN for these.

[Function]
as-capitalized-string keyword => capitalized-string

Kind of the inverse of AS-KEYWORD. Has essentially the same effect as STRING-CAPITALIZE but is optimized for "known" keywords like :CONTENT-LENGTH or :GET.

Acknowledgements

Thanks to Jochen Schmidt's chunking code in ACL-COMPAT for inspiration. This documentation was prepared with DOCUMENTATION-TEMPLATE.

$Header: /usr/local/cvsrep/chunga/doc/index.html,v 1.33 2008/05/29 22:22:59 edi Exp $

BACK TO MY HOMEPAGE