diff options
Diffstat (limited to 'doc/zstd_manual.html')
| -rw-r--r-- | doc/zstd_manual.html | 153 |
1 files changed, 85 insertions, 68 deletions
diff --git a/doc/zstd_manual.html b/doc/zstd_manual.html index 204f56ea5f21..2e77e7742f60 100644 --- a/doc/zstd_manual.html +++ b/doc/zstd_manual.html @@ -1,10 +1,10 @@ <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"> -<title>zstd 1.1.4 Manual</title> +<title>zstd 1.2.0 Manual</title> </head> <body> -<h1>zstd 1.1.4 Manual</h1> +<h1>zstd 1.2.0 Manual</h1> <hr> <a name="Contents"></a><h2>Contents</h2> <ol> @@ -57,46 +57,46 @@ <pre><b>size_t ZSTD_compress( void* dst, size_t dstCapacity, const void* src, size_t srcSize, int compressionLevel); -</b><p> Compresses `src` content as a single zstd compressed frame into already allocated `dst`. - Hint : compression runs faster if `dstCapacity` >= `ZSTD_compressBound(srcSize)`. - @return : compressed size written into `dst` (<= `dstCapacity), - or an error code if it fails (which can be tested using ZSTD_isError()). +</b><p> Compresses `src` content as a single zstd compressed frame into already allocated `dst`. + Hint : compression runs faster if `dstCapacity` >= `ZSTD_compressBound(srcSize)`. + @return : compressed size written into `dst` (<= `dstCapacity), + or an error code if it fails (which can be tested using ZSTD_isError()). </p></pre><BR> <pre><b>size_t ZSTD_decompress( void* dst, size_t dstCapacity, const void* src, size_t compressedSize); -</b><p> `compressedSize` : must be the _exact_ size of some number of compressed and/or skippable frames. - `dstCapacity` is an upper bound of originalSize. - If user cannot imply a maximum upper bound, it's better to use streaming mode to decompress data. - @return : the number of bytes decompressed into `dst` (<= `dstCapacity`), - or an errorCode if it fails (which can be tested using ZSTD_isError()). +</b><p> `compressedSize` : must be the _exact_ size of some number of compressed and/or skippable frames. + `dstCapacity` is an upper bound of originalSize. + If user cannot imply a maximum upper bound, it's better to use streaming mode to decompress data. + @return : the number of bytes decompressed into `dst` (<= `dstCapacity`), + or an errorCode if it fails (which can be tested using ZSTD_isError()). </p></pre><BR> <pre><b>unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize); -</b><p> NOTE: This function is planned to be obsolete, in favour of ZSTD_getFrameContentSize. - ZSTD_getFrameContentSize functions the same way, returning the decompressed size of a single - frame, but distinguishes empty frames from frames with an unknown size, or errors. +</b><p> NOTE: This function is planned to be obsolete, in favour of ZSTD_getFrameContentSize. + ZSTD_getFrameContentSize functions the same way, returning the decompressed size of a single + frame, but distinguishes empty frames from frames with an unknown size, or errors. - Additionally, ZSTD_findDecompressedSize can be used instead. It can handle multiple - concatenated frames in one buffer, and so is more general. - As a result however, it requires more computation and entire frames to be passed to it, - as opposed to ZSTD_getFrameContentSize which requires only a single frame's header. + Additionally, ZSTD_findDecompressedSize can be used instead. It can handle multiple + concatenated frames in one buffer, and so is more general. + As a result however, it requires more computation and entire frames to be passed to it, + as opposed to ZSTD_getFrameContentSize which requires only a single frame's header. - 'src' is the start of a zstd compressed frame. - @return : content size to be decompressed, as a 64-bits value _if known_, 0 otherwise. - note 1 : decompressed size is an optional field, that may not be present, especially in streaming mode. - When `return==0`, data to decompress could be any size. - In which case, it's necessary to use streaming mode to decompress data. - Optionally, application can still use ZSTD_decompress() while relying on implied limits. - (For example, data may be necessarily cut into blocks <= 16 KB). - note 2 : decompressed size is always present when compression is done with ZSTD_compress() - note 3 : decompressed size can be very large (64-bits value), - potentially larger than what local system can handle as a single memory segment. - In which case, it's necessary to use streaming mode to decompress data. - note 4 : If source is untrusted, decompressed size could be wrong or intentionally modified. - Always ensure result fits within application's authorized limits. - Each application can set its own limits. - note 5 : when `return==0`, if precise failure cause is needed, use ZSTD_getFrameParams() to know more. + 'src' is the start of a zstd compressed frame. + @return : content size to be decompressed, as a 64-bits value _if known_, 0 otherwise. + note 1 : decompressed size is an optional field, that may not be present, especially in streaming mode. + When `return==0`, data to decompress could be any size. + In which case, it's necessary to use streaming mode to decompress data. + Optionally, application can still use ZSTD_decompress() while relying on implied limits. + (For example, data may be necessarily cut into blocks <= 16 KB). + note 2 : decompressed size is always present when compression is done with ZSTD_compress() + note 3 : decompressed size can be very large (64-bits value), + potentially larger than what local system can handle as a single memory segment. + In which case, it's necessary to use streaming mode to decompress data. + note 4 : If source is untrusted, decompressed size could be wrong or intentionally modified. + Always ensure result fits within application's authorized limits. + Each application can set its own limits. + note 5 : when `return==0`, if precise failure cause is needed, use ZSTD_getFrameParams() to know more. </p></pre><BR> <h3>Helper functions</h3><pre></pre><b><pre>int ZSTD_maxCLevel(void); </b>/*!< maximum compression level available */<b> @@ -106,28 +106,28 @@ const char* ZSTD_getErrorName(size_t code); </b>/*!< provides readable strin </pre></b><BR> <a name="Chapter4"></a><h2>Explicit memory management</h2><pre></pre> -<h3>Compression context</h3><pre> When compressing many times, - it is recommended to allocate a context just once, and re-use it for each successive compression operation. - This will make workload friendlier for system's memory. - Use one context per thread for parallel execution in multi-threaded environments. +<h3>Compression context</h3><pre> When compressing many times, + it is recommended to allocate a context just once, and re-use it for each successive compression operation. + This will make workload friendlier for system's memory. + Use one context per thread for parallel execution in multi-threaded environments. </pre><b><pre>typedef struct ZSTD_CCtx_s ZSTD_CCtx; ZSTD_CCtx* ZSTD_createCCtx(void); size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx); </pre></b><BR> <pre><b>size_t ZSTD_compressCCtx(ZSTD_CCtx* ctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, int compressionLevel); -</b><p> Same as ZSTD_compress(), requires an allocated ZSTD_CCtx (see ZSTD_createCCtx()). +</b><p> Same as ZSTD_compress(), requires an allocated ZSTD_CCtx (see ZSTD_createCCtx()). </p></pre><BR> -<h3>Decompression context</h3><pre> When decompressing many times, - it is recommended to allocate a context just once, and re-use it for each successive compression operation. - This will make workload friendlier for system's memory. - Use one context per thread for parallel execution in multi-threaded environments. +<h3>Decompression context</h3><pre> When decompressing many times, + it is recommended to allocate a context just once, and re-use it for each successive compression operation. + This will make workload friendlier for system's memory. + Use one context per thread for parallel execution in multi-threaded environments. </pre><b><pre>typedef struct ZSTD_DCtx_s ZSTD_DCtx; ZSTD_DCtx* ZSTD_createDCtx(void); size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); </pre></b><BR> <pre><b>size_t ZSTD_decompressDCtx(ZSTD_DCtx* ctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); -</b><p> Same as ZSTD_decompress(), requires an allocated ZSTD_DCtx (see ZSTD_createDCtx()). +</b><p> Same as ZSTD_decompress(), requires an allocated ZSTD_DCtx (see ZSTD_createDCtx()). </p></pre><BR> <a name="Chapter5"></a><h2>Simple dictionary API</h2><pre></pre> @@ -169,9 +169,10 @@ size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); void* dst, size_t dstCapacity, const void* src, size_t srcSize, const ZSTD_CDict* cdict); -</b><p> Compression using a digested Dictionary. - Faster startup than ZSTD_compress_usingDict(), recommended when same dictionary is used multiple times. - Note that compression level is decided during dictionary creation. +</b><p> Compression using a digested Dictionary. + Faster startup than ZSTD_compress_usingDict(), recommended when same dictionary is used multiple times. + Note that compression level is decided during dictionary creation. + Frame parameters are hardcoded (dictID=yes, contentSize=yes, checksum=no) </p></pre><BR> <pre><b>ZSTD_DDict* ZSTD_createDDict(const void* dictBuffer, size_t dictSize); @@ -399,7 +400,7 @@ typedef struct { ZSTD_allocFunction customAlloc; ZSTD_freeFunction customFree; v </p></pre><BR> <pre><b>ZSTD_CDict* ZSTD_createCDict_advanced(const void* dict, size_t dictSize, unsigned byReference, - ZSTD_parameters params, ZSTD_customMem customMem); + ZSTD_compressionParameters cParams, ZSTD_customMem customMem); </b><p> Create a ZSTD_CDict using external alloc and free, and customized compression parameters </p></pre><BR> @@ -426,12 +427,19 @@ typedef struct { ZSTD_allocFunction customAlloc; ZSTD_freeFunction customFree; v both values are optional, select `0` if unknown. </p></pre><BR> -<pre><b>size_t ZSTD_compress_advanced (ZSTD_CCtx* ctx, - void* dst, size_t dstCapacity, - const void* src, size_t srcSize, - const void* dict,size_t dictSize, - ZSTD_parameters params); -</b><p> Same as ZSTD_compress_usingDict(), with fine-tune control of each compression parameter +<pre><b>size_t ZSTD_compress_advanced (ZSTD_CCtx* cctx, + void* dst, size_t dstCapacity, + const void* src, size_t srcSize, + const void* dict,size_t dictSize, + ZSTD_parameters params); +</b><p> Same as ZSTD_compress_usingDict(), with fine-tune control over each compression parameter +</p></pre><BR> + +<pre><b>size_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx, + void* dst, size_t dstCapacity, + const void* src, size_t srcSize, + const ZSTD_CDict* cdict, ZSTD_frameParameters fParams); +</b><p> Same as ZSTD_compress_usingCDict(), with fine-tune control over frame parameters </p></pre><BR> <a name="Chapter15"></a><h2>Advanced decompression functions</h2><pre></pre> @@ -491,20 +499,29 @@ typedef struct { ZSTD_allocFunction customAlloc; ZSTD_freeFunction customFree; v Note : this use case also happens when using a non-conformant dictionary. - `srcSize` is too small, and as a result, the frame header could not be decoded (only possible if `srcSize < ZSTD_FRAMEHEADERSIZE_MAX`). - This is not a Zstandard frame. - When identifying the exact failure cause, it's possible to used ZSTD_getFrameParams(), which will provide a more precise error code. + When identifying the exact failure cause, it's possible to use ZSTD_getFrameParams(), which will provide a more precise error code. </p></pre><BR> <a name="Chapter16"></a><h2>Advanced streaming functions</h2><pre></pre> <h3>Advanced Streaming compression functions</h3><pre></pre><b><pre>ZSTD_CStream* ZSTD_createCStream_advanced(ZSTD_customMem customMem); +size_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs); </b>/**< size of CStream is variable, depending primarily on compression level */<b> size_t ZSTD_initCStream_srcSize(ZSTD_CStream* zcs, int compressionLevel, unsigned long long pledgedSrcSize); </b>/**< pledgedSrcSize must be correct, a size of 0 means unknown. for a frame size of 0 use initCStream_advanced */<b> size_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs, const void* dict, size_t dictSize, int compressionLevel); </b>/**< note: a dict will not be used if dict == NULL or dictSize < 8 */<b> size_t ZSTD_initCStream_advanced(ZSTD_CStream* zcs, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize); </b>/**< pledgedSrcSize is optional and can be 0 (meaning unknown). note: if the contentSizeFlag is set, pledgedSrcSize == 0 means the source size is actually 0 */<b> size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict); </b>/**< note : cdict will just be referenced, and must outlive compression session */<b> -size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pledgedSrcSize); </b>/**< re-use compression parameters from previous init; skip dictionary loading stage; zcs must be init at least once before. note: pledgedSrcSize must be correct, a size of 0 means unknown. for a frame size of 0 use initCStream_advanced */<b> -size_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs); +size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs, const ZSTD_CDict* cdict, unsigned long long pledgedSrcSize, ZSTD_frameParameters fParams); </b>/**< same as ZSTD_initCStream_usingCDict(), with control over frame parameters */<b> </pre></b><BR> +<pre><b>size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pledgedSrcSize); +</b><p> start a new compression job, using same parameters from previous job. + This is typically useful to skip dictionary loading stage, since it will re-use it in-place.. + Note that zcs must be init at least once before using ZSTD_resetCStream(). + pledgedSrcSize==0 means "srcSize unknown". + If pledgedSrcSize > 0, its value must be correct, as it will be written in header, and controlled at the end. + @return : 0, or an error code (which can be tested using ZSTD_isError()) +</p></pre><BR> + <h3>Advanced Streaming decompression functions</h3><pre></pre><b><pre>typedef enum { DStream_p_maxWindowSize } ZSTD_DStreamParameter_e; ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem); size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize); </b>/**< note: a dict will not be used if dict == NULL or dictSize < 8 */<b> @@ -552,10 +569,9 @@ size_t ZSTD_sizeof_DStream(const ZSTD_DStream* zds); <h3>Buffer-less streaming compression functions</h3><pre></pre><b><pre>size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel); size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel); size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize); </b>/**< pledgedSrcSize is optional and can be 0 (meaning unknown). note: if the contentSizeFlag is set, pledgedSrcSize == 0 means the source size is actually 0 */<b> +size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict); </b>/**< note: fails if cdict==NULL */<b> +size_t ZSTD_compressBegin_usingCDict_advanced(ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict, ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize); </b>/* compression parameters are already set within cdict. pledgedSrcSize=0 means null-size */<b> size_t ZSTD_copyCCtx(ZSTD_CCtx* cctx, const ZSTD_CCtx* preparedCCtx, unsigned long long pledgedSrcSize); </b>/**< note: if pledgedSrcSize can be 0, indicating unknown size. if it is non-zero, it must be accurate. for 0 size frames, use compressBegin_advanced */<b> -size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict, unsigned long long pledgedSrcSize); </b>/**< note: if pledgedSrcSize can be 0, indicating unknown size. if it is non-zero, it must be accurate. for 0 size frames, use compressBegin_advanced */<b> -size_t ZSTD_compressContinue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); -size_t ZSTD_compressEnd(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); </pre></b><BR> <a name="Chapter19"></a><h2>Buffer-less streaming decompression (synchronous mode)</h2><pre> A ZSTD_DCtx object is required to track streaming operations. @@ -640,19 +656,20 @@ ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx* dctx); - Compressing and decompressing require a context structure + Use ZSTD_createCCtx() and ZSTD_createDCtx() - It is necessary to init context before starting - + compression : ZSTD_compressBegin() - + decompression : ZSTD_decompressBegin() - + variants _usingDict() are also allowed - + copyCCtx() and copyDCtx() work too - - Block size is limited, it must be <= ZSTD_getBlockSizeMax() - + If you need to compress more, cut data into multiple blocks - + Consider using the regular ZSTD_compress() instead, as frame metadata costs become negligible when source size is large. + + compression : any ZSTD_compressBegin*() variant, including with dictionary + + decompression : any ZSTD_decompressBegin*() variant, including with dictionary + + copyCCtx() and copyDCtx() can be used too + - Block size is limited, it must be <= ZSTD_getBlockSizeMax() <= ZSTD_BLOCKSIZE_ABSOLUTEMAX + + If input is larger than a block size, it's necessary to split input data into multiple blocks + + For inputs larger than a single block size, consider using the regular ZSTD_compress() instead. + Frame metadata is not that costly, and quickly becomes negligible as source size grows larger. - When a block is considered not compressible enough, ZSTD_compressBlock() result will be zero. In which case, nothing is produced into `dst`. + User must test for such outcome and deal directly with uncompressed data + ZSTD_decompressBlock() doesn't accept uncompressed data as input !!! - + In case of multiple successive blocks, decoder must be informed of uncompressed block existence to follow proper history. - Use ZSTD_insertBlock() in such a case. + + In case of multiple successive blocks, should some of them be uncompressed, + decoder must be informed of their existence in order to follow proper history. + Use ZSTD_insertBlock() for such a case. <BR></pre> <h3>Raw zstd block functions</h3><pre></pre><b><pre>size_t ZSTD_getBlockSizeMax(ZSTD_CCtx* cctx); |
