414 lines
21 KiB
XML
414 lines
21 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
|
|
"http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd">
|
|
<article lang="">
|
|
<articleinfo>
|
|
<title><inlinemediaobject>
|
|
<imageobject>
|
|
<imagedata align="center" fileref="NormLogo.gif" scale="50"/>
|
|
</imageobject>
|
|
</inlinemediaobject> <command>npc</command> User Guide</title>
|
|
|
|
<subtitle>(NORM Precoder User Guide)</subtitle>
|
|
|
|
<titleabbrev><command>npc</command> User Guide</titleabbrev>
|
|
</articleinfo>
|
|
|
|
<sect1>
|
|
<title>Background</title>
|
|
|
|
<para>The NACK-Oriented Reliable Multicast (NORM) protocol is capable of
|
|
supporting robust transmission of content to "silent" receivers that are
|
|
required or only capable of operating in an emission-controlled (EMCON)
|
|
manner. This capability is enabled when the NORM sender is configured to
|
|
proactively transmit Forward Error Correction (FEC) erasure coding content
|
|
as part of its original data transmission. For NACK-based operation, the
|
|
FEC repair packets are usually sent only reactively, in response to repair
|
|
requests (NACKs) from the receiver group. However, hybrid operation with a
|
|
combination of proactive FEC content and additional reactive FEC repairs
|
|
as needed is also supported. Similarly, a mix of nacking and silent
|
|
receivers may be supported with silent receivers capitalizing on the FEC
|
|
repair information sent proactively and/or reactively. The purpose of the
|
|
NORM Pre-Coder (<command>npc</command>) software utility described here is
|
|
to support additional robustness for purely-proactive sessions, where the
|
|
receivers are unable to request repair or retransmission of
|
|
content.</para>
|
|
|
|
<para>The Naval Research Laboratory (NRL) reference implementation of the
|
|
NORM protocol includes support for 8-bit and 16-bit Reed-Solomon FEC
|
|
encoding with additional support for other coding algorithms (e.g.,
|
|
Low-Density Parity Check (LDPC)) planned for the future. The NORM
|
|
specification allows for different FEC algorithms to be applied within the
|
|
protocol. The current Reed-Solomon NORM FEC algorithms in the NRL
|
|
implementation are limited to modest code block sizes (With 16-bit
|
|
Reed-Solomon coding, larger block sizes will be allowed but very high data
|
|
rates may not be possible). For channels with random errors, the current
|
|
NORM FEC codecs are often adequate as there is flexibility in how the
|
|
encoded data can be partitioned into FEC blocks (a block consists of some
|
|
number of data segments (packets)) and the number of FEC parity packets
|
|
that can computed and possibly transmitted per source data block. For
|
|
channels with large bursts of packet loss (with respect to the configured
|
|
NORM FEC block size), it is quite possible that the number of lost packets
|
|
(erasures) that occur within a NORM FEC block may exceed the configured
|
|
erasure-filling capability. The <command>npc</command> utility was created
|
|
to "pre-encode" (and "post-decode") files for NORM transmission to silent
|
|
(non- NACKing) receivers by adding additional FEC encoding, and
|
|
importantly, interleaving of the FEC segments (packets) to re-distribute
|
|
bursts of packet losses as random losses over the entire file. It is thus
|
|
most applicable to very large files (with respect to FEC block
|
|
sizes).</para>
|
|
|
|
<para>The NORM protocol is described in Internet Engineering Task Force
|
|
(IETF) Request For Comments (RFC) RFC 5740 and RFC 5741. NRL provides a
|
|
NORM protocol library with a well-defined API that it is suitable for
|
|
application development. Refer to the NORM website <<ulink
|
|
url="https://github.com/USNavalResearchLaboratory/norm">https://github.com/USNavalResearchLaboratory/norm</ulink>>
|
|
for these other components as well as up-to-date versions of this
|
|
demonstration application and documentation.</para>
|
|
|
|
<para>The <command>npc</command> tool is designed to use in conjunction
|
|
with the NORM protocol and accompanying NORM file transfer examples that
|
|
are part of the NORM source code distribution. However, the encoded file
|
|
format that <command>npc</command> creates can be use with other
|
|
transports as well. The key is to align the segmentation parameters of the
|
|
<command>npc</command> configuration with that of the intended transport
|
|
mechanism.</para>
|
|
</sect1>
|
|
|
|
<sect1>
|
|
<title>Overview</title>
|
|
|
|
<para>The <command>npc</command> utility takes, as input", a file and
|
|
logically divides it into segments, adding cyclic-redundancy checksum
|
|
(CRC) to the segments, encoding the source segments with Reeed-Solomon
|
|
encoding (adding a configurable number of parity segments per FEC source
|
|
block), and interleaves the source and encoding segments to an output
|
|
file. The use of the CRC allows erasure to be detected and also provides
|
|
additional assurance of correct content delivery by possibly detecting bit
|
|
errors that may have been undetected during transport (i.e., link-layer
|
|
framing, Internet Protocol (IP), and/or User Datagram Protocol (UDP)
|
|
checksums). The interleaving by default is a block interleaver using the
|
|
entire file as a logical block, but a limit on the interleaving size can
|
|
be set to help increase the speed of the <command>npc</command>
|
|
encoding/decoding process. This may be useful for extremely large file
|
|
sizes.</para>
|
|
</sect1>
|
|
|
|
<sect1>
|
|
<title>Usage</title>
|
|
|
|
<para>The following is a synopsis of <command>npc</command> usage:</para>
|
|
|
|
<programlisting>npc {encode|decode} input <inFile> [output <outFile>][segment <segmentSize>]
|
|
{[[block <numData>][parity <numParity>]] |
|
|
[auto <parityPercentage>][bmax <maxBlockSize>]}
|
|
[imax <widthLimit>][ibuffer <bytes>][background][help][debug <debugLevel>]</programlisting>
|
|
|
|
<para>The <command>npc</command> utility may be instructed to either
|
|
"encode" a file (add FEC content and interleaving to the given
|
|
<inFile>) or "decode" a file that was previously encoded with
|
|
<command>npc</command>. The ".npc" file extension is suggested to
|
|
delineate files that are of the <command>npc</command> encoded format.
|
|
Note the "output" filename is optional. By default, <command>npc</command>
|
|
will use the filename of the <inFile> as the output filename,
|
|
replacing the '.' extension delimiter with a '_' (underscore) and adding
|
|
the ".npc" extension suffix. The <command>npc</command> format includes
|
|
some minimal "meta-data" in the first encoded <segmentSize> to
|
|
convey the file size and name of the original file. On decoding, if the
|
|
"output" file option is omitted, this "meta-data" is used to name the
|
|
decoded output file.</para>
|
|
|
|
<para>The optional FEC parameters,
|
|
<parameter><segmentSize></parameter>,
|
|
<parameter><numData></parameter>, and
|
|
<parameter><numParity></parameter> control the logical segmentation,
|
|
blocking, and amount of FEC parity content added to the file. For use with
|
|
NORM, it is recommended that the
|
|
<parameter><segmentSize></parameter> value correspond to the same
|
|
segmentation size used for NORM transmission. The
|
|
<parameter><numData></parameter> (source segments per FEC encoding
|
|
block) and <parameter><numParity></parameter> parameters should be
|
|
selected to provide erasure filling coverage for the expected transmission
|
|
packet loss characteristics. Note that when used with proactive NORM FEC
|
|
transmission, the <command>npc</command> encoding provides an "inner" FEC
|
|
code and interleaving and the NORM protocol provides an "outer" FEC
|
|
encoding. The "outer" NORM code might be configured to deal with typical
|
|
random packet loss due to channel BER, etc and the "inner"
|
|
<command>npc</command> interleaving and coding could be correspondingly
|
|
configured to handle expected burst losses (e.g. outages) that might
|
|
occur.</para>
|
|
|
|
<para>The <parameter><auto></parameter> option provides an
|
|
alternative means for setting the FEC encoding protection level instead of
|
|
using the <parameter><block></parameter> and
|
|
<parameter><parity></parameter> options. First the
|
|
<parameter><auto></parameter> option causes <command>npc</command>
|
|
to select a block size corresponding to the entire input file size (plus
|
|
the segment of meta data information that npc adds). Then, the
|
|
<parameter><auto></parameter> option
|
|
<parameter><parityPercentage></parameter> value is used to set the
|
|
number of parity packets per encoding block to the given percentage of the
|
|
automatically determined block size. For exanple, the command
|
|
"<option>auto 100"</option> causes <command>npc</command> to set an
|
|
encoding rate of 100%. I.e., the parity segments sent will equal the
|
|
number of segments in a bock. Note that the
|
|
<parameter><parityPercentage></parameter> can even exceed 100% if
|
|
desired for high levels of loss protection. Also note that the percentage
|
|
here is _not_ a loss protection percentage with a 100%
|
|
<parameter><parityPercentage></parameter> value being able to
|
|
correct up to 50% errored or lost packets within a coding block. With 50%
|
|
uniform random packet loss, this would result in successful file transfer
|
|
about 50% of the time as, per Gaussian distribution, burst error
|
|
probabilities would result in half of blocks arriving with greater than
|
|
50% packet loss and half with less than 50% packet loss.</para>
|
|
|
|
<para>When the <command>npc</command> encoder uses the
|
|
"<option>auto</option>" command the <command>npc</command> decoder MUST
|
|
also use the "<option>auto</option>" command and be configured with the
|
|
same <parameter><segmentSize></parameter> and
|
|
<parameter><parityPercentage></parameter> values. Similarly when the
|
|
"<option>block</option>" and "<option>parity</option>" commands are used
|
|
to explicitly set the <parameter><numdData></parameter> and
|
|
<parameter><numParity></parameter> at the encoder, the decoder MUST
|
|
be configured with the same corresponding options and values, again
|
|
including <parameter><segmentSize></parameter>. And for use with
|
|
NORM protocol transport, the <parameter><segmentSize></parameter>
|
|
parameter SHOULD be matched for best coding gain performance. The NORM
|
|
block size (numData) and parity (numParity) parameters may be set
|
|
indepedently. Basically, the NORM protocol proactive erasure coding can be
|
|
configured to deal with expected short term random packet loss while the
|
|
<command>npc</command> parameters can be configured to counter large burst
|
|
(or outage) losses. The inner/outer encoding approach that the combination
|
|
of <command>npc</command> and NORM provides, can allow for a sort of
|
|
multiplicative coding gain to deal well with both random packet loss and
|
|
bursts or outages with lower FEC overhead. However, when the
|
|
<command>npc</command> coding is configured (e.g., via the "auto" option)
|
|
to encapsulate an entire file into a single logical coding block, the
|
|
desired level of loss protection can be simply "dialed into" the
|
|
<command>npc</command> <parameter><parityPercentage></parameter>
|
|
option. The tradeoff is that the larger FEC block size increases the
|
|
computational requirement for file encoding and decoding. Future versions
|
|
of <command>npc</command> may provide additional FEC code types</para>
|
|
|
|
<para>As basic example usage, to encode a file name "originalFile.txt"
|
|
with the default <command>npc</command> naming convention, FEC, and
|
|
interleaving parameters, use the following syntax:</para>
|
|
|
|
<programlisting>npc encode input originalFile.txt</programlisting>
|
|
|
|
<para>This will produce and output file named
|
|
"<filename>originalFile_txt.npc</filename>" in the current working
|
|
directory. The default <command>npc</command> configuration is
|
|
"<command>auto 100.0</command>" providing 100% parity content which makes
|
|
the encoded file size roughly double the input file. The original file can
|
|
be recovered (decoded) using the syntax:</para>
|
|
|
|
<programlisting>npc decode input originalFile_txt.npc</programlisting>
|
|
|
|
<para>This will decode the ".npc" file, and in this case produce a file
|
|
named "<filename>originalFile.txt"</filename> in the current working
|
|
directory. (The file name information was stored in first "meta data"
|
|
segment of the ".npc" file). This default naming convention can be
|
|
overridden by using the <command>npc</command> "<option>output</option>"
|
|
command. For example, the syntax:</para>
|
|
|
|
<programlisting>npc decode input originalFile_txt.npc output file.txt</programlisting>
|
|
|
|
<para>will produce a file named "<filename>file.txt</filename>" that is
|
|
identical in content to "<filename>originalFile.txt</filename>".</para>
|
|
|
|
<sect2>
|
|
<title>Notes</title>
|
|
|
|
<para>The FEC and interleaving parameters that are used for
|
|
<command>npc</command> encoding MUST be exactly matched to successfully
|
|
decode the encoded file. I.e., if the defaults are used for encoding,
|
|
the defaults must be used for decoding. The parameters that must match
|
|
include <parameter><segmentSize></parameter>,
|
|
<parameter><parityPercentage></parameter> and
|
|
<parameter><maxBlockSize></parameter> (or
|
|
<parameter><numData></parameter> and
|
|
<parameter><numParity></parameter>), and
|
|
<parameter><widthMax></parameter>.</para>
|
|
|
|
<para>It is possible that in some cases it may be beneficial to apply
|
|
more proactive FEC content with the <command>npc</command> program
|
|
instead of with the NORM transport. The trade-offs are
|
|
scenario-specific.</para>
|
|
|
|
<para>The NRL "<command>norm</command>" demonstration application has
|
|
commands included to support transport of <command>npc</command> encoded
|
|
files. The distinction here is that a file that _fails_ NORM transport
|
|
might still be successfully decoded with <command>npc</command>. There
|
|
are two receiver-side <command>norm</command> demo application options
|
|
that apply here:</para>
|
|
|
|
<orderedlist>
|
|
<listitem>
|
|
<para>The "<option>saveAborts</option>" command causes
|
|
<command>norm</command> to not delete (and attempt to postprocess)
|
|
"aborted" files (files that failed reliable NORM transport).</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>The <command>norm</command> "<option>lowDelay</option>"
|
|
command should be applied for silent-receivers to more immediately
|
|
deliver "failed" files to the application for post-processing (i.e.,
|
|
attempted <command>npc</command> decoding)</para>
|
|
</listitem>
|
|
</orderedlist>
|
|
</sect2>
|
|
</sect1>
|
|
|
|
<sect1>
|
|
<title><command>npc</command> Command Reference</title>
|
|
|
|
<para>The following table describes each of the <command>npc</command>
|
|
commands available in the command-line syntax.</para>
|
|
|
|
<informaltable frame="all">
|
|
<tgroup cols="2">
|
|
<colspec colnum="1" colwidth="1*"/>
|
|
|
|
<colspec colname="2" colwidth="2*"/>
|
|
|
|
<tbody>
|
|
<row>
|
|
<entry><para><option>encode</option> |
|
|
<option>decode</option></para></entry>
|
|
|
|
<entry><para>Determine whether <command>npc</command> is to encode
|
|
or decode the given <parameter><inFile></parameter>. This
|
|
option is required and only one should be given.</para></entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><para><option>input
|
|
</option><parameter><inFile></parameter></para></entry>
|
|
|
|
<entry><para>Specifies the file to be processed. Required
|
|
command.</para></entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><para><option>output
|
|
</option><parameter><outFile></parameter></para></entry>
|
|
|
|
<entry><para>Specifies the name of the output file to be produced.
|
|
Overrides the default <command>npc</command> output file naming
|
|
convention. Optional.</para></entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><para><option>segment
|
|
</option><parameter><segmentSize></parameter></para></entry>
|
|
|
|
<entry><para>Sets the segmentation size (e.g., packet payload
|
|
size) in bytes. Note four bytes of the
|
|
<parameter><segmentSize></parameter> are used for a 32-bit
|
|
CRC that <command>npc</command> applies to each segment. (Default
|
|
<parameter><segmentSize></parameter> is 1024
|
|
bytes)</para></entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><para><option>block
|
|
</option><parameter><numData></parameter></para></entry>
|
|
|
|
<entry><para>Specify the number of source data segments (packets)
|
|
per <command>npc</command> FEC coding block. (Default block sizing
|
|
is auto)</para></entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><para><option>parity
|
|
</option><parameter><numParity></parameter></para></entry>
|
|
|
|
<entry><para>Specify the number of FEC parity segments (packets)
|
|
added per <command>npc</command> FEC coding block. (Default is 2
|
|
segments).</para></entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><para><option>auto
|
|
</option><parameter><parityPercentage></parameter></para></entry>
|
|
|
|
<entry><para>Specifies automatic FEC block sizing with
|
|
<parameter><parityPercentage></parameter> indicating the
|
|
percentage of FEC parity segments to include per block. The "auto"
|
|
block sizing sets the block size as large as possible to treat the
|
|
entire files as one logical FEC block to maximize FEC performance.
|
|
The maximum possible block size currently supported by
|
|
<command>npc</command> are blocks where (numData + numParity) is
|
|
less than or equal to 65536. The maximum buffer size can be
|
|
limited by using the <option>bmax</option> command.</para></entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><para><option>bmax
|
|
</option><parameter><maxBlockSize></parameter></para></entry>
|
|
|
|
<entry><para>Limits the maximum block size when the
|
|
<option>auto</option> command is used for automatic block sizing
|
|
(Default is 65536)</para></entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><para><option>imax
|
|
</option><parameter><widthMax></parameter></para></entry>
|
|
|
|
<entry><para>Limits interleaving of encoded file to a maximum
|
|
interleaver width of <parameter><widthMax></parameter>
|
|
segments. A value of ZERO (or less) defaults to
|
|
<command>npc</command> calculating a block interleaver that
|
|
encompasses the entire encoded file size. For extremely large
|
|
files, this option may be beneficial to limit file seeking
|
|
operations required to interleave the file. If the encoded file
|
|
size is less than
|
|
<parameter><widthMax></parameter>*<parameter><widthMax></parameter>
|
|
segments, <command>npc</command> will again calculate its own
|
|
maximum block size. (Default is 1000 segments interleaver depth
|
|
(i.e., about 1 Gbyte interleaver size with the default 1024 byte
|
|
<parameter><segmentSize></parameter> value))</para></entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><para><option>ibuffer
|
|
</option><parameter><bufferSize></parameter></para></entry>
|
|
|
|
<entry><para>This sets the maximum memory (in bytes) that
|
|
<command>npc</command> allocates for encoding. A larger value
|
|
allows <command>npc</command> to perform file input/output with
|
|
less seeking and improved encoding/decoding times can be achieved.
|
|
(Default is 1.5 GByte)</para></entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><para><option>debug
|
|
</option><parameter><debugLevel></parameter></para></entry>
|
|
|
|
<entry><para>Specifies debug output verbosity. Higher number is
|
|
more verbose debugging information. (Default is
|
|
ZERO).</para></entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><para><option>background</option></para></entry>
|
|
|
|
<entry><para>Sets percentage of received messages that are
|
|
randomly dropped (for testing purposes). Default = 0.0
|
|
percent.</para></entry>
|
|
</row>
|
|
|
|
<row>
|
|
<entry><para><option>help</option></para></entry>
|
|
|
|
<entry><para>Displays <command>npc</command> usage
|
|
statement.</para></entry>
|
|
</row>
|
|
</tbody>
|
|
</tgroup>
|
|
</informaltable>
|
|
</sect1>
|
|
</article>
|