BodgeMaster
/
NORM-mirror
mirror of https://github.com/USNavalResearchLaboratory/norm.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

fixes to npc (NormPrecoder) and updated its documentation. Include 'auto' block sizing' feature

pull/91/head
bebopagogo 2025-01-26 20:13:35 -05:00
parent 0c6e516c90
commit 7a6c09ae17
4 changed files with 664 additions and 150 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
doc/npcUserGuide.pdf Normal file
View File

Binary file not shown.

420
doc/npcUserGuide.xml Normal file
View File

@ -0,0 +1,420 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
"http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd">
<article lang="">
<articleinfo>
<title><inlinemediaobject>
<imageobject>
<imagedata align="center" fileref="NormLogo.gif" scale="50"/>
</imageobject>
</inlinemediaobject> <command>npc</command> User Guide</title>
<subtitle>(NORM Precoder User Guide)</subtitle>
<titleabbrev><command>npc</command> User Guide</titleabbrev>
</articleinfo>
<sect1>
<title>Background</title>
<para>The NACK-Oriented Reliable Multicast (NORM) protocol is capable of
supporting robust transmission of content to "silent" receivers that are
required or only capable of operating in an emission-controlled (EMCON)
manner. This capability is enabled when the NORM sender is configured to
proactively transmit Forward Error Correction (FEC) erasure coding content
as part of its original data transmission. For NACK-based operation, the
FEC repair packets are usually sent only reactively, in response to repair
requests (NACKs) from the receiver group. However, hybrid operation with a
combination of proactive FEC content and additional reactive FEC repairs
as needed is also supported. Similarly, a mix of nacking and silent
receivers may be supported with silent receivers capitalizing on the FEC
repair information sent proactively and/or reactively. The purpose of the
NORM Pre-Coder (<command>npc</command>) software utility described here is
to support additional robustness for purely-proactive sessions, where the
receivers are unable to request repair or retransmission of
content.</para>
<para>The Naval Research Laboratory (NRL) reference implementation of the
NORM protocol includes support for 8-bit (and very soon 16-bit)
Reed-Solomon FEC encoding with additional support for other coding
algorithms (e.g., Low-Density Parity Check (LDPC)) planned for the future.
The NORM specification allows for different FEC algorithms to be applied
within the protocol. The current Reed-Solomon NORM FEC algorithms in the
NRL implementation are limited to modest code block sizes (With 16-bit
Reed-Solomon coding, larger block sizes will be allowed but very high data
rates may not be possible). For channels with random errors, the current
NORM FEC codecs are often adequate as there is flexibility in how the
encoded data can be partitioned into FEC blocks (a block consists of some
number of data segments (packets)) and the number of FEC parity packets
that can computed and possibly transmitted per source data block. For
channels with large bursts of packet loss (with respect to the configured
NORM FEC block size), it is quite possible that the number of lost packets
(erasures) that occur within a NORM FEC block may exceed the configured
erasure-filling capability. The <command>npc</command> utility was created
to "pre-encode" (and "post-decode") files for NORM transmission to silent
(non- NACKing) receivers by adding additional FEC encoding, and
importantly, interleaving of the FEC segments (packets) to re-distribute
bursts of packet losses as random losses over the entire file. It is thus
most applicable to very large files (with respect to FEC block
sizes).</para>
<para>The NORM protocol is described in Internet Engineering Task Force
(IETF) Request For Comments (RFC) RFC 3940 and RFC 3941. These are
experimental RFC standards. These documents have been revised in recent
Internet-Drafts and it should be noted that the Naval Research Laboratory
(NRL) implementation of NORM that is represented here has been updated to
reflect the revised protocol. In addition to this demonstration
application, NRL provides a NORM protocol library with a well-defined API
that it is suitable for application development. Additionally, the NRL
source code distribution supports building the NORM protocol as a
component into <emphasis>ns-2</emphasis> and OPNET network simulation
environments. Refer to the NRL NORM website &lt;<ulink
url="http://cs.itd.nrl.navy.mil/work/norm">http://cs.itd.nrl.navy.mil/work/norm</ulink>&gt;
for these other components as well as up-to-date versions of this
demonstration application and documentation.</para>
<para>The <command>npc</command> tool is designed to use in conjunction
with the NORM protocol and accompanying NORM file transfer examples that
are part of the NORM source code distribution. However, the encoded file
format that <command>npc</command> creates can be use with other
transports as well. The key is to align the segmentation parameters of the
<command>npc</command> configuration with that of the intended transport
mechanism.</para>
</sect1>
<sect1>
<title>Overview</title>
<para>The <command>npc</command> utility takes, as input", a file and
logically divides it into segments, adding cyclic-redundancy checksum
(CRC) to the segments, encoding the source segments with Reeed-Solomon
encoding (adding a configurable number of parity segments per FEC source
block), and interleaves the source and encoding segments to an output
file. The use of the CRC allows erasure to be detected and also provides
additional assurance of correct content delivery by possibly detecting bit
errors that may have been undetected during transport (i.e., link-layer
framing, Internet Protocol (IP), and/or User Datagram Protocol (UDP)
checksums). The interleaving by default is a block interleaver using the
entire file as a logical block, but a limit on the interleaving size can
be set to help increase the speed of the <command>npc</command>
encoding/decoding process. This may be useful for extremely large file
sizes.</para>
</sect1>
<sect1>
<title>Usage</title>
<para>The following is a synopsis of <command>npc</command> usage:</para>
<programlisting>npc {encode|decode} input &lt;inFile&gt; [output &lt;outFile&gt;][segment &lt;segmentSize&gt;]
{[[block &lt;numData&gt;][parity &lt;numParity&gt;]] |
[auto &lt;parityPercentage&gt;][bmax &lt;maxBlockSize&gt;]}
[imax &lt;widthLimit&gt;][ibuffer &lt;bytes&gt;][background][help][debug &lt;debugLevel&gt;]</programlisting>
<para>The <command>npc</command> utility may be instructed to either
"encode" a file (add FEC content and interleaving to the given
&lt;inFile&gt;) or "decode" a file that was previously encoded with
<command>npc</command>. The ".npc" file extension is suggested to
delineate files that are of the <command>npc</command> encoded format.
Note the "output" filename is optional. By default, <command>npc</command>
will use the filename of the &lt;inFile&gt; as the output filename,
replacing the '.' extension delimiter with a '_' (underscore) and adding
the ".npc" extension suffix. The <command>npc</command> format includes
some minimal "meta-data" in the first encoded &lt;segmentSize&gt; to
convey the file size and name of the original file. On decoding, if the
"output" file option is omitted, this "meta-data" is used to name the
decoded output file.</para>
<para>The optional FEC parameters,
<parameter>&lt;segmentSize&gt;</parameter>,
<parameter>&lt;numData&gt;</parameter>, and
<parameter>&lt;numParity&gt;</parameter> control the logical segmentation,
blocking, and amount of FEC parity content added to the file. For use with
NORM, it is recommended that the
<parameter>&lt;segmentSize&gt;</parameter> value correspond to the same
segmentation size used for NORM transmission. The
<parameter>&lt;numData&gt;</parameter> (source segments per FEC encoding
block) and <parameter>&lt;numParity&gt;</parameter> parameters should be
selected to provide erasure filling coverage for the expected transmission
packet loss characteristics. Note that when used with proactive NORM FEC
transmission, the <command>npc</command> encoding provides an "inner" FEC
code and interleaving and the NORM protocol provides an "outer" FEC
encoding. The "outer" NORM code might be configured to deal with typical
random packet loss due to channel BER, etc and the "inner"
<command>npc</command> interleaving and coding could be correspondingly
configured to handle expected burst losses (e.g. outages) that might
occur.</para>
<para>The <parameter>&lt;auto&gt;</parameter> option provides an
alternative means for setting the FEC encoding protection level instead of
using the <parameter>&lt;block&gt;</parameter> and
<parameter>&lt;parity&gt;</parameter> options. First the
<parameter>&lt;auto&gt;</parameter> option causes <command>npc</command>
to select a block size corresponding to the entire input file size (plus
the segment of meta data information that npc adds). Then, the
<parameter>&lt;auto&gt;</parameter> option
<parameter>&lt;parityPercentage&gt;</parameter> value is used to set the
number of parity packets per encoding block to the given percentage of the
automatically determined block size. For exanple, the command
"<option>auto 100"</option> causes <command>npc</command> to set an
encoding rate of 100%. I.e., the parity segments sent will equal the
number of segments in a bock. Note that the
<parameter>&lt;parityPercentage&gt;</parameter> can even exceed 100% if
desired for high levels of loss protection. Also note that the percentage
here is _not_ a loss protection percentage with a 100%
<parameter>&lt;parityPercentage&gt;</parameter> value being able to
correct up to 50% errored or lost packets within a coding block. With 50%
uniform random packet loss, this would result in successful file transfer
about 50% of the time as, per Gaussian distribution, burst error
probabilities would result in half of blocks arriving with greater than
50% packet loss and half with less than 50% packet loss.</para>
<para>When the <command>npc</command> encoder uses the
"<option>auto</option>" command the <command>npc</command> decoder MUST
also use the "<option>auto</option>" command and be configured with the
same <parameter>&lt;segmentSize&gt;</parameter> and
<parameter>&lt;parityPercentage&gt;</parameter> values. Similarly when the
"<option>block</option>" and "<option>parity</option>" commands are used
to explicitly set the <parameter>&lt;numdData&gt;</parameter> and
<parameter>&lt;numParity&gt;</parameter> at the encoder, the decoder MUST
be configured with the same corresponding options and values, again
including <parameter>&lt;segmentSize&gt;</parameter>. And for use with
NORM protocol transport, the <parameter>&lt;segmentSize&gt;</parameter>
parameter SHOULD be matched for best coding gain performance. The NORM
block size (numData) and parity (numParity) parameters may be set
indepedently. Basically, the NORM protocol proactive erasure coding can be
configured to deal with expected short term random packet loss while the
<command>npc</command> parameters can be configured to counter large burst
(or outage) losses. The inner/outer encoding approach that the combination
of <command>npc</command> and NORM provides, can allow for a sort of
multiplicative coding gain to deal well with both random packet loss and
bursts or outages with lower FEC overhead. However, when the
<command>npc</command> coding is configured (e.g., via the "auto" option)
to encapsulate an entire file into a single logical coding block, the
desired level of loss protection can be simply "dialed into" the
<command>npc</command> <parameter>&lt;parityPercentage&gt;</parameter>
option. The tradeoff is that the larger FEC block size increases the
computational requirement for file encoding and decoding. Future versions
of <command>npc</command> may provide additional FEC code types</para>
<para>As basic example usage, to encode a file name "originalFile.txt"
with the default <command>npc</command> naming convention, FEC, and
interleaving parameters, use the following syntax:</para>
<programlisting>npc encode input originalFile.txt</programlisting>
<para>The default npc configuration is XXX.</para>
<para>This will produce and output file named
"<filename>originalFile_txt.npc</filename>" in the current working
directory, The original file can be recovered (decoded) using the
syntax:</para>
<programlisting>npc decode input originalFile_txt.npc</programlisting>
<para>This will decode the ".npc" file, and in this case produce a file
named "<filename>originalFile.txt"</filename> in the current working
directory. (The file name information was stored in first "meta data"
segment of the ".npc" file). This default naming convention can be
overridden by using the npc "output" option. For example, the
syntax:</para>
<programlisting>npc decode input originalFile_txt.npc output file.txt</programlisting>
<para>will produce a file named "<filename>file.txt</filename>" that is
identical in content to "<filename>originalFile.txt</filename>".</para>
<sect2>
<title>Notes</title>
<para>The FEC and interleaving parameters that are used for
<command>npc</command> encoding MUST be exactly matched to successfully
decode the encoded file. I.e., if the defaults are used for encoding,
the defaults must be used for decoding. The parameters that must match
include <parameter>&lt;segmentSize&gt;</parameter>,
<parameter>&lt;parityPercentage&gt;</parameter> and
<parameter>&lt;maxBlockSize&gt;</parameter> (or
<parameter>&lt;numData&gt;</parameter> and
<parameter>&lt;numParity&gt;</parameter>), and
<parameter>&lt;widthMax&gt;</parameter>.</para>
<para>It is possible that in some cases it may be beneficial to apply
more proactive FEC content with the <command>npc</command> program
instead of with the NORM transport. The trade-offs are
scenario-specific.</para>
<para>The NRL "<command>norm</command>" demonstration application has
commands included to support transport of <command>npc</command> encoded
files. The distinction here is that a file that _fails_ NORM transport
might still be successfully decoded with <command>npc</command>. There
are two receiver-side norm demo application options that apply
here:</para>
<orderedlist>
<listitem>
<para>The "<option>saveAborts</option>" command causes
<emphasis>norm</emphasis> to not delete (and attempt to postprocess)
"aborted" files (files that failed reliable NORM transport).</para>
</listitem>
<listitem>
<para>The <command>norm</command> "<option>lowDelay</option>"
command should be applied for silent-receivers to more immediately
deliver "failed" files to the application for post-processing (i.e.,
attempted <command>npc</command> decoding)</para>
</listitem>
</orderedlist>
</sect2>
</sect1>
<sect1>
<title><command>npc</command> Command Reference</title>
<para>The following table describes each of the <command>npc</command>
commands available in the command-line syntax.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colnum="1" colwidth="1*"/>
<colspec colname="2" colwidth="2*"/>
<tbody>
<row>
<entry><para><option>encode</option> |
<option>decode</option></para></entry>
<entry><para>Determine whether <command>npc</command> is to encode
or decode the given <parameter>&lt;inFile&gt;</parameter>. This
option is required and only one should be given.</para></entry>
</row>
<row>
<entry><para><option>input
</option><parameter>&lt;inFile&gt;</parameter></para></entry>
<entry><para>Specifies the file to be processed. Required
command.</para></entry>
</row>
<row>
<entry><para><option>output
</option><parameter>&lt;outFile&gt;</parameter></para></entry>
<entry><para>Specifies the name of the output file to be produced.
Overrides the default <command>npc</command> output file naming
convention. Optional.</para></entry>
</row>
<row>
<entry><para><option>segment
</option><parameter>&lt;segmentSize&gt;</parameter></para></entry>
<entry><para>Sets the segmentation size (e.g., packet payload
size) in bytes. Note four bytes of the
<parameter>&lt;segmentSize&gt;</parameter> are used for a 32-bit
CRC that <command>npc</command> applies to each segment. (Default
<parameter>&lt;segmentSize&gt;</parameter> is 1024
bytes)</para></entry>
</row>
<row>
<entry><para><option>block
</option><parameter>&lt;numData&gt;</parameter></para></entry>
<entry><para>Specify the number of source data segments (packets)
per npc FEC coding block. (Default block sizing is
auto)</para></entry>
</row>
<row>
<entry><para><option>parity
</option><parameter>&lt;numParity&gt;</parameter></para></entry>
<entry><para>Specify the number of FEC parity segments (packets)
added per npc FEC coding block. (Default is 2
segments).</para></entry>
</row>
<row>
<entry><para><option>auto
</option><parameter>&lt;parityPercentage&gt;</parameter></para></entry>
<entry><para>Specifies automatic FEC block sizing with
<parameter>&lt;parityPercentage&gt;</parameter> indicating the
percentage of FEC parity segments to include per block. The "auto"
block sizing sets the block size as large as possible to treat the
entire files as one logical FEC block to maximize FEC performance.
The maximum possible block size currently supported by npc are
blocks where (numData + numParity) is less than or equal to 65536.
The maximum buffer size can be limited by using the
<option>bmax</option> command.</para></entry>
</row>
<row>
<entry><para><option>bmax
</option><parameter>&lt;maxBlockSize&gt;</parameter></para></entry>
<entry><para>Limits the maximum block size when the
<option>auto</option> command is used for automatic block sizing
(Default is 65536)</para></entry>
</row>
<row>
<entry><para><option>imax
</option><parameter>&lt;widthMax&gt;</parameter></para></entry>
<entry><para>Limits interleaving of encoded file to a maximum
interleaver width of <parameter>&lt;widthMax&gt;</parameter>
segments. A value of ZERO (or less) defaults to
<command>npc</command> calculating a block interleaver that
encompasses the entire encoded file size. For extremely large
files, this option may be beneficial to limit file seeking
operations required to interleave the file. If the encoded file
size is less than
<parameter>&lt;widthMax&gt;</parameter>*<parameter>&lt;widthMax&gt;</parameter>
segments, <command>npc</command> will again calculate its own
maximum block size. (Default is 1000 segments interleaver depth
(i.e., about 1 Gbyte interleaver size with the default 1024 byte
<parameter>&lt;segmentSize&gt;</parameter> value))</para></entry>
</row>
<row>
<entry><para><option>ibuffer
</option><parameter>&lt;bufferSize&gt;</parameter></para></entry>
<entry><para>This sets the maximum memory (in bytes) that
<command>npc</command> allocates for encoding. A larger value
allows <command>npc</command> to perform file input/output with
less seeking and improved encoding/decoding times can be achieved.
(Default is 1.5 GByte)</para></entry>
</row>
<row>
<entry><para><option>debug
</option><parameter>&lt;debugLevel&gt;</parameter></para></entry>
<entry><para>Specifies debug output verbosity. Higher number is
more verbose debugging information. (Default is
ZERO).</para></entry>
</row>
<row>
<entry><para><option>background</option></para></entry>
<entry><para>Sets percentage of received messages that are
randomly dropped (for testing purposes). Default = 0.0
percent.</para></entry>
</row>
<row>
<entry><para><option>help</option></para></entry>
<entry><para>Displays <command>npc</command> usage
statement.</para></entry>
</row>
</tbody>
</tgroup>
</informaltable>
</sect1>
</article>

2
protolib

@ -1 +1 @@
Subproject commit a28606945f6b507760548b242b6706b51b5b69d3 Subproject commit e5ac157eb61e991c74acfe39e2febe8c09e187dc

392
src/common/normPrecode.cpp
View File

@ -1,6 +1,6 @@
#include "protoApp.h" #include "protoApp.h"
#include "normFile.h" #include "protoFile.h"
// Commment this #define out to use new, faster RS8 codec instead // Commment this #define out to use new, faster RS8 codec instead
// (TBD - provide option use 16-bit Reed Solomon for large block sizes?) // (TBD - provide option use 16-bit Reed Solomon for large block sizes?)
@ -10,6 +10,7 @@
#include "normEncoderMDP.h" #include "normEncoderMDP.h"
#else #else
#include "normEncoderRS8.h" #include "normEncoderRS8.h"
#include "normEncoderRS16.h"
#endif // if/else USE_MDP_FEC #endif // if/else USE_MDP_FEC
#include <sys/types.h> // for BYTE_ORDER macro #include <sys/types.h> // for BYTE_ORDER macro
@ -38,22 +39,22 @@ class NormPrecodeApp : public ProtoApp
bool Encode(); bool Encode();
bool Decode(); bool Decode();
void InitInterleaver(NormFile::Offset numSegments); void InitInterleaver(ProtoFile::Offset numSegments);
NormFile::Offset ComputeInterleaverOffset(NormFile::Offset segmentId, NormFile::Offset numSegments); ProtoFile::Offset ComputeInterleaverOffset(ProtoFile::Offset segmentId, ProtoFile::Offset numSegments);
NormFile::Offset ComputeSegmentOffset(NormFile::Offset interleaverId, NormFile::Offset numSegments); ProtoFile::Offset ComputeSegmentOffset(ProtoFile::Offset interleaverId, ProtoFile::Offset numSegments);
// CRC32 checksum stuff // CRC32 checksum stuff
static const UINT32 CRC32_TABLE[256]; static const UINT32 CRC32_TABLE[256];
static UINT32 ComputeCRC32(const char* buffer, unsigned int buflen); static UINT32 ComputeCRC32(const char* buffer, unsigned int buflen);
static const NormFile::Offset SEGMENT_MIN; static const ProtoFile::Offset SEGMENT_MIN;
static const NormFile::Offset SEGMENT_MAX; static const ProtoFile::Offset SEGMENT_MAX;
// We use these assuming IEEE754 floating point // We use these assuming IEEE754 floating point
static NormFile::Offset ntoho(NormFile::Offset offset) static ProtoFile::Offset ntoho(ProtoFile::Offset offset)
{ {
# if BYTE_ORDER == LITTLE_ENDIAN # if BYTE_ORDER == LITTLE_ENDIAN
NormFile::Offset result; ProtoFile::Offset result;
switch (sizeof(NormFile::Offset)) switch (sizeof(ProtoFile::Offset))
{ {
case 8: case 8:
{ {
@ -78,37 +79,40 @@ class NormPrecodeApp : public ProtoApp
return offset; return offset;
#endif // if/else __BIG_ENDIAN #endif // if/else __BIG_ENDIAN
} }
static NormFile::Offset htono(NormFile::Offset offset) static ProtoFile::Offset htono(ProtoFile::Offset offset)
{ {
return ntoho(offset); return ntoho(offset);
} }
NormFile in_file; ProtoFile in_file;
char in_file_path[PATH_MAX]; char in_file_path[PATH_MAX];
NormFile out_file; ProtoFile out_file;
bool encode; bool encode;
unsigned int segment_size; // should be same as NORM segment size unsigned int segment_size; // should be same as NORM segment size
unsigned int num_data; unsigned int num_data;
unsigned int num_parity; unsigned int num_parity;
double parity_fraction; // invokes "auto" block/parity sizing based on file size
ProtoFile::Offset b_max; // optional max block size for "auto" mode
NormFile::Offset i_max; // max interleaver dimension ProtoFile::Offset i_max; // max interleaver dimension
NormFile::Offset i_buffer_max; // Read buffer max (bigger yields less seeking) ProtoFile::Offset i_buffer_max; // Read buffer max (bigger yields less seeking)
NormFile::Offset interleaver_width; ProtoFile::Offset interleaver_width;
NormFile::Offset interleaver_height; ProtoFile::Offset interleaver_height;
NormFile::Offset interleaver_size; // (width * height) ProtoFile::Offset interleaver_size; // (width * height)
}; // end class NormPrecodeApp }; // end class NormPrecodeApp
// Our application instance // Our application instance
PROTO_INSTANTIATE_APP(NormPrecodeApp) PROTO_INSTANTIATE_APP(NormPrecodeApp)
const NormFile::Offset NormPrecodeApp::SEGMENT_MIN = 8; const ProtoFile::Offset NormPrecodeApp::SEGMENT_MIN = 8;
const NormFile::Offset NormPrecodeApp::SEGMENT_MAX = 8192; const ProtoFile::Offset NormPrecodeApp::SEGMENT_MAX = 8192;
NormPrecodeApp::NormPrecodeApp() NormPrecodeApp::NormPrecodeApp()
: encode(true), segment_size(1024), num_data(196), num_parity(4), : encode(true), segment_size(1024), num_data(196), num_parity(4),
parity_fraction(-1.0), b_max(65536),
i_max(1000), i_buffer_max(1500000000) i_max(1000), i_buffer_max(1500000000)
{ {
in_file_path[0] = '\0'; in_file_path[0] = '\0';
@ -122,6 +126,7 @@ void NormPrecodeApp::Usage()
{ {
fprintf(stderr, "Usage: npc {encode|decode} input <inFile> [output <outFile>]\n" fprintf(stderr, "Usage: npc {encode|decode} input <inFile> [output <outFile>]\n"
" [segment <segmentSize>][block numData][parity numParity]\n" " [segment <segmentSize>][block numData][parity numParity]\n"
" [auto <parityPercentage>]\n"
" [background][help][debug <debugLevel>\n"); " [background][help][debug <debugLevel>\n");
} // end NormPrecodeApp::Usage() } // end NormPrecodeApp::Usage()
@ -136,6 +141,8 @@ const char* const NormPrecodeApp::cmd_list[] =
"+segment", // set segment size (default = 1024) "+segment", // set segment size (default = 1024)
"+block", // set block size (default = 128) "+block", // set block size (default = 128)
"+parity", // set parity per block (default = 2) "+parity", // set parity per block (default = 2)
"+auto", // set auto block/parity mode (specifies parity_fraction as percent)
"+bmax", // limit maximum allowed block size for "auto" operation (default = 65536)
"+imax", // set interleaver max dimension "+imax", // set interleaver max dimension
"+ibuffer", // set imax interleaver buffer (buffer is used if interleaver size fits) "+ibuffer", // set imax interleaver buffer (buffer is used if interleaver size fits)
"-background", // run w/out command shel (Win32) "-background", // run w/out command shel (Win32)
@ -208,24 +215,45 @@ bool NormPrecodeApp::OnCommand(const char* cmd, const char* val)
} }
else if (!strncmp("block", cmd, len)) else if (!strncmp("block", cmd, len))
{ {
int numData = atoi(val); unsigned short numData;
if ((numData < 1) || (numData > 127)) if (1 != sscanf(val, "%hu", &numData))
{ {
PLOG(PL_FATAL, "npc: error: block <numData> out of range\n"); PLOG(PL_FATAL, "npc: error: invalid block <numData> value!\n");
return false; return false;
} }
num_data = numData; num_data = numData;
} }
else if (!strncmp("parity", cmd, len)) else if (!strncmp("parity", cmd, len))
{ {
int numParity = atoi(val); unsigned short numParity;
if ((numParity < 0) || (numParity > 127)) if (1 != sscanf(val, "%hu", &numParity))
{ {
PLOG(PL_FATAL, "npc: error: parity <numParity> out of range\n"); PLOG(PL_FATAL, "npc: error: invalid block <numParity> value!\n");
return false; return false;
} }
num_parity = numParity; num_parity = numParity;
} }
else if (!strncmp("auto", cmd, len))
{
double percent;
if (1 != sscanf(val,"%lf", &percent))
{
PLOG(PL_FATAL, "npc: error: invalid block <auto> value!\n");
return false;
}
if (percent < 0.0)
{
PLOG(PL_FATAL, "npc: error: invalid block <auto> value!\n");
return false;
}
parity_fraction = percent/100.0;
}
else if (!strncmp("bmax", cmd, len))
{
int bMax = atoi(val);
if (bMax <= 0) bMax = 65536;
b_max = bMax;
}
else if (!strncmp("imax", cmd, len)) else if (!strncmp("imax", cmd, len))
{ {
int iMax = atoi(val); int iMax = atoi(val);
@ -318,6 +346,49 @@ bool NormPrecodeApp::OnStartup(int argc, const char*const* argv)
return false; return false;
} }
if (parity_fraction >= 0.0)
{
// Set up auto mode using input file size
// (This uses the max block size possible for the file given "segment_size")
ProtoFile::Offset fileSize = in_file.GetSize();
unsigned int blockSize;
if (encode)
{
// (segment_size - 4) here accounts for 4-byte CRC
blockSize = (unsigned int) (fileSize / (segment_size - 4));
if (0 != (fileSize % (segment_size - 4)))
blockSize += 1;
blockSize += 1; // for meta data segment
}
else
{
// Need to use post decode size
ProtoFile::Offset numSegments = fileSize / segment_size;
ASSERT(0 == (fileSize % segment_size));
blockSize = (unsigned int)((numSegments / (1.0 + parity_fraction)) + 0.5);
}
if (blockSize > b_max) blockSize = b_max;
unsigned int numParity = (unsigned int)((parity_fraction * blockSize) + 0.5);
if ((blockSize + numParity) > 65536)
{
// need to scale down to fit within max possible FEC block size
double scaleFactor = 65536.0 / ((double)(blockSize + numParity));
blockSize = (unsigned int)(scaleFactor * blockSize);
numParity = (unsigned int)(scaleFactor * numParity);
}
//TRACE("auto fileSize:%llu block:%u parity:%u\n", fileSize, blockSize, numParity);
num_data = blockSize;
num_parity = numParity;
}
// Validate num_data / num_parity params
unsigned int totalBlockSize = num_data + num_parity;
if (totalBlockSize > 65536)
{
PLOG(PL_FATAL, "npc: error: numData/numParity total exceeds max block size!\n");
return false;
}
if (encode) if (encode)
return Encode(); return Encode();
else else
@ -335,9 +406,9 @@ void NormPrecodeApp::OnShutdown()
#define DIFF_T(a,b) (1+ 1000000*(a.tv_sec - b.tv_sec) + (a.tv_usec - b.tv_usec) ) #define DIFF_T(a,b) (1+ 1000000*(a.tv_sec - b.tv_sec) + (a.tv_usec - b.tv_usec) )
void NormPrecodeApp::InitInterleaver(NormFile::Offset numSegments) void NormPrecodeApp::InitInterleaver(ProtoFile::Offset numSegments)
{ {
interleaver_width = (NormFile::Offset)(sqrt((double)numSegments)); interleaver_width = (ProtoFile::Offset)(sqrt((double)numSegments));
interleaver_height = numSegments / interleaver_width; interleaver_height = numSegments / interleaver_width;
if (0 != (numSegments % interleaver_height)) interleaver_height++; if (0 != (numSegments % interleaver_height)) interleaver_height++;
// Limit dimension if "i_max" is set to non-zero value // Limit dimension if "i_max" is set to non-zero value
@ -345,19 +416,19 @@ void NormPrecodeApp::InitInterleaver(NormFile::Offset numSegments)
interleaver_height = interleaver_width = i_max; interleaver_height = interleaver_width = i_max;
interleaver_size = interleaver_height * interleaver_width; interleaver_size = interleaver_height * interleaver_width;
PLOG(PL_INFO, "npc interleaver width:%lu height:%lu segments (numSeg:%lld)\n", PLOG(PL_INFO, "npc interleaver width:%lu height:%lu segments (numSeg:%lld)\n",
(unsigned long)interleaver_width, (unsigned long)interleaver_height, numSegments); (unsigned long)interleaver_width, (unsigned long)interleaver_height, numSegments);
} // end NormPrecodeApp::InitInterleaver() } // end NormPrecodeApp::InitInterleaver()
NormFile::Offset NormPrecodeApp::ComputeInterleaverOffset(NormFile::Offset segmentId, NormFile::Offset numSegments) ProtoFile::Offset NormPrecodeApp::ComputeInterleaverOffset(ProtoFile::Offset segmentId, ProtoFile::Offset numSegments)
{ {
ASSERT(0 != interleaver_height); ASSERT(0 != interleaver_height);
NormFile::Offset interleaverWidth = interleaver_width; ProtoFile::Offset interleaverWidth = interleaver_width;
NormFile::Offset interleaverHeight = interleaver_height; ProtoFile::Offset interleaverHeight = interleaver_height;
NormFile::Offset interleaverSize = interleaver_size; ProtoFile::Offset interleaverSize = interleaver_size;
NormFile::Offset blockId; ProtoFile::Offset blockId;
if (i_max > 0) if (i_max > 0)
{ {
blockId = segmentId / interleaverSize; blockId = segmentId / interleaverSize;
@ -366,26 +437,25 @@ NormFile::Offset NormPrecodeApp::ComputeInterleaverOffset(NormFile::Offset segme
else else
{ {
blockId = 0; blockId = 0;
} }
// Check to see if we're in the last block // Check to see if we're in the last block
NormFile::Offset lastSegmentId = numSegments - 1; ProtoFile::Offset lastSegmentId = numSegments - 1;
NormFile::Offset lastBlockId = lastSegmentId / interleaverSize; ProtoFile::Offset lastBlockId = lastSegmentId / interleaverSize;
if ((blockId == lastBlockId) && (0 != (numSegments % interleaverSize))) if ((blockId == lastBlockId) && (0 != (numSegments % interleaverSize)))
{ {
// This block is smaller than our usual interleaver_size, // This block is smaller than our usual interleaver_size,
// so we're going to "square things up" to maximize the // so we're going to "square things up" to maximize the
// distance of this last block within interleaver_size constraint // distance of this last block within interleaver_size constraint
NormFile::Offset lastBlockSize = (numSegments % interleaverSize); ProtoFile::Offset lastBlockSize = (numSegments % interleaverSize);
interleaverWidth = (NormFile::Offset)(sqrt((double)lastBlockSize)); interleaverWidth = (ProtoFile::Offset)(sqrt((double)lastBlockSize));
interleaverHeight = lastBlockSize / interleaverWidth; interleaverHeight = lastBlockSize / interleaverWidth;
if (0 != (lastBlockSize % interleaverHeight)) interleaverHeight++; if (0 != (lastBlockSize % interleaverHeight)) interleaverHeight++;
} }
NormFile::Offset interleaverCol = segmentId / interleaverHeight; ProtoFile::Offset interleaverCol = segmentId / interleaverHeight;
NormFile::Offset interleaverRow = segmentId % interleaverHeight; ProtoFile::Offset interleaverRow = segmentId % interleaverHeight;
NormFile::Offset interleaverId = ((interleaverRow * interleaverWidth) + interleaverCol); ProtoFile::Offset interleaverId = ((interleaverRow * interleaverWidth) + interleaverCol);
if (0 != blockId) if (0 != blockId)
interleaverId += (blockId * interleaver_size); interleaverId += (blockId * interleaver_size);
@ -394,18 +464,18 @@ NormFile::Offset NormPrecodeApp::ComputeInterleaverOffset(NormFile::Offset segme
if (interleaverId >= numSegments) if (interleaverId >= numSegments)
{ {
// We're here because we hit a "hole" in the rectangle // We're here because we hit a "hole" in the rectangle
NormFile::Offset lastSegmentId = numSegments - 1; ProtoFile::Offset lastSegmentId = numSegments - 1;
if (0 != blockId) if (0 != blockId)
{ {
interleaverId = interleaverId % interleaverSize; interleaverId = interleaverId % interleaverSize;
lastSegmentId = lastSegmentId % interleaverSize; lastSegmentId = lastSegmentId % interleaverSize;
} }
// Find non-interleaved position of lastSegmentId within interleaver // Find non-interleaved position of lastSegmentId within interleaver
NormFile::Offset maxRow = lastSegmentId / interleaverWidth; ProtoFile::Offset maxRow = lastSegmentId / interleaverWidth;
NormFile::Offset maxCol = lastSegmentId % interleaverWidth; ProtoFile::Offset maxCol = lastSegmentId % interleaverWidth;
// There may be empty rows if lastSegmentId small wr2 interleaver size // There may be empty rows if lastSegmentId small wr2 interleaver size
NormFile::Offset emptyRows = interleaverHeight - maxRow - 1; ProtoFile::Offset emptyRows = interleaverHeight - maxRow - 1;
NormFile::Offset delta = 1 + emptyRows * interleaverCol; ProtoFile::Offset delta = 1 + emptyRows * interleaverCol;
if (interleaverCol > maxCol) if (interleaverCol > maxCol)
{ {
delta += interleaverRow - maxRow; delta += interleaverRow - maxRow;
@ -417,8 +487,8 @@ NormFile::Offset NormPrecodeApp::ComputeInterleaverOffset(NormFile::Offset segme
} }
// Find interleaved position of lastSegmentId within interleaver // Find interleaved position of lastSegmentId within interleaver
NormFile::Offset lastCol = lastSegmentId / interleaverHeight; ProtoFile::Offset lastCol = lastSegmentId / interleaverHeight;
NormFile::Offset lastRow = lastSegmentId % interleaverHeight; ProtoFile::Offset lastRow = lastSegmentId % interleaverHeight;
// Remap this segment to the "delta" interleaved position after "lastSegmentId" // Remap this segment to the "delta" interleaved position after "lastSegmentId"
lastRow += delta; lastRow += delta;
@ -445,12 +515,12 @@ NormFile::Offset NormPrecodeApp::ComputeInterleaverOffset(NormFile::Offset segme
} // end NormPrecodeApp::ComputeInterleaverOffset() } // end NormPrecodeApp::ComputeInterleaverOffset()
NormFile::Offset NormPrecodeApp::ComputeSegmentOffset(NormFile::Offset interleaverId, NormFile::Offset numSegments) ProtoFile::Offset NormPrecodeApp::ComputeSegmentOffset(ProtoFile::Offset interleaverId, ProtoFile::Offset numSegments)
{ {
NormFile::Offset interleaverWidth = interleaver_width; ProtoFile::Offset interleaverWidth = interleaver_width;
NormFile::Offset interleaverHeight = interleaver_height; ProtoFile::Offset interleaverHeight = interleaver_height;
NormFile::Offset interleaverSize = interleaver_size; ProtoFile::Offset interleaverSize = interleaver_size;
NormFile::Offset blockId; ProtoFile::Offset blockId;
if (i_max > 0) if (i_max > 0)
{ {
blockId = interleaverId / interleaverSize; blockId = interleaverId / interleaverSize;
@ -462,10 +532,10 @@ NormFile::Offset NormPrecodeApp::ComputeSegmentOffset(NormFile::Offset interleav
} }
NormFile::Offset interleaverRow = interleaverId / interleaverWidth; ProtoFile::Offset interleaverRow = interleaverId / interleaverWidth;
NormFile::Offset interleaverCol = interleaverId % interleaverWidth; ProtoFile::Offset interleaverCol = interleaverId % interleaverWidth;
NormFile::Offset segmentId = interleaverCol * interleaverHeight + interleaverRow; ProtoFile::Offset segmentId = interleaverCol * interleaverHeight + interleaverRow;
if (0 != blockId) if (0 != blockId)
segmentId += (blockId * interleaver_size); segmentId += (blockId * interleaver_size);
@ -473,26 +543,26 @@ NormFile::Offset NormPrecodeApp::ComputeSegmentOffset(NormFile::Offset interleav
if (segmentId >= numSegments) if (segmentId >= numSegments)
{ {
// It was a "hole", so find its hole delta // It was a "hole", so find its hole delta
NormFile::Offset lastSegmentId = numSegments - 1; ProtoFile::Offset lastSegmentId = numSegments - 1;
if (0 != blockId) if (0 != blockId)
{ {
segmentId = segmentId % interleaverSize; segmentId = segmentId % interleaverSize;
lastSegmentId = lastSegmentId % interleaverSize; lastSegmentId = lastSegmentId % interleaverSize;
} }
// Here maxRow/maxCol are wr2 _interleaved_ position of lastSegmentId // Here maxRow/maxCol are wr2 _interleaved_ position of lastSegmentId
NormFile::Offset maxCol = lastSegmentId / interleaverHeight; ProtoFile::Offset maxCol = lastSegmentId / interleaverHeight;
NormFile::Offset maxRow = lastSegmentId % interleaverHeight; ProtoFile::Offset maxRow = lastSegmentId % interleaverHeight;
// AS above, this assertion _should_ be true if we're "square" enough // AS above, this assertion _should_ be true if we're "square" enough
// (it does break when interleaver width is much greater than height, // (it does break when interleaver width is much greater than height,
// so, some day (TBD) we may want to generalize this remapping trick more??? // so, some day (TBD) we may want to generalize this remapping trick more???
ASSERT(interleaverCol >= maxCol); ASSERT(interleaverCol >= maxCol);
NormFile::Offset delta = (interleaverCol - maxCol)*(maxRow+1) + interleaverRow - maxRow; ProtoFile::Offset delta = (interleaverCol - maxCol)*(maxRow+1) + interleaverRow - maxRow;
// Then, remap "delta" to find _original_ "hole" position (corrected interleaver position) // Then, remap "delta" to find _original_ "hole" position (corrected interleaver position)
// Here maxRow/maxCol are wr2 _source_ position of lastSegmentId // Here maxRow/maxCol are wr2 _source_ position of lastSegmentId
maxRow = lastSegmentId / interleaverWidth; maxRow = lastSegmentId / interleaverWidth;
maxCol = lastSegmentId % interleaverWidth; maxCol = lastSegmentId % interleaverWidth;
NormFile::Offset emptyRows = interleaverHeight - maxRow - 1; ProtoFile::Offset emptyRows = interleaverHeight - maxRow - 1;
if (delta <= (emptyRows*(maxRow + 1))) if (delta <= (emptyRows*(maxRow + 1)))
{ {
// in first area // in first area
@ -510,7 +580,7 @@ NormFile::Offset NormPrecodeApp::ComputeSegmentOffset(NormFile::Offset interleav
if (0 != blockId) segmentId += (blockId * interleaver_size); if (0 != blockId) segmentId += (blockId * interleaver_size);
ASSERT(segmentId < numSegments); ASSERT(segmentId < numSegments);
} }
NormFile::Offset segmentOffset = segment_size * segmentId; ProtoFile::Offset segmentOffset = segment_size * segmentId;
return segmentOffset; return segmentOffset;
} // end NormPrecodeApp::ComputeSegmentOffset() } // end NormPrecodeApp::ComputeSegmentOffset()
@ -544,35 +614,43 @@ bool NormPrecodeApp::Encode()
struct timeval t1, t2; struct timeval t1, t2;
ProtoSystemTime(t1); ProtoSystemTime(t1);
NormFile::Offset fileSize = in_file.GetSize(); ProtoFile::Offset fileSize = in_file.GetSize();
// We reserve 4 bytes for our CRC (used to detect erasures) // We reserve 4 bytes for our CRC (used to detect erasures)
unsigned int dataSegmentSize = segment_size - 4; unsigned int dataSegmentSize = segment_size - 4;
NormFile::Offset numInputSegments = 1 + fileSize / dataSegmentSize; ProtoFile::Offset numInputSegments = 1 + fileSize / dataSegmentSize;
unsigned int lastFecSegSize = (unsigned int)(fileSize % dataSegmentSize); unsigned int lastFecSegSize = (unsigned int)(fileSize % dataSegmentSize);
if (0 != lastFecSegSize) numInputSegments++; if (0 != lastFecSegSize)
numInputSegments++;
else
lastFecSegSize = dataSegmentSize;
// Calculate FEC block size(s) // Calculate FEC block size(s)
NormFile::Offset numBlocks = numInputSegments / num_data; ProtoFile::Offset numBlocks = numInputSegments / num_data;
unsigned int fecBlockSize = num_data; unsigned int fecBlockSize = num_data;
unsigned int lastBlockSize = (unsigned int)(numInputSegments % num_data); unsigned int lastBlockSize = (unsigned int)(numInputSegments % num_data);
if (0 != lastBlockSize) numBlocks++; if (0 != lastBlockSize)
NormFile::Offset lastBlockId = numBlocks - 1; numBlocks++;
else
lastBlockSize = num_data;
ProtoFile::Offset lastBlockId = numBlocks - 1;
// 0) Calculate "out_file" size and determine interleaver width and height // 0) Calculate "out_file" size and determine interleaver width and height
NormFile::Offset numOutputSegments = ProtoFile::Offset numOutputSegments =
((numBlocks - 1) * (fecBlockSize + num_parity)) + lastBlockSize + num_parity; ((numBlocks - 1) * (fecBlockSize + num_parity)) + lastBlockSize + num_parity;
InitInterleaver(numOutputSegments); InitInterleaver(numOutputSegments);
// 1) Init our FEC encoder // 1) Init our FEC encoder
#ifdef USE_MDP_FEC NormEncoder* encoderPtr;
NormEncoderMDP encoder; if ((num_data + num_parity) > 256)
#else encoderPtr = new NormEncoderRS16;
NormEncoderRS8 encoder; else
#endif // if/else USE_MDP_FEC encoderPtr = new NormEncoderRS8;
NormEncoder& encoder = *encoderPtr;
if (!encoder.Init(num_data, num_parity, dataSegmentSize)) // 4 CRC bytes are _not_ encoded if (!encoder.Init(num_data, num_parity, dataSegmentSize)) // 4 CRC bytes are _not_ encoded
{ {
PLOG(PL_FATAL, "npc: error initializing FEC encoder\n"); PLOG(PL_FATAL, "npc: error initializing FEC encoder\n");
@ -581,7 +659,7 @@ bool NormPrecodeApp::Encode()
// Determine number of segments to allocate for FEC encoding and // Determine number of segments to allocate for FEC encoding and
// interleaver buffering if applicable // interleaver buffering if applicable
NormFile::Offset interleaverBytes = interleaver_size * segment_size; ProtoFile::Offset interleaverBytes = interleaver_size * segment_size;
char* iBuffer = NULL; char* iBuffer = NULL;
bool useBuffering = false; bool useBuffering = false;
if (interleaverBytes <= i_buffer_max) if (interleaverBytes <= i_buffer_max)
@ -625,20 +703,20 @@ bool NormPrecodeApp::Encode()
// (TBD) This could be built directly into iBuffer segment zero // (TBD) This could be built directly into iBuffer segment zero
char metaData[SEGMENT_MAX+4]; char metaData[SEGMENT_MAX+4];
memset(metaData, 0, SEGMENT_MAX); memset(metaData, 0, SEGMENT_MAX);
NormFile::Offset sz = fileSize; ProtoFile::Offset sz = fileSize;
if (sizeof(NormFile::Offset) == 8) if (sizeof(ProtoFile::Offset) == 8)
{ {
sz = htono(fileSize); sz = htono(fileSize);
memcpy(metaData, &sz, 8); memcpy(metaData, &sz, 8);
} }
else if (sizeof(NormFile::Offset) == 4) else if (sizeof(ProtoFile::Offset) == 4)
{ {
sz = htonl((UINT32)sz); sz = htonl((UINT32)sz);
memcpy(metaData + 4, &sz, 4); memcpy(metaData + 4, &sz, 4);
} }
else else
{ {
PLOG(PL_FATAL, "npc: error: unsupported file offset size (%d bytes)\n", sizeof(NormFile::Offset)); PLOG(PL_FATAL, "npc: error: unsupported file offset size (%d bytes)\n", sizeof(ProtoFile::Offset));
return false; return false;
} }
// put in_file_path file name portion into middle section of "metaData" // put in_file_path file name portion into middle section of "metaData"
@ -653,24 +731,23 @@ bool NormPrecodeApp::Encode()
// 2) Read "in_file" segments, encode, and output to "out_file" // 2) Read "in_file" segments, encode, and output to "out_file"
PLOG(PL_ALWAYS, "npc: encoding file ... (progress: 0%%)"); PLOG(PL_ALWAYS, "npc: encoding file ... (progress: 0%%)");
// State to track/display encoding progress // State to track/display encoding progress
NormFile::Offset progressThreshold = numOutputSegments / 100; ProtoFile::Offset progressThreshold = numInputSegments / 100;
double progressIncrement = 100.0; if (progressThreshold < 1) progressThreshold = 1;
if (progressThreshold > 1) ProtoFile::Offset progressCounter = 0;
progressIncrement = (double)numOutputSegments / (double)progressThreshold;
else
progressThreshold = numOutputSegments; // small number of segments
NormFile::Offset progressCounter = 0;
int progressPercent = 0; int progressPercent = 0;
NormFile::Offset blockId = 0; ProtoFile::Offset blockId = 0;
unsigned int parityCount = 0; unsigned int parityCount = 0;
bool parityReady = false; bool parityReady = false;
NormFile::Offset inputSegmentId = 0; ProtoFile::Offset inputSegmentId = 0;
NormFile::Offset outputSegmentId = 0; ProtoFile::Offset outputSegmentId = 0;
while (outputSegmentId < numOutputSegments) while (outputSegmentId < numOutputSegments)
{ {
NormFile::Offset interleaverOffset = ComputeInterleaverOffset(outputSegmentId, numOutputSegments); // The ComputerInterleaverOffset() call here retrieves the _output_ offset location that the
// current input segment will be mapped to. I.e., data is read from the input order, but mapped
// to interleaved output position via this offset
ProtoFile::Offset interleaverOffset = ComputeInterleaverOffset(outputSegmentId, numOutputSegments);
char* segment = useBuffering ? (iBuffer + (interleaverOffset % interleaverBytes)) : iBuffer; char* segment = useBuffering ? (iBuffer + (interleaverOffset % interleaverBytes)) : iBuffer;
if (parityReady) if (parityReady)
{ {
@ -696,25 +773,30 @@ bool NormPrecodeApp::Encode()
else else
{ {
// B) Read in data portion of next "segment" // B) Read in data portion of next "segment"
unsigned int bytesToRead; unsigned int expectedBytes;
if (inputSegmentId != numInputSegments) if (inputSegmentId != numInputSegments)
{ {
bytesToRead = dataSegmentSize; expectedBytes = dataSegmentSize;
} }
else else
{ {
memset(segment, 0, dataSegmentSize); memset(segment, 0, dataSegmentSize);
bytesToRead = lastFecSegSize; expectedBytes = lastFecSegSize;
} }
if (in_file.Read(segment, bytesToRead) != bytesToRead) unsigned int bytesToRead = expectedBytes;
if (!in_file.Read(segment, bytesToRead))
{ {
PLOG(PL_FATAL, "\nnpc: unexpected error reading input file: %s\n", GetErrorString()); PLOG(PL_FATAL, "\nnpc: unexpected error reading input file: %s\n", GetErrorString());
return false; return false;
} }
if (bytesToRead != expectedBytes)
{
PLOG(PL_FATAL, "\nnpc: unexpected end-of-file: %s\n", GetErrorString());
return false;
}
if (0 != bytesToRead) segment[bytesToRead] = '\0';
} }
// C) Encode and check for parity readiness // C) Encode and check for parity readiness
//TRACE("outputSegmentId:%lu\n", outputSegmentId);
encoder.Encode(outputSegmentId % fecBlockSize, segment, parityVec); encoder.Encode(outputSegmentId % fecBlockSize, segment, parityVec);
unsigned int numData = (blockId != lastBlockId) ? fecBlockSize : lastBlockSize; unsigned int numData = (blockId != lastBlockId) ? fecBlockSize : lastBlockSize;
if (numData == ++parityCount) if (numData == ++parityCount)
@ -722,20 +804,29 @@ bool NormPrecodeApp::Encode()
parityCount = num_parity; parityCount = num_parity;
parityReady = true; parityReady = true;
} }
if (++progressCounter >= progressThreshold)
{
progressPercent = int((double)inputSegmentId / (double)numInputSegments);
if (progressPercent < 9)
PLOG(PL_ALWAYS, "\b\b\b%d%%)", progressPercent + 1);
else if (progressPercent < 99)
PLOG(PL_ALWAYS, "\b\b\b\b%d%%)", progressPercent + 1);
if (progressPercent < 99) progressPercent++;
progressCounter = 0;
}
} }
// E) Calculate and add CRC32 checksum to each "segment" // E) Calculate and add CRC32 checksum to each "segment"
UINT32 checksum = ComputeCRC32(segment, dataSegmentSize); UINT32 checksum = ComputeCRC32(segment, dataSegmentSize);
checksum = htonl(checksum); checksum = htonl(checksum);
memcpy(segment+dataSegmentSize, &checksum, 4); memcpy(segment+dataSegmentSize, &checksum, 4);
if (useBuffering) if (useBuffering)
{ {
outputSegmentId++; outputSegmentId++;
if ((0 == (outputSegmentId % interleaver_size)) || (outputSegmentId == numOutputSegments)) if ((0 == (outputSegmentId % interleaver_size)) || (outputSegmentId == numOutputSegments))
{ {
// Output our buffered interleaver block from memory (iBuffer) to "out_file" // Output our buffered interleaver block from memory (iBuffer) to "out_file"
NormFile::Offset bytesToWrite; ProtoFile::Offset bytesToWrite;
if ((outputSegmentId != numOutputSegments) || (numOutputSegments == interleaver_size)) if ((outputSegmentId != numOutputSegments) || (numOutputSegments == interleaver_size))
bytesToWrite = interleaver_size; bytesToWrite = interleaver_size;
else else
@ -764,23 +855,12 @@ bool NormPrecodeApp::Encode()
return false; return false;
} }
outputSegmentId++; outputSegmentId++;
}
}
if (++progressCounter >= progressThreshold)
{
if (progressPercent < 9)
PLOG(PL_ALWAYS, "\b\b\b%d%%)", progressPercent + 1);
else if (progressPercent < 99)
PLOG(PL_ALWAYS, "\b\b\b\b%d%%)", progressPercent + 1);
if (progressPercent < 99) progressPercent++;
progressCounter = 0;
}
} }
if (progressPercent < 10) if (progressPercent < 10)
PLOG(PL_ALWAYS, "\b\b\b100%%)\n"); PLOG(PL_ALWAYS, "\b\b\b100%%)\n");
else else
PLOG(PL_ALWAYS, "\b\b\b\b100%%)\n"); PLOG(PL_ALWAYS, "\b\b\b\b100%%)\n");
in_file.Close(); in_file.Close();
out_file.Close(); out_file.Close();
@ -801,15 +881,15 @@ bool NormPrecodeApp::Encode()
bool NormPrecodeApp::Decode() bool NormPrecodeApp::Decode()
{ {
// 1) Determine file size and init interleaving // 1) Determine file size and init interleaving
NormFile::Offset inputFileSize = in_file.GetSize(); ProtoFile::Offset inputFileSize = in_file.GetSize();
NormFile::Offset numInputSegments = inputFileSize / segment_size; ProtoFile::Offset numInputSegments = inputFileSize / segment_size;
if (0 != (inputFileSize % segment_size)) if (0 != (inputFileSize % segment_size))
{ {
PLOG(PL_FATAL, "npc: error: input file size not integral number of given <segmentSize>\n"); PLOG(PL_FATAL, "npc: error: input file size not integral number of given <segmentSize>\n");
return false; return false;
} }
// Reverse calculate the FEC blocking // Reverse calculate the FEC blocking
NormFile::Offset numFecBlocks = numInputSegments / (num_data + num_parity); ProtoFile::Offset numFecBlocks = numInputSegments / (num_data + num_parity);
unsigned int fecBlockSize = num_data; unsigned int fecBlockSize = num_data;
unsigned int lastFecBlockSize = (unsigned int)(numInputSegments % (num_data + num_parity)); unsigned int lastFecBlockSize = (unsigned int)(numInputSegments % (num_data + num_parity));
if (0 != lastFecBlockSize) if (0 != lastFecBlockSize)
@ -818,17 +898,23 @@ bool NormPrecodeApp::Decode()
lastFecBlockSize -= num_parity; lastFecBlockSize -= num_parity;
numFecBlocks++; numFecBlocks++;
} }
NormFile::Offset lastFecBlockId = numFecBlocks - 1; else
{
lastFecBlockSize = num_data;
}
ProtoFile::Offset lastFecBlockId = numFecBlocks - 1;
// Calculate interleaver dimensions from file size // Calculate interleaver dimensions from file size
// set "interleaver_size", etc // set "interleaver_size", etc
InitInterleaver(numInputSegments); InitInterleaver(numInputSegments);
// 2) init FEC decoder // 2) init FEC decoder
#ifdef USE_MDP_FEC NormDecoder* decoderPtr;
NormDecoderMDP decoder; if ((num_data + num_parity) > 256)
#else decoderPtr = new NormDecoderRS16;
NormDecoderRS8 decoder; else
#endif // if/else USE_MDP_FEC decoderPtr = new NormDecoderRS8;
NormDecoder& decoder = *decoderPtr;
unsigned int dataSegmentSize = segment_size - 4; // leaves space for our CRC unsigned int dataSegmentSize = segment_size - 4; // leaves space for our CRC
if (!decoder.Init(num_data, num_parity, dataSegmentSize)) if (!decoder.Init(num_data, num_parity, dataSegmentSize))
{ {
@ -842,7 +928,7 @@ bool NormPrecodeApp::Decode()
// interleaver buffering if applicable // interleaver buffering if applicable
char* iBuffer = NULL; char* iBuffer = NULL;
bool useBuffering = false; bool useBuffering = false;
NormFile::Offset interleaverBytes = interleaver_size * segment_size; ProtoFile::Offset interleaverBytes = interleaver_size * segment_size;
if ((interleaverBytes <= i_buffer_max))// && if ((interleaverBytes <= i_buffer_max))// &&
//((num_data + num_parity) <= interleaver_size) && //((num_data + num_parity) <= interleaver_size) &&
@ -893,32 +979,29 @@ bool NormPrecodeApp::Decode()
} }
PLOG(PL_FATAL, "npc: decoding file ... (progress: 0%%)"); PLOG(PL_ALWAYS, "npc: decoding file ... (progress: 0%%)");
// State to track/display decoding progress // State to track/display decoding progress
NormFile::Offset progressThreshold = numInputSegments / 100; ProtoFile::Offset progressThreshold = numInputSegments / 100;
double progressIncrement = 100.0; if (progressThreshold <= 1)
if (progressThreshold > 1) progressThreshold = numInputSegments; // small number of segments
progressIncrement = (double)numInputSegments / (double)progressThreshold; ProtoFile::Offset progressCounter = 0;
else
progressThreshold = numInputSegments; // small number of segments
NormFile::Offset progressCounter = 0;
int progressPercent = 0; int progressPercent = 0;
// Read and decode each block in "in_file" // Read and decode each block in "in_file"
NormFile::Offset fecBlockId = 0; ProtoFile::Offset fecBlockId = 0;
NormFile::Offset outFileSize = 0; ProtoFile::Offset outFileSize = 0;
NormFile::Offset lastInterleaverBlockId = numInputSegments / interleaver_size; ProtoFile::Offset lastInterleaverBlockId = numInputSegments / interleaver_size;
NormFile::Offset lastInterleaverBytes = (numInputSegments % interleaver_size) * segment_size; ProtoFile::Offset lastInterleaverBytes = (numInputSegments % interleaver_size) * segment_size;
NormFile::Offset interleaverBlockId = 0; ProtoFile::Offset interleaverBlockId = 0;
unsigned int erasureCount = 0; unsigned int erasureCount = 0;
unsigned int segmentCount = 0; unsigned int segmentCount = 0;
enum State {READING, ADVANCING, DECODING}; enum State {READING, ADVANCING, DECODING};
State state = READING; State state = READING;
NormFile::Offset inputSegmentId = 0; ProtoFile::Offset inputSegmentId = 0;
while ((inputSegmentId < numInputSegments) || (DECODING == state)) while ((inputSegmentId < numInputSegments) || (DECODING == state))
{ {
switch (state) switch (state)
@ -927,15 +1010,21 @@ bool NormPrecodeApp::Decode()
if (useBuffering) if (useBuffering)
{ {
// Read in a full interleaver block // Read in a full interleaver block
NormFile::Offset bytesToRead; size_t bytesToRead;
if (interleaverBlockId != lastInterleaverBlockId) if (interleaverBlockId != lastInterleaverBlockId)
bytesToRead = interleaverBytes; bytesToRead = interleaverBytes;
else else
bytesToRead = lastInterleaverBytes; bytesToRead = lastInterleaverBytes;
if (in_file.Read(iBuffer, bytesToRead) != bytesToRead)
if (!in_file.Read(iBuffer, bytesToRead))
{ {
PLOG(PL_FATAL, "\nnpc: error reading input file: %s\n", GetErrorString()); PLOG(PL_FATAL, "\nnpc: error reading input file: %s\n", GetErrorString());
return false; return false;
}
if (0 == bytesToRead)
{
PLOG(PL_FATAL, "\nnpc: error reading input file: unexpected end-of-file\n");
return false;
} }
interleaverBlockId++; interleaverBlockId++;
state = ADVANCING; state = ADVANCING;
@ -947,7 +1036,7 @@ bool NormPrecodeApp::Decode()
for (unsigned int i = 0 ; i < (numData + num_parity); i++) for (unsigned int i = 0 ; i < (numData + num_parity); i++)
{ {
// Calc offset (de-interleave) and seek // Calc offset (de-interleave) and seek
NormFile::Offset interleaverOffset = ComputeInterleaverOffset(inputSegmentId, numInputSegments); ProtoFile::Offset interleaverOffset = ComputeInterleaverOffset(inputSegmentId, numInputSegments);
// seek to interleaver offset // seek to interleaver offset
if (!in_file.Seek(interleaverOffset)) if (!in_file.Seek(interleaverOffset))
{ {
@ -955,11 +1044,17 @@ bool NormPrecodeApp::Decode()
return false; return false;
} }
// Read segment // Read segment
if (in_file.Read(fecVec[i], segment_size) != segment_size) unsigned int bytesToRead = segment_size;
if (!in_file.Read(fecVec[i], bytesToRead))
{ {
PLOG(PL_FATAL, "\nnpc: unexpected error reading input file: %s\n", GetErrorString()); PLOG(PL_FATAL, "\nnpc: unexpected error reading input file: %s\n", GetErrorString());
return false; return false;
} }
if (bytesToRead != segment_size)
{
PLOG(PL_FATAL, "\nnpc: read() error: incomplete segment (len: %lu out of %lu bytes)\n", bytesToRead, segment_size);
return false;
}
inputSegmentId++; inputSegmentId++;
// Validate checksum (detects errors/ erasures) // Validate checksum (detects errors/ erasures)
@ -988,7 +1083,7 @@ bool NormPrecodeApp::Decode()
unsigned int priorSegmentCount = segmentCount; unsigned int priorSegmentCount = segmentCount;
for (; segmentCount < (numData + num_parity); segmentCount++) for (; segmentCount < (numData + num_parity); segmentCount++)
{ {
NormFile::Offset interleaverOffset = ComputeInterleaverOffset(inputSegmentId, numInputSegments); ProtoFile::Offset interleaverOffset = ComputeInterleaverOffset(inputSegmentId, numInputSegments);
fecVec[segmentCount] = iBuffer + (interleaverOffset % interleaverBytes); fecVec[segmentCount] = iBuffer + (interleaverOffset % interleaverBytes);
inputSegmentId++; inputSegmentId++;
// Validate checksum (detects errors/ erasures) // Validate checksum (detects errors/ erasures)
@ -1045,7 +1140,7 @@ bool NormPrecodeApp::Decode()
if((0 == fecBlockId) && (0 == i)) if((0 == fecBlockId) && (0 == i))
{ {
// First segment of first block is our "meta_data" with file size info // First segment of first block is our "meta_data" with file size info
switch (sizeof(NormFile::Offset)) switch (sizeof(ProtoFile::Offset))
{ {
case 8: case 8:
memcpy(&outFileSize, fecVec[0], 8); memcpy(&outFileSize, fecVec[0], 8);
@ -1078,6 +1173,7 @@ bool NormPrecodeApp::Decode()
{ {
// Last segment, so calculate "lastSegmentSize" // Last segment, so calculate "lastSegmentSize"
segmentSize = (unsigned int)(outFileSize % segmentSize); segmentSize = (unsigned int)(outFileSize % segmentSize);
if (0 == segmentSize) segmentSize = segment_size - 4;
} }
if (out_file.Write(fecVec[i], segmentSize) != segmentSize) if (out_file.Write(fecVec[i], segmentSize) != segmentSize)
{ {
@ -1120,8 +1216,6 @@ bool NormPrecodeApp::Decode()
} // end NormPrecodeApp::Decode() } // end NormPrecodeApp::Decode()
/*****************************************************************/ /*****************************************************************/
/* */ /* */
/* CRC LOOKUP TABLE */ /* CRC LOOKUP TABLE */