NACK-Oriented Reliable Multicast (NORM) implementation & tools (RFCs 5740, 5401)
 
 
 
 
 
 
Go to file
mullerj 4fd768f9cc
.NET NORM Extension (#85)
* .NET Extension Design (#5)

* Designed .NET classes from Java

Co-authored-by: Sergiy <sergiy.yermak.ctr@us.navy.mil>

* Added NormApi static class

* Started on api uml

Co-authored-by: mullerj <mullerj@users.noreply.github.com>
Co-authored-by: Sylvester Freeman III <Slyfree3@users.noreply.github.com>

* Add static methods to api uml

* add static method to normapi uml

* Add enums

* Changes

* Overload methods with defaults

* type marshalling

* change handles to in nuint

* marshall more handles

* Removed in/ref and reformatted.

Co-authored-by: mullerj <mullerj@users.noreply.github.com>

* Removed duplicate enums

* Removed NormEvent from NormApi

* Fixed typo

* design NormInstance methods

Co-authored-by: mullerj <mullerj@users.noreply.github.com>

* Fixed CreateInstance method

* Changed handles to long

* Change id types to long

* Remove condition checks that do nothing

* Removed UserData functions

* Removed NormAlloc() and NormFree()

* Cleaned up some API methods

* Fix misspelling

Co-authored-by: Sylvester Freeman III <Slyfree3@users.noreply.github.com>

* Design NormSession methods

Co-authored-by: Sylvester Freeman III <Slyfree3@users.noreply.github.com>

* Added missing comma

* Uml for enqueue
Co-authored-by: mullerj <mullerj@users.noreply.github.com>

* Updated design based on enqueue test

* Updated EnqueueFile sequence diagram

Co-authored-by: Sylvester Freeman III <Slyfree3@users.noreply.github.com>

* Moved sequence diagrams

* Added constants to NormApi

* Removed method level sequence diagrams

---------

Co-authored-by: Sergiy <sergiy.yermak.ctr@us.navy.mil>
Co-authored-by: syermak <syerm001@odu.edu>
Co-authored-by: mullerj <mullerj@users.noreply.github.com>
Co-authored-by: Sylvester Freeman III <Slyfree3@users.noreply.github.com>
Co-authored-by: Sergiy <47069243+SergiyYermak@users.noreply.github.com>
Co-authored-by: Sylvester Freeman III <Slyfreeman3@gmail.com>

* .NET Extension Design Implementation (#10)

* changes

* Removed GetsNextEvent test

* Implemented GetNextEvent

* Fixed NormStream design

* Fixed HasNextEvent design

* Added NormGetNextEvent overload

* Implemented HasNextEvent

* Updated design

* Added Kernel32 to design

* Modified to use .NET EventWaitHandle instead of kernel32 P/Invoke

* Modified to only copy norm.dll on Windows

* Added copy for non Windows

* add start and stop receiver

* add receiver

* Modified non-Windows to use Socket for HasNextEvent

* Deleted NormFileTests since they break the other tests

* Fixed ReceiverReceives test

* Updated ReceiverReceives test name

* Modified ReceivesFile to get all events instead of adding to array

* Cleaned up NormInstance formatting

* Refactored tests

* Updated NormFile

* Implemented send stream

* Implemented receive stream

* Implemented NormFile Name

* Updated ReceivesFile to use events

* data enqueue

* Fixed merge

* Fixed ReceivesData test

* Modified data test to compare collections without order

* stop instance working

* enqueues data

* Reverted upgrade to .NET 7

* Restartinstance is working

* Modified NormData to read bytes instead of as string

* Removed using statements in NormData

* Updated design

* Updated design

* Switched to nint

* Switched design to nint

* unfinished test for suspendinstance

* working suspend instance

* added resume function to instance

* Updated session send and receive data tests

* Removed launch settings

* session configuration functions

* Modified to use random port due to intermittent failed tests due to session conflict

* Modified to randomize text content

* Implemented NormNode Address

* Updated design for NormNode

* Implemented NormNode Grtt

* Implemented NormNode Id

* implement session configuration functions

* instance

* missing return types

* add missing handle param

* normNode

* Implemented RequeueObject

* Implemented SetWatermark and AckingNode

* Implemented CancelWatermark

* Implemented ResetWatermark

* Implemented SendCommand

* Implemented CancelCommand

* Implemented DebugLevel. Removed DebugPipe

* Updated SetTxPort

* Updated SetRxPortReuse

* Implemented SetEcnSupport

* Added test for RxLoss

* Implemented ReportInterval

* Refactored session calculated properties

* Updated enqueue methods

* Updated StreamOpen design

* Fixed SetWatermark overload

* Implemented GetAckingStatus

* Removed SendCommand overload

* Removed SetSilentReceiver overload

* Implemented NormEvent ToString

* Implemented NormFile Rename

* Implemented SetUnicastNack

* Added ReceivesCommand test

* Implemented SetNackingMode

* Implemented SetRepairBoundary

* Added Fact to SetsRepairBoundary test

* Implemented SetRxRobustFactor

* Implemented NormNode Command

* Implemented FreeBuffers

* Implemented Retain and Release

* Fix using namespaces

* test get object type

* NormObject get size

* NormObject setNackingMode

* NormObject get bytes pending

* NormObjet cancel

* NormObject retain

* NormObject release

* NormObject get sender

* NormObject get hashcode

* normObject equals

* Implemented NormObject Equals

* Removed redundant cast

* Removed unused using directive

* Removed static using

* Added NormApi to global Usings

* Added Enums to global usings

* Cleaned up NormObject Equals

* Cleaned up NormObject GetHashCode() test

* Cleaned up warning

* normStream has vacancy

* normStream read offset

* normStream seek msg start

* NormEventListener interface

* normStream set push enable

* normStream set auto flush

* Simplified NormApi usage

* Updated design of NormStream

* public interface

* Fixed INormEventListener

* fixing name space

* commit initializing  inputstream

* Added StreamBreakException

* Fixed formatting

* Fixed formatting

* Fixed formatting

* Initial implementation of NormOutputStream

* Cleaned up NormObject Equals

* NormInputStream class

Co-authored-by: mullerj <mullerj@users.noreply.github.com>

* normStream data offset

* move offset length arithmetic into write method

* building normInputstream

* all method implemented

Co-authored-by: mullerj <mullerj@users.noreply.github.com>

* NormStreamRead

* Fixed build error

* Completed ReceivesStreamWithOffset test

* Added retries to GetsCommand test

* Updated type for NormStreamRead

* Simplified NormStream Read

* Simplified NormStream Write

* Added dataOffset to DataEnqueue

* Fixed DataEnqueue overload

* Added infoOffset to FileEnqueue

* Added infoOffset to StreamOpen

* Updated design

* Removed extra overload of FileEnqueue

* Removed DataEnqueue overloads

* Removed extra overload from StreamOpen

* Added HasNextEvent overload

* Implemented OpenDebugPipe

* add thread locking

* locking for properties

* Not supported exceptions

* Implementing sychronize

* fixing inputstream

* adding sychronize message

* Reorganized NormApi and updated design

* Modified to skip GetsCommand test on IO exception

* WIP: 537b706 adding sychronize message

* adding message to expections

* Fixed test file cleanup

* Added locking to NormSession methods that use static field

* Changed StartsSender to skippablefact

* Changed to SetsNackingMode to skippable fact

* Converted tests to skippable

* Converted more tests to skippable

* Converted all tests that start sender to skippable

* Updated README about skipped tests

* documentation

* 2 doc

* Cleaned up input and output stream classes

* Cleaned up NormApi

* Updated NormOutputStream Write

* Added document comments to NormApi

* Added comments to NormApi

* Removed unnecessary usings

* Added comments to NormApi

* set up the rest of doc

* api documentation

* fix typo

* fix spacing

* Doc for instance need clean up

* clean up

* document norm session

* NormStream documentation

* commit for node

* NormObject docs

* Updated NormApi comments

* Cleaned up NormInstance comments

* Cleaned up NormNode comments

* Cleaned up NormObject comments

* Cleaned up NormSession comments

* Cleaned up NormStream comments

* Cleaned up NormObject comments

* Adding commits

* Update get commit

* clean up on data and event

* Norm Data change

* clean up normfile docs

* clean up normstream

* clean up api docs

* Fixed data

* space approved

* Cleaned up NormData comment

* documentation clean up

* Update the ReadMe

* Cleaned up comments

* Added comments to NormAckingStatus

* Added comments to NormEventType

* Added comments to NormFecType

* Added comments to NormNackingMode

* Added comments to NormObjectType

* Added comments to NormProbingMode

* Added comments to NormRepairBoundary

* Added comments to NormFlushMode

* Added comments to NormSyncPolicy

* Cleaned up comments.  Fixed default for StartSender

* Added comments to constants

* Moved constants to relevant classes

* Updated design

* Removed NormDataDetachData since it is not referenced

* Updated design

* Updated input and output stream design

* Updated packaging README

* Removed language version since redundant

* Cleaned up project file

* Updated to package from the dotnet directory (test project is set to not pack)

* Added version to project

---------

Co-authored-by: syermak <syerm001@odu.edu>
Co-authored-by: Sylvester Freeman III <Slyfreeman3@gmail.com>
Co-authored-by: Sergiy <47069243+SergiyYermak@users.noreply.github.com>
Co-authored-by: mullerj <mullerj@users.noreply.github.com>

* NormSession typos in comments (#12)

* Fixed default function comments in NormSession

* Fixed typos in NormSession

* Fixed default function comments

* Feature/9 net norm extension target architecture packages (#13)

* Added Win64 project and release configuration

* Added project reference to Win64 project

* Added Win32 project. Moved norm build actions to scripts

* Added linux .NET extension project

* Modified linux project to copy to output directory

* Fixed linux pack script

* Changed Linux project to copy always

* Changed Linux project package path

* Moved Windows libraries to runtimes path

* Renamed runtime specific projects

* Codespace Configurations (#15)

* Created devcontainer.json

* .NET norm extension GitHub actions (#16)

* Added .NET action

* Update dotnet-publish.yml to use source branch (#18)

* .NET norm package publish (#19)

* Added steps to copy package and check workspace

* Changed trigger for testing

* Removed if statement

* Updated dotnet-publish.yml to copy to  $GITHUB_OUTPUT

* Changed from variable to path

* Removed copy command

* Modified to check runner directory

* Added step to publish package

* Added package write permission

* Added job to publish win-x64

* Fixed package win-x64 step

* Added job to publish win-x86

* Added skip-duplicate to publish commands

* Added publish job

* Changed to . for linux find

* Added workflow trigger

* Added branches filter

* Publish test results (#21)

* Added step to publish test results

* Changed upload step to use wildcard

* Added quotes

* Changed directory to relative path

* Fixed upload step

* Added step to publish test results

* Added permissions

* Dev Container extensions (#23)

* Added extensions

* Cut down on extensions

* Added extension

* Removed pull requests extension

* Added workspace recommendation

* Added powershell feature

* Fixed powershell version

* Updated powershell version and added extension

* Updated to latest powershell

* Dev Container performance (#25)

* Fixed devcontainer format

* Added dotnet-linux-x64 devcontainer

* Changed devcontainer image to python

* Pinned OS version

* Updated devcontainer base image

* Reverted changes to devcontainer

* Moved linux devcontainer file

* Updated dotnet-publish to use subfolder

* Revert "Dev Container performance (#25)" (#27)

This reverts commit 2be146ceed.

* Update action versions for dotnet-publish.yml

* Update action versions for dotnet.yml

* Fixed ReceivesData test to use received events for actual data

* NormData Refactor (#30)

* Switched NormData Data property to a method

* Corrected NormData summary comment

* Updated NormData GetData to use Marshal.Copy

* Fixed warnings

* Updated NormData design

* NormDataEnqueue Refactor (#31)

* Switched NormDataEnqueue to use IntPtr

* Updated NormApi design

* Updated GetSession to use lock

* Updated GetSession to TryGetValue

* DataEnqueue Refactor (#33)

* Refactored DataEnqueue to use pointer arithmetic

* NormDataEnqueue Refactor (#34)

* Updated to use SafeBuffer

* Added ByteBuffer and DirectByteBuffer

* Added pointer overload to DataEnqueue

* Updated FileEnqueue() to use pointer arithmetic

* Updated NormStream to use pointer arithmetic

* Updated NormSendCommand to nint

* Updated NormNodeGetCommand to nint

* Updated NormNodeGetAddress to use nint for buffer

* Updated NormStreamRead to use nint for buffer

* Updated NormObjectGetInfo to use nint for buffer

* Updated NormFileGetName nameBuffer to nint

* Updated NormFile.Name to use PtrToStringAnsi

* Updated NormFile.Name to use stackalloc (#37)

* Updated NormNode.Address to use stackalloc and create IPAddress from bytes

* NormObject Refactor

* Updated NormObjectGetInfo to use byte array

* Updated design

* NormStream Refactor

* Updated NormStream Write to use fixed statement

* Updated NormStream Read to use fixed statement

* Updated design

* NormNode Refactor

* Updated NormNodeGetAddress to use byte* to avoid casting

* Updated NormNode GetCommand to use fixed statement

* Updated design

* NormFile Refactor

* Updated NormFileGetName to use sbyte* to avoid casting

* Updated design

* Added workflow_dispatch to workflows

---------

Co-authored-by: Sergiy <sergiy.yermak.ctr@us.navy.mil>
Co-authored-by: syermak <syerm001@odu.edu>
Co-authored-by: mullerj <mullerj@users.noreply.github.com>
Co-authored-by: Sylvester Freeman III <Slyfree3@users.noreply.github.com>
Co-authored-by: Sergiy <47069243+SergiyYermak@users.noreply.github.com>
Co-authored-by: Sylvester Freeman III <Slyfreeman3@gmail.com>
2025-03-26 01:07:19 -04:00
.devcontainer .NET NORM Extension (#85) 2025-03-26 01:07:19 -04:00
.github/workflows .NET NORM Extension (#85) 2025-03-26 01:07:19 -04:00
.vscode .NET NORM Extension (#85) 2025-03-26 01:07:19 -04:00
android pulled updated submodule code 2023-09-03 15:01:44 -04:00
cmake pynorm fixes 2022-08-20 22:36:29 -04:00
doc fixes to npc (NormPrecoder) and updated its documentation. Include 'auto' block sizing' feature 2025-01-26 20:26:49 -05:00
docker feat: adding dockerfile for docker image building (#56) 2022-04-28 22:06:43 -07:00
examples Documentation updates and pynorm example bug fixes (#82) 2024-01-24 13:15:06 -05:00
include pynorm improvement to keep cache sender enqueued objects in pynorm.instance._objects cache 2022-10-04 19:21:41 -04:00
makefiles added examples/chant.cpp for simple command-line NORM chat app 2023-05-31 19:30:36 -04:00
norp Update incl. fix to MacOS Makefile 2024-06-08 17:38:36 -04:00
protolib@e5ac157eb6 fixes to npc (NormPrecoder) and updated its documentation. Include 'auto' block sizing' feature 2025-01-26 20:13:35 -05:00
src .NET NORM Extension (#85) 2025-03-26 01:07:19 -04:00
.gitattributes .NET NORM Extension (#85) 2025-03-26 01:07:19 -04:00
.gitignore .NET NORM Extension (#85) 2025-03-26 01:07:19 -04:00
.gitmodules pynorm fixes 2022-08-20 22:36:29 -04:00
BUILD.TXT Documentation updates and pynorm example bug fixes (#82) 2024-01-24 13:15:06 -05:00
CMakeLists.txt added examples/chant.cpp for simple command-line NORM chat app 2023-05-31 19:30:36 -04:00
Dockerfile pynorm fixes 2022-08-20 22:36:29 -04:00
LICENSE pynorm fixes 2022-08-20 22:36:29 -04:00
NormSocketBindingNotes.txt pynorm fixes 2022-08-20 22:36:29 -04:00
README-DotNet.md .NET NORM Extension (#85) 2025-03-26 01:07:19 -04:00
README-Java.txt pynorm fixes 2022-08-20 22:36:29 -04:00
README-PyNorm.txt Documentation updates and pynorm example bug fixes (#82) 2024-01-24 13:15:06 -05:00
README.md pynorm fixes 2022-08-20 22:36:29 -04:00
TODO.TXT pynorm fixes 2022-08-20 22:36:29 -04:00
VERSION.TXT pynorm fixes 2022-08-20 22:36:29 -04:00
norm.pc.in pynorm fixes 2022-08-20 22:36:29 -04:00
setup.py Documentation updates and pynorm example bug fixes (#82) 2024-01-24 13:15:06 -05:00
waf pulled updated submodule code 2023-01-28 16:09:22 -05:00
wscript fix to waf build of normCast example 2024-12-27 16:32:05 -05:00

README.md

norm

                    NORM SOURCE CODE RELEASE

/*********************************************************************
 *
 * AUTHORIZATION TO USE AND DISTRIBUTE
 * 
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that: 
 *
 * (1) source code distributions retain this paragraph in its entirety, 
 *  
 * (2) distributions including binary code include this paragraph in
 *     its entirety in the documentation or other materials provided 
 *     with the distribution.
 * 
 *      "This product includes software written and developed 
 *       by Code 5520 of the Naval Research Laboratory (NRL)." 
 *         
 *  The name of NRL, the name(s) of NRL  employee(s), or any entity
 *  of the United States Government may not be used to endorse or
 *  promote  products derived from this software, nor does the 
 *  inclusion of the NRL written and developed software  directly or
 *  indirectly suggest NRL or United States  Government endorsement
 *  of this product.
 * 
 * THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR
 * IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
 *
 ********************************************************************/

This is the NRL NORM source code repository.

The Github "Discussions" feature is available here for comment and question in addition to the regular code issue reporting:

https://github.com/USNavalResearchLaboratory/norm/discussions

SOURCE CODE

The "norm" source Git repository includes "protolib" as a git submodule. You can add the "--recurse-submodules" to your git clone command to automate inclusion:

git clone --recurse-submodules https://github.com/USNavalResearchLaboratory/norm.git

The following items can be built from this source code release:

  1. libnorm.a - static NORM library for Unix platforms, or

    Norm.lib - static NORM protocol library for Win32 platforms

(These libraries can be used by applications using the NORM API. The file "normApi.h" provides function prototypes for NORM API calls and the included "Norm Developer's Guide" provides a reference manual for NORM application development using this API. Additional tutorial material and API usage examples will be provided as additional documentation in the future)

  1. norm (or norm.exe (WIN32) - command-line "demo" application built from "common/normApp.cpp"

(The included "Norm User's Guide" provides a rough overview of how to use this demo app. This demo application is useful for file transfers (and streaming on Unix))

  1. normTest (or normTest.exe (WIN32)) - very simple test application

(The "normTest" application (see "common/normTest.cpp") is really just a simple playground for preliminary testing of the NORM API and does not do anything useful. But it does provide a very simple example of NORM API usage. More sophisticated (and better-documented) examples of NORM API usage will be provided in the future.

  1. There is also an "examples" directory in the source code tree that contains some simplified examples of NORM API usage. The example set will be expanded as time progresses.

NRL also has started some apps built around NORM including:

A) A reliable multicast "chat" application (users can share files and images, too) with a GUI. Like IRC with no server needed.

B) An application for reliably "tunneling" real-time UDP packet streams (like RTP video or audio) using NORM's streaming capability.

NACK-Oriented Proxy (NORP)

NORM Proxy code is also included in the 'norp' directory. More info is available in the norp/doc directory, as well as the NORP website: https://www.nrl.navy.mil/itd/ncs/products/norp

Docker

A Dockerfile is provided to facilitate containerized NORM environment testing. Notable build arguments passed to docker build will influence how the image is created.

  • platformName represents the base image name (defaults to ubuntu)
  • platformVersion represents the tagged version for the base image (defaults to latest)
  • version is a 1:1 mapping to this repo's tag (defaults to v1.5.9)
  • configure is piped to ./waf configure (defaults to "")
  • build is piped to ./waf (defaults to --targets=*)

Example usage:

CentOS7 build

docker build --build-arg platformName=centos \
  --build-arg platformVersion=7 \
  --build-arg="--build-java" \
  -t norm:centos-1.5.9 .

Running the data send and recv examples:

# Sender in one terminal
docker run -it --rm --name norm-send <norm:image> normDataSend
# Receiver in another terminal
docker run -it --rm --name norm-recv <norm:image> normDataRecv

OTHER FILES:

NormDeveloperGuide.pdf - PDF version of NORM Developer's Guide (nice for printing)

NormDeveloperGuide.html - HTML version of NORM Developer's Guide with Hyperlinked content (nice for browsing)

NormUserGuide.pdf - Guide to "norm" demo app usage

VERSION.TXT - NORM version history notes

README.md - this file

docker - directory containing docker container support

NOTES:

The NORM code depends upon the current "Protolib" release:

https://github.com/USNavalResearchLaboratory/protolib

It has been addded as a git submodule to the NORM git repository. So, to to build you will need to do the following steps to download the protolib code:

git clone --recurse-submodules https://github.com/USNavalResearchLaboratory/norm.git

Alternatively after a basic "git clone" you can do the folowing to pull in the protolib source:

cd norm git submodule update --init

To keep the 'protolib' submodule up to date, you will need to do the following:

cd norm/protolib git checkout master

This will enable you to issue 'git pull', etc to treat the 'protolib' sub-directory as its own (sub-) repository.