Containers
==========

There are three concepts necessary to reason about containers:

* packet: A variable length chunk of data.  The only "metadata" associated
  with it is a logical timestamp of when the first byte of the data was
  produced.  The type of the packet is *not* stored by the packet itself.
* stream: An ordered sequence of packets.  Each stream is identified by a
  unique id (a random integer).  In general, all streams are handled
  identically and the interpretation of the streams is left to higher levels
  of abstraction.
* session: A set of streams representing a recording session.  E.g., all the
  data (I/Q samples) and metadata (timestamps, device info, center
  frequency, bandwidth, etc.) captured between when the recording started
  and when it was stopped.

Note that there are no other structures.  Any data that is to be stored
must be stored in a packet in a stream.

These concepts are rather generic.  The next section talks about how DSP-
related data and metadata can be mapped to them.


DSP
===

A typical DSP application reads and writes a sequence of samples making up a
signal.  The some of the relevant metadata is often serialized into the
filename, while other metadata is silently discarded.  This is unfortunate.

What streams should there be?
-----------------------------

We certainly want some global metadata - and because of the "everything is a
stream" approach, this means we want a global metadata stream.

It is rather clear that the sequence of I/Q samples should be considered a
stream.  If we are handling multiple sequences of I/Q samples (e.g., we have
a phase-coherent multi-antenna setup), each sequence should be a separate
stream.

Finally, for each I/Q stream, we want a metadata stream with information
such as the center frequency, sample rate, and so on.

It is also possible to include other streams with data that is application
specific.  For example, meteorological observations near the antenna or raw
GNSS packet capture.

Global Metadata Stream
----------------------

The global metadata stream contains information that is valid for the entire
container.  Namely:

* metadata format version number (e.g., 0)
* `generator`: generator software ID string (e.g., "SigDigger v0.1.0-23-g828657f")
* `datetime`: date time of when the container file was created (not necessarily the same
  as when the I/Q samples were recorded)
* `hostname`: hostname?
* `streams`: explicit list of metadata stream unique IDs?

I/Q Data Streams
----------------

These are just I/Q samples.

Per-I/Q Stream Metadata Streams
-------------------------------

* `type`: mime type of the I/Q stream (always "application/octet-stream")
* `ts`: date time of when the packet of samples was created (time of first sample)
* `iq/samples`: number of samples since last metadata update
* `iq/uniq`: I/Q data stream unique id
* `iq/fmt`: I/Q data format (e.g., "cf32" or "cu8")
* `iq/fc`: I/Q center frequency in Hz (e.g., 137500000)
* `iq/fs`: I/Q sample rate in S/s (e.g., 2560000)
* `iq/dev`: I/Q device (e.g., "RTL-SDR v3 00000...")
* `iq/antenna`: I/Q antenna (e.g., "RTL-SDR dipole")
* `iq/???`: LNA device
* `iq/???`: LNB device

TODO: what else?  should some of the following be separate streams?
* ``: lat/lon/alt
* ``: ground speed? azimuth?
* ``: antenna orientation?
* ``: LNA gain?
* ``: LNB freq?

Misc Data Streams
-----------------

These are opaque byte streams with no particular format.  The interpretation
of this data stream is left up to the application.  The associated metadata
stream (see below) identifies the MIME type of this data.

Per-Misc Data Stream Metadata Streams
-------------------------------------

* mime type of the data stream (e.g., TODO)
* optional application specific key-value pairs

TODO: what's in a packet? what's in a stream? what streams are there?

A typical session will contain only three streams:

0. session metadata
1. I/Q samples
2. I/Q metadata

TODO: Note that the ordering of the streams does not matter.


Mapping to Existing Container Formats
=====================================

There are many ways to serialize the abstract concepts of sessions, streams,
and packets.

directory/tar/cpio
------------------

Each packet could simply be stored in a file - either in a plain directory
or inside an archive format such as tar or cpio.  (tar uses a block size,
which wastes a decent amount of space.  cpio doesn't do this padding, and
therefore may be a better choice.  Both are equally simple to implement.)

* packet -> file with filename `<stream unique>-<logical timestamp>`
* stream -> files with filenames `<stream unique>-*`, in lexicographic order
* session -> the directory/tar/cpio containing the files


Ogg
---

* packet -> ogg packet with the data split between the smallest number of
  pages and the granule containing the packet's logical timestamp
* stream -> ogg bitstream with the ogg bitstream serial being set to the
  stream's unique id
* session -> ogg file


Hybrid Sidecar
--------------

It may be beneficial to start with a hybrid approach where the I/Q samples
are appended to a single large file (as is the case today) and all the other
streams are serialized into a richer container.  For example, suppose that
a recording session produces two files:

```
sigdigger_20200520_010415Z_1024000_137500000_float32_iq.raw
sigdigger_20200520_010415Z_1024000_137500000_float32_iq.meta
```

The `.raw` contains the samples as before.  The `.meta` file contains the
remaining streams serialized into structured metadata format (e.g., Ogg +
CBOR).  This would yield all the benefits of having a richer format without
the initial incompatibility with other software.


Other Existing Container Formats
--------------------------------

Here are some other existing container file formats that I looked at and
decided that they are too cumbersome for one or more reasons.

* RIFF: 32-bit file sizes, very messy spec
* RF64: 64-bit version of RIFF, very messy spec
* MKV: requires EBML implementation