The Instrument Model Database (IMO)¶
To run a realistic simulation of an instrument, one needs to known its details: the noise level of the detectors, the angular resolution of the beams, etc. This kind of information is stored in an «instrument model database», called IMO, and the LiteBIRD Simulation Framework provides a few facilities to access it.
Let’s start from a simple example, which will guide us in the following paragraphs:
from litebird_sim import Imo
imo = Imo(flatfile_location="/storage/litebird/my_imo")
scan_params = imo.query(
"/releases/v0.10/satellite/scanning_parameters"
)
metadata = scan_params.metadata
print(metadata["spin_sun_angle_deg"])
# Output: the angle between the sun and the spin axis, in degrees
This example shows how to retrieve the parameters of the scanning
strategy followed by the LiteBIRD spacecraft. Everything revolves
around the class Imo
, which reads the IMO from the files
saved in the folder /storage/litebird/my_imo
.
The call to imo.query
retrieves a specific bit of information;
note that the access to the parameters is done using a file-like path,
/releases/v0.10/satellite/scanning_parameters
. This is not a real
file, but a way to tell the IMO which kind of information is
requested: the /releases/v0.10
specifies the IMO version to use,
and the remaining path /satellite/scanning_parameters
points to
the information you’re looking for.
Local/remote access to the IMO¶
The IMO can be accessed either through an Internet connection or by reading it directly from a file. Each approach has its own advantages and disadvantages:
Having the IMO saved in a local file (like in the example above) is the fastest way to access its contents. However, it might not contain the latest version of the data you’re looking for.
Using a remote IMO through an Internet connection ensures that you have access to the most updated version of the instrument; however, accessing it can be slow, and you’re out of luck if your internet connection is unstable. Here is an example:
from litebird_sim import Imo imo = Imo( url="https://dummy-litebird-imo.org", user="username", password="12345", ) scan_params = imo.query( "/releases/v0.10/satellite/scanning_parameters" )
Once the Imo
object has been created, accessing information
follows the same rules and uses the same syntax for paths.
How objects are stored¶
Information in the IMO is stored using a hierarchical format, and every datum is versioned. There are three fundamental concepts that you need to grasp:
The IMO can store data files and Python-like dictionaries, called «metadata».
Different versions of the same data file can be kept at the same time in the IMO; we use the term quantity to refer to a data file but we don’t care about its version.
Quantities can be stored in hierarchical structures, using the concept of entity, which enable to structure entities in a tree-like shape.
Here is an example:
satellite
|
+--- spacecraft
| |
| +--- *scanning_parameters*
|
+--- LFT
| |
| +--- 40_ghz
| | |
| | +--- *noise_characteristics*
| | |
| | +--- *band_response*
| |
| +--- 50_ghz
| | |
| | +--- *noise_characteristics*
| | |
| | +--- *band_response*
| ...
|
+--- MFT
| |
| ...
|
+--- HFT
|
...
The diagram above shows how different quantities (marked using
asterisks in the diagram: scanning_parameters
,
noise_characteristics
, band_response
) can be structured in a
tree-like structure using entities (the branches in the tree, e.g.,
40_ghz
, LFT
). Different data files can be associated with
quantities like BAND_RESPONSE
.
In the code example at the top of this page, we accessed the scanning
strategy parameters using the string
/releases/v0.10/satellite/scanning_parameters
. The meaning of the
string in terms of entities, quantities, and data files is the
following:
Satellite
is an entity;scanning_parameters
is a quantity, because it’s the last part of the path;Of all the possible versions of the data file that have been saved in the quantity
scanning_parameters
, we’re asking for the one that is part of IMO 0.10 (the stringv0.10
in the path).
Apart from paths like
/releases/v0.10/satellite/scanning_parameters
, there is a more
low-level method to access data files, using UUIDs. Each quantity and
each datafile is identified by a unique UUID, an hexadecimal string
that it’s granted to be unique. This string is assigned automatically
when information is added to the IMO, and it can be used to retrieve
the information later. In Python, you can use the UUID
type from
the uuid library to
encode this information from a string:
from uuid import UUID
# The string below is usually read from a parameter file
my_uuid = UUID("5b9e3155-72f2-4e18-95d4-9881bc3e592d")
# Use "my_uuid" to access data in the IMO
scan_params = imo.query(my_uuid)
The advantage of the latter method is that you can access data files that have not been formally included in a versioned IMO.
IMO and reports¶
When constructing a Simulation
object, you should pass an
instance of a Imo
class. In this way, simulation modules can
take advantage of an existing connection to the IMO.
This has the additional advantage that the report produced at the end of the simulation will include a list of all the data files in the IMO that were accessed during the simulation.
There are cases when you want to query some information from the IMO,
but you do not want it to be tracked. (For instance, you are just
navigating through the tree of entities.) In this case, you can pass
track=False
to the Imo.query()
method: the object you have
queried will not be included in the final report produced by the
Simulation
object.
API reference¶
-
class
litebird_sim.imo.
Imo
(flatfile_location=None, url=None, user=None, password=None)¶ Bases:
object
-
get_list_of_data_files
(quantity_uuid: uuid.UUID, track=False) → List[uuid.UUID]¶ Return a sorted list of the UUIDs of the data files belonging to a quantity.
The result is sorted according to their upload date (oldest first, newest last).
If
track
is True, then the UUID of the object will be kept in memory and will be returned by the methodget_queried_data_files()
. The default isFalse
, as this function is typically used to check which data files are available, not because the caller is going to use each of them.
-
get_queried_data_files
()¶ Return a list of the UUIDs of data files queried so far.
-
get_queried_entities
()¶ Return a list of the UUIDs of entities queried so far.
-
get_queried_quantities
()¶ Return a list of the UUIDs of quantities queried so far.
-
query
(identifier: Union[str, uuid.UUID], track=True)¶ Query an object from the IMO
The value of identifier can be one of the following:
The string
/quantities/UUID
, withUUID
being the UUID of a quantityThe string
/entities/UUID
, which looks for an entityThe string
/format_specs/UUID
, which looks for an entityThe string
/data_files/UUID
, which looks for a data fileA UUID object: in this case, the method assumes that a data file is being queried.
A path in the form
/release/entity/tree/…/quantity
; in this case, the method looks for the data file belonging toquantity
within the entity tree and assigned to the specified release.
The method returns an object belonging to one of the following classes:
DataFile
,Quantity
,Entity
,FormatSpecification
.If
track
is True (the default), then the UUID of the object will be kept in memory and will be returned by the methodget_queried_data_files()
.
-
query_data_file
(identifier: Union[str, uuid.UUID], track=True) → litebird_sim.imo.objects.DataFile¶ Return a
DataFile
object from an UUID.If
track
is True (the default), then the UUID of the object will be kept in memory and will be returned by the methodget_queried_data_files()
.
-
query_entity
(identifier: uuid.UUID, track=True) → litebird_sim.imo.objects.Entity¶ Return a
Entity
object from an UUID.If
track
is True (the default), then the UUID of the object will be kept in memory and will be returned by the methodget_queried_entities()
.
-
query_quantity
(identifier: uuid.UUID, track=True) → litebird_sim.imo.objects.Quantity¶ Return a
Quantity
object from an UUID.If
track
is True (the default), then the UUID of the object will be kept in memory and will be returned by the methodget_queried_quantities()
.
-
-
class
litebird_sim.imo.
FormatSpecification
(uuid: uuid.UUID, document_ref: str, title: str, doc_file_name: Optional[pathlib.Path], doc_mime_type: str, file_mime_type: str)¶ Bases:
object
A format specification for a quantity in the database.
Format specifications are document that detail the file format used to encode a quantity in a file or in a dictionary (metadata).
Any
Quantity
object must point to a valid FormatSpecification object: in this way, the database ensures that the data in the database can be interpreted by users.Fields of this class:
uuid
: a sequence of bytes uniquely identifying this resource (uuid.UUID
type)document_ref
: an unique label identifying the document; the database does not enforce any scheme for this string (apart from its uniqueness).title
: the title of the documentdoc_file_name
: apathlib.Path
object pointing to a local copy of the document, orNone
if no document is provided.doc_mime_type
: the MIME type of the document. This specifies the format of the document (e.g., PDF, Microsoft Word, etc.), and it can be useful when prompting the user to open itfile_mime_type
: the MIME type of the file being described in this specification. For instance, if the document describes the format used to save Healpix maps in FITS files (columns, measure units, etc.), the value offile_mime_type
must beapplication/fits
.
-
class
litebird_sim.imo.
Entity
(uuid: uuid.UUID, name: str, full_path: str, parent: Optional[uuid.UUID])¶ Bases:
object
An entity describing some part of the experiment.
Entities are a generic concept that is used by the IMO to group quantities related to the same part of the instrument (e.g., detector, HWP, telescope, …).
Fields:
uuid
: a sequence of bytes uniquely identifying this resource (uuid.UUID
type)name
: a string descriptive name of the quantity. It can contain only letters, numbers, or the characters_
(underscore) and-
(hyphen).full_path
: a string containing the full path of this entity, considering also the parents in the hierarchical tree of entries.parent
: the UUID of the parent entity, or None if this is a root entity.quantities
: aset
object containing the UUID of each quantity belonging to this entity (see theQuantity
class).
-
class
litebird_sim.imo.
Quantity
(uuid: uuid.UUID, name: str, format_spec: Optional[uuid.UUID], entity: uuid.UUID)¶ Bases:
object
A quantity stored in the IMO database.
Quantities are meant to gather numbers and actual information about entities (
Entity
) in the IMO database. Each quantity must be connected to aFormatSpecification
object, which specifies how the information has been computed and stored.The actual information is kept in a
DataFile
: this further hierarchical level permits to keep several versions of the data.uuid
: a sequence of bytes uniquely identifying this resource (uuid.UUID
type)name
: a string containing the name of the quantity. This must contain only letters, numbers, or the characters_
(underscore) and-
(hyphen).format_spec
: either the UUID of aFormatSpecification
object, or None.entity
: the UUID of aEntity
object.data_files
: a set containing the UUIDs of the data files belonging to this quantity.
-
class
litebird_sim.imo.
Release
(tag: str, rel_date: datetime.datetime, comments: str, data_files: Set[uuid.UUID])¶ Bases:
object
An official release of the IMO.
This class encodes a list of UUIDs for the set of
DataFile
objects that compose an IMO release.Fields:
tag
: an unique string identifying the release, e.g.,v1.0
.rel_date
: a datetime object containing the release date of the IMO.comments
: a free-form string.data_files
: aset
object containing the UUIDs of theDataFile
objects that make this release.
-
exception
litebird_sim.imo.
ImoFormatError
¶ Bases:
Exception
-
class
litebird_sim.imo.
ImoFlatFile
(path)¶ Bases:
object
A class that interfaces with a flat-file representation of an IMO.
-
check_consistency
()¶ Perform some basic checks on the structure of the flat-file IMO
If the structure of the folders is not compliant with the specifications, raise a
ImoFormatError
exception.The checks are not comprehensive, but they should spot the most obvious errors.
-
parse_schema
(schema: Dict[str, Any])¶
-
quantity_path
(uuid: uuid.UUID)¶
-
query
(identifier: Union[str, uuid.UUID])¶ Query an object from the IMO
The value of identifier can be one of the following:
The string
/quantities/UUID
, withUUID
being the UUID of a quantityThe string
/entities/UUID
, which looks for an entityThe string
/format_specs/UUID
, which looks for an entityThe string
/data_files/UUID
, which looks for a data fileA UUID object: in this case, the method assumes that a data file is being queried.
A path in the form
/release/entity/tree/…/quantity
; in this case, the method looks for the data file belonging toquantity
within the entity tree and assigned to the specified release.
The method returns an object belonging to one of the following classes:
DataFile
,Quantity
,Entity
,FormatSpecification
.
-
query_data_file
(identifier: Union[str, uuid.UUID]) → litebird_sim.imo.objects.DataFile¶ Retrieve a data file
The identifier parameter can be one of the following types:
A
uuid.UUID
objectA string representing a UUID
A full path to an object included in a release. In this case, the path has the following form:
/relname/sequence/of/entities/…/quantity
-
query_entity
(identifier: uuid.UUID) → litebird_sim.imo.objects.Entity¶
-
query_quantity
(identifier: uuid.UUID) → litebird_sim.imo.objects.Quantity¶
-
read_schema
()¶ Read the JSON file containing the metadata
-