|
openPMD-api
|
Implementation for the root level of the openPMD hierarchy. More...
#include <Series.hpp>


Classes | |
| struct | ParsedInput |
Public Types | |
| enum class | IterationOpened : bool { HasBeenOpened , RemainsClosed } |
| using | IterationIndex_t = Iteration::IterationIndex_t |
| An unsigned integer type, used to identify Iterations in a Series. | |
| using | IterationsContainer_t = internal::SeriesData::IterationsContainer_t |
| Type for a container of Iterations indexed by IterationIndex_t. | |
| using | iterations_t = decltype(internal::SeriesData::iterations) |
| using | iterations_iterator = iterations_t::iterator |
| using | Data_t = internal::SeriesData |
Public Types inherited from openPMD::Attributable | |
| enum | ReadMode { IgnoreExisting , OverrideExisting , FullyReread } |
| enum class | EnqueueAsynchronously : bool { Yes , No } |
Public Member Functions | |
| Series (std::string const &filepath, Access at, MPI_Comm comm, std::string const &options="{}") | |
| Construct a new Series. More... | |
| Series (std::string const &filepath, Access at, std::string const &options="{}") | |
| Construct a new Series. More... | |
| Series (Series const &)=default | |
| Series (Series &&)=default | |
| Series & | operator= (Series const &)=default |
| Series & | operator= (Series &&)=default |
| operator bool () const | |
| Is this a usable Series object? More... | |
| std::string | openPMD () const |
| Series & | setOpenPMD (std::string const &openPMD) |
| Set the version of the enforced openPMD standard. More... | |
| uint32_t | openPMDextension () const |
| Series & | setOpenPMDextension (uint32_t openPMDextension) |
| Set a 32-bit mask of applied extensions to the openPMD standard. More... | |
| std::string | basePath () const |
| Series & | setBasePath (std::string const &basePath) |
| Set the common prefix for all data sets and sub-groups of a specific iteration. More... | |
| std::string | meshesPath () const |
| Series & | setMeshesPath (std::string const &meshesPath) |
Set the path to mesh records, relative(!) to basePath. More... | |
| bool | hasRankTableRead () |
| chunk_assignment::RankMeta | rankTable (bool collective) |
| Series & | setRankTable (std::string const &myRankInfo) |
| Set the Mpi Ranks Meta Info attribute, i.e. More... | |
| std::string | particlesPath () const |
| Series & | setParticlesPath (std::string const &particlesPath) |
Set the path to groups for each particle species, relative(!) to basePath. More... | |
| std::string | author () const |
| Series & | setAuthor (std::string const &author) |
| Indicate the author and contact for the information in the file. More... | |
| std::string | software () const |
| Series & | setSoftware (std::string const &newName, std::string const &newVersion=std::string("unspecified")) |
| Indicate the software/code/simulation that created the file. More... | |
| std::string | softwareVersion () const |
| Series & | setSoftwareVersion (std::string const &softwareVersion) |
| Indicate the version of the software/code/simulation that created the file. More... | |
| std::string | date () const |
| Series & | setDate (std::string const &date) |
| Indicate the date of creation. More... | |
| std::string | softwareDependencies () const |
| Series & | setSoftwareDependencies (std::string const &newSoftwareDependencies) |
| Indicate dependencies of software that were used to create the file. More... | |
| std::string | machine () const |
| Series & | setMachine (std::string const &newMachine) |
| Indicate the machine or relevant hardware that created the file. More... | |
| IterationEncoding | iterationEncoding () const |
| Series & | setIterationEncoding (IterationEncoding iterationEncoding) |
| Set the encoding style for multiple iterations in this series. More... | |
| std::string | iterationFormat () const |
| Series & | setIterationFormat (std::string const &iterationFormat) |
| Set a pattern describing how to access single iterations in the raw file. More... | |
| std::string | name () const |
| Series & | setName (std::string const &name) |
| Set the pattern for file names. More... | |
| std::string | backend () const |
| The currently used backend. More... | |
| std::string | backend () |
| void | flush (std::string backendConfig="{}") |
| Execute all required remaining IO operations to write or read data. More... | |
| ReadIterations | readIterations () |
| Entry point to the reading end of the streaming API. More... | |
| Snapshots | snapshots () |
| Preferred way to access Iterations/Snapshots. More... | |
| void | parseBase () |
| Parse the Series. More... | |
| WriteIterations | writeIterations () |
| Entry point to the writing end of the streaming API. More... | |
| void | close () |
| Close the Series and release the data storage/transport backends. More... | |
| template<typename X = void, typename... Args> | |
| auto | iterationFlush (Args &&...) |
| This overrides Attributable::iterationFlush() which will fail on Series. | |
| Data_t & | get () |
| Data_t const & | get () const |
| void | setData (std::shared_ptr< internal::SeriesData > series) |
| std::unique_ptr< ParsedInput > | parseInput (std::string) |
| template<typename TracingJSON > | |
| void | parseJsonOptions (TracingJSON &options, ParsedInput &) |
| Parse non-backend-specific configuration in JSON config. More... | |
| bool | hasExpansionPattern (std::string filenameWithExtension) |
| bool | reparseExpansionPattern (std::string filenameWithExtension) |
| template<typename... MPI_Communicator> | |
| void | init (std::string const &filepath, Access at, std::string const &options, MPI_Communicator &&...) |
| template<typename TracingJSON , typename... MPI_Communicator> | |
| std::tuple< std::unique_ptr< ParsedInput >, TracingJSON > | initIOHandler (std::string const &filepath, std::string const &options, Access at, bool resolve_generic_extension, MPI_Communicator &&...) |
| void | initSeries (std::unique_ptr< AbstractIOHandler >, std::unique_ptr< ParsedInput >) |
| void | initDefaults (IterationEncoding, bool initAll=false) |
| std::future< void > | flush_impl (iterations_iterator begin, iterations_iterator end, internal::FlushParams const &flushParams, bool flushIOHandler=true) |
| Internal call for flushing a Series. More... | |
| void | flushFileBased (iterations_iterator begin, iterations_iterator end, internal::FlushParams const &flushParams, bool flushIOHandler=true) |
| void | flushGorVBased (iterations_iterator begin, iterations_iterator end, internal::FlushParams const &flushParams, bool flushIOHandler=true) |
| void | flushMeshesPath () |
| void | flushParticlesPath () |
| void | flushRankTable () |
| void | readFileBased (std::optional< IterationIndex_t > read_only_this_single_iteration) |
| void | readOneIterationFileBased (std::string const &filePath) |
| std::vector< IterationIndex_t > | readGorVBased (bool do_always_throw_errors, bool init, std::optional< IterationIndex_t > read_only_this_single_iteration) |
| Note on re-parsing of a Series: If init == false, the parsing process will seek for new Iterations/Records/Record Components etc. More... | |
| void | readBase () |
| std::string | iterationFilename (IterationIndex_t i) |
| IterationOpened | openIterationIfDirty (IterationIndex_t index, Iteration &iteration) |
| void | openIteration (IterationIndex_t index, Iteration &iteration) |
| iterations_iterator | indexOf (Iteration const &) |
| Find the given iteration in Series::iterations and return an iterator into Series::iterations at that place. | |
| AdvanceStatus | advance (AdvanceMode mode, internal::AttributableData &file, iterations_iterator it) |
| In step-based IO mode, begin or end an IO step for the given iteration. More... | |
| AdvanceStatus | advance (AdvanceMode mode) |
| void | flushStep (bool doFlush) |
| Called at the end of an IO step to store the iterations defined in the IO step to the snapshot attribute and to store that at least one step was written. More... | |
| Series & | setIterationEncoding_internal (IterationEncoding iterationEncoding, internal::default_or_explicit) |
| std::optional< std::vector< IterationIndex_t > > | currentSnapshot () |
| AbstractIOHandler * | runDeferredInitialization () |
| AbstractIOHandler * | IOHandler () |
| AbstractIOHandler const * | IOHandler () const |
| std::optional< std::vector< std::vector< IterationIndex_t > > > | preparseSnapshots () |
| Snapshots | makeRandomAccessSnapshots () |
| Snapshots | makeSynchronousSnapshots () |
| bool | randomAccessSteps () const |
| std::vector< std::string > | availableDatasets () |
| template<typename... MPI_Communicator> | |
| void | init (std::string const &filepath, Access at, std::string const &options, MPI_Communicator &&...comm) |
| template<typename TracingJSON , typename... MPI_Communicator> | |
| auto | initIOHandler (std::string const &filepath, std::string const &options, Access at, bool resolve_generic_extension, MPI_Communicator &&...comm) -> std::tuple< std::unique_ptr< ParsedInput >, TracingJSON > |
Public Member Functions inherited from openPMD::Attributable | |
| Attributable (NoInit) noexcept | |
| Attribute | getAttribute (std::string const &key) const |
| Retrieve value of Attribute stored with provided key. More... | |
| bool | deleteAttribute (std::string const &key) |
| Remove Attribute of provided value both logically and physically. More... | |
| std::vector< std::string > | attributes () const |
| List all currently stored Attributes' keys. More... | |
| size_t | numAttributes () const |
| Count all currently stored Attributes. More... | |
| bool | containsAttribute (std::string const &key) const |
| Check whether am Attribute with a given key exists. More... | |
| std::string | comment () const |
| Retrieve a user-supplied comment associated with the object. More... | |
| Attributable & | setComment (std::string const &comment) |
| Populate Attribute corresponding to a comment with the user-supplied comment. More... | |
| void | seriesFlush (std::string backendConfig="{}") |
| Flush the corresponding Series object. More... | |
| void | iterationFlush (std::string backendConfig="{}") |
| Flush the containing Iteration. More... | |
| MyPath | myPath () const |
| The path to this object within its containing Series. More... | |
| void | touch () |
| Sets the object dirty to make internal procedures think it has been modified. | |
| OpenpmdStandard | openPMDStandard () const |
| protected Series | retrieveSeries () const |
| template<bool flush_entire_series> | |
| void | seriesFlush_impl (internal::FlushParams const &) |
| void | flushAttributes (internal::FlushParams const &) |
| void | readAttributes (ReadMode) |
| template<typename T > | |
| T | readFloatingpoint (std::string const &key) const |
| Retrieve the value of a floating point Attribute of user-defined precision with ensured type-safety. More... | |
| template<typename T > | |
| std::vector< T > | readVectorFloatingpoint (std::string const &key) const |
| Retrieve a vector of values of a floating point Attributes of user-defined precision with ensured type-safety. More... | |
| AbstractIOHandler * | IOHandler () |
| AbstractIOHandler const * | IOHandler () const |
| Writable *& | parent () |
| Writable const * | parent () const |
| Writable & | writable () |
| Writable const & | writable () const |
| void | setData (std::shared_ptr< internal::AttributableData > attri) |
| internal::SharedAttributableData & | get () |
| internal::SharedAttributableData const & | get () const |
| bool | dirty () const |
| bool | dirtyRecursive () const |
| O(1). | |
| void | setDirty (bool dirty_in) |
| void | setDirtyRecursive (bool dirty_in) |
| bool | written () const |
| void | setWritten (bool val, EnqueueAsynchronously) |
| template<typename T > | |
| bool | setAttribute (std::string const &key, T value) |
| Populate Attribute of provided name with provided value. More... | |
| bool | setAttribute (std::string const &key, char const value[]) |
| auto | containingIteration () const -> std::pair< std::optional< internal::IterationData const * >, internal::SeriesData const * > |
| Returns the corresponding Iteration. More... | |
| auto | containingIteration () -> std::pair< std::optional< internal::IterationData * >, internal::SeriesData * > |
Public Attributes | |
| IterationsContainer_t | iterations |
| std::shared_ptr< Data_t > | m_series = nullptr |
Static Public Attributes | |
| static constexpr private char const *const | BASEPATH = "/data/%T/" |
Friends | |
| class | Attributable |
| class | Iteration |
| class | Writable |
| class | ReadIterations |
| class | StatefulIterator |
| class | internal::SeriesData |
| class | internal::AttributableData |
| class | StatefulSnapshotsContainer |
Additional Inherited Members | |
Protected Types inherited from openPMD::Attributable | |
| using | Data_t = internal::AttributableData |
Protected Attributes inherited from openPMD::Attributable | |
| std::shared_ptr< Data_t > | m_attri |
Implementation for the root level of the openPMD hierarchy.
Entry point and common link between all iterations of particle and mesh data.
| openPMD::Series::Series | ( | std::string const & | filepath, |
| Access | at, | ||
| MPI_Comm | comm, | ||
| std::string const & | options = "{}" |
||
| ) |
Construct a new Series.
For further details, refer to the documentation of the non-MPI overload.
| filepath | The file path. |
| at | Access mode. |
| comm | The MPI communicator. |
| options | Advanced backend configuration via JSON. May be specified as a JSON-formatted string directly, or as a path to a JSON textfile, prepended by an at sign '@'. |
| openPMD::Series::Series | ( | std::string const & | filepath, |
| Access | at, | ||
| std::string const & | options = "{}" |
||
| ) |
Construct a new Series.
For details on access modes, JSON/TOML configuration and iteration encoding, refer to:
In case of file-based iteration encoding, the file names for each iteration are determined by an expansion pattern that must be specified. It takes one out of two possible forms:
simData_T.bp becomes simData_50.bp.simData_%06T.bp becomes simData_000050.bp.The backend is determined:
backend, e.g. {"backend": "adios2"}.simData_T.h5.The filename extension can be replaced with a globbing pattern E. It will be replaced with an automatically determined file name extension:
| filepath | The file path. |
| at | Access mode. |
| options | Advanced backend configuration via JSON. May be specified as a JSON/TOML-formatted string directly, or as a path to a JSON/TOML textfile, prepended by an at sign '@'. |
| AdvanceStatus openPMD::Series::advance | ( | AdvanceMode | mode, |
| internal::AttributableData & | file, | ||
| iterations_iterator | it | ||
| ) |
In step-based IO mode, begin or end an IO step for the given iteration.
Called internally by Iteration::beginStep and Iteration::endStep.
| mode | Whether to begin or end a step. |
| file | The Attributable representing the iteration. In file-based iteration layout, this is an Iteration object, in group- based layout, it's the Series object. |
| it | The iterator within Series::iterations pointing to that iteration. |
| std::string openPMD::Series::author | ( | ) | const |
| no_such_attribute_error | If optional attribute is not present. |
| std::string openPMD::Series::backend | ( | ) | const |
The currently used backend.
| std::string openPMD::Series::basePath | ( | ) | const |
| void openPMD::Series::close | ( | ) |
Close the Series and release the data storage/transport backends.
This is an explicit API call for what the Series::~Series() destructor would do otherwise. All backends are closed after calling this method. The Series should be treated as destroyed after calling this method. The Series will be evaluated as false in boolean contexts after calling this method.
| std::string openPMD::Series::date | ( | ) | const |
| no_such_attribute_error | If optional attribute is not present. |
| void openPMD::Series::flush | ( | std::string | backendConfig = "{}" | ) |
Execute all required remaining IO operations to write or read data.
| backendConfig | Further backend-specific instructions on how to implement this flush call. Must be provided in-line, configuration is not read from files. |
| std::future< void > openPMD::Series::flush_impl | ( | iterations_iterator | begin, |
| iterations_iterator | end, | ||
| internal::FlushParams const & | flushParams, | ||
| bool | flushIOHandler = true |
||
| ) |
Internal call for flushing a Series.
Any flushing of the Series will pass through this call.
| begin | Start of the range of iterations to flush. |
| end | End of the range of iterations to flush. |
| flushParams | Flush params, as documented in AbstractIOHandler.hpp. |
| flushIOHandler | Tasks will always be enqueued to the backend. If this flag is true, tasks will be flushed to the backend. |
| void openPMD::Series::flushStep | ( | bool | doFlush | ) |
Called at the end of an IO step to store the iterations defined in the IO step to the snapshot attribute and to store that at least one step was written.
| doFlush | If true, flush the IO handler. |
| bool openPMD::Series::hasRankTableRead | ( | ) |
| IterationEncoding openPMD::Series::iterationEncoding | ( | ) | const |
| std::string openPMD::Series::iterationFormat | ( | ) | const |
| std::string openPMD::Series::machine | ( | ) | const |
| no_such_attribute_error | If optional attribute is not present. |
| std::string openPMD::Series::meshesPath | ( | ) | const |
| no_such_attribute_error | If optional attribute is not present. |
basePath. | std::string openPMD::Series::name | ( | ) | const |
| std::string openPMD::Series::openPMD | ( | ) | const |
| uint32_t openPMD::Series::openPMDextension | ( | ) | const |
| openPMD::Series::operator bool | ( | ) | const |
| void openPMD::Series::parseBase | ( | ) |
Parse the Series.
Only necessary in linear read mode. In linear read mode, the Series constructor does not do any IO accesses. This call effectively triggers the side effects of Series::readIterations(), for use cases where data needs to be accessed before iterating through the iterations.
The reason for introducing this restricted alias to Series::readIterations() is that the name "readIterations" is misleading for that use case: When using IO steps, this call only ensures that the first step is parsed.
| void openPMD::Series::parseJsonOptions | ( | TracingJSON & | options, |
| ParsedInput & | input | ||
| ) |
Parse non-backend-specific configuration in JSON config.
Currently this parses the keys defer_iteration_parsing, backend and iteration_encoding.
| TracingJSON | template parameter so we don't have to include the JSON lib here |
| std::string openPMD::Series::particlesPath | ( | ) | const |
| no_such_attribute_error | If optional attribute is not present. |
basePath. | chunk_assignment::RankMeta openPMD::Series::rankTable | ( | bool | collective | ) |
| no_such_attribute_error | If optional attribute is not present. |
| collective | Run this read operation collectively. There might be an enormous IO overhead if running this operation non-collectively. To make this explicit to users, there is no default parameter. Parameter is ignored if compiling without MPI support, (it is present for the sake of a consistent API). |
| auto openPMD::Series::readGorVBased | ( | bool | do_always_throw_errors, |
| bool | init, | ||
| std::optional< IterationIndex_t > | read_only_this_single_iteration | ||
| ) |
Note on re-parsing of a Series: If init == false, the parsing process will seek for new Iterations/Records/Record Components etc.
If series.iterations contains the attribute snapshot, returns its value. If do_always_throw_errors is false, this method will try to handle errors and turn them into a warning (useful when parsing a Series, since parsing should succeed without issue). If true, the error will always be re-thrown (useful when using ReadIterations since those methods should be aware when the current step is broken). Parameter read_only_this_single_iteration used for reopening an Iteration after closing it.
| ReadIterations openPMD::Series::readIterations | ( | ) |
Entry point to the reading end of the streaming API.
Creates and returns an instance of the ReadIterations class which can be used for iterating over the openPMD iterations in a C++11-style for loop. Series::readIterations() is an intentionally restricted API that ensures a workflow which also works in streaming setups, e.g. an iteration cannot be opened again once it has been closed. For a less restrictive API in non-streaming situations, Series::iterations can be accessed directly. Look for the ReadIterations class for further documentation.
| Series & openPMD::Series::setAuthor | ( | std::string const & | author | ) |
Indicate the author and contact for the information in the file.
| author | String indicating author and contact for the information in the file. |
| Series & openPMD::Series::setBasePath | ( | std::string const & | basePath | ) |
Set the common prefix for all data sets and sub-groups of a specific iteration.
| basePath | String of the common prefix for all data sets and sub-groups of a specific iteration. |
| Series & openPMD::Series::setDate | ( | std::string const & | date | ) |
Indicate the date of creation.
| date | String indicating the date of creation. |
| Series & openPMD::Series::setIterationEncoding | ( | IterationEncoding | iterationEncoding | ) |
Set the encoding style for multiple iterations in this series.
A preview on the openPMD 2.0 variable-based iteration encoding can be activated with this call. Making full use of the variable-based iteration encoding requires (1) explicit support by the backend (available only in ADIOS2) and (2) use of the openPMD streaming API. In other backends and without the streaming API, only one iteration/snapshot may be written in the variable-based encoding, making this encoding a good choice for single-snapshot data dumps.
| iterationEncoding | Desired encoding style for multiple iterations in this series. |
| Series & openPMD::Series::setIterationFormat | ( | std::string const & | iterationFormat | ) |
Set a pattern describing how to access single iterations in the raw file.
| iterationFormat | String with the iteration regex %T defining either the series of files (fileBased) or the series of groups within a single file (groupBased) that allows to extract the iteration from it. For fileBased formats the iteration must be included in the file name. The format depends on the selected iterationEncoding method. |
| Series & openPMD::Series::setMachine | ( | std::string const & | newMachine | ) |
Indicate the machine or relevant hardware that created the file.
| newMachine | String indicating the machine or relevant hardware that created the file (semicolon-separated list if needed).. |
| Series & openPMD::Series::setMeshesPath | ( | std::string const & | meshesPath | ) |
Set the path to mesh records, relative(!) to basePath.
| meshesPath | String of the path to mesh records, relative(!) to basePath. |
| Series & openPMD::Series::setName | ( | std::string const & | name | ) |
Set the pattern for file names.
| name | String of the pattern for file names. Must include iteration regex %T for fileBased data. |
| Series & openPMD::Series::setOpenPMD | ( | std::string const & | openPMD | ) |
Set the version of the enforced openPMD standard.
| Series & openPMD::Series::setOpenPMDextension | ( | uint32_t | openPMDextension | ) |
Set a 32-bit mask of applied extensions to the openPMD standard.
| openPMDextension | Unsigned 32-bit integer used as a bit-mask of applied extensions. |
| Series & openPMD::Series::setParticlesPath | ( | std::string const & | particlesPath | ) |
Set the path to groups for each particle species, relative(!) to basePath.
| particlesPath | String of the path to groups for each particle species, relative(!) to basePath. |
| Series & openPMD::Series::setRankTable | ( | std::string const & | myRankInfo | ) |
Set the Mpi Ranks Meta Info attribute, i.e.
a Vector with a String per (writing) MPI rank, indicating user- defined meta information per rank. Example: host name.
| Series & openPMD::Series::setSoftware | ( | std::string const & | newName, |
| std::string const & | newVersion = std::string("unspecified") |
||
| ) |
Indicate the software/code/simulation that created the file.
| newName | String indicating the software/code/simulation that created the file. |
| newVersion | String indicating the version of the software/code/simulation that created the file. |
| Series & openPMD::Series::setSoftwareDependencies | ( | std::string const & | newSoftwareDependencies | ) |
Indicate dependencies of software that were used to create the file.
| newSoftwareDependencies | String indicating dependencies of software that were used to create the file (semicolon-separated list if needed). |
| Series & openPMD::Series::setSoftwareVersion | ( | std::string const & | softwareVersion | ) |
Indicate the version of the software/code/simulation that created the file.
| softwareVersion | String indicating the version of the software/code/simulation that created the file. |
| Snapshots openPMD::Series::snapshots | ( | ) |
Preferred way to access Iterations/Snapshots.
Single API for all workflows and access modi.
Two fundamental workflows for Iteration/Snapshot processing exist:
operator++() will advance all other iterators as well. (b) Advancing an iterator is collective, all MPI ranks must participate. The workflow is generally managed more automatically, Iteration::open() is not needed, Iteration::close() is recommended, but not needed. In READ_LINEAR mode, parsed Iteration data is deleted upon closing (and will be reparsed upon reopening) for a better support of datasets with many Snapshots/Iterations. This happens only if user code explicitly calls Iteration::close(). This mode generally brings better performance than random access mode due to the more restricted workflow. For accessing some kinds of Series, Synchronous access is even necessary, e.g. for Streaming workflows or variable-based encoding. (In future, random-access of Series with variable-based encoding will be possible under the condition that each Iteration/Snapshot has the same internal structure).As a rule of thumb, the synchronous workflow should be preferred as long as possible. The random-access workflow should be chosen when more flexible interaction with Snapshots is needed (especially for random-access read patterns).
Random-vs.-Synchronous access is determined by the chosen Access modes: Access modes suffixed with _LINEAR will select the linear workflow, those suffixed with _RANDOM_ACCESS will select the random-access workflows. READ_WRITE selects the RANDOM_ACCESS workflow. Legacy access names (CREATE, APPEND, READ_ONLY) map to their RANDOM_ACCESS counterparts.
| std::string openPMD::Series::software | ( | ) | const |
| no_such_attribute_error | If optional attribute is not present. |
| std::string openPMD::Series::softwareDependencies | ( | ) | const |
| no_such_attribute_error | If optional attribute is not present. |
| std::string openPMD::Series::softwareVersion | ( | ) | const |
| no_such_attribute_error | If optional attribute is not present. |
| WriteIterations openPMD::Series::writeIterations | ( | ) |
Entry point to the writing end of the streaming API.
Shorthand for Series::snapshots() for access types CREATE and APPEND called with parameter SnapshotWorkflow::Synchronous, i.e. for streaming-aware data producers.