Hide navigation sidebar
Hide table of contents sidebar
Back to top
Toggle table of contents sidebar

# Repository Structure

- `modules/lib_vnr` - The `lib_vnr` library directory within `https://github.com/xmos/fwk_voice/`. Within `lib_vnr`:

  - `api/` - Header files containing the public API for `lib_vnr`.

  - `doc/` - Library documentation source (for non-embedded documentation) and build directory.

  - `src/` - Library source code.

# Requirements

`lib_vnr` is included as part of the `fwk_voice` github repository and all requirements for cloning and building `fwk_voice` apply. It depends on `lib_xcore_math` and the xmos-ai-tools python package.

# API Structure

The API is split into 2 parts; feature extraction and inference. The feature extraction API processes an input audio frame to extract features that are input to the inference stage. The inference API has functions for running inference using the VNR TensorFlow Lite model to predict the speech to noise ratio. Both feature extraction and inference APIs have initialisation functions that are called only once at device initialisation and processing functions that are called every frame. The performance requirement is relative low, around 5 MIPS for initialisation and 3 MIPS for processing, and as such is supplied as a single threaded implementation only.

# Getting and Building

The VNR estimator module is obtained as part of the parent `fwk_voice` repo clone. It is present in `fwk_voice/modules/lib_vnr`

Both feature extraction and the inference parts of `lib_vnr` can be compiled as static libraries. The application can link against `libfwk_voice_module_lib_vnr_features.a` and/or `libfwk_voice_module_lib_vnr_inference.a` and add `lib_vnr/api/features` and/or `lib_vnr/api/inference` and `lib_vnr/api/common` as include directories.

# VNR Inference Model

The VNR estimator module uses a neural network model to predict the SNR of speech in noise for incoming data. The model used is a pre trained TensorFlow Lite model that has been optimised for the XCORE architecture using the xmos-ai-tools xformer. The optimised model is compiled as part of the VNR Inference Engine. Changing the model at runtime is not supported. If changing to a different model, the application needs to generate the model related files and recompile. This process is automated through the build system, as described below.

## Integrating a TensorFlow Lite model into the VNR module

To integrate the new TensorFlow Lite model into the VNR module:

1. Put an unoptimised model into `fwk_voice/modules/lib_vnr/python/model/model_output/trained_model.tflite`

2. Rerun the build tool of our choice (`make` or `ninja`, for example)

This will use xmos-ai-tools to optimise `.tflite` model for xcore and generate `.cpp` and `.h` files into `fwk_voice/modules/lib_vnr/src/inference/model/`. Those generated files will be picked by the build system and compiled into the VNR module.

The process described above only generates an optimised model that would run on a single core.

Any new models replacing the existing one should have the same set of input features, input and output size, and data types as the existing model. If changes to the features are made, the feature extraction code must be updated. Note that the VNR is used to control the IC behavior, and so its performance may also change.