HomeWorld

MetaProcessor

This is a backup of the main documentation, use GitHub to read the source code and docs.

Initialization

On initial installation, run:

mp config init

This will create a new file located at $XDG_CONFIG_HOME/metaprocessor/config.toml (~/.config/metaprocessor/config.toml if $XDG_CONFIG_HOME is not set).

Interactive setup is also avaiable, run:

mp config init --interactive

If configuration is already present, an error will be thrown. To overwrite the existing configuration, run:

mp config init --force

Edit

To edit the configuration file, run:

mp config edit

This will open the configuration file in the default editor (as specified by the EDITOR environment variable), if the editor is not set, it will use vim.

If the configuration file is not present, an error will be thrown. To create a new configuration file, run:

mp config init

Check

Use the following command to check the configuration file:

mp config check

There are two types of checks:

  1. Unfilled Options check: This is to check if there are any unfilled options in the configuration file. If there are any unfilled options, an error will be thrown.
  2. Invalid Type check: This is to check if the type of the options are correct. If there are any invalid types, an error will be thrown.

If the configuration file is not present, an error will be thrown. To create a new configuration file, run:

mp config init

Show

This command is to show the configuration file:

mp config show

If no flag is specified, the configuration file will be printed to the console in a human-readable format.

If the --json flag is specified, the configuration file will be printed to the console in JSON format, it can be further processed by other programs like jq.

If the configuration file is not present, an error will be thrown. To create a new configuration file, run:

mp config init

Example

[general]
utc-offset = -7.0
gd-location = "~/MetaProcessor/Data/"
gp-regex = "*"
sg-regex = "(END|AEO)[0-9]{3}" # END or AEO followed by 3 digits
sp-regex = "BHC[0-9]{4}-[0-9]" # BHC followed by 4 digits and a dash and a digit
sd-regex = "D[0-9]{3}"         # D followed by 3 digits

[aws]
bucket = "<bucket name>"
access-key = "<access key>"
secret-key = "<secret key>"

Enabling Shell Completion

To enable shell completion, run the following command (modification required, do not copy/paste directly):

eval "$(mp|metaprocessor completion bash|zsh|fish)"

Depending on your preferences, you should use either mp or metaprocessor in the command above (no difference). Also, you should replace bash|zsh|fish with the shell you are using. In the following guides, we will use mp for simplicity.

Bash

To enable shell completion for bash, run the following command:

eval "$(mp completion bash)"

To enable shell completion for bash at the start of each interactive shell session, run the following command:

echo 'eval "$(mp completion bash)"' >> ~/.bashrc

and restart your shell session.

Zsh

To enable shell completion for zsh, run the following command:

eval "$(mp completion zsh)"

To enable shell completion for zsh at the start of each interactive shell session, run the following command:

echo 'eval "$(mp completion zsh)"' >> ~/.zshrc

and restart your shell session.

Fish

To enable shell completion for fish, run the following command:

mp completion fish | source

To enable shell completion for fish at the start of each interactive shell session, run the following command:

mp completion fish > ~/.config/fish/completions/mp.fish

and restart your shell session.

Session Management

What's a "session"?

A session is a data point containing raw IMU data generated by a MetaMotion device. Clinicians uses MetaProcessor to download sessions from MetaMotion devices and then manually typed in the corresponding metadata (e.g. session ID, start/end date etc.) for each session. All the IMU data and metadata are then bundled together and uploaded to a object store service provider.

The session data management tool can list, move, remove, upload, and download session data from local file system or on remote object store service provider.

List Sessions

To list all sessions (local or remote), use the ls command:

mp object ls

To list remote sessions only, use the --remote flag:

mp object ls --remote

To list local sessions only, use the --local flag:

mp object ls --local

The --remote and --local flags are mutually exclusive, to list all sessions, simple omit both flags.

The output of the ls command can be serialized to JSON format using the --json flag:

mp object ls --json

For more information, run mp object ls --help.

Move Objects

Move (or rename) objects managed by MetaProcessor.

On roadmap, not implemented yet.

Remove Objects

Delete objects from MetaProcessor, local file system, or remote object store service provider.

On roadmap, not implemented yet.

Upload Objects

Upload objects to remote object store service provider.

On roadmap, not implemented yet.

Download Objects

Download objects from remote object store service provider.

To download all objects, use the download command (concurrent by default):

mp object download

To download a specific object, use the --key flag:

mp object download --key <object_key>

The object key can be obtained by running the ls command.

The download command is lazy, it will only download the object if it doesn't exist locally, or local object's checksum doesn't match the remote object's checksum.

For more information, run mp object download --help.

Running Workflows

Running MetaProcessor workflows, available workflows are:

Preprocessor

Before running steps/day or uptime algorithms, you need to run preprocessor workflow to prepare all session data for next steps.

To run preprocessor workflow for all sessions stored locally, run:

mp run preprocess

To run preprocessor workflow for specific session, run:

mp run preprocess --key <KEY>

Please note that preprocessor will only run for sessions that are saved locally. If you are getting errors, please check if you have downloaded the session data.

Steps/Day

To run steps/day algorithm for all sessions stored locally, run:

mp run steps

To run steps/day algorithm for specific session, run:

mp run steps --key <KEY>

Please note that steps/day algorithm will only run for sessions that are saved locally, and have been preprocessed. If you are getting errors, please check if you have downloaded the session data and run preprocessor.

Results will be saved to <Session ID>-Results.csv file, to get the results, find the file with corresponding session ID, or, check report generation section.

Upright Position Time

To run uptime algorithm for all sessions stored locally, run:

mp run uptime

To run uptime algorithm for specific session, run:

mp run uptime --key <KEY>

Please note that uptime algorithm will only run for sessions that are saved locally, and have been preprocessed. If you are getting errors, please check if you have downloaded the session data and run preprocessor.

Results will be saved to <Session ID>-Results.csv file, to get the results, find the file with corresponding session ID, or, check report generation section.

Report Generation

After running object download, preprocessor, steps/day and uptime algorithms, you can generate a report for all sessions or specific session.

To generate report for all sessions, run:

mp run report

To generate report for specific session, run:

mp run report --key <KEY>

The report generator will go through all locally stored results (<Session ID>-Results.csv) and generate a .csv file to current working directory, with all saved results.

The format of the report is:

id, day_1, day_2, ..., day_n
<Session ID>, <Feature 1>, <Feature 2>, ..., <Feature n>

For example:

UpTime report may look like mp-report-yyyy-mm-dd-hh-mm-ss-uptime.csv:

id,day_1,day_2,day_3,day_4,day_5,day_6,day_7,day_8,day_9,day_10,day_11,day_12,day_13,day_14,day_15,day_16,day_17,day_18,day_19,day_20,day_21,day_22,day_23,day_24
AEO000-BHC0000-0-D00-Visit-0,0.2302858553377207,0.313004209806334,0.320062796838697,0.2184967770350215,0.1407022845496559,0.1267014793200334,0.241944586278251,,,,,,,,,,,,,,,,,

Steps/Day report may look like mp-report-yyyy-mm-dd-hh-mm-ss-steps_count.csv:

id,day_1,day_2,day_3,day_4,day_5,day_6,day_7,day_8,day_9,day_10,day_11,day_12,day_13,day_14,day_15,day_16,day_17,day_18,day_19,day_20,day_21,day_22,day_23,day_24
AEO000-BHC0000-0-D00-Visit-0,5224,5053,6678,4988,3211,2340,217,,,,,,,,,,,,,,,,,

Use as a Library

Install a stable release of MetaProcessor from PyPI:

pip install metaprocessor

Or install the latest development version from GitHub:

pip install git+https://github.com/stepbrobd/metaprocessor#egg=metaprocessor

Import

Import MetaProcessor into your Python code:

import metaprocessor as mp

steps = mp.steps(*args, **kwargs)
uptime = mp.uptime(*args, **kwargs)

The reset is up to your imagination.

Extending MetaProcessor

You can extend MetaProcessor by forking the repository. If you find your extension useful and should be made generally available, please consider contributing it back to the project.

MetaProcessor is built using Nix, to reduce the complexity of setting up the development environment. To get started, install Nix on your system using either the official installation instructions or the Determinate Nix Installer(recommended).

With Nix installed, you can a start development shell with all the dependencies needed to build MetaProcessor with direnv:

git clone [email protected]:stepbrobd/metaprocessor.git && cd metaprocessor && direnv allow

After the console stops making outputs, MetaProcessor is already installed (editable) in your environment. direnv will load and unload the environment automatically when you enter and leave the directory. To make sure everything is working:

mp --help

After this, change the remote to your fork and start hacking!

Additional Resources:

Contributing

To contribute, please fork the repository and create a pull request.

If you are adding new CLI commands, please add them to metaprocessor/cli.py.

If you are adding modules should be exposed as library functions, please add them metaprocessor/__init__.py and add a corresponding test cases.

Last updated on

# Use Glow to view this page in your terminal
glow http://ysun.co/contents/2023/05/metaprocessor.md
# If you have Nix on your system
nix --extra-experimental-features "nix-command flakes" run "nixpkgs#glow" -- http://ysun.co/contents/2023/05/metaprocessor.md