Documentation for herc-vk4tool

Contents

• Documentation for herc-vk4tool
  • Description
    • Overview
    • Gwyddion Paths
    • Output Format
      • Footnote on the stats Field
      • Footnote on Loading JSON in MATLAB
    • Caveats
  • Syntax
  • Changelog
  • Author
  • License

A tool for interacting with Keyance Profilometry VK4 formatted data files, built on top of Gwyddion’s automation API. This tool is capable of importing VK4 files, extracting the stored data, and producing both MATLAB matrices containing the data and JSON formatted sidecar files containing metadata about each.

Description

Overview

This tool was written to support analysis of VK4 formatted data files programatically via existing tools such as MATLAB/Octave as well as any other tool which is capable of importing MATLAB .mat format matrices (as well as JSON, if metadata is required).

Gwyddion Paths

Throughout this document and the corresponding source code, the term “Gwyddion Path” is used to refer to the notation which Gwyddion uses to refer to data stored within the container object produced by loading a data file. This is because a single container can store many pieces of data, such as various matrices and metadata about them. Gwyddion paths roughly correspond to POSIX paths in terms of notation. Typically, a single fundamental unit of interest will be identified by the path /N, where N is a positive integer. Usually the data associated with this unit of interest is located at /N/data, and other metadata may be stored at other paths relative to /N. A Gwyddion path might identify a Gwyddion DataField object, another Gwyddion container object, a string, or other types of objects or data types.

Output Format

The raw data of the VK4 file which is input is converted into one or more 2-dimensional matrices and stuffed into a single .mat data file for later import into MATLAB (this is accomplished via scipy.io). The matrix names will be set to a prefix (by default matrix, but this can be overridden using a flag) followed by a _ character and then the first part of the key Gwyddion uses to refer to the matrix (this seems to typically be a positive integer)

As much metadata as can be extracted from the VK4 file will also be extracted and stored in a JSON file with the same basename as the generated .mat file. This JSON file contains nested hash tables where the outer keys are again the first part of the Gwyddion “path” to the relevant objects, and the inner keys uniquely identify a metadata field. There are three types of metadata fields which may be populated:

1. Those obtained from /N/meta in the Gwyddion container produced from the VK4 file. These fields each produce an inner key and a string value in the JSON.

2. Those obtained from member functions such as get_xres() of the Gwyddion DataField object corresponding to /N. These values are stored in the format that they are returned by Gwyddion, and the field names in the JSON file are simply the name of the function call with the get_ removed (i.e. the output get_xres() is stored in the key xres)

3. Those obtained from Gwyddion fields which are of type str, these are stored as is and named after all parts of their Gwyddion path excepting the first part. For example, /N/data/title from a Gwyddion container would result in [N][data/title] containing the value of this field.

Additionally, for each matrix in the generated .mat, a version of the raw data which had a degree 2 polynomial regression subtracted from it (to normalize uneven heights) will be stored with the matrix name prefix_N_fit. A third matrix will be created with the name prefix_N_dim which will contain the x and y dimensions in meters as floating point values in that order.

Footnote on the stats Field

As a convenience feature, the output of .get_stats() is recorded for each DataField object which is processed. The value of this field is a list of floating point numbers, and it has the schema (avg, ra, rms, skew, kurtosis) per the Gwyddion documentation.

Footnote on Loading JSON in MATLAB

JSON was selected as the format for the metadata sidecar files because of the fact that there is an existing, open-source JSON parser available for MATLAB/Octave. In the form of JSONlab (https://github.com/fangq/jsonlab).

Caveats

Gwyddion’s internal data structures are not terribly well documented, and this represents a best-effort attempt to rip Gwyddion objects from it’s data model and generalize them into a readily importable format. Some loss is inevitable, and in particular it is not yet clear if all required metadata is preserved by this tool. It is also currently unknown if VK4 format files can store floating point numbers larger than 32 bits, and if so how this tool may affect the precision of such numbers, although it is known to work correctly with 32 bit floating points numbers.

As this tool is very young, it is possible there may be additional edge cases which have not yet been properly accounted for.

NOTE the prefix_N_dim matrix values are not explicitly normalized to meters, they will use whatever SI unit Gwyddion provides them in. In the future, this will will normalize to meters, but in the interim you should check the SI unit field in the JSON metadata file.

Syntax

For usage information, run this command with the ``--help`` flag.

Tip: this tool is also useful for producing outlines of VK4 files. This can
be accomplished by running this program with the ``--verbose`` flag on an
input file without specifying an output file.

Changelog

• 0.0.1

  • 2018-04-25 - Charles Daniels

    • Create initial version.

• 0.0.2

  • 2018-04-26 - Charles Daniels

    • Patch a bug where xres and yres were transposed while reshaping the Numpy matrix representing the image data, which resulted in corrupted .mat contents.

Author

Charles Daniels

License

Copyright 2018 Charles Daniels

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.