********************************** Documentation for ``herc-vk4tool`` ********************************** .. contents:: A tool for interacting with Keyance Profilometry VK4 formatted data files, built on top of Gwyddion's automation API. This tool is capable of importing VK4 files, extracting the stored data, and producing both MATLAB matrices containing the data and JSON formatted sidecar files containing metadata about each. Description =========== Overview -------- This tool was written to support analysis of VK4 formatted data files programatically via existing tools such as MATLAB/Octave as well as any other tool which is capable of importing MATLAB ``.mat`` format matrices (as well as JSON, if metadata is required). Gwyddion Paths -------------- Throughout this document and the corresponding source code, the term "Gwyddion Path" is used to refer to the notation which Gwyddion uses to refer to data stored within the ``container`` object produced by loading a data file. This is because a single container can store many pieces of data, such as various matrices and metadata about them. Gwyddion paths roughly correspond to POSIX paths in terms of notation. Typically, a single fundamental unit of interest will be identified by the path ``/N``, where ``N`` is a positive integer. Usually the data associated with this unit of interest is located at ``/N/data``, and other metadata may be stored at other paths relative to ``/N``. A Gwyddion path might identify a Gwyddion ``DataField`` object, another Gwyddion container object, a string, or other types of objects or data types. Output Format ------------- The raw data of the VK4 file which is input is converted into one or more 2-dimensional matrices and stuffed into a single ``.mat`` data file for later import into MATLAB (this is accomplished via ``scipy.io``). The matrix names will be set to a prefix (by default ``matrix``, but this can be overridden using a flag) followed by a ``_`` character and then the first part of the key Gwyddion uses to refer to the matrix (this seems to typically be a positive integer) As much metadata as can be extracted from the VK4 file will also be extracted and stored in a JSON file with the same basename as the generated ``.mat`` file. This JSON file contains nested hash tables where the outer keys are again the first part of the Gwyddion "path" to the relevant objects, and the inner keys uniquely identify a metadata field. There are three types of metadata fields which may be populated: 1. Those obtained from ``/N/meta`` in the Gwyddion container produced from the VK4 file. These fields each produce an inner key and a string value in the JSON. 2. Those obtained from member functions such as ``get_xres()`` of the Gwyddion ``DataField`` object corresponding to ``/N``. These values are stored in the format that they are returned by Gwyddion, and the field names in the JSON file are simply the name of the function call with the ``get_`` removed (i.e. the output ``get_xres()`` is stored in the key ``xres``) 3. Those obtained from Gwyddion fields which are of type ``str``, these are stored as is and named after all parts of their Gwyddion path excepting the first part. For example, ``/N/data/title`` from a Gwyddion container would result in ``[N][data/title]`` containing the value of this field. Additionally, for each matrix in the generated ``.mat``, a version of the raw data which had a degree 2 polynomial regression subtracted from it (to normalize uneven heights) will be stored with the matrix name ``prefix_N_fit``. A third matrix will be created with the name ``prefix_N_dim`` which will contain the x and y dimensions in meters as floating point values in that order. Footnote on the ``stats`` Field ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ As a convenience feature, the output of ``.get_stats()`` is recorded for each ``DataField`` object which is processed. The value of this field is a list of floating point numbers, and it has the schema ``(avg, ra, rms, skew, kurtosis)`` per the Gwyddion documentation. Footnote on Loading JSON in MATLAB ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ JSON was selected as the format for the metadata sidecar files because of the fact that there is an existing, open-source JSON parser available for MATLAB/Octave. In the form of JSONlab (https://github.com/fangq/jsonlab). Caveats ------- Gwyddion's internal data structures are not terribly well documented, and this represents a best-effort attempt to rip Gwyddion objects from it's data model and generalize them into a readily importable format. Some loss is inevitable, and in particular it is not yet clear if all required metadata is preserved by this tool. It is also currently unknown if VK4 format files can store floating point numbers larger than 32 bits, and if so how this tool may affect the precision of such numbers, although it is known to work correctly with 32 bit floating points numbers. As this tool is very young, it is possible there may be additional edge cases which have not yet been properly accounted for. **NOTE** the ``prefix_N_dim`` matrix values are not explicitly normalized to meters, they will use whatever SI unit Gwyddion provides them in. In the future, this will will normalize to meters, but in the interim you should check the SI unit field in the JSON metadata file. Syntax ====== :: For usage information, run this command with the ``--help`` flag. Tip: this tool is also useful for producing outlines of VK4 files. This can be accomplished by running this program with the ``--verbose`` flag on an input file without specifying an output file. Changelog ========= * 0.0.1 * 2018-04-25 - Charles Daniels * Create initial version. * 0.0.2 * 2018-04-26 - Charles Daniels * Patch a bug where ``xres`` and ``yres`` were transposed while reshaping the Numpy matrix representing the image data, which resulted in corrupted ``.mat`` contents. Author ====== Charles Daniels License ======= Copyright 2018 Charles Daniels Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.