User-defined meta extras

If the standard meta extras provided do not suit your needs, you can make your own. This section explains the context in which custom flash computations happen and the tools available, then takes you through the system defined extras. Finally, documentation is provided for the modules and classes that support meta extras.

The context of meta computations

Let’s start by setting the context for fluorometer event computations. When the LI-6800 triggers a flash, the fluorometer performs it, then sends the results back to the console as a json payload. At this point, flash data appears as in Listing 8‑5.

Listing 8‑5. A flash event as a .json payload, as first received from the fluorometer.

The system then adds some values (Tled through Stops) to the flash data, regardless of event type (Listing 8‑6):

Listing 8‑6. Items added to every flash or dark pulse event.

Then, the meta data is processed, with the standard items coming first, and the extras (if any) following it (Listing 8‑6). Again, for standard flashes or dark pulse, the meta data is automatic, and there are no extras; for a CUSTOM flash, the user defines the meta data and extras.

Listing 8‑7. Items added from meta data. Meta extras in green.

The final step is to add group data: MPF and/or FastKntcs (if appropriate) and FLR.

Listing 8‑8. Fluorometric computations.

Finally, the flash payload is written to file, and an Excel file is written, if specified. The data that a meta extra computation has to work with is anything in the flash data that has come before it.

Meta extra python files

Each available meta extra corresponds to a file in /home/licor/flr/extras/. Adding an extra of your own making involves creating a file for it and adding it to this directory.

When an extra computation is specified in a custom flash meta data, say

+mean 22

the system looks for a file named /home/licor/flr/extras/mean.py. If there is no matching file, then

"mean 22":"Not Supported"

is added to the flash file. Let's look at some of these files.

The basics (example: +fit)

The listing of the file that supports +fit is shown in Listing 8‑9.

fit fits a polynomial to a data set, and reports the coefficients. Recall that the syntax is

+fit[(y target, x target, power)] [codes]

and not specifying any of them (+fit 15 for example) would fit FLUOR as a function of SECS with a straight line.

Let's see how it does this:

Listing 8‑9. Listing of fit.py.

import numpy as np
from flashextra import FlashExtra

class FEFit(FlashExtra):
    def __init__(self, part1, part2, args):
        FlashExtra.__init__(self, part1, part2, args)

    def compute(self, meta):
        indices = meta.getTheseIndices(self.codes)
    y_target = self.getArgValue(0, 'FLUOR').upper()
    x_target = self.getArgValue(1, 'SECS').upper()
    power = int(self.getArgValue(2, 1)) # arguments are always strings, so need to convert
    x_data = meta.getForIndices(x_target, indices)
    y_data = meta.getForIndices(y_target, indices)
    if len(x_data) != len(y_data):
        return 'Mismatched data sizes"'
    if len(x_data) > power:
        fct = np.poly1d(np.polyfit(x_data, y_data, power))
        return list(fct.c)
    return 'Insufficient data'

A meta extra file defines a class, which needs to be a subclass of FlashExtra (a listing and description of which are provided in The FlashExtra class). The important part is the compute() method, which does the work. The compute method is passed a parameter named meta, which is an instance of a class (see The FlashMeta class) that contains all the flash data from the fluorometer and computations up to now (e.g., the listings shown above in The context of meta computations), as well as some handy tools for extracting that data for use.

The first thing compute does is extract the three arguments that may or may not have been passed in. It does this using the getArgValue() method that belongs to the FlashExtra class. The first parameter is the index of the argument, and the second is the default value, in case none was passed in.

The next step is to extract the data to be fit. This uses the get method that belongs to meta. The first argument (e.g., x_target is the name of the time series data, and the second (self.codes) specifies the observations.

self.codes is an internal representation of the code specifiers that the FlashExtra file generated for you on its creation, based on what followed the (in this case +fit) entry in the meta line. Usually, that entry is simple like, 15, but it could be more complicated, like 15[:-10],20,22[4:]. Whatever it was, self.codes contains exactly what you need to get those observations. (If you would like to get a list of the indices self.codes is calling for, there is a method for that. See getTheseIndices( codes)).

The final step is to do the fit, and return the result.

Unless there are errors, compute() returns list of values. What appears in flash data file might be

When compute returns something, its calling context (process() in Listing 8‑16) is what actually adds the result, with a label, to the data file, and that is what is happening here. In the next example, +stats, the compute() method itself writes several things to the flash file.

Multiple outputs (example: +stats)

The listing supporting the +stats extra is shown below.

Listing 8‑10. Listing of stats.py.

from flashextra import FlashExtra
import numpy as np

class FEStats(FlashExtra):
    def __init__(self, part1, part2, args):
        FlashExtra.__init__(self, part1, part2, args)

def compute(self, meta):
    target = self.getArgValue(0, 'FLUOR').upper()
    data = meta.get(target, self.codes)
    if len(data):
        # self.label() might we something like "stats(dc) 15[2:]"
        # Instead of one output, we have five, each with the 'stats' replaced with what it actually is
        # e.g. "count(dc) 15[2:]": 21
        meta.addThis(self.label().replace('stats', 'count'), len(data))
        meta.addThis(self.label().replace('stats', 'min'), min(data))
        meta.addThis(self.label().replace('stats', 'max'), max(data))
        meta.addThis(self.label().replace('stats', 'mean'), np.mean(data))
        meta.addThis(self.label().replace('stats', 'std'), np.std(data))
        return None # we've already written the results
    else:
        return 'No data found' # report this error

In +stats, we again overwrite the compute() method, but this time, we are adding five lines of output, none of which carry the "stats" label. The thing to note here is meta.addThis(), described in addThis(label, value).

Really simple (examples: +mean, +std)

The simplest examples of extras are +mean and +std, because they don't even override the compute method. Rather, they simply pass a function for the default compute to use.

Listing 8‑11. Listing of mean.py.

from numpy import mean
from flashextra import FlashExtra

class FEMean(FlashExtra):
    def __init__(self, part1, part2, args):
        FlashExtra.__init__(self, part1, part2, args, fct=mean)

Listing 8‑12. Listing of std.py.

from numpy import std
from flashextra import FlashExtra
class FEStd(FlashExtra):

    def __init__(self, part1, part2, args):
        FlashExtra.__init__(self, part1, part2, args, fct=std)

Neighbor averaging (examples: +min, +max)

The +min and +max could to the same technique of simply passing in a min or max function to the constructor, but we want to provide the potential for averaging with some neighboring values. So we still pass in a function, but we have to define that function to include the neighbor averaging. The listings below illustrate.

Listing 8‑13. Listing of min.py.

from flashextra import FlashExtra

class FEMin(FlashExtra):
    def __init__(self, part1, part2, args):
        FlashExtra.__init__(self, part1, part2, args, fct=self.na_min)

    def na_min(self, data):
        range_i = int(self.getArgValue(1, 0)) # number of points to average
        v = min(data)
        return self.neighborAverage(data, v, range_i)

Listing 8‑14. Listing of max.py.

from flashextra import FlashExtra

class FEMax(FlashExtra):
    def __init__(self, part1, part2, args):
        FlashExtra.__init__(self, part1, part2, args, fct=self.na_max)

    def na_max(self, data):
        range_i = int(self.getArgValue(1, 0)) # number of points to average
        v = max(data)
        return self.neighborAverage(data, v, range_i)

Using flr_tools (examples: +iv)

The +iv extra computes an initial value of a quantity by fitting a regression, and then computing the value at the very start of the first step involved. There is a subtlety here: the relation between output time and measurement time depends on modulation rate and output rate (see Fluorometry timing details), and a module named flr_tools has some tools that will help, as we see in Listing 8‑15.

Listing 8‑15. Listing of iv.py.

import numpy as np
from flashextra import FlashExtra
import flr_tools

class FEIV(FlashExtra):
    def __init__(self, part1, part2, args):
        FlashExtra.__init__(self, part1, part2, args)

    def compute(self, meta):
        target = self.getArgValue(0, 'FLUOR').upper() # first argument = y variable
        power = int(self.getArgValue(1, 1)) # 2nd argument - power of the fit
        secs, data = meta.getTimeSeries(target, self.codes)

        if len(data) > power:
            target_sec = meta.timeOfFirstCode(flr_tools.codeFromItem(self.codes[0]))
            modrate, outrate = meta.getRates(flr_tools.codeFromItem(self.codes[0]))
            t0 = target_sec + flr_tools.timeShiftForOutrate(modrate, outrate)
            polyfct = np.poly1d(np.polyfit(secs, data, power))
        return polyfct(t0)
    else:
        return 'Insufficient Data'

We get the data we need from a convenient FlashMeta method named getTimeSeries.

Next we get the time associated with the first data value having the code number we specified.

Then, we get the modulation and output rates that were active during the requested step.

Then we compute the time to plug into the polynomial by adjusting the first step's time by the timeshift for that modrate and output rate combination.

The FlashExtra class

The base class FlashExtra for flash extras is shown in Listing 8‑16. This class is designed to provide the computation(s) associated with a meta extra.

Listing 8‑16. Class definition FlashExtra.

class FlashExtra():
    #
    # The base class for flash extras
    #
    def __init__(self, part1, part2, args, fct=None, fail=None):
        # part1 examples: 'hiq' or 'mean(pfd)'
        # part2 examples '3' or '4[2:]'
        # args - list of arugments (if any) found in part1
        # value - optional default value
        self.part1 = part1
        self.part2 = part2
        self.args = args
        self.fct = fct
        
        self.codes = listFromString(part2) # List of CodeSpecifies
        self.fail = fail
        
    def __str__ (self):
        return self.source()
        
    def __repr__ (self):
        return self.source()
        
    def source(self): # as it appeared in extras line
        return '+'+self.part1 + ' ' + self.part2
        
    def label(self): # used in file output
        return self.part1 + ' ' + self.part2
        
    def getArgValue(self, index, default):
        # Returns the ith optional aurgment, if it exists, or default if it doesn't
        rtn = default
        if len(self.args) > index:
            test = self.args[index]
            if len(test) != 0:
                rtn = test
            return rtn

    #
    # This is what is called to do the work
    #
    def process(self, meta):
        try:
            v = self.compute(meta) #
            
            if type(v) != type(None): # write the results
                meta.addThis(self.label(), v)

        except Exception as e:
            print('Exception in FlashExtra.process', str(e))
            meta.addThis(self.label(), str(e)) # report the error

    #
    # Compute a result using self.fct (fct passed in constructor)
    # Alternatively, overwrite compute() for not so simple computations
    #
    def compute(self, meta):
        if type(self.fail) != type(None):
            return self.fail
        
        if type(self.fct) == type(None):
            return 'No fct'

    # Convert CodeSpecifies to a list of indices for the time series data
    target = self.getArgValue(0, 'FLUOR').upper()
    data = meta.get(target, self.codes)
    if len(data):
        return self.fct(data)
    else:
        return 'No data found'

Constructor

The constructor is passed two strings and a list, which come from the code that is parsing the meta commands. These contain the command with any arguments, as typed in the meta command (part1), and the code specifier(s) if any (part2). args is simply a list containing any arguments found in part1.

Optional parameters and code specifiers

The part2 string is converted into list of CodeSpecifer objects (see The CodeSpecifier class) using the listFromString method imported from the flr_tools module.

Most meta extras have optional parameters (+fmean(dc), fmin(,2), etc.). The list of them (args) is passed in when a FlashExtra is created. You can access it directly (self.args) but it is more convenient to use the class method

Pass the index (0, 1, 2, etc.) and the value to use if that index not there. If the first parameter is a time series target and defaults to FLUOR, you would make this call

It is completely up to you what you use optional arguments for.

Computational sequence

When it comes time to compute a meta extra, the process method of FlashExtra is called.

#
# This is what is called to do the work
#
def process(self, meta):
    try:
        v = self.compute(meta) #
        
    if type(v) != type(None): # write the results
        meta.addThis(self.label(), v)
        
    except Exception as e:
        print('Exception in FlashExtra.process', str(e))
        meta.addThis(self.label(), str(e)) # report the error

The process method, in turn, calls the compute method, which is expected to do the work and return a value, or list, or whatever you want it to be. The returned item must to be json-serializable (e.g., int, float, str, are fine. If None is returned, process adds nothing to the flash file (except an error message if warranted).

The parameter meta that is passed is an instance of class FlashMeta (described in The FlashMeta class).

The FlashMeta class

The FlashMeta class serves as a wrapper for the fluorometer event data received from the fluorometer. Below are some useful methods that it provides that you can use when writing your own computations (from within the compute method of a FlashExtra subclass).

addThis(label, value)

Add an entry to the fluorometer event data set (file).

label - The string to use as a label

value - Can be any json serializable type.

Normally, you do not have to call this, as the MetaExtra process method does it for you. If, however, you want to add multiple lines to the file, then you would do so using this method.

get(label, codes=None)

Use this to retrieve values from the flash event.

label - any of the keys in the json representation of the file (see Listing 8‑2).

codes - (Ignored if label is not one of the time series labels.) Can be an int, a CodeSpecifier, or a list of items of either type. If None, returns the entire list of time series values for label.

Returns: A single value or a list of values. If label is not found, then returns 0 if codes=None or else [].

Examples: (in the context of a FlashExtra method)

Listing 8‑17. Examples using get() in a FlashExtra compute method.

class MyCompute(FlashExtra):
    def __init__(self, part1, part2, args):
        FlashExtra.__init__(self, part1, part2, args)

        self.codes = self.getCodes(part2) # This creates a list of CodeSpecifiers that can be used with meta.get()

    def compute(self, meta): # meta points to a FlashMeta data wrapper.
        :
        # Get FLUOR values for the codes asked for by the user
        flr = meta.get('FLUOR', self.codes)
        :
        # get the DC values for code 16 and first ten values of code 17
        dc_all = meta.get('DC', self.getCodes('16,17[0:10]'))
        :
        # get the before-flash light intensity (single value)
        before = meta.get('Pre_Qin') # get the before-flash light intensity (single value)
        :
        # get the computed FMAX value (came from the +fmax meta code)
        fmax = meta.get('FMAX')

There are two additional time series that get() makes available: DC/Q and BLUE. Neither is stored in the file, but is computed when requested. The ith element of these lists is computed from the time lists vectors by the following equations:

8‑31

and

8‑32BLUE_i = PFD_i - REDMODAVG_i - RED_i - REDMODAVG_i

getForIndices(label, indices)

Returns the data set associated with the specified time series table and list of indices.

label - 'SECS', 'FLUOR', 'DC', 'PFD', 'RED', 'REDMODAVG', 'FARRED', 'CODE', 'BLUE' or 'DC/Q'.

indices - List of indices, usually provided by getTheseIndices().

Returns: List of values. If label is not found, then returns [].

Examples: (in the context of a FlashExtra method)

Listing 8‑18. Using getForIndices() and getTheseIndices() in a FlashExtra compute method.

class MyCompute(FlashExtra):
    def __init__(self, part1, part2, args):
        FlashExtra.__init__(self, part1, part2, args)

        self.codes = self.getCodes(part2) # This creates a list of CodeSpecifiers that can be used with meta.get()

    def compute(self, meta): # meta points to a FlashMeta data wrapper.
        :
        # The codespecifier into a list of indices
        ics = meta.getTheseIndices(self.code)
        # Now retrieve several data sets associated with those indices
        flr = meta.getForIndices('FLUOR', ics)
        dc = meta.getForIndices('DC', ics)
        pfd = meta.getForIndices('PFD', ics)
        :
        # get the first 50 PFD values in the flash, regardless of code
        pfd_50 = meta.getForIndices('PFD', [i for i in range(50)])

getPrevCode(code)

Returns the code value of the step that comes immediately before the first step that has that specified code number.

code - an integer code (number).

If the time sequence of codes is [2,2,2,2,3,3,3,3,3,3,4,4,4,4,4,5,5,5,5], then getPrevCode(4) will return 3.

getRates(step_code)

Returns the modulation and output rates for the given step code.

step_code - The step code (integer) in question.

Returns: (modrate, outrate) - Both are numeric values, and both in Hz.

getTheseIndices( codes)

Returns a list of the indices associated with a code specifier (or list).

codes - Can be an int, a CodeSpecifier, or a list of items of either type.

Returns: list of indices

Using getTheseIndices and getForIndices (described next) together is a slightly more efficient way to retrieve multiple data sets associated with a common code specifier.

getTimeSeries(label, codes=None)

This method is much like get(), but it returns two lists instead of one.

label - 'SECS', 'FLUOR', 'DC', 'PFD', 'RED', 'REDMODAVG', 'FARRED', 'CODE', 'BLUE' or 'DC/Q'.

code - Can be an int, a CodeSpecifier, or a list of items of either type.

Returns: two lists: (time, data).

There are slight timing differences for the various time series data in a flash (see Fluorometry timing details), and the returned time list is adjusted accordingly.

Examples: (in the context of a FlashExtra method)

Listing 8‑19. Using getTimeSeries() in a FlashExtra compute method.

class MyCompute(FlashExtra):
    def __init__(self, part1, part2, args):
        FlashExtra.__init__(self, part1, part2, args)

    self.codes = self.getCodes(part2) # This creates a list of CodeSpecifiers that can be used with meta.get()

    def compute(self, meta): # meta points to a FlashMeta data wrapper.
        :
        # Get time and FLUOR values for the codes asked for by the user
        f_secs, flr = meta.getTimeSeries('FLUOR', self.codes)
        :
        # Get time and FLUOR values for the codes asked for by the user
        dc_secs, dc = meta.getTimeSeries('DC/Q', self.codes)
        :
        # Note - f_secs and dc_secs will be different (slightly)

hasThis(label)

Returns True if the specified label is in the data set.

label - the label in question.

The search is case sensitive.

timeOfFirstCode(code)

Returns the time (SECS value) associated with the rst value that has the given step code.

step_code - The step code (integer) in question.

The flr_tools module

The CodeSpecifier class

The CodeSpecifier class converts a string representation of a code specifier, as found on a meta line into an internal representation that can be used to actually perform the work.

Listing 8‑20. CodeSpecifier class definition.

class CodeSpecifier():
    def __init__(self, string): # examples: 50 50[2:] or 49[:-1] or 30[1:-1:10]
        self.code = 0
        self.stringvalue = string

        s1 = 0
        s2 = None
        s3 = None
        # allow for ( ) instead of [ ] or even mixed
        string = string.replace('(', '[').replace(')',']')
        
        if '[' in string:
            i = string.index('[')
            self.code = safeInt(string[0:i], 0)
            if ']' in string:
                j = string.index(']')
                args=string[i+1:j].split(':')
            if len(args) >= 1:
                s1 = safeInt(args[0], None)
            if len(args) >= 2:
                s2 = safeInt(args[1], None)
            if len(args) >= 3:
                s3 = safeInt(args[2], None)
        else:
            if string == '*':
                self.code = '*'
            else:
                self.code = safeInt(string, 0)

        self.slice = slice(s1, s2, s3)
        self.params = [s1, s2, s3]

    def __str__ (self):
        return self.stringvalue
        
    def __repr__ (self):
        return self.stringvalue
    
    def test(self, list_item):
        return list_item[self.slice]

    def makeForFull(self): #n return a CS for getting all the data, (no slicing)
        return CodeSpecifier(str(self.code))

    def filterThis(self, codes):
        if isinstance(self.code, int):
            rtn = [i for i in range(len(codes)) if codes[i] == self.code]
        else:
            rtn = [i for i in range(len(codes))]
        return rtn[self.slice]

timeShiftForModrate(modrate)

Returns time (s) between a step's time and the step's modulation pulse

modrate - Integer value in Hz

For 250000, the pulse is 2 µs before the time stamp

timeShiftForOutrate(modrate, outrate)

Returns the time (s) between the time for an output record and time of the first measurement for that observation

modrate - Integer value of modulation rate in Hz

outrate - Integer value of output rate in Hz

If modrate = outrate, this is 0

codeFromItem(item)

Returns the integer code number associated with a code specifier

item - Can be an int, string, or CodeSpecifier

listFromString(line, codesOnly=False)

Generates a list of CodeSpecifier objects from a string.

line - A string, e.g., "13,4[1:],17"

codesOnly - If True, each CodeSpecifier will not include anly slice information,