Notebook

The Notebook is used to develop and apply custom Python or R scripts. You can access and run any tools that are built into Tovi and develop your own custom code. The purpose of this chapter is to introduce the feature in Tovi. We will not dive into the details of the programming in Python and R. Extensive online documentation is available. See matplotlib.org, for example.

Tips for using the notebook

When you open the notebook for the first time, you will choose which library to use. Currently Tovi supports Python 3.6 and R 3.4.3. Your choice will be remembered in the analysis. If you want to change from one to the other, you are required to create a new analysis. If you choose the R library on macOS, you must install XQuartz (www.xquartz.org) in order to view graphs.

You'll find a row of buttons that allow you to control the notebook:

  • Save
  • Insert, Cut, Copy, or Paste cells
  • Run the selected cells
  • Interrupt the kernel
  • Restart the kernel
  • Re-run all cells
  • Select the view (Code Markdown, or Raw).

Comments may be included in the code. Comments are for humans to read, to help them understand the code. Comments are added in two ways. First, the # character makes a line of text into a comment. Three single quote characters (''') starts a comment that spans multiple lines of text. The same three characters (''') ends the comment. The comments can be added or deleted with no consequence to the code.

Common Python commands

The following table lists some of the commands that you can enter in the notebook. Press Shift + Enter (on Windows) or Shift + Return (on macOS) to send a command. Or click the Play button.

Command Function
ds Lists the Data Structure for the dataset under consideration. Type ds to view the data as text.
import numpy as np NumPy is the core Python package for scientific computing. This command imports the package. Type np to view the directory of the library.
import pandas as pd Pandas is the Python data analysis library. This command imports the library. Type pd to view the directory of the library.
import matplotlib.pyplot as plt The plotting framework is called matplotlib.pyplot. Type plt to view the directory of the library.
import matplotlib.mlab as mlab Python functions written for compatibility with MATLAB. Type mlab to view the directory of the library.

Create windrose plots with Python

To create windrose plots, simply type (or copy and paste) the command into the notebook and click Play or press Shift + Enter.

Copy 
%matplotlib inline
from pyfootprint import utils as u
from pyfootprint import plots as plt
mydata=u.df_from_ds(ds)
plt.plot_windrose(mydata)

Tovi will process the data for a moment and then present the windrose figures.

You can save the process (click Done) and Tovi will add a node to the analysis history.

Compute and plot daily course for LE by month with Python

This code will direct Tovi to compute daily courses of latent energy for each month and then plot the data. Depending on the size of your dataset, this code may take 5 to 10 minutes to finish running.

Copy 
import numpy as np
import matplotlib.pylab as plt
import matplotlib.gridspec as gridspec
import pathlib as pl
def df_from_ds(ds, varlist=None):
    import pandas as pd
    try:
        fo_data = {key: val['Data'] for key, val in ds['full_output'].items()}
    except Exception:
        fo_data = None
    try:
        biomet_data = {key: val['Data'] for key, val in ds['biomet'].items()}
    except Exception:
        biomet_data = None

    if fo_data and biomet_data:
        df = pd.merge(pd.DataFrame(fo_data, index=ds['full_output']['DateTime']['Data']),
                      pd.DataFrame(
                          biomet_data, index=ds['full_output']['DateTime']['Data']),
                      how='outer', left_index=True, right_index=True, suffixes=['', '_biomet'])

    elif fo_data and not biomet_data:
        df = pd.DataFrame(fo_data, index=ds['full_output']['DateTime']['Data'])
    elif biomet_data and not fo_data:
        df = pd.DataFrame(
            biomet_data, index=ds['full_output']['DateTime']['Data'])

    if varlist is not None:
        try:
            return df[varlist]
        except KeyError:
            log.warning(
                "Could not import the requested list of variables. " +"Most likely, at least one of them is not available in the data structure. " +"Importing all variables found in data structure")
            return df
    else:
        return df
        
def daily_courses(data, var, units, 
                  months=['Jan','Dec'], 
                  fill_months=True,
                  stat='mean',
                  nstdev=1,
                  quant_min=0.05, quant_max=0.95, 
                  color='#CC44BB', ecolor= '#888888',
                  eshaded=False,
                  alpha=1.0, qalpha=0.5, ealpha=0.4,
                  ylim=None):

    #Define months to be treated if necessaryif fill_months:
        year_months=['Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec']
        for i in range(0, 12):
            if year_months[i] == months[0]:
                start = i
            if year_months[i] == months[-1]:
                end = i
        months = year_months[start:end+1]
    
    #Figure out y-axis min/max as min/max quantiles if requiredif ylim == None:
        ymin = 1E10
        ymax = -1E10
        for month in months:
            month = month+str(data.index[0].year)
            mini = data[month][var].groupby(data[month][var].index.time).quantile(quant_min)
            maxi = data[month][var].groupby(data[month][var].index.time).quantile(quant_max)
            if np.min(mini) < ymin: ymin = np.min(mini)      
            if np.max(maxi) > ymax: ymax = np.max(maxi)      
        ylim =(ymin, ymax)
                
    #Now makes the plot
        fig, axs = plt.subplots(1, len(months), figsize = (1.5*len(months), 6), sharey=True)
        fig.subplots_adjust(wspace=0.02)
        fig.text(0.125, 0.93, str(data.index[0].year), fontweight='bold')
        j = 0
        for month in months:
            #Initialize axes
            axs[j].text(0.06,0.97, month, transform = axs[j].transAxes)
            month = month+str(data.index[0].year)
            if stat == 'mean':
                mean = data[month][var].groupby(data[month][var].index.time).mean()
            elif stat == 'median':
                mean = data[month][var].groupby(data[month][var].index.time).median()
            mini = data[month][var].groupby(data[month][var].index.time).quantile(quant_min)
            maxi = data[month][var].groupby(data[month][var].index.time).quantile(quant_max)
            err = nstdev * np.sqrt(data[month][var].groupby(data[month][var].index.time).var())
            axs[j].plot(np.linspace(1, 48, 48), mini, ':', lw=1, c=color, alpha=qalpha)
            axs[j].plot(np.linspace(1, 48, 48), maxi, ':', lw=1, c=color, alpha=qalpha)
            axs[j].plot(np.linspace(1, 48, 48), mean, lw=3, c=color, alpha=alpha)
            if not eshaded:       
                axs[j].errorbar(np.linspace(1, 48, 48), mean, yerr = err, 
                        lw=1, c=color, ecolor=ecolor, alpha = ealpha)
            else:
                axs[j].fill_between(np.linspace(1, 48, 48), 
                           mean-err, mean+err,
                           edgecolor='None', color=ecolor, alpha=ealpha)
            axs[j].set_ylim(ylim)
            axs[j].set_xticklabels([])
            axs[j].axhline(0, color='k')
            axs[j].set_xlabel('HH')
            axs[j].xaxis.set_ticks([0, 12, 24, 36])
            axs[j].xaxis.set_ticklabels(['00', '06', '12', '18'])
            if j == 0:
                axs[j].set_ylabel(units)
            j = j + 1

df = df_from_ds(ds)
%matplotlib inline
daily_courses(df, 'LE', 'W m-2')

Have fun with Python when you're done

Here is a sample of code that will instruct Tovi to draw a chart (this is not for processing data). Paste the snippet into the notebook and celebrate your accomplishments!

Copy 
import matplotlib.path as mpath
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt

fig, ax = plt.subplots()

Path = mpath.Path
path_data = [
    (Path.MOVETO, (1.58, -2.57)),
    (Path.CURVE4, (0.35, -1.1)),
    (Path.CURVE4, (-1.75, 2.0)),
    (Path.CURVE4, (0.375, 2.0)),
    (Path.LINETO, (0.85, 1.15)),
    (Path.CURVE4, (2.2, 3.2)),
    (Path.CURVE4, (3, 0.05)),
    (Path.CURVE4, (2.0, -0.5)),
    (Path.CLOSEPOLY, (1.58, -2.57)),
    ]
codes, verts = zip(*path_data)
path = mpath.Path(verts, codes)
patch = mpatches.PathPatch(path, facecolor='r', alpha=0.5)
ax.add_patch(patch)

# plot control points and connecting lines
x, y = zip(*path.vertices)
line, = ax.plot(x, y, 'go-')

ax.grid()
ax.axis('equal')
plt.show()