.. _ipython_directive:

Using the VitalsDataFrame Class in Jupyter
==========================================

The ``VitalsDataFrame`` class provides a direct interface between
the SQLite files generated by the **gfdlvitals** package and Python. 
The class is an extension of a Pandas DataFrame with additonal methods
for smoothing and detrending.

The following imports are needed to start:

.. ipython:: python

    import gfdlvitals
    import matplotlib.pyplot as plt
    from matplotlib.figure import figaspect

Loading SQLite datasets
-----------------------

The package ships with two demonstration datasets of 2 meter
near surface air temperature from a preindustrial control simulation and a
historical similation.  Any path-like string pointing to a SQLite file
can be passed to open_db:

.. ipython:: python

    # Load demonstration datasets
    df_ctrl = gfdlvitals.open_db(gfdlvitals.sample.picontrol)
    df_hist = gfdlvitals.open_db(gfdlvitals.sample.historical)

The details of a dataset are display by calling the object directly:

.. ipython:: python

    df_hist

In this DataFrame we see that the index is the time coordinate using 
the ``cf-time`` package.  The global ``area`` and ``t_ref`` fields are also shown. 

Plotting a field
----------------

Like a Pandas object, the ``.plot()`` is available for plotting a 
variable directly:

.. ipython:: python

   plotargs = {"color":"gray",
               "linewidth":"0.8",
               "ylabel":"DegK"};

   @savefig samplefig.png width=3in
   df_ctrl.t_ref.plot(title="Control",**plotargs);
    
   @suppress
   plt.figure()

   @savefig samplefig2.png width=3in
   df_hist.t_ref.plot(title="Historical",**plotargs);
    
Smoothing the timeseries
-------------------------

.. ipython:: python

   df_hist.t_ref.plot(title="Historical",**plotargs);
   @savefig samplefig3.png width=5in
   df_hist.smooth(20).t_ref.plot(color="red");

Identifying trends
-------------------

.. ipython:: python

   plotargs = {"color":"gray",
               "linewidth":"0.8",
               "ylabel":"DegK"};

   df_ctrl.t_ref.plot(title="Control",**plotargs);
   @savefig samplefig4.png width=3in
   df_ctrl.trend(order=1).t_ref.plot(color="purple");
    
   @suppress
   plt.figure()

   df_hist.t_ref.plot(title="Historical",**plotargs);
   @savefig samplefig5.png width=3in
   df_hist.trend(order=3).t_ref.plot(color="purple");


Detrending
----------

.. ipython:: python

   plotargs = {"color":"gray",
               "linewidth":"0.8",
               "ylabel":"DegK"};

   @savefig samplefig6.png width=3in
   df_ctrl.detrend(order=1).t_ref.plot(title="Control",**plotargs);
    
   @suppress
   plt.figure()

   @savefig samplefig7.png width=3in
   df_hist.detrend(order=3).t_ref.plot(title="Historical",**plotargs);

Removing drift
--------------

.. ipython:: python

   df_hist_detrended = df_hist.detrend(order=1,
                                       reference=df_ctrl,
                                       anomaly=False);

   df_hist.t_ref.plot(title="Historical",**plotargs);
   @savefig samplefig8.png width=5in
   df_hist_detrended.t_ref.plot(color="green",linewidth=0.5);

Statistical comparison of two datasets
--------------------------------------

An instance of the VitalsDataFrame can be passed to a second instance 
and a t-test can be performed to identify differences between 
variables that are common to the two instances. The t-test adjusts the 
degrees of freedom based on the autocorrelation of the timeseries
providing a more stringent threshold for assessing differences. For more 
details and examples of this method, see 
`Santer et al. 2000 <https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/1999JD901105>`_
and 
`Krasting et al. 2013 <https://journals.ametsoc.org/view/journals/clim/26/20/jcli-d-12-00832.1.xml>`_.

In the example below, the test historical dataset is artifically split 
into two 20-year epochs for comparison.
   
.. ipython:: python
   :okexcept:

   df_hist_t0 = df_hist[-40:-20]
   df_hist_t1 = df_hist[-20::]

   pvals = df_hist_t0.ttest(df_hist_t1)

   pvals