Add Top-K Multi-dimensional Motif Discovery Tutorial

[seanlaw]

Demonstrate how to find the top-K motifs and top-K nearest neighbors in multi-dimensional data

[seanlaw]

In the paper, it says:

we tested our method on an electrical load measurement dataset [21]. The dataset consists of electrical load measurements for individual appliances (and an aggregated load) from households in United Kingdom. Five appliances are considered: fridge-freezer, freezer, tumble dryer, dishwasher,
and washing machine. The data was collected from April 19, 2014 to May 15, 2014, where the length is 17,000. The subsequence length was set to 4 hours.

Cross referencing the cleaned version of the REFIT dataset, it was found that only "CLEAN_House3.csv" in the "Access Dataset > REFIT Power Data - 111215" contained the aforementioned appliances. It was not possible to reproduce the paper's data preprocessing steps but this should be sufficient:

%matplotlib inline

import matplotlib.pyplot as plt
import pandas as pd

plt.style.use('https://raw.githubusercontent.com/TDAmeritrade/stumpy/main/docs/stumpy.mplstyle')

start_date, end_date = '2014-04-19', '2014-05-15'
df = pd.read_csv("CLEAN_House3.csv")
df["Time"] = pd.to_datetime(df["Time"])
colnames = {
    "Appliance1": "Toaster",
    "Appliance2": "Fridge-Freezer",
    "Appliance3": "Freezer",
    "Appliance4": "Tumble Dryer",
    "Appliance5": "Dishwasher",
    "Appliance6": "Washing Machine",
    "Appliance7": "Television",
    "Appliance8": "Microwave", 
    "Appliance9": "Kettle",
}
df = (df.rename(colnames, axis="columns")
        .loc[:, ["Time",
                 "Fridge-Freezer",
                 "Freezer", 
                 "Tumble Dryer", 
                 "Dishwasher",
                 "Washing Machine"]]
        .query('Time >= @start_date and Time <= @end_date')
        .groupby(pd.Grouper(key='Time', freq="T")).sum()
     )

for colname in df.columns:
    plt.plot(df[colname])

Alternatively, there is also the PAMAP2 motif example by Jessica Lin (who has worked with Eamonn Keogh in the past). If you look at Section D, I think that the PAMAP2 dataset (Overview found here - Download zip file here) may be good for our tutorial. In the paper, they focused on the "ironing" activity and used a 12.5 second (12500 data point) window size, which they claimed allowed them to identify relevant hand motions (from the Y and Z hand coordinates) that are consistent with ironing while, presumably, other coordinates (from the chest and ankles) may have been irrelevant for ironing.

The dataset is suppose to contain 1.3 million datapoints in length which may be too much for a tutorial but I wonder if we could downsample the data by 10x (i.e., only look at every 10th data point and thereby analyzing only 130K data points for each dimension) and still get be able to convey our point. I wanted to bring it to your attention in case it may be useful.

[SaVoAMP]

Hey!

Since we were discussing this topic in detail here and have already generalized the multidimensional motif discovery to finding the top k motifs and top k nearest neighbors, I would like to help with this tutorial!

[seanlaw]

@SaVoAMP Sounds good! I will assign it to you

[SaVoAMP]

Should I just use the REFIT dataset for the tutorial?

If so, do you think it would suffice to just use the dataset locally and refer to the "download instructions" in a comment?

if not, since the download is a 489 MB archive, I can't import it "on the fly".

Sadly I don't have any servers to host the ´CLEAN_House3.csv´ file on except my own github account.
I presume that the .csv file won't be part of the repo? Maybe Git LFS could be a solution here?

Do you have a link with the cleaned up version of the file that I could use for the tutorial, or should I create my own separate github project containing only the file?
Or would GitHub be unsuitable for that purpose?
I'm asking because my software engineering professor mentioned that GitHub/GitLab are not suited for filehosting and only sourcecode should be uploaded there. I don't know to what extent he meant that.
Or would you suggest using another dataset instead?

[seanlaw]

If so, do you think it would suffice to just use the dataset locally and refer to the "download instructions" in a comment?

Usually, as with our other tutorials, I end up uploading the single CSV file to Zenodo STUMPY collection for each datasete needed. Please do not add the CSV file to the codebase as it will add unnecessary bloat. For now, in your tutorial, please assume that we'll eventually replace the pd.reas_csv line above with a Zenodo dataset URL and no comment should be needed.

Should I just use the REFIT dataset for the tutorial?

Yes, I think we can use the REFIT dataset but only a single interesting CSV that can (hopefully?) get our point/concept across.

Does this answer your question?

I'm asking because my software engineering professor mentioned that GitHub/GitLab are not suited for filehosting and only sourcecode should be uploaded there. I don't know to what extent he meant that.
Or would you suggest using another dataset instead?

They are correct and I agree with them. Git repositories should really only be for code (even Jupyter notebooks are questionable but it's been okay so far for us).

[SaVoAMP]

All right, thank you!

How do you want to visualize the motif pair? Should we plot it inside the (currently very colorful) graph or should I make a zoom-in of the motif pair and only plot that?
As soon as we get matches we need to plot them inside the graph anyway. But if I would visualize all appliances in the same color so that we can see the motif pair and its nearest neighbors with another color, then we could not tell which appliance belonged to which time series.
I would suggest to plot 5 subplots (one for each appliance) and afterwards visualize the motifs and matches with another color, as I have done here, for example:

This was a 30-dimensional data set and after selecting the k dimensions where the motif is represented, I only plotted those dimensions with all its matches. The motif pair can be seen in red.

But the problem with our dataset might be that there are a plenty of spikes in each dimension which may look odd when plotted above each other 🤔

[seanlaw]

For things like this, It's difficult for me to judge without seeing the possible outcomes in context. If it's not too much work, is it possible to submit a PR with a notebook that displays/annotates the pros/cons of each approach to visualizing the results?

In case it matters, we don't necessarily need to use all 9 dimensions from the CLEAN_House3.csv dataset. We could always choose a subset where only 2 dimensions (i.e., washer + dryer) would be identified by MDL.

[SaVoAMP]

Our time series has the length 37,440. But the authors claim that their length is 17,000. If I understand everything correctly, you were converting that to minutes.
I'm trying to plot the motif pair but stuggling since I only work with data frames since the beginning of my thesis.
Since I was totally confused by the differing timesteps, I was only plotting "data points" on the x-axis instead of time (until now). That's why I am struggling to plot the time series in timesteps together with the motif now, since my indices aren't in time steps. Probably it will take some time until I've got it 😆 So far, I'm only managing to decently plot the time series that contain the motif. But as soon as I draw in the motif, everything breaks down 😄

I think we might have the problem, that the flat areas are more similar than the spikes that we want to catch. I'll do a pull request so that you can see what I mean. In essence the motif pair is located at the left of three time series. Don't be surprised that I commented the part of drawing lines at the beginning of the motif pair since I was to stupid to get it working today.

[seanlaw]

Our time series has the length 37,440. But the authors claim that their length is 17,000. If I understand everything correctly, you were converting that to minutes.

I think that's right. In .groupby(pd.Grouper(key='Time', freq="T")).sum(), I'm basically grouping data points to within each minute and then summing up all of the energy consumption during that minute in order to reduce the overall data size.

I think we might have the problem, that the flat areas are more similar than the spikes that we want to catch.

I see. That's a nasty problem to have. I wonder if we can cheat/overcome this by adding ONE additional dimension that is simply the aggregate energy usage (I'm hoping that this will help reduce the flat areas)?

[SaVoAMP]

I don't think so since the similarity still holds in the other dimensions. I think that they will still be chosen and the additional one ignored.