Feature Extraction by Distributed and Parallel means for industrial big data applications

Motivation — Why Feature extraction is necessary?

Feature extraction remains one of the most preliminary steps in machine learning algorithms to identify strong and weak relevant attributes. While many feature extraction algorithms are used during Feature Engineering for standard classification and regression problems, the problem turns increasing difficult for time series classification and regression problems where each label or regression target is associated with several time series. In addition to mapping the data to multiple time series, another problem is to associate the meta-information for each of the time-series.

Trust me such scenarios are quite common with huge datasets obtained from industrial heavy manufacturing equipments, machinery, IoT which often go under maintenance or exibit production line optimization demonstrating different success and failure metrics in different time series.

The main objective in this blog remains to understand the procedures in extracting relevant features from multiple time series and model with real dataset. The blog is structured as follows:

• Evaluate the importance of time series feature extraction for classification and regression problems.
• Model a practical use case of robot execution failure rates for an industrial application.
• Highlight the merits of the model in a distributed real-time environment.

What is Feature Extraction from Time series data and Why it is important?

Feature extraction controls selecting the important and useful features, by eliminating redundant features and noise from the system, to yield the best predicted output. In the context of time series data it aims to :

• Extract characteristics feature from time series, such as min, max, average, percentile or other mathematical derivations.
• Consolidate feature extraction and selection process from distributed heterogeneous sources of information lying on different time-series scale for predicting the target output.
• Allow time series clustering (un-supervised learning) from extracted features based on its relevancy.
• The extracted relevant and non-relevant features can help us to identify new insights at time series properties and dimensions in both classification and regression modeling.

Industrial big data, IoT (IIoT), robotics and other information sources are known to generate large volume of variety information in huge velocity with variability (inconsistent) and veracity (imprecise). Such data needs to be carefully merged and integrated for predicting the system state by analyzing and extracting the most meaningful features from different time intervals.

Feature Extraction and Selection Process

The following figures illustrate the steps involved in feature extraction and selection process. While Feature extraction is used to combine existing features to produce a more useful one, Feature selection helps in selecting the most useful features to train on among existing features.

The below figure gives a detailed understanding of creating feature sets using mathematical operations from n-different time series, followed by the feature aggregation and feature significance/relevance tests to rank them and arrive at the final selected feature list. It also shows high level representation of the full process covering the feature engineering, ranking and testing process that can be used in a scalable distributed environment.

Tsfresh and its usage

I have used Tsfresh to model time series feature extraction and relevancy test. Tsfresh is built as an efficient, scalable feature extraction algorithm for time series classification or regression problems. The algorithm is built with a feature importance filter in the beginning of ML pipeline that extracts relevant features according to its importance scores.

Tsfresh’s algorithm can be summarized to:

• Establish feature mappings by considering additional features and meta-information.
• Evaluate each individual feature vector independently through p-values quantifying its significance for predicting the target output.
• Apply in the context of industrial processes — which may include, prediction of the life span of machines, prediction of the quality of steel billets during a continuous casting process, prediction of success and failure rates of robots and IIoT sensors.

Modeling a practical use-case

Here we model, robot execution failures rates obtained through sensor readings. The execution status collected from 5 different data sources and merged have been known to fail due to :

• Failures in approach to grasp position.
• Failures in transfer of a part.
• Position of part after a transfer failure.
• Failures in approach to ungrasp position.
• Failures in motion with part

Further, this dataset is multi-variate, composed of 6 different time series data.

“Each feature is numeric, representing a force or a torque measured after failure detection; each failure instance is characterized in terms of 15 force/torque samples collected at regular time intervals. The collection starts immediately after any failure is detected. The total observation window for each failure instance was of 315 ms”.

Installation : pip install tsfreshimport tsfresh.examples.robot_execution_failures as robot
robot.download_robot_execution_failures()
df_ts, y = robot.load_robot_execution_failures()

print(df_ts.head())
print(y.tail())

In order to isolate and separately view the successful actions from unsuccessful ones, two different plots are used with two different ids.

normal = df_ts[df_ts.id == 3][['time', 'F_x', 'F_y', 'F_z', 'T_x', 'T_y', 'T_z']]
normal[['time', 'F_x', 'F_y', 'F_z', 'T_x', 'T_y', 'T_z']].plot(x="time", kind="line")
plt.title('Success example (id 3)')
plt.savefig('normal.png')

failure = df_ts[df_ts.id == 20][['time', 'F_x', 'F_y', 'F_z', 'T_x', 'T_y', 'T_z']]
failure[['time', 'F_x', 'F_y', 'F_z', 'T_x', 'T_y', 'T_z']].plot(x="time", kind="line")
plt.title('Success example (id 20)')
plt.savefig('failure.png')

In the next step, the relevant feature set is extracted from the 6 different time-series and shown. Some of the features include range counts, standard deviation, variance, auto-correlation, linear-trends, quantiles and change in quantiles.

from tsfresh import extract_relevant_features
X = extract_relevant_features(df_ts, y, column_id='id', column_sort='time')

print(X.info())
print(X.head())

Feature Significance and Relevance

Feature significance is determined by significance tests that help us to decide if the null hypothesis for a given feature can be rejected/accepted.

In the context of time series feature extraction, a wrongly added feature is a feature Xφ for which the null hypothesis H φ 0 is rejected by the respective feature significance test, even though H φ 0 is true. The risk of such a false positive result are done by hypothesis tests tuned for individual features.

However, when comparing multiple hypotheses and features simultaneously, errors in the inference tend to accumulate. In multiple testing, the expected proportion of erroneous rejections among all rejections is called false discovery rate (FDR), that acts as the last component in filtered feature extraction.

The p-values (known as feature selectors) helps to decide which hypotheses to reject while controlling the FDR. For every feature, an univariate feature selection test is conducted which generate the p values, that are then evaluated by the Benjamini Hochberg procedure to decide which features to keep and which to delete.

FDR_LEVEL = 0.05X = extract_features(df_ts,
                     column_id='id', column_sort='time',X_selected = select_features(X, y)print(X.shape)
print(X_selected.shape)
                     default_fc_parameters=ComprehensiveFCParameters(),
                     impute_function=impute)
X = X.loc[:, X.apply(pd.Series.nunique) != 1]df_pvalues_mann = calculate_relevance_table(X, y, fdr_level=FDR_LEVEL, test_for_binary_target_real_feature='mann')

print("Total \t", len(df_pvalues_mann))
print("Relevant \t", (df_pvalues_mann["relevant"] == True).sum())
print(Irrelevant \t", (df_pvalues_mann["relevant"] == False).sum(),
    "( # constant", (df_pvalues_mann["type"] == "const").sum(), ")")

(88, 1968)

(88, 631) # 631 features selected from 1968

Total 1968
Relevant 631
Irrelevant 1337

The relevant set of features with their p-score (using Mann-Whitney) list is given by:

The non-relevant set of features with their p-score (using Mann-Whitney) list is given by:

“Mann-Whitney U test is the non-parametric alternative test to the independent sample t-test”

It is used to compare two sample means that come from the same population, and used to test whether two sample means are equal or not.

Mann-Whitney U test is used in tsfresh to calculate the feature significance of a real-valued feature to a binary target as a p-value.

m = len(df_pvalues_mann.loc[~(df_pvalues_mann.type == "const")])
K = list(range(1, m + 1))
C = [sum([1.0 / i for i in range(1, k + 1)]) for k in K]rejection_line_mann = [defaults.FDR_LEVEL * k / m * 1.0 / c for k, c in zip(K, C)]

df_pvalues_mann.index = pd.Series(range(0, len(df_pvalues_mann.index)))

df_pvalues_mann.p_value.where(df_pvalues_mann.relevant)\
    .plot(style=".", label="relevant features")

df_pvalues_mann.p_value.where(~df_pvalues_mann.relevant & (df_pvalues_mann.type != "const"))\
    .plot(style=".", label="irrelevant features")

df_pvalues_mann.p_value.fillna(1).where(df_pvalues_mann.type == "const")\
    .plot(style=".", label="irrelevant (constant) features")

plt.plot(rejection_line_mann, label="rejection line (FDR = " + str(FDR_LEVEL) + ")")
plt.xlabel("Feature #")
plt.ylabel("p-value")
plt.title("Mann-Whitney-U")
plt.legend()
plt.plot()

When selecting, fdr_level is a hyper-parameter to tune. It is evaluated to represent expected percentage of irrelevant features among set of all present features. By default, it is set at 5%. However, it might needs to be increased to 0.9, depending on how the chosen classifier is able to deal with non-informative features.

Feature importance filter can also be applied with standard Classification or Regression algorithms. Here I used RandomForestClassifier to evaluate feature importances.

clf = RandomForestClassifier()
clf.fit(X, y)
importance = pd.Series(clf.feature_importances_)
importance.sort_values(ascending=False)
print(importance)

Plotting the relative feature importances through a bar plot gives the following feature importance plot.

imp_frame = importance.to_frame()
imp_frame.plot(kind="bar")
plt.xticks([])
plt.xlabel('Features')
plt.ylabel('Importances')
plt.title('Feature Importance Plot')
plt.savefig('importance.png')

Classification

The standard classifiers Logistic Regression, Boosting and Bagging can be applied on extracted feature set (X). I have used RandomForestClassifier to illustrate the robot execution failure and success classes, after splitting the selected feature set (X) into train and test subsets.

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report
testtrain_xy = train_test_split(X, y,test_size=.4,random_state=50)
X_train, X_test, y_train, y_test = testtrain_xy
print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)

(52, 631) (36, 631) (52,) (36,)

The trained classifier is evaluated on the test data set to evaluate its performance through precision, recall and f1-score.

clf = RandomForestClassifier()
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
print(classification_report(y_test, y_pred))
print(np.sum(y_test == y_pred), y_test.shape)

from sklearn.model_selection import  cross_val_predict
predicted = cross_val_predict(clf, X, y, cv=5)
print(classification_report(y, predicted))

Further feature extraction and model classification can also be performed using Pipeline from scikit-learn, with FeatureAugmentor at the start followed by any classifier like RandomForest.

from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
from tsfresh.transformers import RelevantFeatureAugmenter

In the fit phase, all possible time series features are calculated that is set by the set_params function (if the features are not manually changed by handing in a feature_extraction_settings object). Then, their significance and relevance to the target is computed using statistical methods and only the relevant ones are selected by applying suitable algorithmic (like Benjamini Hochberg) procedure.

In the transform step (fit_transform), the information on which features are relevant from the fit step is used and those features are extracted from the time series. These extracted features are then added to the input data sample that is fed to the classifier.

pipeline = Pipeline([('augmenter', RelevantFeatureAugmenter(column_id='id', column_sort='time')),
            ('classifier', RandomForestClassifier())])

X = pd.DataFrame(index=y.index)

pipeline.set_params(augmenter__timeseries_container=df_ts)
pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)
print(classification_report(y_test, y_pred))

Conclusion

The model can be used in distributed industrial big data applications for the following purposes:

• It is designed to allow consideration of several different time series types per label in addition to several meta-information.
• The parallel distributed functionality helps feature extraction and filtering scalable. Data is processed in distributed architecture for applications where data is fragmented over a widespread infrastructure limiting data aggregation and processing on centralized infrastructure.
• It allows easy combination with domain specific and possibly stateful feature mappings from more specialized machine learning algorithms.
• The algorithm helps to model different industrial big data use-cases with limited domain knowledge, and low computational complexity.
• The model scales linearly with the number of samples and length of time series.

References

1. Christ, Kempa-Liehr, Feindt: Distributed and parallel time series feature extraction for industrial big data applications, arXiv:1610.07717
2. Balancing Small Samples and Big Data : Andreas W. Kempa-Liehr
3. https://archive.ics.uci.edu/ml/datasets/Robot+Execution+Failures