Transfer Learning in various IoT applications using sensor fusion technology

Motivation

Deep learning has been known to learn non-linear data representations for large and complex datasets. It has been successful in exploring hidden features and explaining sources of data variations. As deep learning mechanisms evolved to solve multiple AI problems starting from time series, computer vision and Natural Language Processing (NLP), it became an useful AI technique to co-relate problems in similar domains by applying the knowledge of the source domain. In addition it saved the training cost of learning insufficient datasets from the beginning. In this blog, I explain how transfer learning plays an important role in the field of IoT where sensors:

• Vary and different sensors collect different types of information.

• Have limited power and can give erroneous reading when battery runs low.

• Collecting similar information can be used to learn and monitor related events, activities and tasks.

• Collecting varying environment and system parameters can revolutionize IoT industry by using sensor fusion.

What is Transfer Learning?

Trained ML models for a context have often seen to be useful in related domains which have un-labelled training dataset or historical data of less relevance. To model such contexts, statisticians have come up with an advanced concept of transferring model information of one domain to another similar type of domain through the concept of Transfer Learning. It helps to leverage recent data from similar problems along with historical data to improve predictions of the current problem .

To make the connection between the two problems, transfer learning algorithms help focus the learning process on the more relevant parts of the historical training data. It is highly useful when the historical data is limited, or the environment is uncertain.

Transfer Learning have been known to solve problems by

• Offering a set of algorithms that identify similar areas of knowledge which are “transferable” to the target domain.

• Offering a set of algorithms with similar use-cases, inputs at same or different periods of time.

• Identifying related previous, recent and target tasks and activities from the domain of existing trained model to the target domain.

• Combining algorithms and models from a set of similar domains to the target domain, a concept known as ensemble transfer learning.

• Transfer knowledge across domains or tasks from numerous source domains that have different feature spaces, a concept known as heterogeneous transfer learning.

Examples of Transfer Learning

• Transfer of business knowledge from one country where business is operating to another country where business is new and expanding and they exhibit similar market conditions.

• Transfer of economical, financial and strategic investment knowledge from one sector to another sector across countries which have similar economy, GDP and technological advancements.

• Mapping political affairs across nations. For e.g. 2016 Brexit voting data from the UK would have allowed statisticians to predict U.S. voting trends and more support for Trump when nations (UK and U.S.A.) shared same anti-immigration sentiment.

• Transfer knowledge of working states of machineries and electronic equipments from one industry to another.

• Mapping public sentiments for major events : politics, sports, foreign policies to similar upcoming events.

• Transfer and translate specialized feature vectors on same problem space like : sensor-network-based localization, text classification, image classification, video classification, social network analysis and logical inference.

• Transfer knowledge between entirely different feature spaces, a concept known as translated learning. Such transfers is made possible by learning a mapping function for bridging features in two entirely different domains.

• One very popular example is the type of learning used in recommender systems. Here the collaborative filtering problems to solve the cold start and sparsity problems uses transfer learning. Rating-Matrix Generative Model (RMGM) uses shared latent user and item spaces in order to transfer useful knowledge for collaborative filtering and recommendation engines. RMGM attains this by creating multiple rating matrices from different domains by mapping the users and items.

IOT Sensor Fusion

IoT sensors are used widely, and sensors mimic all sorts of human activity. Technological advancements have led to the discovery of a concept called sensor fusion, which aggregates data collected from individual sensors to build more reliable ML models that gives higher prediction accuracy. Sensor fusion works on the principles of:

• Data integration and aggregation from multiple sensors.

• Offer higher levels of recognition and multitude of new responses.

• Correcting errors from individual sensors by consolidating readings from similar devices. For e.g. gyroscopes suffer from offset drifts over time. However this can be compensated for using a companion accelerometer.

• Transfer related sensor’s knowledge to another domain through transfer learning.

• Use ensemble learning to transfer, aggregate, auto-correct and consolidate information from numerous sensors.

Sensor Fusion and Transfer Learning

Sensor fusion technology is emerging and has been used in both simple and complex IoT fusion devices.

• E-compass is a combo device which is devised using a combination of a 3D magnetometer and 3D accelerometer.

• Fused sensors to combine 3D accelerometers, 3D gyroscopes and 3D magnetometers. It is capable of measuring the components of the magnetic field in a particular direction, relative to the spatial orientation of a given device.

Each of the sensors (can be embedded in different devices) provides unique functionality, but also has limitations:

Accelerometer: x-, y- and z-axis linear motion sensing along 3 different directions. It is highly sensitive to vibration

Gyroscope: pitch, roll and yaw rotational sensing, but zero bias drift

Magnetometer: x-, y- and z-axis magnetic field sensing along 3 different directions. It is highly sensitive to magnetic interference

seNSORS

• A sensor fusion platform can be used to combine IoT sensor/ RFID tag readers’ information from a variety of hospitals to derive an accurate prediction model to monitor movements by doctors and nurses, alerting them if they forget to wash their hands before and after touching patients. Knowledge from a set of devices from one or few hospitals can be transferred to a bigger context of patient monitoring and hospitalisation by means of ensemble transfer learning.

• Autonomous driving technologies use Radar, Camera and LIDAR (Light Detection and Ranging Device). These technologies helps to estimate the location, velocity and trajectory of various classes/objects on the road. Sensor fusion technology has accelerated IoT information learning by combining records from each sensor to generate a probabilistic state of the classes (objects) around it. An example of this includes Kalman Filter.

• IoT sensor fusion technology creates a platform to monitor a person’s vital signs, emotions and movements that could be tracked 24 hours a day. Data from numerous IoT sensors can be aggregated, pre-processed with correct timestamps. Once they are ready to be used in ML they can yield in better prediction models with ensembling techniques. Prediction models created out of sensor fusions can be translated to activity monitoring domain to monitor individual’s health, movement of children, soldiers in battlefield as they help in raising alerts when safety of an individual is at risk.

Mechanisms of ensemble transfer learning

Ensemble learning helps improve predictive power of machine learning models by combining and averaging several models. Ensemble methods aim at decreasing variance (bagging), bias (boosting), or improving predictions (stacking) by means of meta algorithms that embeds independent algorithms (like decision trees) by sequential (boosting) or parallel means (bagging). In the below section, I describe different ensemble techniques of transfer learning using traditional ML.

• Weighted- resampling method called TrResampling for transfer learning where the best base-learner among Naive Bayes, Decision Tree or SVM is chosen for transfer learning. TrResampling allows creation of a new training dataset with same size as original dataset. Samples from original data set with higher frequency exhibits higher probability of getting selected and being classified correctly. TrResampling, works on multiple iterations which generates new training data set at each step, by combining weighted-resamples from the original source data set along with the labeled data in the target data set.

• TrAdaBoost : It works iteratively on large datasets, where data with heavy weights in the source domain are resampled. The TrAdaBoost algorithm is used to adjust the weights of the source data and target data. Multiple classifiers are learned on training data by a given weak learner, and the classifiers produced by the weak learner are combined into a single composite classifier.

• MultiBoosting-based transfer learning algorithm-TrMultiBoosting: It is used in addition to TrAdaBoost to increase the classification accuracy. In TrMultiBoosting, many sub-samples are generated by MultiBoosting on the training data, which are combined as source data sets and labeled data in the target data sets, and the weights of the data in each sub-sample is assigned by a continuous Poisson distribution. It is followed by TrAdaBoost to adjust the weights of the data in the subcommittees.

• Bagging-based transfer learning (BETL): It incorporates bagging algorithm in the ensemble framework. Bagging is very sensitive to noisy training data, and best suited when the base-learner is unstable and situations when the training data overfits. BETL uses bootstrapping to generate random subsets from the source domain and adds the labelled data in the target domain. The combined dataset is used as the training data to learn many initial classifiers.

Future Research

While transfer learning has opened a new research area in utilizing information from similar domain, new research areas which data scientists are actively looking are:

• To analyze domains that are fundamentally different with special emphasis on cross-task and cross-domain similarities and relevances. Here unsupervised learning such as data clustering looks promising. Sensor fusion technology has laid its vast impact on ubiquitous computing and scientists are constantly looking for ways to gather data from socially available IoT devices and apply it on the physical world.

• To explore areas related to Transitive Transfer Learning (TTL), where an intermediate domain is used to reduce large domain distances, when the source and target domains share few factors directly.

• To collaborate deep learning methods with ensemble transfer learning to reduce the dimension and enhance the classification accuracy. For e.g. multiple IoT devices might have redundant feature sets and higher error margins. Such models when trained by ensemble transfer learning (e.g. deep learning based transfer learning algorithm such as convolutional neural networks) helps to remove redundant features and increase prediction performance.

• To identify data relationship between the source domain and target domain to solve the negative transfer problem. To solve negative transfer problem, different IoT datasets with different features (sensor modality, sampling rate, activity duration and environment) should be identified properly by means of feature learning. Once identified the feature set could be used to transform learned representation of activities between source and transfer domain.

• To use Reinforcement Transfer Learning (RTL) to map states and actions between source and target domains by using several sensors. RTL offers scope in IoT based Remote Access Human Control to transfer knowledge across domains. RTL can generate efficient robotic movements by :

1. Boosting performance of an agent in a target task during start by transfer from source task.
2. Increasing learned performance of an agent in the target task.
3. Increasing total reward accumulated by an agent.
4. Reducing learning time via knowledge transfer.
5. Increasing ratio of total reward accumulated by the transfer learner to the total reward accumulated by the non-transfer learner.

References:

1. https://www.cse.ust.hk/~qyang/Docs/2009/tkde_transfer_learning.pdf
2. http://www.deepnets.ineb.up.pt/files/publications/chetak_IWANN2015_ETL_cameraready.pdf
3. https://www.slideshare.net/jins0618/transfer-learning-an-overview
4. http://www.jmlr.org/papers/volume10/taylor09a/taylor09a.pdf
5. https://ieeexplore.ieee.org/document/8194845
6. https://www.mdpi.com/1424-8220/15/12/29858/htm
7. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0130851
8. https://www.einfochips.com/blog/ml-at-the-edge-will-help-unleash-the-true-potential-of-iot/
9. https://www.mouser.in/applications/sensor-fusion-iot/
10. https://people.csail.mit.edu/lpk/papers/MarxRosensteinKaelblingDietterich-final.pdf

Please let me know if there were any mistakes. Suggestions and feedbacks are welcome. Feel free to follow me at linkedin.