Optimistic Machine Learning-privacy satisfying accurate models

Motivation

Machine learning models have been known to leak information about the individual data records on which they were trained. One of the mechanisms include : membership inference attack: 

Where a data record and black-box access to a model is used to determine if the record was in the model’s training dataset. 

Membership inference against a target model uses adversarial machine learning to train custom interface models to recognise differences in the target model’s predictions on the inputs that it trained on versus the inputs that it did not train on. In other words– 

Membership inference problem is converted to a classification problem.

Inference techniques on classification models trained by commercial machine learning services evaluates factors that influences leakage of information:

• Many existing attacks on ML models may not require direct access to the model, however they can be accessed through API.
• Customised NLP based models, such as predictive text, are more vulnerable to attacks as expose sensitive information.
• Datasets with sensitive information and their models should be secured.
• Success of the attacks relates to the classification tasks and metrics of overfitting.
• Different machine learning models, with varying internet architectures tend to remember varied information about their training datasets.

As stated by the author— “Membership information leakage is quantified through the prediction outputs of machine learning models.”

Membership inference attacks requires the following steps:

• Accumulate sufficient training data through public or private datasets.
• Building several shadow models that shares similar characteristics of target model, with similar inputs and outputs of the target model. After models are built, there occurs a sequence of hyper-parameter tuning steps in order to increase precision and recall on samples of the training data.
• Training a discriminator which learns the difference in output between the seen training data and the unseen test or validation data based on the input and output to the numerous shadow models. This discriminator is then used to evaluate the target API and determine if a data point is in or out of the target model, allowing the attacks to be executed in a “black-box” environment.

Inference Target: Stand-alone vs. Federated Learning

There are two major types of training algorithms for deep learning, depending on whether the training data is available all in one place (i.e., stand-alone centralized training), or it is distributed among multiple parties who do not trust each other (i.e., federated learning).

In both cases, the attacker could be the entity who obtains the final trained model. In addition to such attack setting, the attacker can use advanced deep learning measures to observe an updated version of the model. In federated learning, the attacker can be an entity who participates in the training, unknown to the central server or any of the participating entities.

Attack Strategies on Machine Learning Models

• Knowledge of the parameters of SVM and HMM models has been used to dig deeper into statistical information about the training dataset.
• Exploiting changes in the outputs of a collaborative recommender system to infer inputs that caused these changes. These attacks exploit temporal behavior specific to the recommender systems based on collaborative filtering.
• Model inversion, which uses the output of a model applied to a hidden input to infer certain features of this input. For e.g. the model can capture the correlation between the patient’s genotype and the dosage of a certain medicine. Model inversion when applied to face recognition models, sets model’s output is set to 1 for class i and 0 for the rest, thus constructing an input that produces the outputs. Model inversion produces the average of the features that at best can characterize an entire output class. However, model inversion suffers from one major limitation when images in a class are diverse (e.g., if the class contains multiple individuals or many different objects) , the results are not recognizable.
• Model extraction techniques are employed to extract the parameters (such as weight metrics of NN) of a model trained on private data. The attacker’s goal is to construct a model whose predictive performance on validation data is similar to the target model. In Kernel logistic regression (KLR), the kernel function includes a tiny fraction of the training data, directly fed into the model as parameters, extracting them results in the leakage of that particular part of the data.
• Privacy-preserving machine learning focuses on how to learn without direct access to the training data. Secure multiparty computation (SMC) has been used for learning decision trees, linear regression functions, Naive Bayes classifiers, and k-means clustering, to limit information leakage during training. However use of same training algorithm, makes them vulnerable to inference attacks as any conventionally trained model.

Shadow Training

Shadow models imitate the behavior of the target model, but for which we know the training datasets and thus the ground truth about membership in these datasets. In the following example, we first train the attack model using the labeled inputs. and outputs of the shadow models.

• Uses Adversarial Generative mechanisms with black-box access to the target model. It helps in data synthesis. The algorithm in shadow training is unbiased and does not use any information on the distribution of the target model’s training data
• Uses statistics about the population instead of any particular sample. The population metrics helps to generate the target’s training dataset.
• Works on the basis that the adversary has access to a noisy version of the target’s training dataset. This further allows the attacker to run successive queries to the target model in order to exploit the model and infer a given record was in its training dataset.
• Works with minimal knowledge about the deployed target model and also the training dataset.

Attacker Information

Under the assumption the attacker may have access to any one of the following information:

• User knows the type and architecture of the machine learning model, as well as the training algorithm, or
• Has black-box access to a machine learning oracle (e.g., a “machine learning as a service” platform) that was used to train the model and does not know a priori the model’s structure or meta-parameters.

The attacker may have learnt some background knowledge before starting his attacks. That helps him in making guesses, how the target model’s training dataset was drawn. Based on his background knowledge he keeps on drawing independent samples from the population, disjoint from the target model’s training dataset. In addition, the marginal distribution of feature values can also help the attacker.

The below code snippet defines the architecture of the target model. As we define this as a white-box attack, hence the attacker is assumed to know this architecture too. Reference — https://github.com/spring-epfl/mia

CODE

def target_model_fn():
    model = tf.keras.models.Sequential()

    model.add(
        layers.Conv2D(
            32,
            (3, 3),
            activation="relu",
            padding="same",
            input_shape=(WIDTH, HEIGHT, CHANNELS),
        )
    )
    model.add(layers.Conv2D(32, (3, 3), activation="relu"))
    model.add(layers.MaxPooling2D(pool_size=(2, 2)))
    model.add(layers.Dropout(0.25))

    model.add(layers.Conv2D(64, (3, 3), activation="relu", padding="same"))
    model.add(layers.Conv2D(64, (3, 3), activation="relu"))
    model.add(layers.MaxPooling2D(pool_size=(2, 2)))
    model.add(layers.Dropout(0.25))

    model.add(layers.Flatten())

    model.add(layers.Dense(512, activation="relu"))
    model.add(layers.Dropout(0.5))

    model.add(layers.Dense(NUM_CLASSES, activation="softmax"))
    model.compile("adam", loss="categorical_crossentropy", metrics=["accuracy"])

    return model

In membership inference attack with black-box setting, the attacker queries the target model with a data record and obtains the model’s prediction on that record. The predicted output is a vector of probabilities, where each output represents one class. The attack model is then fed with the prediction vector, along with the label of the target record. These two parameters are useful in inferring about each individual record from the neural network and identify whether the record was in or out of the target model’s training dataset.

The below code snippet demonstrates an attack model, that is specific to the class of the input.

CODE

def attack_model_fn():
    AttachModelBundle creates multiple instances of this model for each class.
    """
    model = tf.keras.models.Sequential()

    model.add(layers.Dense(128, activation="relu", input_shape=(NUM_CLASSES,)))

    model.add(layers.Dropout(0.3, noise_shape=None, seed=None))
    model.add(layers.Dense(64, activation="relu"))
    model.add(layers.Dropout(0.2, noise_shape=None, seed=None))
    model.add(layers.Dense(64, activation="relu"))

    model.add(layers.Dense(1, activation="sigmoid"))
    model.compile("adam", loss="binary_crossentropy", metrics=["accuracy"])
    return model

Overview of the attack

The training of target model has some characteristics. One of the primary features is to have target and shadow models’ datasets are disjoint. All models’ internal parameters are trained independently.

The objective of the attacker is to construct an attack model in order to identify a specific record is present in the training dataset and differentiate from non-members, based solely on the target model’s output.

As the attack model is designed for each class, it increases the accuracy of the attack because the target model produces different distributions over its individual output classes depending on the input’s true class.

The first step to train an attack model is to train multiple similar “shadow models”. However, the ground truth for each shadow model is known to outside, i.e., whether a given record was in its training dataset or not.

Supervised training on the inputs and the corresponding outputs (each labeled “in” or “out”) of the shadow models teaches the attack model how to distinguish the shadow models’ outputs.

As the attacker is not able to dig into the internal parameters of the target model, the most important task becomes in distinguishing members from non-members through shadow training. This creates a primary challenge in training attack models.

The Shadow training technique trains the attack model with supervised learning on proxy targets for which the training dataset is known.

Inference attack components

To extract features from the output of each layer, plus the one-hot encoding of the true label and the loss, the following architectural components are used:

• Fully connected network (FCN) submodules with one hidden layer.
• Convolutional neural network (CNN) submodules for the gradients, to compute the gradients on fully connected layers (in the target model).The size of the convolutional kernel is set to the input size of the fully connected layer. Each activation function embodies the correlation of the gradients. The output of each submodule component is reshaped into a flat vector, and then concatenated the output of all components. The outputs of all attack feature extraction components are combined using a fully connected encoder component with multiple hidden layers. The predicted output score (in the supervised attack raining) obtained from the encoder predicts the membership probability of the input data.

Shadow Models

The shadow models are trained in a similar way to the target model. This is easy if the target’s training algorithm (e.g., neural networks, SVM, logistic regression) and model structure (e.g., the wiring of a neural network) are known.

The attacker can use black-box mechanism to use the API service (e.g., Google Prediction API) to train the shadow model as was used to train the target model.

As we know, the attacker can also use shadow models to design the target model, we should try to increase the the attack model’s accuracy by introducing higher number of shadow models.

Training data for Shadow Models

One of the requisites of building shadow models is to have training dataset that exhibits close similarity to the target model’s training data. Several methods for generating such data are stated below:

Model-based synthesis –This mechanism allows to generate synthetic training data for the shadow models by taking help from target model. The intuition is that records that are classified by the target model with high confidence should be statistically similar to the target’s training dataset and thus provide good fodder for shadow models. Synthesis process runs in two phases:

• Search using a hill-climbing algorithm, exploits the space of possible data records and detect right set of inputs with high confidence.
• Sample synthetic data from these records. After this process synthesises a record, the process can be repeated by the attacker till all the shadow model’s data is generated.

Statistics-based synthesis-The attacker may have some statistical information about the population from which the target model’s training data was drawn, like having prior knowledge of the marginal distributions of different features.

Noisy real data-The attacker may have access to similar or noisy data. This could be when the training data for the target and shadow models are not sampled from exactly the same population, or in some cases, the data is sampled in a non-uniform way.

As illustrated in the above figure, each of the shadow models trained, the same set of models are queried with record of training and test data set. Both the predicted outputs are fed to construct the attack training set in a black-box fashion. Ultimately the data set built is used to train the Attack Model

Target Models

Inference attacks can be executed on three types of target models, listed below, with no knowledge on type or structure of the models they create, or even the values of the hyper-parameters used during the training process.

• Machine learning as a service- Google Prediction API allows the user uploads a dataset and obtains an API for querying the resulting model. There are no configuration parameters that can be changed by the user.

• Amazon ML –User cannot choose the type of the model but can control a few meta-parameters, like the maximum number of passes/epochs over the training data and L2 regularization amount.
• Neural networks. Neural networks with locally defined architectures.

Mitigation

• Regularization techniques such as dropout can help defeat overfitting and also strengthen privacy guarantees in neural networks. Regularization is also used for objective perturbation in differentially private machine learning.Well-regularized models should not leak much information about their training data, and inference attack can serve as a metric to quantify this.
• XOR of some input features adds generalization providing information leakage.
• Platform service providers like Google Prediction API and Amazon ML other than taking into account model accuracy, should estimate the risk that it will leak information about its training data. Accordingly, they need to explicitly warn customers about this risk and provide more visibility into the model and the methods that can be used to reduce this leakage.

Testing Success of Attack

CODE

def compute_attack_accuracy():
(X_train, y_train), (X_test, y_test) = get_data()

    # Train the target model.
    print("Training the target model...")
    target_model = target_model_fn()
    target_model.fit(
        X_train, y_train, epochs=FLAGS.target_epochs, validation_split=0.1, verbose=True
    )

    # Train the shadow models.
    smb = ShadowModelBundle(
        target_model_fn,
        shadow_dataset_size=SHADOW_DATASET_SIZE,
        num_models=FLAGS.num_shadows,
    )

    # We assume that attacker's data were not seen in target's training.
    attacker_X_train, attacker_X_test, attacker_y_train, attacker_y_test = train_test_split(
        X_test, y_test, test_size=0.1
    )
    print(attacker_X_train.shape, attacker_X_test.shape)

    print("Training the shadow models...")
    X_shadow, y_shadow = smb.fit_transform(
        attacker_X_train,
        attacker_y_train,
        fit_kwargs=dict(
            epochs=FLAGS.target_epochs,
            verbose=True,
            validation_data=(attacker_X_test, attacker_y_test),
        ),
    )

    # ShadowModelBundle returns data in the format suitable for the AttackModelBundle.
    amb = AttackModelBundle(attack_model_fn, num_classes=NUM_CLASSES)

    # Fit the attack models.
    print("Training the attack models...")
    amb.fit(
        X_shadow, y_shadow, fit_kwargs=dict(epochs=FLAGS.attack_epochs, verbose=True)
    )
data_in = X_train[:ATTACK_TEST_DATASET_SIZE], y_train[:ATTACK_TEST_DATASET_SIZE]
    data_out = X_test[:ATTACK_TEST_DATASET_SIZE], y_test[:ATTACK_TEST_DATASET_SIZE]

Prediction and Evaluation

CODE

# Compile them into the expected format for the AttackModelBundle.
    attack_test_data, real_membership_labels = prepare_attack_data(
        target_model, data_in, data_out
    )

    # Compute the attack accuracy.
    attack_guesses = amb.predict(attack_test_data)
    attack_accuracy = np.mean(attack_guesses == real_membership_labels)

Mitigation Strategies

There are several mitigation strategies against membership inference attacks

• Restrict the prediction vector to top k classes —When the number of classes is large, many classes may have very small probabilities in the model’s prediction vector. Adding a filter to the last layer of the model, can help to restrict the output probabilities of the most likely k classes, thereby lessening the amount of information leaks.
• Coarsen precision of the prediction vector — To implement this, data scientists should round the classification probabilities in the prediction vector down to d floating point digits, as smaller d helps to reduce information and prevent model leaks.
• Increase entropy of the prediction vector — Membership inference attack also exploits the difference between the prediction entropy of the target model on its training inputs versus other inputs. As a mitigation technique for neural-network models, we can modify (or add) the softmax layer and increase its normalizing temperature t > 0. The softmax layer converts the logits computed for each class into probabilities. Knowledge distillation and information transfer between models, would increase the entropy of the prediction vector.
• Use regularization — Regularization techniques are used to overcome overfitting in machine learning.

Conclusion

• This type of inference attacks work in a black-box setting, without any knowledge of the model’s parameters, and infer information about specific records in the training dataset, as opposed to general statistics.
• Overfit models and loss of generalization of inputs is seen to be a detrimental factor for leaking sensitive information.
• Shadow models can be used to generate synthetic or noisy data, from the target model itself, without any prior knowledge about the distribution of the target model’s training data.
• Inference attacks serves to expose leakage metrics and can also be deployed to judge the effectiveness of machine-learning services at cloud.
• Evaluating it on Keras’ CIFAR dataset yields 65.76% accuracy with 6143 loss.

References

1. https://www.cs.cornell.edu/~shmat/shmat_oak17.pdf
2. https://arxiv.org/pdf/1812.00910.pdf
3. https://github.com/spring-epfl/mia
4. https://www.infoq.com/articles/privacy-attacks-machine-learning-models/