P2.2.4 - Evaluating Prediction Performance (5 marks) You will now need to visualise the prediction performance of the mo

[answerhappygod]

P2.2.4 - Evaluating Prediction Performance (5 marks)
You will now need to visualise the prediction performance of the
models, and evaluate them in terms of prediction accuracy, and mean
square error (MSE) metrics. For this purpose, create a function
prediction_perf(gt, MLE_model_predictions) which evaluates the
prediction performance of the reference models. Up to this point,
you should have obtained
N=100 samples of predictions from each model, stored in
MLE_model_predictions.
The ground-truth FTR values from data frames df, stored in
gt.
Assume predicted values for a given model are stored in a
variable P and its average is P¯. The first performance measure
will be the MSE, and will be calculated for each model from the
expression below:
MSE=1N∑i=0N−1(Pi−FTRi)2
In order to obtain the prediction accuracy for each model, you
first should convert continuous prediction results into the binary
form (either 1.0 or 0.0). The binarisation process will follow the
piecewise function below:
Pbinary,i={1.0,0.0,Pi≥P¯otherwisewherei=0,1,…,99
Then the percentage accuracy, Acc% is calculated as
Acc%=100−∑i=099|Pbinary,i−FTRi|
Following these, by using sklearn module methods roc_curve() and
auc() find ROC curve parameters and AUC metric for each prediction
model.
In order to obtain performance analysis results in a neatly way,
you then need to create a new pandas dataframe df_results which
will be in the form of
Consequently, using sklearn method RocCurveDisplay(), the
prediction_perf() function should
Marking for this question
with a condition that all these three operations are performed
in a fully working prediction_perf()
function.
[ ]
def prediction_perf(gt, MLE_model_predictions):
# your code here
return df_results
Dataset: