Test Set:
o Accuracy: 0.9889
o F1 Score: 0.9890
o ROC AUC Score: 0.9889
Test Set:
o Accuracy: 0.9867
o F1 Score: 0.9868
o ROC AUC Score: 0.9867
Analysis: Based on the evaluation metrics, it can be observed that the model trained on df (which includes all transaction types) outperformed the model trained on df2 (which only considers transfers and cash outs) in terms of accuracy and F1 score. This indicates that including other transaction types in the training data helped improve the model's performance. Furthermore, both models demonstrated high accuracy and F1 scores on both the validation and test sets, indicating their effectiveness in detecting fraudulent transactions. The ROC AUC scores also suggest that the models have good discriminatory power.
Feature Importance: By analyzing the feature importances using XGBoost, we identified the following insights:
To further enhance the fraud detection system, we can consider the following steps:
Overall, our objective is to create a sustainable and continuously improving fraud detection system that contributes to a safer and more secure financial ecosystem.