Anamoly Detection in Very Large Scale System using Big Data

Big data refers to a term that is used to describe vast amounts of data that have multiple kinds of Vs: velocity, variety, and volume. It could be semi-structured, unstructured, or even structured, making data analysis difficult. New architecture, methodologies, algorithms, and analytics are needed to extract hidden data and identify assaults on enormous amounts of data. It is quite challenging to identify assaults using conventional methods. This study provides a thorough analysis of malware detection in several sectors using deep learning and provides an overview of deep learning data. In networked computers, there have been more attacks. To protect a network, a strong intrusion detection system (IDS) is necessary. Reviewing the literature reveals that while some research has been conducted in this area, a thorough and in-depth investigation has not yet been carried out. For unanticipated and unpredictable assaults, many academics suggested an IDS employing deep learning, but not for big data. The present research design is based on three ensemble methods, Randam Forest, Decision tree regression, and Gradient Boosting Tree, as well as a deep learning-based intrusion detection system for large datasets named RNN that runs for 1,000 epochs with a learning rate complexity and diversity [0.01-0.5]. It is employed in the creation of the hybrid, safe, and scalable, which is based on big data and deep learning methods. In comparison to using just one classifier, the suggested classifiers provide a more accurate classification. Detection rate (99 percentage), false positive rate (1.5 percent), accuracy (99 percentage), and F-Measure (99.03%) are the experimental results. The results show that new anomaly detection methods work better in the big data context