Skip to content

Project Progress report: "Technical audit, preprocessing and visualization of the CSE-CIC-IDS2018 dataset"

Stage 1. Initialization and configuration of the remote GitHub repository

  • The content of the works: Creation of a remote repository "Nikita7291/dataset" on the GitHub platform to ensure shared access and strict version control of the source code and documentation.
  • Technical results: Initialize the project structure, create a configuration file '.gitignoreto filter temporary files of the Python/Jupyter Notebook environment, and make a primary commit of the basic architecture to themain' branch.

Stage 2. Connecting participants and delimiting access rights

  • The content of the works: Organization of a distributed collaborative development environment. Adding members of the engineering team to the repository with the rights to make changes (Collaborators).
  • Technical results: Setting up a branching policy, delineating areas of responsibility for the parallel management of the analytical and software parts of the project.

Stage 3. Deployment and automation of the static MkDocs website

  • The content of the works: Deployment of the MkDocs static website generator to maintain interactive project documentation. Designing the configuration file mkdocs.yml, defining the structure of the navigation menu (nav) and selecting a visual design theme.
  • Technical results: Setting up an automatic CI/CD pipeline via GitHub Actions. Creating a configuration script using the path .github/workflows/ci.yml'. The pipeline automatically tracks commits to themainbranch, parses and compiles Markdown files from thedocs/directory into valid HTML code, and then dispatches the collected static content to thegh-pages` service branch to update the site on the GitHub Pages server.

Stage 4. Study of the structure and specification of the source dataset

  • Content of the works: Architecture study of the network security reference dataset CSE-CIC-IDS2018. Estimation of the total sample size, traffic distribution structure, and mechanisms for generating network metrics generated by the CICFlowMeter utility when intercepting packets.
  • Technical results: Formation of an understanding of the nature of the target variable Label (differentiation of legitimate traffic Benign and various categories of network attacks) and the structure of the 80 initial features.

Stage 5. Statistical analysis and profiling of features

  • Content of the works: Primary data profiling using the pandas and numpy libraries. Application of methods df.info () and df.describe() to calculate basic statistical metrics (mean, median, standard deviation, minima, maxima, and quantiles).
  • Technical results: Defining data types in columns, identifying features with abnormally high variance, and fixing columns containing non-numeric artifacts in continuous quantities.

Stage 6. Identification of critical defects and anomalies in the data

During a deep technical audit of the data structure, the following critical defects were localized and classified, distorting the mathematical apparatus of machine learning models (in particular, the Random Forest algorithm):

  • Missing values (NaN): The presence of blank records of network flows caused by technical failures during traffic aggregation.
  • Infinite values (inf / -inf): Mathematical artifacts of division by zero in the calculation of speed characteristics (in terms of Flow Packets/s and Flow Bytes/s) that occur when the duration of the flow is zero (Flow Duration = 0).
  • Data type errors: Incorrect interpretation of numeric columns as text (object) due to the presence of header rows that were repeatedly duplicated inside the dataset when gluing the original CSV files.
  • Constant signs: Columns with zero variance (containing the same value in all rows of the sample), which do not carry useful information for training classifiers.

Stage 7. Software anomaly removal (Data Cleaning Pipeline)

Development and application of a Python data purification pipeline to bring the sample to a mathematically correct form:

  1. Replacing and deleting incorrect values:
import numpy as np
import pandas as pd

# Converting infinite values to the standard pass category (NaN)
df.replace([np.inf, -np.inf], np.nan, inplace=True)

# Permanently deleting lines containing omissions
df.dropna(inplace=True)
  1. Correction of data types: Forced conversion of features to the correct types float64 and int64 with pre-filtering and removal of duplicate text headers of the CICFlowMeter utility.
  2. Feature filtering: Extracting and deleting columns with a single unique value ('df[col].nunique() <= 1`) to reduce the dimension of the feature space and prevent overfitting the model.

Stage 8. Building analytical graphs and visualizing distributions

  • The content of the works: Implementation of graphical analysis of the processed data using the libraries matplotlib and `seaborn' to confirm the quality of the cleaning.
  • Technical results:
  • Constructing histograms of the distribution of numerical metrics to assess the skewness and kurtosis of distributions.
  • Creation of span diagrams ("whisker boxes") to detect and isolate extreme statistical outliers.
  • Construction of a correlation matrix (Heatmap) to identify multicollinear features to be excluded before the training stage.
  • Visualization of the class ratio (legitimate traffic / attack) to assess the degree of imbalance of the target variable.