Notes »

data-pre-processing

Step 1: [[Import Required Libraries ]]
  • [[NumPy]]: Numerical Computing
  • [[Pandas]]: Data Manipulation
Step 2: [[Importing Data set]]
Step 3: [[Handling the missing values]]

  • Detecting Missing Values:

    • isnull(): Detects missing values and returns True where they exist.
    • notnull(): Opposite of isnull(), returns True where values are not missing.

  • Dropping Missing Values:

    • If you want to remove rows or columns with missing data, you can use dropna().

  • Filling Missing Values:

    • Instead of dropping rows/columns, you can fill in missing values using fillna()

  • Replacing Missing Values with Interpolation:

    • Means, Median,Mode
    • sklearn.preprocessing
Step 4: Encoding Categorical Data
  • Label Encoder
Step 5: [[Splitting the dataset into test set & training set]]

- train_test_split()

- from sklearn.model_selection import train_test_split
Step 6: [[Feature Scaling]]
  • Min-Max Scaling (Normalization): This technique scales the data between a fixed range, usually 0 and 1.
  • Z-score Normalization (Standardization): This method scales the data such that the mean is 0 and the standard deviation is 1.
  • MaxAbs Scaling: This scales the data by its maximum absolute value, preserving the sign of the data (useful for data that is already centered around 0).
  • Robust Scaling: This technique is less sensitive to outliers as it uses the median and interquartile range (IQR) instead of the mean and standard deviation for scaling.

When to Use Feature Scaling:

  • Algorithms based on distance: Algorithms like k-NN, K-Means, SVM, and PCA rely on the distance between points and can be influenced by the scale of features.

  • Gradient-based algorithms: Algorithms like gradient descent (used in linear regression, logistic regression, and neural networks) converge faster with feature scaling.

  • [[Neural Networks]]: Feature scaling improves performance because it ensures faster convergence and avoids issues with large weight updates.

When Not to Use Feature Scaling:

  • Tree-based algorithms: Algorithms like Decision Trees, Random Forests, and XGBoost do not require feature scaling because they are not distance-based and can handle features with varying magnitudes.

  • Feature scaling ensures that your model performs optimally, treating each feature equally and helping distance- or gradient-based algorithms perform well.