Skip to main content
devn.

Knowledge Discovery in Database: Data Mining and Fundamentals

Understanding the fundamentals of knowledge discovery in database, steps involved in the process and its core part of data mining.

NKNabin Khair
3 min read
Cover image for Knowledge Discovery in Database: Data Mining and Fundamentals

Background

Been long time since i didn't write and especially i have written mostly about AI and Programming related, haven't written something really about the topics that were used to be the hot topics back 10 to 15 years back I guess, where internet was booming and people were getting interested about mobile phones, the services like Google, and other search engines.

Introduction

From the definition of Knowledge Discovery in Database (KDD), it is the process of extracting information, knowledge, and patterns from large datasets that combines the techniques like machine learning, statistics, and pattern recognition. The ideal goal of KDD is to convert the raw data (unorganized, unstructured and semi-structured) into useful knowledge that we can use for our wisdom and decision making.

Process of KDD

The core steps of KDD Process are:

Knowledge Discovery in Database Process

  1. Data Cleaning: The process of removal of noisy and inconsistent data from data collection is called data cleaning. It incudes process likes:
    • Cleaning Missing Values
    • Cleaning Noisy data
    • Remove duplicate values
  2. Data Integration: The process of combining heterogenous datasets from various data sources is called data integration. It includes process likes:
    • Conflict resolution like Naming conflicts
    • Removing redundancy
  3. Data Selection: The process of deciding and then retriving the necessary data from integrated data sources for next stage if KDD is called called data selection. It includes process likes:
    • Identifying relevant data
    • Selecting relevant data
  4. Data Transformation: The process of transforming data from one format to another format that is accepted by the data mining process and its algorithm is called data transformation. It includes process likes:
    • Dimensionality Reduction
    • Data Aggregation
    • Data Normalization
  5. Data Mining: The process of finding hidden patterns, information, and knowledge from the datasets using the techniques like pattern recognition, machine learning, and statistics is called data mining. It includes process likes:
    • Classification
    • Clustering
    • Anomaly Detection
    • Association Rule Mining
  6. Pattern Evaluation: The Process of evaluating discovered pattern and information to determine significance is called data evaluation. It includes process likes:
    • Accessing Pattern Quality
    • Evaluating Pattern Significance
  7. Knowledge Presentation: The Process of representing the extracted knowledge from the data in the way that human can easily understand and use is called knowledge presentation. It includes process likes:
    • Visualization
    • Reporting

Terminology:

  1. Heterogeneous Data: The data that is collected from different sources and in different formats is called heterogeneous data.
  2. Anomaly Detection: The process of identifying the data points that is different from the rest of the data is called anomaly detection.