📊 5 Useful Python Scripts for Automated Data Quality Checks

📌 Introduction

Data quality issues are pervasive and can lead to incorrect business decisions, broken analysis, and pipeline failures. Manual data validation is time-consuming and prone to errors, making it essential to automate the process. This article discusses five useful Python scripts for automated data quality checks, addressing common issues such as missing data, invalid data types, duplicate records, outliers, and cross-field inconsistencies.

📌 Main Content / Discussion

The five Python scripts are designed to handle specific data quality issues.

import pandas as pd
import numpy as np

# Example 1: Missing data analyzer script
def analyze_missing_data(df):
    missing_data = df.isnull().sum()
    return missing_data

# Example 2: Data type validator script
def validate_data_types(df, schema):
    for column, dtype in schema.items():
        if df[column].dtype != dtype:
            print(f"Invalid data type for column {column}")
    return df

# Example 3: Duplicate record detector script
def detect_duplicates(df):
    duplicates = df.duplicated().sum()
    return duplicates

# Example 4: Outlier detection script
def detect_outliers(df, column):
    Q1 = df[column].quantile(0.25)
    Q3 = df[column].quantile(0.75)
    IQR = Q3 - Q1
    lower_bound = Q1 - 1.5 * IQR
    upper_bound = Q3 + 1.5 * IQR
    outliers = df[(df[column] < lower_bound) | (df[column] > upper_bound)]
    return outliers

# Example 5: Cross-field consistency checker script
def check_cross_field_consistency(df):
    # Check for temporal consistency
    df['start_date'] = pd.to_datetime(df['start_date'])
    df['end_date'] = pd.to_datetime(df['end_date'])
    inconsistencies = df[df['start_date'] > df['end_date']]
    return inconsistencies

These scripts can be used to identify and address data quality issues, ensuring that the data is accurate, complete, and consistent.

📌 Conclusion

The five Python scripts discussed in this article provide a comprehensive solution for automated data quality checks. By using these scripts, data analysts and scientists can identify and address common data quality issues, ensuring that their data is reliable and accurate. The main insights from this article include the importance of automating data quality checks, the use of Python scripts for data validation, and the need for consistent data quality practices.
#DataQuality #DataValidation #PythonScripts #AutomatedDataQualityChecks #DataScience #MachineLearning

🔗 Read More https://www.kdnuggets.com/5-useful-python-scripts-for-automated-data-quality-checks

Data validation with Pydantic! 🐍✨

In the early stages of development, data validation usually doesn't cause problems. In many Python projects, validation initially looks simple:

if not isinstance(age, int):
    raise ValueError("age must be an int")

But then come email, JSON from APIs, query parameters, nested objects, configs, nullable fields, and type conversion. At some point, the code turns into a set of if/else and manual checks.

For such tasks, Pydantic is often used. Installation:

pip install pydantic
pip install "pydantic[email]"

Create a model:

from pydantic import BaseModel

class User(BaseModel):
    name: str
    age: int

Now the data is validated automatically:

user = User(
    name="Alex",
    age="30"
)

print(user.age)
print(type(user.age))

The result:
30
<class 'int'>

Pydantic will automatically convert the string "30" to an int. If you pass an incorrect value, you'll get a ValidationError:

User(
    name="Alex",
    age="test"
)

This is especially convenient when working with APIs, JSON, query parameters, and incoming data from outside.

A common production case is checking email:

from pydantic import BaseModel, EmailStr

class User(BaseModel):
    email: EmailStr

User(email="[email protected]")

If the email is invalid, Pydantic will throw a ValidationError. You can set default values:

from pydantic import BaseModel

class Config(BaseModel):
    host: str = "localhost"
    port: int = 5432

And allow None:

from pydantic import BaseModel

class User(BaseModel):
    nickname: str | None = None

This field becomes optional. A practical example is processing an API response:

from pydantic import BaseModel

class Product(BaseModel):
    id: int
    title: str
    price: float

data = {
    "id": "1",
    "title": "Keyboard",
    "price": "99.5"
}

product = Product(**data)

print(product)

The types will be automatically converted. For nested model structures, you can combine:

from pydantic import BaseModel

class Address(BaseModel):
    city: str
    zip_code: str

class User(BaseModel):
    name: str
    address: Address

user = User(
    name="Alex",
    address={
        "city": "Berlin",
        "zip_code": "10115"
    }
)

print(user)

The nested object will also be validated. Serialization in Pydantic v2:

print(user.model_dump())
print(user.model_dump_json())

Pydantic is actively used in FastAPI, ETL, microservices, data pipelines, and API clients.

For working with environment variables in Pydantic v2, a separate package is usually used:

pip install pydantic-settings

It's important to understand: Pydantic is not an ORM and does not replace business logic. Its task is to validate data, convert types, and describe schemas.

🔥 Pydantic significantly reduces the amount of manual data validation and makes processing incoming structures more predictable.

#Python #Pydantic #DataValidation #FastAPI #Coding #DevOps

✨ Join Best TG Channels https://t.iss.one/addlist/0f6vfFbEMdAwODBk
⭐️ Join Our WhatsApp Channel https://whatsapp.com/channel/0029VaC7Weq29753hpcggW2A