Topic: Python SciPy – From Easy to Top: Part 1 of 6: Introduction and Basics

---

1. What is SciPy?

• SciPy is an open-source Python library used for scientific and technical computing.

• Built on top of NumPy, it provides many user-friendly and efficient numerical routines such as routines for numerical integration, optimization, interpolation, eigenvalue problems, algebraic equations, and others.

---

2. Installing SciPy

If you don’t have SciPy installed yet, use:

pip install scipy

---

3. Importing SciPy Modules

SciPy is organized into sub-packages for different tasks. Example:

import scipy.integrate
import scipy.optimize
import scipy.linalg

---

4. Key SciPy Sub-packages

• scipy.integrate — Numerical integration and ODE solvers.
• scipy.optimize — Optimization and root finding.
• scipy.linalg — Linear algebra routines (more advanced than NumPy’s).
• scipy.signal — Signal processing.
• scipy.fft — Fast Fourier Transforms.
• scipy.stats — Statistical functions.

---

5. Basic Example: Numerical Integration

Calculate the integral of sin(x) from 0 to pi:

import numpy as np
from scipy import integrate

result, error = integrate.quad(np.sin, 0, np.pi)
print("Integral of sin(x) from 0 to pi:", result)

---

6. Basic Example: Root Finding

Find the root of the function f(x) = x^2 - 4:

from scipy import optimize

def f(x):
    return x**2 - 4

root = optimize.root_scalar(f, bracket=[0, 3])
print("Root:", root.root)

---

7. SciPy vs NumPy

• NumPy focuses on basic array operations and linear algebra.

• SciPy extends functionality with advanced scientific algorithms.

---

8. Summary

• SciPy is essential for scientific computing in Python.

• It contains many specialized sub-packages.

• Understanding SciPy’s structure helps solve complex numerical problems easily.

---

Exercise

• Calculate the integral of e^(-x^2) from -infinity to +infinity using scipy.integrate.quad.

• Find the root of cos(x) - x = 0 using scipy.optimize.root_scalar.

---

#Python #SciPy #ScientificComputing #NumericalIntegration #Optimization

https://t.iss.one/DataScienceM

Topic: Python SciPy – From Easy to Top: Part 2 of 6: Numerical Integration and Differentiation

---

1. Numerical Integration Overview

• Numerical integration approximates the area under curves when an exact solution is difficult or impossible.

• SciPy provides several methods like quad, dblquad, and trapz.

---

2. Using `scipy.integrate.quad`

This function computes the definite integral of a function of one variable.

Example: Integrate cos(x) from 0 to pi divided by 2

import numpy as np
from scipy import integrate

result, error = integrate.quad(np.cos, 0, np.pi/2)
print("Integral of cos(x) from 0 to pi/2:", result)

---

3. Double Integration with `dblquad`

Integrate a function of two variables over a rectangular region.

Example: Integrate f(x, y) = x times y over x from 0 to 1, y from 0 to 2

def f(x, y):
    return x * y

result, error = integrate.dblquad(f, 0, 1, lambda x: 0, lambda x: 2)
print("Double integral result:", result)

---

4. Using the Trapezoidal Rule: `trapz`

Useful for integrating discrete data points.

Example:

import numpy as np
from scipy import integrate

x = np.linspace(0, np.pi, 100)
y = np.sin(x)

area = integrate.trapz(y, x)
print("Approximate integral using trapz:", area)

---

5. Numerical Differentiation with `derivative`

SciPy’s derivative function approximates the derivative of a function at a point.

Example: Derivative of sin(x) at x equals pi divided by 4

from scipy.misc import derivative
import numpy as np

def f(x):
    return np.sin(x)

dx = derivative(f, np.pi/4, dx=1e-6)
print("Derivative of sin(x) at pi/4:", dx)

---

6. Limitations of `derivative`

• derivative uses finite difference methods, which can be noisy for non-smooth functions.

• Suitable for simple derivative calculations but not for complex cases.

---

7. Summary

• quad is powerful for one-dimensional definite integrals.

• dblquad handles two-variable integration.

• trapz approximates integration from sampled data.

• derivative provides numerical differentiation.

---

Exercise

• Compute the integral of e to the power of negative x squared from 0 to 1 using quad.

• Calculate the derivative of cos(x) at 0.

• Use trapz to approximate the integral of x squared over \[0, 5] using 50 points.

---

#Python #SciPy #NumericalIntegration #Differentiation #ScientificComputing

https://t.iss.one/DataScienceM

Topic: Python SciPy – From Easy to Top: Part 3 of 6: Optimization Basics

---

1. What is Optimization?

• Optimization is the process of finding the minimum or maximum of a function.

• SciPy provides tools to solve these problems efficiently.

---

2. Using `scipy.optimize.minimize`

This function minimizes a scalar function of one or more variables.

Example: Minimize the function f(x) = (x - 3)^2

from scipy import optimize

def f(x):
    return (x - 3)**2

result = optimize.minimize(f, x0=0)
print("Minimum value:", result.fun)
print("At x =", result.x)

---

**3. Minimizing Multivariable Functions**

Example: Minimize f(x, y) = (x - 2)^2 + (y + 3)^2

def f(vars):
    x, y = vars
    return (x - 2)**2 + (y + 3)**2

result = optimize.minimize(f, x0=[0, 0])
print("Minimum value:", result.fun)
print("At x, y =", result.x)

---

**4. Using Bounds and Constraints**

You can restrict the variables within bounds or constraints.

Example: Minimize f(x) = (x - 3)^2 with x between 0 and 5

result = optimize.minimize(f, x0=0, bounds=[(0, 5)])
print("Minimum with bounds:", result.fun)
print("At x =", result.x)

---

5. Root Finding with `optimize.root_scalar`

Find a root of a scalar function.

Example: Find root of f(x) = x^3 - 1 between 0 and 2

def f(x):
    return x**3 - 1

root = optimize.root_scalar(f, bracket=[0, 2])
print("Root:", root.root)

---

6. Summary

• SciPy’s optimization tools help find minima, maxima, and roots.

• Supports single and multivariable problems with constraints.

---

Exercise

• Minimize the function f(x) = x^4 - 3x^3 + 2 over the range \[-2, 3].

• Find the root of f(x) = cos(x) - x near x=1.

---

#Python #SciPy #Optimization #RootFinding #ScientificComputing

https://t.iss.one/DataScienceM

Topic: Python SciPy – From Easy to Top: Part 4 of 6: Linear Algebra with SciPy

---

1. Introduction to Linear Algebra in SciPy

• Linear algebra is fundamental in scientific computing, machine learning, and data science.

• SciPy provides advanced linear algebra routines built on top of LAPACK and BLAS libraries.

• The main sub-package is scipy.linalg which extends NumPy’s linear algebra capabilities.

---

2. Basic Matrix Operations

You can create matrices using NumPy arrays:

import numpy as np

A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

---

3. Matrix Addition and Multiplication

# Addition
C = A + B
print("Matrix Addition:\n", C)

# Element-wise Multiplication
D = A * B
print("Element-wise Multiplication:\n", D)

# Matrix Multiplication
E = np.dot(A, B)
print("Matrix Multiplication:\n", E)

---

4. Using `scipy.linalg` for Advanced Operations

Import SciPy linear algebra module:

from scipy import linalg

---

5. Matrix Inverse

Calculate the inverse of a matrix (if invertible):

inv_A = linalg.inv(A)
print("Inverse of A:\n", inv_A)

---

6. Determinant

Calculate the determinant:

det_A = linalg.det(A)
print("Determinant of A:", det_A)

---

7. Eigenvalues and Eigenvectors

Find eigenvalues and eigenvectors:

eigvals, eigvecs = linalg.eig(A)
print("Eigenvalues:\n", eigvals)
print("Eigenvectors:\n", eigvecs)

---

8. Solving Linear Systems

Solve Ax = b where b is a vector:

b = np.array([5, 11])
x = linalg.solve(A, b)
print("Solution x:\n", x)

---

9. Singular Value Decomposition (SVD)

Decompose matrix A into U, Σ, and V^T:

U, s, VT = linalg.svd(A)
print("U matrix:\n", U)
print("Singular values:", s)
print("V^T matrix:\n", VT)

---

10. LU Decomposition

Decompose matrix A into lower and upper triangular matrices:

P, L, U = linalg.lu(A)
print("P matrix:\n", P)
print("L matrix:\n", L)
print("U matrix:\n", U)

---

11. QR Decomposition

Factorize A into Q and R matrices:

Q, R = linalg.qr(A)
print("Q matrix:\n", Q)
print("R matrix:\n", R)

---

12. Norms of Vectors and Matrices

Calculate different norms:

# Vector norm
v = np.array([1, -2, 3])
norm_v = linalg.norm(v)
print("Vector norm:", norm_v)

# Matrix norm (Frobenius norm)
norm_A = linalg.norm(A, 'fro')
print("Matrix Frobenius norm:", norm_A)

---

13. Checking if a Matrix is Positive Definite

Try Cholesky decomposition:

try:
    L = linalg.cholesky(A)
    print("Matrix is positive definite")
except linalg.LinAlgError:
    print("Matrix is not positive definite")

---

14. Summary

• SciPy’s linalg module provides extensive linear algebra tools beyond NumPy.

• Operations include inverse, determinant, eigenvalues, decompositions, and solving linear systems.

• These tools are essential for many scientific and engineering problems.

---

Exercise

• Compute the eigenvalues and eigenvectors of the matrix \[\[4, 2], \[1, 3]].

• Solve the system of equations represented by:

  2x + 3y = 8

  5x + 4y = 13

• Perform SVD on the matrix \[\[1, 0], \[0, -1]] and explain the singular values.

---

#Python #SciPy #LinearAlgebra #SVD #Decomposition #ScientificComputing

https://t.iss.one/DataScienceM

Topic: Python SciPy – From Easy to Top: Part 5 of 6: Working with SciPy Statistics

---

1. Introduction to `scipy.stats`

• The scipy.stats module contains a large number of probability distributions and statistical functions.
• You can perform tasks like descriptive statistics, hypothesis testing, sampling, and fitting distributions.

---

2. Descriptive Statistics

Use these functions to summarize and describe data characteristics:

from scipy import stats
import numpy as np

data = [2, 4, 4, 4, 5, 5, 7, 9]

mean = np.mean(data)
median = np.median(data)
mode = stats.mode(data, keepdims=True)
std_dev = np.std(data)

print("Mean:", mean)
print("Median:", median)
print("Mode:", mode.mode[0])
print("Standard Deviation:", std_dev)

---

3. Probability Distributions

SciPy has built-in continuous and discrete distributions such as normal, binomial, Poisson, etc.

Normal Distribution Example

from scipy.stats import norm

# PDF at x = 0
print("PDF at 0:", norm.pdf(0, loc=0, scale=1))

# CDF at x = 1
print("CDF at 1:", norm.cdf(1, loc=0, scale=1))

# Generate 5 random numbers
samples = norm.rvs(loc=0, scale=1, size=5)
print("Random Samples:", samples)

---

4. Hypothesis Testing

One-sample t-test – test if the mean of a sample is equal to a known value:

sample = [5.1, 5.3, 5.5, 5.7, 5.9]
t_stat, p_val = stats.ttest_1samp(sample, popmean=5.0)

print("T-statistic:", t_stat)
print("P-value:", p_val)

Interpretation: If the p-value is less than 0.05, reject the null hypothesis.

---

5. Two-sample t-test

Test if two samples come from populations with equal means:

group1 = [20, 22, 19, 24, 25]
group2 = [28, 27, 26, 30, 31]

t_stat, p_val = stats.ttest_ind(group1, group2)

print("T-statistic:", t_stat)
print("P-value:", p_val)

---

6. Chi-Square Test for Independence

Use to test independence between two categorical variables:

# Example contingency table
data = [[10, 20], [20, 40]]
chi2, p, dof, expected = stats.chi2_contingency(data)

print("Chi-square statistic:", chi2)
print("P-value:", p)

---

7. Correlation and Covariance

Measure linear relationship between variables:

x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]

corr, _ = stats.pearsonr(x, y)
print("Pearson Correlation Coefficient:", corr)

Covariance:

cov_matrix = np.cov(x, y)
print("Covariance Matrix:\n", cov_matrix)

---

8. Fitting Distributions to Data

You can fit a distribution to real-world data:

data = np.random.normal(loc=50, scale=10, size=1000)
params = norm.fit(data)  # returns mean and std dev

print("Fitted mean:", params[0])
print("Fitted std dev:", params[1])

---

9. Sampling from Distributions

Generate random numbers from different distributions:

# Binomial distribution
samples = stats.binom.rvs(n=10, p=0.5, size=10)
print("Binomial Samples:", samples)

# Poisson distribution
samples = stats.poisson.rvs(mu=3, size=10)
print("Poisson Samples:", samples)

---

10. Summary

• scipy.stats is a powerful tool for statistical analysis.
• You can compute summaries, perform tests, model distributions, and generate random samples.

---

Exercise

• Generate 1000 samples from a normal distribution and compute mean, median, std, and mode.
• Test if a sample has a mean significantly different from 5.
• Fit a normal distribution to your own dataset and plot the histogram with the fitted PDF curve.

---

#Python #SciPy #Statistics #HypothesisTesting #DataAnalysis

https://t.iss.one/DataScienceM

Topic: Python SciPy – From Easy to Top: Part 6 of 6: Signal Processing, Interpolation, and Fourier Transforms

---

1. Introduction

SciPy contains powerful tools for signal processing, interpolation, and Fourier transforms. These are essential in fields like image and audio processing, scientific simulations, and data smoothing.

Main submodules covered in this part:

• scipy.signal – Signal processing
• scipy.fft – Fast Fourier Transform
• scipy.interpolate – Data interpolation and curve fitting

---

### 2. Signal Processing with `scipy.signal`

Filtering a Signal:

Let’s create a noisy sine wave and apply a low-pass filter.

import numpy as np
from scipy import signal
import matplotlib.pyplot as plt

# Create a sample signal with noise
t = np.linspace(0, 1.0, 200)
x = np.sin(2 * np.pi * 5 * t) + 0.5 * np.random.randn(200)

# Apply a Butterworth low-pass filter
b, a = signal.butter(3, 0.2)
filtered = signal.filtfilt(b, a, x)

# Plot original and filtered signals
plt.plot(t, x, label="Noisy Signal")
plt.plot(t, filtered, label="Filtered Signal")
plt.legend()
plt.title("Low-pass Filtering with Butterworth")
plt.show()

---

Find Peaks in a Signal:

peaks, _ = signal.find_peaks(x, height=0)
print("Peak Indices:", peaks)

---

### 3. Fourier Transform with `scipy.fft`

The Fourier Transform breaks a signal into its frequency components.

from scipy.fft import fft, fftfreq

# Number of sample points
N = 600
# Sample spacing
T = 1.0 / 800.0
x = np.linspace(0.0, N*T, N, endpoint=False)
y = np.sin(50.0 * 2.0 * np.pi * x) + 0.5 * np.sin(80.0 * 2.0 * np.pi * x)

yf = fft(y)
xf = fftfreq(N, T)[:N//2]

plt.plot(xf, 2.0/N * np.abs(yf[0:N//2]))
plt.grid()
plt.title("Fourier Transform of Signal")
plt.show()

---

### 4. Interpolation with `scipy.interpolate`

Interpolation estimates unknown values between known data points.

from scipy import interpolate

x = np.linspace(0, 10, 10)
y = np.sin(x)

# Create interpolating function
f = interpolate.interp1d(x, y, kind='cubic')

# Interpolate new values
xnew = np.linspace(0, 10, 100)
ynew = f(xnew)

plt.plot(x, y, 'o', label="Data Points")
plt.plot(xnew, ynew, '-', label="Cubic Interpolation")
plt.legend()
plt.title("Interpolation Example")
plt.show()

---

### 5. 2D Interpolation Example

from scipy.interpolate import griddata

# Known points
points = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
values = np.array([0, 1, 1, 0])

# Interpolation grid
grid_x, grid_y = np.mgrid[0:1:100j, 0:1:100j]
grid_z = griddata(points, values, (grid_x, grid_y), method='cubic')

plt.imshow(grid_z.T, extent=(0,1,0,1), origin='lower')
plt.title("2D Cubic Interpolation")
plt.colorbar()
plt.show()

---

### 6. Summary

• scipy.signal is used for filtering, finding peaks, convolution, etc.
• scipy.fft helps analyze signal frequencies.
• scipy.interpolate estimates unknown values smoothly between data points.

These tools are critical for real-time data analysis, image/audio processing, and engineering applications.

---

Exercise

• Generate a noisy signal and apply both low-pass and high-pass filters.
• Plot the Fourier transform of a composed signal of multiple frequencies.
• Perform cubic interpolation on a dataset with missing values and plot both.

---

#Python #SciPy #SignalProcessing #FFT #Interpolation #ScientificComputing

https://t.iss.one/DataScienceM

Topic: Handling Datasets of All Types – Part 1 of 5: Introduction and Basic Concepts

---

1. What is a Dataset?

• A dataset is a structured collection of data, usually organized in rows and columns, used for analysis or training machine learning models.

---

2. Types of Datasets

• Structured Data: Tables, spreadsheets with rows and columns (e.g., CSV, Excel).

• Unstructured Data: Images, text, audio, video.

• Semi-structured Data: JSON, XML files containing hierarchical data.

---

3. Common Dataset Formats

• CSV (Comma-Separated Values)

• Excel (.xls, .xlsx)

• JSON (JavaScript Object Notation)

• XML (eXtensible Markup Language)

• Images (JPEG, PNG, TIFF)

• Audio (WAV, MP3)

---

4. Loading Datasets in Python

• Use libraries like pandas for structured data:

import pandas as pd
df = pd.read_csv('data.csv')

• Use libraries like json for JSON files:

import json
with open('data.json') as f:
    data = json.load(f)

---

5. Basic Dataset Exploration

• Check shape and size:

print(df.shape)

• Preview data:

print(df.head())

• Check for missing values:

print(df.isnull().sum())

---

6. Summary

• Understanding dataset types is crucial before processing.

• Loading and exploring datasets helps identify cleaning and preprocessing needs.

---

Exercise

• Load a CSV and JSON dataset in Python, print their shapes, and identify missing values.

---

#DataScience #Datasets #DataLoading #Python #DataExploration

https://t.iss.one/DataScienceM

Topic: Handling Datasets of All Types – Part 2 of 5: Data Cleaning and Preprocessing

---

1. Importance of Data Cleaning

• Real-world data is often noisy, incomplete, or inconsistent.

• Cleaning improves data quality and model performance.

---

2. Handling Missing Data

• Detect missing values using isnull() or isna() in pandas.

• Strategies to handle missing data:

* Remove rows or columns with missing values:

df.dropna(inplace=True)

* Impute missing values with mean, median, or mode:

df['column'].fillna(df['column'].mean(), inplace=True)

---

3. Handling Outliers

• Outliers can skew analysis and model results.

• Detect outliers using:

* Boxplots
* Z-score method
* IQR (Interquartile Range)

• Handle by removal or transformation.

---

4. Data Normalization and Scaling

• Many ML models require features to be on a similar scale.

• Common techniques:

* Min-Max Scaling (scales values between 0 and 1)

* Standardization (mean = 0, std = 1)

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
df_scaled = scaler.fit_transform(df[['feature1', 'feature2']])

---

5. Encoding Categorical Variables

• Convert categorical data into numerical:

* Label Encoding: Assigns an integer to each category.

* One-Hot Encoding: Creates binary columns for each category.

pd.get_dummies(df['category_column'])

---

6. Summary

• Data cleaning is essential for reliable modeling.

• Handling missing values, outliers, scaling, and encoding are key preprocessing steps.

---

Exercise

• Load a dataset, identify missing values, and apply mean imputation.

• Detect outliers using IQR and remove them.

• Normalize numeric features using standardization.

---

#DataCleaning #DataPreprocessing #MachineLearning #Python #DataScience

https://t.iss.one/DataScienceM

Topic: Handling Datasets of All Types – Part 2 of 5: Data Cleaning and Preprocessing

---

1. Importance of Data Cleaning

• Real-world data is often noisy, incomplete, or inconsistent.

• Cleaning improves data quality and model performance.

---

2. Handling Missing Data

• Detect missing values using isnull() or isna() in pandas.

• Strategies to handle missing data:

* Remove rows or columns with missing values:

df.dropna(inplace=True)

* Impute missing values with mean, median, or mode:

df['column'].fillna(df['column'].mean(), inplace=True)

---

3. Handling Outliers

• Outliers can skew analysis and model results.

• Detect outliers using:

* Boxplots
* Z-score method
* IQR (Interquartile Range)

• Handle by removal or transformation.

---

4. Data Normalization and Scaling

• Many ML models require features to be on a similar scale.

• Common techniques:

* Min-Max Scaling (scales values between 0 and 1)

* Standardization (mean = 0, std = 1)

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
df_scaled = scaler.fit_transform(df[['feature1', 'feature2']])

---

5. Encoding Categorical Variables

• Convert categorical data into numerical:

* Label Encoding: Assigns an integer to each category.

* One-Hot Encoding: Creates binary columns for each category.

pd.get_dummies(df['category_column'])

---

6. Summary

• Data cleaning is essential for reliable modeling.

• Handling missing values, outliers, scaling, and encoding are key preprocessing steps.

---

Exercise

• Load a dataset, identify missing values, and apply mean imputation.

• Detect outliers using IQR and remove them.

• Normalize numeric features using standardization.

---

#DataCleaning #DataPreprocessing #MachineLearning #Python #DataScience

https://t.iss.one/DataScience4M

Topic: Handling Datasets of All Types – Part 4 of 5: Text Data Processing and Natural Language Processing (NLP)

---

1. Understanding Text Data

• Text data is unstructured and requires preprocessing to convert into numeric form for ML models.

• Common tasks: classification, sentiment analysis, language modeling.

---

2. Text Preprocessing Steps

• Tokenization: Splitting text into words or subwords.

• Lowercasing: Convert all text to lowercase for uniformity.

• Removing Punctuation and Stopwords: Clean unnecessary words.

• Stemming and Lemmatization: Reduce words to their root form.

---

3. Encoding Text Data

• Bag-of-Words (BoW): Represents text as word count vectors.

• TF-IDF (Term Frequency-Inverse Document Frequency): Weighs words based on importance.

• Word Embeddings: Dense vector representations capturing semantic meaning (e.g., Word2Vec, GloVe).

---

4. Loading and Processing Text Data in Python

from sklearn.feature_extraction.text import TfidfVectorizer

texts = ["I love data science.", "Data science is fun."]
vectorizer = TfidfVectorizer(stop_words='english')
X = vectorizer.fit_transform(texts)

---

5. Handling Large Text Datasets

• Use libraries like NLTK, spaCy, and Transformers.

• For deep learning, tokenize using models like BERT or GPT.

---

6. Summary

• Text data needs extensive preprocessing and encoding.

• Choosing the right representation is crucial for model success.

---

Exercise

• Clean a set of sentences by tokenizing and removing stopwords.

• Convert cleaned text into TF-IDF vectors.

---

#NLP #TextProcessing #DataScience #MachineLearning #Python

https://t.iss.one/DataScienceM

Topic: Handling Datasets of All Types – Part 5 of 5: Working with Time Series and Tabular Data

---

1. Understanding Time Series Data

• Time series data is a sequence of data points collected over time intervals.

• Examples: stock prices, weather data, sensor readings.

---

2. Loading and Exploring Time Series Data

import pandas as pd

df = pd.read_csv('time_series.csv', parse_dates=['date'], index_col='date')
print(df.head())

---

3. Key Time Series Concepts

• Trend: Long-term increase or decrease in data.

• Seasonality: Repeating patterns at regular intervals.

• Noise: Random variations.

---

4. Preprocessing Time Series

• Handle missing data using forward/backward fill.

df.fillna(method='ffill', inplace=True)

• Resample data to different frequencies (daily, monthly).

df_resampled = df.resample('M').mean()

---

5. Working with Tabular Data

• Tabular data consists of rows (samples) and columns (features).

• Often requires handling missing values, encoding categorical variables, and scaling features (covered in previous parts).

---

6. Summary

• Time series data requires special preprocessing due to temporal order.

• Tabular data is the most common format, needing cleaning and feature engineering.

---

Exercise

• Load a time series dataset, fill missing values, and resample it monthly.

• For tabular data, encode categorical variables and scale numerical features.

---

#TimeSeries #TabularData #DataScience #MachineLearning #Python

https://t.iss.one/DataScienceM

Topic: 25 Important Questions on Handling Datasets of All Types in Python

---

1. What are the common types of datasets?
Structured, unstructured, and semi-structured.

---

2. How do you load a CSV file in Python?
Using pandas.read_csv() function.

---

3. How to check for missing values in a dataset?
Using df.isnull().sum() in pandas.

---

4. What methods can you use to handle missing data?
Remove rows/columns, mean/median/mode imputation, interpolation.

---

5. How to detect outliers in data?
Using boxplots, z-score, or interquartile range (IQR) methods.

---

6. What is data normalization?
Scaling data to a specific range, often \[0,1].

---

7. What is data standardization?
Rescaling data to have zero mean and unit variance.

---

8. How to encode categorical variables?
Label encoding or one-hot encoding.

---

9. What libraries help with image data processing in Python?
OpenCV, Pillow, scikit-image.

---

10. How do you load and preprocess images for ML models?
Resize, normalize pixel values, data augmentation.

---

11. How can audio data be loaded in Python?
Using libraries like librosa or scipy.io.wavfile.

---

12. What are MFCCs in audio processing?
Mel-frequency cepstral coefficients – features extracted from audio signals.

---

13. How do you preprocess text data?
Tokenization, removing stopwords, stemming, lemmatization.

---

14. What is TF-IDF?
A technique to weigh words based on frequency and importance.

---

15. How do you handle variable-length sequences in text or time series?
Padding sequences or using packed sequences.

---

16. How to handle time series missing data?
Forward fill, backward fill, interpolation.

---

17. What is data augmentation?
Creating new data samples by transforming existing data.

---

18. How to split datasets into training and testing sets?
Using train_test_split from scikit-learn.

---

19. What is batch processing in ML?
Processing data in small batches during training for efficiency.

---

20. How to save and load datasets efficiently?
Using formats like HDF5, pickle, or TFRecord.

---

21. What is feature scaling and why is it important?
Adjusting features to a common scale to improve model training.

---

22. How to detect and remove duplicate data?
Using df.duplicated() and df.drop_duplicates().

---

23. What is one-hot encoding and when to use it?
Converting categorical variables to binary vectors, used for nominal categories.

---

24. How to handle imbalanced datasets?
Techniques like oversampling, undersampling, or synthetic data generation (SMOTE).

---

25. How to visualize datasets in Python?
Using matplotlib, seaborn, or plotly for charts and graphs.

---

#DataScience #DataHandling #Python #MachineLearning #DataPreprocessing

https://t.iss.one/DataScience4M

Topic: Python PySpark Data Sheet – Part 1 of 3: Introduction, Setup, and Core Concepts

---

### 1. What is PySpark?

PySpark is the Python API for Apache Spark, a powerful distributed computing engine for big data processing.

PySpark allows you to leverage the full power of Apache Spark using Python, making it easier to:

• Handle massive datasets
• Perform distributed computing
• Run parallel data transformations

---

### 2. PySpark Ecosystem Components

• Spark SQL – Structured data queries with DataFrame and SQL APIs
• Spark Core – Fundamental engine for task scheduling and memory management
• Spark Streaming – Real-time data processing
• MLlib – Machine learning at scale
• GraphX – Graph computation

---

### 3. Why PySpark over Pandas?

| Feature | Pandas | PySpark |
| -------------- | --------------------- | ----------------------- |
| Scale | Single machine | Distributed (Cluster) |
| Speed | Slower for large data | Optimized execution |
| Language | Python | Python on JVM via Py4J |
| Learning Curve | Easier | Medium (Big Data focus) |

---

### 4. PySpark Setup in Local Machine

#### Install PySpark via pip:

pip install pyspark

#### Start PySpark Shell:

pyspark

#### Sample Code to Initialize SparkSession:

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("MyApp") \
    .getOrCreate()

---

### 5. RDD vs DataFrame

| Feature | RDD | DataFrame |
| ------------ | ----------------------- | ------------------------------ |
| Type | Low-level API (objects) | High-level API (structured) |
| Optimization | Manual | Catalyst Optimizer (automatic) |
| Usage | Complex transformations | SQL-like operations |

---

### 6. Creating DataFrames

#### From Python List:

data = [("Alice", 25), ("Bob", 30)]
df = spark.createDataFrame(data, ["Name", "Age"])
df.show()

#### From CSV File:

df = spark.read.csv("file.csv", header=True, inferSchema=True)
df.show()

---

### 7. Inspecting DataFrames

df.printSchema()     # Schema info  
df.columns           # List column names  
df.describe().show() # Summary stats  
df.head(5)           # First 5 rows  

---

### 8. Basic Transformations

df.select("Name").show()
df.filter(df["Age"] > 25).show()
df.withColumn("AgePlus10", df["Age"] + 10).show()
df.drop("Age").show()

---

### 9. Working with SQL

df.createOrReplaceTempView("people")
spark.sql("SELECT * FROM people WHERE Age > 25").show()

---

### 10. Writing Data

df.write.csv("output.csv", header=True)
df.write.parquet("output_parquet/")

---

### 11. Summary of Concepts Covered

• Spark architecture & PySpark setup
• Core components of PySpark
• Differences between RDD and DataFrames
• How to create, inspect, and manipulate DataFrames
• SQL support in Spark
• Reading/writing to/from storage

---

### Exercise

1. Load a sample CSV file and display the schema
2. Add a new column with a calculated value
3. Filter the rows based on a condition
4. Save the result as a new CSV or Parquet file

---

#Python #PySpark #BigData #ApacheSpark #DataEngineering #ETL

https://t.iss.one/DataScienceM

Topic: Python Matplotlib – From Easy to Top: Part 1 of 6: Introduction and Basic Plotting

---

### 1. What is Matplotlib?

• Matplotlib is the most widely used Python library for data visualization.

• It provides an object-oriented API for embedding plots into applications and supports a wide variety of graphs: line charts, bar charts, scatter plots, histograms, etc.

---

### 2. Installing and Importing Matplotlib

Install Matplotlib if you haven't:

pip install matplotlib

Import the main module and pyplot interface:

import matplotlib.pyplot as plt
import numpy as np

---

### 3. Plotting a Basic Line Chart

x = [1, 2, 3, 4, 5]
y = [2, 3, 5, 7, 11]

plt.plot(x, y)
plt.title("Simple Line Plot")
plt.xlabel("X Axis")
plt.ylabel("Y Axis")
plt.grid(True)
plt.show()

---

### 4. Customizing Line Style, Color, and Markers

plt.plot(x, y, color='green', linestyle='--', marker='o', label='Data')
plt.title("Styled Line Plot")
plt.xlabel("X Axis")
plt.ylabel("Y Axis")
plt.legend()
plt.show()

---

### 5. Adding Multiple Lines to a Plot

x = np.linspace(0, 10, 100)
y1 = np.sin(x)
y2 = np.cos(x)

plt.plot(x, y1, label="sin(x)", color='blue')
plt.plot(x, y2, label="cos(x)", color='red')
plt.title("Multiple Line Plot")
plt.xlabel("X Axis")
plt.ylabel("Y Axis")
plt.legend()
plt.grid(True)
plt.show()

---

### 6. Scatter Plot

Used to show relationships between two variables.

x = np.random.rand(100)
y = np.random.rand(100)

plt.scatter(x, y, color='purple', alpha=0.6)
plt.title("Scatter Plot")
plt.xlabel("X Axis")
plt.ylabel("Y Axis")
plt.show()

---

### 7. Bar Chart

categories = ['A', 'B', 'C', 'D']
values = [4, 7, 2, 5]

plt.bar(categories, values, color='skyblue')
plt.title("Bar Chart Example")
plt.xlabel("Category")
plt.ylabel("Value")
plt.show()

---

### 8. Histogram

data = np.random.randn(1000)

plt.hist(data, bins=30, color='orange', edgecolor='black')
plt.title("Histogram")
plt.xlabel("Value")
plt.ylabel("Frequency")
plt.show()

---

### 9. Saving the Plot to a File

plt.plot([1, 2, 3], [4, 5, 6])
plt.savefig("plot.png")

---

### 10. Summary

• matplotlib.pyplot is the key module for creating all kinds of plots.
• You can customize styles, add labels, titles, and legends.
• Understanding basic plots is the foundation for creating advanced visualizations.

---

Exercise

• Plot y = x^2 and y = x^3 on the same figure.
• Create a scatter plot of 100 random points.
• Create and save a histogram from a normal distribution sample of 500 points.

---

#Python #Matplotlib #DataVisualization #Plots #Charts

https://t.iss.one/DataScienceM

Topic: Python Matplotlib – From Easy to Top: Part 2 of 6: Subplots, Figures, and Layout Management

---

### 1. Introduction to Figures and Axes

• In Matplotlib, a Figure is the entire image or window on which everything is drawn.
• An Axes is a part of the figure where data is plotted — it contains titles, labels, ticks, lines, etc.

Basic hierarchy:

* Figure ➝ contains one or more Axes
* Axes ➝ the area where the data is actually plotted
* Axis ➝ x-axis and y-axis inside an Axes

import matplotlib.pyplot as plt
import numpy as np

---

### 2. Creating Multiple Subplots using `plt.subplot()`

x = np.linspace(0, 2*np.pi, 100)
y1 = np.sin(x)
y2 = np.cos(x)

plt.subplot(2, 1, 1)
plt.plot(x, y1, label="sin(x)")
plt.title("First Subplot")

plt.subplot(2, 1, 2)
plt.plot(x, y2, label="cos(x)", color='green')
plt.title("Second Subplot")

plt.tight_layout()
plt.show()

Explanation:

* subplot(2, 1, 1) means 2 rows, 1 column, this is the first plot.
* tight_layout() prevents overlap between plots.

---

### 3. Creating Subplots with `plt.subplots()` (Recommended)

fig, axs = plt.subplots(2, 2, figsize=(8, 6))

x = np.linspace(0, 10, 100)

axs[0, 0].plot(x, np.sin(x))
axs[0, 0].set_title("sin(x)")

axs[0, 1].plot(x, np.cos(x))
axs[0, 1].set_title("cos(x)")

axs[1, 0].plot(x, np.tan(x))
axs[1, 0].set_title("tan(x)")
axs[1, 0].set_ylim(-10, 10)

axs[1, 1].plot(x, np.exp(-x))
axs[1, 1].set_title("exp(-x)")

plt.tight_layout()
plt.show()

---

### 4. Sharing Axes Between Subplots

fig, axs = plt.subplots(1, 2, sharey=True)

x = np.linspace(0, 10, 100)

axs[0].plot(x, np.sin(x))
axs[0].set_title("sin(x)")

axs[1].plot(x, np.cos(x), color='red')
axs[1].set_title("cos(x)")

plt.show()

---

### 5. Adjusting Spacing with `subplots_adjust()`

fig, axs = plt.subplots(2, 2)

fig.subplots_adjust(hspace=0.4, wspace=0.3)

---

### 6. Nested Plots Using `inset_axes`

You can add a small plot inside another:

from mpl_toolkits.axes_grid1.inset_locator import inset_axes

fig, ax = plt.subplots()
x = np.linspace(0, 10, 100)
y = np.sin(x)

ax.plot(x, y)
ax.set_title("Main Plot")

inset_ax = inset_axes(ax, width="30%", height="30%", loc=1)
inset_ax.plot(x, np.cos(x), color='orange')
inset_ax.set_title("Inset", fontsize=8)

plt.show()

---

### 7. Advanced Layout: Gridspec

import matplotlib.gridspec as gridspec

fig = plt.figure(figsize=(8, 6))
gs = gridspec.GridSpec(3, 3)

ax1 = fig.add_subplot(gs[0, :])
ax2 = fig.add_subplot(gs[1, :-1])
ax3 = fig.add_subplot(gs[1:, -1])
ax4 = fig.add_subplot(gs[2, 0])
ax5 = fig.add_subplot(gs[2, 1])

ax1.set_title("Top")
ax2.set_title("Left")
ax3.set_title("Right")
ax4.set_title("Bottom Left")
ax5.set_title("Bottom Center")

plt.tight_layout()
plt.show()

---

### 8. Summary

• Use subplot() for quick layouts and subplots() for flexibility.
• Share axes to align multiple plots.
• Use inset_axes and gridspec for custom and complex layouts.
• Always use tight_layout() or subplots_adjust() to clean up spacing.

---

### Exercise

• Create a 2x2 grid of subplots showing different trigonometric functions.
• Add an inset plot inside a sine wave chart.
• Use Gridspec to create an asymmetric layout with at least 5 different plots.

---

#Python #Matplotlib #Subplots #DataVisualization #Gridspec #LayoutManagement

https://t.iss.one/DataScienceM

Topic: Python Matplotlib – From Easy to Top: Part 3 of 6: Plot Customization and Styling

---

### 1. Why Customize Plots?

• Customization improves readability and presentation.
• You can control everything from fonts and colors to axis ticks and legend placement.

---

### 2. Customizing Titles, Labels, and Ticks

import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(0, 10, 100)
y = np.sin(x)

plt.plot(x, y)
plt.title("Sine Wave", fontsize=16, color='navy')
plt.xlabel("Time (s)", fontsize=12)
plt.ylabel("Amplitude", fontsize=12)
plt.xticks(np.arange(0, 11, 1))
plt.yticks(np.linspace(-1, 1, 5))
plt.grid(True)
plt.show()

---

### 3. Changing Line Styles and Markers

plt.plot(x, y, color='red', linestyle='--', linewidth=2, marker='o', markersize=5, label='sin(x)')
plt.title("Styled Sine Curve")
plt.legend()
plt.grid(True)
plt.show()

Common styles:

• Line styles: '-', '--', ':', '-.'
• Markers: 'o', '^', 's', '*', 'D', etc.
• Colors: 'r', 'g', 'b', 'c', 'm', 'y', 'k', etc.

---

### 4. Adding Legends

plt.plot(x, np.sin(x), label="Sine")
plt.plot(x, np.cos(x), label="Cosine")
plt.legend(loc='upper right', fontsize=10)
plt.title("Legend Example")
plt.show()

---

### 5. Using Annotations

Annotations help highlight specific points:

plt.plot(x, y)
plt.annotate('Peak', xy=(np.pi/2, 1), xytext=(2, 1.2),
             arrowprops=dict(facecolor='black', shrink=0.05))
plt.title("Annotated Peak")
plt.show()

---

### 6. Customizing Axes Appearance

fig, ax = plt.subplots()
ax.plot(x, y)

# Remove top and right border
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)

# Customize axis colors and widths
ax.spines['left'].set_color('blue')
ax.spines['left'].set_linewidth(2)

plt.title("Customized Axes")
plt.show()

---

### 7. Setting Plot Limits

plt.plot(x, y)
plt.xlim(0, 10)
plt.ylim(-1.5, 1.5)
plt.title("Limit Axes")
plt.show()

---

### 8. Using Style Sheets

Matplotlib has built-in style sheets for quick beautification.

plt.style.use('ggplot')

plt.plot(x, np.sin(x))
plt.title("ggplot Style")
plt.show()

Popular styles: seaborn, fivethirtyeight, bmh, dark_background, etc.

---

### 9. Creating Grids and Minor Ticks

plt.plot(x, y)
plt.grid(True, which='both', linestyle='--', linewidth=0.5)
plt.minorticks_on()
plt.title("Grid with Minor Ticks")
plt.show()

---

### 10. Summary

• Customize everything: lines, axes, colors, labels, and grid.
• Use legends and annotations for clarity.
• Apply styles and themes for professional looks.
• Small changes improve the quality of your plots significantly.

---

### Exercise

• Plot sin(x) with red dashed lines and circle markers.
• Add a title, custom x/y labels, and set axis ranges manually.
• Apply the 'seaborn-darkgrid' style and highlight the peak with an annotation.

---

#Python #Matplotlib #Customization #DataVisualization #PlotStyling

https://t.iss.one/DataScienceM

Topic: Python PySpark Data Sheet – Part 2 of 3: DataFrame Transformations, Joins, and Group Operations

---

### 1. Column Operations

PySpark supports various column-wise operations using expressions.

#### Select Specific Columns:

df.select("Name", "Age").show()

#### Create/Modify Column:

from pyspark.sql.functions import col

df.withColumn("AgePlus5", col("Age") + 5).show()

#### Rename a Column:

df.withColumnRenamed("Age", "UserAge").show()

#### Drop Column:

df.drop("Age").show()

---

### 2. Filtering and Conditional Logic

#### Filter Rows:

df.filter(col("Age") > 25).show()

#### Multiple Conditions:

df.filter((col("Age") > 25) & (col("Name") != "Alice")).show()

#### Using `when` for Conditional Columns:

from pyspark.sql.functions import when

df.withColumn("Category", when(col("Age") < 30, "Young").otherwise("Adult")).show()

---

### 3. Aggregations and Grouping

#### GroupBy + Aggregations:

df.groupBy("Department").count().show()
df.groupBy("Department").agg({"Salary": "avg"}).show()

#### Using Aggregate Functions:

from pyspark.sql.functions import avg, max, min, count

df.groupBy("Department").agg(
    avg("Salary").alias("AvgSalary"),
    max("Salary").alias("MaxSalary")
).show()

---

### 4. Sorting and Ordering

#### Sort by One or More Columns:

df.orderBy("Age").show()
df.orderBy(col("Salary").desc()).show()

---

### 5. Dropping Duplicates & Handling Missing Data

#### Drop Duplicates:

df.dropDuplicates(["Name", "Age"]).show()

#### Drop Rows with Nulls:

df.dropna().show()

#### Fill Null Values:

df.fillna({"Salary": 0}).show()

---

### 6. Joins in PySpark

PySpark supports various join types like SQL.

#### Types of Joins:

• inner
• left
• right
• outer
• left_semi
• left_anti

#### Example – Inner Join:

df1.join(df2, on="id", how="inner").show()

#### Left Join Example:

df1.join(df2, on="id", how="left").show()

---

### 7. Working with Dates and Timestamps

from pyspark.sql.functions import current_date, current_timestamp

df.withColumn("today", current_date()).show()
df.withColumn("now", current_timestamp()).show()

#### Date Formatting:

from pyspark.sql.functions import date_format

df.withColumn("formatted", date_format(col("Date"), "yyyy-MM-dd")).show()

---

### 8. Window Functions (Advanced Aggregations)

Used for operations like ranking, cumulative sum, and moving average.

from pyspark.sql.window import Window
from pyspark.sql.functions import row_number

window_spec = Window.partitionBy("Department").orderBy("Salary")
df.withColumn("rank", row_number().over(window_spec)).show()

---

### 9. Caching and Persistence

Use caching for performance when reusing data:

df.cache()
df.show()

Or use:

df.persist()

---

### 10. Summary of Concepts Covered

• Column transformations and renaming
• Filtering and conditional logic
• Grouping, aggregating, and sorting
• Handling nulls and duplicates
• All types of joins
• Working with dates and window functions
• Caching for performance

---

### Exercise

1. Load two CSV datasets and perform different types of joins
2. Add a new column with a custom label based on a condition
3. Aggregate salary data by department and show top-paid employees per department using window functions
4. Practice caching and observe performance

---

#Python #PySpark #DataEngineering #BigData #ETL #ApacheSpark

https://t.iss.one/DataScienceM

Topic: Python Matplotlib – From Easy to Top: Part 4 of 6: Advanced Charts – Histograms, Pie, Box, Area, and Error Bars

---

### 1. Histogram: Visualizing Data Distribution

Histograms show frequency distribution of numerical data.

import matplotlib.pyplot as plt
import numpy as np

data = np.random.randn(1000)

plt.hist(data, bins=30, color='skyblue', edgecolor='black')
plt.title("Normal Distribution Histogram")
plt.xlabel("Value")
plt.ylabel("Frequency")
plt.grid(True)
plt.show()

Customizations:

• bins=30 – controls granularity
• density=True – normalize the histogram
• alpha=0.7 – transparency

---

### 2. Pie Chart: Showing Proportions

labels = ['Python', 'JavaScript', 'C++', 'Java']
sizes = [45, 30, 15, 10]
colors = ['gold', 'lightgreen', 'lightcoral', 'lightskyblue']
explode = (0.1, 0, 0, 0)  # explode the 1st slice

plt.pie(sizes, labels=labels, colors=colors, autopct='%1.1f%%',
        startangle=140, explode=explode, shadow=True)
plt.title("Programming Language Popularity")
plt.axis('equal')  # Equal aspect ratio ensures pie is circular
plt.show()

---

### 3. Box Plot: Summarizing Distribution Stats

Box plots show min, Q1, median, Q3, max, and outliers.

data = [np.random.normal(0, std, 100) for std in range(1, 4)]

plt.boxplot(data, patch_artist=True, labels=['std=1', 'std=2', 'std=3'])
plt.title("Box Plot Example")
plt.grid(True)
plt.show()

Tip: Use vert=False to make a horizontal boxplot.

---

### 4. Area Chart: Cumulative Trends

x = np.arange(1, 6)
y1 = np.array([1, 3, 4, 5, 7])
y2 = np.array([1, 2, 4, 6, 8])

plt.fill_between(x, y1, color="skyblue", alpha=0.5, label="Y1")
plt.fill_between(x, y2, color="orange", alpha=0.5, label="Y2")
plt.title("Area Chart")
plt.xlabel("X Axis")
plt.ylabel("Y Axis")
plt.legend()
plt.show()

---

### 5. Error Bar Plot: Showing Uncertainty

x = np.arange(0.1, 4, 0.5)
y = np.exp(-x)
error = 0.1 + 0.2 * x

plt.errorbar(x, y, yerr=error, fmt='-o', color='teal', ecolor='red', capsize=5)
plt.title("Error Bar Plot")
plt.xlabel("X Axis")
plt.ylabel("Y Axis")
plt.grid(True)
plt.show()

---

### 6. Horizontal Bar Chart

langs = ['Python', 'Java', 'C++', 'JavaScript']
popularity = [50, 40, 30, 45]

plt.barh(langs, popularity, color='plum')
plt.title("Programming Language Popularity")
plt.xlabel("Popularity")
plt.show()

---

### 7. Stacked Bar Chart

labels = ['2019', '2020', '2021']
men = [20, 35, 30]
women = [25, 32, 34]

x = np.arange(len(labels))
width = 0.5

plt.bar(x, men, width, label='Men')
plt.bar(x, women, width, bottom=men, label='Women')

plt.ylabel('Scores')
plt.title('Scores by Year and Gender')
plt.xticks(x, labels)
plt.legend()
plt.show()

---

### 8. Summary

• Histograms show frequency distribution
• Pie charts are good for proportions
• Box plots summarize spread and outliers
• Area charts visualize trends over time
• Error bars indicate uncertainty in measurements
• Stacked and horizontal bars enhance categorical data clarity

---

### Exercise

• Create a pie chart showing budget allocation of 5 departments.
• Plot 3 histograms on the same figure with different distributions.
• Build a stacked bar chart for monthly expenses across 3 categories.
• Add error bars to a decaying function and annotate the max point.

---

#Python #Matplotlib #DataVisualization #AdvancedCharts #Histograms #PieCharts #BoxPlots

https://t.iss.one/DataScienceM

Topic: Python PySpark Data Sheet – Part 3 of 3: Advanced Operations, MLlib, and Deployment

---

### 1. Working with UDFs (User Defined Functions)

UDFs allow custom Python functions to be used in PySpark transformations.

#### Define and Use a UDF:

from pyspark.sql.functions import udf
from pyspark.sql.types import StringType

def label_age(age):
    return "Senior" if age > 50 else "Adult"

label_udf = udf(label_age, StringType())

df.withColumn("AgeGroup", label_udf(df["Age"])).show()

> ⚠️ Note: UDFs are less optimized than built-in functions. Use built-ins when possible.

---

### 2. Working with JSON and Parquet Files

#### Read JSON File:

df_json = spark.read.json("data.json")
df_json.show()

#### Read & Write Parquet File:

df_parquet = spark.read.parquet("data.parquet")
df_parquet.write.parquet("output_folder/")

---

### 3. Using PySpark MLlib (Machine Learning Library)

MLlib is Spark's scalable ML library with tools for classification, regression, clustering, and more.

---

#### Steps in a Typical ML Pipeline:

• Load and prepare data
• Feature engineering
• Model training
• Evaluation
• Prediction

---

### 4. Example: Logistic Regression in PySpark

#### Step 1: Prepare Data

from pyspark.ml.feature import VectorAssembler
from pyspark.ml.classification import LogisticRegression

# Sample DataFrame
data = spark.createDataFrame([
    (1.0, 2.0, 3.0, 1.0),
    (2.0, 3.0, 4.0, 0.0),
    (1.5, 2.5, 3.5, 1.0)
], ["f1", "f2", "f3", "label"])

# Combine features into a single vector
vec = VectorAssembler(inputCols=["f1", "f2", "f3"], outputCol="features")
data = vec.transform(data)

#### Step 2: Train Model

lr = LogisticRegression(featuresCol="features", labelCol="label")
model = lr.fit(data)

#### Step 3: Make Predictions

predictions = model.transform(data)
predictions.select("features", "label", "prediction").show()

---

### 5. Model Evaluation

from pyspark.ml.evaluation import BinaryClassificationEvaluator

evaluator = BinaryClassificationEvaluator()
print("Accuracy:", evaluator.evaluate(predictions))

---

### 6. Save and Load Models

# Save
model.save("models/logistic_model")

# Load
from pyspark.ml.classification import LogisticRegressionModel
loaded_model = LogisticRegressionModel.load("models/logistic_model")

---

### 7. PySpark with Pandas API on Spark

For small-medium data (pandas-compatible), use pyspark.pandas:

import pyspark.pandas as ps

pdf = ps.read_csv("data.csv")
pdf.head()

> Works like Pandas, but with Spark backend.

---

### 8. Scheduling & Cluster Deployment

PySpark can run:

• Locally
• On YARN (Hadoop)
• Mesos
• Kubernetes
• In Databricks, AWS EMR, Google Cloud Dataproc

Use spark-submit for production scripts:

spark-submit my_script.py

---

### 9. Tuning and Optimization Tips

• Cache reused DataFrames
• Use built-in functions instead of UDFs
• Repartition if data is skewed
• Avoid using collect() on large datasets

---

### 10. Summary of Part 3

• Custom logic with UDFs
• Working with JSON, Parquet, and other formats
• Machine Learning with MLlib (Logistic Regression)
• Model evaluation and saving
• Integration with Pandas
• Deployment and optimization techniques

---

### Exercise

1. Load a dataset and train a logistic regression model
2. Add feature engineering using VectorAssembler
3. Save and reload the model
4. Use UDFs to label predictions as “Yes/No”
5. Deploy your pipeline using spark-submit

---

#Python #PySpark #MLlib #BigData #MachineLearning #ETL #ApacheSpark

https://t.iss.one/DataScienceM

Topic: Python Matplotlib – From Easy to Top: Part 5 of 6: Images, Heatmaps, and Colorbars

---

### 1. Introduction

Matplotlib can handle images, heatmaps, and color mapping effectively, making it a great tool for visualizing:

• Image data (grayscale or color)
• Matrix-like data with heatmaps
• Any data that needs a gradient of colors

---

### 2. Displaying Images with `imshow()`

import matplotlib.pyplot as plt
import numpy as np

# Create a random grayscale image
img = np.random.rand(10, 10)

plt.imshow(img, cmap='gray')
plt.title("Grayscale Image")
plt.colorbar()
plt.show()

Key parameters:

• cmap – color map (gray, hot, viridis, coolwarm, etc.)
• interpolation – for smoothing pixelation (nearest, bilinear, bicubic)

---

### 3. Displaying Color Images

import matplotlib.image as mpimg

img = mpimg.imread('example.png')  # image must be in your directory
plt.imshow(img)
plt.title("Color Image")
plt.axis('off')  # Hide axes
plt.show()

Note: Image should be PNG or JPG. For real projects, use PIL or OpenCV for more control.

---

### 4. Creating a Heatmap from a 2D Matrix

matrix = np.random.rand(6, 6)

plt.imshow(matrix, cmap='viridis', interpolation='nearest')
plt.title("Heatmap Example")
plt.colorbar(label="Intensity")
plt.xticks(range(6), ['A', 'B', 'C', 'D', 'E', 'F'])
plt.yticks(range(6), ['P', 'Q', 'R', 'S', 'T', 'U'])
plt.show()

---

### 5. Customizing Color Maps

You can reverse or customize color maps:

plt.imshow(matrix, cmap='coolwarm_r')  # Reversed coolwarm

You can also create custom color ranges using vmin and vmax:

plt.imshow(matrix, cmap='hot', vmin=0.2, vmax=0.8)

---

### 6. Using `matshow()` for Matrix-Like Data

matshow() is optimized for visualizing 2D arrays:

plt.matshow(matrix)
plt.title("Matrix View with matshow()")
plt.colorbar()
plt.show()

---

### 7. Annotating Heatmaps

fig, ax = plt.subplots()
cax = ax.imshow(matrix, cmap='plasma')

# Add text annotations
for i in range(matrix.shape[0]):
    for j in range(matrix.shape[1]):
        ax.text(j, i, f'{matrix[i, j]:.2f}', ha='center', va='center', color='white')

plt.title("Annotated Heatmap")
plt.colorbar(cax)
plt.show()

---

### 8. Displaying Multiple Images in Subplots

fig, axs = plt.subplots(1, 2, figsize=(10, 4))

axs[0].imshow(matrix, cmap='Blues')
axs[0].set_title("Blues")

axs[1].imshow(matrix, cmap='Greens')
axs[1].set_title("Greens")

plt.tight_layout()
plt.show()

---

### 9. Saving Heatmaps and Figures

plt.imshow(matrix, cmap='magma')
plt.title("Save This Heatmap")
plt.colorbar()
plt.savefig("heatmap.png", dpi=300)
plt.close()

---

### 10. Summary

• imshow() and matshow() visualize 2D data or images
• Heatmaps are great for matrix or correlation data
• Use colorbars and annotations to add context
• Customize colormaps with cmap, vmin, vmax
• Save your visualizations easily using savefig()

---

### Exercise

• Load a grayscale image using NumPy and display it.
• Create a 10×10 heatmap with annotations.
• Display 3 subplots of the same matrix using 3 different colormaps.
• Save one of the heatmaps with high resolution.

---

#Python #Matplotlib #Heatmaps #DataVisualization #Images #ColorMapping

https://t.iss.one/DataScienceM

Topic: Python Matplotlib – From Easy to Top: Part 6 of 6: 3D Plotting, Animation, and Interactive Visuals

---

### 1. Introduction

Matplotlib supports advanced visualizations including:

• 3D plots using mpl_toolkits.mplot3d
• Animations with FuncAnimation
• Interactive plots using widgets and event handling

---

### 2. Creating 3D Plots

You need to import the 3D toolkit:

from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
import numpy as np

---

### 3. 3D Line Plot

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

z = np.linspace(0, 15, 100)
x = np.sin(z)
y = np.cos(z)

ax.plot3D(x, y, z, 'purple')
ax.set_title("3D Line Plot")
plt.show()

---

### 4. 3D Surface Plot

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

X = np.linspace(-5, 5, 50)
Y = np.linspace(-5, 5, 50)
X, Y = np.meshgrid(X, Y)
Z = np.sin(np.sqrt(X**2 + Y**2))

surf = ax.plot_surface(X, Y, Z, cmap='viridis')
fig.colorbar(surf)

ax.set_title("3D Surface Plot")
plt.show()

---

### 5. 3D Scatter Plot

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

x = np.random.rand(100)
y = np.random.rand(100)
z = np.random.rand(100)

ax.scatter(x, y, z, c=z, cmap='plasma')
ax.set_title("3D Scatter Plot")
plt.show()

---

### 6. Creating Animations

Use FuncAnimation for animated plots.

import matplotlib.animation as animation

fig, ax = plt.subplots()
x = np.linspace(0, 2*np.pi, 128)
line, = ax.plot(x, np.sin(x))

def update(frame):
    line.set_ydata(np.sin(x + frame / 10))
    return line,

ani = animation.FuncAnimation(fig, update, frames=100, interval=50)
plt.title("Sine Wave Animation")
plt.show()

---

### 7. Save Animation as a File

ani.save("sine_wave.gif", writer='pillow')

Make sure to install pillow using:

pip install pillow

---

### 8. Adding Interactivity with Widgets

import matplotlib.widgets as widgets

fig, ax = plt.subplots()
plt.subplots_adjust(left=0.1, bottom=0.25)

x = np.linspace(0, 2*np.pi, 100)
freq = 1
line, = ax.plot(x, np.sin(freq * x))

ax_slider = plt.axes([0.25, 0.1, 0.65, 0.03])
slider = widgets.Slider(ax_slider, 'Frequency', 0.1, 5.0, valinit=freq)

def update(val):
    line.set_ydata(np.sin(slider.val * x))
    fig.canvas.draw_idle()

slider.on_changed(update)
plt.title("Interactive Sine Wave")
plt.show()

---

### 9. Mouse Interaction with Events

def onclick(event):
    print(f'You clicked at x={event.xdata:.2f}, y={event.ydata:.2f}')

fig, ax = plt.subplots()
ax.plot([1, 2, 3], [4, 5, 6])
fig.canvas.mpl_connect('button_press_event', onclick)
plt.title("Click to Print Coordinates")
plt.show()

---

### 10. Summary

• 3D plots are ideal for visualizing spatial data and surfaces
• Animations help convey dynamic changes in data
• Widgets and events add interactivity for data exploration
• Mastering these tools enables the creation of interactive dashboards and visual storytelling

---

### Exercise

• Plot a 3D surface of z = cos(sqrt(x² + y²)).
• Create a slider to change frequency of a sine wave in real-time.
• Animate a circle that rotates along time.
• Build a 3D scatter plot of 3 correlated variables.

---

#Python #Matplotlib #3DPlots #Animations #InteractiveVisuals #DataVisualization

https://t.iss.one/DataScienceM