Topic: Python SciPy – From Easy to Top: Part 5 of 6: Working with SciPy Statistics

---

1. Introduction to `scipy.stats`

• The scipy.stats module contains a large number of probability distributions and statistical functions.
• You can perform tasks like descriptive statistics, hypothesis testing, sampling, and fitting distributions.

---

2. Descriptive Statistics

Use these functions to summarize and describe data characteristics:

from scipy import stats
import numpy as np

data = [2, 4, 4, 4, 5, 5, 7, 9]

mean = np.mean(data)
median = np.median(data)
mode = stats.mode(data, keepdims=True)
std_dev = np.std(data)

print("Mean:", mean)
print("Median:", median)
print("Mode:", mode.mode[0])
print("Standard Deviation:", std_dev)

---

3. Probability Distributions

SciPy has built-in continuous and discrete distributions such as normal, binomial, Poisson, etc.

Normal Distribution Example

from scipy.stats import norm

# PDF at x = 0
print("PDF at 0:", norm.pdf(0, loc=0, scale=1))

# CDF at x = 1
print("CDF at 1:", norm.cdf(1, loc=0, scale=1))

# Generate 5 random numbers
samples = norm.rvs(loc=0, scale=1, size=5)
print("Random Samples:", samples)

---

4. Hypothesis Testing

One-sample t-test – test if the mean of a sample is equal to a known value:

sample = [5.1, 5.3, 5.5, 5.7, 5.9]
t_stat, p_val = stats.ttest_1samp(sample, popmean=5.0)

print("T-statistic:", t_stat)
print("P-value:", p_val)

Interpretation: If the p-value is less than 0.05, reject the null hypothesis.

---

5. Two-sample t-test

Test if two samples come from populations with equal means:

group1 = [20, 22, 19, 24, 25]
group2 = [28, 27, 26, 30, 31]

t_stat, p_val = stats.ttest_ind(group1, group2)

print("T-statistic:", t_stat)
print("P-value:", p_val)

---

6. Chi-Square Test for Independence

Use to test independence between two categorical variables:

# Example contingency table
data = [[10, 20], [20, 40]]
chi2, p, dof, expected = stats.chi2_contingency(data)

print("Chi-square statistic:", chi2)
print("P-value:", p)

---

7. Correlation and Covariance

Measure linear relationship between variables:

x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]

corr, _ = stats.pearsonr(x, y)
print("Pearson Correlation Coefficient:", corr)

Covariance:

cov_matrix = np.cov(x, y)
print("Covariance Matrix:\n", cov_matrix)

---

8. Fitting Distributions to Data

You can fit a distribution to real-world data:

data = np.random.normal(loc=50, scale=10, size=1000)
params = norm.fit(data)  # returns mean and std dev

print("Fitted mean:", params[0])
print("Fitted std dev:", params[1])

---

9. Sampling from Distributions

Generate random numbers from different distributions:

# Binomial distribution
samples = stats.binom.rvs(n=10, p=0.5, size=10)
print("Binomial Samples:", samples)

# Poisson distribution
samples = stats.poisson.rvs(mu=3, size=10)
print("Poisson Samples:", samples)

---

10. Summary

• scipy.stats is a powerful tool for statistical analysis.
• You can compute summaries, perform tests, model distributions, and generate random samples.

---

Exercise

• Generate 1000 samples from a normal distribution and compute mean, median, std, and mode.
• Test if a sample has a mean significantly different from 5.
• Fit a normal distribution to your own dataset and plot the histogram with the fitted PDF curve.

---

#Python #SciPy #Statistics #HypothesisTesting #DataAnalysis

https://t.iss.one/DataScienceM

🤖🧠 PyMuPDF: The Ultimate Python Library for High-Performance PDF Processing

🗓️ 09 Oct 2025
📚 AI News & Trends

If you’re a Python developer working with PDF documents whether it’s for text extraction, data analysis conversion or annotation then you’ve likely encountered the limitations of traditional tools. That’s where PyMuPDF also known as fitz, shines. It’s a lightweight, high-performance Python library that enables comprehensive PDF manipulation with minimal dependencies and maximum flexibility. In this ...

#PyMuPDF #PythonLibrary #PDFProcessing #TextExtraction #DataAnalysis #HighPerformance

🤖🧠 PandasAI: Transforming Data Analysis with Conversational Artificial Intelligence

🗓️ 28 Oct 2025
📚 AI News & Trends

In a world dominated by data, the ability to analyze and interpret information efficiently has become a core competitive advantage. From business intelligence dashboards to large-scale machine learning models, data-driven decision-making fuels innovation across industries. Yet, for most people, data analysis remains a technical challenge requiring coding expertise, statistical knowledge and familiarity with libraries like ...

#PandasAI #ConversationalAI #DataAnalysis #ArtificialIntelligence #DataScience #MachineLearning

💡 Pandas Cheatsheet

A quick guide to essential Pandas operations for data manipulation, focusing on creating, selecting, filtering, and grouping data in a DataFrame.

1. Creating a DataFrame
The primary data structure in Pandas is the DataFrame. It's often created from a dictionary.

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie'],
        'Age': [25, 32, 28],
        'City': ['New York', 'Paris', 'New York']}
df = pd.DataFrame(data)

print(df)
#       Name  Age       City
# 0    Alice   25   New York
# 1      Bob   32      Paris
# 2  Charlie   28   New York

• A dictionary is defined where keys become column names and values become the data in those columns. pd.DataFrame() converts it into a tabular structure.

2. Selecting Data with .loc and .iloc
Use .loc for label-based selection and .iloc for integer-position based selection.

# Select the first row by its integer position (0)
print(df.iloc[0])

# Select the row with index label 1 and only the 'Name' column
print(df.loc[1, 'Name'])

# Output for df.iloc[0]:
# Name       Alice
# Age           25
# City    New York
# Name: 0, dtype: object
#
# Output for df.loc[1, 'Name']:
# Bob

• .iloc[0] gets all data from the row at index position 0.
• .loc[1, 'Name'] gets the data at the intersection of index label 1 and column label 'Name'.

3. Filtering Data
Select subsets of data based on conditions.

# Select rows where Age is greater than 27
filtered_df = df[df['Age'] > 27]
print(filtered_df)
#       Name  Age       City
# 1      Bob   32      Paris
# 2  Charlie   28   New York

• The expression df['Age'] > 27 creates a boolean Series (True/False).
• Using this Series as an index df[...] returns only the rows where the value was True.

4. Grouping and Aggregating
The "group by" operation involves splitting data into groups, applying a function, and combining the results.

# Group by 'City' and calculate the mean age for each city
city_ages = df.groupby('City')['Age'].mean()
print(city_ages)
# City
# New York    26.5
# Paris       32.0
# Name: Age, dtype: float64

• .groupby('City') splits the DataFrame into groups based on unique city values.
• ['Age'].mean() then calculates the mean of the 'Age' column for each of these groups.

#Python #Pandas #DataAnalysis #DataScience #Programming

━━━━━━━━━━━━━━━
By: @DataScienceM ✨

#Pandas #DataAnalysis #Python #DataScience #Tutorial

Top 30 Pandas Functions & Methods

This lesson covers 30 essential Pandas functions for data manipulation and analysis, each with a standalone example and its output.

---

1. pd.DataFrame()
Creates a new DataFrame (a 2D labeled data structure) from various inputs like dictionaries or lists.

import pandas as pd
data = {'col1': [1, 2], 'col2': [3, 4]}
df = pd.DataFrame(data)
print(df)

col1  col2
0     1     3
1     2     4

---

2. pd.Series()
Creates a new Series (a 1D labeled array).

import pandas as pd
s = pd.Series([10, 20, 30, 40], name='MyNumbers')
print(s)

0    10
1    20
2    30
3    40
Name: MyNumbers, dtype: int64

---

3. pd.read_csv()
Reads data from a CSV file into a DataFrame. (Assuming a file data.csv exists).

# Create a dummy csv file first
with open('data.csv', 'w') as f:
    f.write('Name,Age\nAlice,25\nBob,30')

df = pd.read_csv('data.csv')
print(df)

Name  Age
0  Alice   25
1    Bob   30

---

4. df.to_csv()
Writes a DataFrame to a CSV file.

import pandas as pd
df = pd.DataFrame({'Name': ['Charlie'], 'Age': [35]})
# index=False prevents writing the DataFrame index to the file
df.to_csv('output.csv', index=False)
# You can check that 'output.csv' has been created.
print("File 'output.csv' created.")

File 'output.csv' created.

#PandasIO #DataFrame #Series

---

5. df.head()
Returns the first n rows of the DataFrame (default is 5).

import pandas as pd
data = {'Name': ['A', 'B', 'C', 'D', 'E', 'F'], 'Value': [1, 2, 3, 4, 5, 6]}
df = pd.DataFrame(data)
print(df.head(3))

Name  Value
0    A      1
1    B      2
2    C      3

---

6. df.tail()
Returns the last n rows of the DataFrame (default is 5).

import pandas as pd
data = {'Name': ['A', 'B', 'C', 'D', 'E', 'F'], 'Value': [1, 2, 3, 4, 5, 6]}
df = pd.DataFrame(data)
print(df.tail(2))

Name  Value
4    E      5
5    F      6

---

7. df.info()
Provides a concise summary of the DataFrame, including data types and non-null values.

import pandas as pd
import numpy as np
data = {'col1': [1, 2, 3], 'col2': [4.0, 5.0, np.nan], 'col3': ['A', 'B', 'C']}
df = pd.DataFrame(data)
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   col1    3 non-null      int64  
 1   col2    2 non-null      float64
 2   col3    3 non-null      object 
dtypes: float64(1), int64(1), object(1)
memory usage: 200.0+ bytes

---

8. df.shape
Returns a tuple representing the dimensionality (rows, columns) of the DataFrame.

import pandas as pd
df = pd.DataFrame({'A': [1, 2], 'B': [3, 4], 'C': [5, 6]})
print(df.shape)

(2, 3)

#DataInspection #PandasBasics

---

9. df.describe()
Generates descriptive statistics for numerical columns (count, mean, std, min, max, etc.).

import pandas as pd
df = pd.DataFrame({'Age': [22, 38, 26, 35, 29]})
print(df.describe())

Top 100 Data Analyst Interview Questions & Answers

#DataAnalysis #InterviewQuestions #SQL #Python #Statistics #CaseStudy #DataScience

Part 1: SQL Questions (Q1-30)

#1. What is the difference between DELETE, TRUNCATE, and DROP?
A:
• DELETE is a DML command that removes rows from a table based on a WHERE clause. It is slower as it logs each row deletion and can be rolled back.
• TRUNCATE is a DDL command that quickly removes all rows from a table. It is faster, cannot be rolled back, and resets table identity.
• DROP is a DDL command that removes the entire table, including its structure, data, and indexes.

#2. Select all unique departments from the employees table.
A: Use the DISTINCT keyword.

SELECT DISTINCT department
FROM employees;

#3. Find the top 5 highest-paid employees.
A: Use ORDER BY and LIMIT.

SELECT name, salary
FROM employees
ORDER BY salary DESC
LIMIT 5;

#4. What is the difference between WHERE and HAVING?
A:
• WHERE is used to filter records before any groupings are made (i.e., it operates on individual rows).
• HAVING is used to filter groups after aggregations (GROUP BY) have been performed.

-- Find departments with more than 10 employees
SELECT department, COUNT(employee_id)
FROM employees
GROUP BY department
HAVING COUNT(employee_id) > 10;

#5. What are the different types of SQL joins?
A:
• (INNER) JOIN: Returns records that have matching values in both tables.
• LEFT (OUTER) JOIN: Returns all records from the left table, and the matched records from the right table.
• RIGHT (OUTER) JOIN: Returns all records from the right table, and the matched records from the left table.
• FULL (OUTER) JOIN: Returns all records when there is a match in either the left or right table.
• SELF JOIN: A regular join, but the table is joined with itself.

#6. Write a query to find the second-highest salary.
A: Use OFFSET or a subquery.

-- Method 1: Using OFFSET
SELECT salary
FROM employees
ORDER BY salary DESC
LIMIT 1 OFFSET 1;

-- Method 2: Using a Subquery
SELECT MAX(salary)
FROM employees
WHERE salary < (SELECT MAX(salary) FROM employees);

#7. Find duplicate emails in a customers table.
A: Group by the email column and use HAVING to find groups with a count greater than 1.

SELECT email, COUNT(email)
FROM customers
GROUP BY email
HAVING COUNT(email) > 1;

#8. What is a primary key vs. a foreign key?
A:
• A Primary Key is a constraint that uniquely identifies each record in a table. It must contain unique values and cannot contain NULL values.
• A Foreign Key is a key used to link two tables together. It is a field (or collection of fields) in one table that refers to the Primary Key in another table.

#9. Explain Window Functions. Give an example.
A: Window functions perform a calculation across a set of table rows that are somehow related to the current row. Unlike aggregate functions, they do not collapse rows.

-- Rank employees by salary within each department
SELECT
    name,
    department,
    salary,
    RANK() OVER (PARTITION BY department ORDER BY salary DESC) as dept_rank
FROM employees;

#10. What is a CTE (Common Table Expression)?
A: A CTE is a temporary, named result set that you can reference within a SELECT, INSERT, UPDATE, or DELETE statement. It helps improve readability and break down complex queries.

• Group data by a column.

df.groupby('col1')

• Group by a column and get the sum.

df.groupby('col1').sum()

• Apply multiple aggregation functions at once.

df.groupby('col1').agg(['mean', 'count'])

• Get the size of each group.

df.groupby('col1').size()

• Get the frequency counts of unique values in a Series.

df['col1'].value_counts()

• Create a pivot table.

pd.pivot_table(df, values='D', index=['A', 'B'], columns=['C'])

VI. Merging, Joining & Concatenating

• Merge two DataFrames (like a SQL join).

pd.merge(left_df, right_df, on='key_column')

• Concatenate (stack) DataFrames along an axis.

pd.concat([df1, df2]) # Stacks rows

• Join DataFrames on their indexes.

left_df.join(right_df, how='outer')

VII. Input & Output

• Write a DataFrame to a CSV file.

df.to_csv('output.csv', index=False)

• Write a DataFrame to an Excel file.

df.to_excel('output.xlsx', sheet_name='Sheet1')

• Read data from an Excel file.

pd.read_excel('input.xlsx', sheet_name='Sheet1')

• Read from a SQL database.

pd.read_sql_query('SELECT * FROM my_table', connection_object)

VIII. Time Series & Special Operations

• Use the string accessor (.str) for Series operations.

s.str.lower()
s.str.contains('pattern')

• Use the datetime accessor (.dt) for Series operations.

s.dt.year
s.dt.day_name()

• Create a rolling window calculation.

df['col1'].rolling(window=3).mean()

• Create a basic plot from a Series or DataFrame.

df['col1'].plot(kind='hist')

#Python #Pandas #DataAnalysis #DataScience #Programming

━━━━━━━━━━━━━━━
By: @DataScienceM ✨

📌 NumPy for Absolute Beginners: A Project-Based Approach to Data Analysis

🗂 Category: DATA SCIENCE

🕒 Date: 2025-11-04 | ⏱️ Read time: 14 min read

Master NumPy for data analysis with this project-based guide for absolute beginners. Learn to build a high-performance sensor data pipeline from scratch and unlock the true speed of Python for data-intensive applications.

#NumPy #Python #DataAnalysis #DataScience

• (Time: 90s) Simpson's Paradox occurs when:
a) A model performs well on training data but poorly on test data.
b) Two variables appear to be correlated, but the correlation is caused by a third variable.
c) A trend appears in several different groups of data but disappears or reverses when these groups are combined.
d) The mean, median, and mode of a distribution are all the same.

• (Time: 75s) When presenting your findings to non-technical stakeholders, you should focus on:
a) The complexity of your statistical models and the p-values.
b) The story the data tells, the business implications, and actionable recommendations.
c) The exact Python code and SQL queries you used.
d) Every single chart and table you produced during EDA.

• (Time: 75s) A survey about job satisfaction is only sent out via a corporate email newsletter. The results may suffer from what kind of bias?
a) Survivorship bias
b) Selection bias
c) Recall bias
d) Observer bias

• (Time: 90s) For which of the following machine learning algorithms is feature scaling (e.g., normalization or standardization) most critical?
a) Decision Trees and Random Forests.
b) K-Nearest Neighbors (KNN) and Support Vector Machines (SVM).
c) Naive Bayes.
d) All algorithms require feature scaling to the same degree.

• (Time: 90s) A Root Cause Analysis for a business problem primarily aims to:
a) Identify all correlations related to the problem.
b) Assign blame to the responsible team.
c) Build a model to predict when the problem will happen again.
d) Move beyond symptoms to find the fundamental underlying cause of the problem.

• (Time: 75s) A "funnel analysis" is typically used to:
a) Segment customers into different value tiers.
b) Understand and optimize a multi-step user journey, identifying where users drop off.
c) Forecast future sales.
d) Perform A/B tests on a website homepage.

• (Time: 75s) Tracking the engagement metrics of users grouped by their sign-up month is an example of:
a) Funnel Analysis
b) Regression Analysis
c) Cohort Analysis
d) Time-Series Forecasting

• (Time: 90s) A retail company wants to increase customer lifetime value (CLV). A data-driven first step would be to:
a) Redesign the company logo.
b) Increase the price of all products.
c) Perform customer segmentation (e.g., using RFM analysis) to understand the behavior of different customer groups and tailor strategies accordingly.
d) Switch to a new database provider.

#DataAnalysis #Certification #Exam #Advanced #SQL #Pandas #Statistics #MachineLearning

━━━━━━━━━━━━━━━
By: @DataScienceM ✨

📌 Beyond Numbers: How to Humanize Your Data & Analysis

🗂 Category: DATA SCIENCE

🕒 Date: 2025-11-07 | ⏱️ Read time: 16 min read

Just as an optical illusion can deceive the eye, raw data can easily mislead. To make truly effective data-driven decisions, we must learn to humanize our analysis. This means looking beyond the raw numbers to add critical context, build a compelling narrative, and uncover the deeper story hidden within the figures. By focusing on the 'why' behind the 'what', we can avoid common interpretation pitfalls and unlock more powerful, actionable insights.

#DataAnalysis #DataStorytelling #BusinessIntelligence #DataLiteracy

🤖🧠 PokeeResearch: Advancing Deep Research with AI and Web-Integrated Intelligence

🗓️ 09 Nov 2025
📚 AI News & Trends

In the modern information era, the ability to research fast, accurately and at scale has become a competitive advantage for businesses, researchers, analysts and developers. As online data expands exponentially, traditional search engines and manual research workflows are no longer sufficient to gather reliable insights efficiently. This need has fueled the rise of AI research ...

#AIResearch #DeepResearch #WebIntelligence #ArtificialIntelligence #ResearchAutomation #DataAnalysis