p2j - Python-to-Jupyter parser with zero intervention
The purpose of this package is to be able to run a code on Jupyter notebook without having to copy each paragraph of the code into every cell. It's also useful if we want to run our code in Google Colab. This parser isn't perfect, but you would be satisfactorily pleased with what you get.

$ pip install p2j
$ p2j script.py
$ Notebook script.ipynb written.

There are various data viewer tools for vscode, I like this one:
Data Preview
which allow to view the data in a nice table, also provide some quick visualization panels.

About 48 hours left to apply for Computational Neuroscience training program (January 22 to February 9, 2024) as a student!

Apply here by 23:59 Sunday December 17th anywhere on earth:
https://academy.neuromatch.io/pilot

اگر به یادگیری روش های درست کد نویسی pep8 علاقه دارید، سایت زیر به فارسی مطالب مفیدی دارد.

pep8.ir

As before we saw the process of fining the best hyperparameters for a machine learning model using gridsearch, now we consider the same example by Genetic Algorithm.

References:
0. GitHub notebook with direct link to colab.
1. Tune Your Scikit-learn Model Using Evolutionary Algorithms
2. Understanding the evaluation process

Using conda for your environments? want to just add a package for testing but not break your existing one? dislike waiting half hour for conda to make up its mind about dependencies? you can tell it to copy an existing environment without trying to download any new versions with --clone and --offline:

conda create --name copy_of_foo --clone foo --offline

credit: Marmaduke

Downloading entire playlist from YouTube using yt-dlp:

# brew install yt-dlp
yt-dlp  playlisturl

use -c for continuing and -N for multi-threaded fragment downloads:

yt-dpl -c -N 5 playlisturl

ممکنه از این پلی لیست خوشتون بیاد.

Latexify

Pandas cheat sheet.

How to pass class of parameters to jitted python function?
Here are some examples.
It's used for writing cleaner functions which get many parameters.
COLAB
GitHub

#numba #jit

Basic Dictionary commands (1):

# Create a dictionary
d = {'name': 'Max', 'age': 28}
d = dict(name='Anna', age=27)
squares = {x: x*x for x in range(6)}

# Reading 
print(d['name'])
print(d.get('age'))
print(d.get('address')) # returns None
print(d.get('address', 'not found')) # returns 'not found'

# Adding/Modifying
d['address'] = '123 N. Elm St.'
d['name'] = 'Anna'

# Updating
d.update({'name': 'Max', 'age': 28})
d.update(address='123 N. Elm St.')
d['age'] = 30

# Deleting
del d['name']
d.pop('age')
d.popitem() # removes a random key-value pair
d.clear() # removes all items

Basic Dictionary commands (2):

# Looping
for key in d:
    print(key)
for value in d.values():
    print(value)
for key, value in d.items():
    print(key, value)
    
# Copying
d_copy = d.copy()
d_copy = dict(d)

# Merging
d = {'name': 'Max', 'age': 28}
d1 = {'name': 'Anna', 'age': 27}
d.update(d1)
d = {**d, **d1}

# Dictionary Comprehension
d = {x: x**2 for x in range(10)}
d = {k: v**2 for k, v in zip(['a', 'b'], range(4))}
d = {k: v for k, v in d.items() if v % 2 == 0}

Copy and deep copy in dictionaries:

# copy and deepcopy
import copy
d = {'x': [1, 2, 3]}
d1 = d.copy()
d2 = copy.deepcopy(d)

d['x'].append(4)
print(d1) # {'x': [1, 2, 3, 4]}
print(d2) # {'x': [1, 2, 3]}

activate "sticky scroll" in vscode through ctrl+shirt+p panel.

#Datashader simplifies creating meaningful visuals from large datasets by breaking down the process into clear steps. It automatically generates accurate visualizations without the need for manual parameter tweaking. Computations are optimized using Python, Numba, Dask, and CUDA, making it efficient even with huge datasets on standard hardware.

https://datashader.org/
Gallery

The walrus operator, introduced in Python 3.8, is represented by ":=". It allows the assignment of a value to a variable within an expression.

## f(x) is called 3 times
foo = [f(x), f(x)**2, f(x)**3]

## two lines of code
y = f(x)
foo = [y, y**2, y**3]

## walrus operator
foo = [y := f(x), y**2, y**3]

Walrus operator [1]

# Avoiding inefficient comprehensions

results = []
for x in data:
    res = f(x)
    if res:
        results.append(res)
        
# f(x) is called twice
results = [f(x) for x in data if f(x)]        
# walrus operator
results = [res for x in data if (res := f(x))]

# Unnecessary variables in scope
match = pattern.search(data)
if match:
    do_sth(math)

if (match := pattern.search(data)):
    do_sth(match)
    
# Processing streams in chunks
chunk = file.read(8192)
while chunk:
    process(chunk)
    chunk = file.read(8192)

# walrus operator
while chunk := file.read(8192):
    process(chunk)

Datasets for machine learning typically contain a large number of features, but such high-dimensional feature spaces are not always helpful.

In general, all the features are not equally important and there are certain features that account for a large percentage of variance in the dataset. Dimensionality reduction algorithms aim to reduce the dimension of the feature space to a fraction of the original number of dimensions. In doing so, the features with high variance are still retained—but are in the transformed feature space. And principal component analysis (PCA) is one of the most popular dimensionality reduction algorithms.

Here's a simple example in Python demonstrating PCA for dimensionality reduction before training a scikit-learn classifier.
Github

You may also need to read more about PCA here.

Applications for Students & Teaching Assistants are Open!

1️⃣ 3-week Courses (July 8 - 26, 2024):
- Computational Neuroscience: Explore the intricacies of the brain's computational processes and join an engaging community of learners.
- Deep Learning: Delve into the world of machine learning, uncovering the principles and applications of deep learning.

2️⃣ 2-week Courses (July 15 - 26, 2024):
- Computational Tools for Climate Science: Uncover the tools and techniques driving climate science research in this dynamic two-week course.
- NeuroAI - Inaugural Year!: Be part of history as we launch our first-ever NeuroAI course, designed to explore the intersection of neuroscience and artificial intelligence.

https://neuromatch.io/courses/

Incremental principal component analysis (IPCA) is typically used as a replacement for principal component analysis (PCA) when the dataset to be decomposed is too large to fit in memory.

IPCA builds a low-rank approximation for the input data using an amount of memory which is independent of the number of input data samples. It is still dependent on the input data features, but changing the batch size allows for control of memory usage.

I have made some changes on the example from sklearn documentation so one does not need to load the whole dataset in memory.

python
X_ipca = np.zeros((X.shape[0], n_components))
for i in range(3):
    ipca.partial_fit(X[i*50:(i+1)*50])
    
for i in range(3):
    X_ipca[i*50:(i+1)*50] = ipca.transform(X[i*50:(i+1)*50])

GitHub