On previous posts we explained about query slowness. Here we try to explain different parts of the function.

db.currentOp: in progress operations in mongoDB is displayed by this command. The response of the command is in json format, so you
can use command like db.currentOp()['inprog']. The response has many useful informations like lock status, numYields and so on.
The part we are interested in is opid part. opid is the pid number of the query operation. op section of each operation shows the type of the query. It can be an internal database command, insert command and or query. secs_running of the operation is the part that we can check whether a query has taken a long time or not. It is in second.


db.killOp : killing an operation is just as simple as giving the opid number to killOp as below:

db.killOp(6123213)

This is all we've done in previous posts, to kill slow queries in mongoDB.

#mongodb #mongo #currentOp #killOp #opid

#ebook #book #mongodb #mongo #pdf

#ebook #book #mongodb #mongo #pdf

See live disk IO status by using iostat:

iostat -dx 1

The output has many columns. The part I'm interested in for now is r/s which refers to read per second and w/s which is write per
second. To see size per second in read and write see columns rkB/s, wkB/s in their corresponding order.

NOTE: if you don't have iostat on your linux os install it on debian by issuing apt-get install sysstat command.


#linux #debian #iostat #read_per_second #write_per_second #sysstat

Benchmark disk performance using hdparm & dd.

In order to get a meaningful result run the test a couple of times.


Direct read (without cache):

$ sudo hdparm -t /dev/sda2
 /dev/sda2:
  Timing buffered disk reads: 302 MB in  3.00 seconds = 100.58 MB/sec

And here's a cached read:

$ sudo hdparm -T /dev/sda2
 /dev/sda2:
  Timing cached reads:   4636 MB in  2.00 seconds = 2318.89 MB/sec

-t: Perform timings of device reads for benchmark and comparison
purposes. For meaningful results, this operation should be repeated
2-3 times on an otherwise inactive system (no other active processes)
with at least a couple of megabytes of free memory. This displays
the speed of reading through the buffer cache to the disk without
any prior caching of data. This measurement is an indication of how
fast the drive can sustain sequential data reads under Linux, without
any filesystem overhead. To ensure accurate measurements, the
buffer cache is flushed during the processing of -t using the
BLKFLSBUF ioctl.

-T: Perform timings of cache reads for benchmark and comparison purposes.
For meaningful results, this operation should be repeated 2-3
times on an otherwise inactive system (no other active processes)
with at least a couple of megabytes of free memory. This displays
the speed of reading directly from the Linux buffer cache without
disk access. This measurement is essentially an indication of the
throughput of the processor, cache, and memory of the system under
test.


You can use dd command to test your hard disk too:

$ time sh -c "dd if=/dev/zero of=ddfile bs=8k count=250000 && sync"; rm ddfile

rm ddfile removes the test file created by dd command of=ddfile. of param stands for output file.


These are some useful and simple disk benchmarking tools.

#linux #benchmark #hdd #dd #hard_disk #hdparm

You can see both direct and cache read using command below:

$ sudo hdparm -Tt /dev/sda

 /dev/sda:
 Timing cached reads:   12540 MB in  2.00 seconds = 6277.67 MB/sec
 Timing buffered disk reads: 234 MB in  3.00 seconds =  77.98 MB/sec

#linux #benchmark #hdd #hdparm #sda

#ebook #book #epub #python #keras

دوستانی که با الگوریتم زیر آشنایی دارند و چهت پروژه ارشد میتونند کمکی کنند لطفا با کاربر زیر در تماس باشند (هزینه توافقی):

Metropolis–Hastings algorithm (Markov chain Monte Carlo (MCMC) method)

👤 @shararehadipour

MYSQL insert

If you have bulk scripts for your email/sms like me, and you are sending to thousands of users, you will difintely will get stuck in the middle of the bulk notification or it will be very slow and unacceptable. First of all you must initiate only ONE mysql connection. If you are inserting your data one by one, you're dead again! try to use executemany that will insert data into mySQL in bulk not one by one:

client.executemany(
       """INSERT INTO email (name, spam, email, uid, email_content)
       VALUES (%s, %s, %s, %s, %s)""",
       [
       ("Ali", 0, '[email protected]', 1, 'EMAIL_CONTENT'),
       ("Reza", 1, '[email protected]', 2, 'EMAIL_CONTENT'),
       ("Mohsen", 1, '[email protected]', 3, 'EMAIL_CONTENT')
       ] )

Other note for bulk insertion is to avoid disk IO in case possible, and use redis, memcached or so on for inserting some data like user's phone or emails. It will tremendously improve performance of your bulk script.

#python #mysql #executemany #redis #bulk #email #sms

It's Dangerous!

Today I had to implement email unsubscription and for that I had to pass some data alongside the link as a token in emails. The best candidate for this scenario is itsdangerous. If I don't use it have to store tokens in a redis DB and link those tokens with received token from emails as ubsubscription link. These all adds complexity.

Sometimes you just want to send some data to untrusted environments. But how to do this safely? The trick involves signing. Given a key only you know, you can cryptographically sign your data and hand it over to someone else. When you get the data back you can easily ensure that nobody tampered with it.

Granted, the receiver can decode the contents and look into the package, but they can not modify the contents unless they also have your secret key. So if you keep the key secret and complex, you will be fine.

Internally itsdangerous uses HMAC and SHA1 for signing by default and bases the implementation on the Django signing module. It also however supports JSON Web Signatures (JWS). The library is BSD licensed and written by Armin Ronacher though most of the copyright for the design and implementation goes to Simon Willison and the other amazing Django people that made this library possible.


Example Use Cases:
- You can serialize and sign a user ID for unsubscribing of newsletters into URLs. This way you don’t need to generate one- time tokens and store them in the database. Same thing with any kind of activation link for accounts and similar things.

- Signed objects can be stored in cookies or other untrusted sources which means you don’t need to have sessions stored on the server, which reduces the number of necessary database queries.

- Signed information can safely do a roundtrip between server and client in general which makes them useful for passing server-side state to a client and then back.


To install it using pip:

pip install itsdangerous

Sample code:

>>> from itsdangerous import URLSafeSerializer
 >>> s = URLSafeSerializer('secret-key')
 >>> s.dumps([1, 2, 3, 4])
 'WzEsMiwzLDRd.wSPHqC0gR7VUqivlSukJ0IeTDgo'
 >>> s.loads('WzEsMiwzLDRd.wSPHqC0gR7VUqivlSukJ0IeTDgo')
 [1, 2, 3, 4]

#python #itsdangerous #URLSafeSerializer

What is cronjob?
cron is a unix, solaris, Linux utility that allows tasks to be automatically run in the background at regular intervals by the cron daemon.

What is crontab?
Crontab (CRON TABle) is a file which contains the schedule of cron entries to be run and at specified times.

What is a cron job
Cron job or cron schedule is a specific set of execution instructions specifing day, time and command to execute.

To see list current cronjobs use crontab -l.

Edit crontab file, or create one if it doesn’t already exist by issuing the command below:

crontab -e

It may get opened by nano by default. In case you want to change your default editor for crontab use the below command:

export EDITOR=vim

NOTE: if you want to persist this data you have to put the above export command inside of ~/.bashrc


The general form of a cronjob is like below:

* * * * *   command to be executed

In total we have 5 stars. From left to right, first star is the minute that you want to run your cronjob at (min (0 - 59)).

Second star is hour (0 - 23).

Third star is day of month (1 - 31).

Fourth star refers to month (1 - 12).

And last star which refers to day of week (0 - 6). Be careful that 0 is sunday!


Now let's create a sample cronjob that reset our eshop service at 22:00:00 everyday:

0 22 * * * svc -k /etc/service/eshop

The other stars say that run this script every day, every day of month and every month.

#linux #cron #cronjob #crontab

Do you know what kind of RAID technology is suitable for you? Do you know whether RAID 0 is fault tolerant or not? What kind of raid duobles, triples your read performance or write performance?

This is just a tip of the iceberg. To know these parameters and memorizing them is hard, so there is an online tool just for this purpose:

https://www.raid-calculator.com/

#hardware #raid #disk #raid_calculator

To rename a window in tmux, press Control+b then press comma as below:

^b + ,

Now you can edit your window name and hit enter to save the new name.

#tmux #window

To see your current database in MySQL:

mysql> SELECT DATABASE();
 +------------+
 | DATABASE() |
 +------------+
 | my_data    |
 +------------+
 1 row in set (0.00 sec)

As shown above my current database is my_data. I usually forgot what db I'm currently on, when I use tmux and my session has been kept open for weeks. That's why! :)

#mysql #database #current_database #current_db

Change stream is a new method on MongoDB 3.6 that you can use to watch real-time data modifications to get changes. In older versions you had to tail the oplog.

for change in db.collection.watch():
     print(change)

The ChangeStream iterable blocks until the next change document is returned or an error is raised. If the next() method encounters a network error when retrieving a batch from the server, it will automatically attempt to recreate the cursor such that no change events are missed. Any error encountered during the resume attempt indicates there may be an outage and will be raised.

#mongodb #mongo #mongo36 #change_stream #stream #etl

Did you push a very large file into git? Does everyone yell at you about your commit and your uselessness? Are you a junky punky like me that just ruin things? Oh I'm kidding...

Because of that big file cloning the repo again would take a long long time. Removing the file locally and pushing again would not solve the problem as that big file is in Git's history.

If you want to remove the large file from your git history, so that when everyone clone the repo should not wait for that large file, just do as follow:

git filter-branch --tree-filter 'rm path/to/your/bigfile' HEAD

 git push origin master --force

I should note that you should be in the root of git repo.

If you need to do this, be sure to keep a copy of your repo around in case something goes wrong.

#git #clone #rm #remove #large_file #blob #rebase #filter_branch

How much do you know rebooting/shutting down your linux server?

* First of all you should be root or sudoer to be able to reboot the server.


The command below reboot the server immediately:

reboot

In case you want to reboot in a specific time, you can use shutdown command! Yes you have to use shutdown for rebooting server, it has historical reasons.

shutdown -r time "message"

time parameter can be now, or in the format of hh:mm, or in the format of +m which stands for minute. now is a shortcut for +0. The message part is the part that will be broadcast to all users who are logged in.

NOTE: shutdown command is recommended over reboot command.


So by the explanation given so far you can reboot your system after 5 minutes by the below command:

shutdown -r +5 "Server is going down for kernel upgrade. Please save your work ASAP."

To see last reboots history log:

last reboot

To reboot a remote server you can use ssh:

ssh root@server1 /sbin/reboot

#linux #cyberciti #reboot #shutdown #remote_reboot

#programmer_jokes #fun

#programmer_jokes #fun