Generate random valid users agent for your bot crawlers in Python:

https://pypi.org/project/user_agent/

#python #user_agent #crawl #web_scraper

How to Load & Parse YAML file in Python?

To install YAML library:

pip install pyyaml

To load a YAML file:

>>> import yaml
 >>> print yaml.load("""
 ... name: Vorlin Laruknuzum
 ... sex: Male
 ... class: Priest
 ... title: Acolyte
 ... hp: [32, 71]
 ... sp: [1, 13]
 ... gold: 423
 ... inventory:
 ... - a Holy Book of Prayers (Words of Wisdom)
 ... - an Azure Potion of Cure Light Wounds
 ... - a Silver Wand of Wonder
 ... """)

To convert a dictionary to a YAML file:

>>> print yaml.dump({'name': "The Cloak 'Colluin'", 'depth': 5, 'rarity': 45, 'weight': 10, 'cost': 50000, 'flags': ['INT', 'WIS',     'SPEED', 'STEALTH']})

#python #yaml #pyyaml #load #dump

How to localize a UTC date?

import pytz
 import datetime

 date_format = '%Y-%m-%d %H:%M:%S'
 dt = datetime.datetime.strptime('2018-12-12 00:00:00', date_format)
 utc_tz = pytz.timezone('UTC')
 utc_d = utc_tz.localize(dt)
 teh_tz = pytz.timezone('Asia/Tehran')
 dt = utc_d.astimezone(teh_tz)

#localize #timezone #pytz #astimezone #replace

#ebook #download #sql #postgreql #pdf

#pdf #ebook #postgresql

https://developer.mozilla.org/en-US/docs/Learn/Server-side/Django

#django #tutorial #mozilla

ngrok something that ROCKS and is priceless!

As ngrok itself says:

Public URLs for exposing your local web server

Have you had an experience of when you need to test a tool and the 3rd party needs you to give a public accessible URL? Well, what would you do when you are developing a project on your PC, Laptop? You don't have any URL to give to 3rd party tools. Here ngrok truely rocks!

Downlaod it from the URL below:

https://ngrok.com/

And extract the download binay for your OS. In order to run the ngrok just do like below:

./ngrok http 5000

The above command will output some data and a part which is Forwarding for http and https:

Forwarding                    https://76eec0f8.ngrok.io -> localhost:5000
 Forwarding                    https://76eec0f8.ngrok.io -> localhost:5000

The URL 76eec0f8.ngrok.io is the part that you will give to the 3rd party. and all the other part of the URL path will be forwarded
to port 5000 on your local server.

Yes, I know it's like a magic.

#ngrok #port #http #ngrok_io

https://stackoverflow.com/questions/23554872/why-does-pycharm-propose-to-change-method-to-static

#python #pycharm #static #staticmethod

Why some sites does not show in iFrame HTML tag?

The first reason you need to check is to make sure that you are not loading an HTTP website from within an HTTPS website, otherwise
you will get and error. You can see the error by using your browser Developer Tools -> Console.

If you are loading an HTTPS from inside of HTTPS and it is not loading, then take a look at the header X-Frame-Option in response header. If it exists it probably has avalue of either DENY or SAMEORIGIN.

DENY: No one can load the website in iframe. Even the same domain page wont be able to load.

SAMEORIGIN: only a page which is in same domain can load this website in iframe.

#html #iframe #X_Frame_Option

#template #idea

#template #idea

I sometimes forgot to pull data from git before start working on a project. To minimize the headache of merge conflict or having to reset head and stash your data you can set a cronjob to run every 10 minute or so. And inside of your bash script CD into eaach of your folders and issue git pull command.

If you have similar issues and you're solving it in a different manner, I'd be happy to hear about it. :)

#linux #mac #bash #script #git #cron #crontab #cronjob

jsonschema slack channel:

https://join.slack.com/t/json-schema/shared_invite/enQtMjk1NDcyNDI2NTAwLTcyYmYwMjdmMmUxNzZjYzIxNGU2YjdkNzdlOGZiNjIwNDI2M2Y3NmRkYjA4YmMwODMwYjgyOTFlNWZjZjAyNjg

#slack #jsonschema

Read all about Jinja2 templates:

https://jinja.pocoo.org/docs/2.10/templates/

#jinja2 #pocoo

How to create a simple Chrome extension?

First of all you need to enable developer mode in Chrome in order to debug your extension before publishing. Head over to chrome:// extensions in your browser, or simply click “More Tools” and “Extensions” on the Chrome menu. This should lead you to the Extension Management page, where you can turn on Developer Mode (it should be in the top right corner).

OK! Create a directory and put a file called manifest.json into it with the below content:

{
    "name": "Name of the extension",
    "description": "Description of the extension",
    "version": "1.0",
    "background": {
       "scripts": [ "background.js" ]
    },
    "browser_action": {
       "default_icon": "icon.png",
       "default_title": "Your Extension Title"
    },
    "icons": {
       "128": "icon_128.png"
    },
    "manifest_version": 2,
    "permissions": ["activeTab", "tabs", "contextMenus"]
 }

background section is used to link a javascript file to your extension (You need to create a file background.js in your newly created directory).

permissions section is used when you want specific permissions in your js file. contextMenus is used to work on menu items related to your extension. tabs is used to open a new tab on chrome browser.

In the next post I will explain the JS section.

#chrome #extension

Now let's write the main JS code in order to add menu items into Chrome extension. Create background.js file in the same directory as manifest.json with the content below:

if (chrome && chrome.browserAction){
     var baseUrl = "https://www.example.com"
     chrome.runtime.onInstalled.addListener(function() {
         chrome.tabs.create({ url: baseUrl + "/extension-installed/" });
     });

     chrome.runtime.setUninstallURL(baseUrl + "/extension-uninstalled/");

     // open a new page on left clicking on the extension itself
     chrome.browserAction.onClicked.addListener(function(activeTab) {
         chrome.tabs.create({ url: baseUrl + "/my-home-page" });
     });

     chrome.contextMenus.create({
         "id": "first_item",
         "title": "This is the first item",
         "contexts": ["browser_action"]
     });

     chrome.contextMenus.create({
         "id": "second_item",
         "title": "This is the second item",
         "contexts": ["browser_action"]
     });

     chrome.contextMenus.onClicked.addListener(function(info)
     {
         urls = {
             first_item: baseUrl + "/first-item",
             second_item: baseUrl + "/second-item"
         };
         chrome.tabs.create({url: urls[info.menuItemId]});
     });
 }

There are many events that based on that event you can do an action. Let's review some of the codes above.

chrome.runtime.onInstalled.addListener: on extension installation open a url using chrome.tabs. As you recall from the previous post we have added tabs to the permission section of manifest.

chrome.runtime.setUninstallURL: open a url on extension uninstallation.

chrome.browserAction.onClicked.addListener: open a url when your extension in toolbar is clicked (at the right of the address bar).

chrome.contextMenus.create: create an item with specific id and title. contexts sections tells the browser to show the menu just when user right click on the extension in toolbar. You can add page to have the extension menu item on the main page of browser itself.

chrome.contextMenus.onClicked.addListener: When one of the menu items is clicked open the corresponding URL.

Now to test it go to chrome://extensions/. Make sure developer mode is enabled. Click on Load unpacked and select your extension folder and open it. It will be installed by now.

Enjoy working with Chrome Extension! :)

#chrome #xtension

In order to list all logical volumes (LVM) in Linux you can issue the below command:

$ lvscan
  ACTIVE            '/dev/mysql-vg/swap_1' [31.88 GiB] inherit
  ACTIVE            '/dev/mysql-vg/root' [27.94 GiB] inherit
  ACTIVE            '/dev/mysql-vg/var' [833.64 GiB] inherit

#linux #lvscan #lvm #logical_volume

Python Microservices Developement

#python #ebook #epub #pdf #microservice #flask

How to configure a Delayed Replica Set Member?

Let's assume that our member is third in the array of replica members:

cfg = rs.conf()
 cfg.members[2].priority = 0
 cfg.members[2].hidden = true
 cfg.members[2].slaveDelay = 3600
 rs.reconfig(cfg)

The priority is set to 0 (preventing to be elected as primary).

The hidden to true in order to hide the node from clients querying the database.

And finally slaveDelay to number of seconds that we want it to get behind of Primary Node.

The use case for this is to have a replication that is used for analytical purposes or used for backup and so on.

#mongodb #mongo #replica #replication #primary #delayed_replica_set #slaveDelay