What is the general or best way to do deploy application to staging and to production
I'll try to elaborate as much as possible:
Our app is dockerized and we are using docker-compose to deploy (k8s not required right now). Everything runs on Azure :VMs,container registry as well as Azure devops for build pipelines, there are two servers for staging and production and both of them have different FQDN configured in azure front door.
Now the problem we are facing is when we are building the containers we can either build it for production or staging because both have diff domain names and other conf files. So I can't leverage the full use of azure devops release pipeline, where we can drop the artifact to staging and then use the same containers for production.
What could be the best solution:
Should we maintain two different branches like master for staging and one branch for release in the production.
Or should the devs find a way to make sure the same containers work in both env by taking variables during run time.
​
How orgs do this in general do they use the same build or maintain diff pipeline/build to diff env or is there a better solution ?
https://redd.it/11fwufz
@r_devops
I'll try to elaborate as much as possible:
Our app is dockerized and we are using docker-compose to deploy (k8s not required right now). Everything runs on Azure :VMs,container registry as well as Azure devops for build pipelines, there are two servers for staging and production and both of them have different FQDN configured in azure front door.
Now the problem we are facing is when we are building the containers we can either build it for production or staging because both have diff domain names and other conf files. So I can't leverage the full use of azure devops release pipeline, where we can drop the artifact to staging and then use the same containers for production.
What could be the best solution:
Should we maintain two different branches like master for staging and one branch for release in the production.
Or should the devs find a way to make sure the same containers work in both env by taking variables during run time.
​
How orgs do this in general do they use the same build or maintain diff pipeline/build to diff env or is there a better solution ?
https://redd.it/11fwufz
@r_devops
Reddit
r/devops on Reddit: What is the general or best way to do deploy application to staging and to production
Posted by u/defact0o - No votes and 1 comment
Using OpenTelemetry with Splunk Enterprise
Hi Everyone!
Recently we have explored the use of OpenTelemetry and Open Source solution to minimize the cost of our monitoring. While we are not removing our commercial solution due to some obvious benefits, we want to slowly introduce the use of Open Source to our infrastructure (and eventually transition to it).
Currently we are using Splunk Enterprise (with Splunk ITSI) to ingest logs, query the metric, and create a KPI on ITSI. We want to utilize OpenTelemetry to ingest data from Splunk Forwarder, and push it to Loki or Prometheus, whichever is available.
Our goal is to forward data from the agent directly to OpenTelemetry, instead of pushing the data to indexer, such that:
Splunk Universal (or Heavy) Forwarder -> OpenTelemetry -> Prometheus / Loki
Instead of:
Splunk Universal Forwarder -> Splunk Indexer
We are able to achieve the ingestion to Loki, however, the data ingested seems to be cooked, and we are getting hex data (as shown here). Anyone who has similar cases before and was able to successfully perform this framework?
https://redd.it/11ex85r
@r_devops
Hi Everyone!
Recently we have explored the use of OpenTelemetry and Open Source solution to minimize the cost of our monitoring. While we are not removing our commercial solution due to some obvious benefits, we want to slowly introduce the use of Open Source to our infrastructure (and eventually transition to it).
Currently we are using Splunk Enterprise (with Splunk ITSI) to ingest logs, query the metric, and create a KPI on ITSI. We want to utilize OpenTelemetry to ingest data from Splunk Forwarder, and push it to Loki or Prometheus, whichever is available.
Our goal is to forward data from the agent directly to OpenTelemetry, instead of pushing the data to indexer, such that:
Splunk Universal (or Heavy) Forwarder -> OpenTelemetry -> Prometheus / Loki
Instead of:
Splunk Universal Forwarder -> Splunk Indexer
We are able to achieve the ingestion to Loki, however, the data ingested seems to be cooked, and we are getting hex data (as shown here). Anyone who has similar cases before and was able to successfully perform this framework?
https://redd.it/11ex85r
@r_devops
pasteboard.co
Pasteboard - Uploaded Image
Simple and lightning fast image sharing. Upload clipboard images with Copy & Paste and image files with Drag & Drop
a silly little idea for a gh bot
yesterday I had an idea to build a github bot or whatever that would automaticly assign a reviewer to a pr based on the diff content. I wonder if it is something that anyone else would want, or am I just thinking of solving a non existent problem
https://redd.it/11ei8dc
@r_devops
yesterday I had an idea to build a github bot or whatever that would automaticly assign a reviewer to a pr based on the diff content. I wonder if it is something that anyone else would want, or am I just thinking of solving a non existent problem
https://redd.it/11ei8dc
@r_devops
Reddit
r/devops on Reddit: a silly little idea for a gh bot
Posted by u/pawellisso - No votes and no comments
Is devops really that difficult???
I've been thinking of learning devops from scratch and when I started to do my research on what tools technologies are needed- IT IS OVERWHELMING. 1. So many softwares tools to play around with
2. 50% of the terms were new to me
3. I read these tools have a lot of depth and are complex.
Someone help me out hereee
https://redd.it/11e68hg
@r_devops
I've been thinking of learning devops from scratch and when I started to do my research on what tools technologies are needed- IT IS OVERWHELMING. 1. So many softwares tools to play around with
2. 50% of the terms were new to me
3. I read these tools have a lot of depth and are complex.
Someone help me out hereee
https://redd.it/11e68hg
@r_devops
Reddit
r/devops on Reddit: Is devops really that difficult???
Posted by u/rohost14 - No votes and no comments
How does DevOps enable continuous integration and continuous delivery (CI/CD)?
Hoping to get a good response from fellow members on this question as I need the answer for my project. Some references related to this will be added advantage.
https://redd.it/11e0egb
@r_devops
Hoping to get a good response from fellow members on this question as I need the answer for my project. Some references related to this will be added advantage.
https://redd.it/11e0egb
@r_devops
Reddit
r/devops on Reddit: How does DevOps enable continuous integration and continuous delivery (CI/CD)?
Posted by u/risabh07 - No votes and no comments
Report on aws cost and usages
Aws currently generates a very nice cost and usage report however if your account uses a lot of services , it will lay out the 9 highest cost of services you used but for the rest it will group the services and name it “others”. Is there a way for AWS not to do this or is there a third party tool that will individually lay out the usage of the services/resources and the cost?
https://redd.it/11g1z2d
@r_devops
Aws currently generates a very nice cost and usage report however if your account uses a lot of services , it will lay out the 9 highest cost of services you used but for the rest it will group the services and name it “others”. Is there a way for AWS not to do this or is there a third party tool that will individually lay out the usage of the services/resources and the cost?
https://redd.it/11g1z2d
@r_devops
Reddit
r/devops on Reddit: Report on aws cost and usages
Posted by u/eyesniper12 - No votes and 2 comments
Atlantis on GCP multiple service accounts
I'm setting up infra automation for google cloud with Atlantis running on GKE, and I'm a bit stumped on how to manage the service accounts. Different environments are in different projects, so my idea was to create the service accounts with terraform (no biggie with for_each and the official module), and then inject it as K8S secret to the Atlantis pod (and consider using vault in the future). From here on I'm very unsure if I can make the workflows will be able to pick the right credentials. I'm not a fan of TF workspaces, but nothing else comes to mind. The best would be something short lived obviously. How's everyone dealing with this?
https://redd.it/11g39h3
@r_devops
I'm setting up infra automation for google cloud with Atlantis running on GKE, and I'm a bit stumped on how to manage the service accounts. Different environments are in different projects, so my idea was to create the service accounts with terraform (no biggie with for_each and the official module), and then inject it as K8S secret to the Atlantis pod (and consider using vault in the future). From here on I'm very unsure if I can make the workflows will be able to pick the right credentials. I'm not a fan of TF workspaces, but nothing else comes to mind. The best would be something short lived obviously. How's everyone dealing with this?
https://redd.it/11g39h3
@r_devops
Reddit
r/devops on Reddit: Atlantis on GCP multiple service accounts
Posted by u/salvaged_goods - No votes and no comments
How to get a joined volume Python Virtual environment to Airflow Docker working with externalpythontask?
# GOAL
​
\- Have a local python environemnt that I can swap up and install things to it
\- withouth needing to build a new image -> stopping the runing container -> starting new container
​
# DONE
​
\- I use the docker version of airflow 2.4.1
\- I have succesfully joined the Python Virtual environment to Airflow Docker as a volume you can see I in the docker-compose.yml
\- After restarting docker with the new yml file it works fine.
\- I can jump in to the container activate manually the python environment import and run python libraries perfectly fine.
​
# CHALLANGE
​
\- The problem comes when I try to run my test dag with the new venv2
\- The DAG works with the original external python environemnt that is installed via the Dockerfile but the goal would be to not to need this as mentioned before
\- My guess is that this error happens because the python environemnt does not activated.
​
# Files and ERRORS
​
docker-compose.yml
​
​
version: '3'
x-airflow-common:
&airflow-common
image: ${AIRFLOWIMAGENAME:-myown-image-apache/airflow:2.4.1}
build: .
environment:
&airflow-common-env
AIRFLOWCOREEXECUTOR: CeleryExecutor
AIRFLOWDATABASESQLALCHEMYCONN: NOTPUBLIC
#ORIGINAL: postgresql+psycopg2://airflow:airflow@postgres/airflow
# For backward compatibility, with Airflow <2.3
AIRFLOWCORESQLALCHEMYCONN: NOTPUBLIC
#ORIGINAL postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOWCELERYRESULTBACKEND: NOTPUBLIC
# ORIGINAL db+postgresql://airflow:airflow@postgres/airflow
AIRFLOWCELERYBROKERURL: redis://:@redis:1111/0
AIRFLOWCOREFERNETKEY: ''
AIRFLOWCOREDAGSAREPAUSEDATCREATION: 'true'
AIRFLOWCORELOADEXAMPLES: 'false'
AIRFLOWAPIAUTHBACKENDS: 'airflow.api.auth.backend.NOTPUBLIC'
PIPADDITIONALREQUIREMENTS: ${PIPADDITIONALREQUIREMENTS:-}
AIRFLOWCOREENABLEXCOMPICKLING: 'NOTPUBLIC'
AIRFLOWSMTPSMTPHOST: NOTPUBLIC
AIRFLOWSMTPSMTPPORT: 222
AIRF LOWSMTPSMTPUSER: "NOTPUBLIC"
AIRFLOWSMTPSMTPPASSWORD: NOTPUBLIC
AIRFLOWSMTPSMTPMAILFROM: [email protected]
AIRFLOWWEBSERVERBASEURL: NOTPUBLIC
AIRFLOWWEBSERVERWEBSERVERSSLCERT: /opt/airflow/certs/NOTPUBLIC.pem
AIRFLOWWEBSERVERWEBSERVERSSLKEY: /opt/airflow/certs/NOTPUBLIC.pem
AIRFLOWCOREMAXACTIVERUNSPERDAG: 1
AIRFLOWCOREDEFAULTTASKEXECUTIONTIMEOUT: 21600
AWSSNOWPLOWACCESSKEY: NOTPUBLIC
AWSSNOWPLOWSECRETKEY: NOTPUBLIC
AIRFLOWSCHEDULERMINFILEPROCESSINTERVAL: 180
#AIRFLOWSCHEDULERDAGDIRLISTINTERVAL: 600
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- routtofolder/NOTPUBLIC1:/opt/airflow/NOTPUBLIC1
- routtofolder/NOTPUBLIC2:/opt/airflow/NOTPUBLIC2
- /routtofolder/NOTPUBLIC3:/opt/airflow/NOTPUBLIC3
- ./venv2:/opt/airflow/venv2 #########################################THIS IS THE PROBLEMATIC PART
user: "${AIRFLOWUID:-50000}:0"
dependson:
&airflow-common-depends-on
redis:
condition: servicehealthy
postgres:
condition: servicehealthy
​
​
my example DAG that I want to work:
​
from future import annotations
import logging
import sys
import tempfile
from pprint import pprint
from datetime import timedelta
import pendulum
from airflow import DAG
from airflow.decorators import task
from airflow.operators.pythonoperator import PythonOperator
from airflow.models import Variable
import requests
from requests.auth
# GOAL
​
\- Have a local python environemnt that I can swap up and install things to it
\- withouth needing to build a new image -> stopping the runing container -> starting new container
​
# DONE
​
\- I use the docker version of airflow 2.4.1
\- I have succesfully joined the Python Virtual environment to Airflow Docker as a volume you can see I in the docker-compose.yml
\- After restarting docker with the new yml file it works fine.
\- I can jump in to the container activate manually the python environment import and run python libraries perfectly fine.
​
# CHALLANGE
​
\- The problem comes when I try to run my test dag with the new venv2
\- The DAG works with the original external python environemnt that is installed via the Dockerfile but the goal would be to not to need this as mentioned before
\- My guess is that this error happens because the python environemnt does not activated.
​
# Files and ERRORS
​
docker-compose.yml
​
​
version: '3'
x-airflow-common:
&airflow-common
image: ${AIRFLOWIMAGENAME:-myown-image-apache/airflow:2.4.1}
build: .
environment:
&airflow-common-env
AIRFLOWCOREEXECUTOR: CeleryExecutor
AIRFLOWDATABASESQLALCHEMYCONN: NOTPUBLIC
#ORIGINAL: postgresql+psycopg2://airflow:airflow@postgres/airflow
# For backward compatibility, with Airflow <2.3
AIRFLOWCORESQLALCHEMYCONN: NOTPUBLIC
#ORIGINAL postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOWCELERYRESULTBACKEND: NOTPUBLIC
# ORIGINAL db+postgresql://airflow:airflow@postgres/airflow
AIRFLOWCELERYBROKERURL: redis://:@redis:1111/0
AIRFLOWCOREFERNETKEY: ''
AIRFLOWCOREDAGSAREPAUSEDATCREATION: 'true'
AIRFLOWCORELOADEXAMPLES: 'false'
AIRFLOWAPIAUTHBACKENDS: 'airflow.api.auth.backend.NOTPUBLIC'
PIPADDITIONALREQUIREMENTS: ${PIPADDITIONALREQUIREMENTS:-}
AIRFLOWCOREENABLEXCOMPICKLING: 'NOTPUBLIC'
AIRFLOWSMTPSMTPHOST: NOTPUBLIC
AIRFLOWSMTPSMTPPORT: 222
AIRF LOWSMTPSMTPUSER: "NOTPUBLIC"
AIRFLOWSMTPSMTPPASSWORD: NOTPUBLIC
AIRFLOWSMTPSMTPMAILFROM: [email protected]
AIRFLOWWEBSERVERBASEURL: NOTPUBLIC
AIRFLOWWEBSERVERWEBSERVERSSLCERT: /opt/airflow/certs/NOTPUBLIC.pem
AIRFLOWWEBSERVERWEBSERVERSSLKEY: /opt/airflow/certs/NOTPUBLIC.pem
AIRFLOWCOREMAXACTIVERUNSPERDAG: 1
AIRFLOWCOREDEFAULTTASKEXECUTIONTIMEOUT: 21600
AWSSNOWPLOWACCESSKEY: NOTPUBLIC
AWSSNOWPLOWSECRETKEY: NOTPUBLIC
AIRFLOWSCHEDULERMINFILEPROCESSINTERVAL: 180
#AIRFLOWSCHEDULERDAGDIRLISTINTERVAL: 600
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- routtofolder/NOTPUBLIC1:/opt/airflow/NOTPUBLIC1
- routtofolder/NOTPUBLIC2:/opt/airflow/NOTPUBLIC2
- /routtofolder/NOTPUBLIC3:/opt/airflow/NOTPUBLIC3
- ./venv2:/opt/airflow/venv2 #########################################THIS IS THE PROBLEMATIC PART
user: "${AIRFLOWUID:-50000}:0"
dependson:
&airflow-common-depends-on
redis:
condition: servicehealthy
postgres:
condition: servicehealthy
​
​
my example DAG that I want to work:
​
from future import annotations
import logging
import sys
import tempfile
from pprint import pprint
from datetime import timedelta
import pendulum
from airflow import DAG
from airflow.decorators import task
from airflow.operators.pythonoperator import PythonOperator
from airflow.models import Variable
import requests
from requests.auth
How to get a joined volume Python Virtual environment to Airflow Docker working with external_python_task?
# GOAL
​
\- Have a local python environemnt that I can swap up and install things to it
\- withouth needing to build a new image -> stopping the runing container -> starting new container
​
# DONE
​
\- I use the docker version of airflow 2.4.1
\- I have succesfully joined the Python Virtual environment to Airflow Docker as a volume you can see I in the docker-compose.yml
\- After restarting docker with the new yml file it works fine.
\- I can jump in to the container activate manually the python environment import and run python libraries perfectly fine.
​
# CHALLANGE
​
\- The problem comes when I try to run my test dag with the new venv2
\- The DAG works with the original external python environemnt that is installed via the Dockerfile but the goal would be to not to need this as mentioned before
\- My guess is that this error happens because the python environemnt does not activated.
​
# Files and ERRORS
​
docker-compose.yml
​
​
version: '3'
x-airflow-common:
&airflow-common
image: ${AIRFLOW_IMAGE_NAME:-myown-image-apache/airflow:2.4.1}
build: .
environment:
&airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: NOTPUBLIC
#ORIGINAL: postgresql+psycopg2://airflow:airflow@postgres/airflow
# For backward compatibility, with Airflow <2.3
AIRFLOW__CORE__SQL_ALCHEMY_CONN: NOTPUBLIC
#ORIGINAL postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: NOTPUBLIC
# ORIGINAL db+postgresql://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:@redis:1111/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.NOTPUBLIC'
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
AIRFLOW__CORE__ENABLE_XCOM_PICKLING: 'NOTPUBLIC'
AIRFLOW__SMTP__SMTP_HOST: NOTPUBLIC
AIRFLOW__SMTP__SMTP_PORT: 222
AIRF LOW__SMTP__SMTP_USER: "NOTPUBLIC"
AIRFLOW__SMTP__SMTP_PASSWORD: NOTPUBLIC
AIRFLOW__SMTP__SMTP_MAIL_FROM: [email protected]
AIRFLOW__WEBSERVER__BASE_URL: NOTPUBLIC
AIRFLOW__WEBSERVER__WEB_SERVER_SSL_CERT: /opt/airflow/certs/NOTPUBLIC.pem
AIRFLOW__WEBSERVER__WEB_SERVER_SSL_KEY: /opt/airflow/certs/NOTPUBLIC.pem
AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG: 1
AIRFLOW__CORE__DEFAULT_TASK_EXECUTION_TIMEOUT: 21600
AWS_SNOWPLOW_ACCESS_KEY: NOTPUBLIC
AWS_SNOWPLOW_SECRET_KEY: NOTPUBLIC
AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL: 180
#AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 600
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- routtofolder/NOTPUBLIC1:/opt/airflow/NOTPUBLIC1
- routtofolder/NOTPUBLIC2:/opt/airflow/NOTPUBLIC2
- /routtofolder/NOTPUBLIC3:/opt/airflow/NOTPUBLIC3
- ./venv2:/opt/airflow/venv2 #########################################THIS IS THE PROBLEMATIC PART
user: "${AIRFLOW_UID:-50000}:0"
depends_on:
&airflow-common-depends-on
redis:
condition: service_healthy
postgres:
condition: service_healthy
​
​
my example DAG that I want to work:
​
from __future__ import annotations
import logging
import sys
import tempfile
from pprint import pprint
from datetime import timedelta
import pendulum
from airflow import DAG
from airflow.decorators import task
from airflow.operators.python_operator import PythonOperator
from airflow.models import Variable
import requests
from requests.auth
# GOAL
​
\- Have a local python environemnt that I can swap up and install things to it
\- withouth needing to build a new image -> stopping the runing container -> starting new container
​
# DONE
​
\- I use the docker version of airflow 2.4.1
\- I have succesfully joined the Python Virtual environment to Airflow Docker as a volume you can see I in the docker-compose.yml
\- After restarting docker with the new yml file it works fine.
\- I can jump in to the container activate manually the python environment import and run python libraries perfectly fine.
​
# CHALLANGE
​
\- The problem comes when I try to run my test dag with the new venv2
\- The DAG works with the original external python environemnt that is installed via the Dockerfile but the goal would be to not to need this as mentioned before
\- My guess is that this error happens because the python environemnt does not activated.
​
# Files and ERRORS
​
docker-compose.yml
​
​
version: '3'
x-airflow-common:
&airflow-common
image: ${AIRFLOW_IMAGE_NAME:-myown-image-apache/airflow:2.4.1}
build: .
environment:
&airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: NOTPUBLIC
#ORIGINAL: postgresql+psycopg2://airflow:airflow@postgres/airflow
# For backward compatibility, with Airflow <2.3
AIRFLOW__CORE__SQL_ALCHEMY_CONN: NOTPUBLIC
#ORIGINAL postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: NOTPUBLIC
# ORIGINAL db+postgresql://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:@redis:1111/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.NOTPUBLIC'
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
AIRFLOW__CORE__ENABLE_XCOM_PICKLING: 'NOTPUBLIC'
AIRFLOW__SMTP__SMTP_HOST: NOTPUBLIC
AIRFLOW__SMTP__SMTP_PORT: 222
AIRF LOW__SMTP__SMTP_USER: "NOTPUBLIC"
AIRFLOW__SMTP__SMTP_PASSWORD: NOTPUBLIC
AIRFLOW__SMTP__SMTP_MAIL_FROM: [email protected]
AIRFLOW__WEBSERVER__BASE_URL: NOTPUBLIC
AIRFLOW__WEBSERVER__WEB_SERVER_SSL_CERT: /opt/airflow/certs/NOTPUBLIC.pem
AIRFLOW__WEBSERVER__WEB_SERVER_SSL_KEY: /opt/airflow/certs/NOTPUBLIC.pem
AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG: 1
AIRFLOW__CORE__DEFAULT_TASK_EXECUTION_TIMEOUT: 21600
AWS_SNOWPLOW_ACCESS_KEY: NOTPUBLIC
AWS_SNOWPLOW_SECRET_KEY: NOTPUBLIC
AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL: 180
#AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 600
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- routtofolder/NOTPUBLIC1:/opt/airflow/NOTPUBLIC1
- routtofolder/NOTPUBLIC2:/opt/airflow/NOTPUBLIC2
- /routtofolder/NOTPUBLIC3:/opt/airflow/NOTPUBLIC3
- ./venv2:/opt/airflow/venv2 #########################################THIS IS THE PROBLEMATIC PART
user: "${AIRFLOW_UID:-50000}:0"
depends_on:
&airflow-common-depends-on
redis:
condition: service_healthy
postgres:
condition: service_healthy
​
​
my example DAG that I want to work:
​
from __future__ import annotations
import logging
import sys
import tempfile
from pprint import pprint
from datetime import timedelta
import pendulum
from airflow import DAG
from airflow.decorators import task
from airflow.operators.python_operator import PythonOperator
from airflow.models import Variable
import requests
from requests.auth
import HTTPBasicAuth
my_default_args = {
'owner': 'Anonymus',
'email': ['[email protected]'],
'email_on_failure': True,
'email_on_retry': False,
}
with DAG(
dag_id='test_connected_env',
schedule='10 10 * * *',
start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
catchup=False,
#execution_timeout=timedelta(seconds=60),
default_args=my_default_args,
tags=['sample_tag', 'sample_tag2'],
) as dag:
#@task.external_python(task_id="test_external_python_venv_task", python=os.fspath(sys.executable)) # ORIGINAL
#@task.external_python(task_id="test_connected_env_task", python='/opt/airflow/venv1/bin/python3') ### installed via pip via Dockerfile, this works perfectly fine
u/task.external_python(task_id="test_connected_env_task", python='/opt/airflow/venv2/bin/python3')
def go(): # this could be any function name
#import package here
print("My Start")
#if you want to test the error
# print(1+"Airflow")
import pandas as pd
print(pd.DataFrame({'a':[1,2,3], 'b':[4,5,6]}))
import numpy as np
print(np.array([1,2,3]))
return print('my end')
external_python_task = go()
​
​
​
ERROR that I get:
​
*** Reading local file: /opt/airflow/logs/dag_id=test_connected_env/run_id=manual__2023-03-02T14:15:16.674123+00:00/task_id=test_connected_env_task/attempt=1.log
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: test_connected_env.test_connected_env_task manual__2023-03-02T14:15:16.674123+00:00 [queued]>
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: test_connected_env.test_connected_env_task manual__2023-03-02T14:15:16.674123+00:00 [queued]>
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1362} INFO -
--------------------------------------------------------------------------------
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1363} INFO - Starting attempt 1 of 1
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1364} INFO -
--------------------------------------------------------------------------------
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1383} INFO - Executing <Task(_PythonExternalDecoratedOperator): test_connected_env_task> on 2023-03-02 14:15:16.674123+00:00
[2023-03-02, 14:15:18 GMT] {standard_task_runner.py:54} INFO - Started process 15812 to run task
[2023-03-02, 14:15:18 GMT] {standard_task_runner.py:82} INFO - Running: ['airflow', 'tasks', 'run', 'test_connected_env', 'test_connected_env_task', 'manual__2023-03-02T14:15:16.674123+00:00', '--job-id', '142443', '--raw', '--subdir', 'DAGS_FOLDER/test_connected_env_task.py', '--cfg-path', '/tmp/tmp1t0wy5hy']
[2023-03-02, 14:15:18 GMT] {standard_task_runner.py:83} INFO - Job 142443: Subtask test_connected_env_task
[2023-03-02, 14:15:18 GMT] {dagbag.py:525} INFO - Filling up the DagBag from /opt/airflow/dags/test_connected_env_task.py
[2023-03-02, 14:15:18 GMT] {task_command.py:384} INFO - Running <TaskInstance: test_connected_env.test_connected_env_task manual__2023-03-02T14:15:16.674123+00:00 [running]> on host 0ad620763627
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1590} INFO - Exporting the following env vars:
[email protected]
AIRFLOW_CTX_DAG_OWNER=Anonymus
AIRFLOW_CTX_DAG_ID=test_connected_env
AIRFLOW_CTX_TASK_ID=test_connected_env_task
AIRFLOW_CTX_EXECUTION_DATE=2023-03-02T14:15:16.674123+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=manual__2023-03-02T14:15:16.674123+00:00
[2023-03-02, 14:15:18 GMT] {python.py:725} WARNING - When checking for Airflow installed in venv got Command '['/opt/airflow/venv2/bin/python3', '-c', 'from airflow import version; print(version.version)']' returned non-zero exit status
my_default_args = {
'owner': 'Anonymus',
'email': ['[email protected]'],
'email_on_failure': True,
'email_on_retry': False,
}
with DAG(
dag_id='test_connected_env',
schedule='10 10 * * *',
start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
catchup=False,
#execution_timeout=timedelta(seconds=60),
default_args=my_default_args,
tags=['sample_tag', 'sample_tag2'],
) as dag:
#@task.external_python(task_id="test_external_python_venv_task", python=os.fspath(sys.executable)) # ORIGINAL
#@task.external_python(task_id="test_connected_env_task", python='/opt/airflow/venv1/bin/python3') ### installed via pip via Dockerfile, this works perfectly fine
u/task.external_python(task_id="test_connected_env_task", python='/opt/airflow/venv2/bin/python3')
def go(): # this could be any function name
#import package here
print("My Start")
#if you want to test the error
# print(1+"Airflow")
import pandas as pd
print(pd.DataFrame({'a':[1,2,3], 'b':[4,5,6]}))
import numpy as np
print(np.array([1,2,3]))
return print('my end')
external_python_task = go()
​
​
​
ERROR that I get:
​
*** Reading local file: /opt/airflow/logs/dag_id=test_connected_env/run_id=manual__2023-03-02T14:15:16.674123+00:00/task_id=test_connected_env_task/attempt=1.log
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: test_connected_env.test_connected_env_task manual__2023-03-02T14:15:16.674123+00:00 [queued]>
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: test_connected_env.test_connected_env_task manual__2023-03-02T14:15:16.674123+00:00 [queued]>
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1362} INFO -
--------------------------------------------------------------------------------
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1363} INFO - Starting attempt 1 of 1
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1364} INFO -
--------------------------------------------------------------------------------
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1383} INFO - Executing <Task(_PythonExternalDecoratedOperator): test_connected_env_task> on 2023-03-02 14:15:16.674123+00:00
[2023-03-02, 14:15:18 GMT] {standard_task_runner.py:54} INFO - Started process 15812 to run task
[2023-03-02, 14:15:18 GMT] {standard_task_runner.py:82} INFO - Running: ['airflow', 'tasks', 'run', 'test_connected_env', 'test_connected_env_task', 'manual__2023-03-02T14:15:16.674123+00:00', '--job-id', '142443', '--raw', '--subdir', 'DAGS_FOLDER/test_connected_env_task.py', '--cfg-path', '/tmp/tmp1t0wy5hy']
[2023-03-02, 14:15:18 GMT] {standard_task_runner.py:83} INFO - Job 142443: Subtask test_connected_env_task
[2023-03-02, 14:15:18 GMT] {dagbag.py:525} INFO - Filling up the DagBag from /opt/airflow/dags/test_connected_env_task.py
[2023-03-02, 14:15:18 GMT] {task_command.py:384} INFO - Running <TaskInstance: test_connected_env.test_connected_env_task manual__2023-03-02T14:15:16.674123+00:00 [running]> on host 0ad620763627
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1590} INFO - Exporting the following env vars:
[email protected]
AIRFLOW_CTX_DAG_OWNER=Anonymus
AIRFLOW_CTX_DAG_ID=test_connected_env
AIRFLOW_CTX_TASK_ID=test_connected_env_task
AIRFLOW_CTX_EXECUTION_DATE=2023-03-02T14:15:16.674123+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=manual__2023-03-02T14:15:16.674123+00:00
[2023-03-02, 14:15:18 GMT] {python.py:725} WARNING - When checking for Airflow installed in venv got Command '['/opt/airflow/venv2/bin/python3', '-c', 'from airflow import version; print(version.version)']' returned non-zero exit status
1.
[2023-03-02, 14:15:18 GMT] {python.py:726} WARNING - This means that Airflow is not properly installed by /opt/airflow/venv2/bin/python3. Airflow context keys will not be available. Please Install Airflow 2.4.1 in your environment to access them.
[2023-03-02, 14:15:18 GMT] {process_utils.py:179} INFO - Executing cmd: /opt/airflow/venv2/bin/python3 /tmp/tmdqmf6q9rg/script.py /tmp/tmdqmf6q9rg/script.in /tmp/tmdqmf6q9rg/script.out /tmp/tmdqmf6q9rg/string_args.txt
[2023-03-02, 14:15:18 GMT] {process_utils.py:183} INFO - Output:
[2023-03-02, 14:15:18 GMT] {process_utils.py:187} INFO - My Start
[2023-03-02, 14:15:18 GMT] {process_utils.py:187} INFO - Traceback (most recent call last):
[2023-03-02, 14:15:18 GMT] {process_utils.py:187} INFO - File "/tmp/tmdqmf6q9rg/script.py", line 38, in <module>
[2023-03-02, 14:15:18 GMT] {process_utils.py:187} INFO - res = go(*arg_dict["args"], **arg_dict["kwargs"])
[2023-03-02, 14:15:18 GMT] {process_utils.py:187} INFO - File "/tmp/tmdqmf6q9rg/script.py", line 30, in go
[2023-03-02, 14:15:18 GMT] {process_utils.py:187} INFO - import pandas as pd
[2023-03-02, 14:15:18 GMT] {process_utils.py:187} INFO - ModuleNotFoundError: No module named 'pandas'
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1851} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/decorators/base.py", line 188, in execute
return_value = super().execute(context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 370, in execute
return super().execute(context=serializable_context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 175, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 678, in execute_callable
return self._execute_python_callable_in_subprocess(python_path, tmp_path)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 426, in _execute_python_callable_in_subprocess
execute_in_subprocess(
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/process_utils.py", line 168, in execute_in_subprocess
execute_in_subprocess_with_kwargs(cmd, cwd=cwd)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/process_utils.py", line 191, in execute_in_subprocess_with_kwargs
raise subprocess.CalledProcessError(exit_code, cmd)
subprocess.CalledProcessError: Command '['/opt/airflow/venv2/bin/python3', '/tmp/tmdqmf6q9rg/script.py', '/tmp/tmdqmf6q9rg/script.in', '/tmp/tmdqmf6q9rg/script.out', '/tmp/tmdqmf6q9rg/string_args.txt']' returned non-zero exit status 1.
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1401} INFO - Marking task as FAILED. dag_id=test_connected_env, task_id=test_connected_env_task, execution_date=20230302T141516, start_date=20230302T141518, end_date=20230302T141518
[2023-03-02, 14:15:18 GMT] {warnings.py:109} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/utils/email.py:120: RemovedInAirflow3Warning: Fetching SMTP credentials from configuration variables will be deprecated in a future release. Please set credentials using a connection instead.
send_mime_email(e_from=mail_from, e_to=recipients, mime_msg=msg, conn_id=conn_id, dryrun=dryrun)
[2023-03-02, 14:15:18 GMT] {email.py:229} INFO - Email alerting: attempt 1
[2023-03-02, 14:15:18 GMT] {email.py:241} INFO - Sent an alert email to ['[email protected]']
[2023-03-02, 14:15:18 GMT] {standard_task_runner.py:102} ERROR - Failed to execute job NONPUBLIC for task test_connected_env_task (Command '['/opt/airflow/venv2/bin/python3', '/tmp/tmdqmf6q9rg/script.py', '/tmp/tmdqmf6q9rg/script.in', '/tmp/tmdqmf6q9rg/script.out', '/tmp/tmdqmf6q9rg/string_args.txt']' returned non-zero exit status 1.; 15812)
[2023-03-02, 14:15:18 GMT]
[2023-03-02, 14:15:18 GMT] {python.py:726} WARNING - This means that Airflow is not properly installed by /opt/airflow/venv2/bin/python3. Airflow context keys will not be available. Please Install Airflow 2.4.1 in your environment to access them.
[2023-03-02, 14:15:18 GMT] {process_utils.py:179} INFO - Executing cmd: /opt/airflow/venv2/bin/python3 /tmp/tmdqmf6q9rg/script.py /tmp/tmdqmf6q9rg/script.in /tmp/tmdqmf6q9rg/script.out /tmp/tmdqmf6q9rg/string_args.txt
[2023-03-02, 14:15:18 GMT] {process_utils.py:183} INFO - Output:
[2023-03-02, 14:15:18 GMT] {process_utils.py:187} INFO - My Start
[2023-03-02, 14:15:18 GMT] {process_utils.py:187} INFO - Traceback (most recent call last):
[2023-03-02, 14:15:18 GMT] {process_utils.py:187} INFO - File "/tmp/tmdqmf6q9rg/script.py", line 38, in <module>
[2023-03-02, 14:15:18 GMT] {process_utils.py:187} INFO - res = go(*arg_dict["args"], **arg_dict["kwargs"])
[2023-03-02, 14:15:18 GMT] {process_utils.py:187} INFO - File "/tmp/tmdqmf6q9rg/script.py", line 30, in go
[2023-03-02, 14:15:18 GMT] {process_utils.py:187} INFO - import pandas as pd
[2023-03-02, 14:15:18 GMT] {process_utils.py:187} INFO - ModuleNotFoundError: No module named 'pandas'
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1851} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/decorators/base.py", line 188, in execute
return_value = super().execute(context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 370, in execute
return super().execute(context=serializable_context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 175, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 678, in execute_callable
return self._execute_python_callable_in_subprocess(python_path, tmp_path)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 426, in _execute_python_callable_in_subprocess
execute_in_subprocess(
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/process_utils.py", line 168, in execute_in_subprocess
execute_in_subprocess_with_kwargs(cmd, cwd=cwd)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/process_utils.py", line 191, in execute_in_subprocess_with_kwargs
raise subprocess.CalledProcessError(exit_code, cmd)
subprocess.CalledProcessError: Command '['/opt/airflow/venv2/bin/python3', '/tmp/tmdqmf6q9rg/script.py', '/tmp/tmdqmf6q9rg/script.in', '/tmp/tmdqmf6q9rg/script.out', '/tmp/tmdqmf6q9rg/string_args.txt']' returned non-zero exit status 1.
[2023-03-02, 14:15:18 GMT] {taskinstance.py:1401} INFO - Marking task as FAILED. dag_id=test_connected_env, task_id=test_connected_env_task, execution_date=20230302T141516, start_date=20230302T141518, end_date=20230302T141518
[2023-03-02, 14:15:18 GMT] {warnings.py:109} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/utils/email.py:120: RemovedInAirflow3Warning: Fetching SMTP credentials from configuration variables will be deprecated in a future release. Please set credentials using a connection instead.
send_mime_email(e_from=mail_from, e_to=recipients, mime_msg=msg, conn_id=conn_id, dryrun=dryrun)
[2023-03-02, 14:15:18 GMT] {email.py:229} INFO - Email alerting: attempt 1
[2023-03-02, 14:15:18 GMT] {email.py:241} INFO - Sent an alert email to ['[email protected]']
[2023-03-02, 14:15:18 GMT] {standard_task_runner.py:102} ERROR - Failed to execute job NONPUBLIC for task test_connected_env_task (Command '['/opt/airflow/venv2/bin/python3', '/tmp/tmdqmf6q9rg/script.py', '/tmp/tmdqmf6q9rg/script.in', '/tmp/tmdqmf6q9rg/script.out', '/tmp/tmdqmf6q9rg/string_args.txt']' returned non-zero exit status 1.; 15812)
[2023-03-02, 14:15:18 GMT]
{local_task_job.py:164} INFO - Task exited with return code 1
[2023-03-02, 14:15:18 GMT] {local_task_job.py:273} INFO - 0 downstream tasks scheduled from follow-on schedule check
https://redd.it/11g5j1e
@r_devops
[2023-03-02, 14:15:18 GMT] {local_task_job.py:273} INFO - 0 downstream tasks scheduled from follow-on schedule check
https://redd.it/11g5j1e
@r_devops
Reddit
r/devops on Reddit: How to get a joined volume Python Virtual environment to Airflow Docker working with external_python_task?
Posted by u/glassAlloy - No votes and no comments
Devops age restrictions
Hello guys, do employees look for employees in 20s 30s to work as devops Or can you move up to dev ops in your 40s 50s ? What your expiernece been like.
https://redd.it/11g5ng9
@r_devops
Hello guys, do employees look for employees in 20s 30s to work as devops Or can you move up to dev ops in your 40s 50s ? What your expiernece been like.
https://redd.it/11g5ng9
@r_devops
Reddit
r/devops on Reddit: Devops age restrictions
Posted by u/Titanguru7 - No votes and 8 comments
S3 bucket lifecycle policy
Can we test the S3 bucket lifecycle policy in a dry run immediately?
I have a policy that deletes files older than 30 days. I want to test it immediately to ensure that the policy works Is it possible?
https://redd.it/11g7ffl
@r_devops
Can we test the S3 bucket lifecycle policy in a dry run immediately?
I have a policy that deletes files older than 30 days. I want to test it immediately to ensure that the policy works Is it possible?
https://redd.it/11g7ffl
@r_devops
Reddit
r/devops on Reddit: S3 bucket lifecycle policy
Posted by u/anacondaonline - No votes and 3 comments
Help Me Upgrade My DevOps/K8S Communities?
I remember the good old days where I could sit in an IRC channel with like minded people and have open conversations about topics related to the channels.
There was no marketing, blog post links, just a bunch of people with similar interests hanging out and periodically discussing relevant things.
I keep seeking to replicate this experience, but I find that almost anything labeled with DevOps or K8S has become just a marketing channel for weak marketing content, posts for webinars, and basically trying to force content to me rather than engage in discussions.
For example, just take a look at the #devops feed on Twitter - it feels like miles of billboards for devops companies.
LinkedIn is similarly flooded with links to marketing content that contains just enough buzzwords to catch the Google bots, but to leave the reader feeling more lost than ever.
Are there any good Slack/Discord/IRC communities?
Dev.to used to be appealing but now I feel like it's just a lot of useless content as well.
I'd love to join something and contribute my experiences as well as help others where I can!
Would you please share your top DevOps/K8S communities with me?
https://redd.it/11di9fy
@r_devops
I remember the good old days where I could sit in an IRC channel with like minded people and have open conversations about topics related to the channels.
There was no marketing, blog post links, just a bunch of people with similar interests hanging out and periodically discussing relevant things.
I keep seeking to replicate this experience, but I find that almost anything labeled with DevOps or K8S has become just a marketing channel for weak marketing content, posts for webinars, and basically trying to force content to me rather than engage in discussions.
For example, just take a look at the #devops feed on Twitter - it feels like miles of billboards for devops companies.
LinkedIn is similarly flooded with links to marketing content that contains just enough buzzwords to catch the Google bots, but to leave the reader feeling more lost than ever.
Are there any good Slack/Discord/IRC communities?
Dev.to used to be appealing but now I feel like it's just a lot of useless content as well.
I'd love to join something and contribute my experiences as well as help others where I can!
Would you please share your top DevOps/K8S communities with me?
https://redd.it/11di9fy
@r_devops
DEV Community
A space to discuss and keep up software development and manage your software career
Is GitLab Premium worth it at its new price?
Per GitLab's blog, GitLab Premium is getting a price hike from $19/user/month to $29/user/month, with a "transitional" price of $24/user/month.
Their article talks about all the features they've added to Premium since 2018, but I feel like the company has changed significantly since it went public. We've seen no movement on any of the features we care about - mainly related to packaging - and I can't remember the last time that Premium actually got a new feature we appreciated.
Am I alone in feeling like its value is running a bit thin at this price? Do competing products provide a better value than GitLab Premium at its new price?
https://redd.it/11gadwc
@r_devops
Per GitLab's blog, GitLab Premium is getting a price hike from $19/user/month to $29/user/month, with a "transitional" price of $24/user/month.
Their article talks about all the features they've added to Premium since 2018, but I feel like the company has changed significantly since it went public. We've seen no movement on any of the features we care about - mainly related to packaging - and I can't remember the last time that Premium actually got a new feature we appreciated.
Am I alone in feeling like its value is running a bit thin at this price? Do competing products provide a better value than GitLab Premium at its new price?
https://redd.it/11gadwc
@r_devops
about.gitlab.com
New pricing for GitLab Premium | GitLab
Learn more about the GitLab Premium updates.
Best way to provision a vm in hyper v?
The current approach looks like the following.
Pipeline agent builds a golden image with Packer and uploads it to an image repository. Then Terraform connects via winrm to the hyper v host, downloads the prepared .vhdx file from the image repository and creates the vm with the image attached to it.
Any suggestions to improve this process? I am not sure if something like WDS or Powershell DSC will be better to provision the VMs.
https://redd.it/11gcvey
@r_devops
The current approach looks like the following.
Pipeline agent builds a golden image with Packer and uploads it to an image repository. Then Terraform connects via winrm to the hyper v host, downloads the prepared .vhdx file from the image repository and creates the vm with the image attached to it.
Any suggestions to improve this process? I am not sure if something like WDS or Powershell DSC will be better to provision the VMs.
https://redd.it/11gcvey
@r_devops
Reddit
r/devops on Reddit: Best way to provision a vm in hyper v?
Posted by u/Ok-Mine-6491 - No votes and no comments
How to exempt github actions bot from the rules of the protected brances
I developed a workflow with github actions that for each push builds some stuff and it is supposed to push the changes automatically. This worked fine until the workflow had to be merged on a branch which was protected. and did not allow pushing without a pull request first. Is there any way I could allow to bot to carry on with the push, any workaround?
I did find an action that creates a pull request automatically to the branch you push to, from another branch that this action creates (and then deletes). So I could use this with the 2 default branches in my repo and everytime something pushes into those, the action creates a separate branch and generates a PR from there. But this was not the approach I wanted because I wanted this thing to get built on every push.
Is there really no workaround on how to exempt github actions bot from being counted in the protection rules of the branches? I know we can exept users in the organization, but I really dont want to add a user as the author only for this reason.
https://redd.it/11gdhxx
@r_devops
I developed a workflow with github actions that for each push builds some stuff and it is supposed to push the changes automatically. This worked fine until the workflow had to be merged on a branch which was protected. and did not allow pushing without a pull request first. Is there any way I could allow to bot to carry on with the push, any workaround?
I did find an action that creates a pull request automatically to the branch you push to, from another branch that this action creates (and then deletes). So I could use this with the 2 default branches in my repo and everytime something pushes into those, the action creates a separate branch and generates a PR from there. But this was not the approach I wanted because I wanted this thing to get built on every push.
Is there really no workaround on how to exempt github actions bot from being counted in the protection rules of the branches? I know we can exept users in the organization, but I really dont want to add a user as the author only for this reason.
https://redd.it/11gdhxx
@r_devops
Reddit
r/devops on Reddit: How to exempt github actions bot from the rules of the protected brances
Posted by u/Acrobatic-Ad-6556 - No votes and 1 comment
What is the best open-source CI/CD platform?
I've used Gitlab in the past, but that is not open-source to the best of my understanding. And I don't like Jenkins and had a lot of confusion around installing CircleCI, any other suggestions?
https://redd.it/11gez33
@r_devops
I've used Gitlab in the past, but that is not open-source to the best of my understanding. And I don't like Jenkins and had a lot of confusion around installing CircleCI, any other suggestions?
https://redd.it/11gez33
@r_devops
Reddit
r/devops on Reddit: What is the best open-source CI/CD platform?
Posted by u/pred135 - No votes and 2 comments
KodeKloud or ACloudGuru
Hi, I need hands on training on Docker, Kubernetes/Kubectl, Jenkins, Terraform, Azure and possibly soon AWS since my company is about to migrate into it. I'm not chasing any certifications and I'm not looking to be a pro in a short mean of time. I just need practical knowledge of general DevOps toolchain. Would you say KodeKloud has better designed hands on labs than ACloudGuru? I brought these two because they are also offering a discount in their plans right now and they seem to be most popular ones also. I see majority of SE people on Reddit preferes KodeKloud but just wanted to get a direct recommendation for a learning platform like these? Thanks a lot!
https://redd.it/11gbhtj
@r_devops
Hi, I need hands on training on Docker, Kubernetes/Kubectl, Jenkins, Terraform, Azure and possibly soon AWS since my company is about to migrate into it. I'm not chasing any certifications and I'm not looking to be a pro in a short mean of time. I just need practical knowledge of general DevOps toolchain. Would you say KodeKloud has better designed hands on labs than ACloudGuru? I brought these two because they are also offering a discount in their plans right now and they seem to be most popular ones also. I see majority of SE people on Reddit preferes KodeKloud but just wanted to get a direct recommendation for a learning platform like these? Thanks a lot!
https://redd.it/11gbhtj
@r_devops
Reddit
r/devops on Reddit: KodeKloud or ACloudGuru
Posted by u/davegurney2 - No votes and 6 comments
DevOps challenges - what do you think? can you contribute ?
Hi people, today I bring to us interest topic (at least for me). Surely, sometime in your career you had concern about side/pet projects or "how can I increase my technical skills?". So, like a frontend engineer cloning a Twitter or some social medial, what kind of challenges you consider interest for DevOps engineer? I mean, in my case I think something like that:
Put an API in some git-based source code repository hosting (BitBucket, GitHub, GitLab)
Create a Pipeline with some CI/CD tool (Bitbucket pipelines, GitHub Actions, Jenkins)
Setup a VM in some cloud provider (AWS, Azure, GCP)
Deploy this API in the VM mentioned above
and so on..
have you seen some github repository with this challenges?
can you share some challenge that you consider interesting?
greetings :)
https://redd.it/11glzlq
@r_devops
Hi people, today I bring to us interest topic (at least for me). Surely, sometime in your career you had concern about side/pet projects or "how can I increase my technical skills?". So, like a frontend engineer cloning a Twitter or some social medial, what kind of challenges you consider interest for DevOps engineer? I mean, in my case I think something like that:
Put an API in some git-based source code repository hosting (BitBucket, GitHub, GitLab)
Create a Pipeline with some CI/CD tool (Bitbucket pipelines, GitHub Actions, Jenkins)
Setup a VM in some cloud provider (AWS, Azure, GCP)
Deploy this API in the VM mentioned above
and so on..
have you seen some github repository with this challenges?
can you share some challenge that you consider interesting?
greetings :)
https://redd.it/11glzlq
@r_devops
Reddit
r/devops on Reddit: DevOps challenges - what do you think? can you contribute ?
Posted by u/FernandoJaimes - No votes and no comments