Reddit DevOps

How to get a joined volume Python Virtual environment to Airflow Docker working with externalpythontask?

# GOAL



\- Have a local python environemnt that I can swap up and install things to it

\- withouth needing to build a new image -> stopping the runing container -> starting new container



# DONE



\- I use the docker version of airflow 2.4.1

\- I have succesfully joined the Python Virtual environment to Airflow Docker as a volume you can see I in the docker-compose.yml

\- After restarting docker with the new yml file it works fine.

\- I can jump in to the container activate manually the python environment import and run python libraries perfectly fine.



# CHALLANGE



\- The problem comes when I try to run my test dag with the new venv2

\- The DAG works with the original external python environemnt that is installed via the Dockerfile but the goal would be to not to need this as mentioned before

\- My guess is that this error happens because the python environemnt does not activated.



# Files and ERRORS



docker-compose.yml





version: '3'
x-airflow-common:
&airflow-common
image: ${AIRFLOWIMAGENAME:-myown-image-apache/airflow:2.4.1}
build: .
environment:
&airflow-common-env
AIRFLOWCOREEXECUTOR: CeleryExecutor
AIRFLOWDATABASESQLALCHEMYCONN: NOTPUBLIC
#ORIGINAL: postgresql+psycopg2://airflow:airflow@postgres/airflow
# For backward compatibility, with Airflow <2.3
AIRFLOWCORESQLALCHEMYCONN: NOTPUBLIC
#ORIGINAL postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOWCELERYRESULTBACKEND: NOTPUBLIC
# ORIGINAL db+postgresql://airflow:airflow@postgres/airflow
AIRFLOWCELERYBROKERURL: redis://:@redis:1111/0
AIRFLOWCOREFERNETKEY: ''
AIRFLOWCOREDAGSAREPAUSEDATCREATION: 'true'
AIRFLOWCORELOADEXAMPLES: 'false'
AIRFLOWAPIAUTHBACKENDS: 'airflow.api.auth.backend.NOTPUBLIC'
PIPADDITIONALREQUIREMENTS: ${PIPADDITIONALREQUIREMENTS:-}
AIRFLOWCOREENABLEXCOMPICKLING: 'NOTPUBLIC'
AIRFLOWSMTPSMTPHOST: NOTPUBLIC
AIRFLOWSMTPSMTPPORT: 222
AIRF LOWSMTPSMTPUSER: "NOTPUBLIC"
AIRFLOWSMTPSMTPPASSWORD: NOTPUBLIC
AIRFLOWSMTPSMTPMAILFROM: [email protected]
AIRFLOWWEBSERVERBASEURL: NOTPUBLIC
AIRFLOWWEBSERVERWEBSERVERSSLCERT: /opt/airflow/certs/NOTPUBLIC.pem
AIRFLOWWEBSERVERWEBSERVERSSLKEY: /opt/airflow/certs/NOTPUBLIC.pem
AIRFLOWCOREMAXACTIVERUNSPERDAG: 1
AIRFLOWCOREDEFAULTTASKEXECUTIONTIMEOUT: 21600
AWSSNOWPLOWACCESSKEY: NOTPUBLIC
AWSSNOWPLOWSECRETKEY: NOTPUBLIC
AIRFLOWSCHEDULERMINFILEPROCESSINTERVAL: 180
#AIRFLOWSCHEDULERDAGDIRLISTINTERVAL: 600
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- routtofolder/NOTPUBLIC1:/opt/airflow/NOTPUBLIC1
- routtofolder/NOTPUBLIC2:/opt/airflow/NOTPUBLIC2
- /routtofolder/NOTPUBLIC3:/opt/airflow/NOTPUBLIC3
- ./venv2:/opt/airflow/venv2 #########################################THIS IS THE PROBLEMATIC PART
user: "${AIRFLOWUID:-50000}:0"
dependson:
&airflow-common-depends-on
redis:
condition: servicehealthy
postgres:
condition: servicehealthy

&#x200B;

&#x200B;

my example DAG that I want to work:

&#x200B;

from future import annotations

import logging
import sys
import tempfile
from pprint import pprint
from datetime import timedelta
import pendulum
from airflow import DAG
from airflow.decorators import task
from airflow.operators.pythonoperator import PythonOperator
from airflow.models import Variable
import requests
from requests.auth

4 views15:33