Configuring nginx with docker-compose

I have a simple app of 3 containers which all run in the same AWS EC2 server. I want to configure Nginx to act as a reverse-proxy serving the same domain however I'm pretty new with Nginx and don't know how to set the conf file correctly.

Here is my docker-compose file:

version: "3"
services:

nginx:
container_name: nginx
image: nginx:latest
ports:
- "80:80"
volumes:
- ./conf/nginx.conf:/etc/nginx/nginx.conf

frontend:
container_name: frontend
image: myfrontend:image
ports:
- "3000:3000"

backend:
container_name: backend
depends_on:
- db
environment:
DB_HOST: db
image: mybackend:image
ports:
- "8400:8400"

db:
container_name: mongodb
environment:
MONGO_INITDB_DATABASE: myDB
image: mongo:latest
ports:
- "27017:27017"
volumes:
- ./initialization/db:/docker-entrypoint-initdb.d
- db-volume:/data/db

volumes:
db-volume:

The backend fetches data from the database and sends it to be presented by the frontend.

Here is what I tried to do with my nginx.conf file (which is obviously wrong):

events {
worker_connections 4096;
}

http {
server {
listen 80;
listen [::]:80;

server_name myDomainName.com;

location / {
proxy_pass https://frontend:3000/;
proxy_set_header Host $host;
}

location / {
proxy_pass https://backend:8400/;
proxy_pass_request_headers on;
}

}
}

Any help would be greatly appreciated.Note: I want all containers to run behind the same domain name

https://redd.it/faxz3q
@r_devops

Getting an SSL error when trying to push my Kafka Message to the Cloud via my python script.

I've followed all of the proper instructions via the Aiven Getting Started Page (I'm using their script as a skeleton) & even their youtube tutorial

[https://www.youtube.com/watch?v=QBFWgvudgaE](https://www.youtube.com/watch?v=QBFWgvudgaE)

[https://help.aiven.io/en/articles/489572-getting-started-with-aiven-kafka](https://help.aiven.io/en/articles/489572-getting-started-with-aiven-kafka)

Here's my code:

​

# This script connects to Kafka and send a few messages

from kafka import KafkaProducer

producer = KafkaProducer(
bootstrap_servers="kafka-385d27c1-mkramer789-8285.aivencloud.com:29668",
security_protocol="SSL",
ssl_cafile="/Users/mike/Desktop/AivenKeys/ca.pem",
ssl_certfile="/Users/mike/Desktop/AivenKeys/service.cert",
ssl_keyfile="/Users/mike/Desktop/AivenKeys/client.keystore.p12"
)

for i in range(1, 4):
message = "message number {}".format(i)
print("Sending: {}".format(message))
producer.send("demo-topic", message.encode("utf-8"))

# Force sending of all messages

producer.flush()

Heres the error:

Traceback (most recent call last):
File "aiven_producer.py", line 5, in <module>
producer = KafkaProducer(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/kafka/producer/kafka.py", line 380, in __init__
client = KafkaClient(metrics=self._metrics, metric_group_prefix='producer',
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/kafka/client_async.py", line 242, in __init__
self.config['api_version'] = self.check_version(timeout=check_timeout)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/kafka/client_async.py", line 907, in check_version
version = conn.check_version(timeout=remaining, strict=strict, topics=list(self.config['bootstrap_topics_filter']))
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/kafka/conn.py", line 1228, in check_version
if not self.connect_blocking(timeout_at - time.time()):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/kafka/conn.py", line 337, in connect_blocking
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/kafka/conn.py", line 398, in connect
self._wrap_ssl()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/kafka/conn.py", line 478, in _wrap_ssl
self._ssl_context.load_cert_chain(
ssl.SSLError: [SSL] PEM lib (_ssl.c:3965)

https://redd.it/fav8fh
@r_devops

Looking for some job advice. Been working for 3 months and I real haven’t done any DevOps work.

This is my first job. Very cool fintech company. I’m on a 6 month contract and I get payed $25hr. Ever since I got here I’ve been pulled in every direction doing everything but DevOps. I just got back from a business trip to a major client where I mostly performance tuned their environments and set them up to scale out. I pretty much by myself reduced their user wait times by a factor of 10 so I’m coming back with a great success. I had a great time doing this but it wasn’t even in the application I work in. After doing similar work for a few other clients I’m now the go to guy for solving client problems. Currently in my local office I am assigned critical work on almost every project. For example they’re tasking people to move our client hosted application to azure to sell prod side. They have this big org task chart with tons of business tasks then in the middle is a single box “put application on cloud” with my name above it. I have a few other DevOps that work in different countries but I’m always assigned the work because they don’t answer their emails.

The people I work with everyday are getting payed $150k-500k a year. I am very confident in my value an know that they do not want to lose me. Is my hourly pay something I can ask to renegotiate mid contract? I’ve been keeping track of all the major success I’ve accomplished and have a solid portfolio.

I’m not really doing any DevOps work. I’ve setup Jenkins to do auto deployments for QA but besides that I’m basically just a problem solver. I do a lot of work performance tuning and spend a large amount of time face to face/calling clients. Should I change my job title? What would I change it to? I actually love working with clients even the difficult ones.

TLDR
3 months into my first job on a contract. I am the go to guy for client issues and performance tuning in my company on the North American side and do almost no DevOps work in the project I was hired for. I get payed $25/hr and my desk mates are making 150k. Commonly the project managers from other projects will come and task me with major jobs. Is it crazy to ask to get paid more? Also should I change my job title to something more fitting for what I’ve accomplished?

https://redd.it/fasiib
@r_devops

Docker Swarm Tutorial | 3 Node: 1 Manager and 2 Worker Swarm Setup

Docker Swarm Tutorial | 3 Node: 1 Manager and 2 Worker Swarm Setup

Watch the full video here: [https://youtu.be/wLZ4j730bDo](https://youtu.be/wLZ4j730bDo)

\#DevOps , #CICD , #Docker, #Jenkins, #Pipeline, #Container,#thetips4you , #dockerswarm

https://redd.it/fay01n
@r_devops

Reduce redis costs in dev

Hi Devops

I work at a very large enterprise in the UK as a platform engineer. I'm very much new to the role and the nuances of the environment.

We currently have a setup that's dev - stage - prod and the way the pipeline is configured is that everything that's deployed to dev is eventually deployed to stage and prod. Infrastructure included. This is to avoid disparity between environments.

The problem thks brings with it is though is that the environments cost a fortune because the infrastructure is always deployed with the code.

Currently we are being stung with huge costs for redis infra in dev, and its hard to determine whether or not we can shutoff or delete these clusters.

My question is, is there a way to Mock redis in dev only so thst instead of provisioning infrastructure for redis we just fake it? And save the redis deployment for staging and prod?

https://redd.it/fb6l8d
@r_devops

How to learn about networking ?

Hello all,

I am a non-CS graduate devops engineer and I am seeing a lot of jobs out there require quite a bit of networking knowledge. At my current job I do a lot of development and don't really get to work with network layer. It is really a big gap in my knowledge and I'd like to start learning. Where do I start ? What kind of tasks I should be able to do ? Can we include networking and security related "getting into devops" links to the mega post ?

https://redd.it/faqiyo
@r_devops

Any tool chain to manage deployment/ roll out steps?

So in my organization we do roll out of tools (off the shelf).

There are like 40+ steps which need to be followed. The rollout itself us automated but the pre and post steps have to be done personally by a team member.

We use (unfortunately) excel for managing these steps.

I am wondering what kind of tool sets are available to kind of document these roll out steps and manage them (reuse for next releases, review etc).

How does your organization manages these?

https://redd.it/fb804g
@r_devops

Heroku to AWS Migration?

Hi, Imma final year at college doing some freelance development on the side.. I created the backend service for an app with Golang + Postgres stack that is hosted on my personal heroku account. How do i migrate it to AWS? Resarch tells me there are several options available like

a) Start an EC2 Instance with Postgres and Golang running

b) EC2 instance with golang and a Amazon RDS running postgres

c)Dockerize the whole thing and deploy

Which do you think is the best option or is there anything else altogether that i should know of

https://redd.it/faq31p
@r_devops

As a DevOps how do you deal with the management asking you to do Service Desk / Help Desk stuff?

I am seeking for advise. New user here and sorry if I am breaking any rules, feel free to delete this post if so.

Before reading further, I am aware that DevOps is a culture and not a job title, however I will be talking about a job title because that's what the job titles are named these days.

TL;DR: Those of you who work as a DevOps or Ops engineer, but at the company where you work for, apart from the usual DevOps tasks you are also expected to do the shitty Service Desk/Help Desk tasks (like fixing the laptops of the other incompetent coworkers, fix that video call, fix broken WiFis and broken LAN connectors, and other shitty stuff), how do you deal with it? Also have you ever tried to change this or you just leave?

&#x200B;

The long story:

I always use to ask these questions before I start working somewhere during the job interview. *"What is expected of me for this position? What kind of tasks are given to the team? How is success measured?"* The last question give's me an oversight how do they measure success and if they have a KPI which is also another shitty trend.

Thing is, the DevOps is really a mixed concept among the companies. Usually (most often, actually) a company posts a job as a DevOps but what they really need is a one man army, where you are expected to do everything from Service Desk and fixing other people laptops, broken LAN wall jacks, IT Procurement and then pps stuff containers, Kubernetes, Jenkins, Sysadmin things like fixing broken Linux servers, administering mail servers, user account access, managing the physical equipment in the datacenter and what else. Now, please do not misinterpret me here. It is not that I complain because of these tasks, the problem arises when ALL of them are given to a single team, and when ALL of them are measured in the KPI equally for all team members.

With my 10 years of experience I have knowledge of many legacy and modern technologies so I rarely get rejected from a job and when I get and answer like that, usually I am the one who rejects the position.

In reality few companies offer a job position that really is DevOps (or Ops), and in most occurrences they ask for a combination of Service Desk/Help Desk and Ops. Myself for an IT guy of 10 years, to me it makes absolutely no difference if someone comes to my desk with his laptop asking me "please install the printer" or if he comes and asks me to mop the floor in his office.

To summarise IMO, it is humiliating and disrespectful from the company to ask these kinds of tasks from experienced engineers.

Thus I come to the last part of my story and my current job position. I was referred into this company by a friend of mine (who is also my coworker right now and in the same team). During the interview I knew almost everything they asked me and in the end they gave me a demo task that was totally DevOps oriented. It was really a joy for me to work on it and I completed it even ahead of schedule. The salary they offered was really competitive so I took the offer.

All this pleasant experience was so good at the beginning that I completely forgot to ask the main three questions that I mentioned above, so guess what? I landed in the same type of environment that I was trying for so hard to avoid.

What could I do and how to deal with this without leaving? (I have my reasons of why I do not wish to leave at least for another 6 months)

https://redd.it/faauvl
@r_devops

Rancher as a Kubernetes Dashboard

So at work we have Kubernetes clusters being managed by Rancher. A bad experience with Rancher and certificates has left everyone there very sour with it. So there's a plan to migrate our clusters to EKS.

However, there are less technical people in our company that like having a visualization of the clusters that isn't just via kubectl. Heck, even I really like some features that Rancher has, like the "press the + button to deploy more pods" one.

So I was thinking... Would it be possible to have clusters running on EKS, and just having Rancher be like a Kubernetes Dashboard for them? Rancher wouldn't manage any of those clusters, it would only be an interface, kinda like the actual Kubernetes Dashboard.

The thing that is important is that authentication via LDAP is a must, and I found it easier to setup in Rancher than in Kubernetes Dashboard.

Is something like this possible? If yes, how hard/flexible is it to configure?

https://redd.it/fa6unj
@r_devops

Should I modify a AWS DMS task?

I have AWS DMS task running in CDC mode for sometime and now we need to sync two more tables which were not included initially. Should I stop and modify current task or should I create a new task for the new tables?

https://redd.it/fa7db7
@r_devops

(Dead) Snakes on a… Debian System

> Python 3.6 / 3.7 / 3.8 Debian packages for both Stretch and Buster

The Deadsnakes PPA project originally built older Python releases for Ubuntu, so you could e.g. run unit tests on a new release using a Python version found on older releases (i.e. the ‘dead’ snakes).

Nowadays, the project also builds newer Python versions ahead of what a certain release offers as its default.

The packages contain the minor Python version in their name (e.g. python3.6) and can thus be installed concurrently to the default python3 ones. Originally based on the Debian source packages, they can also be used on Debian and not just on Ubuntu.

[more…](https://jhermann.github.io/blog/python/deployment/2020/02/26/deadsnakes_on_debian.html)

https://redd.it/fbatbc
@r_devops

What exactly is an “Infrastructure Engineer”, and can one become a Junior Infrastructure Engineer within 10 months of study?

I’ve found a lot of conflicting information as to what it is, so I’m a bit confused as to exactly what it is.

https://redd.it/fa5kx8
@r_devops

Kubernetes Namespaces Explained in 15 mins

This is a [**short but thorough video**](https://youtu.be/K3jNo4z5Jx8) about Kubernetes Namespaces. **What they are and how they can help you manage your Kubernetes resources.**

In detail I'll go through the following topics:

* What are Kubernetes Namespaces?
* Explain the 4 Default Namespaces
* Why to use Namespaces with 4 Use Cases
* Characteristics of Namespaces
* Demo of how to create components in Namespaces
* How to change the active namespace, which makes it easier to work with namespaces

&#x200B;

Hope this tutorial is helpful for some of you 🙂 Also appreciate any feedback and suggestions.

https://redd.it/fbfpf4
@r_devops

Deploy to Kubernetes from CI/CD

Which is the best way for deploying to Kubernetes? Using Ansible, Helm or something else ? Also if I am not using a docker-compose file (and over that compose file Kompose for creating deployment and service files) is there a easy way for creating these deployment and services files from a java(spring) config yaml files(also know as application.yaml)

Also which is the best way for deploying to Kubernetes using Jenkins. Calling deployment and service files from Jenkinsfile or integrate Jenkins and Kubernetes with Helm or something else ?

https://redd.it/fbceyj
@r_devops

Intro to DevSecOps?

Any quality literature, website content or videos on implementing DevSecOps? I want to go beyond adding code scanning to a Pipeline. Any info would be appreciated.

https://redd.it/fbcpzl
@r_devops

Kubernetes isn't for me, what is?

I've been looking hard at kubernetes but my environment is basically two or more VMs that I get by ordering them from my hypervisor team, in a shared hypervisor environment.

So maybe there's a possibility to get a VMware API access but it would be complex since it's a shared environment.

In essence I have VMs to work with and I can't provision new ones easily. So let's say I have two to begin with.

I want to deploy 4-5 micro services on them to begin with. Maybe more in the future. I'm using Gitlab for CI/CD.

I probably want to route traffic between these deployed services, some talk to each other, some talk to the outside, some receive requests from the outside.

I have a docker registry too.

What are my options?

https://redd.it/fbag5z
@r_devops

Functional programming in DevOps

Hi everyone!

How prevalent are purely functional programming languages in the DevOps space?

I know a little Java, Ruby, Python, PowerShell, Shell scripting and am interested in learning Go or Rust.
I'm wondering if I should invest some time in learning a functional language too, maybe Scala or Haskell.

How relevant are functional concepts to build/deployment/etc.. in DevOps?
My gut feel is Python and Bash are more suited to this kind of thing; tying things together with scripting and Docker for consistent environments.

What's everyones thoughts about functional languages? Are there some concepts which are useful to know?

https://redd.it/fb9vao
@r_devops

How to i find contracts for designing new pipelines?

I'm a freelancer and generally the contracts that I can find through my external sales organization lands me maintenance positions within old legacy systems. What I want to do to test my skills and stroke my creative mind is to design new systems, selecting state of the art tools and putting it all together. That's what I enjoy and I hate myself every time I have to mess around with some old crap like jenkins, or bitbucket, or god forbid perforce.

I'm not really sure how to look for these types of contracts. I don't know if I maybe should select for a particular industry, if I should market myself as an expert on a particular tool that I like, or any other trick to attract the right recruiters on LinkedIn.

https://redd.it/fb9drd
@r_devops

Kubernetes and Spring Boot MVC

My company is deploying their app as one huge monolythuc mvc web app, deployed on kubernetes. Does it make sence to do this? No microservices are involved.

https://redd.it/fbn63o
@r_devops

I have a python script on AWS EC2 that connects to a website via websockets that is hosted on Digital Ocean. Every method I've tried to start it up on boot up has resulted in failure.

I'll try to be thorough in my description of the problem, what I've tried so far, and what I think the problem(s) is. The flow of information look like this: `transformer.py` starts up, loads autobahn for the websocket connection, tries to find another half of the websocket waiting on the other side of a URL, and attempts to connect to that. If the handshake goes through it it connections, if it doesn't connect then it either results in an error or just hangs. The websocket connection used to connect to AWS Lambda and it worked just fine; it was only once I had it connect to my own website that the autostart stopped working. When I start the script manually it connects just fine, but when try to start the script automatically on boot up I get some weird connection issues. This is a stripped down version of [`transformer.py`](https://transformer.py):

from autobahn.asyncio.websocket import WebSocketClientProtocol
from autobahn.asyncio.websocket import WebSocketClientFactory
import asyncio
import json

uri = "wss://<domain_name>.com/ws/ai/"
domain = "<domain_name>.com"
port = 443

class ClientProtocol(WebSocketClientProtocol):
def onOpen(self):
message = input("Enter Prompt: ")
message = {"action": "handlePrompt", "prompt": message}
payload = json.dumps(message, ensure_ascii=False).encode("utf8")
self.sendMessage(payload, isBinary=False)
print("Sent: " + str(json.loads(payload)))

def onMessage(self, payload, isBinary):
print("Text Message Received: " + str(json.loads(payload)))


if __name__ == "__main__":

factory = WebSocketClientFactory(uri)
factory.protocol = ClientProtocol

carousel = asyncio.get_event_loop()
socket = carousel.create_connection(factory, domain, port, ssl=True)
carousel.run_until_complete(socket) # This line is where the error below appears
carousel.run_forever()
carousel.close()

This is the error that I get with autostartup:

Traceback (most recent call last):
File "transformer.py", line 358, in <module>
carousel.run_until_complete(socket)
File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "/usr/lib/python3.6/asyncio/base_events.py", line 820, in create_connection
sock, protocol_factory, ssl, server_hostname)
File "/usr/lib/python3.6/asyncio/base_events.py", line 846, in _create_connection_transport
yield from waiter
ConnectionResetError

The file structure looks like this:

/home/ubuntu/project/transformer.py
/home/ubuntu/project/start_script.sh
/home/ubuntu/project/venv/

`start_script.sh` is executable. The contents are:

#!/bin/sh
cd ~
cd home/ubuntu/project/
venv/bin/python3 transformer.py

While the autostartup worked with AWS Lambda, I was using a cron job to run a shell script on reboot:

@reboot /home/ubuntu/project/start.sh | bash &

I later found out that it's kind of pointless to pipe the script to bash, but neverminding that it worked. Once I switched the websocket info over to my own website the above error appeared. I switched the cron job to:

@reboot /home/ubuntu/project/start.sh > /home/ubuntu/startup.log 2>&1

This results in the error above. I then tried to use the package `supervisor`. We use this on our webserver and are very familiar with how it works. The thought process behind this was that maybe there were some user permission errors and [`transformer.py`](https://transformer.py) needed to be run as ubuntu. This resulted in the same error. So at this point I was thinking maybe it has something to do with the loading order of modules for linux. Like, maybe the networking part of linux isn't fully loaded by the time the connection is