how to create csv file using ansible data and python?

I have created a yaml file to get ansible facts. I need to create a csv file using python to add all the ansible facts i have. In my yaml file, I have lineinfile command for header and data input and creating a file and deleting a file command. i need a python script for csv file format.

https://redd.it/et5vau
@r_devops

Multiple instances of application with different states and databases?

Hi,

I have an application defined in a docker-compose.yml and now I want to add an orchestrator.

My use case: I'm administering this application for a organization with many different subdivisions. Right now nobody is using this application. I want to be able to run multiple instances of this application on my server, where every subdivision in the organization, that wants to use the application, gets one instance with its own configuration, userbase and database independent from all other instances (maybe each configured instance could even be scaled in the traditional sense, but that's not top priority).

My ideal setup would be:

1. A subdivision wants to also use the new application
2. I run some commands and tweak the configuration for this individual instance
3. The orchestrator starts the application and manages the state (configuration, userbase and database) independent from all the other instances

Now my questions are:

* Which orchestrator would be able to fulfill my use case?
* Are there any resources out there how to achieve something like this?
* EDIT: What would be better in this situation: one database for all with different namespaces, or a database for each

https://redd.it/etp65k
@r_devops

Tech stack agnostic tool for tracking builds/artifacts/tests/releases?

Hi

My situation is that we have a number of different tech stacks, each with their own build/release processes and cycles. I've gotten most of my org to centralise around a handful of domain specific techs, but even so a change that goes to production might be any of a Java app, a Go app, a react front end, or libraries for any of those languages, most deployed in containers on k8s, some on VMs, and some in AWS lambda, native iOS or Android builds, terraform projects, shell scripts in crontabs and probably a few things I can't remember.

Luckily, everything is in a source control repo somewhere, and almost everything has a decent CI process, and many things have pretty solid CD processes. So it's not a complete disaster.

What we don't have is a holistic view across all these areas about what was released, when, by whom, how it was built and tested, and what branch of what repo it came from.

Does anybody know of any tool (open source, SaaS, whatever) that would support ingesting this such data from a range of sources and tools, for developing an audit trail and providing metrics and dashboards around release velocity?

To be specific, I am not looking for a CI/CD tool. Most teams already have their own, and while I would like to consolidate everybody to a single platform, for now I'm just trying to measure what's going on.

Any help or suggestions or help are welcome.

https://redd.it/etq0d3
@r_devops

Can't connect to MariaDB docker container

A while back I created my own project that used a simple apache + php + mariadb architecture to print out a search made through a web browser. It consists of a simple html form page with an AJAX javascript file to retrieve a json object generated from a php page that queries mariadb.

I thought it would be a fun project to try and "dockerize" this small webapp, but I'm currently stuck on maria db (I've also switched over to nginx, but I genuinely don't think that's the issue, as you'll see in a moment).

Currently, everything works as I'd expect. Nginx serves the web page and will send any php script requested by the browser to php-fpm etc. The problem I'm having is that I keep getting:

mariadb\_1 | 2020-01-25 17:52:00 9 \[Warning\] Access denied for user 'php'@'[172.19.0.3](https://172.19.0.3/)' (using password: YES)

Whenever I use the html form. Even worse, when I connect directly to the container via:

sudo docker container exec -it mariadb\_1 mysql -u php -p (my password)

I get:

ERROR 1045 (28000): Access denied for user 'php'@'localhost' (using password: YES)

The same is true for using the root user and password:

ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

Its also true when I assign an empty password in environment variables:

ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)

I have no dockerfile for mariadb, but here is my docker-compose.yml:

version: '3'
services:
mariadb:
image: mariadb:10.4.11-bionic
networks:
- backend
environment:
- MYSQL_ROOT_PASSWORD=password
- MYSQL_DATABASE=my_db
- MYSQL_USER=php
- MYSQL_PASSWORD=password
nginx:
build: './nginx/'
ports:
- "8080:80"
networks:
- frontend
volumes:
- ./public/html:/usr/share/nginx/html
php:
build: './php/'
networks:
- frontend
- backend
volumes:
- ./public/php:/var/www/html
networks:
- fontend
- backend

I've done a lot of searching and I just can't figure out what's wrong. Does anyone have any idea of where I should go from here? I'm starting to get frustrated. As far as I can tell everything is working, and I'm getting a response from the mariadb container each time something tries to connect to it. The passwords are hardcoded, and I just can't figure out what's wrong.

EDIT: I noticed that MYSQL\_USER was entered twice, overriding the user with the password. I've updated accordingly, but I still can't connect with root (or, obviously, the php user).

EDIT: Is there a way that I can check that these variables are being set properly, maybe by bashing into the container?

https://redd.it/etve9w
@r_devops

Maven in the container?

We are working down the container / K8 path and whiles going through trials and tests one of our developers included the Maven builds in the container (as command lines to add the maven binaries and then pull in what’s needed before running the output .jar).
Rational being that the build team have everything.
Initially I was thinking no way for a host of reasons, but wondered if anyone else had seen this pattern.

https://redd.it/eu1xzq
@r_devops

I'm a .NET Developer. Any suggestions how I can learn/incorporate more DevOps in my work?

Hello everyone, I want to transition to DevOps/Cloud eventually and I'm eager to learn new skill sets. Any suggestions or advice is much appreciated.

I write mostly API's. At work, our deployment process is fairly simple if not archaic. It's just making backups then publishing from visual studio to a network folder. We do a sql schema compare between test and prod then manually create a deployment/rollback script.

We don't write a lot of tests, use testing automation/tools, etc ... We have a QA team and most of our testing is end-to-end.

Honestly typing all this out hurts, it seems very old fashioned nowadays. My company sucks in regards of helping their people learn new skills and a big part of me suggests to go elsewhere that better suits my career goals.

Anyway, again thanks for any advice or suggestions.

https://redd.it/etzc8o
@r_devops

Recommended Application Performance Monitor?

I’m looking to get more insight into how our systems are performing so we can respond more quickly and effectively to problems. I’m hoping to get some suggestions on products to try out since there seem to be a ton of options out there. Today we monitor the infrastructure with CloudWatch, I want to add tooling to the applications themselves to get more insights.

Architecture Basics:
- Python Flask and Starlette based services
- Background jobs with Celery
- Everything hosted in Docker containers running in AWS Fargate
- MySQL (AWS Aurora)
- Redis for caching (ElastiCache)

Must Have:
- Dashboard containing response times, background jobs, deployment markers, custom metrics
- Alerting on anomalies and/or custom thresholds
- Easy developer experience- a couple lines of code to add more metrics

Should Have:
- SaaS option
- CloudWatch integration
- Custom actions on alerts (e.g. call a Lambda which I can configure to scale a container)

Any suggestions you have are appreciated, thanks!

https://redd.it/eu8hgp
@r_devops

Nginx unable to get SSL cert to work

I'm trying to get a kubernetes cluster using the nginx ingress controller and cert manager to get an SSL cert from lets encrypt. Currently is appears the cert has been issued however I am unable to get it to load properly inside Chrome or via curl.

​

Below are some config files that show the current config and following that some describes.

​

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: letsencrypt-prod
namespace: cert-manager-prod
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: letsencrypt-prod
namespace: cert-manager-prod
spec:
commonName: [email protected] <- This is not actually what I have just for example reasons
secretName: letsencrypt-prod-tls
issuerRef:
name: letsencrypt-prod

&#x200B;

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
# add an annotation indicating the issuer to use.
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
# needed to allow the front end to talk to the back end
nginx.ingress.kubernetes.io/cors-allow-origin: "https://portal.<domain>.com"
nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, PATCH, OPTIONS"
# needed for monitoring - maybe
prometheus.io/scrape: "true"
prometheus.io/port: "10254"
#for nginx ingress controller
ad.datadoghq.com/nginx-ingress-controller.check_names: '["nginx","nginx_ingress_controller"]'
ad.datadoghq.com/nginx-ingress-controller.init_configs: '[{},{}]'
ad.datadoghq.com/nginx-ingress-controller.instances: '[{"nginx_status_url": "https://%%host%%:18080/nginx_status"},{"prometheus_url": "https://%%host%%:10254/metrics"}]'
ad.datadoghq.com/nginx-ingress-controller.logs: '[{"service": "controller", "source":"nginx-ingress-controller"}]'
name: prod-ingress
namespace: production
spec:
rules:
- host: api.<domain>.com
http:
paths:
- backend:
serviceName: api
servicePort: 8090
path: /
tls: # < placing a host in the TLS config will indicate a certificate should be created
- hosts:
- api.<domain>.com
secretName: prod-ingress-cert # < cert-manager will store the created certificate in this secret

&#x200B;

me@LAPTOP-LINUX:~/Projects/k8/tmp$ kubectl get certificates
NAME READY SECRET AGE
prod-ingress-cert False prod-ingress-cert 30m

&#x200B;

Name: prod-ingress-cert
Namespace: production
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1alpha2
Kind: Certificate
Metadata:
Creation Timestamp: 2020-01-26T21:29:07Z
Generation: 1
Owner References:
API Version: extensions/v1beta1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: prod-ingress
UID: adec15ea-4cde-4376-8e6a-306586e57119
Resource Version: 2780
Self Link: /apis/cert-manager.io/v1alpha2/namespaces/production/certificates/prod-ingress-cert
UID: c86bf4aa-8435-4da0-a822-1a0e9d1df38e
Spec:
Dns Names:
api.<domain>.com
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt-prod
Secret Name: prod-ingress-cert
Status:
Conditions:
Last Transition Time: 2020-01-26T21:29:07Z
Message: Waiting for CertificateRequest "prod-ingress-cert-1024021523" to complete
Reason: InProgress
Status

: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal GeneratedKey 30m cert-manager Generated a new private key
Normal Requested 30m cert-manager Created new CertificateRequest resource "prod-ingress-cert-1024021523"

\`\`\`

&#x200B;

That's everything I can think of that might be of use in resolving this issue. I should note I have a similar setup running on development (well did till I took the cluster down today) that is/was working.

https://redd.it/eueoxc
@r_devops

(A few) Ops Lessons We All Learn The Hard

https://www.netmeister.org/blog/ops-lessons.html

To be frank, It is not «a few», it is 88! I was thinking about choosing the points I agree with the most and write these down, but I realize it would be hard to choose a top three. Even a top ten. I might as well make cron send me the link once month as a quick and steady reminder. This just hit too close to home.

https://redd.it/eu7wv2
@r_devops

Code challenge sites for Devops/sysadmin scripting (Python)

Hey guys,

Wondering if anyone knows some good sites to practice and expand Python scripting skills with exercises related to linux admin/ devopsy tasks. Basically all things parsing files and linux related.

[PyBites](https://codechalleng.es/bites/75/) is pretty good, with some relevant questions. Example questions is "*Parse Unix cal to a weekday mapping*" or implementing grep.

Another one is [CodeSignal](https://app.codesignal.com/challenges/page/1) which has good questions in a 'Devops' category.

Alot of the other sites such as Codewars and edabit only have algorithmic type questions.

https://redd.it/euaua7
@r_devops

I'm a (36m) Sr. SRE / DevOps Engineer . Where do I go from here ?

Greetings and thanks for reading.

I am currently employed as a 100% remote "Sr. SRE" for a company that is primarily V.C. funded.

My base salary is 145k, I sometimes receive a bonus, and all my remote expenses (phone, internet, cowork space) are covered. I live in a lower cost of living area. All in all, it's not a bad setup. But as someone who is approaching middle-age, I am beginning to wonder how I can continue to increase my earnings, stay relevant, and not burn out.

&#x200B;

I have ambitions for technical leadership / management but I'm not sure how to get there. In my current role, I am an important contributor and regularly consulted about implementation details etc but the reality is that I am at the bottom of the hierarchy for my department. I believe my salary is in the upper tier for individual contributors within the company, but there is nobody below me in the org chart.

&#x200B;

Trying to start my own company sounds like a big risk now that I have a wife, child and mortgage, but I certainly don't hate the idea.

I fear the potential for age discrimination if I don't advance one way or another.

&#x200B;

Some stats abut me:

\- Almost 36 years old

\- Didn't go to University, have worked since I was a teenager.

\- Have been a linux hobbyist since 1998

\- I have something like 15 years of job experience that basically goes tech support -> sysadmin/I.T. -> devops -> SRE over the span of my career, with a little bit of training/mentoring and technical leadership sprinkled in.

\- I'm decent at python, for systems and webdev stuff

\- AWS DevOps Pro certified

\- Have some experience training/mentoring jr. engineers with sysadmin, config management, and basic programming

&#x200B;

Technologies I regularly utilize:

AWS, GCP, Ansible, Terraform, Salstack, Packer, Jenkins, Kubernetes/Docker, Nginx, Apache, postgres, mysql, monitoring tools like nagios, icigina2, checkmk, ELK stack ...

&#x200B;

I feel very comfortable working on large scale, production infrastructure. I consider myself to be a master of troubleshooting and very resourceful at learning new technologies on my own.

&#x200B;

If you've made it this far in my post, thanks for reading. Am I underpaid, overpaid? What can I do to move onward and upward in my career?

&#x200B;

Thanks so much.

https://redd.it/eua83x
@r_devops

Best way to create automations on the cloud?

I want to create an automation on a website without web hooks (pixieset). How can I create an automation on the cloud for this?

https://redd.it/euhe10
@r_devops

Secure website

Hello I hope someone can help me I get a task for a work I try to land.




I need to secure a website ruining in dockers, I'm using this image




[https://github.com/TrafeX/docker-php-nginx](https://github.com/TrafeX/docker-php-nginx)




I get some troubles making the SSL, I wonder if I can make the SSL in the same docker file,




Also if you have any other best security practices that wanna share I could appreciate any help.




Thanks

https://redd.it/eu6z6y
@r_devops

DevOps/SRE Python online learning platform

Hi all,

I am a DevOps/SRE with 10y+ exp. I want to challenge myself in order to progress in Python.

But i struggle to find some good online formation tool in order to get exercises (devops oriented or not) in Pyhon.

I have found things like "PyBites". There is other or better site like that ?

&#x200B;

Thanks for your feedback.

https://redd.it/euovp2
@r_devops

Best way to learn Containers / Kubernetes?

I've been searching for a new job as a DevOps Engineer and one of the things I'm lacking that everyone wants is experience with containers / Kubernetes. What are some good resources for learning these?

Thanks!

https://redd.it/euqbac
@r_devops

What’s your testing practices for Infra,Jenkins Pipelines etc?

We have a combination of Terraform/ansible modules for infrastructure provisioning &management.As it gets more and more complex,we are looking to invest in more unit and integration tests.Our eco system consists of Jenkins pipelines for CI/CD,a customer vue app we wrote to be the front end of the orchestration layer.We are looking to add more testing across our eco system.Planning to leverage local stack,Inspec and puppeteer for the front end app.

What are your testing strategies?What are the testing tools you use?What are your learnings?Was there any tools any one used for Jenkins pipelines?

https://redd.it/euproz
@r_devops

AWS Fargate loadbalancing need advice

Hey,

I have a website, frontend on S3 and 4 microservices for it which I want to put in Fargate. Really only one of them will most likely need to ever be scaled which is where my question arises:

How do I loadbalance the services? So far I can see 3 solutions:

\- put each container inside its own service, put loadbalancer in front of each one and don't worry about it (but each loadbalancer costs money)

\- put all containers in one service with a single loadbalancer in front and scale the whole thing (but I need an expensive machine)

\- use route 53 for service discovery and nginx container to dynamically resolve DNS records and hope it works because there is no algorithm behind this?

How do you guys set up your projects?

https://redd.it/euotug
@r_devops

Issue tracking for open-source projects

I have an open source project and we manage release planning in jira, but managing 2 repositories for issue tracking is cumbersome. Is there a way I can overcome this. How do open-source projects generally manage their issues for every release cycles ??

https://redd.it/euonmc
@r_devops

Guide to cracking interviews from insiders?

I have been in the process of doing interviews for Devops roles and been having mixed success. There are many rounds to even get to the onsite but typically hit a brick wall.

I seem to get picked by HR managers as my resume has the key words (buzz words use to screen) and then do countless calls set up by recruiters.

For the assessment, I do technical tests and for the most part my answers are right, but the managers are so picky that even if you answer is correct you don’t get selected. The questions they ask are not super hard. Most frequent URL from a file, ping IP addresses in a loop, replace a word using sed command. I ask is this correct, and you here the usual (we’ll get back to you).
There doesn’t seem to be any objective standard and it seems very arbitrary. Do you want me to write the code first time right without any errors? Did I take too long?
(You get a typical thank you applying while we were impressed , by your evaluation we have decided to move on.. yadda yadda .. very impersonal and no sort of helpful feedback.


When I go onsite, they start asking me super curve ball questions and sometimes obscure out of left fields. I’m not a networking guy.. yet they ask me What port is SNMP, what is difference between router and a switcher, what level are they on? And nuances between Entrypoint and CMD? And what are inodes?
The worst is when you are expected to memorise command line functions such as using stat or how to sort logfile and knowing the right arguments.

Things you usually look up on Wikipedia as I don’t memorise the documentation. Then I do white boarding and draw my setup - it’s a Kubernetes cluster with pods running micro services. Then would you make this multi regional or zonal, are you using VPC peering?
Basically I have to explain the architecture and theory behind this as If a lecturer, I’m a practitioner of tools

I am bewildered as how to play this game. I miss some super easy questions (such as what’s the docker process running in background- dockerd ..how do assign Kubernetes pods to nodes ..using labels). Obviously I try to go prepared but you can never know what they are going to throw and it you.

The recruiters are horrible and pitch you to get their quota, sometimes they even send the wrong job description and make appointments without your permission. Obviously they don’t want to gamble and make the wrong decision on a hire, the managers though are the real ones in charge and decide if they hire you. And it’s not easy to get past them.

Obviously I know this whole process is a vetting out competition trying to weed out weak candidates, but at least make it skills based and something you will do on the job. I can make a CI /CD pipeline, set up a cluster , do automated deployments all day, set up monitoring tools.

But asking me what is a replica set and deployment difference ( trick question ) .. they are the same.. or what is Kubernetes operator? It starts becoming more a sadist ritual on trivial jargon that an evaluation of your competence.

Sometimes it seems like companies don’t know what they want - or they send the wrong message about their company. Are they trying to measure ability or the persons ability for cultural fit? I’m even afraid to answer the right answer, as they will follow up with a bunch of even harder questions until I choke. I start thinking the Gods of tech are cursing me and delighted in making me look very foolish ...

I’m starting to think who these guys hire because it’s the unicorn who uses their exact tech stack, needs zero training, and subject matter expert in several domains with exceptional knowledge of system architecture.

I’m not trying to make light of a job selection process as it’s challenging, personally believe it’s about compromise .. if person suffices and satisfices rather than creating a mythical tech worker based on a checklist.

https://redd.it/eum46s
@r_devops

How to pass a variable from a Root Module down to a child module? Terraform

I'm not understanding the usage of Output Variables correctly or something ... the documentation is confusing and I can't find examples online.

Structure:

|- /api/datasources

|--------------------- dev.tf

main.tf

I understand how to broadcast the output, but how do I consume it from the child element?

namely ... SSM parameter value..

-----------------------------

module "transfer" {

source = "./api/datasources"

api_id = "${var.api_id}"

}

--------------------------

output "api_id" {

value = data.aws_ssm_parameter.myapi.value

}

------------------

The child module (datasources) needs to consume the output of this ... but HOW DO I pass it thru the module?

How do I consume the api_id inside of /api/datasources/dev.tf?? Here is dev.tf:


---------------------
resource "aws_instance" "name"{

api_id = var.api_id

}
---------------------

Help... please

https://redd.it/euv4n3
@r_devops