DevSecOps (Security in DevOps)

I have done implementation of security tools in DevOps process that helps to make DevOps more secure
Here is the github link
https://github.com/Rishabh-Tamrakar/DevSecOps

For now I have updated only 1st page that will explain process that I followed
Configuration of tools will be updated soon (most Probably this weekend)

Do tell me your thoughts about it and what more can be added, ideas are most welcome.

https://redd.it/etbe2k
@r_devops

Easiest way to deploy and host a running Docker Container?

I'm a bit overwhelmed by everything AWS has to offer, but am trying to find a way to deploy a Docker image to have the container running in AWS. The image is stored and updated in Docker Hub, and built via a CircleCI job.

Should I use Elastic Beanstalk, ECS, or something else?

Also, I was originally in favor of AWS, but if any of you have suggestions to keep the cost down, and improving the ease of maintainability, I'm definitely interested in hearing what's worked best for you!

Thanks

https://redd.it/eteld9
@r_devops

Containers that crash "by design"

Hi Everyone. I'm having a "discussion" with our dev team with regards to a container that was deployed to a non production k8s environment, which immediately went into crash-loop-back-off. I assumed the deployment had failed, but in fact, the pod was just waiting for a certain condition (a queue defined in a rabbitmq cluster) to become "true"

The conditions for the pod to continue running (and not crash) are then triggered by someone doing something somewhere in the UI of another service, which may happen 5 mins later, or 5 months later.

In the meantime, its crash-loop-backoff city for our pod.

The developers are claiming this is Kubernetes best practice to allow kubernetes to restart the container should it fail, but:

1) I can't find mention of this specific use case, and

2) It feels like its not in the spirit of what was intended when the best practices were laid out. E.g. "if your application fails due to some completely unforeseeable event, then kubernetes should restart it"... 100% agree with that, but this is "We're going to run code that we know will likely fail, and just let kubernetes restart it until the conditions exist that make it happy"

My concerns are that having a failing pod being a "normal situation" will reduce in alert spam, and eventually, people ignoring those alerts. Further, unless the devs tells us, how are we to know (and remember that those pods "might be in crash loop back off for some time" and this is ok)

My suggestion to the dev team is to have the container "sleep" (and reflect this state in the liveness probe) so that kubernetes will not direct any traffic to it, but similarly, it wont be stuck in crash-loop-backoff hell.

Thoughts?

https://redd.it/etcx9h
@r_devops

Completely random. Can someone tell me how kubernetes is pronounced?

Every time I see it, I read it as cue-bur-neets , but I overheard someone the other day and they pronounced it like cue-bur-net-ees

Please reply with a 1 for my interpretation or a 2 for the way I heard someone say it. If both wong let a man know lol.

Tia

https://redd.it/etfolj
@r_devops

How do you organize the work?

I’m the first DevOps Engineer at a company working on a newish product. It’s beyond MVP stage and so far the developers have been doing all the infrastructure work. Realizing that’s not sustainable they’ve brought me on.

There’s so much to do, immediate needs and lower priority tasks I want to keep pushing along. These aren’t really Jira tasks inside they product board since they don’t deal with product features or bug fixes. How are you all organizing infrastructure work?

https://redd.it/etfca3
@r_devops

DevOps Stack for Microservices

Hello,

New to DevOps. Do you have a recommendation for a stack to support Microservices and cross cloud deployments? Even a past thread or links will be helpful.

Thanks in advance!

https://redd.it/etblml
@r_devops

Docker future stuff

I've been hearing some rumors about Docker Swarm being end-of-life'd by Mirantis and docker-compose being integrated into 'docker stack'.

It'd be nice to not have to install a separate tool for compose, and Mirantis not wanting to double Kubernetes/Swarm efforts makes sense; however, does anyone have any real facts or source of truth behind any of this?

https://redd.it/etdaue
@r_devops

How much do you value WFH

I’m a relatively Jr DevOps engineer who worked my way up from IT at my current company. I’m making 95K in the Northeast area. My current job is great and I’m still learning and getting good mentorship. I typically WFH 3-4 days a week which is a huge benefit. (Although slightly isolating and depressing in the winter)

I recently got an offer for 135K + 15% bonus doing similar tasks at a more mature tech company. They would like me to be in the office though. My commute is a 10 minute drive to the station and then about an hour of sitting on a train if I was to accept.

40K more seems like a lot and I’m wondering if anyone has had to sacrafice WFH benefits for a large jump in pay and was it worth it? FWIW I don’t have any kids so when I’m home it’s just me and my cat 🐈.

https://redd.it/et6hq2
@r_devops

how to create csv file using ansible data and python?

I have created a yaml file to get ansible facts. I need to create a csv file using python to add all the ansible facts i have. In my yaml file, I have lineinfile command for header and data input and creating a file and deleting a file command. i need a python script for csv file format.

https://redd.it/et5vau
@r_devops

Multiple instances of application with different states and databases?

Hi,

I have an application defined in a docker-compose.yml and now I want to add an orchestrator.

My use case: I'm administering this application for a organization with many different subdivisions. Right now nobody is using this application. I want to be able to run multiple instances of this application on my server, where every subdivision in the organization, that wants to use the application, gets one instance with its own configuration, userbase and database independent from all other instances (maybe each configured instance could even be scaled in the traditional sense, but that's not top priority).

My ideal setup would be:

1. A subdivision wants to also use the new application
2. I run some commands and tweak the configuration for this individual instance
3. The orchestrator starts the application and manages the state (configuration, userbase and database) independent from all the other instances

Now my questions are:

* Which orchestrator would be able to fulfill my use case?
* Are there any resources out there how to achieve something like this?
* EDIT: What would be better in this situation: one database for all with different namespaces, or a database for each

https://redd.it/etp65k
@r_devops

Tech stack agnostic tool for tracking builds/artifacts/tests/releases?

Hi

My situation is that we have a number of different tech stacks, each with their own build/release processes and cycles. I've gotten most of my org to centralise around a handful of domain specific techs, but even so a change that goes to production might be any of a Java app, a Go app, a react front end, or libraries for any of those languages, most deployed in containers on k8s, some on VMs, and some in AWS lambda, native iOS or Android builds, terraform projects, shell scripts in crontabs and probably a few things I can't remember.

Luckily, everything is in a source control repo somewhere, and almost everything has a decent CI process, and many things have pretty solid CD processes. So it's not a complete disaster.

What we don't have is a holistic view across all these areas about what was released, when, by whom, how it was built and tested, and what branch of what repo it came from.

Does anybody know of any tool (open source, SaaS, whatever) that would support ingesting this such data from a range of sources and tools, for developing an audit trail and providing metrics and dashboards around release velocity?

To be specific, I am not looking for a CI/CD tool. Most teams already have their own, and while I would like to consolidate everybody to a single platform, for now I'm just trying to measure what's going on.

Any help or suggestions or help are welcome.

https://redd.it/etq0d3
@r_devops

Can't connect to MariaDB docker container

A while back I created my own project that used a simple apache + php + mariadb architecture to print out a search made through a web browser. It consists of a simple html form page with an AJAX javascript file to retrieve a json object generated from a php page that queries mariadb.

I thought it would be a fun project to try and "dockerize" this small webapp, but I'm currently stuck on maria db (I've also switched over to nginx, but I genuinely don't think that's the issue, as you'll see in a moment).

Currently, everything works as I'd expect. Nginx serves the web page and will send any php script requested by the browser to php-fpm etc. The problem I'm having is that I keep getting:

mariadb\_1 | 2020-01-25 17:52:00 9 \[Warning\] Access denied for user 'php'@'[172.19.0.3](https://172.19.0.3/)' (using password: YES)

Whenever I use the html form. Even worse, when I connect directly to the container via:

sudo docker container exec -it mariadb\_1 mysql -u php -p (my password)

I get:

ERROR 1045 (28000): Access denied for user 'php'@'localhost' (using password: YES)

The same is true for using the root user and password:

ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

Its also true when I assign an empty password in environment variables:

ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)

I have no dockerfile for mariadb, but here is my docker-compose.yml:

version: '3'
services:
mariadb:
image: mariadb:10.4.11-bionic
networks:
- backend
environment:
- MYSQL_ROOT_PASSWORD=password
- MYSQL_DATABASE=my_db
- MYSQL_USER=php
- MYSQL_PASSWORD=password
nginx:
build: './nginx/'
ports:
- "8080:80"
networks:
- frontend
volumes:
- ./public/html:/usr/share/nginx/html
php:
build: './php/'
networks:
- frontend
- backend
volumes:
- ./public/php:/var/www/html
networks:
- fontend
- backend

I've done a lot of searching and I just can't figure out what's wrong. Does anyone have any idea of where I should go from here? I'm starting to get frustrated. As far as I can tell everything is working, and I'm getting a response from the mariadb container each time something tries to connect to it. The passwords are hardcoded, and I just can't figure out what's wrong.

EDIT: I noticed that MYSQL\_USER was entered twice, overriding the user with the password. I've updated accordingly, but I still can't connect with root (or, obviously, the php user).

EDIT: Is there a way that I can check that these variables are being set properly, maybe by bashing into the container?

https://redd.it/etve9w
@r_devops

Maven in the container?

We are working down the container / K8 path and whiles going through trials and tests one of our developers included the Maven builds in the container (as command lines to add the maven binaries and then pull in what’s needed before running the output .jar).
Rational being that the build team have everything.
Initially I was thinking no way for a host of reasons, but wondered if anyone else had seen this pattern.

https://redd.it/eu1xzq
@r_devops

I'm a .NET Developer. Any suggestions how I can learn/incorporate more DevOps in my work?

Hello everyone, I want to transition to DevOps/Cloud eventually and I'm eager to learn new skill sets. Any suggestions or advice is much appreciated.

I write mostly API's. At work, our deployment process is fairly simple if not archaic. It's just making backups then publishing from visual studio to a network folder. We do a sql schema compare between test and prod then manually create a deployment/rollback script.

We don't write a lot of tests, use testing automation/tools, etc ... We have a QA team and most of our testing is end-to-end.

Honestly typing all this out hurts, it seems very old fashioned nowadays. My company sucks in regards of helping their people learn new skills and a big part of me suggests to go elsewhere that better suits my career goals.

Anyway, again thanks for any advice or suggestions.

https://redd.it/etzc8o
@r_devops

Recommended Application Performance Monitor?

I’m looking to get more insight into how our systems are performing so we can respond more quickly and effectively to problems. I’m hoping to get some suggestions on products to try out since there seem to be a ton of options out there. Today we monitor the infrastructure with CloudWatch, I want to add tooling to the applications themselves to get more insights.

Architecture Basics:
- Python Flask and Starlette based services
- Background jobs with Celery
- Everything hosted in Docker containers running in AWS Fargate
- MySQL (AWS Aurora)
- Redis for caching (ElastiCache)

Must Have:
- Dashboard containing response times, background jobs, deployment markers, custom metrics
- Alerting on anomalies and/or custom thresholds
- Easy developer experience- a couple lines of code to add more metrics

Should Have:
- SaaS option
- CloudWatch integration
- Custom actions on alerts (e.g. call a Lambda which I can configure to scale a container)

Any suggestions you have are appreciated, thanks!

https://redd.it/eu8hgp
@r_devops

Nginx unable to get SSL cert to work

I'm trying to get a kubernetes cluster using the nginx ingress controller and cert manager to get an SSL cert from lets encrypt. Currently is appears the cert has been issued however I am unable to get it to load properly inside Chrome or via curl.

​

Below are some config files that show the current config and following that some describes.

​

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: letsencrypt-prod
namespace: cert-manager-prod
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: letsencrypt-prod
namespace: cert-manager-prod
spec:
commonName: [email protected] <- This is not actually what I have just for example reasons
secretName: letsencrypt-prod-tls
issuerRef:
name: letsencrypt-prod

&#x200B;

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
# add an annotation indicating the issuer to use.
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
# needed to allow the front end to talk to the back end
nginx.ingress.kubernetes.io/cors-allow-origin: "https://portal.<domain>.com"
nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, PATCH, OPTIONS"
# needed for monitoring - maybe
prometheus.io/scrape: "true"
prometheus.io/port: "10254"
#for nginx ingress controller
ad.datadoghq.com/nginx-ingress-controller.check_names: '["nginx","nginx_ingress_controller"]'
ad.datadoghq.com/nginx-ingress-controller.init_configs: '[{},{}]'
ad.datadoghq.com/nginx-ingress-controller.instances: '[{"nginx_status_url": "https://%%host%%:18080/nginx_status"},{"prometheus_url": "https://%%host%%:10254/metrics"}]'
ad.datadoghq.com/nginx-ingress-controller.logs: '[{"service": "controller", "source":"nginx-ingress-controller"}]'
name: prod-ingress
namespace: production
spec:
rules:
- host: api.<domain>.com
http:
paths:
- backend:
serviceName: api
servicePort: 8090
path: /
tls: # < placing a host in the TLS config will indicate a certificate should be created
- hosts:
- api.<domain>.com
secretName: prod-ingress-cert # < cert-manager will store the created certificate in this secret

&#x200B;

me@LAPTOP-LINUX:~/Projects/k8/tmp$ kubectl get certificates
NAME READY SECRET AGE
prod-ingress-cert False prod-ingress-cert 30m

&#x200B;

Name: prod-ingress-cert
Namespace: production
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1alpha2
Kind: Certificate
Metadata:
Creation Timestamp: 2020-01-26T21:29:07Z
Generation: 1
Owner References:
API Version: extensions/v1beta1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: prod-ingress
UID: adec15ea-4cde-4376-8e6a-306586e57119
Resource Version: 2780
Self Link: /apis/cert-manager.io/v1alpha2/namespaces/production/certificates/prod-ingress-cert
UID: c86bf4aa-8435-4da0-a822-1a0e9d1df38e
Spec:
Dns Names:
api.<domain>.com
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt-prod
Secret Name: prod-ingress-cert
Status:
Conditions:
Last Transition Time: 2020-01-26T21:29:07Z
Message: Waiting for CertificateRequest "prod-ingress-cert-1024021523" to complete
Reason: InProgress
Status

: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal GeneratedKey 30m cert-manager Generated a new private key
Normal Requested 30m cert-manager Created new CertificateRequest resource "prod-ingress-cert-1024021523"

\`\`\`

&#x200B;

That's everything I can think of that might be of use in resolving this issue. I should note I have a similar setup running on development (well did till I took the cluster down today) that is/was working.

https://redd.it/eueoxc
@r_devops

(A few) Ops Lessons We All Learn The Hard

https://www.netmeister.org/blog/ops-lessons.html

To be frank, It is not «a few», it is 88! I was thinking about choosing the points I agree with the most and write these down, but I realize it would be hard to choose a top three. Even a top ten. I might as well make cron send me the link once month as a quick and steady reminder. This just hit too close to home.

https://redd.it/eu7wv2
@r_devops

Code challenge sites for Devops/sysadmin scripting (Python)

Hey guys,

Wondering if anyone knows some good sites to practice and expand Python scripting skills with exercises related to linux admin/ devopsy tasks. Basically all things parsing files and linux related.

[PyBites](https://codechalleng.es/bites/75/) is pretty good, with some relevant questions. Example questions is "*Parse Unix cal to a weekday mapping*" or implementing grep.

Another one is [CodeSignal](https://app.codesignal.com/challenges/page/1) which has good questions in a 'Devops' category.

Alot of the other sites such as Codewars and edabit only have algorithmic type questions.

https://redd.it/euaua7
@r_devops

I'm a (36m) Sr. SRE / DevOps Engineer . Where do I go from here ?

Greetings and thanks for reading.

I am currently employed as a 100% remote "Sr. SRE" for a company that is primarily V.C. funded.

My base salary is 145k, I sometimes receive a bonus, and all my remote expenses (phone, internet, cowork space) are covered. I live in a lower cost of living area. All in all, it's not a bad setup. But as someone who is approaching middle-age, I am beginning to wonder how I can continue to increase my earnings, stay relevant, and not burn out.

&#x200B;

I have ambitions for technical leadership / management but I'm not sure how to get there. In my current role, I am an important contributor and regularly consulted about implementation details etc but the reality is that I am at the bottom of the hierarchy for my department. I believe my salary is in the upper tier for individual contributors within the company, but there is nobody below me in the org chart.

&#x200B;

Trying to start my own company sounds like a big risk now that I have a wife, child and mortgage, but I certainly don't hate the idea.

I fear the potential for age discrimination if I don't advance one way or another.

&#x200B;

Some stats abut me:

\- Almost 36 years old

\- Didn't go to University, have worked since I was a teenager.

\- Have been a linux hobbyist since 1998

\- I have something like 15 years of job experience that basically goes tech support -> sysadmin/I.T. -> devops -> SRE over the span of my career, with a little bit of training/mentoring and technical leadership sprinkled in.

\- I'm decent at python, for systems and webdev stuff

\- AWS DevOps Pro certified

\- Have some experience training/mentoring jr. engineers with sysadmin, config management, and basic programming

&#x200B;

Technologies I regularly utilize:

AWS, GCP, Ansible, Terraform, Salstack, Packer, Jenkins, Kubernetes/Docker, Nginx, Apache, postgres, mysql, monitoring tools like nagios, icigina2, checkmk, ELK stack ...

&#x200B;

I feel very comfortable working on large scale, production infrastructure. I consider myself to be a master of troubleshooting and very resourceful at learning new technologies on my own.

&#x200B;

If you've made it this far in my post, thanks for reading. Am I underpaid, overpaid? What can I do to move onward and upward in my career?

&#x200B;

Thanks so much.

https://redd.it/eua83x
@r_devops