Department moving to devops: Help needed.

Hello all, sorry for the long post.

This is a Xpost on r/devops, r/sre and r/sysadmin as i think this can be related to all 3 communities.

​

I am in charge of 8 ops engineers in an environment that is still working with development and operation separated.

​

Finally i was able to convince upper management that this is not viable anymore as the job market is now toward dev and ops together.

This means i need to provide a clear direction where i want to go with ops engineer and help them fill the gaps.

​

Before looking at the directions i would like to give you a bit of context: I work in a big company, within the company there's our department with ten squads of 5-6 people in which there's 1 ops, 1 product owner, 3-4 devs, 1 business analyst. As a working methodology we follow Scrum, PO is responsible of the backlog.

​

The squad itself is called "devops squad", meaning dev were developing and the ops was solving incidents, creating and maintaining CI/CD, performing standby shifts and dealing with some extra documentation. Infra is (actually was) maintained by a different department and we are (were) their customers.

​

As infrastructure responsibilites have been moving to our department we needed someone to start focusing on them: we don't really deal with bare metal, we are more taking care of VM's (creation, decomm, updates), and soon we will need to think on how to manage k8s clusters.

​

As a side note Security, Network and firewalls are being taken care by other departments, of course a bare minimum of knowledge on those fields is required but it is not someone we need to look at 24/7. At least not yet

​

Due to the above infra requirements i was able to get some of those ops, make them work in a central squad to provide infrastructure management and a general support to the squad.

This has helped migrating to an idea were developers were supposed to start thinking of basic operations

​

This central squad is trying to follow as much SRE concepts: automation, monitoring, incident automation BPM and so on. the part that is lacking most is the 50% time on developmenmt, due to the fact that some of the people are above 50 year old and some concepts takes a lot of time to absorb, and old habits take lots of time to change.

​

Now, as the shift toward devops is being pushed more and more i will be asked to provide a direction on the ops people. Of course there will be a huge focus on what they will develop, as opposed to the fact that most of the complains from developers is that they don't want to do incidents because they need to code.

​

I need to provide a clear answer to this; i don't want those people to become just developers, or have them being absorbed back in to squads becaus as a result old habits will be enforced and we will go to the situation: "this guy who has to fix incidents because he doesn't know how to code but he doesn't have time to learn to code because he needs to fix incidents and by the way he knows infrastructure better than us so why bother" .

I know this because i run this same experiment and this is what happened.

​

I would like to keep this central squad, i need to find a proper purpose to this and justified it; at the same time i need to instruct and help people to move away from this mentality of separation between dev and ops: this needs to be done on technical people but mostly important to the non technical people (PO's BA's).

​

One of the purpose i am thinking of is to make it become the SRE, and fully adopt SRE scope and purpose. I am totally unaware of what could be the issues i will face with this choice.

​

For sure i will need learning materials for those who are not used to coding, and help them understand concepts as branching strategies, automated testing and so on.

So far bigges coding challenges we faced are scripts under 300 lines.

​

Has anyone experienced something like this and could share their thoughts,

their experiences ?

https://redd.it/g8aq9i
@r_devops

Upgrading AMI's of k8s cluster provisioned with RKE (community) terraform provider.

Currently in the process of implementing rancher server. The initial cluster is provisioned using RKE (terraform provider atm) and then we place Rancher server on top. Maybe I'm overlooking something from the documentation or I'm just thinking about it wrong, but has anyone performed an underlying rolling update of the underlying ec2 instances?

https://redd.it/g9dqy9
@r_devops

What are my options for deploying my web backend that uses Docker?

Hopefully this is the right place to ask such questions. Essentially I have a RESTful API built with Django and MongoDB that I'd like to deploy into a VM I have. (no major cloud providers, I literally just have root access to RHEL VM) I already dockerized and tested it and now I'd like to automate the deployment process. (Repo is on Github) What are the options that make the most sense? Will I need some container orchestration tool like K8s or Docker Swarm? Do I do some webhook when something gets pushed to master? I'm kinda lost as to what I can do because DevOps tooling can get confusing for beginners. Also at some point, I'd like to automate the deployment of frontend side of things as well, but that's on a different repo (don't think it makes too much of a difference).

https://redd.it/g9anbo
@r_devops

What are the top 3 things you wish current APM tools did better?

I find current APM tools like DataDog to be complex. My top 3 wishes:

1. Simpler pricing. I always keep fearing that I will be charged for something which I don't know about
2. Simpler dashboards. Which show me relevant issues and I don't have to go through multiple graphs
3. Show me what you are doing to my infra? How much RAM is the agent using - what is the extra load on my infra due to using an APM.

What are your top 3?

https://redd.it/g98m1z
@r_devops

How I manage secrets in git

I created a little Git repo showing how I store encrypted secrets in git and then decrypt them at runtime on EC2/ECS/Kubernetes.

​

[https://github.com/noqcks/GitSecrets](https://github.com/noqcks/GitSecrets)

​

I created this because sops and other git secret managers make it easy to store the git secrets, but make little mention of operating them once you store them! The documentation can also be quite verbose and confusing.

​

I've used this previously for personal projects and places I've worked.

​

What do you think? How do you manage secrets in production?

https://redd.it/g96ywi
@r_devops

How does google cloud platform detect detect which npm script to run ?

Hi,

​

I just started learning to use Google App Engine. I deployed a Node app, which had a npm script called "dev" so to launch the app using the following command: \`npm run dev\`. However I never mentionned that command to GCP. Did it guess which one to pick ? How does it work ?

​

Thanks.

https://redd.it/g97xfe
@r_devops

Want to get into Devops and want to learn Language, Why python?

Hi All,

Sorry that this post may seem a bit of repetition to others posts, however, I'm hoping someone can answer the specifics of my question, so I can hopefully make the right choice on where to begin.

I have the ability to understand/read basics of code, But in no way am I a developer and would really need to start from scratch for learning a language. And generally speaking everywhere i have worked have been Microsoft one-stop shops, (other than AWS, but they moved to Azure when migrating to O365)

I've had "DevOps" roles in the past. I've worked with AWS, Created networks in the cloud, and those things were generally pretty easy for me to get my head around.I also moved onto a second job in which I was working within a Deployments team, again, advertised as DevOps but it was more Aspects of the role fit into DevOps, than being a DevOps role itself.

Sooooo in regards to languages, I've tried to have a search online regarding what is the best language to learn for starting in Devops, A lot of people are recommending Python.

**But what I don't understand is how does python integrate with things like Azure or AWS?Generally speaking, I thought these platforms mainly used C# or Java?**

**So how can you write in python and it still be used effectively?**

​

Thanks in Advance

https://redd.it/g94qzs
@r_devops

The tool that really runs your containers: deep dive into runc and OCI specifications

Did you know that regardless of which container tool you use - Docker, Podman, CRI-O - it most likely uses runc under the hood?

In my new article, I am showing what runc is and how to use it, as well as a couple of low level things like working with OCI Runtime bundles directly.

I hope, this article will demystify containers for you a bit!

https://mkdev.me/en/posts/the-tool-that-really-runs-your-containers-deep-dive-into-runc-and-oci-specifications

https://redd.it/g97jk4
@r_devops

nginx deployment in multiple environment

I'm doing a AWS CI/CD CodePipeline and CodeBuild with Docker. Dockerfile uses a base NGINX server image. It also has "ADD server.conf" command which copy the config file into the image on Docker build.

Now problem is , server.conf has a proxy_pass url (backend service) . This proxy_pass url is different in Dev , Stage and Production.


Is it possible to parameterize this proxy_pass url in server.conf and value can be replaced at runtime in the target environment? How it is done ? What is the best practice here ?

https://redd.it/g91ror
@r_devops

git tagging strategy

I use Jenkins for CI. When I am about to release I check the latest tag and I increment it using an awk 1 liner. It works. I was wondering if I should maintain a file called "release.txt" which will have:

`githash, tag, message`

`abcdef, v0.1, initial release`

`sdfsdf, v0.2, fixed a bug`

`qqwef, v0.3, added a new feature`

​

Then I run a script which will create the appropriate tag and message. Any thoughts about doing a software release this way? Or is there a better way?

https://redd.it/g90wi1
@r_devops

DevOps playground

Hi guys
Is there an online sandbox kind of place where we can practice DevOps, like IaaS, configuration management etc?

https://redd.it/g946sl
@r_devops

Secure your API from DDoS attacks with NGINX and fail2ban

Hi everyone!

Last week our production environment API was attacked by a DDoS attack. I wrote a blog post detailing how we fixed it by using NGINX and fail2ban.

If you have any suggestions or it worked for you, please let me know! I'm not an expert, so I will gladly take suggestions.

[https://blog.rogs.me/2020/04/secure-your-django-api-from-ddos-attacks-with-nginx-and-fail2ban/](https://blog.rogs.me/2020/04/secure-your-django-api-from-ddos-attacks-with-nginx-and-fail2ban/)

Thanks!

https://redd.it/g94fo3
@r_devops

Let’s talk runbooks for a sec.

Everywhere I’ve worked, we had runbooks. I mean, we were super strict about creating runbooks as part of an operational readiness process before releasing a new service. The idea was then to constantly update the runbooks as we went.

However, my current role, we put a little bit of information into the alert (Datadog monitor-> PagerDuty) itself. But almost feels like an after thought and it barely gets updated.

So my question: what do you guys do? Do you have runbooks? If you have runbooks, do you even follow them?

https://redd.it/g92iof
@r_devops

Running Folding@Home on AWS with AWS CDK

I had some credits that were expiring and decided to use them to run Folding@Home on AWS. I did the setup manually at first, and figured this is a good time to get some hands-on on AWS CDK.

My blog post on the same: https://sathyasays.com/2020/04/26/folding-at-home-aws-cdk/

The code is here: https://github.com/SathyaBhat/folding-aws

The code is rough and there's fair bit of hardcoding - PRs welcome!

https://redd.it/g91kum
@r_devops

New AppSec training platform. Thoughts welcome.

So, a short background about me. I am working my way into pentesting bit by bit (already an IT pro) through online courses, just for the sake of it. For the time being, and until I have sufficient knowledge and experience, I do not plan of focusing on a cybersec career, so I am just enjoying the ride of up-skilling.

That being said, I bumped into this platform on another channel: [https://application.security/](https://application.security/) and even though they do not offer pentesting courses, they do offer secure development training. I was aware of some more well known and established companies, but not of this one.

They seem to be aiming for group registrations but they do have a free section where anyone can practice OWASP best practices.

Just thought it might prove useful to someone here.

\- I am in no way affiliated with the specific platform/company

https://redd.it/g8y83t
@r_devops

[Article] So You Inherited an AWS Account

I've found that many AWS security resources tend to be oriented towards developers who are deploying new services or launching new accounts. They tout a variety of best practices and security controls that should (rightfully) be used.

But what happens if the previous account owner leaves, your company acquires another company, or you are somehow given responsibility for a production AWS account that has been running for years and need to quickly secure it? Many of those same security controls become the goal, but not the reality. This is especially true if the previous owners did not have a strong security posture and you're now responsible for implementing security controls while simultaneously keeping production infrastructure running.

This is a guide for developers who find themselves in this position. It covers the immediate must-dos, along with a roadmap for monitoring and migrating the account to a more secure standard.

[https://medium.com/@matthewdf10/so-you-inherited-an-aws-account-e5fe6550607d?sk=138a8800de70d07e158e918d503ff69a](https://medium.com/@matthewdf10/so-you-inherited-an-aws-account-e5fe6550607d?sk=138a8800de70d07e158e918d503ff69a)

https://redd.it/g9nkqh
@r_devops

Good literatur for getting started with microservices?

I'm a Junior Developer with solid experience in Docker and some first experiences with Swarm who wants to design and develop scalable high-availability microservice applications. Do you have some book recommandations for me?

https://redd.it/g9ml53
@r_devops

Creating a DigitalOcean Droplet with Terraform

I wrote up a three part starter post series on creating a DigitalOcean droplet using Terraform. I went through creating a droplet, attaching a volume, and using cloud-init to customize the droplet.

Would enjoy any wonderful comments from the glorious reddit community.

[https://bitleaf.io/blog/creating-a-digitalocean-droplet-with-terraform-part-1-of-3/](https://bitleaf.io/blog/creating-a-digitalocean-droplet-with-terraform-part-1-of-3/)

https://redd.it/g9o84o
@r_devops

Deploy and Manage Azure Infrastructure Using Terraform, Remote State, and Azure DevOps Pipelines (YAML)

While I found a ton of articles to achieve this, I did not find many that were not only a complete step-by-step guide but used YAML pipelines instead of the classic GUI pipelines. So just wanted to share if anyone else was looking to do the same.

[**link**](HTTPS://WWW.THELAZYADMINISTRATOR.COM/2020/04/28/DEPLOY-AND-MANAGE-AZURE-INFRASTRUCTURE-USING-TERRAFORM-REMOTE-STATE-AND-AZURE-DEVOPS-PIPELINES-YAML/)

https://redd.it/g9na34
@r_devops

Getting started with GitHub Actions: concepts and tutorial

If you heard about GitHub Actions, you already know it is a task automation system. The question is when and how to use it? [Here is a handy tutorial to learn how to use GitHub Actions](https://www.padok.fr/en/blog/github-actions).

https://redd.it/g9mex0
@r_devops