Good tool for RCA of an outage?

Our prod system has an outage yesterday that caused our website to go down. I just started on boarding so getting familiar with their application architecture. One of the engineers resolved it draining and scaling down replicas to 0 and back to normal. I also added more instances to the auto scaling group and issue was resolved.

Trying to determine the causes of the issue is challenging, we are using fluentbit and cloud watch agent on the eks cluster, so all metrics and logs are available. Can see CPU spiking up and network traffic going up on pods to cause the ELB to go down. Going through logs hasn’t provided anything interesting, also used log insights but it’s just a lot of noise.

Are there any good AIOps tools that can be used postmortem? I’m looking into sysdig and dynatrace,
Not sure if they can be used after an incident. This is the second time this week this issue happened, my assumption is a service is making too many calls to the API and causing resource utilization to spike. Our website just launched so we don’t have much external traffic, it seems like an internal configuration issue.

https://redd.it/n2bh2x
@r_devops

People always talking about Istio this, Istio that

People in /r/devops telling me, always on Twitter, Istio this, Istio that. They say that Consul don't be putting in no work. Shut the hell up! Yall DevOps Engineers aint know shit. All yall talk about Consul aint no hitta. Consul aint this, Consul a fake. Shut the hell up! Yall dont even run a service mesh. Yall know that Consul can update 172K services in under 1 second since I dont know when. Stop playing Consul like that. Consul is a savage out there. If I catch another user talking sweet about Consul I’m reporting they ass! I’m not playing no more.

hashicorp.com/cgsb

https://redd.it/n2vo79
@r_devops

Automatic Semantic Version in Gitlab ci


Hi folks,

I have gitlab Repo for electron Desktop app. I have created the gitlab ci pipeline of it which creates the desktop application package for all Operating system (Mac, win and linux). It is working good.

But now I want to add automatic semantic version with tag in my pipeline. So when pipeline run it should automatically create a semantic version and tag on pipeline success.

Can any one provide me a useful resource as a example for my better understanding. Thanks

https://redd.it/n2baaj
@r_devops

If you could have a tool to solve a problem you have right now. What would it be and what would it do?

Just thought this would be fun to get other perspectives

https://redd.it/n276yp
@r_devops

Triggering a jenkins build from a Bitbucket pull request

Hi All,

Sorry if this has been asked here (I'm sure it has), but I am having some issues trying to get a jenkins build to cooperate appropriately when a pull request is created or updated.

I've got the build triggering successfully, but now I am having trouble getting it to build the correct revision, which in my case is the pull request source branch.

My end goal is to have Jenkins start a build when a pull request is created, and then either approve or deny the pull request based on the result of the build.

My stretch goal is to have a Jenkins bot actually comment on the PR with the build status, primarily so that it can indicate any issues. I have worked with a setup like this before, so I know it's possible, but I was not the one to configure it so I'm kind of at square 1 here.

I am attempting to use this plugin to achieve the job triggering at least: https://plugins.jenkins.io/bitbucket-push-and-pull-request/

The job actually triggers, but when trying to use the environment variable they reference, ${BITBUCKETSOURCEBRANCH} I am getting a failure because it's not expanding to anything. I thought maybe just leaving it blank and letting Jenkins figure it out was the right move, but that just builds master. I'm unsure where to go from here other than to ask people smarter than I for help. Please let me know what additional information I can provide.

Bitbucket version: 7.6.4

Jenkins version: 2.235.1

https://redd.it/n24e1k
@r_devops

Tired of doing configuration management or yaml engineering, what's next?

Devops Engineer who is bored writing Ansible code or tweaking yaml/json config etc. I do find terraform a bit better as I can relate the abstraction to Go progamming (lists -> slices, maps -> structs etc).

How do I do get into more programming level tasks apart from writing occasional CLIs, lambda code but at the same time not be an API dev?

https://redd.it/n3a4zv
@r_devops

I am confused on how to write code on Jenkins to connect to sonarqube

I have bitbucket sending an alert to Jenkins once a commit is made. This will kick off the build in Jenkins. I am just not sure on how to write the code so that I can connect to sonarqube. Any advice will be appreciated

https://redd.it/n3kqtu
@r_devops

Transitioning to DevOps/Infrastructure from Help Desk?

I've been working in tech for about a year, 1/2 a year at an internship pushing rails web apps to Ubuntu servers while also helping with internal pentesting. The other half so far has been at a help desk role because the company I interned at couldn't sustain itself during the beginning of COVID. Help Desk has been great and all, I got to learn more about Windows environments, but it's not where I want to BE.

I'm used to scripting all my troubles away and using new and innovating technologies to solve problems, but that's just not what my job wants/requires. I still script (Powershell for the win!), but due to the human need of supporting oneself via monetary compensation, I can't spend ALL my time spinning up Linux boxes and configuring them with my ansible playbooks. However, I'd love to make a career of it. It's very much something I love and enjoy, and I do have experience from my internship and self-study/projects. Are there any tips to make a successful transition from help desk to DevOps?

​

tl;dr: I've got experience with cloud infrastructure from an internship but couldn't continue it due to COVID, so I've been working Help Desk to make ends meet but I really want to get back to the DevOps side of things.

https://redd.it/n3jejx
@r_devops

Looking for freelance DevOps/Aws/ci cd projects.

Hi guys,

I want to learn devops and looking to solve problems , build your ci cd flow or solve any challenge you or your devops team is facing.

More than any thing I am looking to gain some more experience with real world use cases.

I am up for volunteering positions for any freelancer outthere who take projects via upwork or freelancer, would be glad to work alongside.

Skills : aws, jenkins ,Docker , Kubernetes, cloudformation

Any leads or opportunities will be much appreciated.

Thanks

https://redd.it/n3f3k7
@r_devops

Actually working, resource-efficient, user-friendly local development environment for a HA multi-master Kubernetes (K3s) cluster

After long nights of troubleshooting and reading Github issues, I can finally share my work: https://github.com/medmouine/vagrant-k3s-HA-cluster


This repository contains all the configuration and scripts you need to run a Kubernetes (K3) cluster locally in a highly available, load-balanced manner.


It is the easiest, best (and probably the only) way to run an efficient (low resources, user-friendly, up-to-date) Kubernetes cluster on a local development environment with just a few CLI commands.


Hope this will be helpful for your personal or professional projects.

See README for details.

https://redd.it/n3ghq1
@r_devops

Terraform AWS FIPS provider

Hey guys, this was a royal pain in the ass to type up and I figured you guys may find it helpful. I've had some compliance requirements and FIPS 140-2 validated encryption is a requirement. I went through the AWS docs and got every AWS FIPS endpoint into the AWS provider.

You can maybe modularize something like this but I haven't ever tried setting up a provider in a module so I'm not sure if that's possible.

Here's a blog link: https://blog.kwnetapps.com/terraform-aws-fips-provider/

Here's a link to the github repo: https://github.com/Kaydub00/terraform-aws-fips

Now, there's probably more to meeting these requirements for your org, but if you need to meet these requirements and you're using TF and AWS you'll need this. Granted I've never been asked by an auditor to see this stuff, you may get an auditor who knows their stuff.

https://redd.it/n3af0g
@r_devops

Monorepo for microfrontends, how to set it up?

We have split several applications with microfrontends. Now these microfrontends can be reused across different applications. However until now each application has its own repo.

Should we swich to a single monorepo in order to make this more effective and manageable? how?

We are working on Azure Devops and frontends in Angular.

Any suggestion welcome

https://redd.it/n36euv
@r_devops

Which cloud provider to choose for an EdTech business?

Hello,

I wanted to know which cloud provider to choose for a better service and easy experience to host an EdTech business'? I am aware of 300 free credits provided by GCP for the initial signin, but AWS is one of the leading provider, hence, I'm confused.

https://redd.it/n32zn6
@r_devops

how to troubleshoot airflow csv inserts to db ?

i.e

I managed to extract info from db A and dump to csv.
Now airflow will pickup the csv ( success ) but unable to insert ( usually it's working ).

I alread created the table before the insert.

If the input csv has say 50 cols, how do we need which columns has issues? I suspect is incompatible columns. I have several timestamp and the rest are char.

Thanks.

https://redd.it/n31uql
@r_devops

Traditional on premises to Cloud

I’m a network engineer would love to more about devops. I don’t see any use to go above VM’s for small org like us. But want to learn so we may find a way thats useful for us and also reduce the cost in the long run. Where would small orgs can use devops? How containers came into play? Is there a good book that gives us the idea of moving to cloud and from on premises and from VM’s to containers? Where the regular IT is going in terms of research? If we don’t have a app on our own and uses only third party apps? Any useful going this devops route?

https://redd.it/n43nia
@r_devops

Are there any reputable degree programs out there that specifically focus on DevOps?

The main thing holding me back from getting a degree is I don't want to be learning outdated or irrelevant concepts that I'll never end up using again. I know of WGU's Cloud Computing program so if anyone has some opinions on it feel free to speak up, or if there are any good alternatives out there from other reputable colleges would be great to know.

https://redd.it/n46fmb
@r_devops

Open Source Programming Help Needed!!

Hi everyone,

We are a team of graduate students at UCSD working on a combined Open Source analysis tool that would make the process of scanning large banks of OS code for security vulnerabilities and license compliance issues.

We have a starting point, but are looking for help with our next steps. We have built a short anonymous survey of 16 questions (less than 10 minutes). Any help would be greatly appreciated. Thanks!

The link to the survey is below:

https://ucsd.co1.qualtrics.com/jfe/form/SV\_6eUZXbkJsoQ7cOO

https://redd.it/n4b9br
@r_devops

What can I do if a company warned me about copyright infringement on a dictionary?

I developed a dictionary, a simple collection of words which I scrapped from several dictionaries and removed all the visual garbage and advertisements, as a lover of linguistics I am. Now the legal department of one of those dictionaries sent me a legal warning to shutdown the site.

I am totally lost, thus I pose the following questions for which I am grateful that someone could help

Can they shutdown my website? The TLD is .pt and the server is in Germany
If I move the server to Russia, for example, with a TLD .org, will it help?

Thanks a lot

https://redd.it/n411h7
@r_devops

Thoughts on internal developer platforms?

I wanted to ask regarding thoughts about introducing internal developer platforms.

The goal of DevOps at the end of the day is to not be a bottle neck and to create pace. The way you create pace is by automating everything so that developers can move at their own pace.

Wouldn't introducing IDPs be a good way to do this?

In an ideal world, DevOps engineers can template out infrastructure modules for common use cases (think containers, static sites, etc.) Developers then can just deploy the needed infrastructure that will setup and run their applications without needing to be exposed to all the intricacies of infrastructure. If this is true, wouldn't IDPs be the best abstraction layer for developers so that they're not forced to be come infrastructure experts?

Has anyone here implemented / built out an IDP for your company? If so, how successful has it been?

https://redd.it/n3zbop
@r_devops

Can single Loki instance works with multiple projects?

I am wondering whenever I can run Loki instance for all my projects and correctly authorize them with something like access token, or I will have to create as many Loki instances as many projects I have (which does not make sense for Grafana single DataSource).

https://redd.it/n4f2i7
@r_devops

C is missing from the majority of CI/CD implementation

The vast majority of CI/CD implementations that I saw, especially among enterprise companies, are completely missing the point of merging to the mainline frequently (e.g., at least once a day). Yet, everyone claims to be doing at least CI, if not CD. Why is that?

https://youtu.be/0ivcSjpUzl4

https://redd.it/n3yg6v
@r_devops