Has anyone implemented ‘Central’ CI/CD Pipelines

I want to bring control over our pipelines and have thought up the idea of running all our CI/CD pipelines from our admin cluster.

When a developer performs a PR, this will trigger the application build pipeline to run in the admin cluster. The deployment of the container will be directed to the Dev or Test cluster.

Has anyone deployed this or similar pattern, and are there any drawbacks to it?

Thanks

https://redd.it/sq0l62
@r_devops

To All the people who transitioned to DevOps...

To All the people who transitioned to DevOps( QA->DevOps, Dev->DevOps, Others IT,NonIT->DevOps) \- We need one help from all of you.

How you guys learned the real time experience/production troubleshooting experience for DevOps Job before moving to DevOps jobs

Doing Reading About DevOps culture ,Side Projects and practising tools is enough to handle real time scenarios in DevOps Jobs?

Basically this question is from People who is not having support from organisation (not allowing internal transition) and only way to get into DevOps by switching company

https://redd.it/sqj4c2
@r_devops

Achieving good development to production flow with kubernetes

Hi,

I want to help companies get from X to Y where Y is hopefully a much faster and much more productive place for developers and operations. I think I know what Y is, but I'm not sure how to get there.


Y would look something like this (1-2 development, 3-4 ops):

1. Developer is working on a service with his IDE while the service is in the dev k8s cluster in the developers namespace (not on his laptop on minikube) and he can test it by API or by going to the dev website. This seems to be the most undeveloped area of the kubernetes ecosystem.
2. This one is already accomplished: when developer is done, he pushes and pipeline is triggered to test code and if it passes, creates PR. When dev is done, he puts relevant people as reviewers and assigns code review status in jira.
3. Service's PR is approved and ci is triggered to create image + update services repo with new tag for a tool like argocd to do CD. This will hopefully shift us to gitops.
4. Argocd style tool will trigger and deploy and rollback if pre-defined rules are met (latency spike for over 1 minute / x amount of 4zz-5zz errors in y minutes....) + run qa tests in some cases.
5. I want a secret update to trigger re-creation of all deployments that use that secret (currently we use vault for secrets if it helps).

What tools / practices could help me achieve Y state? 

Thanks for any advice!

https://redd.it/sqo09t
@r_devops

How would you go about a full Operating System upgrade on a large fleet of servers?

How would you automate the upgrade of OS of a large fleet of servers? Can it be done with Go? What would the best practices be?

https://redd.it/sqnshr
@r_devops

Is the book "Terraform: Up & Running" still relevant in 2022?

I am looking for resources to learn Terraform. I see that the book "Terraform up and running, 2nd edition" (https://www.amazon.com/gp/product/1492046906) has good reviews but it was written in 2019 when Terraform was still in version 0.12. Now it is in version 1.1.2.

I don't know how much difference that makes for someone who wants to get an overall tour of the tool. Is the book still relevant or should I be looking for more up to date resources?

https://redd.it/sqqfz0
@r_devops

A dashboard to handle SSL certificate

Hello all,

We have a serious problem in managing SSL certificates in our environments, especially the ones used by ibm websphere, is there any dashboard solution that i can use where i can see all my SSL certificates and sending me alerts when the expirations day is close??

Thank you all,

https://redd.it/sqq7xe
@r_devops

Is the book "Terraform: Up & Running" still relevant in 2022?

I am looking for resources to learn Terraform. I see that the book "Terraform up and running, 2nd edition" (https://www.amazon.com/gp/product/1492046906) has good reviews but it was written in 2019 when Terraform was still in version 0.12. Now it is in version 1.1.2.

I don't know how much difference that makes for someone who wants to get an overall tour of the tool. Is the book still relevant or should I be looking for more up to date resources?

https://redd.it/sqqfz0
@r_devops

Automating GitHub users' access

Does anyone know of an open source project for automating GitHub user access via the API and pull requests?
I worked at a company a while back that had an automated tool for this.

e.g. Are you a new team member?...add your name to the json file of this meta repo, make a PR and the lambda function (?) will add you to the relevant repos for your team. If a team member leaves, anyone can make a PR and the former team member's access is gone. The tool we had even handled AWS permissions.

What I like about this approach is that the burden doesn't fall on one individual (admin, DevOps, etc) and it could tie into LDAP/AD.

I've googled for another similar project but didn't find anything. I thought I'd check here before building it from scratch.

https://redd.it/sqsou2
@r_devops

Which company is killing at devops?

Note: This is not a job post.
I was just curious to know if as a devops engineer, you could land in one dream company which would it be?

Which company is killing at devops? I know about Netflix, Spotify and Amazon though.

https://redd.it/sqs9qy
@r_devops

What AWS service is crucial / very useful in day to day operations, but often overlooked?

This is an open question, it could be anything from WAF to AWS config idk. As per title, what AWS service has changed a lot for you in your day to day operations, but is unpopular?

https://redd.it/sqt5j8
@r_devops

Rebasing and force pushing

When do you decide to do a rebase rather than just push another commit to go back to the state you want. And how do you rebase and force push safely?

Edit: I’ll add a particular example. Say you have a feature branch where master has been updated since you branched it. Do you rebase the feature branch and force push or merge.

https://redd.it/sqwzo5
@r_devops

What is your coding skill level and what is your language of choice?

As I'm trying to land a job at one of the bigger companies (e.g., Disney, Netflix, TikTok, Amazon, Google, Tesla, etc.) instead of the smaller startups that I've worked at most of my career as a DevOps engineer, Cloud architect, or systems engineer, one my of main weaknesses (that I've noticed) is coding. Specifically around solving exercises on leetcode, hackerrank, etc. Even though I know how to code in Python and PowerShell, in my day-to-day, I'm using logic to interact with other systems. Moreover, I understand and leverage fundamental programing concepts like if statements, loops, variables, functions, etc. I don't really deal too much with classes and beyond. Unfortunately, since most bigger companies use those types of sites like leetcode and hackerank as a benchmark, what advice do you have on those types of problems? Admittedly, I'm not too good in math and I didn't goto college. I'm more self taught and have been in IT a long time. I was thinking of focusing on pre-algebra, then college algebra, then business calculous. This should help me better be able to identify how to solve these problems, then the syntax seems like the easier portion. The problem is i have no idea what approach to take when i look at this problems on those sites.


In summary, which would you recommend? Should i
1. stare at leetcode problems until i get it?

2. Ensure that i have the fundamental knowledge of pre-algebra, college algebra, and business calculous which would make it more approachable and I only have to focus on the syntax of the language?

3. Something else i haven't thought of...

​

I know this is a long post but I want to be intentional as possible and learn from my peers. What are your thoughts?

https://redd.it/sqx7bx
@r_devops

Very Sus/Scammy-Sounding Interviewees This Past Year

I've interviewed dozens of people over the past few years. What I've noticed recently, as DevOps-type jobs' pay has gone through the stratosphere, is what I can only classify as job seekers who are straight up lying or being coached in interviews.

Here's what I've noticed about how it goes with these types of interviews:

1. The interviewee sits so close to the screen you can only see just their face, and usually no neck, shoulders, or torso visible.
2. When asked to summarize their job history by an interviewer, the interviewee reads a prepared script that is keyworded all about the same, the commonality of the language pattern having a lot of "such as" phrases. Following their eyes on camera, I can tell they start with their eyes at the top and their eyes go down the screen as they read.
3. When asked to quantify their proficiency in a certain area (say from 1-10), they usually say somewhere from 6-8. Upon asking them to explain how to use said tool or to explain when you would decide to use a part of the tool over another, they seem to immediately shift their view, as if to start searching for the answer. After a few seconds, they say "can you hear me?" like we had lost connection. When watching them, I can tell there is no loss of connection. After we go back and forth saying we can hear them now, they suddenly have a canned, but still very generic response to the question asked. For example, if I were to ask when to use an Ansible role, they would eventually come up with what an Ansible role is defined as and say that. Knowing what to look for now, I asked them to explain to me how to use an Ansible role in an actual playbook. They usually at this point start rambling in generalities and when pressed, cannot provide the answer.
4. Any topic the interviewee knows they immediately say what the answer is; any one they don't, they will use the excuse of a connectivity issue or that they could not hear. But it's only for something they don't know.
5. All the questions are answered in general, sometimes highly-keyworded terms. Never any specifics you would expect if someone had hands-on knowledge.

Has anyone else noticed these patterns lately? What do you do to quickly weed out these types of job seekers? Is this a widely known thing?

Thanks!

https://redd.it/sqz6u6
@r_devops

Picking between a contract or staying as an employee (does the contract sounds more DevOps?)

I recently interviewed for a 12 month contract position and got an offer. I'm currently an employee for a different company.

The contract sounds more interesting as it'll involve embedded device deployments using OCI runtimes where my current place doesn't use containers because of separation of concerns span across too many shared scripts executed by several pieces of software that isolating the dependency trees would required time the project does not have time to allocate.

Mainly I prefer OCI and CI/CD development work over pure configuration management of VMs with no other type of work in sight for the next year aside from writing network policy scripts on a fairly large and confusing code base that management agrees needs refactoring but there's no time for it.

I'm trying to ask how'd you determine what road you'd accept to continue down on? The OCI & CI/CD development 12 month contract or the nice stable job where it's clear what you'll be exposed to over the next year with maybe the opportunity to help refractor the code base into using OCI containers.

Container building, managing them, managing the VMs and CI/CD development is what I thought I was hired for in my current DevOps role. But it's turning out to be VM package maintenance, patch package upgrades, and network policies. Is that the typical DevOps employee role because of its long term needs are more firefighting than a contract to create reproducible environments to run software in? I can't tell if I'm being shortsighted or too new to the field or if staying leads to what I thought DevOps would be when someone else is offering the work on a contract (so you know to some degree it's not being changed on you but you probably will deal with feature creep).

Thank you for your input.
I did software development for 8 years before getting into DevOps.

https://redd.it/sr03nd
@r_devops

Hashi Boundary vs. Teleport vs. StrongDM;

Is anyone using these tools to manage cloud application access? I am wondering what mass like or ⠀dislike about these tools or if any of them are any good.

https://redd.it/sqyk1r
@r_devops

Is it okay to half-ass a solution, when it’s temporary?

Yeah, the big catch right, it’s never temporary..

So, situation:

Platform: windows server, php.

In a recently acquired business, their publishing to both prod and UAT (and I’ll assume dev as well, though the devs themselves have access to that and do their own publishing) is done by RDPing onto the servers, where they have git installed, and do a “git pull” to bring it up to date.

On UAT, they actually charge branches to publish to it what they want.

“It’s always been this way” - but to me, this is horrid! Beyond horrid, in fact.

First, I should ask, am I off base here? Is this valid/normal?

It’s really only come to my attention lately when the guy in my team from that business went on leave, and I was suddenly pestered by the dev team to routinely (2-3 times per day) do a git pull for them on the UAT servers.

I’m big on automating the hell out of everything, and this just tells me this team member is spending way too much time doing this manually.

Now I’d love to remove git from the servers, use bit bucket pipelines to stage a deployment and push it to the servers, but I’m told by the dev lead they’re “extremely close” to having the code able to run in containers. (Frankly, I don’t know what would really prevent php doing that in the first place, but not my place to argue), meaning these servers all get destroyed very soon. I hope.

So rather than investing time to build all of that, so I was thinking, would it be “valid” to use a pipeline commit trigger that just remotely executed the git pull on those servers, so when a commit is merged in, it will auto-pull it on the servers. It still feels icky to me, but I could knock that up really fast without any risk to operations and minimal work, while waiting for the containers, or is it better to do it right and proper now, even if the containers are imminent?

https://redd.it/sr88yw
@r_devops

How much of an upgrade would this be?

My current machine has 32 GB of memory and here are the specs:

​

SMBIOS 2.7 present.

​

Handle 0x1100, DMI type 17, 34 bytes

Memory Device

Array Handle: 0x1000

Error Information Handle: 0x0000

Total Width: 64 bits

Data Width: 64 bits

Size: 16384 MB

Form Factor: DIMM

Set: None

Locator: DIMM 0

Bank Locator: Not Specified

Type: RAM

Type Detail: None

Speed: Unknown

Manufacturer: Not Specified

Serial Number: Not Specified

Asset Tag: Not Specified

Part Number: Not Specified

Rank: Unknown

Configured Clock Speed: Unknown

​

Handle 0x1101, DMI type 17, 34 bytes

Memory Device

Array Handle: 0x1000

Error Information Handle: 0x0000

Total Width: 64 bits

Data Width: 64 bits

Size: 16384 MB

Form Factor: DIMM

Set: None

Locator: DIMM 1

Bank Locator: Not Specified

Type: RAM

Type Detail: None

Speed: Unknown

Manufacturer: Not Specified

Serial Number: Not Specified

Asset Tag: Not Specified

Part Number: Not Specified

Rank: Unknown

Configured Clock Speed: Unknown

​

Architecture: x86_64

CPU op-mode(s): 32-bit, 64-bit

Byte Order: Little Endian

CPU(s): 8

On-line CPU(s) list: 0-7

Thread(s) per core: 2

Core(s) per socket: 4

Socket(s): 1

NUMA node(s): 1

Vendor ID: GenuineIntel

CPU family: 6

Model: 79

Model name: Intel(R) Xeon(R) CPU E5-2686 v4 @

2.30GHz

Stepping: 1

CPU MHz: 2300.084

BogoMIPS: 4600.16

Hypervisor vendor: Xen

Virtualization type: full

L1d cache: 32K

L1i cache: 32K

L2 cache: 256K

L3 cache: 46080K

NUMA node0 CPU(s): 0-7

​

​

I want to switch to this one, so I was wondering how much of a speed boost I would get. I am worried, because this one only has 16 GB, so I have no idea how it would perform.

​

c5.2xlarge 8 16 EBS-Only Up to 10 Up to 4,750

​

https://aws.amazon.com/ec2/instance-types/c5/c5.2xlarge

https://redd.it/sr89vo
@r_devops

"Daddy, what do you do at work?"

"Well sweetie I just stand here on this balance board and type stuff, occasionally breaking for more coffee."

"Why were you yelling at that man on the phone?"

"Oh, well you see princess, he's on what we call 'the CloudFlare sales team' and he won't stop calling daddy 10 times a week."

Seriously though, my kids are genuinely curious about what I do. Every time they ask, I try to answer and realize what I just said made no sense to them.

FWIW, they're 8 and 11. My son (11) is even starting to occasionally ask "so what are you working on right now?" It sucks because I'd really love to answer him but even when I try to dumb it down a few levels, nothing I say makes sense to him. So I usually just mutter something like, "ugh... trying to fix this shit." Lol

How do you all explain your job/tasks to your kids?

https://redd.it/sra77u
@r_devops

How do use Go or Python in your work?

What are the tasks that Go or Python help you solve in your work as a devops eng?

https://redd.it/srfk74
@r_devops

I have this idea. Thoughts?

Kubernetes ecosystem is pretty saturated. Everything you can think of there is already tool ready for you. However this is not the case for Linux automation / configuration management ecosystem. There are Chef, Puppet, Ansible. But they feel like they're not enough or at least not on par with k8s ecosystem. I wish if there was a tool like ArgoCD but for configuration management / state management for Linux itself. K8s is cool and all, but it runs on Linux. Linux servers have to be provisioned, automated & maintained in a long term. This is no easy task. Currently I'm working with two major tools. Puppet & Ansible. They're both useful in their own terms. IMHO Ansible's agentless mode comes with both advantage & disadvantage. In k8s ecosystem I just use ArgoCD and connect my git repository and forget about it. Unless there is an error on ArgoCD, I don't care. I know it's applied automatically and running healthy. However I cannot do the same for Linux server provisioning. Ansible doesn't have an agent, it's one shot operation. Puppet has agent but it's not realtime, AFAIK it runs on certain interval, default on 30 mins right? So what I really want to have is something like "ArgoCD but for Linux automation".. Imagine you define your Linux server's state in your git repository and your tool handles rest of it in realtime. It ensures your Linux server's state matches what you have defined in your git repository. Does this make sense? I don't think this kind of workflow doesn't exist currently unless I'm missing something. What's your opinion on this approach?

If there were a such tool would you use it? Is it already possible with certain tools? If yes please let me know. If no I'm willing to create an open source tool for this exact use case. Please let me know your opinion.

https://redd.it/sqnd12
@r_devops

Need help with side project deployment

I have several side projects and i am in the process of deploying on right now. I am trying to use GCP free tier. I dont want to use heroku. Any suggestions on how can i setup alerting, automation, logging and other required stuff ? Or Any guide i can follow. I will be deploying my other projects soon and this one is taking too much time.

https://redd.it/sqq6pf
@r_devops