How to Approach Kubernetes Infrastructure With Container Sprawl in Mind

Just came across this interesting piece that discusses how to tackle container sprawl in Kubernetes infrastructure. It dives into some key aspects like resource management, logging and monitoring, scalability, security, and networking. I found it pretty useful as I've been dealing with container sprawl issues in my own projects. Curious to hear if any of you have faced similar challenges and how you've addressed them in your Kubernetes environments.

Source: https://www.devopsinstitute.com/how-to-approach-kubernetes-infrastructure-with-container-sprawl-in-mind/

https://redd.it/13f54vg
@r_devops

Oh, it's just Github having their 'routine' daily outages

In all seriousness, what's happening there?

https://redd.it/13f6f56
@r_devops

What would you want to be hired as?

What would you call this role? DevOps (not really a role), SRE... not really, Platform Engineer... maybe but also has some ops.

EDIT: This person is joining a company where others have been doing this for several years, they are not the only person doing it, just the first person dedicated to it that isn't also managing teams, or also doing product development.

This is a startup where we're starting to distribute the hats people wear but still have a ways to go, and it does have a division between ops and dev, though it's getting blurrier every month.

Responsibilities of the role are to help with all of the following but doesn't need to lead/own it:

1. AWS Cloud/Infra management. EC2, RDS, S3, MWAA, IAM using CloudFormation and the console.
2. Person that Operations can escalate to. The Ops team is trained on the application and the data but have limited AWS/Infra and real Observability/tuning experience.
3. Improve observability, ELK, Py Dash and alerting, PagerDuty for example
4. CI/CD Pipeline development and maintenance excluding writing test automation.
5. Manual deployment on areas without automation, until they help setup the automation
6. GitOps
7. Performance tuning and optimizations' in production ranging from concurrency settings for the application to rightsizing AWS Infra
8. Development Env, Docker management
9. Escalation point for Development when they have Infra/Pipeline issues. (not on call)
10. Security implementation around architecture and software, not business risk.

What other responsibilities am I forgetting?

No "On call" time but when DNS messes up and we lock ourselves out of the office like Facebook did, they would be called along with every up from them through the CTO.

https://redd.it/13exmrp
@r_devops

Hashicorp Vault TCP listener

I was wondering if my tcp listner defined in my vault config has to be HTTPS? The reason being is Im trying to utilize a pre-existing nginx (gitlab omnibus nginx) to serve as a reverse proxy for a vault instance running on the same host but every example Im seeing has the TCP listener configured for HTTP. So Im not sure what is the best case for production use.

https://redd.it/13eq2ur
@r_devops

Run ClickHouse like a Cheapskate – 6 Ways to Save Money While Delivering Real-Time Analytics

ClickHouse analytics can be fast and really cheap if you do it right. This webinar digs into the cheap part showing tricks any dev can apply to save up to 90% on cost. Here are three of many we'll discuss. First, do free as-in-beer development using open source. Second, optimize compute, storage, and memory on ClickHouse itself. Third, move off AWS or GCP completely to cheap hosting at vendors like Hetzner.

Join us LIVE for free TOMORROW 10th May. RSVP now: https://hubs.la/Q01KS44X0

https://redd.it/13d01w9
@r_devops

What should I do next?

I am currently working as a devops person in a small startup. Though my role is called a devops person I am doing most of works like sysadmin stuffs. I am learning everything side by side. Currently I learned Jenkins, ansible,docker,k8s(still learning), terraform (still learning), Prometheus and grafana. I learned AWS and got certified. I learned all these by watching tutorials so I haven't done any real time projects yet. May be I've just learned basics alone. Considering what I mentioned above as learned, what should I learn next? Should I learn coding like python?
Or I should build projects with the tools that I've learned? Which should be my first priority?
I am planning switch company after 4 or 5 months. Which should be the priority? Learn next tool or continue building projects?

https://redd.it/13fb696
@r_devops

AI tools / plugins for devops engineers?

Hi everybody,

Since generative AI seems become more and more mainstream I was wondering which AI tools / plugins y'all use in you daily business to make your life easier?

I know about ChatGPT and GitHub Copilot, however Codepilot seems to be more dev focused. I also tried to make a short Google research but there seem to be tons of AI tools out there, therefor I was wondering if you could give me your honest opinion and recommendations about some tools / plugins you learned to love :)

If you need some details for recommendations: our main techstack currently revolves around Google Cloud, Kubernetes, Terraform and some basic Cloud Functions written in Python / NodesJS. But I'm curious in general about your tips and tricks.

Thx!

https://redd.it/13clymd
@r_devops

Ask for recommendation Suggestion on learning a new skill in DevOps

Hello knowledgeable community members,I recently secured a position as a system administrator in a Linux infrastructure environment (seem quite large, > 10k servers). It appears that the company I'll be working for has a significant on-premises infrastructure rather than relying heavily on the cloud. With about a month left before my graduation, I have an opportunity to learn a new skill.

During my previous work experience, I had exposure to Linux servers and Ansible. While pursuing my education over the past two years, I continued to deepen my knowledge of Ansible and explored additional technologies such as Docker, Terraform, and GitLab CI/CD pipelines. I worked on my own projects involving Terraform+Ansible Infrastructure as Code (IaC) for server provisioning and system configuration, docker image building and container deployment. However, I'm aware that there's still much to learn in these areas. I don't think I have in-depth knowledge of everyone here.

I also had the chance to experiment with AWS and completed projects like deploying a high availability and scalable WordPress website, setting up an S3 bucket with CloudFront, utilizing OAC and WAF, and implementing a simple newsletter system using Lambda functions, API Gateway, SES, SNS, and DynamoDB.

I also played around with some monitoring tools like Prometheus, Grafana, Loki, Protmail, and Telegraf during these two years. e.g. Doing some deployment and making it work with my homelab environment, getting the metrics and set up the alert with alertmanager and send to the my own slack channel. e..g CPU/memory spike alert, disk usage etc.

Regarding my programming skills, I admit that I have room for improvement. I have worked with Python and bash scripting in the past, but unfortunately, I haven't had many opportunities to utilize them extensively in my work. Also, I have found ChatGPT to be incredibly helpful, especially when designing scripts for specific tasks and integrating them into cron jobs. Do you have any suggestions for me to learn programming skill in this particular area?

Now, I am seeking guidance on the next steps in my career development. Should I focus on further honing the skills I already possess or should I pursue a new skill set? I don't want to squander the upcoming free month, and my goal is to transition to a junior "DevOps" role next year. Automation is an area I particularly enjoy and excel in. I have discovered a great deal of satisfaction in working in this area.

I would greatly appreciate any advice or suggestions from this knowledgeable community. Thank you in advance for your insights and support.

https://redd.it/13fdklg
@r_devops

Coming from non IT background. Should I start with learning Linux? How is Imran Afzal’s video lectures on Udemy to learn Linux?

A friend suggested to begin with Linux to get into devops. Would like to know this community’s view. Thanks

https://redd.it/13cjzmm
@r_devops

Create windows node cluster Kubernetes in windows host

I have an IIS app with a docker image in windows/amd64 (arch). Looks like I need to create Kubernetes windows node cluster to orchestrate this docker in a windows host.

https://redd.it/13ff8gg
@r_devops

Is Dynatrace that bad?

I see mostly disparaging comments here about Dynatrace (most related to pricing). My company is positioning to use Dynatrace (money isn't an object for them), so pricing aside what can I expect?

My background is mostly working with open source stacks like Prometheus, Loki, Grafana, Tempo, etc.

https://redd.it/13c6467
@r_devops

How to increase visibility on GitFlow?

I'm working on a project that is relatively big, but very unstructured. Before I joined the team, they were working in some sort of trunk-based development, but with too many compromises. There are no feature flags, or even enough test coverage to confidently merge to master or do the deployments.

I decided that, given the lack of tests and bad integration pipelines, we're moving to Git Flow to enable more clarity about branches and environments until we build more maturity and implement the necessary stuff for TBD.

My question is, how to increase visibility about what is merged to where with GitFlow? Do you guys have any strategy to deal with that?

I have worked many years with both, GitFlow and TBD, but my main struggle with the former was to have a clear overview of the contents of each environment, especially for non-technical stakeholders like Product or QA.

https://redd.it/13fhpmn
@r_devops

How do you structure deployment workflows?

We've been busy the past year migrating our apps from docker containers on VMs to Kubernetes. We have Helm charts that get deployed via Terraform (helm provider) using Github Actions. We use three branches (main, acceptance and production), and only merge commits upstream. This kind of works, but leaves plenty of room for human error, as when we want to do a production deployment, the changes first have to go from feature branch > main (deploying to staging cluster) > acceptance (deploying to acceptance cluster) > production (deploying to production cluster).

Sometimes changes already get merged to the main branch to only be deployed on staging, but a deployment to acceptance comes in between and inadvertently deploys unwanted changes to acceptance.

I'm looking for a way to streamline this whole process, while retaining the advantages of automated pipelines.

https://redd.it/13fiv7o
@r_devops

Azure Pipeline Help!

Guys, I am an aws guy new on azure domain. Need to create a simple pipeline for both PHP and vue.js app. Basically, it is CD part, not needed for CI or build stages. Please check the below scenarios and help me how should i process or should I use ansible? Also let me know how permission should handle for this operation. Thanks.For PHP, steps are below:

pipeline will ssh/connect to azure vm server.

cd to app directory

git pull

composer install

For vue.js app, steps are below:

cd to app directory

sudo npm install

sudo npm run build

pm2 delete "name" 2> /dev/null || true

pm2 start npm --name "name" -- run start

https://redd.it/13fjy8b
@r_devops

How to deal with a junior devops who is also a tech lead

I am doing a contract with a tech startup and am a senior devops there. I have a junior who has been doing tech stuff for 3-4 years now and is decent ish at figuring out stuff. He questions literally everything and doesn't want to listen or accept an answer to anything. I have the choice of either getting frustrated for an hour or just letting him do his way in things and dealing with it later.

He also happens to be the tech lead and fairly up there in management buddy buddy with the CEO and CTO.

He is a big one of scrum, agile, stories and sprints so he hates how I have done a very fluid system of design on how we work because he can never commit any real time to working and my hours are on an as available status right now at about 20 hours a week.

I normally do a description, task details, research if needed, any instructions on how to do it if I have them and a testing plan is needed - some stuff I just figure he'd know to check like if we update a version of something or a node count etc.

I do have some tickets in there for more general stuff to be worked out as more details are available. We have something about docker today he brought in the CTO for and the result was what I said don't use docker but I had it as part of the ticket to research and test better ways though the devs wanted docker for it since that was their first google search result.

I'm wondering does anyone know how to deal with this? I don't want to cause conflict or get him in trouble or be rude it's just come to where some days I say fuck it and log off even though I have time and need to get work done.

https://redd.it/13fs2ne
@r_devops

Incident Write-ups

I'd like to share my insights on how to document an incident in preparation for a post-mortem!

https://certomodo.substack.com/p/incident-write-ups?sd=pf

(cross-posted from r/SRE)

https://redd.it/13fu6wy
@r_devops

Looking for a Windows utility to recursively search and replace text

We are moving a site in IIS to another server, and for testing we need to use a second URL as we cannot take the first site down. The new site has a login field that is currently failing, and we think it’s due to the way the login is potentially referencing the old URL somewhere for the return traffic. We found some hard coded references to the site name in some custom files already, but there are too many to search by hand. The person that made the site is no longer with the company and no other web developers, so by process of elimination I’m the SME. Can anyone recommend a method or utility in Windows 2019 that can recursively search a folder and file contents, and find and replace (or list) all specific instances of a search word or phrase? I told the network guy that he broke it, so I have a few days before he works it out.

Thanks!

https://redd.it/13ftur7
@r_devops

Is My Workload Normal

Hey everyone,

I wanted to share my current experience and workload in a new position I've held for about four months now. I've been in DevOps for about four years, and in my last role, I was part of a two-man team that migrated a multi-billion-dollar company from on-prem to AWS within a year, alongside some straightforward DevOps tasks. So, I'm no stranger to responsibility. However, my current role has left me questioning whether the expectations placed on me are typical for a DevOps role, or if I'm being stretched too thin.

Currently, I wear multiple hats: I'm the lead architect for an Azure migration project, responsible for implementing DevSecOps, and in charge of both the development and operations (infrastructure) teams for DevOps implementation.

Additionally, I've been assigned tasks that veer into the realm of a security architect rather than a traditional DevOps engineer. Here's a snapshot of what that entails:

- Integrating data privacy, industry regulations, and PCI impact assessments into the SDLC.
- Incorporating a standard set of security, privacy, and industry compliance requirements into every design.
- Defining a security component library for approved components and calls.
- Maintaining a common data catalog and defining data classification for each data item.
- Leveraging threat models to inform application security requirements and security architecture.
- Including mobile security requirements in designs.
- Implementing and managing various automated security tests, static application security testing (SAST), and dynamic application security testing (DAST) in the testing environment.
- Running denial of service and security stress testing before production deployment.
- Monitoring file integrity and handling reports of security vulnerabilities from end-users.

And the list goes on, with a variety of security testing and scanning procedures.

All these tasks are on top of my responsibilities as the lead architect for the Azure migration and my regular work as a DevOps engineer. The cherry on top? The deadline for these projects is set for November.

So, I'm reaching out to this community to understand if my experience is common. Are these expectations reasonable, or am I being spread too thin? I'd love to hear your thoughts and any advice you might have.

Thanks in advance for your insights!

https://redd.it/13fwwwd
@r_devops

Are these two equivalents and when should you use one over the other?

docker run -d --name sql-container --network mydockernetwork --restart always -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=pass' -e 'MYSSQL_PID=Express' -p 1433:1433 mcr.microsoft.com/mssql/server:2017-latest-ubuntu

Is there any advantage in running a docker command like above instead of doing it like this:


version: '3'
services:
sql-server:
image: mcr.microsoft.com/mssql/server:2017-latest-ubuntu
container_name: sql-container
environment:
ACCEPT_EULA: Y
SA_PASSWORD: pass
MSSQL_PID: Express
ports:
- "1433:1433"
restart: always
networks:
- mydockernetwork
networks:
mydockernetwork:

https://redd.it/13fy5z2
@r_devops

Best practices for CICD process

We are a spring boot shop for the most part. Can any of you good people point me towards an article or a book that details a good CI process?

It, best way to deal with PR, how much validation should be done at the PR vs post merge? We keep updating our process ad hoc, looking for something to compare it too or use for reference.

I have spent considerable time online and this question is surprisingly hard to answer via google.

Thank you in advance.

https://redd.it/13fy776
@r_devops

The Cloud Native Playground

The Cloud Native Playground is in public beta - https://play.meshery.io. If you’d like to explore the Cloud Native Computing Foundation’s graduated, incubation, and sandbox projects (as well as many other open source projects), and you’re willing to share feedback on your experience (particularly around the multi-player visual Kubernetes designer), submit the form on the page and we’ll see to getting you access (this playground runs on a live cluster in which you can deploy your infrastructure designs, hence the need to sign up).

https://redd.it/13fzrod
@r_devops