What's going to be the next big thing after DevOps?

Hey guys, a question that just sprung up in my head - today there's a lot of hype around DevOps, but what will be the next big thing?

https://redd.it/ztbpod
@r_devops

Need Help in selecting a CICD tool AWS CodePipeline Suite or Gitlab

Learner Here, Starting a small project and would like to learn and implement CICD for a project .

Need some help in deciding a CICD tool for getting things started for a web app project which relies almost AWS Infra (Server less). So would like to hear some thoughts on which tool should I be considering for a small team (3) and benefits and pitfalls of AWS toolkit over Git lab.

Would like to hear some insights from people using these tools in daily work and know more about ease of setting things up and cost wise

https://redd.it/zu7zmc
@r_devops

How to calculate hardware requirements for a hypervisor?

hello guys,

Wish everyone is doing okay.

I would like to setup a baremetal hypervisor that would virtualize \~25 vm for 25 users.

The vms mostly would be used for sending email , working with docs and will have software package like microsoft office,adobe etc and all of them running Win 10 Enterprise.

How to calculate minimal hardware requirements for hypervisor server and what should one take in consideration, are there any better ways?

https://redd.it/zu7wsu
@r_devops

Billions of unnecessary files in GitHub

It seems a lot of people who use #GitHub don't know about .gitignore or they misunderstand how to use it. This leads to billions of unnecessary files: generated files, 3rd party installations, caches etc. to be added to git and GitHub.

Not only is this a waste of space and a waste of network bandwidth when you clone a project, it also makes tracking the real changes more difficult.

https://dev.to/szabgab/billions-of-unnecessary-files-in-github-i85

https://redd.it/ztfayw
@r_devops

Advice needed for someone willing to get into Devops

Hello.

I've been working on sysadmin role for almost 8 years, mostly Windows Servers, Hyper-V, Networking and Firewalls stuff, had some certifications in the past, MCSA, CCNA, some firewalls certifications and so on, have played a little with Azure and O365, nothing too deep. All that time been working for a small consultancy company with small business customers. Before that I've worked as web developer with PHP, so I think have a decent programming logic.

Now I am trying to migrate to a "DevOps role" (I understand that there is no such thing as a role and I also understand the DevOps culture, thanks to this sub), so I have changed job to a bigger company (9000+ employees) with an internal dev team, multi-cloud environment (mostly OCI) and lots of oportunities to put in practive some DevOps tools, but right now there is no such "devops role" in this company, I am hired as a midle Infrastructure Analyst.

So I started learning some basics stuff (linux mainly), and after 6 months I already deployed some automation using Ansible and Powershell, developed a web app using Python and Flask that integrates with Terraform to provision VMs to our local VMware Infrastructure (VxRail), learned Azure and got two certifications (AZ-104 and AZ-400), and was studying for CKA to learn Kubernetes (passed on first attempt), I like sutdying for certifications because this way I can have something like a study plan.

But now comes the point where I currently am, with this little experience I can't even be called to an proper DevOps role interview, I know there are lots of tools and expertise that I still have to learn and this gets me overwhelmed, I am planning on learning Terraform and Jenkins deeper and study for AWS Certifications (Sysops and Devops) but I can't really feel secure that this is going to give somewhere because I feel that I am always running from behind, feeling like I have lost the timming.

Sorry for the long post and thanks to anyone that read it.

https://redd.it/zto2yx
@r_devops

Should we be afraid of AI and a more advanced ChatGPT automating all engineers out of a job within the next 10 years?

Basically the title

https://redd.it/ztot6t
@r_devops

Alerting rules "libraries, compendiums, or bundles:" where to find already-written, useful alerting rules for Prometheus, especially for the AWS Cloudwatch Datasource metrics?

Prometheus has no out of the box alerting rules. The list of such "libraries of useful rules" I have so far is:

1. https://awesome-prometheus-alerts.grep.to/ (many examples, my primary reference)
2. https://alex.dzyoba.com/blog/prometheus-alerts/ (examples, links to other pages)

Does anyone have a source for Cloudwatch alerting rules, or did you write them all by hand? Any other lists of pre-written rules for other exporters and/or Data Sources? I've decided on Alertmanager instead of using Grafana Alerts, so I'm not really looking into Grafana Alerts.

https://redd.it/zsvasl
@r_devops

is eBPF the cool thing now?

Lately I am getting a lot of popups on media prophesying the rise of eBPF era and the doom of sidecars approache.

Any thoughts or experience about this?

https://redd.it/zucclf
@r_devops

December Challenge: Kubernetes Certified within 3 weeks

I would like to share my victory with the community when it comes to a challenge I set for myself for December. I am currently a Junior DevOps engineer. I have been working for 1 and a half years total and I decided to challenge myself to get all the 3 Kubernetes certifications (CKA, CKAD and CKS in this order) within 3 weeks.


My challenge (so my studying as well) started on December 1st and then I had my CKA on the 5th, my CKAD on the 12th and the CKS on the 19th. I managed to pass all of them and I also wrote a bit about the experience I had on my blog:


\- CKA: https://mirceanton.com/posts/2022-12-06-my-cka-experience/
\- CKAD: https://mirceanton.com/posts/2022-12-13-my-ckad-experience/
\- CKS: https://mirceanton.com/posts/2022-12-20-my-cks-experience/

https://redd.it/zumyqi
@r_devops

A question that puzzles my mind, how can a cloud server like gdrive transmit data without congestion? even the best SSD would get slow if there are multiple reads and writes at the same time.

As the title says,
A file from gdrive could get more than 10 connections at the same time, and that single drive might be hosting tons of other data too which might be active too, so I don't see how one drive could survive hundreds of simultaneous data access.
Unless maybe active data above the threshold could be moved temporarily into multiple drives without us realizing it?


Thanks!

https://redd.it/zuob2s
@r_devops

'Best' Build System Language & Interface?

So in my workplace we have two teams (broadly comprising application side and operating system side for embedded control systems). We both contribute to a sort of internal 'platform' of scripts for various parts of build, test and analysis on internal products we deploy to.

The operating system side have implemented a lot in 'make' (and I mean not just interface but a lot of logic and parallelisation within the makefiles) - and frankly it makes my head spin and only the one person on the team can understand the bizarre ways they've stretched the capabilities of make to its limits. On the application side we have primarily implemented in PowerShell (mainly because it is a good language for integrating with Azure DevOps pipelines and maintaining our own Windows machines), with an 'interface' maintained with lots of one liner batch scripts. However, whenever we have a new hire or someone from the dev teams who are dipping their toes in DevOps, they rarely know any powershell and need to learn a lot from scratch.

Basically we've started butting heads over implementation and it's starting to become very counterproductive. Some things we've both implemented in our own way because we haven't been able to align which is a pretty big waste of time.

Ideally in the new year I would like to propose a rewrite in something we can both agree on. I was wondering what languages and build systems people think are clear and easy enough to maintain while still being powerful. Is there really a benefit to using a proper build framework over a bunch of script files? For reference, some of our code is in C, some is in Java (Eclipse plugins for an internal version of Eclipse we produce), some MATLAB scripting, but mostly Simulink models (yes an interesting mix I know). So there's quite a lot of calling external executable tools on binary files (particularly MATLAB).

I'd be interested in what people have done in their workplaces and how it has worked out for them?

https://redd.it/zuezi1
@r_devops

What's the best resource to learn EKS (paid or free)?

Hi,
I am looking for a course which covers everything needed to setup a production ready EKS cluster.
Thanks

https://redd.it/zugcu8
@r_devops

Best/low maintenance devops toolchain for basic sass?

I'm looking for a new devops stack, I got pretty deep into k8's, and I feel like it's just too much overhead. It is cool, and does a lot of great stuff, but it seems like there are other solutions that are better for basic sass app type stuff.

I start a decent amount of projects, and so I'm looking for something that's fairly low maintenance. I know nothing is 0 maintenance, but the less the better. I do lean towards AWS, but I wouldn't mind flexibility in different cloud providers as well. If a solution is really elegant though, only supports one provider, i could get behind that.

I've been doing a little research, and these are some of the options I'm considering.

1. App engine/Elastic beanstalk with terraform
2. cloud formation, with lambda
3. CDK, with lambda and api gateway
4. serverless with CDK
5. SAM
6. Some other combination of the above technologies.

The only requirement I have is that I should be able to develop offline. I can't stand having to be online and deploying things in order to develop / test.

Currently, I'm kind of leaning towards serverless with CDK (if I can use CDK offline). Anyone have any experience with that? Did you have any long term issues or drawbacks?

I typically also use hasura with postgres as well, so I don't want some premade firebase type solution

https://redd.it/zumr7u
@r_devops

Looking for insight into this DevOps books

Hello everyone one i was looking for reviews and searching these books and came here looking for more thoughts on this

https://www.fanatical.com/en/bundle/dev-ops-bundle-3-rd-edition

That's all thank you!

https://redd.it/zuzzug
@r_devops

Beanstalk Blue/Green Deployments - Inactive Environment Receives Traffic Up to 48 Hours After Deploy

We've implemented B/G deployments with 2 Beanstalk environments using CNAME swap as described in AWS docs https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.CNAMESwap.html

I see in the end of this article they have an explanation for what we experience.

> After Elastic Beanstalk completes the swap operation, verify that the new environment responds when you try to connect to the old environment URL. However, do not terminate your old environment until the DNS changes are propagated and your old DNS records expire. DNS servers don't always clear old records from their cache based on the time to live (TTL) that you set on your DNS records.

We observe in load balancer logs that inactive environment continues receiving traffic in the next 2 days after deploy. At first, it's something like up to 10% of all traffic and then it falls down to zero as time goes on. We have TTL of few minutes set on our end in route53, so it looks like it's a client side DNS cache that doesn't refresh DNS records.

I'm worried about this as it means that if we deploy some hotfix for example, these 10% of clients with stale DNS records won't see it immediately. Before we implemented B/G deployments, we haven't had this class of issues at all.

Did anyone have similar concerns about client side DNS caching in regards to B/G deployments and are there some techniques that may be useful to address these issues?

https://redd.it/zv4zjy
@r_devops

Storing Grafana Loki log data in a Google Cloud Storage bucket & possible alternatives

Hi all. I've been using Grafana Loki for a while to collect logs in Kubernetes and search them using Grafana.

I like that it's a lightweight solution but I had a problem where it wasn't easy to predict the amount of storage required depending on the growing amount of logs and the retention we configure. Someone on Slack suggested that I switch to object storage so I don't have to worry about the size of the volume and I can store an unlimited amount of logs, which is nice.

So today I tried setting up the Loki distributed Helm chart using a Google Cloud Storage bucket (since we use GKE and the Google Cloud Platform for everything); I got it working (I also took notes in a blog post if anyone is interested at https://vitobotta.com/2022/12/25/storing-grafana-loki-log-data-in-a-google-cloud-storage-bucket/).

To confirm that it reads the logs from the bucket I tried uninstalling Loki completely and in fact I was able to search through old logs after reinstalling it once again.

So it seems to work for the goal I was trying to achieve.

However before I commit to this I was wondering, would there be any advantage by switching to something else like ElasticSearch or other? All I want to do is be able to collect an indefinite amount of logs and search those logs easily and quickly. Is Loki good enough for this or should I try alternatives as well? if yes which one(s) do you recommend? Thanks!

https://redd.it/zv3sv7
@r_devops

Which stack to use for long term metrics storage?

Which storage to use with Prometheus?

1. Cortex (development stopped?)
2. Thanos
3. Uber M3
4. Grafana Mimir

Which one are you using? Which is one more future proof?

https://redd.it/zut835
@r_devops

Devops mentoring

I am looking to build out a mentorship program. I was wondering what people would either like to see out of mentorship program or what they have done in the past as part of their program.

https://redd.it/zvcpe2
@r_devops

CI : Run tests in parallel

Question about CI/CD best practices: Let's suppose I have an application/service that has unit tests and security tests and we have also SonarQube. Currently, I am new in my team and I want to suggest some improvements as we are migrating our pipelines from Jenkins to Gitlab CI/CD.

I think to save us milliseconds, maybe it would be better to run security tests and unit tests in parallel ( we also have some projects that use e2e and integration tests ).

Any advice or guidance on this ?

https://redd.it/zujltc
@r_devops

Check out rbac-police for k8s rbac testing

This isn't my tool, but I watched the talk on it from blackhat US this year (it's on YouTube).

It's quite interesting. It looks at the cluster roles in kubernetes and then let's you know which pods you can "escape" from and takeover the kubernetes cluster.

I ran it this week against 2 nginx ingress deployments. One was deployed from the nginx helm chart, the other from the kubernetes nginx helm chart. The official one has two ways of taking over the cluster, but the kubernetes one has none. Obviously I've switched to using that one.


https://github.com/PaloAltoNetworks/rbac-police

https://redd.it/zstcwa
@r_devops

Advice/Tips needed for new hire

I’m starting a new position (first job in swe) as a devops engineer. I’m looking for advice that will help me become productive and efficient in this role. I would also like to know what types of questions I should be asking and things to do to go “above and beyond”. What are some things that you would have done if you were a new hire or things that you would want your employees to do? Any feedback is appreciated and thank you for your time!

https://redd.it/zvi7i5
@r_devops