How do you guys ensure database in production and uat are identical?

My company is facing this issue.

We have many databases in production, using SQL Server and Stored Procedures.

Our operations team may change the database stored procedure in production during maintenance or an issue.

But usually these dont get implemented in UAT due to human error.

How do u guys handle this?

https://redd.it/1c6vtky
@r_devops

Pointing F5 Big Ip to a Kubernetes Cluster

In my company, we are provisioning a Kubernetes cluster with Kubespray on VMware VMs. The ingress path for this cluster, both for the management API (kubectl and port 6443) and the Ingress for applications, will be handled by F5.

We are struggling since we get:

k get nodes -v=10  
I0404 10:59:25.328555   21011 loader.go:395\] Config loaded from file:  kubeconfig.conf  
I0404 10:59:25.329875   21011 round\_trippers.go:466\] curl -v -XGET  -H "User-Agent: kubectl/v1.29.1 (darwin/arm64) kubernetes/bc401b9" -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" 'https://xxxx.xxxxx.es:6443/api?timeout=32s'  
I0404 10:59:25.332888   21011 round\_trippers.go:495\] HTTP Trace: DNS Lookup for xxxx.xxx.es resolved to \[{192.168.xx.xx }\]  
I0404 10:59:55.332089   21011 round\_trippers.go:508\] HTTP Trace: Dial to tcp:192.168.xxx.xxx:6443 failed: dial tcp 192.168.xx.xx:6443: i/o timeout  
I0404 10:59:55.332242   21011 round\_trippers.go:553\] GET https://xxx.xxx.es:6443/api?timeout=32s  in 30002 milliseconds  
I0404 10:59:55.332276   21011 round\_trippers.go:570\] HTTP Statistics: DNSLookup 2 ms Dial 29999 ms TLSHandshake 0 ms Duration 30002 ms  
I0404 10:59:55.332296   21011 round\_trippers.go:577\] Response Headers:  
E0404 10:59:55.333498   21011 memcache.go:265\] couldn't get current server API group list: Get "https://xxx.xxx.es:6443/api?timeout=32s": dial tcp 192.168.xxx.xxx:6443: i/o timeout  
I0404 10:59:55.333532   21011 cached\_discovery.go:120\] skipped caching discovery info due to Get "https://xxx.xxxx.es:6443/api?timeout=32s": dial tcp 192.168.xxx.xxx:6443: i/o timeout

If we try a cURL:

curl -vvvv https://xxx.xx.es:6443  
\*   Trying 192.168.xx.xx:6443...  
\* Connected to xxx.xxx.es (192.168.xxx.xxx) port 6443 (#0)  
\* ALPN, offering h2  
\* ALPN, offering http/1.1  
\*  CAfile: /etc/ssl/certs/ca-certificates.crt  
\*  CApath: /etc/ssl/certs  
\* TLSv1.0 (OUT), TLS header, Certificate Status (22):  
\* TLSv1.3 (OUT), TLS handshake, Client hello (1):  
\* OpenSSL SSL\_connect: Connection reset by peer in connection to xxx.xxx.es:6443   
\* Closing connection 0  
\* TLSv1.0 (OUT), TLS header, Unknown (21):  
\* TLSv1.3 (OUT), TLS alert, decode error (562):  
curl: (35) OpenSSL SSL\_connect: Connection reset by peer in connection to xxx.xxx.es:6443 

We suspect about certificates but appreciate any pointers or documentation you might have.

Note: This F5 is shared among other infrastructure. So installing the f5 k8s controller is not an option for now.

https://redd.it/1c6wt01
@r_devops

Stupid question: k3s identity provider keycloak

Currenlty I have kubectl hooked up to keycloak and works pretty well but now got me thinking I can remove the need for passwords using the cluster api oidc as an identity provider.


my first usecase is to authenticate the standard serviceaccount token on minio, so that minio will validate the token with keycloak and keycloak with the cluster-api? anyone done this?


Then we coudl rolebase, postgres, minio for backups and a few other usecases I have all by serviceaccount token and a role annotation





​

https://redd.it/1c6wrdy
@r_devops

Semantic Release with Maven

I am assigned a task to implement semantic versioning to a maven project at my organization, I am using the semantic-release npm package to achieve the same. So far, I have achieved the desired behavior but one thing is bothering me a lot.

We did not use to follow any commit message standard here and the current pom.xml version is 2.5.

I want semantic release package to detect the current version of pom.xml and prepare the next version accordingly, instead of starting the versioning from scratch. How can I achieve that? I am using Jenkins as a CI agent and .releaserc.js for semantic release configuration. Please let me know the possible solutions, is there any possible way to detect versions from tags either?

https://redd.it/1c6xw61
@r_devops

K8s EKS Learning Path

Hi All

I'm a python dev who has devops experience and don't have k8s experience.

I have experience on whole aws intra including ecs. But no experience on k8s.

I have a deadline to be proficient on k8s in next 20 days for a job offer on devops ( in-company migration)

I know k8s is vast. Is there any way I can learn k8s in this time frame ?

Cloud used here is aws and eks is used. Please share your thoughts on this and please share resources.


I know this is unrealistic timeframe but I have to be proficienct enough to manage clusters if given.

I'm willing to give 6hrs a day for next 25 days to learn.

Kindly give your suggestions.

https://redd.it/1c70vpb
@r_devops

How do I fix these deployment/git workflow issues?

We have a massive and very old .NET project at work. It's codebase is a mess, there's no test suite, it's running on very old tech, and we're startup so don't have the capacity to really change any of this. But, it's our most important service.

Some quick info on our infra, we have `prod`, `uat` and `dev` environments. QA testing is done manually on `uat`.

Our git branches are setup as so:

* `master` branch -> `prod`
* `dev` branch -> `uat`
* PR against `dev` branch -> `dev`

So when we're working on a feature branch, we can deploy to `dev` via a PR, once checks have passed and it's been approved it's merged into the `dev` branch and deployed to `uat`.

**First issue**: this deployment strategy means that if I'm working on feature 1 and deploy to dev, someone else can then work on feature 2 and deploy to `dev`, overriding my feature. And when deployments are 40 minutes, this can be a bit of a blocker (and it means we can't QA test on `dev`).

**Second issue**: if feature 1 and 2 pass code review and goes out to `uat` for QA testing. But feature 2 takes longer to test, or a bug is found, feature 1 is now blocked from going to `prod`. Our temp fix for this is to cherry pick feature 1 onto a "release" branch and merge into `master` (`prod` deployment) but this is causing merge conflicts and is a bit messy

I'm pretty new to DevOps and have been asked to solve these issues. Been reading up on GitFlow and trunk based solutions but can't seem to get around our blockers

https://redd.it/1c71mpm
@r_devops

Best way to store nested json data

What's a better way to store nested jaon data

Ive been using azure tables recently, they fit my needs of storing and recalling data when I need it in powershell/python/js.

I have a system that manages renewals for a suite of products, when a renewal is coming up, I store the renewal data as a Json in a single cell.

Later the rowkey is used as a query parameter and the json is retrieved and rendered.


Over simplification but that's the idea.

I know I could flatten the json and store the data flat, but that is so much manipulation and translation that it doesn't t seem worth it.

Without flattening it, I can't directly query it in the table.

Should I be using a no-sql database to store Json always?

I do not know any no-sql, but I'm willing to start learning if that's really the best path.



https://redd.it/1c72n45
@r_devops

Understanding AWS End of Service Life Is a Key FinOps Responsibility by Marry Henry

Mary Henry from Fairwinds wrote an article about AWS increasing their pricing on EKS extended support, meaning, running out of support Kubernetes versions on EKS.

Never had that on AWS myself, but we ran into it once or twice on Azure, simply because we had no time to upgrade 🫣

AWS wants $0.60 instead of $0.10 per hour in this case. I can't remember we paid anything extra on Azure. That is insane to increase to 6x the price 🤯

https://www.fairwinds.com/blog/understanding-aws-end-of-service-life-is-a-key-finops-responsibility

https://redd.it/1c72l1y
@r_devops

How Jersey Mike's Rebuilt their Infrastructure during COVID

During the COVID pandemic, Jersey Mike’s ran an advertising campaign offering 50% off all sandwiches.

The ad went viral, jumping from an average of 800 orders a day to 80,000+ orders within the first four hours.

Jersey Mike's rebuild their infrastructure to meet high user demand, optimize cloud spend, and reduce the operation workload for their team.

https://www.fullstackexpress.io/p/how-jersey-mikes-rebuilt-infrastructure-covid

https://redd.it/1c723dz
@r_devops

Prometheus instance with data

Hey everyone

Does anybody know a good option for creating a Prometheus instance loaded with sample data for testing?

Thanks!

https://redd.it/1c76hrb
@r_devops

Have we reached a stage of AI/ML models for DevOps related tasks?

Like as simple as providing a solution for certain use cases like provisioning storage capacity as needed or open up n/w ports for a new service that gets spinned up?


I want to see if there are any development out there around data science to deal with day-to-day tasks or even provisioning infrastructure when a use case is provided.


Cheers!

https://redd.it/1c789lc
@r_devops

How is Your Company's Use of AWS Evolving?

Are your companies increasing their use of AWS services, maintaining their current level of usage, or are there instances where projects are being moved back on-premises?

I'm interested in understanding the reasons behind these decisions as well. Whether it's due to cost, security concerns, performance issues, or any other factors....

If you're comfortable sharing, I'd appreciate if you could also mention the industry your company operates in and the scale of your AWS usage. :-)

https://redd.it/1c75bpv
@r_devops

Service Mesh Preferences- Istio, Linkerd, or Something else?

Ok so we're not a service mesh, nor do we have a stake in either of these products so don't come for us in the comments (though- if you want to talk Gateways, hit us up).

We recently received a guest blog that covered a full guide on traffic management in Istio: https://www.getambassador.io/blog/istio-kubernetes-traffic-management-guide

Now, we've played well with both Istio and Linkerd from an integration + open source perspective, but we're curious...do y'all have one you prefer? Or maybe a third service mesh option you prefer more? Any downsides to either that you've seen?

Give us the best and worst you've had from using either of these. :)

https://redd.it/1c7a2t6
@r_devops

Resume/CV Review

https://imgur.com/a/fdFWyDk

[Reposting because wrong link\]

Hello all, this is my resume (version 8). I have been approached by recruiters and applied to a lot of positions. However no one has come back with an interview for me, really unsure what it is. I am a DevOps Engineer of 3 years really hoping someone can shed some light weather I am trolling with my resume or improvements. Let me know. Many thanks.

https://redd.it/1c7elez
@r_devops

Best way to learn Windows IIS

Hey all! I need to learn Windows IIS and am trying to figure out the best way to do it. Is there a recommended book, tutorial, YouTube series I should be looking at?


Context. I've been hired as a lead software engineer for a brand new department. I need to be able to deploy legacy code (OLD asp) *and* modern django applications on an air gapped system that utilizes Windows Server and Windows IIS.


All my experience is Linux and Mac. I haven't touched any devops stuff in a *really* long time, and back then I did LAMP, mostly small scale and from the developer side.

We are building and supporting internal tools for support staff. We aren't expecting high traffic.

\-------------------------------------

Bonus question/thought. No one around me has any idea of how modern software engineering is supposed to work. The deployment process from the guy I replaced was to write code in production, in the production environment, and hit save (no, I'm not kidding). I'm trying to set up something resembling modern day development practices from scratch.

Is there any benefit to even using Windows Server/IIS? Or is the group I'm working for throwing away money?

https://redd.it/1c7drx5
@r_devops

Safe terraform apply in CI

I know that some shops do terraform apply as part of their CI.


How do ensure that your CI doesn't do something horrific in production? In terraform plan we trust?

Do you use GitHub Actions? What has been your experience?

https://redd.it/1c7ismb
@r_devops

Blue Green and CD on EKS

Hey guys, so at my current company - we are having some tension between product and qa/platform. Currently we are doing manual code freezes so qa can thoroughly test a bunch of things before we push to prod. This is causing tension as product engineering wants things out ASAP. Problem being is we’re getting tons of bugs due to this. We are severely lacking the CD portion of our pipeline, and one of my plans (in agreement with the VP of infra) is to get B/G deploys rolled out using something like Argo rollouts (paired with ArgoCD). This way we can push to “prod” and have QA run as many tests as possible without causing a code freeze. We use automated tests like cypress, but another step will be automating more manual QA processes down the line.

In the meantime is this a good idea, would love to hear pushback. Currently we only have a dev env, and while yes we can do a staging - my exp in past is that things can pass staging but break in prod due to minor discrepancies.

Would love to hear what you guys have been doing, especially revolving b/g and CD. We are GitHub actions shop and deploy with helm.

Also given I have a chance to really architect this however I’d like, let me know how you would do deployments if you had no limitations. My last job we did GitHub actions and argocd as well, but did not have any B/G as there were not frequent app breaking changes.

(Note I am new to this company so I did not set any of the current processes up).

https://redd.it/1c7nnhb
@r_devops

Seeking Advice from Respected Experienced DevOps Engineers.

I’m a level 1 seller on Fiverr, offering Linux system and server administration services. To be honest, I haven't completed any course fully yet; I've only done about 30 to 50 percent of each course I started, mainly due to procrastination. However, since I'm good at problem-solving and using Google searches, I accept orders from buyers for projects that I have never worked on and have no idea how to handle.

I’m very interested in DevOps, as I believe that to apply for a job in this field, you must have skills in CI/CD, IaC, etc., and I’ve noticed many Linux sysadmins are moving into DevOps. I've realized I need to learn things properly. I’m currently about 40% through the RHCSA and my goal is to complete it without leaving it incomplete. After that, I'm considering pursuing Docker and Kubernetes as they are critical for DevOps roles. I believe good Linux skills are a prerequisite for DevOps.

I'm also practicing networking fundamentals on EVE-NG. I would like to know if I’m on the right path now. Hearing about your success story, the path you followed, and any recommended courses would be much appreciated. Thank you so much!"

This revised message reflects your updated goals and emphasizes your commitment to completing the RHCSA before moving on to Docker and Kubernetes. It’s great to see that you're setting specific objectives to guide your learning path in DevOps.



https://redd.it/1c7qrsi
@r_devops

Running Kong on GKE

Hi, new to Kong, love the concepts. Experimented with Gateway APIs on KIND, worked great. Now need to move it to GKE. There are several things I would like to do that I'm not sure how:

1. How to make the kong-proxy service to use the reserved static ip in GKE? Before I would use the kubernetes.io/ingress.regional-static-ip-name annotation on Ingress, but not sure how to do that with Gateway APIs

2. What is the right way of running multiple replicas of kong? I saw there is replicaCount value in the chart, but there is also a cluster.enabled mode. Should it be enabled as well?

3. Related to 2, how to run those replicas so that they are spread across zones & nodes?

4. I'm installing using kong chart from https://charts.konghq.com and with pretty lean values below. What are other fields you would suggest to specify for a typical production use.

ingressController:
  enabled: true

5. Want to apply CRDs separately. What would be the best place to find them?

Thank you!

https://redd.it/1c7qoin
@r_devops

How can we update docker containers on edge device with no Internet or should we simply get rid of docker?

We need to deploy docker container on edge devices which wont be having Internet. These devices occasionally connect to a network and one of the device (lets call it H) on the network will have internet access. So, I want to know how we can update docker containers in such scenario. I imagine following two approaches:

1. Create tar of image. Copy it to edge device (say over USB) and then update the image on the edge device.
2. Create local registry on device H. Pull the updated image from remote registry to local registry on device H. Make edge device pull only updated layers from this local registry on H.

I feel option 2 is good given it only moves updated layers between devices making update size small, while tar contains all layers resulting in tar of size 300 MB. So, option 2 was good option till we thought device H will be x86 Windows device. But now we are told that it can be Android or iPad companion device for edge device. We cannot run docker registry on Android or iPad right? So what solution we have remained with for updating docker container on edge device?

We did various docker related POCs. But now after knowing the fact that device H can be android or iPad device, we may have to get rid of docker completely and deploy apps say through other non container approaches say snap etc.

Should we let edge device access Internet through device H say through tethering? We don't want Internet on edge device for security reason, but then should we restrict the Internet access on edge to only servers hosting docker registry? Or there can be any better solution without requiring Internet on edge device at all?

https://redd.it/1c7sb25
@r_devops

Salary Expectation for 8 yoe in India

Hi Everyone,

May I ask what should be the expected CTC for a Devops Engineer in Bengaluru (India).

Thanks



https://redd.it/1c7s4e0
@r_devops