How do you decouple metrics generation from structured logging?

Where I work, I often see monitoring code that records the same fact twice: one for logging, and one for metrics, for example (Python-ish code):

base_logger = get_logger()
REQUESTS_RECEIVED = prometheus_client.Counter('requests_received', 'Number of requests received', ['method', 'status'])

def middleware(request, next_call):
logger = base_logger.bind(method=request.iss.onethod, endpoint=request.url)
logger.debug("starting to handle request")
response = next_call(request)
logger.debug("successfully handled request", status=response.status)
REQUESTS_RECEIVED.labels(request.iss.onethod, response.status).inc()

Having thought about it a little, it seems that it'd be better for the application to export only structured events. Some downstream process then receives these events, processes them, and then generates metrics based on defined rules. For example, one could define a rule for the above `REQUESTS_RECEIVED` metric, and another for a `REQUEST_TIME` metric based on how long it took for a request to be handled. In particular, those metrics could be derived from multiple events in aggregate.

Does something exist already to do this?

https://redd.it/12tuwyk
@r_devops

How to create a multi developer bitbucket workflow

Hi guys,

I have started at an agency who has multiple developers all over the world.

They have been all cloning sites from the repo to their local machines, making changes and committing. But you know there are a lot of problems with this. People forget to rebase to main. Developers are all using different versions of Apache and mysql not to mention php.

And then they commit to main and FTP to staging thinking “oh it worked in local”, something breaks and hopefully they notice before pushing to production.

So.

This is the process I am setting up:

1. Using source tree on local to commit to bitbucket.
2. Bitbucket pipeline runs automatically to staging.
3. Bitbucket pipeline is configured to manually deploy to production.

But the missing pieces of the puzzle are still different Apache versions, MySQL and php. So

Is there a software that a team of devs can use that will automatically set the versions of AMP so when they commit we are getting consistency from each developer?

Or is there a way to require the versions in the Git config file.

Thanks very much 😊

https://redd.it/12tvrcg
@r_devops

what is needed to be an SRE/Devops/platform in MAANG

Hey All,


I think this will be usefull for many

Any one who works as SRE/Devops/platform anything that they call these days in MAANG
what is the necessary skill needed to get inside these companies as SRE/Devops/platform engineer


I have a total of 6.5 years of experience in various cloud and devops tool including terraform , ansible, packer k8s docker CI/CD jenkins/ansible/team city . I mostly work in central team so i get requirement to automate CI/CD end to end so i use mix of shell /python /powershell and mix of tools


Now i want to move higher up in companies , I am a decent coder but have never done web dev something of that sort , mostly mine was scripting . I am good in fixing systems ie something breaks , i can dig deeper like dns /lb something got broke i will get the first call to fix since good at linux/networking and troubleshooting


I heard MAANG kind of companies just evaluate problem solving and never care about experience is it true , if so then how should i start and what all i need to revise please help


and how should i prepare now for these roles in these companies ?

https://redd.it/12twdrf
@r_devops

Bit of a "wow" this company is great to work for post, feel free to share stories yourselves

So i started at a big financial tech firm recently, Ive had my share of shitty jobs in the past where blame culture is massive, but I was shocked a few weeks ago by this new company.


Someone in another team fat fingered an update in a tool which took down everything in production, in such a away that the automated systems couldn't bring the apps back up successfully, so a bit of manual work had to be done, it was over a weekend so it luckily wasn't a lot of traffic hitting the app. But it was still a big outage.


In other places I've worked even though It was accidental with no malice intent, that guy would be fired immediately and the issues swept under a rug(fixed quietly) .


In this company they didn't name names, in a company wide email, they said it was a fault in an automated system and if anyone was to blame it would have been them as managers for not foreseeing this would be a problem (email sent by ceo, cto, coo, etc)

Only a few people in cloud ops with the correct access to this tool could see who did it.

The guy later said he was called into a meeting with all the ceo, coo, types, him thinking the worst.

They reassured him the blame was on them, thanked him for alerting them to the issue, and seeing how distraught he was offered the next few day off paid so he could return to work in a good frame of mind.

The guy is still at his job and is part of the team fixing the issue, still has all his admin accesses too.

https://redd.it/12txhbs
@r_devops

How DevOps and QA can be friends

Fluid 🌊
Flexible ✍🏻
And fun? 😁

Join us for an epic conversation with Doug Simons - the inventor of SenseTalk - to discover the hottest trends in #lowcode #softwaretesting and how they empower your dev and QA to work together more efficiently.


Register now: ow.ly/sBQN50NGXMx

https://redd.it/12tuvxy
@r_devops

Looking for the cheapest deployment option

Hey guys. I need to deploy a service into AWS and I'm divided between ECS Fargate and Lambda functions. It's a simple service which gets e-commerce orders and transfer to a database, parsing the information. It will probably be called between 10 to 50 times a day.
I already have the container ready to be deployed but I'm also thinking about lambda functions due to its low price... What do you guys think? The service itself takes just a few 'ms' to run.

https://redd.it/12u1xt6
@r_devops

Anybody adopted OpenTelemetry for all observability signals (logs, metrics, and traces)? If so, any thoughts?

Looks like there have been a lot of recent advancements with OTel, and I know traces were always had first-class support. I'm curious if anybody has also adopted OTel to handle their metrics and logs as well. If so, what's the good/bad/ugly?

https://redd.it/12u3g7z
@r_devops

How do you deploy secret files within a pipeline?

I currently have a bitbucket pipeline that deploys to docker hub a image and thats fine and all but now i need a way to also include a file (a .crt certificate) while building the image and deploying it to docker hub.

I am imagining holding said file in a bucket, something called PRODUCT-secrets, then copying all contents of the bucket to a folder within the image, but that would kinda suck to do.

The other alternative of course if to simply commit the file in the source code but then every developer would have access to that file, making it a security risk.

There is also the other alternative which is to have the files in the VM that would have the image be deployed onto and bind mount them into the container, but then when the VM gets destroyed the files would be lost, kinda bad as well.

​

Is there a better way to do this?

https://redd.it/12u5fpb
@r_devops

Broken websites: are they 2x as frustrating because we know how to fix them?

Specifically: 404 on https://careers.etsy.com/ca/

Or broken profile views when an ad blocker is installed: https://www.reddit.com/r/Etsy/comments/11512uh/favorites\_not\_loading\_properly/

Sometimes it seems like reddit or some other "social media" site is the only way to report these things.

Is there an alternative? Like a "report problem" link with capcha. Who does this well?

Or hell, just checking your links with a scanner or checking your logs for errors.

https://redd.it/12udo4x
@r_devops

How Useful has Distributed Tracing Been for You?

Do you think it’s worth investing in? What pain points do you have?

https://redd.it/12ufbqh
@r_devops

Five Rookie Mistakes with Kubernetes on AWS

We recently setup a new Kubernetes cluster on AWS and made some rookie mistakes. In a nutshell:

Not setting resource memory limits
Using EBS storage with more then one AZ
Using the default instance templates
Not using version control for your config

Here is the full story Five Rookie Mistakes with Kubernetes on AWS

Can you think of any others or any that you have stumbled over yourself?

https://redd.it/12ugpqp
@r_devops

6 months, 1 failed attempt and a lot of studying - got my RHCSA

Finally got this one over the line today, what a rollercoaster 🎢.

Would recommend to anyone though, feels like the biggest cert I've achieved to date. Onwards to the next (CKA + CKS babyyy!)

Cheers

Good night

https://redd.it/12ujta0
@r_devops

Mentors?

I was wondering if anyone can suggest any mentorship programs, either paid or free. I've got so little time nowadays that working on a single project for too long is difficult and I usually lose interest. I was hoping to find someone who can help me structure some kind of plan and aid in my career. Any recommendations are appreciated, even if they're not related to mentorship!

https://redd.it/12udkts
@r_devops

First DevOps Job is a dumpster fire

I started my first DevOps job a few months ago, and it has rapidly shown itself as a total shit show. The team is heavily demoralized, with almost half on the verge of quitting and/or being fired. We are underappreciated and overstressed, with no upper management buy-in for additional personnel or helpful tools.


The team is too busy dealing with day-to-day fires to build appropriate automation, additional tasks and responsibilities keep getting added on, etc. I was specifically hired to improve automation, but spend so much time dealing with daily work that I am unable to. The job is also 100% onsite for anyone who isn’t management or related to the CEO, and the overall compensation is poor.


While I am learning a lot about DevOps tooling, ways of thinking, etc, I am also definitely getting fed up with the overall culture and environment here.

My question is, how long should I stay here before moving on to a better position/company? Is there anything I can do in the meantime to make the future transition easier or to improve things where I am?

https://redd.it/12umxp0
@r_devops

Intermittent DNS issues in EKS

So I've got a simple pytest that deploys a curlimages/curl pod, execs into it and sequentially curls several AWS endpoints (like 'https://s3.us-west-2.amazonaws.com' etcetc).

It mostly works. However, sometimes it doesn't, and that's fun. Sometimes curl returns 'Could not resolve host' - for different endpoints, on different nodes etc.

I've tried many things and none have helped. Some coredns tweaks (using EKS coredns addon, EKS 1.25 if that helps), using an ubuntu image instead of curl based on alpine (read some scary stuff), looking into coredns logs (failed curls don't show up there so I guess coredns is off the hook?), adding some sleeps between execs, using FQDNs ('https://s3.us-west-2.amazonaws.com.', adding more coredns pods (up to 10 on a 10 node cluster)...

More than that, I can't reproduce this manually (i.e. running multiple infinite loops of kubectl exec -ti curlpod -- curl -vvv url doesn't produce errors **at all**). That's the strangest thing - both pytest and manual kubectl execs run locally from my machine, and pytest fails fairly consistently, while manual repro doesn't, at all.

Y'all, I hate computers sometimes.

https://redd.it/12unh2a
@r_devops

Hey guys, just landed a gig as a DevOps release engineer! I'm super stoked but also pretty nervous. Any seasoned vets out there have any tips or advice for a newbie like me? Would love to hear your thoughts!

Hey everyone, I just got a sweet job offer at a company that makes iOS and Android apps. Despite not having many options in these tough times, I accepted the offer and I'm excited to get started.

The company's infrastructure is all on-premise, with Jenkins as the main tool and some Python scripts thrown in to maintain the apps. My main task will be writing pipelines to build the apps with "xcode," but I don't have any experience with mobile app development.

Do you guys have any recommendations for materials I can use to get up to speed before I start next year? Any fellow release engineers out there with advice for a newbie like me? Thanks in advance!

https://redd.it/12ubjcr
@r_devops

I’m looking for my first engineering role and I feel like I’m being marketed for roles I’m not qualified for

Quick background about my skill set
* Work at a large company in a non-technical IT team
* I have all AWS associate certs but I’m not a great coder (basic-intermediate Python and HTML/CSS). I’ve done a few small projects (ex. Cloud Resume challenge)
* I use Linux often and am relatively comfortable with it and some very basic bash

Quick overview of my company IT teams
* We have teams of Tier 1-2 Application Engineers across IT that own 1-2 web apps. They basically just code and use CICD to push small stuff through with a handful of AWS services. Nothing flashy. This is what I envisioned my first step being.
* We also have mixed teams (Technical Project Managers, Scrum Masters, and often just one tier 2-3 Cloud Engineer). These engineering roles are closer to DevOps with Docker, Ansible, Kubernetes and seem quite advanced. This is what is being marketed to me.

My issue - I’m being pushed towards the latter (mixed team) based on my manager’s networking and opportunities arising in my department/subdivision. He seems to think I can do it. The hiring managers seem to think so too, but I really don’t. I envisioned my first engineering role to be on a team of others I can learn from. Instead, it seems like everyone wants me in a team where I’m alone to build things by myself. This is really daunting to me as someone who can barely code. Yeah I know a lot about AWS, but I’ve never even deployed to prod before. I’m going through courses now for front end to complement my Python but it’s going to take me months to be even be intermediate. I can’t help but feel like all of these people are overestimating me. My manager talks like a Tier 1 Application Engineer is beneath me (which sounds crazy to me).

Are they overestimating my ability like I think they are? Or am I underestimating myself?

https://redd.it/12uaax5
@r_devops

How can I pass a terraform variable in to a custom _data script block in a terraform file

Please accept my apologises if I explain incorrectly I’m a self taught DevOps engineer coming from a traditional windows infrastructure engineer background got here with some guidance’s from this page. I have a script that installs a application on a azure Linux vm silently with a configuration using terraform ..I’m using custom_ data this works great but it can only silently install the package with me giving it the password for service account , instance name, etc. This is all sensitive information that I’d rather the bash script gets from variables in the variable.tf file. Is there any way to pass through these variable in bash script. For example the password for the service account in bash script i’d like to use my variable.tf file to pass on to bash script which would be var.password. Hitting a dead end try to add terraform variable from variable.tf to my script.sh file using custom_data ..Happy to hear any other elegant way to achieve adding configuration like this .. I have terraform ,azure,GitHub actions to achieve this task unfortunately I don’t have ansible or cloud-init to work with. Any help would be appreciated

https://redd.it/12uvqeu
@r_devops

Advice after failure

Failed my apprenticeship in the uk

I was a devops apprentice and I worked in cloud.

I tried extremely hard to pass. I worked really hard on my project and created a good presentation.

The feedback says I didn’t show that I met the knowledge of monitoring, CICD, and unit testing.

In the presentation I talked about installing monitoring tools (azure log analytics with terraform), I talked about configuring notifications argocd notifications and I debugged them with logs showing the credential had expired. I showed the CICD process and I talked about unit testing with kustomize and validation scripts.

It seems like the assessor was extremely harsh or got confused during my presentation because it wasn’t clear enough. In the report they don’t mention any of the above they just hold onto the fact that some of the stuff was broken / being tested by other people at the time of the demo even though I showed previous runs from when it was working.

They’re telling me to resit it I have to pick a brand new project and spend 3 months preparing.

I already was offered a full time position which I started a month or two ago.

I guess I’m just looking for advice.

1) Do you think this is going to affect my employment in any way?

2) Should I try to keep it a secret at work?

3) Should I try to retake it with a new project and spend another 3 months on it or focus on new things?

The qualification was going to be a Level 4 apprenticeship.

Thanks.

https://redd.it/12uw3wp
@r_devops

Building from source or not?

Wrote up a counter argument to "Build Once, Deploy Anywhere" artifact promotion here:
https://dabase.com/blog/2023/build-from-source/

Perhaps I'm being naive about:
A) Build + test can be fast
B) Builds can be reproducible?

https://redd.it/12uy6q4
@r_devops

Terraform giving 403 AuthorizationFailure after accidentally deleting the private endpoint to a storage account

I added a wrong configuration (multiple subresource names for an endpoint) and applied instead of planned. My old private endpoint got deleted and now I get this 403 error whenever I try to reapply with the good configuration.

I am applying it from a github workflow (that I did not create because i am a beginner). Can anyone give me a suggestion? I also tried creating the endpoint manually, but for some reasons the organization rules i am under did not allow me to add manually a private dns zone. So now i get Failure sending request, status code = 0, context deadline exceeded. Help please?

https://redd.it/12u3kdh
@r_devops