What are the best tracing tools for microservices?

This article covers some of the best options - curious to know if anything is missing from this list.

https://redd.it/13e404b
@r_devops

AWS WAF and SQLiMatchStatement

I'm working on some WAF rules that should be blocking SQL injection attempts, but for some reason they don't work with form input data (application/x-www-form-urlencoded). We're using the match type "Contains SQL injection attacks" vs writing our own.

The website uses a basic html form to post data to an endpoint behind the WAF. The form takes basic data (first name, last name, address, etc) with ~8 fields total.

If I input SQL injection in any of the first 3 fields (ex: SELECT * FROM INFORMATION_SCHEMA;--) the WAF correctly returns a 404.

If I enter the injection string in any of the other fields, the WAF ignores it and returns a 200 status.

Ex:

200: curl 'https://local.test' --data-raw 'A=1;B=2;C=SELECT * FROM INFORMATION_SCHEMA;--'

404: curl 'https://local.test' --data-raw 'A=1;B=2;C=3;D=SELECT * FROM INFORMATION_SCHEMA;--'

I can't find any documentation on how the match type works beyond a very basic overview. I suspect this is to prevent people from trying to circumvent it. But it makes no sense to me why only the first 3 fields of the form input are checked.

Has anyone encountered this before? Is there some trick to checking form data for SQLi, or at least better documentation on what is being blocked?

https://redd.it/13dxx6p
@r_devops

Should I continue in DevOps?

So, I'm about to finish a two-year apprenticeship in a large company. I've been working as a DevOps there, and I'm wondering if I should continue in this field. For context, I'm over 30 years old, and I made a late career change. I only started working in the IT industry about three years ago, roughly, and it has always been through apprenticeships. I have primarily a networking and systems-oriented background.

But in my current job, even after almost two years, I still need help with certain topics. The infrastructure is very complex, and I still make mistakes or don't know where to turn to (we're talking about a company with over 1000 employees). Even after one year of apprenticeship, some of my colleagues (not all of them) complained about my lack of autonomy.

However, I do have autonomy, I understand the tools being used, even though I occasionally make coding mistakes which are becoming rarer. Where I have questions is primarily about how a specific service works, and since I don't have much knowledge in development, I also have some inquiries at that level.

In comparison, another employee who was an apprentice before me for six months and has been hired since is doing much better. However, he has a development background and many years of experience in that field. But he is constantly used as an example compared to me, to tell me that I'm not progressing fast enough.

I enjoy DevOps, and I really like creating infrastructure as code, but I'm wondering if I should continue in this path considering the feedback I've been receiving. To be honest, I mostly feel that some of my colleagues are being unfair, and that it's normal for an apprentice to have questions and need guidance on certain topics, even after almost 2 years. Some other colleagues have confirmed this viewpoint.

But perhaps after 2 years, all apprentices in the world already have complete autonomy. If that's the case, then maybe this sector is not where I belong. What do you think?


( I apologize if there are any language errors, English is not my native language.)

https://redd.it/13dig37
@r_devops

Branching Strategy for multiple APIGW endpoints split in different resources

Hi Everyone, I'm looking for a good way of Branching strategy in order to maintain the liason between APIGW split in different resources containing several endpoints. We are using CodePipeline to orchestatte the CodeCommit repos, CodBuild and CodeFormation. The solution involves APiGW, Lambdas pointing to a specific ECR images. I'd appreciate any recommendations for the Branching strategy. Thanks. Kind regards

https://redd.it/13eaz54
@r_devops

Suggestions for improvements to the environment

Hey people!

I currently work in a government agency and am solely responsible for the DevSecOps area in an environment that is migrating from monolithic applications to microservices.

We have two Kubernetes clusters, one for production and one for development. Both are single node.

I'm wanting to migrate each of them to at least have a control plane and two workers.

Allied to this, I also have to define the monitoring tools. I thought about using ELK stack.

Also wanted to use a management tool. We have RHEL licenses but not Openshift, so I'm considering using Rancher.

​

I would like your opinion on cluster architecture and management tools.

https://redd.it/13d1eg6
@r_devops

What DevOps tool you wish that existed? I'll create it for free!

I'm not kidding. I have enough DevOps and programming experience, in addition to time and will.

I have recently tried to create a cross-platform cloud management tool accepting any feature requests but it didn't gain any traction even after open-sourcing it and making it as easy as possible to build and understand.

So I'm thinking now: maybe that is not the tool that everybody needs! Maybe they need something like a Kafka manager (with features that current ones don't have), a real-time messages editor/monitor, a tracing tool, a deployment tool, etc...

And you might ask: why am I doing that? Well, I'm a senior dev and I just don't like the way my career is growing. It is growing at the "normal" pace, but I'm trying to find a way into entrepreneurship. So my plan for this app to be free + open-source for the first two years and then I'll start adding enterprise features for paid customers complete with source-code, while still maintaining the free version (I promise!). So let the ideas come!

https://redd.it/13ed4ie
@r_devops

Has anyone switched to a product team from devops? What were your reasons and how are you liking it?

I'm curious if any others have come from building products as a SWE to DevOps, and then gone back. What did you like about each role? What didn't you like?

​

For me it feels like product work is much more flexible with the implementation details, and creating/scaling infra or setting up monitoring and alerts is very rigid, there's not much room for creativity, however, there are moments of pure automation joy when everything works and needs little maintenance.

https://redd.it/13eesqi
@r_devops

How do you deploy multiple preview environments across multiple repos and branches?

Hi. Currently working on creating a better way of managing our preview environments.

Our current setup is having 10 preview environments which can be set to a branch on demand, using a web-ui (Forge). However, as this is kinda unscalable and not dev-friendly, I am working on a way to automate this using Github branches and automatic deployment to AWS.

So, when someone creates a PR on frontend, I will deploy everything needed to AWS (like an S3 bucket). However, I also want it to "connect" to its own, isolated, API (say, an EC2 instance). This seems simple, but we ALSO want to be able to select on which branch that API is checked out, as our devs' workflow is often that they work on api and frontend simultaneously, so we can't just make their frontend PR-deployment connect to api-master.

Does this make sense?

I basically have every building block already done, except for this part - the "workflow" part as we call it.

Would love some insight into this! If you need more info on specifics in order to give a meaningful answer, please let me know!

https://redd.it/13efp40
@r_devops

I had to do mandatory military service which wasted a year of my life (had no access to a smartphone or internet). Have a career fair coming up but I don't even know what kind of jobs to apply for.

######My resume

----
After I graduated from university, I did 13 months of military service, training police officials with my IT skills. No phone or internet access and only a week of vacation monthly. Tried to study but limited access made it unfeasible, so I postponed it till I was done with my service.

Now, I'm currently following the "devops roadmap" that recommends foundational knowledge in Python, Linux, Networking, AWS, and IaaC. I have a solid grasp in python and linux (studied for the RHCSA), and I'm now prepping for the AWS SAA cert while grinding leetcode.


Questions:

1. What kind of jobs do I even apply for? I don't think I'm ready for a developer position. And yet, I don't have the necessary experience for a system administration one either.


2. Should I continue learning the rest of the roadmap?

https://redd.it/13egfeu
@r_devops

DevOps to ‘DevOps’

Yo Dev-Ops cats.


I hope some of you can hear me out and maybe relate to me a little. And if possible offer some advice, or encouragement.

When I got into the industry I started as a software developer. I was doing C#, a little javascript, and a little Java. I was new and young and it was awesome. I could write run-on sentences for days describing how happy it made me. It’s like a toddler describing Christmas to an alien.

Eventually I started learning the Ops side of the project I was on. That was exciting. It was new technology, it was meaningful to my project. I could see a direct impact of what the Ops had on the Dev work. And why they would go hand in hand, praise the DevOps!

Here’s where the storm clouds roll in. Once I learned these Ops skills, like CI/CD, Docker, Docker-Compose, other sys admin things, I was sort of pulled. The company said “this is great! We have a guy who understands this new (ish) technology and can reliably use it for a project!” So they set me loose. I was now doing Ops for every project it felt like. I thought wow this is interesting. But boy did I burn out fast. I didn’t realize that I lost all of my passion and drive because I was missing real DevOps. Not ‘devops’.

I was missing having a meaningful relationship with the project that I was working on. I missed doing Ops for something I was working on. Not Ops for someone else’s problem. This model of taking real budding DevOps programmers and making them ‘devops’ or really just ops guys can’t be something only I went through.

I just don’t know what to do. I feel like I had so much momentum as a developer(and originally more of true DevOps programmer). But I feel like a baby who had his lollipop taken away. Personally I’m tired of feeling bad for myself. I’m going to find a way to break back into a real DevOps role. But I’d appreciate any thoughts, advice, relating to’s, or even ‘this is the wrong place for this bud’.

As for the title I describe this as being a DevOps guy put into a ‘devops’ role looking to return to the DevOps guy. Like Star Wars:

Star Wars 4: A New DevOps

Star Wars 5: Empire ‘devops’ <- I’m here

Star Wars 6: Return of the Dev-Ops



Help a DevOps guy out.

P.S. on mobile, sorry for format issues. Plz comment and I can fix in the morning. Been up all night with an existential crisis (this is it this is the existential crisis).

Edit 1-3: formatting

https://redd.it/13ei26p
@r_devops

Digest #102: Are Terraform’s days numbered? - Same argument that language X may die b/c of new language Y

Its not apples to apples. But these headlines are stupid for a devops bulletin, misleading to some that may not have a grasp on what it means for them, if they are just getting used to TF.

Just b/c Prime goes Monolith, after rapid development and exhausting, enough, mitigation of performance to realize, that its time.

TF doesn't only deploy/use micro services, lambda, step functions and containers.

</rant> I guess. You see this so often on subs like r/AskProgramming

https://redd.it/13d71dy
@r_devops

For a career in DevOps what certification should I acquire?

Hello,

I am new to the field of DevOps. I have AWS Cloud Practitioner certificate and am working on some hands on project with help of ACloudGuru training platform. No professional experience though.

At associate level, what certificate should I apply for Associate Developer Or Associate SysOps Administrator or Associate Solution Architect? Which is more in demand? My company may sponsor the certification exam if I can get it before July. Hence, the hurry 😅

Any other suggestions for a beginner in Aws and DevOps are welcome.

Thanks in advance.

edit: help me choose the best associate level certificate

https://redd.it/13elq9u
@r_devops

Automate creation of Project Agent Pools (VMSS)

Hi all.


Has anyone managed to link up a scale set to a project agent pool via command line?

I have a pipeline that creates the gold image, adds that to a VMSS via Terraform but i would like to finish the workflow by having this scale set added to a scale set pool so they can be used within Devops on other projects. I have tried to google but come up short.

Thanks.

Below is a screenshot of what i am wanting to automate.

https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/media/scale-set-agents/create-agent-pool.png?view=azure-devops

https://redd.it/13en9sp
@r_devops

FYI: GitHub appears to be down at the moment (https://www.githubstatus.com/)

I'm getting HTTP 500s when viewing any repository on the site.

https://redd.it/13enp35
@r_devops

3 straight days of degraded github performance

That's it, I'm bring back the Travis deploy scripts. Suddenly Travis' 90% uptime seems amazing.

https://redd.it/13enths
@r_devops

Leaving job after 6 months

This place is a total mess. 0 documentation, 0 onboarding, 0 work life balance. They completely lied to me during the interview process and their job description was entirely misleading. I've stuck around so far hoping that it's going to get better, but it's only gotten worse. There is 0 work life balance so I'm looking at new jobs.

What do I put on my resume for this place? I've managed to learn a few of their tools here and onboarded myself since their process was entirely non existent but not much more than that...

https://redd.it/13eph8w
@r_devops

Friend & I built a production debugging & monitoring alternative to Datadog, New Relic (based on OpenTelemetry + Clickhouse)

While Github is down, figured it might be some good down time to share what my friend & I have been working on...

We've been building a Datadog alternative to have one place to monitor and debug production apps, in an actually affordable way (Currently 9x cheaper compared to DD).

We’ve previously ran the numbers looking at Datadog for some of our services and realized our Datadog bill would rival our AWS EC2 bills! (and I know we aren’t the only ones with that problem). Yet we also knew it was hard to get the end-to-end visibility we often needed to debug complex race conditions and data-driven edge cases from other vendors.

So we’ve decided to spend time crafting the production debugging product we needed internally, and share it as a viable alternative for others as well.

It’s built on top of OpenTelemetry, Clickhouse and S3. This ensures we’re able to scale indefinitely, with minimal cost, and still have tons of flexibility to build a complex product on top of it all. With it, we’re able to easily tie together charts, logs, traces, and session replays, all in one place.

If this is interesting to y’all - would love to hear what everyone thinks:

https://hyperdx.io/

https://redd.it/13espyq
@r_devops

Needing help with the non-technical side of pipelines.

Please forgive any non devops terms, I’m not a SWE.

I’ve been desperately searching for information on how the CI CD process works from a people/processes perspective. What is the process when code/application fails a test or check?

What I really need to know is what happens when code fails a test? Who is notified? How are they notified? Who is suppose to fix it? How are the failures and fixes typically documented? Is there a person to make final approval before deployment? I realize this will highly vary for different environments and products but is there a standard guideline for this?

I’ve stacks upon stacks information that tells me what x product checks for and how to configure for checks. I’ve got 100 different diagrams that show me how code is created then tested then deployed. But almost none of it goes through the process of code failure and fixes.

I am hoping to find something like this:

1. Developer creates/ updates code
2. Code is ran through tests/checks
3. If Code fails test
A. Developer/Manager/Princess Peach is noticed of failure
B. Failed code is sent back to (original developer? backlog software? Ticketing program? Manual spreadsheet tracking?)
C. Code is fixed by
D. Return to step 2
4. If code passes all tests
A. Code is sent to passed code repository
B. is notified of code awaiting final approval
C. Code is approved and sent for deployment
5. Code is deployed to production environment.

It is possible I’m looking at the whole process wrong and I am open to any feedback on that too. Maybe there is no process? Maybe if the code passes the tests it is just dropped into the production environment automagically and there is zero human intervention?


Thanks for the help.

https://redd.it/13eqkut
@r_devops

Anyone using Terraform in a single production environment with multiple devs that need to test changes?

We're leveraging a single production environment using CI/CD, feature flags, and other best practices. The biggest challenge we have is testing changes via Terraform, whether it's application or infrastructure. The main problem is that dev A is working on something and deploys changes directly to prod (from local) for testing, but dev B also needs to deploy, undoing dev A's changes.

Right now we're just closely collaborating to not step on each other and I can't imagine there's much of a better way.

https://redd.it/13evbwf
@r_devops

What's your opinion on self-hosted Github Enterprise?

Is there feature parity between GE and Github? Is it reliable?
Does it use some of the global Github services and does it go down when Github goes down? (seen that in some of the SaaS offerings)

https://redd.it/13eyvp2
@r_devops

5 hours interview for Principal DevOps Engineer?

I already cleared HM , and HR phone screen.

What is this 5 to 6 hours over different days with different folks in cross functional roles about?

Any guidance? I have done 1 hour ones before usually 3 to 4 panel Including the Hiring Manager.

How to prepare? It's virtual but still kinda daunting and making me nervous.

https://redd.it/13ey6mn
@r_devops