Automated post-deployment monitoring vendors

Hi all, I'm looking for some help sourcing some vendors general strategies for doing post-deployment monitoring.

**Tl;dr: I'm looking for a system that can handle pre-deployment checks, the actual deployment, and post-deployment monitoring with automated rollbacks.**

I recently joined a start-up with a manual deployment process. The start-up currently does scheduled releases on 2 days during the week, and we deploy the entire stack of a handful of microservices at the same time. Aside from the obvious problem to solve which is to move to a place where we can deploy individual services independently, I am looking for a platform where I can do pre-deployment checks (check list of things to verify it is safe to deploy), the deployment itself, and then post-deployment monitoring with auto-rollbacks.

To elaborate a bit on the pre and post-deployment phases: I want the ability check if there is an active OpsGenie incident declared, if it is a weekend deployment, etc. before we do an automated deployment. In post-deployment monitoring, I want the ability to configure an alarm that, when triggered, initiates a rollback automatically. This alarm would be monitored for a specified amount of time post-deployment (measured in hours or by some other trigger, like meeting a condition such as "1000 requests processed").

What I've looked at so far:

* **Argo Workflows**: promising amount of integrations, but very general purpose. My impression is that it can only deploy to K8s targets... am I wrong? We deploy more than just K8s workloads. I suppose I could write custom tasks to deploy to non-K8s... but Octopus can do this natively.
* **Octopus**: Can handle almost all of the pre-deployment requirements I have, can deploy to virtually any target (including K8s), but is unfortunately lacking any sort of post-deployment monitoring (verified after talking to a solutions architect at Octopus).
* **Codefresh**: Just bought by Octopus. Seems like it's also K8s specific.

I'd really appreciate any leads on systems out there! I come from one of the FAANGs where we had all of this (and more) but it was all internal tooling - does anything exist in the market?

https://redd.it/1bt08qs
@r_devops

Project advice for SE wanting to switch to DevOps

Hello people,

I am currently working as a software engineer, mainly using Rails. I used Java and React in my previous jobs.

Lately for the past 3 months, I got interested in the cloud and started studying for the AWS SAA certificate, and also trying to get better with Linux and Terraform on the side. I have some small project ideas to get your opinions on it:

- Setting up a React project I did earlier on AWS cloud with EC2, ASG, Load Balancer, and Route 53(with my own domain).

- Setting up a simple frontend plus backend project with basic CRUD stuff, that also uses RDS and CloudFront on top of the other things I used for the previous project. I want to write some basic tests for the backend so that I can set up a CI/CD with them.

I think of doing these projects manually via the Management console first, then learn how to automate it with Terraform.

I also want to practice using VPC, but don't know how I could use it in these projects. I also wonder how I could utilize Ansible in projects like this, would I even need it?

Do you think these projects are good to implement for getting into DevOps, or are they way too simple?

I am open to any positive or negative feedback. Feel free to roast me if I said something clueless.

Thanks in advance.


https://redd.it/1bt1mj7
@r_devops

In which order would you learn these?

Terraform, Docker, Kubernetes, Ansible, CI/CD, Prometheus/Grafana

 

I recently passed AWS Solutions Architect Associate and have Python development experience, RHCSA Linux, and Network+ under my belt.

I'm thinking of learning Terraform for a cloud-agnostic IaC (I already know cloudformation), followed by jumping into Docker containers and Kubernetes since that seems to be extremely in-demand within the industry. After that I might look into CI/CD with Gitlab and observability monitoring with Grafana. Thoughts?


Also, is Ansible stilll in demand in this day and age? The RHCE exam - which is the sequel to the RHCSA - is basically an Ansible exam. Knowing I've recently gotten the RHCSA, would it be a good idea to do the RHCE next, or should I skip that and focus more on Docker and Kubernetes?

https://redd.it/1bt3om5
@r_devops

The Latest Innovation in Incident Response - Most Privilege Access

Least privilege access is important but has shown its limitations during incidents.

As a result entitle.io developed a new feature that instantly empowers EVERYONE in the organization with full admin rights to combat cyber threats collectively.

Here’s how it works: https://www.entitle.io/lp/most-privilege-access

https://redd.it/1bt3iig
@r_devops

Inputs into API calls

We have a use case where our CICD pipelines will “deploy” to API end points. We’re unsure how we should store the API inputs in our Gitlab CICD repos. I figure we’ll need to house the API version number along with with PUT command’s payload.

Has anyone here worked with something similar? My go to would be a JSON input that we would then parse into the request but curious what others have seen/used in similar situations.

https://redd.it/1bt74gp
@r_devops

Teams being separated from O365

I see that it is a constant gripe in this sub about Slack and Teams. Well seems like the tyrants have made a decision to split Teams from the cost of m365. Its to only affect new customers.

Probably some very upset people out in the world right now reading this.

Realigning global licensing for Microsoft 365

https://redd.it/1btc167
@r_devops

Job searching with an inflated job title!

Hello, I’m in a bit of unique position in my career right now and I’m wondering if anyone can relate/ give any advice.

Graduated college with a CS degree in 2020 and got hired straight away at a big company into a cloud engineering role (with zero experience in the cloud). Immediately I come to realize I’m out of my depth as the role is really an Observability DevOps one and I am doing none of the programming I had been taught in school.

Our team uses a wide variety of tools and technologies, such as Azure, GCP, Kubernetes, Docker, Terraform, Jenkins, Helm, ArgoCD, Grafana, Opentelemetry, etc…I learned a little bit of everything, enough to keep afloat and be a useful enough member to the team.

Then, about a year ago, all the senior members of my team decided to move on to new opportunities. My management was unable/unwilling to find replacements, and instead decided to promote me and my coworker to Senior Engineers.

So, now here I am, a senior DevOps engineer with 4 total years of experience. I’m very aware of how inflated my title is. I am familiar with many DevOps concepts but I lack a foundation in Dev or Ops.

Now, I’m looking for a job where I would prefer to be a Junior engineer and even take a paycut just to get some mentorship and experience with fundamentals.

What I’m really struggling with is how to convey this to recruiters, and how to find roles that strike a good balance between letting me get back to the basics, while still letting me market the skills I do have.

Has anyone experienced something similar? 😭




https://redd.it/1btgj5z
@r_devops

Is there too much saturation for a senior admin to move into DevOps?

Just curious about how saturated the DevOps market is right now? I’m a senior systems engineer with 11 years of experience, but most of that is on-prem and not with IaaS. Jobs in my area are all on-prem except for SaaS apps. I’ve been automating with PowerShell for years and have experience with Terraforms VMware provider and using Ansible. No container experience outside of Docker. Any chance I could break into a DevOps or SRE position, or are the cards stacked against me now?

https://redd.it/1btkzgl
@r_devops

Career Advice of Scrum/PM and DevOPs

I wanted to know the input of some people as I see careers (PM/Scruum) which asks to know DevOPs, so I got the following inquiries:

Should I as a Scrum/PM, learn DevOps?

Is DevOps something worth knowing to help a team of developers? (As PM/Scrum) If, yes is there any path/certification which can help with this?

Any advice can be helpful.



https://redd.it/1btm8eg
@r_devops

Business strategy, Devops, and dollars

I've been struggling for the past few years around the business justification behind devops that I'd be interested in hearing people's opinion about. Its fun and exciting to geek out about the engineering involved with the field, but I feel that it very rarely one of those mission critical aspects that actually matters to my employers and the people writing my paycheck.

Can I shift-left people, process, and technology on a CI/CD pipeline? Yeah, no problem. How about geo-distributed/zero-downtime deployments? You bet. Event-driven distributed systems with HA/SLAs/DR/Business Analytics/FinOps while still adhering to regulations, corporate policies, and ITIL change management while taking an iterative approach towards delivery to navigate uncertainty that can still provide some amount of value if the project/program gets cancelled half way through? Its a hassle and I hope all interested parties are ready for a challenging conversation, but sure. But to what end? How does this move the needle in a way that matters?

What I've struggled with boils down to a single question: in what ways is devops part of your business strategy? By that I mean, the arena your company competes in and how you intend to win over others in the space. If you envision the conversations between your management, executives, board members, or shareholders, how does devops come up? My experience is that quality, availability, support, and analytics is implied or assumed at best. More often than not, its considered part of the cost of doing business that they'd just as soon (and often do, sometimes even rightfully so) deprioritize when the going gets tough. I can't imagine devops showing up as a line item on an A3 or a future state operating model.

I feel like the total cost of ownership behind devops adoption is in the tens of millions of dollars range for most medium-sized businesses (say a 5-person "devops" team, development changes, cloud/tooling spend along with custom integrations, education/training, countless hours in meetings, the opportunity cost of time being taken away from feature delivery, etc). Its been a trend over the past decade that even though I've worked in this space for a long time as an engineer, advisor, and starting to border on organizational management that if asked to justify my paycheck for a particular employer or the overall value of the devops in general, I simply couldn't do it. DevOps techniques and capabilities aren't cheap -- a single, very talented, senior level engineer probably costs the company ~200-250k in total compensation. This is a space I work in and can really matter to some companies, sectors, and products but not the type of "industry best practice" that every business gets a measurable ROI from contrary to popular belief/salesmanship/conferences/etc.

Is this just me being out in left field or chasing my own tail around impossible expectations? I'd be interested to hear how others grapple this question.

https://redd.it/1btodmf
@r_devops

How do you run one-off scripts in Github Actions?

I have a Flask app that uses Flask-Migrate and SQLAlchemy however, one thing that is constantly bugging me is that whilst I have a Github Actions to build, test and deploy, handling data migrations is something that I can never do within the pipeline.

Two specific examples of my what I want to use a one-off scripts :

1. I can generate Alembic migrations but I am never sure where exactly to run a SQL script to backfill data? And ensure it only runs the one-time I do so.
2. Updating the db say I wanted to update the db to facilitate a new feature (no schema change) or cleaning up bad data. Again running once.

I have seen examples where people used their migration tool to also manage existing data, correct me if I am wrong, these tools are supposed to only handle schema changes and not the existing data?

How do you usually do it? Currently, I just manually inspect by connecting prod db through DataGrip but I know this isn't best practice.

​

https://redd.it/1btuphk
@r_devops

What do you guys like getting at trade shows?

Hi all,

I'm helping the GTM team at a trade show. Being an engineer myself, I hate being sold to, but you know, business... Anyhow, they've asked what would be some good things to bring to the booth to start conversations. I've been to some of these things and know that most booths will bring t-shirts, stickers, pens, cables, etc.

Is there anything useful you've gotten at a trade show? Anything a bit more unique or something that stands out?

Any help is greatly appreciated! Thanks in advance.

https://redd.it/1btyh9r
@r_devops

Introducing Kusion v0.11.0: might be helpful for boosting collaboration between application developers and platform engineers

Declaration: I'm one of the members of the KusionStack team

Hi guys, I'm happy and humble to introduce the latest version of Kusion, an opensource platform orchestration tool that might be a booster for collaboration between application developers and platform engineers. Platform engineers can design, build and publish a reusable building block for application resources and config data, which can be easily adopted by application developers. More info can be found here.

https://redd.it/1btz48i
@r_devops

Do you enjoy Gitlab CI?

I am really annoyed with it. I didn't use much else, so hard to compare. But what really puts me off is the way it handles variables - one has to work around the fact that it doesn't expand them when passing them to downstream - typical thing: define some workflow, set up some stuff like image names by using stuff like commit or tag. Pass such name to downstream multi-project pipeline and guess what. Nothing works because it gets passed as string to downstream and it is expanded in downstream project, with its variables. Or am i missing a clever trick?

Other annoyance is that i have to either build super image containing everything and do everything in one job (build and package mostly) or pass the stuff around as artifacts, which is painfully slow.

Sorry, if i am annoying you,, i really wanted to vent my frustration. By the way i am grateful that GitLab is open source and free and overall pretty OK. I'd love to hear your experience!

https://redd.it/1bu1hlw
@r_devops

Resume Feedback Please

I'm currently looking for either a Cloud Engineering or DevOps Engineering position, I've not been getting much of a response. A few lukewarm responses but no actual interviews in a while.

I have a strong background in System Administration working at a Managed Service Provider, with one of them requiring some Cloud administration. I switched over to Cloud Engineering and ended up being the 'devops' guy in the projects that I was working on because of my sysadmin background. I'm looking to switch to another devops role.

Would one of you kind folks scan my resume please? I'm going to write it from scratch. But I'd love some feedback, maybe there is something that I'm missing.

​

Full Resume

Imgur01

Imgur02

https://redd.it/1bu31dt
@r_devops

How does your team approach problems you don't have much experience with ?

So, recently landed in a team that seems to be responsible for everything from start to an end while consisting of few ppl.

Managing DBs, profiling queries ? Check

Defining standards for security ? Check

Managing two clouds in parallel ? Check

Managing On-Call, Monitoring, Logs, ES ? Check

Implementing Zero-Trust ? Check

Implementing Developer Platform ? Check

Maintaining Terraspace, Kafka, Rabbit, Kubernetes, Clickhouse, Vault, Okta, Backstage, Jenkins, GitLab and few more.

A lot of moving parts. All of us have pretty solid understanding of the bigger picture, but when it comes to details - when stuff is actually implemented, a lot of those details seem to fall through and are not taken into consideration during plannings/brainstorming.

This is mostly due to inexperience with quite a few of those managed or not services (team rotated over the years).

I'm curious how other teams are managing complex setups to make railroad tracks meet in place instead of having to adjust everything at the end coz stuff was not discussed and aligned correctly beforehand?!

I know it's sometimes impossible to think about everything, but would like to decrease as much as possible those turnouts at the later stages of development.

What's your go to strategy ?

https://redd.it/1bu2p62
@r_devops

Devops US citizen sponsored abroad?

I'm a senior engineer and a US citizen. I'd like to move long term to another country in EU or Asia. I'm aware of the pay difference and the "digital nomad visas". I'm asking specifically if any Devops / SRE US citizens have been sponsored by a country abroad?

https://redd.it/1bu49ns
@r_devops

Az Solution Architect exam, Memorization is sick

So I'm studying to take the Az Solution Architect expert exam, and I'm tired having to Memorize SKU features of each service and it seems that it's a requirement for the exam.
Really guys if IT is about memorization then what's the difference between us and medicine field, this should be included in resources alongside the exam "same as k8s resources in CKA exam"; because in actual world we won't be remembering these as lawyers remember articles to perform in court. On the contrary we will just open Microsoft docs and check about SKUs features and that's it!

https://redd.it/1bu7cd9
@r_devops

Cloud Engineer in Singapore

Hello Everyone!

I am curious as to how someone from the States can get a job in Singapore as a Cloud Engineer/DevOps Engineer. I am also curious as to what the salaries are in Singapore, and overall how the market is over there.

I am a Cloud/DevOps Engineer with five years of experience looking to move to Singapore, thus wanted to learn about some of the experiences y'all have had.

Thanks!

https://redd.it/1bu8os3
@r_devops

How to focus GitLeaks on a smaller subset

We have a few mega-repos with several if not dozens of solutions. I wanted to see if I could make a GitLeaks YAML task that would focus in on a particular path. The documentation seems to say that we could do it like
scanlocation: '$(Build.SourcesDirectory)/path/to/your/subfolder'


But in practice it doesn't seem to be finding it. Has anyone done here had some experience with this?

https://redd.it/1bua53g
@r_devops

Terraform workspace: how to do it?

Hello,

I am experimenting with workspace for handling three environments (dev/test, staging and prod)...

I got it work under the assumption that the three environments are similar in terms of type and quantities of services... but this is not the case in real life...

let me explain:

my terraform (for now/testing) is pretty simpler:

* **in dev/test**: it creates an S3 Bucket
* **in staging:** it creates one S3 Bucket + 2 VMs
* **in prod:** it creates - one S3 Bucket + 4 VMs

the idea is to have a terraform that allows to create 1 S3 + 4 VMs, but in each environment it should just create a subset...

how do I get this done? is there a way if .tfvars does not have a variable, it skip the instruction?

any suggestions?



https://redd.it/1bu9uvm
@r_devops