Automate creation of Project Agent Pools (VMSS)

Hi all.


Has anyone managed to link up a scale set to a project agent pool via command line?

I have a pipeline that creates the gold image, adds that to a VMSS via Terraform but i would like to finish the workflow by having this scale set added to a scale set pool so they can be used within Devops on other projects. I have tried to google but come up short.

Thanks.

Below is a screenshot of what i am wanting to automate.

https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/media/scale-set-agents/create-agent-pool.png?view=azure-devops

https://redd.it/13en9sp
@r_devops

FYI: GitHub appears to be down at the moment (https://www.githubstatus.com/)

I'm getting HTTP 500s when viewing any repository on the site.

https://redd.it/13enp35
@r_devops

3 straight days of degraded github performance

That's it, I'm bring back the Travis deploy scripts. Suddenly Travis' 90% uptime seems amazing.

https://redd.it/13enths
@r_devops

Leaving job after 6 months

This place is a total mess. 0 documentation, 0 onboarding, 0 work life balance. They completely lied to me during the interview process and their job description was entirely misleading. I've stuck around so far hoping that it's going to get better, but it's only gotten worse. There is 0 work life balance so I'm looking at new jobs.

What do I put on my resume for this place? I've managed to learn a few of their tools here and onboarded myself since their process was entirely non existent but not much more than that...

https://redd.it/13eph8w
@r_devops

Friend & I built a production debugging & monitoring alternative to Datadog, New Relic (based on OpenTelemetry + Clickhouse)

While Github is down, figured it might be some good down time to share what my friend & I have been working on...

We've been building a Datadog alternative to have one place to monitor and debug production apps, in an actually affordable way (Currently 9x cheaper compared to DD).

We’ve previously ran the numbers looking at Datadog for some of our services and realized our Datadog bill would rival our AWS EC2 bills! (and I know we aren’t the only ones with that problem). Yet we also knew it was hard to get the end-to-end visibility we often needed to debug complex race conditions and data-driven edge cases from other vendors.

So we’ve decided to spend time crafting the production debugging product we needed internally, and share it as a viable alternative for others as well.

It’s built on top of OpenTelemetry, Clickhouse and S3. This ensures we’re able to scale indefinitely, with minimal cost, and still have tons of flexibility to build a complex product on top of it all. With it, we’re able to easily tie together charts, logs, traces, and session replays, all in one place.

If this is interesting to y’all - would love to hear what everyone thinks:

https://hyperdx.io/

https://redd.it/13espyq
@r_devops

Needing help with the non-technical side of pipelines.

Please forgive any non devops terms, I’m not a SWE.

I’ve been desperately searching for information on how the CI CD process works from a people/processes perspective. What is the process when code/application fails a test or check?

What I really need to know is what happens when code fails a test? Who is notified? How are they notified? Who is suppose to fix it? How are the failures and fixes typically documented? Is there a person to make final approval before deployment? I realize this will highly vary for different environments and products but is there a standard guideline for this?

I’ve stacks upon stacks information that tells me what x product checks for and how to configure for checks. I’ve got 100 different diagrams that show me how code is created then tested then deployed. But almost none of it goes through the process of code failure and fixes.

I am hoping to find something like this:

1. Developer creates/ updates code
2. Code is ran through tests/checks
3. If Code fails test
A. Developer/Manager/Princess Peach is noticed of failure
B. Failed code is sent back to (original developer? backlog software? Ticketing program? Manual spreadsheet tracking?)
C. Code is fixed by
D. Return to step 2
4. If code passes all tests
A. Code is sent to passed code repository
B. is notified of code awaiting final approval
C. Code is approved and sent for deployment
5. Code is deployed to production environment.

It is possible I’m looking at the whole process wrong and I am open to any feedback on that too. Maybe there is no process? Maybe if the code passes the tests it is just dropped into the production environment automagically and there is zero human intervention?


Thanks for the help.

https://redd.it/13eqkut
@r_devops

Anyone using Terraform in a single production environment with multiple devs that need to test changes?

We're leveraging a single production environment using CI/CD, feature flags, and other best practices. The biggest challenge we have is testing changes via Terraform, whether it's application or infrastructure. The main problem is that dev A is working on something and deploys changes directly to prod (from local) for testing, but dev B also needs to deploy, undoing dev A's changes.

Right now we're just closely collaborating to not step on each other and I can't imagine there's much of a better way.

https://redd.it/13evbwf
@r_devops

What's your opinion on self-hosted Github Enterprise?

Is there feature parity between GE and Github? Is it reliable?
Does it use some of the global Github services and does it go down when Github goes down? (seen that in some of the SaaS offerings)

https://redd.it/13eyvp2
@r_devops

5 hours interview for Principal DevOps Engineer?

I already cleared HM , and HR phone screen.

What is this 5 to 6 hours over different days with different folks in cross functional roles about?

Any guidance? I have done 1 hour ones before usually 3 to 4 panel Including the Hiring Manager.

How to prepare? It's virtual but still kinda daunting and making me nervous.

https://redd.it/13ey6mn
@r_devops

Hey fellow devs! Just curious if any of you freelancers out there are charging for value instead of time when it comes to devops. I’m trying to explore my options and would love to hear your experiences and thoughts on this approach. Thanks in advance for any help!

Hey guys, I run a software dev agency and we specialize in developing solutions and managing client servers. As we've grown, we've noticed our clients requesting more complex systems. Right now, we charge by time but I'm considering switching to a devops model where we charge by value. For instance, setting up a server is a breeze for me with the scripts I have. Are any of you charging fixed fees and can share some pricing ideas? I'm based in the US. Thanks!

https://redd.it/13f1fsc
@r_devops

Opinions on job decision

Would you recommend …

Job A)
- Industry/production environment
- Salary increase 28k, no on-call, same hours
- IT director pushing shift to Linux/OSS
- Phenomenal mindset towards documentation
- There is source control/CI systems implemented but barely used and barely configured yet
- Having the freedom to do things to my preference
- Basically being „the“ guy to introduce DevOps methodology and possibly having a great impact on policies and standards

Job B)
- Finance sector
- Salary increase 40-45k, no on-call, more weekly hours
- Already established DevOps culture
- Highly complex workflows
- Very restricted due to regulations
- Being able to learn from experienced peers
- Basically being „one“ of the DevOps guys


Any opinions are welcome

https://redd.it/13f3opj
@r_devops

How to Approach Kubernetes Infrastructure With Container Sprawl in Mind

Just came across this interesting piece that discusses how to tackle container sprawl in Kubernetes infrastructure. It dives into some key aspects like resource management, logging and monitoring, scalability, security, and networking. I found it pretty useful as I've been dealing with container sprawl issues in my own projects. Curious to hear if any of you have faced similar challenges and how you've addressed them in your Kubernetes environments.

Source: https://www.devopsinstitute.com/how-to-approach-kubernetes-infrastructure-with-container-sprawl-in-mind/

https://redd.it/13f54vg
@r_devops

Oh, it's just Github having their 'routine' daily outages

In all seriousness, what's happening there?

https://redd.it/13f6f56
@r_devops

What would you want to be hired as?

What would you call this role? DevOps (not really a role), SRE... not really, Platform Engineer... maybe but also has some ops.

EDIT: This person is joining a company where others have been doing this for several years, they are not the only person doing it, just the first person dedicated to it that isn't also managing teams, or also doing product development.

This is a startup where we're starting to distribute the hats people wear but still have a ways to go, and it does have a division between ops and dev, though it's getting blurrier every month.

Responsibilities of the role are to help with all of the following but doesn't need to lead/own it:

1. AWS Cloud/Infra management. EC2, RDS, S3, MWAA, IAM using CloudFormation and the console.
2. Person that Operations can escalate to. The Ops team is trained on the application and the data but have limited AWS/Infra and real Observability/tuning experience.
3. Improve observability, ELK, Py Dash and alerting, PagerDuty for example
4. CI/CD Pipeline development and maintenance excluding writing test automation.
5. Manual deployment on areas without automation, until they help setup the automation
6. GitOps
7. Performance tuning and optimizations' in production ranging from concurrency settings for the application to rightsizing AWS Infra
8. Development Env, Docker management
9. Escalation point for Development when they have Infra/Pipeline issues. (not on call)
10. Security implementation around architecture and software, not business risk.

What other responsibilities am I forgetting?

No "On call" time but when DNS messes up and we lock ourselves out of the office like Facebook did, they would be called along with every up from them through the CTO.

https://redd.it/13exmrp
@r_devops

Hashicorp Vault TCP listener

I was wondering if my tcp listner defined in my vault config has to be HTTPS? The reason being is Im trying to utilize a pre-existing nginx (gitlab omnibus nginx) to serve as a reverse proxy for a vault instance running on the same host but every example Im seeing has the TCP listener configured for HTTP. So Im not sure what is the best case for production use.

https://redd.it/13eq2ur
@r_devops

Run ClickHouse like a Cheapskate – 6 Ways to Save Money While Delivering Real-Time Analytics

ClickHouse analytics can be fast and really cheap if you do it right. This webinar digs into the cheap part showing tricks any dev can apply to save up to 90% on cost. Here are three of many we'll discuss. First, do free as-in-beer development using open source. Second, optimize compute, storage, and memory on ClickHouse itself. Third, move off AWS or GCP completely to cheap hosting at vendors like Hetzner.

Join us LIVE for free TOMORROW 10th May. RSVP now: https://hubs.la/Q01KS44X0

https://redd.it/13d01w9
@r_devops

What should I do next?

I am currently working as a devops person in a small startup. Though my role is called a devops person I am doing most of works like sysadmin stuffs. I am learning everything side by side. Currently I learned Jenkins, ansible,docker,k8s(still learning), terraform (still learning), Prometheus and grafana. I learned AWS and got certified. I learned all these by watching tutorials so I haven't done any real time projects yet. May be I've just learned basics alone. Considering what I mentioned above as learned, what should I learn next? Should I learn coding like python?
Or I should build projects with the tools that I've learned? Which should be my first priority?
I am planning switch company after 4 or 5 months. Which should be the priority? Learn next tool or continue building projects?

https://redd.it/13fb696
@r_devops

AI tools / plugins for devops engineers?

Hi everybody,

Since generative AI seems become more and more mainstream I was wondering which AI tools / plugins y'all use in you daily business to make your life easier?

I know about ChatGPT and GitHub Copilot, however Codepilot seems to be more dev focused. I also tried to make a short Google research but there seem to be tons of AI tools out there, therefor I was wondering if you could give me your honest opinion and recommendations about some tools / plugins you learned to love :)

If you need some details for recommendations: our main techstack currently revolves around Google Cloud, Kubernetes, Terraform and some basic Cloud Functions written in Python / NodesJS. But I'm curious in general about your tips and tricks.

Thx!

https://redd.it/13clymd
@r_devops

Ask for recommendation Suggestion on learning a new skill in DevOps

Hello knowledgeable community members,I recently secured a position as a system administrator in a Linux infrastructure environment (seem quite large, > 10k servers). It appears that the company I'll be working for has a significant on-premises infrastructure rather than relying heavily on the cloud. With about a month left before my graduation, I have an opportunity to learn a new skill.

During my previous work experience, I had exposure to Linux servers and Ansible. While pursuing my education over the past two years, I continued to deepen my knowledge of Ansible and explored additional technologies such as Docker, Terraform, and GitLab CI/CD pipelines. I worked on my own projects involving Terraform+Ansible Infrastructure as Code (IaC) for server provisioning and system configuration, docker image building and container deployment. However, I'm aware that there's still much to learn in these areas. I don't think I have in-depth knowledge of everyone here.

I also had the chance to experiment with AWS and completed projects like deploying a high availability and scalable WordPress website, setting up an S3 bucket with CloudFront, utilizing OAC and WAF, and implementing a simple newsletter system using Lambda functions, API Gateway, SES, SNS, and DynamoDB.

I also played around with some monitoring tools like Prometheus, Grafana, Loki, Protmail, and Telegraf during these two years. e.g. Doing some deployment and making it work with my homelab environment, getting the metrics and set up the alert with alertmanager and send to the my own slack channel. e..g CPU/memory spike alert, disk usage etc.

Regarding my programming skills, I admit that I have room for improvement. I have worked with Python and bash scripting in the past, but unfortunately, I haven't had many opportunities to utilize them extensively in my work. Also, I have found ChatGPT to be incredibly helpful, especially when designing scripts for specific tasks and integrating them into cron jobs. Do you have any suggestions for me to learn programming skill in this particular area?

Now, I am seeking guidance on the next steps in my career development. Should I focus on further honing the skills I already possess or should I pursue a new skill set? I don't want to squander the upcoming free month, and my goal is to transition to a junior "DevOps" role next year. Automation is an area I particularly enjoy and excel in. I have discovered a great deal of satisfaction in working in this area.

I would greatly appreciate any advice or suggestions from this knowledgeable community. Thank you in advance for your insights and support.

https://redd.it/13fdklg
@r_devops

Coming from non IT background. Should I start with learning Linux? How is Imran Afzal’s video lectures on Udemy to learn Linux?

A friend suggested to begin with Linux to get into devops. Would like to know this community’s view. Thanks

https://redd.it/13cjzmm
@r_devops

Create windows node cluster Kubernetes in windows host

I have an IIS app with a docker image in windows/amd64 (arch). Looks like I need to create Kubernetes windows node cluster to orchestrate this docker in a windows host.

https://redd.it/13ff8gg
@r_devops