How to sclae OsX builds?

Hi all! I'm almost 1.5yrs into my position as a DevOps engineer at a large company. I was lucky enough to get this gig straight out of college. I know I'm not anywhere near close to knowing all I need to know in this discipline but I'm learning more every day.

Currently, we have an application that we build on macOS, Windows, and Linux. For Macs, we currently have a bunch of minis sitting in a data center and our Jenkins master will target an available mini to run a build on (a very simplified explanation but that's pretty much the gist). My issue is, I don't think this method is scalable. We have more and more developers coming on every year so we have to buy more minis to keep up with the demand. We've thought about using Macstadium but from testing, we've seen our build times almost triple. I've been trying to do research into finding a scalable solution, preferbly with cloud tools, but I'm coming up empty.

Any ideas or advice would be greatly appreciated!!

https://redd.it/dtkfbp
@r_devops

How would you bundle many applications within kubernetes. Service mesh?

I'm just wondering if it's a good idea to manage a high number of applications running on their own scaling up and down and each of them connecting to each other all on plain kubernetes. Or maybe using a service mesh would be a better fit?

https://redd.it/dtjztd
@r_devops

Budgeting question for devops leaders

How do you go about forecasting budget for your cloud spend and what metrics are you using to make those estimates?

Are you 100% usage driven? How to you estimate and account for new features in a coming year that aren't yet planned? Having the moment to moment flexibility is nice, but finance folks still want annual budgets and want teams to stick to those budgets. Obviously if there is a good reason (business growth) the finance teams are happy, they just want to understand the why when it comes to overages, but things get complicated when you're in the millions of dollars in spend across hundreds of deployable assets spread across dozens of teams with no concept of 'chargebacks' in place to hold any team accountable.

https://redd.it/dtm5z6
@r_devops

Employer changed Stock options vesting policy

My offer letter said my 5k stocks would start vesting after 6 months, then every 6 months after that. I worked at the company for 10 months and didn't like it, so I started looking for another job and got one.

When I talked to the COO about it, he said they changed to policy to start vesting after one year. Then basically said there's nothing I can do about it.

So, is there something I can do about it?

https://redd.it/dtmsc9
@r_devops

Sources for a really deep dive into containers?

I’m looking for some reading material that shows exactly how containers work, in depth. Not a how to, but a pretty low level explanation. Can someone recommend me something?

https://redd.it/dti2lo
@r_devops

Ansible's log_plays callback plugin doesn't log the TASK banner, possible to get that functionality?

Ansible's log_plays callback plugin does exactly what we're looking for, logging of all the play results per hostname into its own separate file. However, the output cuts off the "TASK [Here is the name of the task]******************" portion for each task. Has anyone encountered this and changed the plugin to achieve this, or know of a plugin and/or configuration that achieves that per hostname logging but would give us the TASK banner portion?

https://redd.it/dtfz6r
@r_devops

Containerization Security Challenges

what are the container security risks that is commonly faced and any recommendation for precautions?

https://redd.it/dtfc7n
@r_devops

Question: Having daily issue with Kubernetes Cluster (Nodes are in not ready state)

Is it a good practice to scale down master node as well as worker node every day when not required?

Or should I only scale down the worker nodes?

Kindly explain why should or shouldn't I scale down the master node.

The issue is every morning my cluster is up but PODs are getting failed every time (I scale down master as well as worker node daily)

https://redd.it/dtbs2q
@r_devops

How to work with a turd such as Microsoft Azure?

FYI it's my rant at Azure....


I'm constantly forced to produce automation solutions for Azure and I can't comprehend why people are using it. Things are constantly broken, changes at the back-end are happening without considerations for users. i.e. sudden changes to API versions without regression.

Every Monday is a battle as something is breaking usually with random timeouts or errors , I have stopped counting how many times I had to increase retries beyond sane logic.

​

With recent Ignite the amount of breaking code which had been pushed to production is insane. I'm constantly finding new bugs for previously working feature, yet nowhere to report it. Phone support is as useless as the code changes.


I'm interested to hear if I'm the only one facing those problems and how are people mitigating incompetence of Azure engineers.

https://redd.it/dtgans
@r_devops

Anybody in the field have tattoos?

I want to get my knuckles done and possibly my hand within 2 - 3 years. How much has it affected your hiring chances etc?

https://redd.it/dtvmnf
@r_devops

Difficulty in understanding concepts

I work with cloud technologies but I feel I don't understand the concepts or terminologies at depth. And I'd like to fill that gap so that I'm more effective at my job. Not that I don't get it but my understanding is shallow. Example would be, recently i was reading up on managed identities in cloud and service principals. I realized that at the end, although I felt I understood at least 60% of what I read but I'm far from being able to implement it. I'm a year out of university and working with cloud technologies for the first time.


Any suggestions?
Thanks in advance

https://redd.it/dtar41
@r_devops

Comparing two PostgreSQL databases with Python

After an infrastructure migration we had to figure out whether data was lost during that process. We wanted to ensure the new PostgreSQL database had exactly the same data as before.

Just sharing our process over here: [https://medium.com/trinity-mirror-digital/comparing-two-postgresql-databases-with-python-a98926ba9945](https://medium.com/trinity-mirror-digital/comparing-two-postgresql-databases-with-python-a98926ba9945)

https://redd.it/dt1lud
@r_devops

Managing secrets via AWS Param Store

I've decided that I am going to use AWS Param Store to handle my secrets for my Lambda function and am looking for the best way to manage them. Like is it something I do just manually via the CLI or is there a better way of managing them?

https://redd.it/dty0yp
@r_devops

Cost insigt into Google BigQuery

I`m interested in approaches/tooling to see the drivers behind shared BQ database.

We have a BQ database used for insights into viewers patterns among other things. It is used for our own recommendations software project, the major driver and owner of the database. In addition to that it is used by our analytical department for ad-hoc queries. The challenge is to see what is driving the cost precisely. The software running queries is ofcourse easier to keep track of, they are encoded and you can guestimate. The analysts on the other hand run ad-hoc queries and link more tables creating heavier queries. What I`m looking for is a breakdown of each.

So far I´ve thought about using audit logs and separating by accounts of queries ran, but can´t get the straight cost out of GCP spend logs.

Any tips?

https://redd.it/du0kx5
@r_devops

I just made the first step to my goal of becoming an AWS Certified DevOps Engineer!

I passed my Cloud Practitioner exam!! I've been dabbling in AWS for years now and I love how it feels to make those big, impactful changes on a system. Right now I'm a Software Engineer and I've never gotten an official certification or a college degree, so taking this first step has made me happy beyond belief.

My next step is AWS Developer Associate, so if anyone has any words of wisdom for me, let me know.

Anyway, I just wanted to share my successes with everyone cause I couldn't hold it in any longer.

Thanks!

https://redd.it/du3tum
@r_devops

Version control for mono-repo with microservices

Hey guys,

I have been trying to finalize the **version control theory for the new repository structure** that I am working on which has microservices(*currently 3 but can be more*). **The decision that has been taken collectively is to go ahead with MONO-repos**. We can debate about pros and cons of mono-repo vs multi-repo but clearly, the opinions are very much divided if you check the internet.

No, lets talk about how we can create versioning control for End to end product, each microservice and how to maintain the codebase so that in case point in time, we know which microservice was at which version for a global version of product. Let me put an example below:

>Suppose for product ABC, we have 3 microservices. At beginning, ABC and all 3 microservices are at 0.0.0 (as per semantic versioning scheme). Now, we can't control the order in which we update each microservice but what SHOULD be our strategy to bump up the **MAJOR.MINOR.PATCH** part of our product as a whole.

It will also be good to **tag GIT** (source repo) along the process so that we can fetch the code from GIT in case of bugs. And **what happens when we fix them**, precisely, how to update the overall version of product?

Additionally, **how do we handle inter-dependency amongst microservices**. There may not be all microservices that are dependent on each other, but surely there will be few. How do we define this dependency part in our repo. It can be defined in each parent folder of each microservice or how?

>I have researched on google and learnt about Lerna but it seems good fit for webservices based projects. Have read that google and facebook both are using monorepo so I reckon they must be using some tools to achieve this. Any clues about them?

How are you handling this kind of requirement at your workplace? I am sure everyone must be having/ or had this pain while migrating to microservices based deployment from monolithic.

https://redd.it/dtum27
@r_devops

Do you know any server admin panel tool/service with devops features ?

Is there any service, where I can add my own server and run/schedule simple administration commands, git operations, docker, run builds, manage nginx, see stats and similar stuff ? I am thinking about writing my own simple script for that, at least project + git + docker management.

https://redd.it/dtth0u
@r_devops

Cloud migrating production in 18 hours without a plan, and how it made me a better programmer ;)

We had a b2b webapp product in cloud_provider_a which started having trouble serving requests, timing out.

My team (supporting another product in cloud_provider_b) was tasked with setting up a duplicate of prod in cloud_provider_b. I can't even remember why mgmt thought that would help.

By the end of the day, by hook or by crook the call was made to switch Production traffic into a newly made stack in a different cloud than where that application was currently hosted. Despite protest from the engineering and QA teams.

Its all a blur now. Things went horribly wrong for two days. On the third day my team (with no prior knowledge of how this product works) found that the docker containers running the app had an nginx config with a low worker_connections setting.

We changed that one line and the app is back up. But the cost has been high, our customers had a far higher level of disruption these 3 days. Our engineers feel burnt out and disrespected, and our QA teams are appalled.

AMA I guess

https://redd.it/dtoqti
@r_devops

New to Bit Bucket Pipelines -Why can't I read my build logs?

I can access them however every instance of the letter `c` in the log is replaced with `$en$en$en$encryption_keyryption_keyryption_keyryption_key`


Example:
>To see the full sta$en$en$en$encryption_keyryption_keyryption_keyryption_keyk tra$en$en$en$encryption_keyryption_keyryption_keyryption_keye of the errors,

Translates to:

>To see the full stack trace of the errors,

Anyone have any ideas how I can make it readable?

https://redd.it/dt1ah7
@r_devops

Reality of software developers having no guardrails

My company is at a crucial point where the software development side of the business is fed up over the engineering side not providing more self-services platform (compute, networking) and infrastructure (databases, message queues, etc) causing efforts to take significantly longer and don’t invoke “developer joy”. Everything becomes a ticket with the over-utilized, engineering support centers that have partly automated processes, who feel they aren’t brought in on new initiatives to understand priorities or account for capacity. Lastly, there is always complaints when the business asks for anything seen as “undue” process: change management, security, SOX compliance, etc; like there is no justification for any of these things.

Because of this, the software development teams are complaining to leadership that AWS is the only way to solve everything. Comments about having self-service PaaS/IaaS and being able to turn up resources without capacity planning delays have some weight, however most of our software developers are junior without much experience around system administration or operating production services. Most dev managers give into whatever they or their teams think they need to solve problems individually with little thought to broader business or platform needs such that every service looks and operates differently.

Given all of this, I wonder what others in a DevOps / Platform engineering role have done when faced with this culture crisis. Where have you seen guardrails be necessary between software developers and build/release/deliver processes? How much should a software developer know beyond pure product development; networking, OS, automation, packaging, etc?

https://redd.it/du6olo
@r_devops