Triggering K8s Jobs internally?

We at present host of containers with ECS, and are looking to move into K8s.

Our batch layer at present has a controller container, that triggers tasks based on need, via AWS's apis.

In order to move this into K8s, we plan on creating yamls for Jobs, but I was wondering how we go about triggering the jobs themselves. I mean we could ask the devs to program it, but I guess we need to store the kubeconfig as a secret or configMap in the cluster to be able to load it? Or is there a better way?

https://redd.it/owdek2
@r_devops

DevOps, SRE, or Platform Engineering - how to choose your poison

Hi folks!

I've been a software developer for quite a long time, but relatively recently, I've moved into the SRE role. My new job responsibilities were quite clear from the beginning, thanks to the Google SRE book (easy to understand and apply) and the already established SRE culture in the company. But the new domain itself was quite vague. I'd been struggling to understand the difference between DevOps and SRE; both seemed to be the same but felt quite different based on the job offers I was getting periodically. So, I was confused about how to position myself in the new market and what skills should I invest time into? Basically, same feelings that I had when I was just starting my software dev career - I had no good understanding of the potential areas of focus. Do I wanna be a backend, a frontend, a full-stack, or a data engineer? What's the difference between these roles? Are there other options? So, after a year of being a true SRE and then a year more of being an SRE in Platform Engineering, and periodic analysis of the market, I managed to come up with a more or less consistent picture in my head. So, sharing it here in case someone is in the same trouble of grasping the domain for the sake of self-orienting.

P.S. All things in the post are IMO, and probably other people may have totally different experiences. So, let's try to keep it an open forum.

https://redd.it/owdyxl
@r_devops

Question about secrets inside git repositories and how to deal with them

Hi,

I'm having some problems when removing secrets such as API keys and credentials from files in our Github repositories.

According to multiple people/articles the best course of action is to remove the offending files (e.g. ".env" and "google-services.json") and add them to .gitignore in order to prevent them to be reintroduced to the repository. In my point of view it will only result in the secret being shared in many other places such as instant messagers since developers need them to test their applications.

What do you guys do in this case? Is there a standard way to go about this problem?

PS: I was looking into a tool called BlackBox by StackExchange which keeps the file inside the repository, but it's encrypted using GPG, which seems to be a good strategy keeping the secret safe and the secret sprawl under control.

https://redd.it/owfipu
@r_devops

DevOps stress level, what do you think?

Hi all,
How high, in your opinion, are the stress levels for a DevOps job, and what they are impacted by?
Meaning, what can contribute to increased stress and what can help reduce it?


This came up today when I was talking to a friend of mine, who argued that DevOps was one of the most stressful roles in the dev organization (hence he can't quit smoking :) ).


Unhealthy lifestyle choices aside, what do you think?

https://redd.it/owf7hb
@r_devops

Scaling Prometheus to handle high cardinality data at high scale

https://engineering.razorpay.com/scaling-to-trillions-of-metric-data-points-f569a5b654f2

What were your experiences on Prometheus for such use cases?

https://redd.it/owfa50
@r_devops

Favorite cloud logging service for small team/project?

Looking for opinions on cloud-based logging systems for a 3-person dev team running a service that has pretty low request volume for now. We don't have any complex analytics needs yet; nor do we have thousands of hosts to aggregate across. Mostly need something that will let us do reasonable full-text search across a dozen or so hosts and do simple "regex X matched more than Y times in the last hour" kinds of alerting. Real-time tailing of logs would be nice to have as well.

One slightly unusual requirement is that our system has a mix of AWS hosts and a small number of on-premises servers at customer sites, and we want all the logs to go to one place.

When I last evaluated services like this at a previous company, I ended up choosing Scalyr and was mostly happy with it, but that was 7 years ago so maybe there are new entrants I haven't heard of.

Self-hosting an ELK stack or such is an option but not our preference given our small size and our lack of experience operating our own logging and alerting infrastructure, plus the on-premises part of our setup.

https://redd.it/owg7nj
@r_devops

An alternative to latest Linus vide about Marvis?

https://www.youtube.com/watch?v=KrhzKOiRzV8

This looks like some kind of proxy packed up with a lot of monitoring tools, but it ended up with a nice UI.

Wondering if there is something similar for home network

https://redd.it/owgzjl
@r_devops

Is DevOps as a profession too beholden to Big Cloud?

This has been bothering me for some time now, and I just cannot find anyone talking about it in the context of DevOps as a professional role.

We all know Amazon, Microsoft and Google absolutely dominate the cloud space. Simultaneously the same companies also happen to dominate incredibly large swaths of the tech space in general.

Some of us may also know the rising controversies around "Big Tech" and the consequences of their extraordinary power with regards to privacy, deplatforming, but also technical over-dependence of the entire internet's infrastructure on the few big players. AWS is basically a global example of a Single Point of Failure. AWS has an outage, third of the internet is disrupted.

Yet the few professions (like DevOps and SREs) which could do something about it seems to just happily contribute to the situation.

Almost EVERY job offer lists AWS, Azure or GCP as a requirement or a preference, and seemingly everyone who is into DevOps seems to be perfectly happy with the situation despite the fact that there are plenty of other clouds, the likes of Digital Ocean, Linode, Hetzner or even IBM Cloud. And we should be the first ones to understand the concept of not putting all eggs into one basket when it comes to infrastructure.

I almost feel like DevOps Engineers effectively work FOR big tech as their unpaid salespeople, all too eager to just jump on the bandwagon of big cloud and never stop to think if maybe they could differentiate a little bit, that maybe AWS, Azure or GCP aren't always the only or even the best solution.

I understand being pragmatic in your professional life, but there must be some space for a wiggle room, for influencing your organisations away from centralization both within your current roles and in your choices of new roles. But most of all, I feel like this is absolutely worth talking about, which nobody seems to be doing.

I wonder what others think? Am I off the mark somehow?

https://redd.it/owm5pz
@r_devops

8 tricks to use S3 more effectively with Python

Leverage the power of S3 in Python by:

1. Reading objects without downloading them
2. Downloading files to a temporary directory
3. Specifying content type when uploading files
4. Retrieving only objects with a specific content-type
5. Hosting a static HTML report
6. Generating presigned URLs for temporary access
7. Uploading large files with multipart upload
8. Making use of S3 Transfer Acceleration

​

I've put together a "how to" on each of these 8 methods (incl code snippets) here: https://dashbird.io/blog/aws-s3-python-tricks/

https://redd.it/owfa67
@r_devops

Is leaving the only way to get a big pay day?

I've been with the same company for coming up on 10 years and they've treated me well in terms of increasing pay over the years, especially considering I started at an entry-level salary with no related educational or professional background; however, I've seen people jump ship and get big pay days. I've also had co-workers get offers and use those as leverage to stay, but something about doing that for the sake of a raise feels slimy to me. The trouble is, I can't imagine walking into my boss's ~~office~~ Slack channel to ask for a 10-20k raise just for the sake of being on par with someone who left for the same position at another company.

I know my boss has been pleased with my growth and contributions and I feel like I've taken advantage of opportunities to take ownership of things over the past few years and while I've had a couple of small raises during that time, I know that connections of mine with similar experience are making 10, 20 or even 30k more at other companies in the area, so my question is - is leaving the only realistic option if I want a big pay day?

I love everything about the company I work for. The technology, the culture and the work-life balance are ideal, but I just continually get that feeling that I could be making a lot more elsewhere; that being said, I'm very non-confrontational, so I'd hate to blow it asking for too much and have them assume that I'm on my way out if they can't deliver what I'm asking for because I'd much rather have a job than be forced to go find something else.

https://redd.it/owp0xl
@r_devops

Any good docker box for production Wordpress?

Looking for some good boxes using varnish and nginx. I am also looking for something better with more advanced caching.

https://redd.it/owoex8
@r_devops

Any good LEMP stack for leaning purpose?

I have a really old application, and I want to update the docker-compose file, but I would like to see a few examples to test out various things, I have a lot of images and some of the images may not be compatible with other images, so I need to figure out a way on how to fix everything in our dockerized application.

https://redd.it/owqnwm
@r_devops

Transitioning from sysadmin to DevOps.

Yeah I know DevOps isn't supposed to be a title, been it's been morphed into one. I'm a system admin and have been for about 10 years. Sysadmin jobs are basically all dead at this point, and I have like 40 years until I retire, so I have to retool at some point. The only scripting skills I have are in PowerShell and the only tooling that I have any experience with is Ansible.

Has anyone else in this sub come from a background of system administration, if so what skills carried over? I assume that most jobs under the DevOps umbrella want expert level skills right out of the gate, so I have no idea how anyone new breaks into this field.

https://redd.it/owrsa3
@r_devops

How do I setup a "service account" to hit the compute API? (Azure hellscape)

I can't figure out how to hit the compute api. I just want to get a hello world query where I get back a list of VMs. In AWS, you simple create a service account and apply the given READ permissions.

In Azure, I'm having trouble figuring out how they want you to do that. I'm using the new typescript packages "@azure/arm-compute". It requires a tenantId, clientId, clientSecret and subscriptionId which I've found and seems to be authenticating okay.

However, I get back the following error because the "Azure AD" "App registration" doesn't have permissions for

```"error":{"code":"AuthorizationFailed","message":"The client '123 with object id 'abc' does not have authorization to perform action 'Microsoft.Compute/virtualMachines/read' over scope '/subscriptions/123/resourceGroups/testing_group/providers/Microsoft.Compute' or the scope is invalid. If access was recently granted, please refresh your credentials."}}

```

I can't for the life of me figure out how to give this "Azure AD" "App registration" the relevant permissions. You would think it'd be the easiest thing in the world, but it seems hidden.

When I go to "API permissions" under the application, I hit "add permission" and then "Azure Service Management", it won't show me any permissions and I can't select "Application permissions" only "Delegated permissions".

Does anyone know what's going on? Also can anyone point me to how I get paid support from azure? I'll literally pay whatever to get this solved lol but even paying them money they make hard to find how to do that.

https://redd.it/owtj1z
@r_devops

setting up s3 bucket to handle cross origin resource policy

Hello,

​

I am facing issues playing audio files from my s3 bucket. I was wondering if I set up the cross origin resource policy poorly.

​

Here is my server.js file:

​

app.use(cors());


app.use(function(req, res, next) {
res.header("Cross-Origin-Resource-Policy", "cross-origin");
res.header("Cross-Origin-Embedder-Policy", "require-corp");
res.header("Cross-Origin-Opener-Policy", "same-origin");
next();

});

​

The error chrome throws me is:

​

GET https://monkeys.s3.amazonaws.com/sounds/1/6.wav?AWSAccessKeyId={test}&Expires=1627949062&Signature=HkT2CyazqovtjPVpDNKCu9Nsyk8%3D&response-content-disposition=attachment net::ERR_BLOCKED_BY_RESPONSE.NotSameOriginAfterDefaultedToSameOriginByCoep
manage:1 Uncaught (in promise) DOMException: Failed to load because no supported source was found.


My bucket is set up like this in the cors section:

​

[
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"GET",
"PUT",
"POST",
"DELETE"
],
"AllowedOrigins": [
"https://telecurve.herokuapp.com/manage"
],
"ExposeHeaders": []
},
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"GET",
"PUT",
"POST",
"DELETE"
],
"AllowedOrigins": [
"https://telecurve.herokuapp.com/"
],
"ExposeHeaders": []
},
{
"AllowedHeaders": [],
"AllowedMethods": [
"GET"
],
"AllowedOrigins": [
"*"
],
"ExposeHeaders": []
}
]

​

Anything im missing? I've been at this for days and can't seem to find out why. I have more posts detailing the code side if you need more context! Desperately need help.

https://redd.it/owrege
@r_devops

AppOps: The evolution of DevOps

Hey everyone, I read this blog and its really amazing. The way author has explained everything regarding how DevOps can be used more efficiently and how it has evolved itself to AppOps is really great.

What do you guys think, AppOps tools like Devtron or Shipa as mentioned in the blog, will be the future ? Will these tools be able to remove the friction between the developers and operations team by providing a flexible & seamless experience ?


https://betterprogramming.pub/devops-appops-f096cdbb02ac \- Blog

https://redd.it/owwis9
@r_devops

How do you know what tools are right for you?

Hi all :)

I'm in the process of evaluating a new tool for my company, and I'm searching the internet for tips and best practices for choosing the right DevOps tools. Not necessarily the "best" tools but the right tools.


I've found some interesting articles, and frankly, I enjoy delving into the minds of likeminded devs to compare their thought process to mine, so I wanted to share some of the articles I found and ask you guys - what factors do YOU take into consideration when evaluating a new tool? is it just cost vs. value, or do you have more bespoke criteria? do you factor in your team size or any existing tools you already have? I'm curious to know what should be my focus.

Here's some of what I've found and hopefully you guys could add some more insights on top of it =\]
Just to clarify, I'm interested in the process itself and not just what tools you have.
* https://www.helpsystems.com/blog/5-rules-choosing-best-devops-tools
* https://techbeacon.com/app-dev-testing/how-find-right-devops-tools-your-team
* https://hackernoon.com/moving-from-circleci-to-buildkite-everything-you-need-to-know-o73r37ie
* https://www.threatstack.com/blog/tips-on-choosing-the-best-free-devops-tools


Much obliged!

https://redd.it/owggnf
@r_devops

The DORA DevOps performance metrics

Feedbacks are very important in DevOps. As metrics provide feedbacks, they are essential to the success of DevOps. But they must be appropriate, and it’s not an easy task. DORA assessed teams to understand how they develop, deliver and operate software (“Accelerate State of DevOps Report”).

The report identifies 4 DevOps performance metrics that can best help the organisation to achieve its goals.The performance of these metrics is measured on 4 levels.

Read more...

https://redd.it/owy2l5
@r_devops

Is there alternative to OTP / 2FA?

I am really annoyed each service requires me enter one time password after enabling 2 factor authorization across all company's WEB assets: it takes 10-15 min each day to login to all those services...

I wonder if there are alternatives. We don't use 2FA when work with SSH keys but WEB apps love 2FA (Jira, WEB cloud consoles, etc).

I'm using browser built-in password manager, There are few OTP plugins for Firefox/Chrome:

* https://addons.mozilla.org/en-US/firefox/addon/auth-helper/
* https://chrome.google.com/webstore/detail/authenticator/bhghoamapcdpbohphigoooaddinpkbai?hl=en

I haven't used KeePass though it also supports 2FA via plugins:

* https://keepass.info/plugins.html

Is it a good idea to delegate 2FA to on PC apps? Today I read 2FA tokens from mobile device and it make sense (intruders supposedly don't have access to phone if PC is compromized).

https://redd.it/owyfpj
@r_devops

Help How can an associate DevOps interview a senior DevOps ?!

As the title states.
I am the only DevOps in the startup I am working at, I have been telling my manager to bring a senior DevOps to learn from because I am still an associate, good thing we have 2 senior candidates but now I am asked to interview both of them, I have no idea how to interview someone who has +10 years experience in everything I know in life, XD.


How can I interview these candidates and what questions should I ask them, btw I am not intevirweing them alone, I have one of the senior developers with me.

https://redd.it/owcfxu
@r_devops