Handling secrets in Flux v2 repositories with SOPS

This is part 2 of my series on “GitOps with Flux v2”: "Handling secrets in Flux v2 repositories with SOPS"

If you’re not familiar with what Flux is and how it helps you build GitOps workflows on Kubernetes, feel free to read part 1 here: “Introduction to GitOps on Kubernetes with Flux v2”.

In today’s guide we will look at Mozilla SOPS and learn how to incorporate it with Flux v2 to store encrypted secrets in our GitOps repositories and have Flux decrypt them automatically during deployments.

Hope this is helpful to someone.

https://redd.it/lx0qfl
@r_devops

Artifact/Package versioning

How does everyone handle versioning of their artifacts/packages?

Are you using semantic versioning and increasing with every change and just deploying once it passes your pipeline? How do you track what needs to be deployed then or what's in dev/production?

Do you not version artifacts at all and just rely on your source code versioning? Same questions as above.

Do you add a tag/version as it moves through your pipe? One for feature branch, dev, master, etc, so that you always know what's where and just have your pipeline change the tag?

Something else?

In my case specifically, it's just RPMs right now and maybe eventually Conan packages. No continuous delivery to customers (no connectivity to them) atm.

I understand it's a choice to what fits your organization, just looking to hear some pros and cons and issues you've done with different methods.

https://redd.it/lz3ysl
@r_devops

Junior Devops salaries/life in London?

Just curious for some outside perspective on this, as I’m moving to London at the end of the year. I’ve gotten different answers from different people so it’s hard to be sure what’s what.

Also, how is the work there? I’ve heard work-life balance can be better but I guess it depends. Thanks

https://redd.it/lz7mrw
@r_devops

Looking for non-dev friendly batch job operation service

My organization runs a lot of containerized batch jobs, mostly for importing and exporting data from third-party APIs on behalf of our customers. Today, jobs are both provisioned and operated by our devops people. The main causes of failure in these jobs are external and I would like our support organization to handle most of these failures, since the customer usually wants a notification and explanation and the main resolution is anyway to re-run the job with a slightly different configuration.

Thus, I would like a system somewhat as Airflow or Argo to manage these jobs. But unlike those systems, I would like one which (falling importance):

- has easy to use (i.e. point-and-click web UI) job provisioning
- allows devops to operate infrastructure and configuration (e.g. job templates)
- supports both scheduled and manual runs
- provides easy access to job log files and basic metrics (e.g. RAM consumed)
- has a reasonable API for programmatic access

Hashicorp Nomad combined with Hashi-ui (https://github.com/jippi/hashi-ui) comes relatively close, but is disqualified because it provides no support for easy to use provisioning. Azkaban comes relatively close, but seems not to have strong supoort for containers.

Does anyone know of such a system or service, preferrably a FOSS one?

(My understanding of Airlow, Argo and Celery springs from research rather than operational experience, so info on how to extend these to fill my need would also be an appreciated answer.)

https://redd.it/lz5y66
@r_devops

Incremental introduction of IaC and DevSecOps to a traditional IT department

I've been in IT Operations since 1996. I've always enjoyed scripting a process, whether it was in perl (young people, that was the python of the day) or vbscript, etc. Through the years, I have seen new languages evolve and have really been excited about the idea of coding everything and getting away from "right-click to glory" and "lemme ssh in and fix that" style Operations.

My current challenge is convincing my leadership that DevSecOps is not just for companies that produce software, but for traditional IT shops as well. I've done all the Powerpoint slides on how IaC and CaC are going to increase reliability, configuration management, <insert itsm lingo> but I can't seem to get any momentum.

Side-note, I'd say that 85% of our Operations is maintaining existing rather than building new things.

I'm looking for a quick, flashy, smallest thing that could work to show everyone that it is not only possible, but better.

A good robust ecosystem takes buy-in, hours and dollars; and I want to get there, but I need a spark and some kindling.

Creating "one-offs" is never a good thing, so there is going to be resistance to "use this whole other process to do the one thing" so it has to be Culture Changing from the start.

Any advice, war-stories, or README.md's would be greatly appreciated

https://redd.it/lz5qg2
@r_devops

Easiest way to make file local

We have many servers spread worldwide by supermicro. We need to install Ubuntu on all of them from an ISO. We have the ISO auto install seed ready.

Only problem is, we can't attach the ISO file locally to all the remote servers as it would take forever.

What would be the easiest way of setting up the ISO locally on each location? The plan is to do make the file accisable from each location and run an ansible that would auto install the OS through the ipmi.

Thanks ahead!

https://redd.it/lza3h4
@r_devops

Self-hosted tools similar to bitbucket?

I'd like a graphical interface for my git repos at home.

What are the free offerings?

https://redd.it/lzc7u2
@r_devops

Continuous Deploy workflow for deploying to Virtual Machines and Kubernetes?

What are people's strategy to deploying to Virtual Machines and Kubernetes? My company is currently transitioning to Kubernetes but there are some applications that are just not ready to be put onto Kubernetes and some that just may never be put onto Kubernetes because of various reasons like being a legacy application that leaks and needs to be rewritten or needing very specific drivers, etc...


I'm looking to unify the process in which we deploy to Kubernetes and Virtual Machines and curious if anyone has done this yet. We're using GitLab CI for our build pipelines right now.
Current solutions I'm looking into:
Octopus Deploy - Started out as virtual machine deployments, now they support Kubernetes.
GitLab CI + Helm + Ansible - This is expanding on our current solution of using GitLab CI and Helm to deploy to Kubernetes, just when there is a Virtual Machine deployment we'll have an ansible playbook to make sure that the VM has all the require pre-req packages, firewall rules, services accounts etc... then pull the application down from some raw package store as a zip, extract, and run (general through systemd).


What other solutions have you guys used or can think of? I'm also looking into Spinnakers but I'm not sure that it does what I need. I'd like something that follows a similar flow for both Kubernetes and VM to abstract as much as we can from our Developers who deploy.

https://redd.it/lz4ex9
@r_devops

How secure is your Kubernetes cluster?

Do you know? Or you don't even sure? Who's responsible for security in your company? When you apply new configurations to your cluster, do you have some kind of methodology to validate security?

https://redd.it/lyyoqb
@r_devops

Pick a stack: kubernetes + istio vs FAAS using AWS... And why?

Would you rather go kubernetes & istio with the benefits of cloud portability and doing more management versus going serverless on AWS, with the benefits of pay for what you use & offloading so much of the management due to the use of highly managed services (dynamodb, sns, sqs, API gateway)

https://redd.it/lz1z12
@r_devops

How to handle base OS patches with Immutable Infrastructure?

We're currently in the process of delivering an AWS setup which will require two custom-built AMIs. Figuring out how to build and deploy these with a CI pipeline is fairly straightforward, at least from a logical perspective - but how do you handle continued maintenance on these systems?

Say the base OS has a security problem, now you need a new AMI. Something has to automatically build, test and deploy it without producing downtimes while you're asleep and I don't really see how that fits into our current setup.

Can someone link to a decent guide explaining the general approaches to the issue? We're using Terraform, Packer and Gitlab CI, so fairly standard stack. I'd assume we need some kind of additional service that triggers watches builds unless there's some Gitlab functionality I'm missing here.

https://redd.it/lyye2c
@r_devops

What kind of development you do as DevOps?

I initially got into DevOps from working as developer in a small startup who just didn't have enough people to do DevOps. So I got mixed tasks. and I thought it meant "DevOps", as I would do development of backends, and also infrastructure management.

Right now I work in a bigger organization as "DevOps" where I get to actually develop services and also operate others. But my team members really rather not develop anything. And I noticed it's common here. Lots of people who don't want to develop software, rather operate it.

So I got the hunch that "Dev" in "DevOps" doesn't necessary mean development in the sense of building services, but rather as IaaS and configuration instead of running code. or scripting.

I was wondering about all of you people, how much development do you get to do? and what kinds?

https://redd.it/lyxf5g
@r_devops

Goodbye minikube

I’ve been using minikube as my local cluster since I started to learn Kubernetes. But I’ve decided to let it go in favor of kind. Here’s the story.

A couple of weeks ago, I gave my talk on Zero Downtime on Kubernetes. A demo is included in the talk, as with most of my presentations. While rehearsing in the morning, the demo worked, albeit slowly. Two days before that, I had another demo that also uses Kubernetes and it was already slow. But I didn’t take the hint. 🤦‍♂️

Read the whole story

https://redd.it/lztyr4
@r_devops

What about EFG stack?

Hello everyone
What do you think about Elasticsearch, Fluentd, Grafana stack?
We are currently using ELK and Prometheus/Grafana stacks in our company. I know core concepts and core modules of Kubernetes and we are going to install Openshift into our Infrastructure. I read some documents about Kubernetes log aggregation that says EFK stack is the best solution at this moment where we set Fluentd as daemonset, Elasticsearch as remote storage or statefulset etc...
The question is I couldn't find much EFG stack. Is this because there are no Fluentd datasource for Grafana but served as plugin through Loki?
Is it bad idea to consider as stack? If it is possible we may consider to ditch kibana

https://redd.it/lzwvmc
@r_devops

Securing a digital ocean server -- need help

Hello, I need help securing my digital ocean server.

So far this is what I've done:

disabled root login
disabled password authentication
installed fail2ban which blocks IP addresses permanently after 3 failed login attempts within the previous 30 days

What I want to do next:

Change default SSH + Mysql ports
Deny access to all ports except SSH/SFTP + Mysql + HTTP/S
Whitelist a single dedicated IP address for these 4 ports, deny all other IPs other than localhost*

Anything else I should consider?

*I think I can login locally as root by accessing my droplet console from my digital ocean web account directly which is fine, because I have to login with Google SSO and my Google account has 2-step auth

https://redd.it/lzxij1
@r_devops

Interviewing for Systems Dev Engineer - Amazon

Hi,
Does anyone here have experience with interviewing for Sys Dev Eng role with Amazon? What to expect during the interview process? - DS/algos, Linux fundamentals, system design, sql querying.

Looking at the job posting it looks like more of a devops Eng role. (But their devops Eng roles look like support Eng roles)

https://redd.it/lzrr86
@r_devops

Has anyone worked for or with Replicated, the maintainers of the KOTS and kURL projects?

Their recruiters have reached out and while I think the software looks solid and the team sounds great, I admit my knowledge of them exists in a vacuum, so I’d like to gather any experiences anyone has had with them for my own edification.

https://redd.it/lzthkj
@r_devops

How to handle development/staging deployments when production deployment in a Kubernetes cluster

I used to deploy traditional micro services stack in three deployment environments(development, stating, production) using docker-compose and I used three cloud VMs as the hardware base.


However currently I'm converting orchestration from docker-compose to Kubernetes. As per my current experience Kubernetes clusters are massively expensive compared to a single VM.

So is there any cost effective or proven approach to solve my current concern. As an example, dev and staging run in single nodes? while production runs in a full capable cluster?

In that case how would developers deploy services on their local machines when needed?

https://redd.it/lzguz7
@r_devops

Have I pigeonholed myself with SOC work??

I am currently a SOC analyst that came up from helpdesk. I feel like I put myself in a bad place. My two helpdesk roles had zero opportunities to touch server/network infrastructure along with the fact that they always hired externally for everything beyond helpdesk with no opportunity to seize to show we could develop in to those positions. They were strictly silo'd off from us and the leadership very specifically did not want to develop anyone.

I had a hard time finding sysadmin gigs even with a homelab and some basic powershell automation experience. So I took a SOC role. But now I'm in a weird place. I'm interested in DevOps, I have SOC experience yet not enough to advance in Security quite yet, no production experience with servers and networking, and everything around me wants people with experience.

I live in DFW more specifically and there's not a lot of junior roles out here and unfortunately I can't move due to my family.

I guess I'm just kind of lost. Do I go back to helpdesk/desktop stuff and hope I find something better than my last two situations where I can find more autonomy? Do I just keep up with personal projects and hope someone eventually takes a chance on me?

https://redd.it/lzhhax
@r_devops

We are ready to upgrade the exporterhub.io !

Hey folks!

We are ready to upgrade about of exporter-hub as below.

\- https://github.com/NexClipper/exporterhub.io

Exporter-hub is a web-service and open-source project for ‘Easy find and apply of Prometheus metric exporter’.

So, If you need some function or improvement about this? feel free send me a comment :)

Thanks!

https://redd.it/m04fes
@r_devops

How to monitor my dockerized app with ELK ?

I have a dockerized rails app that I want to monitor. Already have the ELK suite hosted on elastic.co.

Which agent should I run on my app server ? Is it Beats ? Is there a docker image so that I can add it as a service ? I’m having a really hard time finding this info.

If anyone has an example docker app, that would be really cool.👍

https://redd.it/lzzrx0
@r_devops