Is my raise enough?

I’m heavily involved in AWS, CI/CD, and architecting and engineering solutions at an east coast higher ed (not in NYC)that’s hemorrhaging IT staff and has no concrete WFH policy. Is a 6.5% raise and a 4% bonus split over two payments good or am I being too critical? Salary now at 130k. Recruiters contact me for positions starting at 150k.

https://redd.it/u5monh
@r_devops

Need help setting up self hosted loki logging mechanism for docker containers.

I am trying to create a monitoring mechansim for our microservices. After an extensive research, I found out using prometheus + loki + grafana is the most scalable, least challenging and most maintainable solution for us since we are going in the self hosted direction.

My current problem is that while prometheus + grafana seemed pretty straightforward, I cannot add loki to the mix. The documentation is a bit confusing.

​

There are apparently two options for Loki + docker for logging.

1. Using promtail + loki for logging, and mounting docker container log files in promtail environment.
2. Using loki logging driver for containers that I want to log. (source)

The first one is not only confusing, but also way less maintainable, since I probably have to lunch a new promtail instance + config files for each time a new type of microservice is getting deployed. Plus there's this hassle of service discovery and container filtering that I have to write manually in the promtail config yaml that is daunting to say the least.

​

The second one while seems pretty straightforward if you use the centralized grafana solution, I cannot find any examples of it being done in a self hosted situation. Has anyone done this? If so, I need a bit of an explaination to how to exactly set up a very small, reproducible example.

https://redd.it/u5qd19
@r_devops

DevOps Junior, Why is BASH something I need to learn?

Good morning / day / evening!

I started learning BASH, and I know the very basics (variables, functions). But I do not know, what can be done with BASH. Why is BASH important for DevOps? I's need someone to explain (preferably someone who works as a DevOps Engineer, or has some experience at a company), what is BASH useful for. To a common young adult.. I do not have any idea what can it be used for.

Also, if you could please make it like:

Beginner: (let's me know what can a Beginner use BASH for)

Advanced: (what are some advanced things you can do with BASH)

Above Advanced: (What are some Above Advanced things one can do with BASH)

I appreciate your time, and will appreciate your effort in a well written comment!

Have a blessed day!

https://redd.it/u5tk9l
@r_devops

Minimal viable continuous delivery example repo

[https://github.com/KarmaComputing/minimalcd](https://github.com/KarmaComputing/minimalcd)
Have been working on this hope someone finds is useful for learning intro concepts to continuous delivery/devops.

It is a minimal web application with state (database) and pipelines which:

* Automatically generates releases based on semantic version for every merge into the main branch (using [intuit/auto](https://github.com/intuit/auto))
* Database migrations are [version controlled](https://github.com/KarmaComputing/minimalcd/tree/main/src/migrations/versions) and ran upon app startup
* This repository uses [alembic](https://alembic.sqlalchemy.org/en/latest/) (python) but you might use [alembic/doctrine](https://github.com/doctrine/migrations) (php), flyway/liquibase (java) - the concept is the same
* When a pull request is opened, a [preview application](https://github.com/KarmaComputing/minimalcd/actions/workflows/pr-preview.yml) is automatically built, with a url so people can view the proposed new version (you might use Jenkins X preview environments, or ArgoCD/kubernetes namespaces for this in larger envionrments)
* When a pull request gets merged into the main branch, the latest application is automatically deployed (using [Dokku](https://dokku.com/)). ([Pipeline Code](https://github.com/KarmaComputing/minimalcd/actions/workflows/deploy.yml) / [UI](https://github.com/KarmaComputing/minimalcd/actions/workflows/deploy.yml))
* You might use Kubernetes with ArgoCD (the underlying concepts are the same)
* A backup/snapshot of any database is taken pre and post each release
* Codebase is regularly automatically scanned for known security issues
* At each release a container is built and published to a container registry ([Pipeline Code](https://github.com/KarmaComputing/minimalcd/blob/main/.github/workflows/publish-container.yaml) / [UI](https://github.com/KarmaComputing/minimalcd/actions/workflows/publish-container.yaml))

​

If people like it/think it'll be useful for learning then I'd like to expand it with further examples/questions

https://redd.it/u5zqz7
@r_devops

Where are you in your career and how did you get there?

I would like to know your stories.

1) Where are you currently in your career? Please include title, company, salary range (<$100k, $100-$200k, $200k-$300k, >$300k)

2) How did you get there? Please include certs, schooling, or experience (don’t need to go to deep into experience just summarize)

3) How do you like where your career has taken you? Talk about work/life balance, stress levels, compensation, etc.

https://redd.it/u62enz
@r_devops

Internal and External APIs in the same webserver

1. Is it a bad practice to host Internal and External APIs on the same web server?
2. We have decided to do this anyway. What is the ideal way to solve this? Some options I could think of are:
1. Using different ports for internal and external APIs
2. Using AWS ALB to create rules? Not sure if this is possible.

https://redd.it/u6h2fx
@r_devops

Atlassian outage prompting moves?

With the Atlassian outages and the information that has been spread around related to their internal practices, are people looking to find alternative solutions?

My wife's company left BitBucket and moved to GitLabs but it is hard to find a good alternative to Confluence and Jira.

https://redd.it/u6jk4i
@r_devops

Getting a take-home assignment for DevOps Engineer III interview process: what to expect?

As the title says, I'm currently in the interview process for a DevOps/SRE role, and part of it is getting a take home assignment to do. I was wondering if anyone here had experience with doing a take home assignment when interviewing with a company, and what I could expect?


I interviewed a few years ago for a more junior role and the assignment was building a simple instance in AWS using Terraform, but I feel that since this role is more "senior", it will be way more complex. Maybe something like designing a complete infrastructure with app servers, load balancers, and complete AWS/GCP config. I have no idea.

Any help would be appreciated!

https://redd.it/u6x89q
@r_devops

What OS are you running on your work laptop?

I have never had the option to use a Unix OS on my work laptop at any of my jobs so far... I find this quite frustrating and I usually end up carrying out a large chunk of my work via an Ubuntu/CentOS hop-on server.

So I'm curious how common it is for IT companies to allow their users to use Unix OS distributions on their machines...

https://redd.it/u70kfi
@r_devops

What OS are you running on your work laptop?

I have never had the option to use a Unix OS on my work laptop at any of my jobs so far... I find this quite frustrating and I usually end up carrying out a large chunk of my work via an Ubuntu/CentOS hop-on server.

So I'm curious how common it is for IT companies to allow their users to use Unix OS distributions on their machines...

https://redd.it/u70kfi
@r_devops

Management always assigning epics/bodies of work to individuals instead of the team as a whole.

This is my third team I've been on now. I have worked under at least 7 managers. The way the work has always been assigned no matter which manager I've had goes like this:

Manager needs something done and either assigns one of the teams members in sprint planning or some other planning session. "X I need you to work on this" sometimes they'll say "I need X to take the lead on this". Or in one on ones they ask the person to do some work for them.

I get that this is just a management style and probably works for lots of folks. But I'm wondering if it's rare to find a manager who just throws a body of work at the team and let's them figure out how to manage it amongst themselves?

I'm not a fan of the X person is responsible for this. It tends to lead to silos in my experience.

https://redd.it/u76jn5
@r_devops

Every monitoring system has an inbox, but why isn't there a central inbox for all my monitoring systems?

Like, Sentry, Datadog, Pagerduty etc. all have an inbox view for the alerts they generate, but there isn't a central inbox. The current way seems to be dump all notifications from all these sources to a team slack channel.
Why do we think this hasn't been centralized yet?
Some points that come to mins are, maybe it would be impossible to merge notification from all these sources because of the distict nature of the notifications and different workflow that happen for each source.

https://redd.it/u7042j
@r_devops

Hi r/devops! I've got a bit of a specific question that I'm having a hard time googling for... Which tends to be when I come to Reddit! I'm building a Pi Kubernetes cluster and I'm wondering about hardware/application monitoring.

Essentially, I want to monitor each of my Pis in addition to the Kubernetes plane and the apps running on it.

Let's say I use Prometheus to monitor all of it. Would I have one Prometheus instance running on the OS to monitor the hardware and a separate instance running on the Kubernetes cluster to monitor my apps & the cluster health? Or could I somehow monitor the underlying OS with a Prometheus running in the Kubernetes cluster?

Does it even make sense to monitor the Kubernetes cluster from within the Kubernetes cluster or should that monitoring live on the OS for reliability? Should I have a completely separate couple of nodes that only host prometheus and monitor the K8s nodes?

Any thoughts, experiences, and opinions are appreciated.

Oh, and by the way, I do understand that this is just a small home cluster so a lot of these choices are inconsequential, but the purpose of the cluster is to learn about best practices, so I'd like to set up the monitoring with that in mind. :)

https://redd.it/u7gq68
@r_devops

Hashicorp survey for 2021 says that there is lack of skills in multi-cloud.

Last year Hashicorp did a survey and among other things (e.g. cost) one the main issues is the lack of skills in the multi-cloud. Yesterday there was a new survey for 2022, but now it's closed.

I think that although the services operational costs might have decreased a bit since last year, the lack of skills is still an issue. But this is actually a good thing, because it's a hint for many devops and SW engineers to know where they should focus and learn new skills and also take advantage of that fact to set their salaries requirements.

DevSecOps and security seems to be the second pain point in the industry, but the survey was before the Log4Shell, which caught almost everyone with the pants down. Therefore, maybe this year security will pop in the first place.

https://redd.it/u7qs7v
@r_devops

Joining DevOps!

Hi guys,

I've managed to get a job as a Junior DevOps with a small company. I didn't expect to get it because 1 , I wasn't looking for a new job (got headhunted by a recruiter) and 2, I didn't have any DevOps skill.

I did the interview because I thought I got nothing to lose anyway and plus I am always interested in DevOps, 2 Interviews later (really nice and friendly team) I got the job!

During the interviews I've made sure they were aware that I don't have any DevOps experience and my cloud skill is a bit rusty due to my current company don't have any cloud infra, but I am willing to learn (which I am and I really excited for it!)

I will start in a month time, now the question is where should I start? I've asked them about which DevOps tool/program that they are using so I can start learning prior to the start date and they said they use Docker, BitBucket and Git.

Background about my experience, been in IT for about 10 years (5 in support and 5 in Sysadmin)

Any pointer would be helpful!

https://redd.it/u7p6w9
@r_devops

Keycloak config management?

Hey, anyone uses here uses Keycloak at their workplace?

How do you guys handle configuration drift across different environments? People at my workplace keep changing some stuff and then when a release moves to another env, keycloak issues arise. I've looked at keycloak terraform provider however it might be too complicated for devs without any terraform experience.

Could anyone give some advice on keycloak config management?

https://redd.it/u7vowm
@r_devops

Error: Issues are disabled for this repo, but they are

so I just moved a github actions workflow from one internal org repo to a new one, (also internal org) and all of a sudden I get the following error message for the action:

Error: Issues are disabled for this repo

Issues is enabled on the settings page in general/features, and the workflow was working fine before in other repos. I'm running the following segment is what causing the issue:


- name: ZAP Scan
uses: zaproxy/[email protected]
with:
token: ${{ secrets.TEST_PAT }}
docker_name: 'owasp/zap2docker-stable'
target: 'https://localhost:8000'
rules_file_name: '.zap/rules.tsv'
cmd_options: '-a'

Any idea where I should look at next to solve this issue? as mentioned I never encountered it before, and I was running it in different repos with the same settings in the same org, so this really comes out of nowhere to me.

https://redd.it/u85z4g
@r_devops

What to ask the company recruiter of a startup

It's hard to tell from the outside if a startup is in decent shape. What questions have people asked the company's recruiter that gave them an idea of the health of the startup overall?

https://redd.it/u88e67
@r_devops

Teaching junior developers

As an industry we tend to select for the autodidact who can teach themselves. I think we do this for two reasons. First since best programming practices aren't settled yet we want people to be able to adapt to change. Secondly it saves a certain amount of effort training.

However this seems short sighted on the effort front. Surely teaching will go faster than letting them discover on their own. Even accounting for both peoples time. Also while not everything is settled there is quite alot that is. TDD is good. IaC is good.

So I think I need to learn how to teach others well. Teach coding techniques, technologies, tools, and practices (like DevOps). How do you go about teaching DevOps to juniors?

https://redd.it/u89dv5
@r_devops

As the owner/lead architect of the cloud infrastructure for a rapidly growing startup, what type of process or concerns do you have around devs (indirectly) proposing infrastructure changes?

To provide a little context, I'm the tech lead of our DevOps team. We've tripled our Engineering team over the last year. With all this new blood comes all the new ideas of how we can improve things. I've historically tried my best to embrace innovation but we just wrapped up a project for a "minor" change that had crazy scope creep and turned into a major ordeal.

To avoid future problems of the same nature, I wanted to put together a list of things for the team to consider when we introduce new changes.

"Infrastructure changes" are anything from deploying a new microservice, to switching from Nginx to API Gateway, leveraging a new AWS service, and everything in between.

* Does this change require modifications to the local dev or sandbox environments?
* Does this change require modifications to automated testing in CI/CD?
* Does this change require modifications to how we deploy to production?
* Does this change impact our security posture in any way?
* Does this change use new AWS services or different features of AWS services than we currently use?
* Does this change serve to fulfill a currently unmet need?
* What alternatives have been considered?

This list isn't exhaustive but I'm interested to hear if you all have a similar set of concerns?

Cheers!

https://redd.it/u88ds5
@r_devops

Switching from fullstack to devops

I've been into web dev since 10 years, I am very strong in both backend and frontend, currently I am mainly working with React and Laravel.

Recently I got an interview for a Devops position. I'm not going to lie, I only started devops since 5 months when our devops guy left. Mainly what I do in my current company is to create Dockerfiles, docker-compose files, modify some k8s configuration files, configure Jenkins so it runs integration testing and deploys automatically, set up composer packages, git repositores, etc ... more like a dev guy is messing around with the already setup tools and fixing some stuffs.

To be honest I didn't even apply for this Devops position, my whole resume is about Backend Frontend webdev, but in my last position I simply put in the Jenkins, Docker, Kubernetes in my position description so this HR guy gave me a Devops interview. I thought no shit I will give it a try, if I fail anyway I should just see what are the options out in the market. So we had a two hours interview, it's a big company with 30 offices around the world, they said currently they do a lot of tasks manually and they need a dedicated guy who will only do Devops. I explained them what I am doing exactly in my current position, and for my surprise they liked it. I even told them do you guys want some Devops guy who are very experienced, or would it be okay if I learn on the job. They said they are still interested in someone who are like me, strong developer background and transition into Devops.

I looked into the K8s configurations, I wouldn't be able to set these up from scratch right now, just only if I deep dive into it. With Docker and Docker composer I am quite comfortable, I also wrote a slick Jenkins script that does all the integration testing and deploy to the k8s. But to be honest I don't know what the fuk I am doing for why. As I said I am developer who was given these infrastructure and started gaining experience these ways. But when I check AWS certification question, I am like what the fuck am I reading. So I have doubts that I should go into a Devops position just because I modified a K8s configuration and I knew how to setup a docker-compose.yml

What do you guys think, would I be suffering if I accept this job? I have one month notice in my current company, the package is nice and I thought maybe I would study for the AWS Certified DevOps Engineer in this one month if I accept the offer so I don't get into a job like a complete idiot. \\

What do you guys think?

https://redd.it/u8jzel
@r_devops