Operational stability from a data perspective.

At my company, we have a DevOps team that's applying the 3 ways of DevOps to our monolithic architecture. They've successfully achieved daily deploys, sensible change management, and implemented several processes to quickly recover from disastrous situations.

Unfortunately, the same care hasn't been given to other parts of our ecosystem. Specifically, our data architecture is kind of a mess right now. Our current architecture has already started fires that were tough and mind-numbing to put out. But in the midst of the chaos, emerged a data pipeline that was golden from a DevOps perspective.

I wrote an article which talks about how to build such pipelines: [https://medium.com/@talham7391/structuring-a-robust-data-pipeline-24ff67783782](https://medium.com/@talham7391/structuring-a-robust-data-pipeline-24ff67783782)

Here are some of the main points:

\- In order to build good data pipelines, data can't be an afterthought when designing systems that will inevitably generate data. Such practices will result in ad hoc, hard to test, and brittle data pipelines that will break.

\- If your company is shifting from monolithic to service-oriented architecture, you might be heading towards Data Armaggedon - a place where everyone is freaking out because data is disorganized and hard to access. Make sure you think about your data needs when making the shift.

\- Data pipelines, if they are critical to the business, should be treated the same as other critical services. Namely, you need to make sure you think about testing, automated deployments, observability, quick recovery mechanisms, etc.. etc...

If you're liking what you've read so far, I think you'll like the article! I go into more detail and I also talk about certain properties data pipelines should have that will make them a joy to work with.

https://redd.it/fczq6o
@r_devops

K8s terms for any newbies looking to learn

Came across this list. Solid overall. Good for anyone new to DevOps or K8s looking for a quick understanding of some industry terms. Missing a few things like Kube-proxy, DeamonSet, toleration & PSP tho.

​

[https://www.portshift.io/blog/ultimate-devops-devsecops-kubernetes/](https://www.portshift.io/blog/ultimate-devops-devsecops-kubernetes/)

https://redd.it/fdu62z
@r_devops

An argument for building event driven microservices and avoiding distributed monoliths!!

Based on my relatively short experience of working with [Space Cloud](https://github.com/spaceuptech/space-cloud) and microservices, i quickly realized what Kelsey Hightower meant by _distributed monoliths_.

Here's my two cents on how you can overcome that problem by going event driven. https://medium.com/spaceuptech/the-right-way-of-making-microservices-communicate-b6cd517ae702

Would love to hear your thoughts and experience! All feedback is welcomed!

https://redd.it/fdvz6u
@r_devops

Kubernetes clicked when I learned about Deployments and Services

https://ljvmiranda921.github.io/notebook/2020/01/18/kubernetes-deployments/

https://redd.it/fdykn8
@r_devops

Former webhosting guy wanting to get into Devops after several years afk. Advice? Feeling knowledge is a mile wide but an inch deep.

HI folks. I have been doing data center technician work for several years and am looking to switch back to a Linux oriented job, more specifically devops. I originally was a network engineer for a major telecom after going through a ccna internship. I used that experience to get work at a small webhosting company several years ago and was trained to basically run it for the owner. It was a very simple and easy to manage and never got to expand knowledge past basic wp/ static site hosting (apache,bind9,postfix,etc..). After that I worked as a jr devops engineer for about two months before being fired for essentially not being able to keep up. I felt discouraged and went back to basically plugging stuff in for various datacenters for about 3 years.

I am at a point where I recognize my lacking skills but have immense ability to improve. I am unemployed currently and would like to do something in the next 2 months to get me on track to have a solid career in devops/ system engineering. I have considered going back to school for CS but am not prepared to do 4-5 years while working fulltime.

I am considering the LPIC path which I think I could at the very least knock out LPIC1 in the next 3 months. Then get a devops specific cert.

Does this sound like a good plan? Can I land a job with an LPIC and limited experience from several years ago, within the year? Would it be better to just brush up on linux learn python and go straight for a devops cert?

TLDR; Can I get a job with one or two of these certs? Should I just go for a devops cert right away?
https://www.lpi.org/our-certifications/lpic-1-overview
https://www.lpi.org/our-certifications/exam-701-objectives

https://redd.it/fe0i8u
@r_devops

CI/CD for Java Dropwizard + React app

My application is composed of three parts: a React frontend, Java Dropwizard REST service, and a Java scheduler (Quartz).

I was thinking of using Azure DevOps to deploy everything so I could have CI/CD, but they don't have Dropwizard as a framework and less customization.

Was wondering if anyone has any suggestions on what to use in this case? Should I use Docker?

https://redd.it/fe07rk
@r_devops

Jenkinsfile SCM best practice

Hi. We are planning to make a switch from TeamCity to the Jenkins. One of the most interesting features to us are Pipelines/ aka Jenkinsfiles.

I would like to hear how do you organize Jenkins in terms of scm? Do you have one repo containing multiple jenkinsfiles stored per project, or you have single Jenkinsfile within source code of project ( separate repo)?

Thank you for your opinions!

https://redd.it/fe1dzc
@r_devops

Best Way To Monitor Developer Environment on TV's?

Hi. So here at the office, we'd like to have some developer stats on the TV's in the office, so people can keep track of things and whatnot. We currently use Dashkiosk on pretty much 1 or 2 TV's right now to essentially test this out and have run into some HTML5 bugs with it. We have recent Vizio TV's, so they have an "OK" web browser built-in that we'd prefer to use over buying a dozen or so Intel Compute sticks. Is there a plugin for Dashkiosk that would resolve our HTML5 problems, or should we be using something else entirely, like DashPlay or another program?

https://redd.it/fe18fa
@r_devops

Pros/Cons: Containers vs immutable virtual machine images

Hi /r/DevOps,

I just started a new gig and find myself in a position where I need to pitch a pros/cons list of building a microservices application using containers or virtual machine images.

I'm familiar with both and have used them both to great success, but I have to convince management that containers are purpose built for our architecture. It's more or less a mix of python, Java, and node apps.

We're going to have about 70 total services, and if we follow the pattern of 1 service to one container/one image, it's a no brainer to me that containers (either on swarm or k8s) will win out in terms of operational overhead and cost.

How would you present pros/cons of each approach?

https://redd.it/fe4y75
@r_devops

Just got an offer as a Devops team lead, now what?

I have been escalating real quick In my career, I came out of college 4 years ago and now I got in a pretty good company by working 1 year in each of the following roles

Software developer > Scrum master > Profesional services consultant > Sysadmin > Devops team lead

This is my first time a as a team lead I might not be SR enough as many people here but you can count my experience x2. I'm really dedicated and work really hard from the beginning to the end of the day

I will be in charge of a team of 3 people working together with other devops teams in other countries but to be fair I'm a bit clueless where to start in this new role any advice is welcome especially in the "lead engineer" part

https://redd.it/fe99fw
@r_devops

Python For DevOps: Learn Ruthlessly Effective Automation

Has anyone had the pleasure of reading this book? I was wondering what people thought of it

​

[https://www.oreilly.com/library/view/python-for-devops/9781492057680/](https://www.oreilly.com/library/view/python-for-devops/9781492057680/)

https://redd.it/fdv7vv
@r_devops

Python script for bulk find and replace in multiple files in multiple git Repositories

This script is useful when you want to change multiple part of multiple files in multiple git repositories in bulk. Script is creating a new branch from the given base branch and committing changes into it. Maybe push to remote also can be added with small change.

BTW, any feedback would be much appreciated.

I will drop this here maybe it can be helpful for someone else:

[Medium Post](https://itnext.io/bulk-find-and-replace-in-multiple-files-in-multiple-git-repositories-487c7c433b75)

[Github Repo](https://github.com/omerkarabacak/bulk-find-and-replace-in-git-repositories)

https://redd.it/fdxt9a
@r_devops

Authfailure: AWS was not able to validate the provided access credentials

I am trying to create my Gitlab CI/CD pipeline with AWS. The goal is to Terminate the Existing EC2 Instance, Run a new instance from a template, then Associate an Elastic IP to the new EC2. The runner I'm using is a Docker runner using the python:latest image. When I run my CI/CD pipeline I get

An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials My .gitlab-ci.yml is as follows:
```
stages:
- build
AWS_Install:
image: python:latest
stage: build
tags:
- Docker
script:
- pip install awscli
- export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
- export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
- export AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION
- echo "running script :)"
- OLDEC2=$(aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" --query "Reservations[*].Instances[*].[InstanceId]" --output text)
- aws ec2 terminate-instances --instance-ids "$OLDEC2"
- sleep 200.0
- aws ec2 run-instances --launch-template LaunchTemplateId=[launch-template-id],Version=12
- sleep 120.0
- NEWEC2=$(aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" --query "Reservations[*].Instances[*].[InstanceId]" --output text)
- aws ec2 associate-address --allocation-id [allocation-id] --instance-id "$NEWEC2" --allow-reassociation
```
```
What I've checked/tried:
- AWS credentials: They are correct and valid
- aws configure: Everything sets correctly (checked using aws configure get)
- Ensured UNIX line endings were being used
- Adding a variable section to the YAML file to include environment variables
- Hardcoding credential values
- New user on AWS with all necessary credentials
- Using export to get the environment variables
- Running everything in one script rather than having a before script
- Having multiple stages/Jobs
```

https://redd.it/fdwxo3
@r_devops

Company would like to have a call to make a formal offer to start working, and answer any questions you may have at this point of the process.

I don't really want to have a video call and have to answer straight in the moment what I think, how much I want, etc.

I'm a much stronger negotiator when it's all written and digital. Any tips on how to handle this?

Should I e-mail back:

>Is it possible to get the offer in writing before having the discussion tomorrow?

I feel it's a strategic move to talk me down financially. I'm definitely looking to maximize the offer and I'm quite introverted, so this would make me feel uncomfortable.

https://redd.it/fdwel5
@r_devops

What does a typical raise/bonus look like after 1 year at your company?

I’m pretty new to devops field and young so i was wondering what i should i expect and if what i got was too low or just right.

https://redd.it/fdvr7l
@r_devops

When to use devops? (noob question)

Hi All,

I' trying to learn about DevOps and have a very basic understanding about the concept around it. However, I cannot wrap my head around where exactly DevOps principles can be applied in a process flow. Hoping to pick all your brains.
Lets say I have a process flow that does the following for the organization I work for:
-inbound files are received in my organization that contain information that need to eventually go into our back end (lets pretend a purchase order from a customer).
-files then need to be 'massaged' and reformatted so our back end can read it. I assume at this point I can create a service using DevOps tools to accomplish this.
-Then send file across to back end application (like netsuite, SAP, etc.).

So in the end, I would really be applying DevOps to one portion of my project. Does that sound right? I know this is a vague example. I am just trying to understand where you would use DevOps in a project.

https://redd.it/fduwpz
@r_devops

Alerting - an overlooked support tool

Found this piece on alerting useful, thought I'd share it for others. Check [it out here](https://caylent.com/alerting-handling-on-call-management-and-notifications).

https://redd.it/fed354
@r_devops

Use KubeEdge to Collect the value of the Temperature Sensor connected to the RaspberryPi to Cloud



KubeEdge is an open source system extending native containerized application orchestration and device management to hosts at the Edge. It is built upon Kubernetes and provides core infrastructure support for networking, application deployment and metadata synchronization between cloud and edge. It also supports **MQTT** and allows developers to author custom logic and enable resource constrained device communication at the Edge.

Using the device management capabilities of KubeEdge, we can use the Raspberry Pi on the edge to transfer the value of the temperature sensor to the cloud.

A application called **Temperature Mapper** contains code to collect temperature from a temperature sensor which connected to a raspberry Pi through gpio. Depending on the expected state of the temperature sensor, the program collect the temperature.

The details for how to setup: [https://github.com/kubeedge/examples/tree/master/kubeedge-temperature-demo](https://github.com/kubeedge/examples/tree/master/kubeedge-temperature-demo)

[https://github.com/kubeedge/kubeedge](https://github.com/kubeedge/kubeedge)

https://redd.it/fdrd6x
@r_devops

Tell me about your artifact repository

Where is located? or what service do you use?
What tool do you use for ci?
How do you connect to it from ci tool, dev machines?

Small java shop with old and locked down on prem nexus repository and jenkins.
We dont have time to keep these updated, operational and they are running in old expensive setup.
We want to take advantage of the services available from different ci tools, but are in doubt what the best path forward is.

https://redd.it/fees6p
@r_devops

Most frecuent devops tasks

Hi all,

What's the most popular or frecuent task/configuration/deployment that you make as devops role? and components or languajes that it included?

https://redd.it/fe9tpl
@r_devops

SREs of Reddit who don't tag their cloud resources, why don't you tag your resources?

I have also gad a phase where I didn't care much about tagging my resources but I managing resources became a very real headache very quick. I am trying to get some more perspective around why people don't tag their resources.

Is it because you are too lazy and tagging is a lot of effort? Do you not know what tags to use?

Follow up - people who have standard tagging policies at their workplace, what do these policies look like and how are they enforced?

https://redd.it/fe9bes
@r_devops