Jenkins Deleting ssh.exe!

Any job I run on one of my windows agents deletes my C:\\Program Files\\Git\\usr\\bin folder and thus my ssh.exe so I cannot clone a repo via ssh. I am baffled...anyone have any ideas?

https://redd.it/fafgtn
@r_devops

What should I look out for as a candidate in a devops job interview ?

I've got close to 2.5 yoe in technical support, I recently interviewed at a startup and the interview was way too easy. So I started having doubts if I'm doing the right thing?
The things that were asked very basic tcp/udp difference, one simple coding question, how to design a backup systems and some questions about my current role.

https://redd.it/fafu87
@r_devops

How to grow as an SRE/DevOps Engineer

This post is more so for exploring what I can do to grow as an SRE/DevOps Engineer.
A little about me: I have a B.S. in Computer Science and have moved into a DevOps/SRE role where I am writing a decent amount of code/scripts (python, bash), rolling out a decent bit of infrastructure (chef, cloud services) and troubleshooting application/network issues. I would consider myself on the above-average side of programming but the average to below-average side of Linux knowledge. I am very comfortable making my way around the Linux OS but struggle with some of the more in-depth troubleshooting/concepts from a knowledge standpoint.
The inspiration for this all is from interviewing for SRE positions at a couple of different companies and either making the on-site, pretty much acing the coding portion and then not passing through the architectural side, or rarely not making it to the on-site. The frustration for me is that I am given little to no feedback and struggle to find where to learn/gain experience that is needed to grow (large scale systems) and ultimately pass through all the interviews.
I generally have used Linux academy to learn but am constantly looking for more places to learn from.


Any insight/perspective/guidance/encouragement would be greatly appreciated.

https://redd.it/fakrm2
@r_devops

Lack of tooling that helps understand large codebases

**How does the code base work?**

There is a lack of good in-depth tolling for understanding big codebases for architects/team leads/developers. On one side we have simple tools for test coverage and on the other side, complex and expensive products like code climate focused on velocity.

But there is a problem left unsolved in the middle. As an engineer working in a bigger team, it’s hard to get an answer about how does part of codebase work. Or why & how & in what places is this piece of the system used.

Keep in mind that you can use static analysis and “find all” to find specific usage of a function but what I am interested in is connecting concepts/modules together and showing their relationships in easy to understand way.

**It’s a new decade and I still don’t have a way to have:**

* A clickable diagram explaining all subsystems of my app and their dependencies to one another
* A simple way to see the high-level flow for a particular feature. User registration code diagram for an example
* A simple way to see which parts of code are undertested / overtested. Code coverage is a bad metric in general. It does not tell you anything about the popularity/usage of tested code
* A way to have a good understanding of what side effects will I have to think about **before** starting the work on a feature. To measure scope and time needed. (Oh shit user registrations need to be exported to intercom...)

**What general questions should you be able to answer easily?**

* I want to change the functionality of this feature for new requirements. How can I know all other subsystems interacting with the feature?
* I want to quickly get a high-level overview of major subsystems. What are the dependencies, are there any hidden connections that people tend to forget? What should I watch out for?
* How can I easily know where does the feature starts and ends? I want to know how the card payments work. Where to start?
* How much effort will be needed to refactor this feature? Are there any connections/interloop that I should be careful about?

**Real-world example**

What parts of the codebase do Stripe payment gateway touch and how is it implemented at a high level? What do I need to watch out for when migrating to Braintree?

**Can it be better?**

It took me 4 months just to grasp the major subsystems and their interactions with each other in my company. There has to be a better way.

I understand that deep and comprehensive docs could be an answer, but really how many companies will do documentation at all levels of the tech stack? Usually what I see is that you have a few people at the company who have an understanding of the architecture and you kind of have to piece it all together from them.

Why can’t we automate at least part of this process? Something simple to give to the developers, so they can get a quick understanding of the features and their relations in the whole codebase.

https://redd.it/fah0jw
@r_devops

Looking for feedback from someone who is a user of Stackstorm / Rundeck

Hello, Just discovered stackstorm, but I want to know if someone can recomend which is better, rundeck/stackstorm or if them are pretty the same?

​

I know stackstorm is event-drivven but can also run scheduled simple tasks as rundeck? Also, can rundeck be event-triggered like stackstorm?

​

Thanks

https://redd.it/fakc5k
@r_devops

DevOps Roadmap & Me

I was given a really strange opportunity because of my powershell automation experience to be part of an amazing company, and I was wondering besides what’s on the monthly thread and the Roadmap:

What can I learn that will ensure I am successful meshing with my team?

I’ve wiped my PCs and servers and started running Linux only, learning bash.
I’ve started AWS and GCP free credits and am about to start learning K8s on AWS and go from there...

The employer knew I had no experience but they were confident in my ability to learn fast and enjoyed all the servers I have at home that I self learn with.

—

Any advice and encouragement is really appreciated.

https://redd.it/fanf2m
@r_devops

Azure Devops Agent installation (linux) – need help/advise

Hi we currently making use of Azure devops and have the agent deploy to a number of windows servers and everything works great. We make use of the install flags to configure it to run as a service and use a service account which has permissions to do what is required (ie local admin).

We have the need to install it on a linux machine, however the documentation is a little lacking. Firstly it seems to run as the user which configured the agent (ie my user account) and looking thought the documentation there seems to be no flag to change who the agent runs as.

I am looking for some advice on how best to install this agent. Should I install / run it as root or a dedicated account. If a dedicated account how do we deal with permissions (ie I do not see any options in pipeline to say sudo this task.

Any advice/recommendations would be great thanks

https://redd.it/fam2dq
@r_devops

What's your email QA process?

I run a small SaaS business which sends out report emails every day. And I haven't come up with a good process for email QA.

Last night we sent a blast of badly formatted emails to our subscribers, although after we did some testing on our server. The emails were badly formatted because we did not handle some corner cases. We checked some of the test sent emails, and they seemed to look good. But it turned out most did not.

Needless to say, this was embarrassing at best, churns at worst.

We need to set up some formal process for email testing to ensure this won't happen again (hopefully).

Here is what we think could work:

1. clone a testing VPS from the production server, with all the data and code;
2. update the code on the testing VPS with our latest changes;
3. redirect all the emails to some test email address which we will examine later;
4. run the code, check the test emails, make sure they are good;
5. push the code changes to the production server;

Care to share your process?

BTW: email QA is different than otherwise like dashboard, website, etc. Once you send a poor email message by accident, it stays in the user's inbox forever. There is nothing you can do about it. You can't recall it. Every bad email sent to our customer feels like a punch in the gut to me.

Updated: my main goal here is to implement a solid email QA process to get rid of manual testing as much as possible. Manual testing sucks most of the time.

Thanks good people.

https://redd.it/faof7d
@r_devops

Firewalling a docker container?

I have been given a system where DokuWiki lives in a docker container on an Unraid server. An nginx proxy handles directing what little internet traffic reaches the Unraid server. Notably, for the Wiki, you can only access it via port 80 (you can't ssh in or anything unless you RDP to a Windows machine from which you can SSH in).

The problem here is, the wiki holds sensitive data, and folks want it locked down completely from Internet traffic, with only a one or two external IPs which can get through. All employees are remote and there's no VPN for several reasons, so this is those employees dedicated static IPs.

Now, the question is: What would *you* do block port 80 traffic and only allow certain IPs? Should I just modify something within the nginx configuration?

Another option I have is simply to ditch docker and make a tiny VM, then write a few firewall rules in there. Anyways, thoughts?

https://redd.it/falyxx
@r_devops

Career in devops

Hello guys, I working in it field from last 3 years mostly in help desk positions. Right now due to some personal issues I cannot work for 5-6 months so I thought to study and complete some certification to get better job in cloud. I have basic knowledge of programming languages such as java, c#, html and css. My question is that what would be the path to get my career start in cloud and move to devops? Do I need to learn programming if yes than which languages? Should I give aws solutions architect ? Please guide me.


Thanks in advance

https://redd.it/falka0
@r_devops

Struggling trying to get my local Kafka messager project uploaded in the cloud using Aiven. Can anyone help?



https://redd.it/falhyl
@r_devops

Gitlab CI: feel like I'm going crazy, how do I build a docker Image?

I'd like to just build a Docker image for a node.js app. The Dockerfile builds on my computer. I have gone through the Gitlab CI Docker documentation and it feels like it's taking me to the stars and back but I'm having difficulty understanding whether I need to register a runner or where to get started. (https://docs.gitlab.com/ee/ci/docker/using_docker_build.html)

My assumption was that it would be something like the following. The build-node stage is not exactly what I'm using but it is working correctly -- there are also testing and lint stages that are working, too. The docker image build is where I'm tripping up.

//gitlab-ci.yml

build-node:
stage: build
before_script:
- yarn install --ignore-engines --frozen-lockfile
script:
yarn build

build-docker:
stage: dockerize
image: some-standard-gitlab-docker-build-image
script:
- docker build --rm -f ./docker/Dockerfile -t my-app:latest .

https://redd.it/falcop
@r_devops

'grep' to find how many users are in a group number?

I am having trouble trying to use the grep command to find how many users are in a GID...

Example: how many users are in group 101?

https://redd.it/fagb0h
@r_devops

How to get better at searching within JIRA

We painstakingly document a fix for issues with full screenshots, comments, etc. But when it comes to search, I can clearly see that JIRA fails even the most basic searches. We are using the cloud JIRA BTW, which is painfully slow to load on top of other things.

Is there a better way to search for keywords or groups of keywords? is there a way to use google to search within our JIRA cloud? Confluence is slightly better at search, but JIRA is the worst.

https://redd.it/faj9zf
@r_devops

Gateway with throttling/rate limiting - help us decide!

We're building a SaaS solution and have recently moved past MVP phase. Our customer base grows and we have noticed that some of them are abusing/overloading the system, which of course leads to "noisy neighbors" type of problems.

To overcome the issues, I have been assigned to scout out the throttling possibilites. However, our stuff runs in AWS and so far we have been using AWS ALB as "gateway" (it has various rules so that `/api/service1/` is handled by `service_1` application running on ECS, `/api/service2` goes to `service_2` and so on. Internal communication between services is handled by AWS App Mesh - Envoy in disguise). Since we did not need a real gateway when building MVP, load balancer worked just fine.

This has to change however, due to throttling/rate limiting requirement. Natural move would be replacing AWS ALB with Amazon API Gateway. That would come with a hefty price and I thought of adding our gateway between the load balancer and microservices instead.

The question is: **What are your best practices/go-to technologies when it comes to rate limitng/throttling the requests?**

I've looked into solutions and Nginx, HAProxy or Zuul(our apps are Kotlin-based so JVM technology like Zuul could fit in just fine) popped up. Which one would you recommend or avoid? Zuul seems too big for now, while Nginx or HAProxy could be not developer-friendly (our team has no classic SysOps - all the infrastructural work is done by Sofware Developers/SRE utilising Terraform/AWS CDK), but maybe that's just a wrong feeling. We are not afraid of any technology. :P

The throttle/rate limitng mechanism should work based on header with `tenant-key`. I'd also love to have different limit for different endpoints.

Thank you for your time!

https://redd.it/fastey
@r_devops

metrics for docker performance on different linux flavors

Curious if anyone has any experience gathering metrics or gauging how docker performs on certain distros of linux. I wanted to install a few different linux flavors on the same machine, probably ubuntu server, debian, and centos, and spin up some containers and I was wondering if anyone knows how a way to compare the different distros. As in what metrics would you gather to say that docker runs better on debian than ubuntu, or better on ubuntu than centos. How would you gather these metrics?

https://redd.it/faiwep
@r_devops

Quantifying business impact of my role

I got a disappointing performance review, despite my boss, teammates, and practically every dev team recognizing my work. However, the company calculator seems to value putting out fires rather than preventing them. For goal-setting, I want to know what kind of Key Performance Indicators (KPIs) that I can measure from my work to show what I'm actually doing.

Right now 2/3 of my team has been conscripted to project work, and I am basically one of two people dealing with all the support requests and automation improvements. I deal with the CICD pipeline and managing a lot of cloud-based resources.

What should I be measuring?

https://redd.it/fahw07
@r_devops

Including Mainframe in CI/CD with Zowe CLI

What do you think about using the Zowe open source framework to incorporate mainframe into enterprise CI/CD toolchains? The Zowe CLI is a lot like CLIs for AWS, Azure, K8s, etc. Here's a [simple CI example using Jenkins](https://medium.com/zowe/continuous-integration-for-a-mainframe-app-800657e84e96).

The mainframe frequency is not the same as cloud, mobile, etc. but curious to hear your thoughts.

https://redd.it/fafr5h
@r_devops

The woes of Wix, and how client lost all subdomains!

The client decided it was time to spruce up the old image by getting a new website to replace the old WordPress site. Now this is nothing unusual - but what is unusual is having one of the board-members do it. A board-member with somewhat limited technical insight…

How hard can it be, amirite?! He went forth with audacity and bravado. And indeed, the end result wasn’t bad as such. Video, animations, nicely organized. The advantage of tools like Wix - it makes cookie-cutter webdesign quite simple and easy. Anyone can do it, and indeed anyone did.

The problems started when going live. This fresh web-dev found that with Wix, it is not as easy as just updating your DNS to whatever VM is hosting the generated website. Oh no. Wix wants to be the DNS SOA (Start of Authority) and NS (Name Server) provider for the already existing domain. A domain I might add that was in production, with a host of services attached as sub-domains.

#### The disconnect

DNS SOA changed over, happy green web-dev cum board-member proudly showed his work - it going live on a Sunday evening… When exactly zero people are at work just in case, you know, something should happen. And it did. It soon transpired that the link to the subdomain that host the webapp suddenly stopped working. As did the logging server, the metrics server, the gitlab server. CI/CD pipeline, Prod and test environments both. In short: Everything. The kit and kaboodle. It all ground to a screeching halt. Except the website - that worked very well. Turns out, if you change DNS SOA and NS - any registered subdomains done by the previous DNS SOA and NS don’t carry over. They are wiped.

#### Tried turning it off and on again?

The technical staff having had no involvement thus far in this comedy of errors, soon got distracted from whatever they were doing at nearly nine o’clock on a Sunday evening. The problem soon got triaged down to one of three possible items - based merely on behaviour, as at this point in time what changes were made were not known. Indeed, it was not known that any change had taken place to begin with. Actual access to the relevant consoles and dashboards of various services was not possible outside the office, so was essentially working blind. Anyhow, the shortlist was: DNS error, Server-App error or TLS/SSL Certificates expired error. The two latter got eliminated off the list in short order, and what one was left with was the first. Which was top of the agendum in the wee hours of the morning after.

What was done though - at once - was quickly set up a temporary DNS link for the client facing web-app, so that at least the customers could get to it. Hence there was little customer facing down-time. Perhaps an hour, at a time of day with very little traffic on the service provided.

Fixing the issue itself was not a problem as such. Once access was had early next morning, the DNS SOA and NS got set back to the original provider. This takes a while to propogate around the globe, so the time after was spent hunting for the original web-site. None working there were involved in the making or deploying the original WordPress website - so no-one got a clue where it is hosted. So for the timebeing, the domain was set to point to a silly little 404 page. That is, until Tuesday rolled by - and a quick and dirty new site was created in Hugo by yours truly, pushed onto Netlify and is now running live. This bare-bones site will work as the foundation to build more onto. In a proper way, and not using managed tools like Wix.

#### The end and lessons learned

First off, don’t leave the keys to the kingdom in the hands of anyone who do not know how to operate the realm. No permanent damage done luckily, just created a bit of excitement and activity. Oh and postponed already scheduled tasks by a couple of days.

Moving forwards, I think CEO and Board-Member both have a new found appreciation of my refusing to deploy to production policy on Fridays, Weekends and day before any national Holiday. I

f something goes sideways, one need time to fix it within normal working hours and normal working days. And of course, I do not think they’ll meddle in technical stuff again without first consulting with the people possessing the proper expertise.

Oh, and a little bit of knowledge is a very dangerous thing, but we already knew that… :)

https://redd.it/fa9274
@r_devops

Have anyone tested Azure DevOps + Wordpress integration?

Hello guys,

I'm currently learning & working on making our wordpress site a DevOps friendly environment. I've been trying to integrate Azure DevOps with our Wordpress site.

Here's what I'm thinking of doing:

I will have to init a git on my Kinsta hosting via SSH.

Then connect the repository to DevOps Azure.

​

Do you guys have any experience integrating Wordpress with DevOps? If yes, what are your suggestion?

I have experience with Gitlab. Looking for the suggestion of the wiser & more experienced one. THank you.

https://redd.it/fadew4
@r_devops