Incident Write-ups

I'd like to share my insights on how to document an incident in preparation for a post-mortem!

https://certomodo.substack.com/p/incident-write-ups?sd=pf

(cross-posted from r/SRE)

https://redd.it/13fu6wy
@r_devops

Looking for a Windows utility to recursively search and replace text

We are moving a site in IIS to another server, and for testing we need to use a second URL as we cannot take the first site down. The new site has a login field that is currently failing, and we think it’s due to the way the login is potentially referencing the old URL somewhere for the return traffic. We found some hard coded references to the site name in some custom files already, but there are too many to search by hand. The person that made the site is no longer with the company and no other web developers, so by process of elimination I’m the SME. Can anyone recommend a method or utility in Windows 2019 that can recursively search a folder and file contents, and find and replace (or list) all specific instances of a search word or phrase? I told the network guy that he broke it, so I have a few days before he works it out.

Thanks!

https://redd.it/13ftur7
@r_devops

Is My Workload Normal

Hey everyone,

I wanted to share my current experience and workload in a new position I've held for about four months now. I've been in DevOps for about four years, and in my last role, I was part of a two-man team that migrated a multi-billion-dollar company from on-prem to AWS within a year, alongside some straightforward DevOps tasks. So, I'm no stranger to responsibility. However, my current role has left me questioning whether the expectations placed on me are typical for a DevOps role, or if I'm being stretched too thin.

Currently, I wear multiple hats: I'm the lead architect for an Azure migration project, responsible for implementing DevSecOps, and in charge of both the development and operations (infrastructure) teams for DevOps implementation.

Additionally, I've been assigned tasks that veer into the realm of a security architect rather than a traditional DevOps engineer. Here's a snapshot of what that entails:

- Integrating data privacy, industry regulations, and PCI impact assessments into the SDLC.
- Incorporating a standard set of security, privacy, and industry compliance requirements into every design.
- Defining a security component library for approved components and calls.
- Maintaining a common data catalog and defining data classification for each data item.
- Leveraging threat models to inform application security requirements and security architecture.
- Including mobile security requirements in designs.
- Implementing and managing various automated security tests, static application security testing (SAST), and dynamic application security testing (DAST) in the testing environment.
- Running denial of service and security stress testing before production deployment.
- Monitoring file integrity and handling reports of security vulnerabilities from end-users.

And the list goes on, with a variety of security testing and scanning procedures.

All these tasks are on top of my responsibilities as the lead architect for the Azure migration and my regular work as a DevOps engineer. The cherry on top? The deadline for these projects is set for November.

So, I'm reaching out to this community to understand if my experience is common. Are these expectations reasonable, or am I being spread too thin? I'd love to hear your thoughts and any advice you might have.

Thanks in advance for your insights!

https://redd.it/13fwwwd
@r_devops

Are these two equivalents and when should you use one over the other?

docker run -d --name sql-container --network mydockernetwork --restart always -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=pass' -e 'MYSSQL_PID=Express' -p 1433:1433 mcr.microsoft.com/mssql/server:2017-latest-ubuntu

Is there any advantage in running a docker command like above instead of doing it like this:


version: '3'
services:
sql-server:
image: mcr.microsoft.com/mssql/server:2017-latest-ubuntu
container_name: sql-container
environment:
ACCEPT_EULA: Y
SA_PASSWORD: pass
MSSQL_PID: Express
ports:
- "1433:1433"
restart: always
networks:
- mydockernetwork
networks:
mydockernetwork:

https://redd.it/13fy5z2
@r_devops

Best practices for CICD process

We are a spring boot shop for the most part. Can any of you good people point me towards an article or a book that details a good CI process?

It, best way to deal with PR, how much validation should be done at the PR vs post merge? We keep updating our process ad hoc, looking for something to compare it too or use for reference.

I have spent considerable time online and this question is surprisingly hard to answer via google.

Thank you in advance.

https://redd.it/13fy776
@r_devops

The Cloud Native Playground

The Cloud Native Playground is in public beta - https://play.meshery.io. If you’d like to explore the Cloud Native Computing Foundation’s graduated, incubation, and sandbox projects (as well as many other open source projects), and you’re willing to share feedback on your experience (particularly around the multi-player visual Kubernetes designer), submit the form on the page and we’ll see to getting you access (this playground runs on a live cluster in which you can deploy your infrastructure designs, hence the need to sign up).

https://redd.it/13fzrod
@r_devops

Testing terraform code

My organization is starting to come up with more complicated terraform code that includes lots of conditional logic and dynamic blocks and stuff. What's the best way to perform automated testing to make sure it will render the resources that I want? I was looking at terratest, but it actually creates resources in the account and I think that's a little more burdensome than I want. Ideally it would be just something that runs terraform plan with a couple different sets of inputs and compares the output to a set of rules.

https://redd.it/13g3vug
@r_devops

Fluent-bit request repeated too quickly

I'm experimenting with Fluent-Bit as a logging alternative to Splunk UF. The initial goal is to load this onto a Docker container, and ship the logs generating by the app to Splunk.

It's not a perfect method, and we're slowly migrating towards containers for some of our applications. So... you know. Baby steps.

I'm testing Fluent-Bit out on an AWS V2 OS / Server. However every time I start the application I get the following error:

start request repeated too quickly.
unit fluent-bit.service entered failed state.
Failed to start Fluent Bit.

I'm not sure why starting up the app fails. Hoping someone can point me in the right direction to solve this issue.

https://redd.it/13fz7ds
@r_devops

Filtering metrics by labels in OpenTelemetry Collector

https://last9.io/blog/filtering-metrics-by-labels-in-opentelemetry-collector/

https://redd.it/13ftjz7
@r_devops

Orangescrum not installling in Centos 7?

The official tutorial for automated installation of centos 7 doesn't work. So, I had to refer to online tutorials.

I installed apache 2.4,php 7.2, mysql 5.7.
Then I downloaded .zip source code from here

https://github.com/Orangescrum/orangescrum.
Then, I setup some permissions and ownership as per documentation.

$ sudo mv ~/orangescrum-main /var/www/html && sudo chown root:root -R /var/www/html
$ sudo chmod -R 0777 /var/www/html/orangescrum-main/{app/Config,app/tmp,app/webroot}

Then I setup a virtual host orangescrum.conf in /etc/httpd/conf.d.

<VirtualHost :80>
ServerName orangescrum.example.com
DocumentRoot /var/www/html/orangescrum-main
<Directory /var/www/html/orangescrum-main>
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Order allow,deny
allow from all
</Directory>
</VirtualHost>


Then I created mysql database

CREATE DATABASE orangescrum;
CREATE USER 'orangescrumuser'@'localhost' IDENTIFIED BY 'yourpassword';
GRANT ALL PRIVILEGES ON orangescrum. TO 'orangescrumuser'@'localhost' IDENTIFIED BY 'yourpassword' WITH GRANT OPTION;
FLUSH PRIVILEGES;
EXIT;

Then I imported the database.

$ mysql -u root -p orangescrum < /var/www/html/orangescrum-main/database.sql

Then I updated orangescrum credentials inside /var/www/html/orangescrum-1.6.1/app/Config/database.php

'login' => 'orangescrumuser',
'password' => 'yourpassword',
'database' => 'orangescrum',

But it's not working and I'm getting the below output when I enter my ip address in browser.

![image1]1


How do I fix this issue? Is there a way?

1: https://i.stack.imgur.com/7FQg1.png

https://redd.it/13g8z3n
@r_devops

What are industry practices for how to trigger Cloud Stack tools?

We are starting to get into more complicated use cases for Ansible, terraform, and Cloud Formation with AWS. We started out by using terraform and triggering it from within the AWS Console.

However, that began to become very tedious. Other teams we have noticed are using answerable playbooks from within their get hub CI CD actions. That seems like a pretty nifty way to always pair your cloud setup and configuration with the underlying code.

Are there other industry practices that people use to make cloud resource set up and configuring more manageable?

https://redd.it/13ftdvz
@r_devops

mssql: Error 10054: A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)

mssql: Error 10054: A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)

docker run -d --name sql-container --network mydockernetwork --restart always -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=pass' -e 'MYSSQL_PID=Express' -p 1433:1433 mcr.microsoft.com/mssql/server:2017-latest-ubuntu

Cannot connect to mssql in localhost using VSCode mssql plugin. I use sa as username and pass as password, and localhost as name and <blank> as db name, but it gives me the above error message. Not sure if something is wrong with the image.

https://redd.it/13g11cr
@r_devops

Do you have a personal portfolio website?

I wonder if it's the same for Devops as it is for software developers. Do you have a personal website? Did it help you advertise yourself and find a job? Do you have suggestions on how to achieve it easily and for free?

https://redd.it/13gdwfr
@r_devops

Any AWS networking visualization tool?

Basically something that shows me if something in VPC X can reach something in VPC Y, in a specific port if necessary. It needs to be aware of:

- Subnets
- Security Groups
- Route Tables
- Transit Gateways
- Peering

Etc.

Everytime I'm architecting an app networking-wise on AWS I have to draw by hand what I'm trying to do and how to achieve it. It just seems absurd to me that something like this doesn't exist.

https://redd.it/13gducr
@r_devops

Throw in the key

In Swedish there is a saying that you're fast if "you can lock a drawer but manage have time to throw in the key".

If I create a AWS VPC with private subnets, how do I access them from Ansible if I lock down SSH to only local addresses using a Security Group? Do I have a local Ansible Control Host? Should I keep SSH open an all machines and give them public IP addresses although this is a huge security risk? Is there anyway to throw in the key while locking the drawer?

As a AWS+Terraform+Ansible newbie, I'm a little bit lost here, does anyone have advice or can you point me in the direction to go?

https://redd.it/13gjwn7
@r_devops

Where are the recipes for "Hybrid" Delivery?

Everyone talks about and uses recipes for Continuous Delivery.

Does anyone have any pointers to tooling or recipes for delivering complex systems in releases?

My team manages \~dozens of components that we do CI/CD into a staging environment, but release into production every 2 weeks, and we like it that way; we don't really want to move to continuous delivery to production.

Our current system is a set of scripts with a decade old sqlite database with Jenkins:

Devs trigger a build in Jenkins.
Jenkins deploys to staging via Ansible.
Jenkins then adds the artifacts to sqlite as "latest artifact deployed to staging as part of XXX release".
On release day, once all tickets are resolved, we run a script that pushes the latest artifacts for each project up to the production repo.
At this point devs can begin working on tasks for the next release. Before this point devs need to be working in a branch and not building changes for the following release into staging.
Release night I do database migrations and then run Ansible scripts that release the updated packages.

This is all implemented with custom built scripts.

I'd like to do some enhancements to this (adding docker containers as a deployable, for example. Integrating the database migrations which right now are done largely by hand). But ideally I'd like to follow some existing patterns or use tooling rather than custom building if possible. It seems like something that should be a well worn path, but all I ever see talked about is CI/CD.

Anyone have pointers to what I'm missing?

https://redd.it/13gl6t4
@r_devops

Is this a good CI/CD setup to build and deploy services to Kubernetes?

So at the company I'm working for we have many applications and services that we deploy to k8s clusters on GCP (GKE). We follow this approach:

We dockerize the application
Create a helm chart
Create a Google cloud build pipeline that gets triggered on commits to certain files on some branches, the pipeline steps:

1. Build docker image with a new tag and push it to the registry
2. Upgrade helm chart setting the image.tag variable of the service container with the new tag

The application code and helm chart are all present in the same Git repo. The issues I see with this approach:

You always need to trigger the whole pipeline and build new docker images even if you're just modifying the helm chart
We don't separate the versioning of application and the versioning of helm charts and we don't have a registry for helm charts
It's kinda messy to have helm, Dockerfile, and application code all in one repo

Now I'm still a junior DevOps engineer and I don't like the way things are but they say we don't have the resources now to do better. Anyways, I wanted to ask you guys what better practices could we have done and what other problems could this setup result in?

https://redd.it/13gjsz5
@r_devops

Work like balance as a devops engineer?

I’m a backend dev with 2+ yoe. I’m looking to transition to devops and one of my important criteria is WLB. So I just wanted to ask the experienced devops peeps here about the WLB in this field.

https://redd.it/13gmayq
@r_devops

Would you say that learning how to develop browser extensions is a useful skill in devops?

Just a random question I thought about today, noticed we use a lot of sites that we don't have much control over, figured learning JS may help in this regard (automation, make things easier for our team, etc) and it never hurts to have another programming language under one's belt.

Do any of you do this?

https://redd.it/13gq6gg
@r_devops

Migrating from Nexus to Harbor: A Comparison of Features and Easy Steps

In this article, I would like to share my experience of migrating from Nexus docker registry to Harbor and the steps involved. I will also explain some of the most useful features of Harbor by comparing them to Nexus and why you should consider using it:

https://mallakimahdi.wordpress.com/2023/05/13/migrating-from-nexus-to-harbor-a-comparison-of-features-and-easy-steps/

https://redd.it/13gpzk6
@r_devops

Got the $3000 bill on my AWS account

Hello guys,
I left my services open and I got the bill of $3000 bill on my mail id. so, will they make any legal actions if I don't pay the bills. There is no payment methods added so it can not be deducted automatically.

https://redd.it/13glmng
@r_devops