What is behind FB outage? DNS?

What is behind FB outage? DNS?

System is too fragile, even "best" engineers had hard time to mitigate an issue timely.

https://redd.it/q1pby7
@r_devops

Is a CNAME visible for web server?

I am bit too lazy to test that out

If I set a CNAME foo.baz.com for aux.baz.com will the webserver see the referer / source as foo.baz.com, or it will always be a aux.baz.com?

Specifically in nginx there is a `server_name` config parameter and I am not sure if I should handle it as both foo.baz.com and aux.baz.com, or aux.baz.com is enough

https://redd.it/q2i3bg
@r_devops

I built an API management tool that would log every request, automatically generate documentation, provide API health analytics and test your API in real-time.

Hey community!

For the past 10 years i have been running a development and design company. We’ve completed 100+ projects and most of them involved developing a mobile app or website with an API in the background. While developing APIs we noticed that a lot of time is “wasted” on non development related tasks like: writing and maintaining documentation, providing support to other developers, debugging user reports, trying to recreate user steps and similar. We also noticed that our clients never fully understood what APIs were, how they work, how they are being used and what data is being sent back and forth. When we would give our API to a third-party for integration no matter how good we prepare the documentation or try to make everything bulletproof they would still have questions, we would still have to provide support, they would still be confused. Recently we saw a huge increase in webhook popularity where we simply had no way of debugging or even seeing what data we received from the webhook itself.

Because of this we started developing Treblle. To solve our own problems. We simply could not find a solution that would allow us to do what we wanted and needed(please correct me if i’m wrong). There were a plethora of tools that log code based errors or tools that do automated periodical monitoring but nothing that would log every request, generate documentation, provide analytics and insights and similar.

I would really appreciate your feedback on our tool. If anyone is interested in giving it a go please let me know i can hook you up with the biggest package.

https://redd.it/q2j0bg
@r_devops

Started my first Devops job, but feel a bit overwhelmed 😅

Hello!

I started a few days ago as a Junior DevOps engineer.

For years and years I’ve been an application support engineer, but I dealt with AWS on a daily basis and knew programming specifically python and PHP rather well.

I do have experience with Terraform and deploying to a Ci/CD stream but it’s very limited, hence why I’m at a junior level.

My new company everyone has been super nice and helpful to me, but I’m still feeling a bit overwhelmed with everything.

Almost everyone in the company is at a senior level and there’s so much to learn!

I haven’t yet really been tasked with anything major, but I am a bit concerned I could be dropped in the deep end a bit! I’ve gone from being the most senior person technically in my last company to being the least technical here and just worried I won’t be good enough and that could even potentially lead to me losing the role.

I’ve been learning via cloud guru and various other services when I get free time to try and learn as much as I can!

Has anyone else felt like they have been in this position before and any tips or tricks to help?

https://redd.it/q2kouf
@r_devops

seeking to be developer rather than support production engineer

I'm currently working as support production engineer, my responsibility is implementing the software, and customer support for productions issues.

The technology that i use , are below : git , linux operation and have basics knowledge of database administration, also we are simply working with kubernetes and docker, one of my responsibility to prepare pipeline for the software , but its not what you think at all as we copying and pasting staffs😂.

We can say that i work as junior devops with more OPS as we don't write codes. I feel that the devops is a culture not a role

I have been graduating since one year before , and this is my first job, i feel that like dev things more than supporting, i don't want to be stacked as support , i think that the future is development, i talked to my manager that i would like to go to the dev teams but he doesn't like the idea , he said that he preferred me to stay as support engineer as it better than development.

My question, do you advise me to switch to dev, considering that I may leave the company, btw im afraid to leave because i don't have skills to be a developer ,now im learning now some staff related to dev.

Im really tired of thinking about this, and i need advises , or a roadmap developer that helps me to proceed and success 🙌.

Thanks all.

https://redd.it/q2li5p
@r_devops

Help with Rails app GitHub action.

Hello everyone, I’m noobie trying to write a GitHub action for a Rails app, I’m following different internet tutorial but nothing is working for me. The idea is make a self hosted app on a Droplet using GitHub action. The app is a basic Rails project with PG as database. I’m getting many errors on the way but now I’m stuck on this “FATAL: password authentication failed for user "postgres"”. I don’t know how to set the password on the action because in my local is working fine.

My current action:

name: Deploy

on:
push:
branches: [ main ]

jobs:
test:

runs-on: ubuntu-latest

services:
postgres:
image: postgres
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432

strategy:
matrix:
ruby-version: ['2.7.2']

steps:
- uses: actions/checkout@v2

- name: Set up Ruby
uses: ruby/setup-ruby@473e4d8fe5dd94ee328fdfca9f8c9c7afc9dae5e
with:
ruby-version: ${{ matrix.ruby-version }}
bundler-cache: true # runs 'bundle install' and caches installed gems automatically

- name: Install PostgreSQL client
run: |
sudo apt-get -yqq install libpq-dev

- name: Build App
env:
PGHOST: localhost
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
RAILS_ENV: test
run: |
bin/rails db:setup

- name: Run tests
env:
PGHOST: localhost
PGUSER: postgres
PGPASSWORD: postgres
RAILS_ENV: test
run: bundle exec rake

I appreciate your help. Thank you.

https://redd.it/q2p2ux
@r_devops

What questions would you ask someone hired as a devops trainer?

I think there are large differences between the day to day use of devops, and a solid understanding of how to train others in devops.

What would you ask a devops trainer during an interview, now that you know the principles and use them?

https://redd.it/q2q7sl
@r_devops

Gathering opinions on kafka management tools

Hey folks,

I'm working on a UI tool for managing apache kafka clusters and I wanna gather feedback from you on this topic.

The question is, do you use any UI management tools for kafka? If that's the case, could you name the tool(s) and tell me:

1. What nice features do they have important for you?
2. And what do they lack? What do you really miss in them?

Any feedback would be much appreciated. Thanks!

https://redd.it/q2qz6u
@r_devops

Grow from Senior DevOps to DevOps Lead

Hey,

it looks like my soft skills are very useful to manage the team and the head of engineering told me I'm really good at that. I'd like to become a Lead in this company and lead the team of 5 DevOps. However this is new role for me and I'm not sure if this is the path I want to grow further.

How management roles are better/worse than pure engineering roles? Especially salary wise? I still see myself growing salary wise, but I'm not sure I can get more by moving to more managing roles than to continue growing as Senior DevOps.

Any thoughts or real life examples could be really helpful!

https://redd.it/q2nhir
@r_devops

How do you manage JIRA tickets across different teams - support/devops/engineering?

Looking for experiences/suggestions on structuring JIRA for work that flows across multiple teams, i.e. support to devops to engineering.


External and internal customers report production issues and request work into a JIRA Service Desk project run by tier-1 support. Some things get resolved by tier-1 support but most require escalation to functional teams.


The DevOps JIRA project has become the dumping group for anything needing escalation from support. Outages, functional bugs, requests for new environments - it all ends up with DevOps to fix or escalate.


If DevOps need support from other team such as infrastructure or software engineering then DevOps create a new ticket with that team, linking to the old DevOps ticket which in turn links to the Support ticket.


This process results in loads of linked tickets, with each team having to click through lots of related tickets to try and find the latest updates. Also the DevOps tickets end up hanging around for ages waiting for the other functional teams to resolve theirs, which for things like functional bugs could take months or even years.


Somehow I need a cleaner way of managing work that ends up spanning multiple functional teams with different reporting lines.


Options seem to be:


1 - Having multiple queues in the Support JIRA Service Desk for the different teams, but this means each team has to look in two places for work - their own team JIRA project and the Support JIRA project;


2 - Having one combined "escalation" project for the functional teams to use, which is only a slight variation on (1);


3 - Moving JIRA tickets between projects, which avoids having lots of linked tickets but ends up with issues pinging back and forth between the different functional teams JIRA projects;


4 - Something else I can't think of....


How have other people solved this project? How do you handle JIRA tickets that end up spanning multiple functional teams for resolution?

https://redd.it/q2t6zm
@r_devops

Any good references for automating the buildout of DataDog dashboards? Preferably with Jenkins.

Thanks

https://redd.it/q2nur8
@r_devops

Overcoming Organizational Momentum

i wash hired help modernize the entire stack of a proprietary monolithic application based web presence for a state government department and they asked specifically to incorporate the devops tooling that's popular in the software development world since a huge majority of the organization's budge goes into software development. the tools are straight forward and path to getting it setup, technologically, is a little challenging because our vendors have never heard of this stuff and have legal protectionist government contracts against having to update anything.

however the biggest challenge by far is getting my colleagues and immediate management to understand how this works (upper management understands how this works and is forcing this upon my department). i've created functional ansible modules that are already useful and shared them with my colleagues; i've created example pipelines in gitlab that show how this is supposed to work to the software developers; i've created proof of concept environments showing how much easier it is to us these tools compared to the very legacy procedures that they're used to.

despite all of it; there's no improvements to my department's understanding. I can see NO ONE using the ansible modules during collaboration times during regular maintenance or outages; NO ONE has logged into gitlab to inspect the pipelines in almost a year now; and the proof of concept is just sitting there with no plans to expand it beyond proof of concept even though upper management loves it.

nearly all of my colleagues are close to retirement age and have been working in this department for 15+ years; am i wrong in thinking that's why they don't want to update? is there anything else i can try? does this spell doom for my future with this organization?

i suspect that if it keep my mouth shut and just collect a paycheck i could stay there until i retire like the rest of these guys.

https://redd.it/q2n0kb
@r_devops

Thoughts on Telepresence for local development integration in K8s?

I’m curious to hear people’s opinions and/or experiences with using telepresence by ambassador for local dev integration with their dev cluster(s).

I’m a junior DevOps engineer and was recently tasked with implementing a method for local dev integration. By all accounts, telepresence seems like an awesome and powerful tool, but I am having a ton of difficulty getting it set up and working correctly on my machine for a POC and am wondering what the general consensus is.

https://redd.it/q2wwc4
@r_devops

History of CI/CD on Windows?

Hello,

I am hoping someone with some knowledge and experience of DevOps can shed some light on the history of CI/CD in Windows environments. At my current workplace we are still using the copy and paste method for deploying all of our software. We will be going from TFS (only used to store code) to Azure DevOps Server and I'd like to be able to share some insight and provide some information as to why CI/CD in general, can be a good thing.

Specifically, I was hoping to gain some knowledge about the different "phases" of DevOps/CI-CD on Windows and what each new "phase" solved in the previous. I'm ignorant to how deployments were managed over the years but I'd imagine it would be something like copy-paste -> script -> script orchestration tool -> TFS -> Azure DevOps.

Any insight would be greatly appreciated. Thanks!

https://redd.it/q308te
@r_devops

Can you enter a container and execute commands on Gitlab CI while it's running a pipeline?

I have a particular environment with a strict firewall. Making small incremental changes and then running a job takes forever, it would be much easier if I could execute commands in the container and see results in real time. Is it possible? If not what's the second best thing? Is there a CICD provider that allows that?

https://redd.it/q2ldyx
@r_devops

Understanding Octopus Deploy Targets

After reading this post, I'm not de-mystified from the pricing from Octopus deploy
https://www.reddit.com/r/devops/comments/dejlae/understanding\_octopus\_deploy\_pricing/


I have 3 clusters of EKS (dev, stage, prod) - with varying number of nodes powering it.
I have some serverless functions on AWS

I have several namespace within EKS and about 50 or so microservices but I'm probably going to increase this in the future.


According to Octopus Deploy - What is a Target? Is it the 50 microservice? is the ENV? is it ENV + Lambda? is it the very ultra expensive path: ENV + Services + lambda + ???

https://redd.it/q31uxy
@r_devops

Need suggestions regarding my terraform code

I'm working on bringing up a kubernetes cluster infrastructure through terraform.This is what I have right now:

1. I have created separate directories for each component and each component has a terraform.tfvars file.
2. If I have to bring up a new cluster for a different environment, I'll have to copy the root directory and modify the terraform.tfvars files for the environment specific configurations.
3. Then go to each component directory and do "terraform apply".
4. Using Ansible for software provisioning.

What I'm trying to achieve is to bring up a new cluster for a new environment with one click. At the moment, I'm considering to use "terragrunt". Any suggestions would be highly appreciated.

k8s-msvcs

├── bastion

│   ├── backend.tf

│   ├── main.tf

│   ├── provider.tf

│   ├── terraform.tfvars

│   └── variables.tf

├── elb

│   ├── backend.tf

│   ├── main.tf

│   ├── provider.tf

│   ├── terraform.tfvars

│   └── variables.tf

├── iam

│   ├── backend.tf

│   ├── iam_policies

│   ├── main.tf

│   ├── provider.tf

│   └── variables.tf

├── master_nodes

│   ├── backend.tf

│   ├── main.tf

│   ├── provider.tf

│   ├── terraform.tfvars

│   └── variables.tf

├── security_groups

│   ├── sg

│   └── sg_rules

└── worker_nodes

├── backend.tf

├── main.tf

├── provider.tf

├── terraform.tfvars

├── utils

└── variables.tf

https://redd.it/q32k54
@r_devops

Openstack popularity? Career advise

I'm currently working with k8s, AWS and Azure and I see the market for those is very hot. I'm looking for a change to get a little better pay, but mostly for career growth, and I have this offer to join a team in a software development company running Openstack and looking to start migrating their services to k8s (within Openstack)

I feel like I might be taking a step backwards with going away from public cloud, but still the opportunity seems promising in that I could be part of the team migrating to k8s as well as introducing devops practices to the company.

Advice?

edit: I'll say that it's a government company (european) so they're stuck running a private cloud

https://redd.it/q33jit
@r_devops

As a sysadmin what’s your main job responsibilities or day to day tasks right now ? In your humble opinion what’s the main differences between a junior, intermediate and senior sysadmin today ?

1) Do you do a lot of the advanced stuffs AWS / Azure, Powershell, database, server hardening, Ansible, Active Directory on premise with exchange or system center, advanced storage clusters Kubernetes things like that etc or it’s mostly routine things changing password, updates, checking sercurity, hyper v or something else ?

2) In your humble opinion what’s the main differences between a senior sysadmin and a junior starting out learning windows server / linux etc or a intermediate ? Do you feel it’s more about knowing things or getting experience or it’s about understanding concept and know where to go what will work what will break etc

3) Do you prefer on premise or in the Cloud ? Have you ever worked in a large Data Centers with a Hybrid solutions of on premise with the Cloud ? Do you miss the old day of pure sysadmin with no cloud everything was in the building so to speak. What’s your opinion about scripting and automation, do you consider yourself a advanced scripter / programmer in Bash / Powershell ?

4) Prefer Linux or Windows Server and why ? What's your opinion about some youtuber or microsofties who knows just Azure and nothing of Linux would you consider them real sysadmin or ?

https://redd.it/q33z40
@r_devops

Automated build and release of Python packages using CircleCI and bump2version

Hi,

I am in the process of trying to imagine how I can change the development process for our team of 3 developers. We create and maintain a number of python packages that we deploy to a private PyPI server.

What I would like to do is to automate the versioning of the packages. We have a master and a development branch currently.

At the moment we manually update the version number in a setup.py file and tag the corresponding commit and CircleCI deploys it based on that.

Is it viable/possible to set up a development workflow where the CI/CD tool tags and updates the version number?

I've been looking at bump2version, but if the new version number needs to be updated for developers too, does that mean their local branches will often be one or several commits behind because of the version commits?

How do other people do it?

https://redd.it/q2kzhv
@r_devops

What are you using for handling incidents in the cloud?

How well is it working?

https://redd.it/q35u6k
@r_devops