DevOps Degree Thesis help

Hello, DevOps Redditors!
I am in my final year of bachelor studies in CS and for the topic of my degree thesis, I chose to write about DevOps concepts, environment, and this Covid remote work situation (not yet refined the research question).
My colleague and I are in seek of information about experiences from DevOps (worldwide) in terms of the environment you work in, common tools that you use, problems that you face due to the Covid restrictions, etc.
We are doing the thesis in Sweden and possibly will cooperate with the DevOps teams from Ikea and Ericsson but we would like to get the experience outside Sweden as well.


So please write here:
What is (in your opinion) essential for a successful DevOps environment? (tools, utilities, ...)
What are the most common tools that the DevOps environment includes today?
How has the Covid pandemic affected DevOps?


Thanks in advance!

https://redd.it/l1znvl
@r_devops

Advice on monitoring an external API

I'm looking setting up a system to monitor an external API for a service we use. This will query the API to get back results on unauthorized config changes or unusual activity. It will generate alerts that get sent to our Slack channel if something doesn't look right.

I've already got a basic Python app working that queries the API. I hope to extend it in the future but just want to containerise what I've got for now.

Here is the basic plan based on systems we currently use:

- Create Python application to query the external API
- Store results in a local DB that can be used to compare with existing results
- Create alerts in Slack for data that isn't right
- Run the application in a container in our kubernetes cluster
- Run the container as a cronjob at set intervals to check for changes

What I'm asking is, is best to run this as a cronjob? Is there maybe some type of framework I can use where the container is running constantly and sending queries at set intervals? Also any other advice on the setup would be appreciated.

Thanks!

https://redd.it/l1zgec
@r_devops

Importance of issue tracking survey for university

Hello everyone, we are a group of students in IT. As part of a case study for university, we want to determine the impact of issue tracking and how useful it can be for bug investigation.

We would be very grateful if you can take the time to fill this survey, it should take about 5 minutes.

Link to the survey : https://forms.gle/TtfcbQWa1sUieR9bA

https://redd.it/l2krdh
@r_devops

Looking for a logging/alerting solution

We have an AWS-based product with a small but growing number of customers. It’s a niche product; the maximum number of customers we’re expected to ever acquire is in the low hundreds. We have a hand-rolled alerting solution at present but it needs overhauling and is going to be replaced with a COTS product.

One of our key requirements is that we have a lot of customer-based metrics that we want to monitor/alert on. For example, one part of our system processes incoming documents and we have a queue-depth per customer of “documents to process”. Pseudocode for inserting these metrics would be:

insert-metric --metric-name document-queue --tag “environment:prod” --tag “customer:customerA” --value 0
insert-metric --metric-name document-queue --tag “environment:prod” --tag “customer:customerB” --value 100
insert-metric --metric-name document-queue --tag “environment:prod” --tag “customer:customerC” --value 0

We would then want to define an alert that says “if document-queue is over 1000 for a 5-minute window, generate an alert containing the environment, customer name and queue depth”.

Alerting by customer is important to the boss because he wants to know if there is an issue, exactly which customers are affected so he can personally reach out to them if required.

We have approximately 25 other metrics to monitor, some of which have more “tags” e.g.

insert-metric --metric-name outstanding-async-jobs --tag “environment:prod” --tag “customer:customerA” --tag “job_type:asset_report” --tag “job_id:12345” --value 123

We started off looking at cloudwatch but we’d need to create alarms for each environment/customer/metric combination and the costs would start to pile up as well. Even if we automate the alarm creation it doesn't make sense for us to have to create a bunch of new alarms every time we acquire a new customer.

In summary, our requirements are:

1. Support many custom metrics with alarms, without having to create the alarms for every “tag” up-front
2. Support synthetic monitoring of web UI and REST API
3. Cheaper than cloudwatch :)
4. If it can be managed via Terraform, even better

We started looking at Datadog, New Relic etc. but their marketing-websites and pricing pages are a bit impenetrable unless you understand their product well. Given our (relatively simple) requirements, does anyone have a recommendation?

https://redd.it/l2lmxv
@r_devops

A tech IP lawyer panel on Elasticsearch not being open source anymore

This Tuesday - A webinar where we will go through the history of the Elasticsearch license, the gradual change from full Apache to SSPL license (not open source) and have an open Q&A with an IP expert lawyer to answer any specific use case and whether is poses a threat to your business.

https://land.coralogix.com/panelelknotopensource/

https://redd.it/l2m259
@r_devops

Formulate DevOps questions differently.

Hi folks

Could any of you make bottom questions easier for me.

I do not understand what should be in content, what should I focus on.

I just know that the content of questions must be practically applicable.

If you know a link where I can learn more about practical application of DevOps in an organization is also welcome.

Q 1. Provide a description of a current value stream that you identify within your organization, including a classification of the process activities.

Q 2. Make an analysis of the current situation. What is going well now and what could be improved, taking into account what you think are relevant and elementary DevOps aspects?

Q 3. Map out the desired situation, indicate what the most ideal situation is and what you think is the highest achievable within your organization.

Q 4. Give a brief description of how you intend to optimize the situation outlined using DevOps.

https://redd.it/l1uufq
@r_devops

Best SSO solution for a +50 company

I'm collecting opinions on the best SSO solution for a mid-sized company that only has a few DevOps specialists.

I have created a (very) preliminary list of solutions. I want to test them for ease of integration, costs, ability to integrate with other services ect. These are:

* Azure Idap
* KeyCloak
* AWS Cognito
* Azure Active Directory
* AWS SSO

We definitely want to integrate with Github, Gmail, Vault, AWS and Graylog.

https://redd.it/l2ohox
@r_devops

Use cut to add fields between strings

I have a string a.b.d and I want to add .c after the first two fields to get a.b.c.d.

This is the first step I guess.

~$ echo "a.b.d" | cut -d"." -f1-2
a.b

Problem is how to continue with this? How do I append .c to the output above and then append the rest of the string f3-end?

Thanks ahaed.

# EDIT:
The only thing that's consistent are the dots. The string can might as well be gasg1.123.sgd1fsa. or asdp[q=aaa.0.asd1

https://redd.it/l1xlz7
@r_devops

User Telemetry on OSS project? PRO or CONS?

I'm trying to figure out how to have more information on who is using my open-source project.

Until now I have only the download metrics on how many people downloaded a certain version of my App, but I still no have any evidence on how many users are using it daily?

Could it be seen badly if some sort of Anonymous analytics will be added to the Desktop App?

PS: I explain better my project, it is an Electron Desktop App for developers, and with some Analytics I can retrieve very useful information that can impact the UX/UI side.

Furthermore, some of those analytics can help me in prioritizing some features.


What's your opinion?

https://redd.it/l2r18a
@r_devops

Pomerium — open source identity-aware access proxy — now supports TCP

I wanted to share update about Pomerium that I'm really excited about.

Pomerium now supports internal access for any TCP-based application or service such as, SSH, RDP, or any Databses like Redis, MySQL, Postgres! And as with with HTTP, every session is authenticated, authorized, and encrypted. This has been one of the most requested features since the project's genesis.


- ▶️ Check out the demo
- 📢 Read the announcement

Thanks again to all our users and to everyone who contributed to the project so far. Happy to answer any questions!

https://redd.it/l2re7f
@r_devops

Looking for UI for building yaml by schema

If you think about that, if you have a yaml schema, you already know how the yaml will look like, and what item/type can be inserted in any point.

So if I have a service with long configurable yaml, I want to help my users with a UI that helps them to generate the yaml. You know, like the Swagger tools.

Do you know such as tool/library? That I can give it a schema and it will create a form to generate the yaml file?

https://redd.it/l2ozgc
@r_devops

Devops Career - stuck in mediocre companies

Hi,

I've been devops engineer working with AWS: mainly containers with ECS, terraform iaac, ci/cd with Gitlab,etc. However, I feel that I've been stuck in jobs that are not that great. I will go through my job history:

1. Joined as a devops engineer in a big corporation. My team was basically responsible for creating IAM users/ policies/ etc. There were some interesting projects but most of the job was very boring and slow moving. However, the nice part was that we were a team of devops engineers and all could learn and share experiences and it was good as a junior engineer.
2. Joined a scale-up style company. I was placed as a single devops engineer in a team of backend developers and was responsible for modernizing the infrastructure. It was very challenging and interesting task I was communicating with developers and suggesting the best tools in AWS to use for the job: SQS for asynchronous communication, Cloudfront with s3 for fronted deployment,etc. I liked the job, however after modernizing all the infrastructure, creating ci/cd pipelines, monitoring with cloudwatch I didn't have enough to do, also my manager was not very supportive.
3. This leads to my current job. I joined the small company that deploy one project to the cloud and didn't have a devops engineer before. It was very challenging but interesting at first: all of the infrastructure were manually provisioned using aws console, deployments were not automated and were failing all the time. So I had to do everything myself in a moving system. However, after 8 months I was done with creating IaaC using terraform, CI using gitlab, etc. With AWS ecs containers there is also not much work, our system has very stable traffic so no autoscalling is needed, also we only deploy to production every couple of months.

I'm thinking I should start looking for a new job soon because it feels that my career is not progressing forwards. The last two jobs looks like could have been done by consulting companies not a permanent devops position. I mostly enjoy solving engineering problems. However, in current job I don't feel challenged at all. What should I look for when finding a new role? Do you have any tips?

Thanks!

https://redd.it/l2o49l
@r_devops

Helm Umbrella charts and ArgoCD/Flux

Hi everyone,

Current setup

60 microservices
Helm umbrella chart, all microservices as subcharts

​

I would like to move to the 'GitOps' model where the developer would use a PR to change the version of an image/Chart and that would be reflected in the environment.

​

1. Can I use ArgoCD/Flux v2 with my existing Helm umbrella chart (with its subcharts) or do I have to add individual helm charts in each of the 60 microservices repos?
2. Will the microservices be displayed as one app or the tools will figure out there are 60 of them?
3. Anyone have a similar case they tried out? Which one (argo/flux) worked best for this scenario

​

Thanks

https://redd.it/l2rb1y
@r_devops

React App Slow Startup Time in Docker Container on k8s

This maybe better suited for a react sub but wanted to start asking infra folks.

I'm experiencing very slow start up times for a production webapp built with react in a docker container. Base image is node:14.2.0-alpine3.10. Experiencing maybe 5 minute avg startup time in a deployed container on kubernetes. Locally around 5m as well.

Does this seem reasonable for anyone else deploying react applications in docker? Even testing locally the startup time for my react app feels atrocious. Looking for any optimization tips for the container or resources to try to improve these times.

Edit: Dockerfile & Package.json for context

https://redd.it/l2wml0
@r_devops

SigNoz - Open-source alternative to DataDog. Would love some feedback!

Hi r/devops

Wanted some feedback on our project to build an open source alternative to DataDog.

In my past roles, I spent a lot of time debugging issues in production. To solve this, we tried to build something using Prometheus & Grafana, but it didn’t give us the complete experience as an APM product like DataDog does. Hence, we thought of building an end to end open-source product with seamless integration of metrics & traces.

https://github.com/SigNoz/signoz

Would love to get any feedback from you on the project and if it is useful/not useful

https://redd.it/l2ss9o
@r_devops

Fulfilling the promise of CI/CD

https://stackoverflow.blog/2021/01/19/fulfilling-the-promise-of-ci-cd/

Does your org continuously deploy? I've been doing doing so for the past 10 years, but had to stop at my last job because a random auditor decided it was too risky. They made me install an "approval" gate, which was really just theater and caused changes to batch up.

https://redd.it/l2sl3k
@r_devops

What do you do?

As DevOps Engineers, I wonder how do you reply to this common question to people that are not familiar with the Tech industry.

I use to give them a long answer including software deployment, automation, pipelines, etc. but I noticed that it was confusing for many of them.

Nowadays I just reply that I'm a Software Engineer that works specifically on the backend side.

https://redd.it/l2rmys
@r_devops

Running docker from scripts

I'd like to set up my pipeline so that I can run most of it on my own machine. I've noticed that most of the CI systems use their own config syntax to define running containers -- I was thinking about using docker-in-docker so that my own code spins up the various dependency services.

Is this a terrible idea for any reason? I'm surprised I haven't seen more people do it that way. I don't particularly dislike the yaml config syntax, but AFAIK none of the CI platforms have local running as a first-class feature, so I always end up duplicating stuff. Take running integration tests, for instance: I need to spin up my application along with some related services. I've got a script that does this locally, for development, and then the yaml that does it in CI. They get out of sync over time, and it's kinda awkward.

https://redd.it/l323z5
@r_devops

Encrypted Registry Account Credentials with Codeship and Github Container Registry

Hi everyone,

apologies of this has been asked before.

I have been going through the documentation for codeship to see if I can pull an image for docker from a private github container registry (GHCR) when I am automating my build process.

The documentation specifies docker registries but nothing for the GHCR.

Has anyone used Codeship to pull docker images from the GHCR?

The documentation specifies that the most common way authenticate with image registries is to provide your account credentials via an encrypted dockercfg file. The example is as follows:

{

"auths": {

"`https://index.docker.io/v1/`": {

"auth": "your_auth_string",

"email": "your_email"

}

}

}

(sorry about the indentation)

This file will later be encrypted as per the instructions in the docs which can be found here

I was wondering if anyone has used this for GHCR, and if they have my question is whether one can simply replace the URL with the GHCR URL and pass it the email and personal access token used to access the packages or in this case images in the GHCR?

https://redd.it/l2pdz4
@r_devops

AWS Beanstalk environment management config support Japanese language???

Config Environment properties Japanese.

I have config title email in ENV but when i get email from it converted to "???". Any solutions?

https://redd.it/l33lf7
@r_devops

quick file sharing solution :-)

Have you ever neeeded to quickly copy a file from one computer to another, but... the boxes did not want to talk to each other? (For example one is Windows machine, and the other one is livecd-booted Linux without samba client...) Yes, me too... :-)

Have fun: https://hub.docker.com/r/michabbs/trashbox

https://redd.it/l2vxi3
@r_devops