many services, single host: what tooling to use? Ansible, terraform, docker, bash scripts?

Hi everyone,

I'm posting here to ask some advice because I don't see my particular setup being discussed often (if ever) in DevOps forums or articles. Some background: I'm a freelance web developer that builds a lot of different web apps for different clients, but that are each small enough to be able to host on a single server, that is: I'm hosting all apps I ever made on a single DigitalOcean host. Each service is contained within a Docker container and all networks, env variables and secrets are defined in several Ansible playbooks.

The problem is, with every service/app I add, my collection of Ansible playbooks becomes more a mess. There's dependencies between playbooks, for example: all services rely on a single Traefik reverse proxy (or nginx, I'm thinking of switching back to that) being started and my vars-files are starting to become a big list of variables. Aside from that, writing an Ansible playbook where I'm just starting containers feels like a bit of a pain compared to writing a docker-compose.yml and having a quick look at Terraform makes me think that this declarative approach is much easier to manage and reason about than the way Ansible does things.

Is there a better approach to doing this? Should I seriously look into Terraform? Kubernetes seems way overkill as I don't need any orchestration as the trafic I deal with is so minimal. Would it be better to just manually copy docker-compose files and nginx configs to my host and keep each apps config in their own repo? Should I try to make better use of Ansible roles and split out my configs in that way, defining common components like firewall and reverse proxy in a common role? Curious to hear how other people would approach this!

https://redd.it/10qpri2
@r_devops

What are the best devops stacks these days?

I'm working at a small company and I've been recently upping my skills especially in the cloud department and I'm pushing to go serverless, probably by using Google Cloud Run and AlloyDB. This has led me to question our use of Azure DevOps since it was here before I got hired and I never got to see alternatives.

I've heard that nowadays the best integrated solutions are either Gitlab with its CI/CD or GitHub Actions but it's unclear whether it's fully mature. My main issue with ADO is obviously the cluttered UI and the convoluted pipelines but that may be the nature of it. Also Microsoft which makes me more interested in Gitlab as a default but I don't want to choose a lesser solution out of spite for MS

Our DevOps is relatively simple, we manage projects using Agile, backlogs and the usual stuff. We have a CI pipeline that compiles everything, runs unit tests, publishes artifacts and then deploys on our on-prem windows server IIS instance for testing. Once it's validated main gets PRed into a release branch and after manual approved everything gets deployed to our production VPS.

In the future we will move for serverless and thus the pipeline will also build docker images and upload to GCR for production. For testing we will either retain an on-prem solution or find something else.

So I guess I'm looking for the pros & cons of each from actual user perspective in the context of our workflow

https://redd.it/10qq6om
@r_devops

How do you cope with your employer forcing you to use their shit ass laptop ?

It’s frustrating limiting, I can’t connect multiple monitors to it, and I can’t install any basic tool on it that are from my point of view absolutely necessary.

https://redd.it/10qoukj
@r_devops

´include:´ keywords limit (250) in gitlab ci pipeline

Hello DevOps community,

I just switched to DevOps and trying to move ahead in this exciting field. As part of my new tasks, I must look after a HUGE pipeline that creates GCP projects for the whole company using terraform as IaC tool. A week ago, it reached the include: keyword limit, despite the CI files are already DRY'd as much as possible; mainly do to the fact that the company is huge and there is a lot of demand for new GCP projects everyday, and for every project we create an individual ci file that performs terraform init, plan and apply. An idea that came to my mind is, to write a simple code (maybe in python or go), that on every pipeline run checks wether a file has changed or not, and depending on the changes generate a new ci file that triggers the whole init, plan and apply flow again, but it may take me too long to develop that code. Any suggestion on how to approach this issue is welcome and really appreciated!

https://redd.it/10qoyvs
@r_devops

Best DevOps blogs to follow?

In an ever-changing industry, keeping your knowledge in point is very important. This Subreddit is a great way to keep up to date, but I was wondering what are the best DevOps blogs to follow?

Currently, I follow the AWS DevOps blog to keep with new services and architectures.

Please share your recommendations.

https://redd.it/10qvf4b
@r_devops

How to protect your database (from yourself)

Hi everyone,

My name is Rotem, I'm one of the creators of Atlas, an open-source database schema management tool that we're building at my company.

We created the project out of frustration from the amazing gap between the maturity of DevOps tooling in many layers of the software stack and what we could use for our databases.

One of the areas we focus on is increasing dev-velocity by making schema changes much much safer by automating the verification of schema changes. To do this, we've built a bunch of infrastructure for doing static code analysis of schema changes.

You can catch a quick demo of it in a video I uploaded today:

How to protect your database (from yourself) - YouTube

Since we've all experienced that database outages or failed migrations are a particularly painful subject, I wanted to ask, how does your team do CI for database schema changes?

https://redd.it/10qvd83
@r_devops

prometheus + grafana stack in k8s. is it a silver bullet?

I'm <3 years junior DevOps and I have experience with 2 companies that both have 20\~30 devs. Both companies use prometheus + grafana stack for their monitoring especially k8s nodes & pods. However having used these stack I got some curious that are other companies using same thing as me. Because these were not that bad in most cases though, but some trivial bugs or several-years-old issues gave me a hard time. (or maybe they were all my fault)

And it seems there are 4 major way to deploy prometheus based monitoring stack in k8s..
prometheus-operator/prometheus-operator,prometheus-operator/kube-prometheus,bitnami/charts/kube-prometheus,prometheus-community/helm-charts/kube-prometheus-stack,... they really confuse me

\---

So I wonder

\- Are you using prometheus + grafana for k8s monitoring?

\- If yes, are you using them without any problem?

\- If not, what are you using now?

&#x200B;

Thanks

https://redd.it/10qy7j9
@r_devops

What’s a project, personal or professional, that you can never find the time/resources for? Why?

For me it would be to update my personal website with a blog portion that contains well documented posts of my completed projects. I have the projects done. I have the knowledge and experience. I know what to do to get this all completed. I don’t really have any excuse, especially considering that if the big project was for work I’d already have the documentation completed.

https://redd.it/10r0i5w
@r_devops

Monthly 'Shameless Self Promotion' thread - 2023/02

Feel free to post your personal projects here. Just keep it to one project per comment thread.

https://redd.it/10r0ixm
@r_devops

Best Free Full-Stack monitoring Suite?

We currently have a hybrid Datacenter-AWS environment. We are currently using Dynatrace, but it is too expensive. What suite do you use to monitor Applications, Servers, Containers, networking, DB, logs, etc?

https://redd.it/10r1ca2
@r_devops

Test environment for branches?

Is there a standard procedure for manual testing different branches in a repo?

For example I have Azure DevOps server and I have an environment set up for my dev branch (site, db) so people can click stuff in the current general state of the app.

If I start implementing a feature on a separate branch, how would I handle the environment? Like say I want to show off the progress to see if this is functionally the right direction or whatever.

I would need to provision a new environment, frontend, backend, db, domains, etc. How would I achieve this in an automated manner?

https://redd.it/10r043l
@r_devops

looking for ways to learn AWS or other cloud service providers in a production environment

I have good experience managing infrastructure in a private cloud for more than 5 years, with strong k8s and docket fundamentals. However my job didn't involve me working on any public cloud providers. Every job I am looking for asks for strong experience in atleast one of the CSPs. So what is a good way to learn and use one of the CSPs at production level so that I can justify my experience. Don't want just introduction to AWS or something else.

https://redd.it/10r5tzt
@r_devops

Code Automation Help

Hi all,

I'm trying to figure out what the most efficient way is to automate my app deployment process. Currently using GitLab to store code for an app and using an AWS EC2 instance that it gets deployed on, I'm currently limited to EC2 only. My current process is, I have a Packer script that builds out the AMI image and includes a line to clone my git repo for the app code.

This is a manual process each time I update so I'd like to change it where whenever new code is committed in GitLab, it can automatically kick off the Packer script to create the newly updated AMI image. Is this possible to integrate with GitLab CI/CD or any other ways of doing it?

Thanks

https://redd.it/10r83zs
@r_devops

Frequent Targetgroupbinding reconciliation on an EKS Cluster

Hi Everyone,

Viewing my event logs, my EKS cluster tends to perform a lot of reconciling (less than 15mins continuously) by Targetgroupbinding with exposed service containers.

I understand it has to perform leader board election but checking if this is normal behavior

https://redd.it/10r9jst
@r_devops

What to expect for first DevOps interview? (with DoD-based contractor)

Hi everyone, while I think I have a federal "DevOps" role in the bag (seems to be taking forever for a solid offer letter though). I was invited by two DoD based companies to do an interview with them for mid-DevOps roles. I'm DevOps now, I think?

I use docker, Ansible, I create bash or python scripts to lockdown systems or to install sw prereqs, I do site reliability (but without monitoring software), I troubleshoot finicky service scripts, I ACAS scan systems for security vulnerabilities, I create and backup Linux systems, and of course more. So I do DevOps stuff but then I look at these job listings I applied for and I don't know cloud (@lat least no work experience), I don't know load balancers, I haven't produced code for a project, I don't know kubernetes, I don't practice leetCodes. Am I walking into a mind field for this interview? I have "PowerShell" scripting on my resume but now that I think of it, I have scripted anything in PowerShell since I made my few "Windows baseline configuration" scripts 2-3 years ago. 😶 I just use them quite often.

DevOps broadness is awesome when you're on the jobs but nerve-racking when you are trying to land a position. My current position was for Systems Administration but I kind of got thrown onto a very fast moving agile development project where I learned my above skills.

So what is expected of me if I don't know some of these technologies, what are some ways you guys navigate unknown technology questions in interviews?

https://redd.it/10ra89y
@r_devops

Monitoring mobile apps

How do you monitor your mobile apps to see if it has issues loading?
Story: We have mobile apps available from both Playstore and App store. We get issues sometimes when the app doesn't load at all or shows a blank page. Web application(browser based) loads fine at all times. Hence all our monitoring checks pass as it's looking at web applications only.

My question is how do you ensure uptime on mobile apps? Please advise.

https://redd.it/10qxsh4
@r_devops

Folks In Healthcare Industries, what do you use for application monitoring?

Tl;dr: Large hospital, primarily Windows Desktop/Server, Epic. We have infrastructure monitoring covered, what do you recommend for application monitoring when there is TONS of health care applications already in the environment?

Without disclosing much, I work a hospital. Rather large one. We're an Epic shop. We run our own datacenter. Been here for 4 years, came onto the DevOps team recently. Started in operations, went over to the systems administration team so it's a huge plus as a lot of the monitoring tools I used to be the operator on, I am now the maintainer.

I had a realization today that we don't have a clear tool to do application monitoring. Infrastructure is certainly covered as we have vROps, Loginsight, NagiosXI and other solutions at our disposal to monitor the underlying infrastructure. I've tried working with Telegraf, which is integrated with vROps now; had my gripes and such with it but it does OK; just not the best or what I would want for my org for an application monitoring perspective. Best we can do right now is Windows Services monitoring, but I know we can do better than that.

There must be at least 200+ applications in the environment total. That's just a guess. I checked out roadmap.sh and it mentions Jaeger, New Relic, App Dynamics, Instana, OpenTelemtry. Checked out Dynatrace but the agent is 5gigabytes.... just seems a bit big.

What's your take?


Edit: I have a test environment at work that is usable. Additionally, my own homelab at home. So I'm ready to tear at something. Just wanted to gear it towards Healthcare applications specifically.

https://redd.it/10rdzc2
@r_devops

When did you shift to devops and where do you see yourself going amidst recession and do you thinks devops will stand strong amidst Artificial intelligence in coming years. Give solutions as well to overcome to all these questions please. Thank you!

..

https://redd.it/10rd6tk
@r_devops

Continuous Delivery vs Deployment

https://adamdingman.net/posts/continuous-delivery-deployment/

https://redd.it/10rgrsa
@r_devops

I was exploring Nginx web server and got error as permission denied. I don't find where. I have given every permission.

This is my nginx.conf:

events{}

http {

server {

listen 80;

server_name localhost;

root /home/mrburnwal/Downloads/DhirajCV-master;

}

}

Error from error.log:

2023/02/02 09:06:11 [notice] 810306#810306: signal process started
2023/02/02 09:06:18 [error] 810307#810307: *8 "/home/mrburnwal/Downloads/DhirajCV-master/index.html" is forbidden (13: Permission denied), client: 127.0.0.1, server: localhost, request: "GET / HTTP/1.1", host: "localhost"

and my file permission that:

drwxrwxrwx 2 mrburnwal mrburnwal 4096 Nov 20 2021 css
-rwxrwxrwx 1 mrburnwal mrburnwal 2915 Nov 20 2021 index.html
drwxrwxrwx 2 mrburnwal mrburnwal 4096 Nov 20 2021 media

it should have run successully with this config. But getting errors. Ans yes, I have reloaded nginx after making changes as well.

systemctl reload nginx

https://redd.it/10rfzo9
@r_devops

My 2023 plan

I'm a noob to devops, but not a noob to software development. Anyway, my current job is titled as devops but it's actually code fixes and maintenance and shit, hardly an opportunity to learn cool new shit, particularly cloud native stuff, which I'm interested in. I've made a plan to learn cloud native stuff in my free time. Here's the plan below and let me know what you think:

Goal: deploy and manage full stack app in k8s

\- pick a cloud from either Azure, AWS, GCP

\- learn ArgoCD

\- learn Rancher

\- try out GitLab CI/CD

https://redd.it/10rig0b
@r_devops