Alert/event aggregators?

What is your stance on event correlation vendors? I’m talking about those that aggregate and correlate alerts, perform root cause analysis etc. For anyone using these vendors, are they providing actual value they claim? I’m talking about SNOW ITOM, Moogsoft, Big panda and similar. Interested in hearing opinions.

https://redd.it/y06mgf
@r_devops

Apache 403 Error You don't have permission to access this resource.

Hi all

I wrote an ansible playbook that deploys a wordpress host with the click of a button. It worked except when I went to add the ssl cert. I have had to play about with the apache conf file and it has resulted in numerous errors including not being able to read the website style, and not redirecting properly and now with my latest iteration im getting a 403 error.

​

No doubt its something misconfigured in the .conf file which ive show below.

​

:/srv/www# ls -lrth
total 4.0K
drwxr-xr-x 5 www-data nogroup 4.0K Oct 9 13:10 wordpress

​

<VirtualHost :80>
ServerName mysite.co.uk
ServerAlias www.mysite.co.uk
DocumentRoot /srv/www/wordpress
RewriteEngine on
RewriteCond %{SERVER_NAME} =www.mysite.co.uk [OR]
RewriteCond %{SERVER_NAME} =mysite.co.uk
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>
<VirtualHost :443>
ServerName mysite.co.uk
ServerAlias www.mysite.co.uk
DocumentRoot /srv/www/wordpress



Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/mysite.co.uk/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/mysite.co.uk/privkey.pem
<Directory /srv/www/wordpress>
Options FollowSymLinks
AllowOverride all
DirectoryIndex index.php
Require all granted
</Directory>
<Directory /srv/www/wordpress/wp-content>
AllowOverride all
Options FollowSymLinks
Require all granted
</Directory>

</VirtualHost>

&#x200B;

/etc/apache2/sites-enabled# ls -la /etc/apache2/sites-enabled
total 28
drwxr-xr-x 3 root root 4096 Oct 9 21:58 .
drwxr-xr-x 9 root root 4096 Oct 9 21:56 ..
drwxr-xr-x 2 root root 4096 Oct 9 21:37 backups
-rw-r--r-- 1 root root 345 Oct 9 21:03 wordpress-le-ssl.conf
lrwxrwxrwx 1 root root 33 Oct 9 21:27 wordpress.conf -> ../sites-available/wordpress.conf
-rw-r--r-- 1 root root 424 Oct 9 14:07 wordpress.conf.bak
-rw-r--r-- 1 root root 1841 Oct 9 18:40 wordpress.conf.bak.09101940
-rw-r--r-- 1 root root 1242 Oct 9 21:56 wordpress.test.conf

any help would be appreciated. Thanks in advance

&#x200B;

***********updating with more info***************

Apache up and running

/etc/letsencrypt/live# systemctl status apache2
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2022-10-09 21:59:24 UTC; 2h 21min ago
Docs: https://httpd.apache.org/docs/2.4/
Process: 5080 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
Process: 5552 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=0/SUCCESS)
Main PID: 5085 (apache2)
Tasks: 7 (limit: 1143)
Memory: 18.7M
CPU: 737ms
CGroup: /system.slice/apache2.service
├─5085 /usr/sbin/apache2 -k start
├─5562 /usr/sbin/apache2 -k start
├─5563 /usr/sbin/apache2 -k start
├─5564 /usr/sbin/apache2 -k start
├─5565 /usr/sbin/apache2 -k start
├─5566 /usr/sbin/apache2 -k start
└─5567 /usr/sbin/apache2 -k start
Oct 09 21:59:24 ip-172-31-2-184 systemd1: Starting The Apache HTTP Server...
Oct 09 21:59:24 ip-172-31-2-184 systemd1: Started The Apache HTTP Server.
Oct 10 00:00:18 ip-172-31-2-184 systemd1: Reloading The Apache HTTP Server...
Oct 10 00:00:18 ip-172-31-2-184 systemd1: Reloaded The Apache HTTP Server.

however there are some errors in the logs. They look to confirm what i suspected, and thats that there is something wrong in the .conf file. client denied by server configuration

Mon Oct 10 00:00:18.307276 2022 mpm_prefork:notice pid 5085 AH00163: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2 configured -- resuming normal operations
Mon Oct 10 00:00:18.307301 2022 core:notice pid 5085 AH00094: Command line:

'/usr/sbin/apache2'
Mon Oct 10 00:00:28.478617 2022 authz_core:error pid 5563 client 205.210.31.158:61332 AH01630: client denied by server configuration: /srv/www/wordpress/
Mon Oct 10 00:09:24.974556 2022 authz_core:error pid 5562 client 35.195.135.67:59210 AH01630: client denied by server configuration: /srv/www/wordpress/xmlrpc.php
Mon Oct 10 00:19:08.730886 2022 authz_core:error pid 5565 client 120.24.52.65:37542 AH01630: client denied by server configuration: /srv/www/wordpress/xmlrpc.php
Mon Oct 10 00:22:30.532975 2022 authz_core:error pid 5566 client 192.241.214.214:53134 AH01630: client denied by server configuration: /srv/www/wordpress/actuator

&#x200B;

https://redd.it/xzz1nx
@r_devops

Do we need leaders in a devops team?

So working in a smal devops team , working to move a onprem winform app to the cloud. And in the team we all have strengths and weaknesses. And for the last year we have been pushing out value every day. And out of the blue one of the devs got promoted to department lead, and the dev should now be our leader. And this got me thinking, would the power shift in the devops team be a bad thing? Do we need leaders, we have product owners.

https://redd.it/xzptla
@r_devops

How to deal with an already shitty status quo?

Whenever I check any devops resources, with enough digging, there's sufficient resources out there covering most crucial (and usually problematic) aspect of the development life cycle. They usually paint the perfect picture, assuming that all IT projects will start after reading that very devops book, article, vid ... my point is what if you ended up with a very large codebase,without automated tests, opposing goals across operations, devs, business? How to improve assuming that the worst case has already happened ex: deployments take days, require planning, lots of time wasted, bugs, conflicts, inability to make small changes and test one's work, ... you get the picture.

https://redd.it/y0bzon
@r_devops

I just canceled a technical interview due to the “assignment”. Sanity check

So I’ve been talking to this slightly post-startup company in Northern Europe which needed someone to come in and lead the migration work to Kubernetes and basically own the whole platform from code to production.

This is something I already have experience doing so it seemed like a great fit. First interview goes well, I chat with the CTO who I would be reporting to. Company seems progressive and with a good product.

Then comes the technical interview invite. They want me to design the golden path for their company from code to production and give a 40 minute presentation on it.

I’m initially skeptical as this sounds like “hire a consultant for free”. And when I start calculating how long it will take me to research, design and create this presentation I come to a complete halt.

I ask them if a 10 minute presentation will suffice. Their reply is basically, no but you can have more time to prepare if you’d like.

I said no thanks and canceled the interview. To me this sounds both ethically questionable and completely disrespectful of my time. Do companies really demand this much time from candidates these days?

https://redd.it/y0gows
@r_devops

Patterns for deploying application infrastructure in ArgoCD based GitOps setup

So hello dear YAML engineers, I am currently working on a project where I am facing some dilemmas, in how to handle the continous delivery of applications in my setup. To be more precise I am unsure of the best way to handle the deployment of infrastructure which is ancilliary to application, such as databases, redis caches and similar. More details about setup and more concrete questions in section below.

&#x200B;

Current tech-stack is basically following:

\- AWS fully in cloud

\- Kubernetes

\- Terraform for infrastructure

\- ArgoCD + Argo Rollouts for deployments

\- Jenkins for building and triggering stuff

\- Low level infrastructure such as networking, DNS setup, security groups, EKS cluster provisoning, logging and supporting functions are taken care of by separate platform team. We basically get a fully functional EKS cluster and AWS account, and are free to hack away at it, but only for purposes of application. We collaborate with platform team in case that we need some changes on networking ( which is basically never ), but we are otherwise free to provision anything and everything we need for our applications and we are free to install what we want in the clusters made available to us.

&#x200B;

Current setup is that we have three repositories like following:

\- One repository is for application source code, build scripts, tests and etc. Plain and simple.

\- Second repository is for declarative description of resources, which we need to have deployed by Argo in an App-of-Apps pattern ( here app-of-apps is needed since we operate the same application in multiple different configurations )

\- Last is our infrastructure repository, where we manage our Terraform code, with one folder per cluster.

On the infrastructure side we are currently only provisoning application-related infrastructure, namely RDS instances, Elasticache for Redis, DynamoDB tables and Elasticsearch clusters from time to time.

My questions are following:

\- Would it make sense to keep the infrastructure code togehter with application code? If yes, then where would I provide the values to my Terraform modules, which differ on per-environment basis? Concretely if have nonproductive and productive environment, which are based on same TF module ( which should now live with application source code), but are supplied with different values for variables, where would I place these values and where would I specify the version of the module I want to have for which of the environments?

\- Does it make sense to treat deployment of application-related infrastructure as part of the continuous delivery pipeline? I would have something like a presync hook for ArgoCD, where Terraform resources are applied before application is deployed. Alternative to this would be having fixed "infrastructure environments" ( consisting only of ancilliary services used by application ) and then deploying application "into these environments", which would mean simply configuring the application with correct URLs.

I would also be interessted if there is tooling concetrated on deploying the application together with application-relevant infrastructure, while excluding the "lower-level" infrastructure. I have heard of KubeVela, but I have read very few experience reports online.

&#x200B;

I am thrilled to hear your 2 cents on this.

https://redd.it/y0glfi
@r_devops

What are the biggest security challenges for CI/CD in 2022?

I am currently thinking about how to improve our build environment with regards to "I don't trust the admin", but "I want the admin to maintain my nodes". What are your challenges regarding security & CI/CD in 2022?

Maybe even challenges when adopting a SaaS CI/CD provider and rolling/maintaining your own platform instead?

View Poll

https://redd.it/y0cre3
@r_devops

Skipping GitHub Actions jobs while maintaining branch protection rules that require them

How to work around some quirks and limitations of GitHub Actions to skip CI jobs that aren't necessary in certain scenarios, without breaking branch protection rules that normally require those CI jobs to succeed:

https://blog.pantsbuild.org/skipping-github-actions-jobs-without-breaking-branch-protection/

https://redd.it/y0ku2a
@r_devops

Gradle and Software composition analysis

Why do SCA tools require a gradle build before scanning whereas for other projects just scanning package.json is sufficient without build?

https://redd.it/y0n45s
@r_devops

Best conventions to go to 2023

What conventions would you consider the best to attend in 2023? Looking for generalized conventions dealing with all facets of DevOps.

https://redd.it/y0nxgs
@r_devops

Diagram-as-code for cloud architecture

For visualizing cloud architecture and generating a diagram using a small DSL.

https://docs.tryeraser.com/docs/examples

https://redd.it/y0k2gz
@r_devops

loops in packer - dynamic sources/builds

Hi,

Looking for a bit of advice on how best to approach this.

Initially I had a packer file that runs ansible against a docker image, pushing the new image to a repo.

I then needed to do this for a number of images, so I added a source for each one and created a separate post processor to tag the specific images before pushing them.

The problem I now have is that the list of images is changing regularly, I have written a python file to loop over the different images/tags and pass the details into the initial packer file, running the builds one at a time but I was wondering if there is a way to do this natively in packer a d run the builds in parallel.

The list of images would be something like Ubuntu:focal, Ubuntu:bionic, Ubuntu:jammy though it would likely be longer and a mixture of images.

The python script works, just wondering if there is a better way.

Thanks

https://redd.it/y0swq2
@r_devops

Need help wrapping my head around massively bloated legacy application

I can't give too much detail as to not give away where I work since it's quite a large company.

But basically I'm on a very small team supporting somewhere between 15-20 applications. We're always so swamped - I've gotten better with dealing with the stress though. And learned a lot.

However there is one application that is just scary and it's partially broken and has been for months. My team lead made it but was gone for some time and it was changed a lot. It's an old web app that basically does a lot of HR processing.

We have multiple tickets for it but it's just been ignored until recently they are really on us to fix it. My team lead has begun training me on the app and I have many tickets and enhancements assigned to me for it.

While there is indeed a dev, test and prod version. I'm not sure that it's the same.

The documentation is a little sparse and there are dozens of stored procedures, batch jobs, and it's using SSIS as well which I've not used till now.

Basically all of the work is being handed down to me and I don't know if it's because I'm burnt out or if I just know I'm not capable of fixing this application but I just don't know where to start and am overwhelmed. Looking for some sort of guidance from someone on how to approach massive legacy projects like this that are broken and when documentation is sparse and there is just so much data and processing being done?

https://redd.it/y0s6t4
@r_devops

DevOps - Dev & Ops, Build & Maintain vs Ops/Customer Support - Prod Access

I know everyone has their own interpretation of DevOps, NoOps, etc.

Lets see how the following theory can work "Developers (a team) build and maintain a product." This team is cross functional but conceptually all are the same 'role' but have different strengths and have the same access otherwise we're just grouping Developers and Operation contributors together in a team.

Need to deploy a new service, the team spins up a new server.

Server runs out of memory and hangs, the team restarts it.

Team then builds new automation which they deploy to automatically restart it if it runs out of memory again.

Team gets tired of doing manual deploys and builds automated deploy, but still needs access to do the deploy since well they are running it and can modify it and what is in it.

All of that is focused on engineering and technology, not business and I think we all agree that falls under DevOps and access that the team needs to have which is already pretty high access since they built the deploy pipeline, the server, and everything that gets put onto it, meaning even if they don't have access to they prod keys, they can deploy scripts that do.

Lets say we're a movie theater and customer calls up our customer services/support line and complains that they reserved seat 2 but their virtual ticket says seat 20. Hopefully there is a tool that lets CS reassign the seat but we're a startup and haven't built it yet. Movie is later today so we don't have time to build a tool.

CS calls us up and needs help, do we have access to modify the ticket mysql database directly to change the seat. `UPDATE tickets SET seat = 2 WHERE id = ?;`

If we don't, who does?

In theory no one does, there is a tool for everything and never any bugs get through our process so there is never need for one off fixes but lets face it, that'll never happen.

Does your answer change if this is a bank or hospital instead of a movie theater?

Does your answer change if the organization is 3000 people vs 30 people?

https://redd.it/xzlhyk
@r_devops

Exited from consulting to a DevOps role in the industry, missing some aspects of consulting, how to get the best of both worlds?

Hello!

After spending 2 years in consulting right after my CS degree (ACN, cloud domain), I've accepted an industry role as a DevOps Engineer, that came with a title/rank bump, huge salary bump, and is fully remote - my new employer is in another country and doesn't have a branch in mine.

While I really enjoy the higher level of responsibility that came with the role and nature of the work - we're involved in day to day infrastructure operations and tech decisions need to be made carefully to not screw anything over, ability to gain much more breadth in my role, get very strong technical skills, getting to know the nuances of the industry of my current employer, and being around tech experts, I really miss some aspects of consulting and I'm also worried my career trajectory could be potentially slowed down.

I really loved the constant rotation of projects - some focused on really niche aspects of the cloud, variety of tasks (yeah even the ppts...), hybrid nature of the work, exposure to execs, clients PoC's/workshops, and sometimes building things from scratch. I think that also helped speed up my career trajectory a ton.

I'd like to stay in my current role for a bit, because the pay is great, and I really want to gain more breadth in my domain, but I cannot see myself growing into a ultra senior tech role in the industry/become a tech lead.

My ultimate goal is to combine tech with soft skills and get into management or solutions architecture.

In consulting, there was a ton of networking opportunities, management skill building just happened organically, because that is the requirement for higher levels.
I saw some of my supervisors just exit to former clients or using those networks at Director level. In a technical industry role, I feel that those opportunities are kind of limited or slowed down and the higher you go the more pigeonholed you get into pure tech roles.

I also don't want a pure remote job forever - I really miss the professional face to face interactions and also feel it's harder to build business relationships without meeting in person.

Thinking of ways to get the best of both worlds for now and next steps to take to stay on track.

I've had an idea to go to tech conferences/meetups and try to talk to other departments at my company and try to get into some BD/sales work to substitute for the aspect i liked about consulting.

What else would you recommend me to do to stay on track?
I guess going back to consulting doesn't make sense for now since I would still be at consultant level, not gaining tech depth, and lower pay.

Thanks for your insights!

https://redd.it/xzj9dt
@r_devops

DevOps Career

Hello everyone, I am trying to become a DevOps. I have an engineering degree, a bit of programming knowledge, Cloud Practitioner, Solutions Architect Associate, and Developer Associate certificates from AWS. What do you think I need to learn to be able to find a job as a DevOps?

https://redd.it/xzm3qt
@r_devops

What Certifications to do ?

Hi I recently passed AWS SAA exam and now thinking of doing another one in order to get into devops role maybe a junior DevOps engineer. My background is 5+ yrs in IT QA engineer. While keeping my job I am thinking of preparing and giving these certifications:


1. CKA

2. Terraform associate

3. Docker associate


please advise me if its a right thing to do

https://redd.it/y11l7q
@r_devops

How to learn DevOps being broke

Hello folks!

Not sure if anyone already asked this, but today I was talking with a friend and she's trying to find her path into SRE positions, but the openings always ask to have knowledge (and some experience) around some of the big cloud providers.

&#x200B;

As we're from a third-world country (hello from Argentina) paying services like AWS/GCP and even DO can be pretty hard for someone that lives with the exact amount to survive.

&#x200B;

So here is my question, is there any way to learn how to use these cloud providers in a cheap way?

https://redd.it/xz8lh4
@r_devops

Is devops really fancy name for sysadmin in many companies?

Is it true?

https://redd.it/y14d3z
@r_devops

Question: Docsets for SQL Server and other proprietary programs

Hi, i am looking for docsets for non open software. Docsets are offline documentations for tools like zeal or dash. Does anyone hast a tipp or source? Thank you very much!

https://redd.it/y15faz
@r_devops