3 years in Devops, still need help from seniors to get work done.

I've been in Devops for 3 years. It's my first job in this industry. I'm on my second job, been here a year and my current project for about 5 months. Its a devsecops project.

There's a lot of moving parts in this project and I still find myself asking the seniors for help and guidance and I honestly hate it. I pick up a ticket in the sprint and can't seem to complete it on my own as there's lots of moving parts.

For example, if I want to update some TF code I need to be careful of the dependencies and implications. Hence why I need senior help. It was the same at my first job but back then I was still junior so I still had that excuse. I've been bought in as a mid-level engineer for this project but I don't feel like I'm operating at that level.

I know you always hear that you never know enough in IT and that your always winging it but I feel constantly needing help is not good. What are things people did early on in there careers to level themselves up.

The project stack is AWS, Terraform, Team city.

How do I become more autonomous and be less reliant on others?

https://redd.it/xy8ljw
@r_devops

How to get the current date in the make file in windows

This line is working fine in bash script

date +"%y-%m-%d"

But I cannot get the current date in a variable in a make file like this

currenttime = $(shell date +"%y-%m-%d") or currenttime = $(date +"%y-%m-%d")

Please help me make it work. I tried escaping the % character by adding another % or / in front of it

https://redd.it/xyd138
@r_devops

How Pytorch team at Facebook tracks CI/CD metrics at scale

Interesting read from a couple of Facebook engineers about tracking and sharing CI/CD metrics at scale: "The scalability problem now really hit us. Remember all the decompressing and de-aggregating and re-aggregating that was happening for every JSON? We would have had to do that massaging for potentially hundreds of thousands of JSONs"

https://rockset.com/blog/pytorch-infra-journey-to-rockset/

https://redd.it/xyejie
@r_devops

I see the Drone CI extension on Docker Desktop. Does this mean we can do CI from our laptops?

I just upgraded my Docker Desktop and I see the Drone CI Docker Desktop extension.

I think this is a big thing as we can do CI from our laptops now just by installing this extension in a minute.

The official information about this extension from Docker is here: https://hub.docker.com/extensions/drone/drone-ci-docker-extension

https://redd.it/xy28o1
@r_devops

Help finding an article about not having metrics displays in the office

I'm looking for an article that I read a while back where the gist of it was that there were no metrics dashboards in the office and instead (if I remember correctly) they completely relied on that the alerting system would let them know if something was going wrong. I think this blog was talking about the culture at Netflix but I can't be sure.

Anyway, I can't for the life of me find this article now and was hoping that somebody might be able to help me dig it up.

https://redd.it/xxwla5
@r_devops

Debugging semantic release not detecting older tags

Hey,

In this article, I walk through debugging your git repository to recover from the situation where semantic-release doesn’t detect older tags.

https://alimoezzi.medium.com/debug-semantic-release-not-detecting-older-tags-b7653caf60db

https://redd.it/xy4gsh
@r_devops

ECS tasks don't start on new Cluster Instance HELP

Hello, folks!

I need to scale our solution but i 'm running in a weird problem. I changed the Auto Scaling Group to have two ec2 instances running instead of just one, but the new ec2 instance inst running any of the services defined in the cluster.

Things I've already checked

The instance is attached to the cluster
The services are all running (all tho in tasks in just one instance)
The new ec2 instance have the ec2 agent running
I'm almost sure my Launch configurations are fine. Because before this when I had only one instance, if I killed the instance another one would spawn with running services.

What do you think could be problem?

https://redd.it/xy3xy3
@r_devops

Guideline for distribute apps on IBM Cloud catalog

Does anyone have experience distributing your apps on IBM Cloud catalog ? https://cloud.ibm.com/catalog.

If yes, would you please share your experience? Some documents for self-research also helps, since I have been struggling to find any resources to do it myself.

https://redd.it/xynbn3
@r_devops

Is devops-guy the next generation of anykey-guy?

N.B. This post is about 'devops-guy' job, not about 'devops syntergy bla-bla-bla', and it's not about deveopers with ssh access to production.

My first job was 'system administrator assistant', which is the person to crawl under table to fix cables, to replace cartridges, to help with 'odd windows', etc. They called this job 'anykey'. Person to come and to help user react properly on message 'Press any key to continue'.

20 years later, I'm venerable devops guy shoveling yamls like pro.

There is hype, and aura of decent money, few offers from HRs, etc, etc.

But! Recently someone said that 'devops is anykey'. And it start to crawl in my mind. I see all symptoms for 'anykey'.

1. Called when 'something is oddly broken' in CI/CD
2. Not participating in core company activity (there are programmers who are dedicated to that)
3. Tasked with half-routine half-creativity thing 'make this work'.
4. There are other people on the team with not-the-less qualification but with 'other goals', so 'devops-guy' is hired for reducing routine stress from other personnel.
5. Invent specific rules to enforce and acts as a 'enforcing proxy' for others (no, you can't do latest on production), which are important, but trivial.
6. Always works on verge of incompetence. Are you really knows npm, maven, cargo, mezon, poetry, dh_python, opam, cabal? Really-really? Do you even knows all programming languages they use?
7. Invent small-scale automation to solve local problems (yes, I wrote automated print-server enrollment script in 2004, I just finished writing workflow generator...)
8. Extensive vendor knowledge for AWS/GCP/Azure is as important as knowing about 'Federated forest of domain trees' in AD. Super-duper important until no one need it anymore.

The more I write those, the more I believe it's true. It's well-paid now (but not much as some solidity-geek), but its rapidly been get fixed by the industry. I feel that the whole 'devops guy' thing is industry reaction on imperfections of the existing tooling. Tooling get better, devops-guys get less of importance. It required a skilled 'anykey' to install workstation in 2002, now people don't understand 'skilled' part, because "it's just works", and "everyone knows how to connect to wifi nowaday".

So, I don't see any sings of obsoleting in next 5 years, but I'm pretty sure, that 10+ is going to be very much different.

https://redd.it/xynxqo
@r_devops

How to test Kubernetes api?

hello there,

i have a web app that runs a lot of background jobs on Kubernetes. in our code we connect to Kubernetes and run a job on it.

i want to test this on our development setup. can i run a mini-kubernetes or something on docker to mock this ?

what do you use for this kind of scenarios ?

or should i just create another cluster just for testing things out ?

https://redd.it/xyp9ai
@r_devops

How to upgrade your Helm Charts?

Just a short article on upgrading your Helm charts using the helm diff extension

https://renjithvr11.medium.com/how-to-upgrade-your-helm-charts-2856afb5d400?sk=547e1562762c56639fe8e47d3258e116

https://redd.it/xy1wha
@r_devops

SSL Cert issue Ubuntu EC2 Wordpress site

Hi all

Im having some issues installing an ssl cert on my wordpress site hosted on a ubuntu ec2 instance

the site works but is not secure until i add the following to my /etc/apache2/sites-enabled/wordpress.conf file

<VirtualHost \*:**443**\>

DocumentRoot /srv/www/wordpress

**SSLEngine on**

**SSLCertificateFile /etc/apache2/certificate/apache-certificate.crt**

**SSLCertificateKeyFile /etc/apache2/certificate/apache.key**

<Directory /srv/www/wordpress>

&#x200B;

<VirtualHost *:443>
DocumentRoot /srv/www/wordpress
SSLEngine on
SSLCertificateFile /etc/apache2/certificate/apache-certificate.crt
SSLCertificateKeyFile /etc/apache2/certificate/apache.key
<Directory /srv/www/wordpress>
Options FollowSymLinks
AllowOverride Limit Options FileInfo
DirectoryIndex index.php
Require all granted
</Directory>
<Directory /srv/www/wordpress/wp-content>
Options FollowSymLinks
Require all granted
</Directory>
</VirtualHost>

once those changes are made and apache is restarted my ip no longer shows my wordpress site but instead just shows the default apache page.

root@ip-172-31-2-98:/etc/apache2/sites-enabled# sudo a2enmod ssl
Considering dependency setenvif for ssl:
Module setenvif already enabled
Considering dependency mime for ssl:
Module mime already enabled
Considering dependency socache_shmcb for ssl:
Module socache_shmcb already enabled
Module ssl already enabled
tcp6 0 0 :::80 :::* LISTEN 30744/apache2
tcp6 0 0 :::443 :::* LISTEN 30744/apache2

Any help would be appreciated. Thanks in advance.

https://redd.it/xyso1c
@r_devops

DataDog Custom Metrics

Hey

I'm currently using DataDog Cluster Agent installed via helm Chart in Kubernetes and Metrics are scrapped by the Agent. 

I have checked multiple docs from DataDog about the custom metrics and how it affects the Billing, and I'm confused. Anyone knows what's the best way to avoid sending metrics to be counted as custom metrics or how to reduce the amount of them? What is the native protocol to send metrics to avoid being counted as custom metrics? Is it a DagStatsD integration? 

I was looking here:

1. 

>This client automatically injects telemetry about itself in the DogStatsD stream. Those metrics will not be counted as custom and will not be billed

https://docs.datadoghq.com/developers/dogstatsd/high_throughput/?code-lang=go#client-side-telemetry
https://github.com/DataDog/datadog-go#monitoring-this-client

2. 

>If a metric is not submitted from one of the more than 500 Datadog integrations it’s considered a custom metric(1).
A custom metric is uniquely identified by a combination of a metric name and tag values (including the host tag).

https://docs.datadoghq.com/account_management/billing/custom_metrics/

3. But later when I open the integration page for example OpenMetrics I find it confusing

>All the metrics retrieved by this integration are considered custom metrics.

https://docs.datadoghq.com/integrations/openmetrics/

4. It seems that the way to reduce the volume is by using Tags configuration.

>Note: Only configured metrics contribute to your Ingested custom metrics volume. If a metric is not configured with Metrics without Limits™, you’re only charged for its indexed custom metrics volume.

https://docs.datadoghq.com/metrics/metrics-without-limits/

https://redd.it/xyugd0
@r_devops

Moving beyond solo scripting into CICD

I currently work in the networking field of IT operations. My background is in IT, not computer science. That said, I have learned a couple of languages along the way as I've scripted a lot of components of my roles. I primarily use Python (Nornir and custom-fit REST APIs), as well as Ansible CLI.

How do I learn CICD, or Git, or how to be collaborative when most of my automations are solo endeavours? In the networking world, I'm one of very few who knows any programming at all, but I'd like to understand how to make these developments collaboration focussed moving forward so I can "graduate" from a "script developer" to a something closer to holding a DevOps hat.

Right now, I occasionally use Git to a very small degree when developing my automations to track my changes, milestones, feature adds, etc. That's it. No hooks, no automated unit testing, etc. These have been topics that I just haven't learned yet as programming hasn't been my primary skill, but rather associative to my networking studies.

https://redd.it/xy0k62
@r_devops

How do you block any ticket merge when the unit tests don't run in a Github repository?

I am guessing this is done through a Github action, but I would like to know how the solution looks like to implement such a CI/CD solution to prevent faulty code from being merged into production.

https://redd.it/xxpfrs
@r_devops

Schema migrations on RDS from github actions

We run a number of laravel apps that require schema migrations on deployment from time to time. These RDS instances are in private subnets so obviously github actions can't reach them.

Our current solution is for the GH workflow to trigger a jenkins task from a jenkins machine we have in the same vpc.

Not particularhappy with this solution (although it works fine) and wondering if anyone has any other suggestions or ideas on how to approach this

https://redd.it/xz4gvm
@r_devops

DevSecOps Azure

Goodnight guys! I just had a 1:1 with my manager and we talked about career progression since I’m in my early career. I work at a consulting firm and you can play many roles. I work under the azure security team and wanna do DevOps. In the future I would like to be a solutions architect. He gave me some topics I should like into and get comfortable with Azure Bicep & DevOps, Terraform, and Python. He told me I can use Python as a wrapper for Terraform, I’ve been Googling and I don’t know what exactly that’s means any suggestions? Links? Videos? Guides? On how to do that. Was also thinking about doing that coursera Google python automation course to get my python skills up.

https://redd.it/xz88j3
@r_devops

Great resource

https://github.com/bregman-arie/devops-exercises

https://redd.it/xz9sa7
@r_devops

How do you separate your vpc network?

What are the factors you're looking at for a microservice application? Below are some of the things that comes into mind:

* Each application has its own VPC network
* Everything resides in the same VPC network but each application is separated using different sets of subnets
* ...

https://redd.it/xzc9dz
@r_devops

Please suggest infrastructure for a SaaS app

Hello,

I'm here looking for help/suggestions after lurking around a lot.

I've created a web app which consists of the following components (docker containers):

|App|CPU (limit)|Memory (limit)|
|:-|:-|:-|
|backend PROD|250m|512Mi|
|2 x workers PROD|250m|256Mi|
|API PROD|250m|512Mi|
|backend DEMO + API DEMO|250m|256Mi|
|2 x worker DEMO|250m|128Mi|
|3 x JS apps (js, SSR, static)|200m|256Mi|
|helper|200m|256Mi|
||||

Right now they are all hosted on a Linode kubernetes cluster connector to DigitalOcean managed DB.

Why? I wanted to learn a little bit of kubernetes and needed a secure place to store my data (with backups and point in time recovery if needed).

But I'm looking for alternatives, the cost of all this is almost $80 per mo (cluster, db, load balancer).

I need a place where I can just throw a dockerfile and in return I get a domain.

I've managed to move JS apps to Vercel hobby, which is more than enough for me. But I still have the other components.

What would you do to lower the cost of this and also managed less infrastructure? (I don't wanna deal with VPS upgrades, hardening, patches...etc)

&#x200B;

PS. currently looking fly.io , they have some cheap VMs I can use for workers ($2/mo) and even some free ones. I'll make some calculations.


LE: JS apps were moved to vercel

&#x200B;

Thank you.

https://redd.it/xxucvm
@r_devops

Choosing the most convenient software for my business

Hello there. I just started testing the Azure DevOps platform. It looks great but from experienced users in these platforms, how to decide which one would suit my business more? Also, what things should I consider when I choose a DevOps platform? I use Azure Cloud for my work and GitHub, Gitlab, Kubernetes, and Docker. I tried Jira and ClickUp apart from Azure DevOps (a longer time than AzureDevops). I want to establish my startup in one single DevOps platform.

https://redd.it/xxtx8r
@r_devops