CODEGURU profiler - heap usage not accurate?

Im profiling some code with CodeGuru because I suspect a memory leak. The app running grows to over 10GB heap (java) but when I check in CodeGuru it says its only using 28 MB.

my profiler setup within the code is

new Profiler.Builder().profilingGroupName(groupName).withHeapSummary(true).build().start()

are there any tips on how to get a more accurate reading of memory?

https://redd.it/z9yvus
@r_devops

Why do people use so many abbreviations?

It is not devops related, but it sucks! I am in the industry for 12 years and i used and learned so many god damn abbreviations. I mean sure if you are working on in your team on a certain cloud vendor or a certain tooling, most likely everyone understand what service you mean, but if you fire those abbreviations to some one who‘s not into it, the person will most likely not understand what you are talking about and in most cases are not even able to google it, because that abbreviation has 20 wikipedia entries with not relation to what you are referring to.
Most people can type faster then they speak and for sure type faster then thy think, so why make life harder by using stupid abbreviations to outsiders? It doesn‘t make you look smarter, it is just annoying.

https://redd.it/z9ucvr
@r_devops

testing pyramid doesn't quite work for infrastructure

Active monitoring is cheap, there are many options and you should be using it, despite being at the top of the testing pyramid! https://youtu.be/9kDTAnRmeic

https://redd.it/zaehd0
@r_devops

unable to resolve Kubernetes cluster API URL dns

I ran my Kops create cluster on aws. This was working fine 2 hours back but all of sudden I get this error on running Kops validate:
unable to resolve Kubernetes cluster API URL dns: no such host

https://redd.it/zae390
@r_devops

A doubt on the env / secrets process

Hello folks

I am new to this company where they are using Gitlab CI Variables to pull the env variables. This method is done via a Gitlab Auth Token and is being used for all environments including local development.

This is the first time I am seeing this kind of a workflow. I am not a DevOps person, but I understand this side of things (a bit).
What can be the pitfalls of this approach and what is a better approach ?

​

Thanks

https://redd.it/zah5ye
@r_devops

Handling software lifecycle and CD with devices in client premises

Hey guys,

I am very much used to and spoiled by all the cloud providers, all my experience regarding automation lies with them...

​

Now I am faced with an issue: I have an artifact (think .NET or Java), and I need to deploy it to multiple devices located in different client premises. I also need some way to handle software updates in the least painful way. No k8s!

​

What are the best practices for CD in this case? In my mind there should be a centralized CD server that can connect to all the machines and do the rollouts.

https://redd.it/zaip7t
@r_devops

Taking Your DevOps Tools to the Dark Side

DevSecOps 2022 - Conf42 presentation yesterday on taking DevOps to the Dark Side from Mike, the Head of RAV (Reliability, Automation, Visibility) at NetFoundry, by using the open source solution OpenZiti.

https://youtu.be/T8UjctrFPAk?list=PLIuxSyKxlQrAhxoNbHJutmUl6RYr8vD6O&t=114

https://redd.it/zal3ov
@r_devops

How can I access VPN protected repo through ArgoCD?

I have a repository on Azure DevOps which can only be accessed using a VPN. Can someone please tell me the best way to connect to that repo from ArgoCD?

https://redd.it/zakpq7
@r_devops

Python for Devops learning dilemma

How much Python is one expected to know for a DevOps role?

I'm trying to get the most bang for my buck. I want to effectively but efficiently study Python for a Devops role. But I don't have much time. At the end of the day, I'm looking for the most practical approach. The one that will put food on the table.

 

I've read past posts and they seem to mention the book "Automate the Boring Stuff" as the go-to recommendation for Python.

But if anyone read that book before, the author pretty much states in the intro that it's a book for office workers, not for learning programming, and that if the reader's intention is to learn good software engineering, they won't get it from that book.

So I'm kinda confused. I thought good programming ability was an important aspect of DevOps? If so, then why are people recommending the book if it teaches basic automation more than programming principles?

 

As a fresh university graduate wanting to get into devops, my dilemma is:

1. Should I be studying python as a tool for automation (i.e. using the recommended book mentioned above or the Google IT Automation with python course) or

2. Should I be studying python as a means of programming and software engineering (in that case, what kind of resources should I be looking at? ) I read a previous top-post on this sub talking about an Amazon interview where they were asked about Data Structures and Design Pattens.

 

Note: I've taken several CS courses in my uni that taught the fundamentals of programming, but it was in C++ and I honestly can't remember much as I haven't used much of the knowledge since, given that I'm an IT major.

https://redd.it/zajzge
@r_devops

How to Get Started in DevOps: Tips and Advice from a DevOps Engineer

As a devops engineer, I'm often asked for advice on how to get started in the field. Devops is a rapidly growing and exciting field, and there are many opportunities for those who are interested in working with computers and systems.

To get started in devops, you'll need to have a strong foundation in computer science and programming. You should be comfortable working with a variety of languages and technologies, and you should be able to learn new tools and techniques quickly. In addition to technical skills, you'll also need to have good communication and problem-solving skills, as devops engineers often work closely with other teams and stakeholders.

Once you have the necessary skills and knowledge, there are several ways you can begin your devops career. Many companies have dedicated devops teams, and you can search for job openings in your area. You can also consider working as a freelancer or contractor, which can be a good way to gain experience and build your portfolio.

Another option is to join a startup or a small company that is looking to build out their devops capabilities. These companies often have a more agile and flexible approach to devops, and you can have a greater impact on their systems and processes.

Regardless of the path you choose, it's important to stay up-to-date with the latest developments in the field. You can do this by attending conferences and meetups, reading blogs and forums, and joining online communities. You can also continue to learn and improve your skills by taking online courses and certifications.

Working in devops can be challenging, but it can also be incredibly rewarding. You'll have the opportunity to work on complex and interesting projects, and you'll be able to make a real difference in the way that systems and processes are managed. If you're interested in starting a career in devops, I encourage you to take the first step and start learning and gaining experience. Good luck!

https://redd.it/zap4w1
@r_devops

Monthly 'Shameless Self Promotion' thread - 2022/12

Feel free to post your personal projects here. Just keep it to one project per comment thread.

https://redd.it/zaspwq
@r_devops

Timescaledb to Thanos

Testing out Thanos to replace timescaledb in our Telegraf + Prometheus + TimescaleDB... stack.

Is it absurd to think that the design could be adding the following?

* sidecars to exiting prometheus docker compose setups
* a remote docker compose setup for minIO server on its own (new host)
* another remote docker compose setup for the rest of the desired Thanos components (new host)

My disconnect is specifically with the sidecar and how it can effectively communicate with the container services on the remote 'Thanos' server.

https://redd.it/zauqj4
@r_devops

Career pivot - Dev -> DevOps

I have over 10 years of development experience with a mix of .Net, Node.js, Angular, SQL etc and in my current role working as a devops engineer writing terraform and powershell scripts, implementing ci/cd and automating cloud deployments and creating and managing kubernetes clusters in cloud. I seem to be enjoying this better and planning to continue in this path. My question is what the career trajectory is for a devops engineer at this point in their career? Should I go the cloud architect route? Whats skills should I concentrate on developing? What am I giving up and what am I gaining? In my future job will the hiring managers see my development experience as a plus for the devOps role. Also how does an interview for a devops engineer compare to an interview of a full stack developer (which can be draining with leetcode, live coding etc) with same amount of experience?

https://redd.it/zawjjh
@r_devops

Snapshot Error

I am running into issue with a python script to Prue old snapshots. The error is saying a "snapshot doesn't exist" but when I check the AWS console I can still find it. I could pass another condition to by pass this error but trying to understand why it would happen in the first place.

This is the code I am using.

import boto3
from datetime import datetime

client = boto3.client("ec2")
snapshots = client.describesnapshots()


for snapshot in snapshots['Snapshots']:
a = snapshot['StartTime']
b = a.date()
c = datetime.now().date()
d = c-b
try:
if d.days > 365:
id = snapshot['SnapshotId']
client.deletesnapshot(SnapshotId=id)
except Exception as e:
if 'InvalidSnapshot.InUse' in e.message:
print( "Skipping This snapshot")
continue

https://redd.it/zau5k5
@r_devops

Incident review: Intermittent downtime from repeated crashes

Sharing a post-mortem of a recent incident where our app went down repeatedly due to a poison pill event in our async workers.

https://incident.io/blog/intermittent-downtime

Hopefully nice from a general incident coordination perspective, and gives a picture of how we run things. Also talks about the mitigations we took, which are generally useful if you run similar services.

Feedback welcome, of course!

https://redd.it/zater3
@r_devops

What is the correct approach to implement scheduled pipeline?



So basically, let's say we have project of end2end tests called A-framework
. This project has a pipeline that will be run when it receives a commit on the master branch and that pipeline will:

1. Build and push a docker image with the latest hash commit as the tag of that image
2. Package and push a Helm Chart with the docker image that we just built

And on the other hand, we have a project called A-pipeline that has a pipeline that will run the A-framework tests every 30 minutes on a Kubernetes cluster, using Helm. Currently, this pipeline will always use the latest commit of the A-framework project ( by cloning the A-framework project and extract the latest commit hash of the master branch, then run it in the Helm upgrade command ) and I got a task to make it possible to rollabck to a previous version.

The logic I came up with is using the values.yaml file in the A-pipeline project. If there is an imageTag field specified with a hash commit, it will be assumed that it's the hash commit of A-framework that we want to use for this scheduled tests. If not, the pipeline will just run the latest commit of A-framework.

I got a counter proposition from the reporter of the ticket because it's 'time consuming' to keep cloning the project A-framework and look for the latest hash commit each time we are using the latest version and instead he came up with a complex workflow I didn't understand.

He wants the change to be made in the A-framework pipeline, not in the A-pipeline, by pushing the new Helm Chart with the hash commit specified in the values.yaml in A-framework and have A-pipeline always use the latest Helm chart. And in order to determine if we are using the older commit, specified in values.yaml or the current commit ( latest one ), we have to look for the image in the private repository ( What image ? I don't know. He said it's the one in values.yaml but in that case it's always a rollback and that version will always be used ). If it's found in the private repository, then it's a rollback, the Helm Chart should be bumped and the A-pipeline must use that version. Otherwise, if it's not found, then it's a rollout and the current image being built should be used. Now, when it's not found, then the only possible scenario is that it's a commit in a branch that has not been merged yet, and in that case, have the Helm Chart pushed with the latest commit... a hassle and not even clear, at least for me.

Honestly, the second one is confusing and it doesn't really make sense for me. We are delegating the decision of the choosing the version of the scheduled tests in A-pipeline
to A-framework, which is a repository of end2end codes and its pipeline should stick to building and pushing the image and the Helm chart new version after each merged branch. Why would the first approach be rejected despite it being simpler, clearer and easier to implement ? The Gitlab repository is not even that big to clone. It will be run on Gitlab-CI and the gitlab-ci-runner will be purged after it's done... no memory or time issue...

Any thoughts please ?

https://redd.it/zb22bv
@r_devops

Regarding a question i was asked in my recent interview!

I was interviewing for a company for a SRE role and i was asked this question.

1) How will you 'productionise' a monolith application deployed on a single VM manually? Assume there is no CI/CD, infrastructure automation, scaling etc. in place.

2) How do you go about making systems reliable? Target is to go from current availability(Unknown) SLO to 99.5 percentile?

I am quite curious how would the members of this community would answer this question.

https://redd.it/zaq73j
@r_devops

Managed Kubernetes with zero nodes

Hello! I would need a managed Kubernetes service that would allow me to have the cluster always running with minimal costs. Basically what I need are some web apps I'll only use for myself and I want the cluster to be up and configured and only scale up if/when needed. I understand there must be a master node, but most cloud providers don't allow you to have a very small (minimum price) master node and then more expensive node pools. Pretty much I would need to have a minimal price on the cluster when I'm not using it and pay for it when using it. Couldn't really find any cloud provider that offers that, but maybe I was searching wrong.


Oh, and I won't care if the compute is more expensive for this service, as the cluster will more often than not be in idle, I'm fine if when I use it, it is a bit more expensive. Thank you!

https://redd.it/zakopk
@r_devops

Am I Misunderstanding Trunk Based Development?

Maybe a dumb question, but I'm trying to convert my team from GitFlow with branches per environment to something like trunk-based development but still possibly with feature branches.

I've been asked to present a comparison of the two flows for a developer, which has me researching and trying to ancipate concerns.

As a result, I have a couple questions:

1) I've read that a build should be made for each PR and tested. Does that mean it is assumed that the dev has immediately before pulled the latest changes from main into their branch? If they don't do that, and the build runs and succeeds, the tests would have operated on an "older version" of main, so do the build and tests then run a second time and therefore take longer to give devs feedback? Would I want a pipeline step to automatically pull latest main changes into the feature branch before building?

2) In the tag promotion method, a commit is subsequently given tags/labels like dev, stage, qa rejected, production etc. As I understand, this replaces a git branch per environment. So how well does that work with many many devs working off the same trunk? Let's say Carl commits to main and it gets tagged as dev. He checks dev to make sure things look right, then hits approve on the pipeline and it gets promoted to stage for QA to review. But then Lacey commits to main with the dev label and approves it and now QA has two commits to review the functionality of but both are in the middle of their own pipelines and QA doesn't easily know which one came first. Which means we know how Lacey's code works alongside Carl's but not the other way around. Am I missing something here that makes this work with dozens of devs?

I appreciate any insight. Still researching but hopeful to get some expertise or detailed articles that address these types of issues.

https://redd.it/zb8d45
@r_devops

Best practice around maven pom files as devops?

I recently got a project to work on a pipeline for azure function apps and this is my first time working from the devops side of things. I create a pom file for a function app that has the base dependencies and I am able to build a project zip it up using assembly plugin via maven. This pom should work for all function apps and the devs should just have to add their dependencies. Now it could become more complex depending on the project. My question is what is the best practive around this as devops? Do I just tell devs to copy my maven file in their project in their dependencies? Can I upload my pom file to artifactory and have my pipeline pull it down and then have devs create a pom that calls my parent pom? If I follow this apporach how do I prevent them from overriding dependencies from my pom? Trying to get ideas around best practice. Please let me know if you have experience around this.

https://redd.it/zb7q14
@r_devops