Intermittent DNS issues in EKS

So I've got a simple pytest that deploys a curlimages/curl pod, execs into it and sequentially curls several AWS endpoints (like 'https://s3.us-west-2.amazonaws.com' etcetc).

It mostly works. However, sometimes it doesn't, and that's fun. Sometimes curl returns 'Could not resolve host' - for different endpoints, on different nodes etc.

I've tried many things and none have helped. Some coredns tweaks (using EKS coredns addon, EKS 1.25 if that helps), using an ubuntu image instead of curl based on alpine (read some scary stuff), looking into coredns logs (failed curls don't show up there so I guess coredns is off the hook?), adding some sleeps between execs, using FQDNs ('https://s3.us-west-2.amazonaws.com.', adding more coredns pods (up to 10 on a 10 node cluster)...

More than that, I can't reproduce this manually (i.e. running multiple infinite loops of kubectl exec -ti curlpod -- curl -vvv url doesn't produce errors **at all**). That's the strangest thing - both pytest and manual kubectl execs run locally from my machine, and pytest fails fairly consistently, while manual repro doesn't, at all.

Y'all, I hate computers sometimes.

https://redd.it/12unh2a
@r_devops

Hey guys, just landed a gig as a DevOps release engineer! I'm super stoked but also pretty nervous. Any seasoned vets out there have any tips or advice for a newbie like me? Would love to hear your thoughts!

Hey everyone, I just got a sweet job offer at a company that makes iOS and Android apps. Despite not having many options in these tough times, I accepted the offer and I'm excited to get started.

The company's infrastructure is all on-premise, with Jenkins as the main tool and some Python scripts thrown in to maintain the apps. My main task will be writing pipelines to build the apps with "xcode," but I don't have any experience with mobile app development.

Do you guys have any recommendations for materials I can use to get up to speed before I start next year? Any fellow release engineers out there with advice for a newbie like me? Thanks in advance!

https://redd.it/12ubjcr
@r_devops

I’m looking for my first engineering role and I feel like I’m being marketed for roles I’m not qualified for

Quick background about my skill set
* Work at a large company in a non-technical IT team
* I have all AWS associate certs but I’m not a great coder (basic-intermediate Python and HTML/CSS). I’ve done a few small projects (ex. Cloud Resume challenge)
* I use Linux often and am relatively comfortable with it and some very basic bash

Quick overview of my company IT teams
* We have teams of Tier 1-2 Application Engineers across IT that own 1-2 web apps. They basically just code and use CICD to push small stuff through with a handful of AWS services. Nothing flashy. This is what I envisioned my first step being.
* We also have mixed teams (Technical Project Managers, Scrum Masters, and often just one tier 2-3 Cloud Engineer). These engineering roles are closer to DevOps with Docker, Ansible, Kubernetes and seem quite advanced. This is what is being marketed to me.

My issue - I’m being pushed towards the latter (mixed team) based on my manager’s networking and opportunities arising in my department/subdivision. He seems to think I can do it. The hiring managers seem to think so too, but I really don’t. I envisioned my first engineering role to be on a team of others I can learn from. Instead, it seems like everyone wants me in a team where I’m alone to build things by myself. This is really daunting to me as someone who can barely code. Yeah I know a lot about AWS, but I’ve never even deployed to prod before. I’m going through courses now for front end to complement my Python but it’s going to take me months to be even be intermediate. I can’t help but feel like all of these people are overestimating me. My manager talks like a Tier 1 Application Engineer is beneath me (which sounds crazy to me).

Are they overestimating my ability like I think they are? Or am I underestimating myself?

https://redd.it/12uaax5
@r_devops

How can I pass a terraform variable in to a custom _data script block in a terraform file

Please accept my apologises if I explain incorrectly I’m a self taught DevOps engineer coming from a traditional windows infrastructure engineer background got here with some guidance’s from this page. I have a script that installs a application on a azure Linux vm silently with a configuration using terraform ..I’m using custom_ data this works great but it can only silently install the package with me giving it the password for service account , instance name, etc. This is all sensitive information that I’d rather the bash script gets from variables in the variable.tf file. Is there any way to pass through these variable in bash script. For example the password for the service account in bash script i’d like to use my variable.tf file to pass on to bash script which would be var.password. Hitting a dead end try to add terraform variable from variable.tf to my script.sh file using custom_data ..Happy to hear any other elegant way to achieve adding configuration like this .. I have terraform ,azure,GitHub actions to achieve this task unfortunately I don’t have ansible or cloud-init to work with. Any help would be appreciated

https://redd.it/12uvqeu
@r_devops

Advice after failure

Failed my apprenticeship in the uk

I was a devops apprentice and I worked in cloud.

I tried extremely hard to pass. I worked really hard on my project and created a good presentation.

The feedback says I didn’t show that I met the knowledge of monitoring, CICD, and unit testing.

In the presentation I talked about installing monitoring tools (azure log analytics with terraform), I talked about configuring notifications argocd notifications and I debugged them with logs showing the credential had expired. I showed the CICD process and I talked about unit testing with kustomize and validation scripts.

It seems like the assessor was extremely harsh or got confused during my presentation because it wasn’t clear enough. In the report they don’t mention any of the above they just hold onto the fact that some of the stuff was broken / being tested by other people at the time of the demo even though I showed previous runs from when it was working.

They’re telling me to resit it I have to pick a brand new project and spend 3 months preparing.

I already was offered a full time position which I started a month or two ago.

I guess I’m just looking for advice.

1) Do you think this is going to affect my employment in any way?

2) Should I try to keep it a secret at work?

3) Should I try to retake it with a new project and spend another 3 months on it or focus on new things?

The qualification was going to be a Level 4 apprenticeship.

Thanks.

https://redd.it/12uw3wp
@r_devops

Building from source or not?

Wrote up a counter argument to "Build Once, Deploy Anywhere" artifact promotion here:
https://dabase.com/blog/2023/build-from-source/

Perhaps I'm being naive about:
A) Build + test can be fast
B) Builds can be reproducible?

https://redd.it/12uy6q4
@r_devops

Terraform giving 403 AuthorizationFailure after accidentally deleting the private endpoint to a storage account

I added a wrong configuration (multiple subresource names for an endpoint) and applied instead of planned. My old private endpoint got deleted and now I get this 403 error whenever I try to reapply with the good configuration.

I am applying it from a github workflow (that I did not create because i am a beginner). Can anyone give me a suggestion? I also tried creating the endpoint manually, but for some reasons the organization rules i am under did not allow me to add manually a private dns zone. So now i get Failure sending request, status code = 0, context deadline exceeded. Help please?

https://redd.it/12u3kdh
@r_devops

Question How to approach a system architecture design problem when needing to balance expandability and development

I have a project I want to take seriously which involves designing an app that performa similarly to google maps (mostly geo queries).

I need to store data about sites as well as their position and allow searching and filtering by location (as well as other parameters).

### Some technical data:

- Support around 1,000 monthly users (similar use time to door dash or uber.

- Support around 1,000 geolocations

- Support fast image queries (around 5,000 images), editing them is rare.

- Support user authentication and authorization.


Since I’m working alone I need to build something that will serve as a prototype and would be useful to convert into a real usable system by someone more experienced than me.

### My current plan:

1) Use a no-sql database and basic geohashing to store data.

2) Store the database in a centralized server (since all locations will probably start in 1 city as a pilot).

3) Use a rest api to interact with the database (mostly due to scalability issues I might face).
Avoid micro-services for now (I’m really not sure about this so I need your help).

4) Use an SQL database with blobs to store images. Alternatively use buckets, depends on how much hard it’ll be to implement and how important it would be.

5) Use a third party service (such as firebase) to support push notifications and 2 factor authentication.


I’m a software engineer and most of my experience has been on building web apps, so I need some feedback about my plan.

- What can I give up on to make development easier and faster, while not requiring to rewrite everything later?

- Are there any things that I must do that aren’t in my list?

https://redd.it/12v00z8
@r_devops

What are good job boards for senior (ICs and managers/directors) EU-based remote jobs?

Hey all! I'm looking for information and opinions from both hiring managers/recruiters and job seekers. What are the best places to look for/post jobs for remote EU-based SRE/DevOps roles, ideally at a senior IC level and above, and for the management side, e.g. Manager, Director, VP.

Most job boards I've seen are overwhelmingly US-focused and have very little traction with EU roles/viewers. Some don't even have a way to filter only EU-based posts. The best I've seen so far is LinkedIn, but I feel like there must be something better. What say you, r/DevOps?

(and yes, I used the search, and I considered other subreddits but this one seems appropriate given this is a discussion. mods correct me if I'm wrong)

https://redd.it/12uzvdy
@r_devops

Are there things like Ephemeral Virtual Machines that can be used in CI system?

So I have prepared an Ubuntu-based VM via Packer + Ansible for our team at work. It all works smoothly, Here is the catch, the team wish to do some integration tests in the VM.

AFAIK packer in this case won't be suitable because on succesful runs, packer (with the qemu plugin) will try to generate the a new VM on the test server for me which is not what I want. I want something where I can spin up my VM, push the test dirs into it, trigger them and throw the VM out on successful testing.

The only solution I can think of at the moment is spinning the VM via a bash script with qemu-system-x64 commands and then achieve what I want.

Is there some tool that can integrate well with my GitLabCI system to achieve this ephemeral VM logic?

I thought about Vagrant but I doubt it might be a good solution since I see it more as a development tool.

https://redd.it/12v4gwe
@r_devops

Github actions conditional matrix

Hi all,

Trying to reign in the github actions runners and can't seem to figure out how to do it. Or even if it's possible
Here's a trimmed version of the workflow file

​

name: Build on Pull Request

on:
pull_request:
paths:
- 'path/test1/**'
- 'path/test2/**'
- '.github/workflows/this-file.yaml'

jobs:
build:
runs-on: ubuntu-latest

permissions:
id-token: write
contents: read
pull-requests: write

strategy:
matrix:
env:
- "dev"
- "prod"
dirs:
- dummy
exclude:
- env: ${{ github.base_ref == 'main' && 'dev' || github.base_ref == 'develop' && 'prod' }}



steps:

- name: Build
run: build something
working-directory: ${{ matrix.dirs }}
- name: Validate
run: validate something
working-directory: ${{ matrix.dirs }}

The problem is that if i want to make it more complex for other items I am having trouble understanding how to do it.
For example if i wanted to change the format to


env:
- account: "1234567890"
name: "dev"
- account: "0987654321"
name: "prod"




How can I go about getting the same behaviour. The documentation for [exclusions](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs#excluding-matrix-configurations) is extremely lightweight and deals more with hardcoded values than conditional logic.


trying to add logic with fails with my attempts.

strategy:
matrix:
env:
- account: "1234567890"
name: "dev"
- account: "0987654321"
name: "prod"
dirs:
- dummy
exclude:
- env:
- account: "1234567890"
name: "dev"
if: github.base_ref == 'main'
- env:
- account: "0987654321"
name: "prod"
if: github.base_ref == 'develop'


Any help would be appreciated

https://redd.it/12v5bcm
@r_devops

Nomad agent dev - missing drivers (docker)

Hello,

I'm desperately trying to run ***any*** job with `nomad agent -dev`. I am using nomad `v1.5.3`.

Unfortunately I get something like:

```
2023-04-22T18:12:02+02:00: Task Group "redis-test-web" (failed to place 1 allocation):
* Constraint "missing drivers": 1 nodes excluded by filter
2023-04-22T18:12:02+02:00: Evaluation "949981aa" waiting for additional capacity to place remainder
```

The information I could get was to reboot and/or restart the `docker` service (did not work). I even uninstalled and reinstalled `nomad`, but didn't work either.

So in summary:
* Rebooting didn't work
* My user is in the `docker` group
* Reinstalling `nomad` didn't work/help either

Where can I get more information ? I tried the logs (terminal output) but it didn't really help me.

I am using `archlinux`.

Thank you very much in advance for any help

https://redd.it/12vb0x3
@r_devops

Do you know, or or is it worth knowing, any front-end languages?

I know languages like Python, Bash, etc are useful for DevOps, but what about front-end languages like basic HTML, CSS, Javascript?

https://redd.it/12vd3ni
@r_devops

Observability Getting metrics for an endpoint "when in" a trace of another endpoint

Hi there.

Given I have two apps, A and B, and the controller of one endpoint in A makes a call to an endpoint in B, how can I get a metric of the amount of time the endpoint in B contributes to the latency of the endpoint in A.

Say the controller in A calls 7 times (yes! :( ) the endpoint in B. This endpoint in B is called by other apps as well. How can I know the latency of calling B 7 times introduces into the latency of the controller in A?

https://redd.it/12vg0m4
@r_devops

Finding balance on Enterprise Priorities vs Team level Priorities for new SRE TEAM?

Hello all!

We are forming a new SRE team and looking for some guidance on managing various secondary initiatives I am learning about with this new team. We are a team of 8 with a wide variety of skill sets but new to SRE space outside a few people who have prior experience in this area.


We have the VP saying to increase tech quality, velocity, and reliability enterprise wide.


We are taking over some leftover tools, we can choose our own tooling and have the ability to set enterprise patterns for SRE. We are mostly an Elk shop with some prom and Grafana. We have some stuff that was done before but half baked.


We are meeting with other app development teams and understanding the lay of the land. We have various teams asking us for some one off work or other requests like that. We are still doing support requests for the tools assigned to us and we have a few people that know them but need to get other team members up to speed.


Implementing SRE principles and writing automation to resolve biggest pain points is priority number 1.

**BUT**

What I can't figure out is how to balance the secondary stuff I know is very important as well :

* Increasing user experience within tooling
* Coding standards for team
* Removal of tech debt from tooling we are responsible and inherited?
* Building out proper pipelines for our team and upgrades to the tooling (How far do I take this?)
* Development of documentation around current tooling?
* Moving by hand infra processes into IAC?
* Refactoring previous automation that needs a revamp?

How would you all prioritize this stuff?

https://redd.it/12vi36j
@r_devops

Gitlab Project Automatically Creating Basic TF Project

In order to make a basic TF Template, we have a gitlab repo that has a .sh script that will automatically create a TF project for us and a gitlab pipeline for that project. This gitlab project that creates the projects and pipelines also has its own pipeline

My question is where/how do I run this script? I feel like this is very basic, but I am confused on how exactly to proceed. Do I run it through its own pipeline?

https://redd.it/12vjuew
@r_devops

If im making mistakes every week, should i quit devops ?

Im into this new job 5 months. Before that i have 3 YoE in devops but much less hands on work(especially coding).

This new job (top tier and much bigger system), ive been making very rookie mistakes pretty much everyweek. By mistakes i mean small things like reverting PRs, deployments getting stuck to big things like production outage.
Root cause is because im not fully aware of what im doing or i cant think ahead. (E.g. tested properly on dev but things that would break in prod). Part of it is that i am new, but there are many other new folks who dont have these issue.
Also i am much much slower at writing/reading code. Anyways in easy words, quality of work i deliver is pretty junior.

Worst part is i put a lot of effort in work. Im not kidding i work till midnight quite often and rarely finish work before 8pm. But i still see no improvement. To be really honest, i feel like i dont have the brain for cs or some thing is fundamentally different from other engineers. The feeling of getting no output from putting extensive amount of effort.. feels like i might need to consider switching careers at this point. Also this is happening every single week only to me on the team. Doesnt this mean that i have a problem? Its hard to justify at this point lol. Is this a valid reason to switch careers? Or is this normal

https://redd.it/12vkhi2
@r_devops

Can someone tell me how to fix this issue?

https://stackoverflow.com/questions/62414739/mongodb-authentication-failing-on-accessing-database-while-using-docker

​

Not sure why it doesn't work as intended and I need to create an user manually. This is really mind boggling.

​

What's the easiest way to go about this and what docker-compose and docker configs do you use to set it up? I keep getting Authentication Failed on node.js.

https://redd.it/12vjq9n
@r_devops

error no. 2002 can't connect to mysql server on localhost 10061

My primary OS is windows 10. Inside it, I use a virtual machine, namely vmware workstation pro. And I am using centos 7 minimal on it.

I've successfully installed mysql in centos 7. Now, my goal is to connect my mysql server to "sqlyog" client.

When I try to do it, I get the following error:

error no. 2002 can't connect to mysql server on localhost 10061

I've seen an exact similar issue here(https://faq.webyog.com/content/23/59/en/i-get-error-no-2002-can_t-connect-to-local-mysql-server-through-socket-.html). I edited the /etc/my.cnf file with this:

[mysqld]socket=/var/lib/mysql/mysql.socket[client]socket=/var/lib/mysql/mysql.socket

But still the same issue continues.

https://redd.it/12w57a1
@r_devops

Ping Connection Pool failed for banksmart. Class name is wrong or classpath is not set for : com.mysql.cj.jdbc.MysqlXADataSource Please check the server.log for more details.

The error:

>Ping Connection Pool failed for banksmart. Class name is wrong or classpath is not set for : com.mysql.cj.jdbc.MysqlXADataSource Please check the server.log for more details.

Here's the JDBC Resources settings:

https://imgur.com/a/St7UY1V

Here's the JDBC connection pools settings:

https://imgur.com/a/wQLZLe2

Here's the server.log logfile.


https://pastebin.com/Ju3GQtCd

https://redd.it/12w6h00
@r_devops

Free Udemy Coupon Collections

1. [2023\] AWS Certified Cloud Practitioner (CLF-C01) - Exams - https://coursetreat.com/udemycourse/aws-certified-cloud-practitioner-clf-c01-practice-exams-i/

2. The Ultimate Pandas and PostgreSQL Bootcamp - Data Analysis - https://coursetreat.com/udemycourse/pandas-and-postgresql-basic-to-advanced/

3. The Complete Pandas Bootcamp 2023 for Data Analysis - Python - https://coursetreat.com/udemycourse/pandas-for-data-analysis-in-python/

4. Start Your Own Mortgage Loan Processing Business - https://coursetreat.com/udemycourse/start-your-own-mortgage-loan-processing-business/

5. Sales For Entrepreneurs: How to Sell Without Being Sleazy - https://coursetreat.com/udemycourse/sales-for-entrepreneurs-tareq-hajj/

6. SQL / PostgreSQL Bootcamp : Go From Beginner to Advanced - https://coursetreat.com/udemycourse/sql-postgresql-crash-course-for-absolute-beginners/

7. PHP with MySQL 2023: Build 5 PHP and MySQL Projects - https://coursetreat.com/udemycourse/php-with-mysql-2022-build-5-php-and-mysql-projects/

8. NumPy Bootcamp for Data Science and ML in Python - 2023 - https://coursetreat.com/udemycourse/numpy-for-data-science-and-machine-learning-in-python/

9. NumPy & Pandas Masterclass for data analysis and ML | 2023 - https://coursetreat.com/udemycourse/numpy-and-pandas-for-beginners/

10. Mastering Data Visualization with Python using Seaborn Lib. - https://coursetreat.com/udemycourse/data-visualization-in-python-using-seaborn-library/

11. Master the Art of Closing Hard Equity Mortgage Loans - https://coursetreat.com/udemycourse/master-the-art-of-closing-hard-equity-mortgage-loans/

12. Digital Forensics for Pentesters - Hands-on - https://coursetreat.com/udemycourse/digital-forensics-for-pentesters-hands-on-learning/

13. Data Analysis with Pandas, NumPy and PostgreSQL -Masterclass - https://coursetreat.com/udemycourse/numpy-pandas-postgresql-basic-to-advanced-for-beginners/

14. Curso Completo de Bases de datos MongoDB y NoSQL. - https://coursetreat.com/udemycourse/curso-completo-de-bases-de-datos-mongodb-y-nosql/

15. CISSP Exam - Covering all domains - 150 Ques - 2023 Edition - https://coursetreat.com/udemycourse/cissp-exam-covering-all-domains-150-ques-2023-edition/

16. Build Responsive Real World Portfolio WordPress Website - https://coursetreat.com/udemycourse/build-responsive-real-world-portfolio-wordpress-website/

17. The Complete Computer Forensics Course for 2023 PRO | CFCT+ - https://coursetreat.com/udemycourse/computer-forensics-beginner-to-advanced-cfct-masterclass/

18. Online Reputation Management Guide: Rank Highly on Google - https://coursetreat.com/udemycourse/online-reputation-management-guide-rank-highly-on-google-tareq-hajj/

19. Adobe Photoshop CC For Absolute Beginner to Advanced - https://coursetreat.com/udemycourse/adobe-photoshop-cc-for-absolute-beginner-to-advanced/

20. 3 Step Innovation for Uncertain Business - Road to Davinci - https://coursetreat.com/udemycourse/3-step-innovation-for-uncertain-business-road-to-davinci/

21. Wordpress Create Your Own Website Elementor - https://coursetreat.com/udemycourse/how-to-create-a-website-with-wordpress-2022-fast/

22. The Complete, Step-By-Step course to Ace your Job Interview - https://coursetreat.com/udemycourse/interview-hero/

23. SEO - The Complete SEO Course Beginners to Advanced 2023 - https://coursetreat.com/udemycourse/seo-the-complete-seo-course-beginners-to-advanced-2022/

24. Positive Parenting- Basic Understanding - https://coursetreat.com/udemycourse/positive-parenting/

25. MicroSoft Teams Masterclass: Connect With MS Teams - https://coursetreat.com/udemycourse/microsoft-teams-masterclass-connect-with-ms-teams/

26. Master the Art of Writing Fascinating Copy for Persuasion - https://coursetreat.com/udemycourse/master-the-art-of-writing-fascinating-copy-for-persuasion/

27. Easiest Side Gig 2023 - Passive Income from User Testing - https://coursetreat.com/udemycourse/easiest-side-gig-2022-passive-income-from-user-testing-course/

28. CSS, Bootstrap And