DevOps pragmatism?

What are your thoughts?


Should I be more pragmatic?


I tend to stand firm when I see something is missing transparency and ask people to do the least amount of work to make something easy to understand whenever we get back to code. Especially from a security point of view.

A coworker upgraded an ye olde application which is not documented in any fashion, other than it exists in a git repository. It wasn't added to a CI/CD pipeline in any form, nor is there added any documentation or pushed a package. The upgrade ended up on a production server, since it was time essential to get the application going again.

From my point of view, there should be at least be a CI build and an application package which ends up in a package repository, even if the application is small and cannot be deploy with a CD tool. Me and my coworker discussed it with our team, but the team decided it wasn't worth the effort since the application/code actually belongs to another team.

I feel like I'm nagging about something that should be the minimum effort. But am I just being an inconvenience?


https://redd.it/ssz4ul
@r_devops

The DevOps handbook - summary

# The DevOps handbook - summary

DevOps handbook is one of the most important DevOps books on the market. It combines the most important DevOps ideas, examples from known companies, and best practices. It does not have any code so it is an ideal book for non-technical readers that would like to know what their teams are talking about. Bellow, you find a summary with the ideas that stick with me the most.

## Flow and Feedback

The first thing we want to accomplish in DevOps is to have fast feedback on our work. The first frontier, before the ultimate testing on production, is a set of well-written fast tests. Those tests should be able to run in parallel and they should be run on each commit automatically. Tests that are not run are useless, and the only way to enforce running those tests is using automation. This allows us to have basic confidence in our code and the fast feedback provided by fast pipeline should not break the flow. The testing should be not limited to unit tests but it can (and should) test also security, integration, and other aspects of our application. Important is to run different tests on different occasions (run slow tests less often) to not break the developer`s flow.



> ...a small number of reliable, automated tests are almost always preferable over a large number of manual or unreliable automated tests.



## Get it out!

You will never test everything. In production, someone will always do something you do not expect, try to hack you, or will not use your shiny feature at all. And the only way to test this is to go to production as fast as possible. This is the only way to test (almost) every possible scenario and see the outcomes. Of course, going fast to production have its dangers but we can mitigate them using tools such as shadow releases, canary testing, or blue-green release pattern. With frequent releases, we generally get smaller release sizes which is associated with a higher deployment success rate.



> ...when we increase our deployment batch size, our change success rates go down and our incident counts and MTTR go up—the opposite of the outcomes we want.

## Observer and learn

If we put our code into production we need to have a way to observe the code running. Other than standard metrics such as CPU and Memory usage, percentage of a cache hit, and other technical measures we should also have business metrics available. Does our new feature that is promoted everywhere have poor usage? It might be caused by a technical issue such as poor performance or distorted render on some browsers. The second case can be that we simply build the wrong thing. Both outcomes are valuable for us developers so we should have access to those metrics to be able to learn from them.

> ...business metrics create context for our infrastructure metrics, enabling Development and Operations to better work together toward common goals.

## Involve everyone

One of the most important lessons for me was how important is to make everyone more exposed to other team's problems. In the end, we all work toward a common goal. By being more exposed to problems of developers, testers, or ops we can often find better solutions to problems, fix the problems earlier, cheaper and build empathy for other teams.


> ...how to make Dev work more visible to Operations. To accomplish this, we explored three broad strategies, including creating self-service capabilities to enable developers in service teams to be productive, embedding Ops engineers into the service teams, and assigning Ops liaisons to the service teams when embedding Ops engineers is not possible.



## Conclusion

The DevOps handbook is one of the technical books that will fill your head with ideas. It will not teach you how to create an ideal pipeline, nor it will give you the answers to all problems but it will tickle those parts of your brain that know that our job can be done better.


> Our call to action is this: no matter what role you play in your organization, start finding people around you who want to change

how work is performed. Show this book to others and create a coalition of like-minded thinkers to break out of the downward spiral.

Orignaly published on my substack, if you like the post consider subscribing
https://rejmank.substack.com/p/the-devops-handbook-summary?r=438li

https://redd.it/ssz1y9
@r_devops

You started a new job, what are the first tools you install on your machine?

I'm curious to know what DevOps engineers do to setup their workspace when they are handed with a new laptop. What tools do you use daily to boost your productivity?

For example, oh my zsh, WSL for Windows, VS Code with X extensions, etc etc.

I'm part of a DevOps IT team for R&D (I'm personally IT) and the engineers at my team always have wacky setups that help them to almost never use their mouse, or never completely type anything without some auto-completion. Basically wizards.

Thanks :)

https://redd.it/st22ia
@r_devops

How long did you wait between your final interview and job offer?

I've been interviewing for this tech company in London for the past 3 weeks, the stages went like this:

​

1. Screening call with Founder of company (invitation to 2nd round 1 week later)
2. Technical interview with DevOps engineer (invitation to culture fit interview day after)
3. Culture fit interview with 2 other Devs at the company last Friday - hopefully will hear back this week (it's Tuesday).

The position is a junior DevOps engineer with a focus on AWS, Kubernetes and Python. I'm a certified SAA and after 5 years working as a sys admin this would be my first devops job.

I think I'm just being anxious and their are clearly a lot of other unseen factors involved, but it would be good to know what the rough estimate is between a final interview and job offer in the devops world. Thanks.

https://redd.it/ssz74u
@r_devops

Shameful cloud sourcing request regarding Azure Pipelines and Docker login task

The default behavior of the docker login task is absolutely absurd when you're not using ACR.

It dumps the service connection credentials into the pipeline logs. Does anyone know of a way to suppress this? If not ; someone made an issue regarding this already and it'd be nice to get more attention to this?

https://github.com/microsoft/azure-pipelines-tasks/issues/14322

https://redd.it/st6x6h
@r_devops

Packer unable to read the subnet in a different resource group

Hi All,
I am getting an error like
"Invalid resource reference" .. /subscriptions/12fasd4324-213asd4e21dad342-dsa/resourseGroups/network-rg/Providers/...../subnets/existing-subnet1 referenced by resource /subscriptions/../Microsft.Network/networkInterfaces/pkrni88im9aen8 was not found .Please make sure the referenced resource exist and both are in same region


The referenced resource existing-subnet1 is present and also all these things are in Eastus2
I am not sure whats the issue
Can you guys please help


Important points:
I cant use public ip thats the reson i included virtual network details
the virtual network is in a different resource group but everything is in a same subscription

​

is the error because packer is not able to read that vnet ?
Attached code below

{

"builders": [{

"type": "azure-arm",

​

"client_id": "f5b6a5cf-fbdf-4a9f-b3b8-3c2cd00225a4",

"client_secret": "0e760437-bf34-4aad-9f8d-870be799c55d",

"tenant_id": "72f988bf-86f1-41af-91ab-2d7cd011db47",

"subscription_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx",

"virtual_network_name" : "existing-vnet1"

"virtual_network_subnet_name" : "existing-subnet1"

"virtual_network_resource_group_name" : "network-rg"

​

"managed_image_resource_group_name": "myResourceGroup",

"managed_image_name": "myPackerImage",

​

"os_type": "Linux",

"image_publisher": "Canonical",

"image_offer": "UbuntuServer",

"image_sku": "16.04-LTS",

​

"azure_tags": {

"dept": "Engineering",

"task": "Image deployment"

},

​

"location": "East US",

"vm_size": "Standard_DS2_v2"

}\],

"provisioners": [{

"execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh '{{ .Path }}'",

"inline": [

"apt-get update",

"apt-get upgrade -y",

"apt-get -y install nginx",

​

"/usr/sbin/waagent -force -deprovision+user && export HISTSIZE=0 && sync"

\],

"inline_shebang": "/bin/sh -x",

"type": "shell"

}\]

}

https://redd.it/sszwe6
@r_devops

Path based routing in local

How can we test path based routing from our local systems, Is it possible?

https://redd.it/st1cl5
@r_devops

Is a Customer Success Engineer job as a new grad better than no job?

I apologize in advance if this post doesn't belong in the DevOps community but I thought it might be related.

The CSE job description says, I have to :

* maintain websites on cloud PaaS
* access cloud servers using terminal system (Nginx, Apache, etc.)
* work on chats and support tools (such as Zendesk)

There's a 5 month probation period and then a 2 yrs contract (and if I leave before that I have to pay back 5 months' salary). I have been job hunting for around 1.5 months and even though I have gotten calls from a few good places, I couldn't pass the interview because my DSA problem solving skills are not that good yet.

Now, I plan to study DSA for 3/4 months and then apply to jobs I am interested in but I am not confident about the future. I am planning to accept the job and then practice DSA at the same time, then apply elsewhere. Does that sound like a good plan? Although, I am interested in a SWE job but I do not mind going into the DevOps route either. But does the above job description even sound good for my career?

I just graduated 4 months ago and money is not a problem atm. I am just scared that I might be unemployed for a year or more if I let go of this job. Any insight is appreciated. Thanks for your time.

https://redd.it/stbpuc
@r_devops

Aws image builder pipeline for CIS images.:

errors with TOE because of the working directory being set to /tmp - where else can it be set for the CIS ?

https://redd.it/std77u
@r_devops

Defining responsibilities in a DevOps team...

So, I know it's far too common for "DevOps engineers" to get thrown under the bus and become a multitool on steroids thay can fix anything involving technogy.

I'm in a position where in my new team I might be clable to restore some sanity. The issue is in a team of about 10, the work is going through about 3 people who have to fix everything and dev pretty much everything.

How do you introduce a fairer distribution of work and responsibilities in a DevOps team without creating silos? Everyone will still be working on one project and having CICD is critical, but I don't think it's fair for our Ops Engineer, who has a full work load already, to spend half their week in meetings rewriting a junior devs Java because said Jr Dev complained to the boss about getting help and the (departing boss) understood DevOps to be "everyone knows how to do everything and should therefore work on everything".

https://redd.it/stcjh2
@r_devops

How good do you have to be at linux or powershell for a devops engineer?

I know the more you know the better. But I am trying to find out the time I should split between coding, learning devops and bash? Thanks

https://redd.it/stfr2g
@r_devops

How do you make an infinitely scalable Wordpress box?

How do you make an infinitely scalable Wordpress box? I have a box, and I have a cronjob that runs a stored proc from a MSSQL server, I want the other instances to just be slave instances of the master and not run any cronjob. Is there a tutorial on how to do this?

https://redd.it/stgnuj
@r_devops

DevOps Certification for someone wanting to become a Digital Nomad?

I am thinking of switching over to DevOps Engineer from my current Data Engineer job. I think DevOps gives more opportunities for remote work (thus helping to become a Digital Nomad!) and is less boring compared to being an SQL monkey as a Data Professional. Besides, I found it's very interesting to work with Infra and Production problems rather than PoC and pure data modeling jobs.

So, which DevOps certification should I get? GitLab is the best DevOps platform (CI/CD) now and they have some certifications. AWS on the other hand has cloud dominance and they offer DevOps Professional-level certifications. Moreover, Kubernetes is ever more important nowadays and GitOps is the future (that includes Terraform).

Please suggest to me which DevOps certification should I get that will cover Linux, Git, CI/CD, Docker, Terraform, Ansible, Kubernetes, etc?

https://redd.it/stjtjj
@r_devops

Env Variables

I know this can be somewhat opinionated but let's say we have an application with ten env variables of those 2 are only sensitive. Is it worth storing all of them in vault/secrets manager or set only the secret env variables in a secret manager and letting the other non-sensitive be placed in the cicd config?

https://redd.it/sti7g8
@r_devops

What are some other good DevOps skills test to add to this list?

After seeing something similar in this sub, I created the following skills test for new candidates. [Link\]

The goal was to make something easy but also a fun little exercise. What would be some other good additions that I may be missing?

Also feedback is (probably) welcome!

https://redd.it/stjbpa
@r_devops

Separate service account for each integration?

Service accounts are accounts that are not used by humans and are used to integrate a tool with another. For example database account to connect to external db, LDAP binding service account, SonarQube service account to integrate with Jenkins.

Is it best practice to use a separate service account for each of the individual integrations? Or is it okay to use a single service account?

Edit: Also, for certain integrations with Jenkins, the credentials is not required, but a generated token is used instead. In this case, what is the best practice? Can I just use the admin account for the tool to generate a token? Or should I create a service account for the tool to generate a token to be used in Jenkins?

https://redd.it/stjb7y
@r_devops

monitoring AND restarting services (small scale)

hi, we have just a few servers with a few small business scale apps (nginx, fpm, mysql, few docker containers, some other services).


what would you recommend for monitoring AND automatically restarting these services?
I have found this thread:
https://www.reddit.com/r/devops/comments/d83v9w/monitoring\_and\_autorestart\_of\_services/


I am looking for something that is:
\- simple to install and manage
\- can restart failing services if needed
\- will notify if an issue cannot be solved automatically (mail, telegram, webhook?)
\- generally, will allow me not to spend much time on this (small company, much responsibilities, limited resources, time)


I've seen folks are using prometheus with grafana now, but as I understand it does not handle automatically bringing the services back.

I used to use monit few years ago and it was doing it's job. just wondering if there is something new around?


thanks

https://redd.it/stqxon
@r_devops

What is distroless in Docker? Does it contain graalVM?

In my office where I work, we use google kubernetes engine and I used that to my quarkus application.

I should built my quarkus on native application. It's fine with quarkus 2.1.1.Final. However due to nexus scan violation detects some dependency, I think better to moved to latest quarkus core because it has latest dependencies. But when jenkins tried to built it fails with error "GraalVM installed is 21.0.0, quarkus support GraalVM xxx" (I am forgot the exact version lol)

Jenkins script is executing some dockerfile and inside of it written "FROM our repository/image name" the image name is "distrolesswithbashandsh:1.5"

On jenkins script we also has agent kubernetes yamlFile and it's container image named "centosquarkusmaven21java11:latest"

I have no experience with kubernetes server so I wonder where's is graalVM located? Does it's located on kubernetes system itself or in one of those container image?

https://redd.it/stt8v0
@r_devops

Awesome OpenTelemetry

Just created my first "awesome" list about OpenTelemetry.

Checkout Awesome-OpenTelemetry to find hopefully good and helpful resources.

An awesome list is a list of awesome things curated by the community.

You can read more in the Awesome Manifesto

https://redd.it/stuaxy
@r_devops

Free Secure Software Development Fundamentals Courses

The Open Source Security Foundation (OpenSSF) has developed a trio of free courses on how to develop secure software. These courses are part of the Secure Software Development Fundamentals Professional Certificate program.  There’s a fee if you want to try to earn a certificate (to prove that you learned the material). However, if you just want to learn the material without earning a certificate, that’s free; simply audit the course. You can also start for free and upgrade later if you pay within the upgrade deadline. All three courses are available on the edX platform.

The courses included in the program are:

​

1. Secure Software Development: Requirements, Design, and Reuse (LFD104x)
2. Secure Software Development: Implementation (LFD105x)
3. Secure Software Development: Verification and More Specialized Topics (LFD106x)

https://redd.it/stwefa
@r_devops