Azure Cognitive Search API kill-switch

Hello fellow SRE’s,

Has anyone tried to implement a kill-switch for Azure’s [Cognitive Search API](https://azure.microsoft.com/en-us/services/search/) so far? Basically, we have a Java SpringBoot microservice running in GCP GKE that’s calling Cognitive Search API but after scaling the app out our cost ramped up quite fast & we can’t control it.

Is there a way one could restrict API calls to Cognitive Search in any way if a certain billing threshold is reached? I have no experience with Azure so far...

Thank you

https://redd.it/fvwy7b
@r_devops

Tutorial: Standing up an EKS Cluster with Terraform

I've been playing with EKS lately and wanted to ensure I define my IaC so I wrote the following article on how to do this with terraform. Hopefully its helpful! [https://link.medium.com/0doYsmoXp5](https://link.medium.com/0doYsmoXp5)

https://redd.it/fuzgno
@r_devops

Misconfiguration in the Cloud

‘Soon most of the attacks on the cloud environment will be the result of misconfigurations, lack of customizable security profiles, and auto-remediation by organizations in their day-to-day” — 

Just wrote a new article about Misconfiguration in the cloud any feedback or new ideas on how to mitigate it will be very helpful

[https://medium.com/@fernando0stc/misconfigurations-in-the-cloud-be-prepared-before-they-cause-you-headaches-1a92c2979eff](https://medium.com/@fernando0stc/misconfigurations-in-the-cloud-be-prepared-before-they-cause-you-headaches-1a92c2979eff)

https://redd.it/fuym1z
@r_devops

How does your company enable you to properly implement DevOps/SRE principles?

I've been very frustrated with the way my company and team(s) sees DevOps/SRE. Doing anything the "right way" is shot down for reasons ranging from red tape to "because we said so". I've pushed very, very hard to change things to the best of my ability for as long as I've worked here, and in some ways things have improved, but only marginally so. I just do not have any buy in from the teams I work with or anyone in management/leadership positions as much as they love to talk about us "doing devops" because we have a decent CI/CD flow and a few other things. I've basically thrown my hands up and transitioned back to a standard SWE position. Which, that being said, I do enjoy regular feature work a lot, but the difference in support and treatment for SWE vs. DevOps/SRE at my company is absolutely staggering.

Would love to hear stories from the other side of the spectrum where y'all are being actively supported and enabled to improve things.

https://redd.it/fuxebu
@r_devops

Is there any way to launch predefined bash scripts from Slack?

Hello, redditors,

I want to let employees run some predefined bash scripts right from Slack.

Do you know any good software to do it?

What about security?

I saw solutions for running SQL queires like SQLBot, but nothing for bash.

Scenarious:

/slackbot server1 reboot

/slackbot server2 show nginx status

/slackbot server1 addtorepo [[email protected]](mailto:[email protected])

https://redd.it/fvz7nj
@r_devops

Flash sales on A Seat at the Table book on 23.04 and 24.04

Hey everyone,

to give you heads up, the IT Revolution books are doing World Book Day promotion on April 23 and 24, and the book *A Seat at the Table* by Mark Schwartz will be $0.99.

https://redd.it/fvz0m5
@r_devops

permission denied when reading root owned bind mounted file in container

Dear all,

​

I have a weird situation where I need to use an in house binary (let's call it binary x) shipped in a docker container. Said container runs the binary as an unprivileged user with the same name as the binary. The binary needs a username and password to function. Not wanting to store the username and password in the container, I thought about storing them in a root owned, mode 500, on the host and bind mount it in the container for use.

​

The issue I'm having is that, since the container username is unprivileged, it can't read the file.

For kicks, I tried another container that runs its app as root but still can't read the bind mounted file (permission denied).

​

How can I read a root owned file, mode 700 from the host machine in a container that either has or does not have its own root user?

https://redd.it/fw2f4l
@r_devops

Jenkins needs access to my AWS pem key in linux users .ssh directory. What's best practices for giving Jenkins access to pem keys on a linux box?



https://redd.it/fvxlp9
@r_devops

How much distributed tracing costs: using open source like Jaeger or paid products like DataDog?

Wanted to get your inputs on 2 solutions for tracing needs:

1. **Open Source solutions like Jaeger -** Is there a blog or link which analyses the cost of storage of traces to any of the backend like Cassandra or Elastic? Do you use **any compression techniques** before storing to DB?
2. **Paid products like Datadog -** Most of the vendors charge on a per-host basis. What's the underlying logic for charging like that?
If I were to use a vendor for this, I would like to be charged according to the number of spans sent for storage like in logs, say $2 per million spans or $0.2 per GB. Only Datadog seems to charge on such basis and asks for [$1.7 per million of spans](https://docs.datadoghq.com/account_management/billing/apm_distributed_tracing/) apart from $31 per host which covers 1M spans only. Does DataDog give you control of which spans to visualise & store?


* Does any vendor give you control of spans to process or a clear pricing estimate for the tracing part?

What other things should I look when using Jaeger or buying Datadog? My primary need is to monitor and debug my applications.

https://redd.it/fw105j
@r_devops

Junior employee, feeling a bit stuck.

Let me preface this by saying I may not be "DevOps" in the way that a lot of you are. I understand the argument that "true" DevOps roles are typically the coupling of sysadmin experience with dev/scripting skills. But I work in a team that is trying to automate a lot of ops tasks. Hence, I come across the same tools and technologies that are mentioned here.

With the world being shut down, I've become pretty bored. I figured it'd be more productive to use my spare time towards becoming a better engineer instead of drinking box wine and watching Netflix. So since I may be using tools like Ansible, Kubernetes, and Docker in the future I've been trying to learn these via Linux Academy. Although I end up hitting a point where I can *follow along* with the tutorials but I am not fully grasping the content. It's as if I'm missing some sort of underlying prerequisite knowledge needed to fully understand these tools. Or as the saying goes, putting the cart before the horse.

Just to give you some background on me. I studied Bachelor of Computing and I was really into dev work at university. I made all sorts of web apps for internships, personal projects, and freelance work. I'm talking Python, JavaScript & Node.js, LAMP stack, and so on. I've also worked help desk in your standard Microsoft environment.

I've searched this subreddit and there's so much conflicting advice of "learn k8s/ansible/terraform" vs "no learn Linux/networking first." I did the LPI Linux essentials course on Linux Academy. It was really top quality and I got a lot from it, so should I continue down the path of increasing my Linux skills? Or should I keep soldiering on with the more tool-based Kubernetes/Ansible courses?

Any help is appreciated, cheers.

https://redd.it/fvyd4c
@r_devops

Local Azure Devops Server and CI/CD to Containers and Servers

Hello, long time systems guy and just getting into and understanding the whole CI/CD concept. We currently do manual deploy of .net websites and services in IIS. We run team foundation server but have the licence to upgrade to Devops Server 2019. I have been struggling for a week now trying to get just a basic website to deploy to a container or IIS site via a pipeline. I've been using the Microsoft resources and a few older blogs but can't seem to find one that walks me from start to finish. I have a Devops server 2019, a physical 2019 container server with the latest docker and 3 virtual IIS web servers. I could list all of the errors I'm having but we might be here a while. The question is, does anyone have any resources that will walk me through the commit to a repository and the deployment to a container and IIS?

https://redd.it/fw79do
@r_devops

Opinionated Infrastructure as Code

Terraform, Pulumi, CDK, etc. help us to deploy and manage infrastructure through code.

After working in that space for quite a while, I found myself implementing a lot of infrastructure components over and over again. Some examples might be VPC's, Load Balancers, Databases, Private Buckets, etc.

For Terraform we have modules that help us to abstract common patterns. Still, I am struggling to find good modules that are tested and follow best practices.

I would like to see some kind of library or framework that helps me to easily deploy infrastructure that follows best practices such as the AWS well-architected framework or Google Cloud Whitepapers. I would appreciate a solution that helps me to quickly span up production-grade infrastructure through code without the necessity to implement everything from scratch.

Also, I would like to see a low-code solution that helps me to quickly bootstrap new environments but delivered as code.

Are there any existing solutions out there and what would you like to see to help you to become more productive?

https://redd.it/fvxcnp
@r_devops

Need advice on how to handle CVEs and automatic scanning

I'm sort of new to all this. I'm trying to get some sense of the state of security of all the machines I manage. I figured one way to do that is to automatically and periodically scan for CVEs. So I tried openSCAP and AWS inspector. And they just spam with huge lists of discovered problems on a fully updated system. Weirdly enough, what openSCAP calls medium, inspector happily calls high. So hard to judge severity just by looking at the list.

I've looked through the lists themselves, and there's one thing in common for all the problems: there are no patches yet, or state is some form of "needs confirmation." It seems unreasonable to hunt down the patches in all the relevant repositories and compile all the things myself, including dependencies. So I see two ways of handling those problems: ignore and wait for patches from upstream, or disable the package.

Ok, I can do that once, maybe twice. But how do I keep track of all that continuously without going insane? It's hard to believe that I'm the first to stumble on that problem, but googling does not help. It seems that I'm looking for the wrong things here.

What I'd like to be able to do is to track the state of those findings in time as I periodically scan my systems. So, once ignored item stays ignored, an item that does not yet have patches is marked one way, and the one that has a patch available is marked differently.

But maybe I'm approaching this all wrong? Any advice is welcome.

My machines are all Ubuntu if it matters. But it'd imagine, that situation is the same in other worlds.

https://redd.it/fvy0hz
@r_devops

How to develop more in-depth, SRE- relevant Linux knowledge?

Hi all,
I recently started my first SRE job straight out of college. While I have been using Linux as my primary OS for the past couple of years and I can use it comfortably, I feel my knowledge of linux is a bit lacking for an SRE. Can you all amazing people of r/devops suggest some resources or methods to learn more about how Linux stuff like process and memory management, user permissions, logging, networking, security etc?

Thank you for taking the time to read this!

https://redd.it/fvt6b7
@r_devops

Uninstall kustomize leftovers

Hi experts, what sort of techniques you used to uninstall Kustomize leftovers. Most of the time if you want to get rid of other resources installed by your Kustomize declaration, you execute \`kustomize build | kubectl delete -f -\` and followed by \`kustomize build | kubectl apply -f -\`. But this doesn't guarantee that it will remove those previous installed items that is not part of the manifest anymore or being renamed.

https://redd.it/fvvmz6
@r_devops

Deploys at Slack

Interesting post on the deploy process at Slack. I like how they go over how the deploy process evolved from rsync to something much more complicated:

https://slack.engineering/deploys-at-slack-cd0d28c61701

https://redd.it/fwbkhs
@r_devops

Stuck on SSH using CircleCI

Hi all,

I've just started using CircleCI to deploy code for me for personal projects. I have managed to get my config to deploy when using the \`circleci local execute\` cli but pushing it it always falls. I have tweaked the config as below to use the projects SSH key.

I generated a new SSH key and uploaded to the " Additional SSH Keys" section. When uploading the key I set the hostname to "[my-server.net](https://my-server.net)" (changed for obvious reasons). I then used the fingerprint that the web page showed and added to the config. I can log into the sever I need to using the key in question.

​

When running after a push it gets stuck at:

The authenticity of host 'my-server(123.123.123.123)' can't be established.
ECDSA key fingerprint is SHA256_VALUE.
Are you sure you want to continue connecting (yes/no)?

I then stop the job and then re-start allows SSH access to the run.

When I try to ssh in from within CircleCI I get the error:

username@my-server: Permission denied (publickey).

Can anyone suggest what I'm doing wrong?

​

version: 2.0
jobs:
build:
docker:
- image: cimg/base:2020.01
steps:
- checkout
- add_ssh_keys:
fingerprints:
- "my key fingerprint"
- run:
name: "Create build"
command: tar -czvf /tmp/build.tar.gz .
- run:
name: "Upload to remote server"
command: scp /tmp/build.tar.gz username@my-server:~/
- run:
name: "Install build"
command: |
ssh username@my-servertar -xvf build.tar.gz -C /var/www/vhosts/my-server/
ssh username@my-server"cd /var/www/vhosts/my-server/ && composer install"
- run:
name: "Reload NGINX"
command: ssh my-serversudo service nginx reload

Any suggestions would be massively appreciated.

https://redd.it/fwax4g
@r_devops

AWS CLI permission error

AWS CLI access denied - Workspaces

I’m getting an AWS CLI access denied error when attempting to get info about my AWS Workspaces machines.

Only thing I don’t get is that I am an aws administrator? Already double checked.

I’m running version 2.

I don’t want to check my workspaces one by one :(

Command I’m running

Aws workspaces describe-workspaces

https://redd.it/fw9g6u
@r_devops

Voice of reason

[Nothing new](https://i.imgur.com/XHdcKaG.jpg)

https://redd.it/fwennq
@r_devops

Need suggestions

I'm getting offer for a U.S client in my state(remotely in India) as a developer (Android,IOS,React, React Native,Angular). I'm in my last semester of B.tech and i was always interested in DevOps. Is it to be good idea to start as developer. I think it would be difficult to change profile after having experience as a developer . Do give your thoughts (i hope i won't regert this opportunity as recession is coming)

https://redd.it/fw7mp3
@r_devops

Packer hangs on building an Ubuntu 18 template on vmware (vsphere-iso: Waiting for SSH to become available...)

Hi,

I'm trying to build a vmware image of Ubuntu 18 with Packer , but it keeps failing with :

**vsphere-iso: Waiting for SSH to become available...**

I'm running Vmware vcenter 6.7, and packer 1.5.5 on a centos 8 host.
I have build centos7 and centos8 templates successfully.



Here is my variables file (variables.json)

{
"vsphere_server": "192.168.0.51",
"vsphere_username": "[email protected]",
"vsphere_password": "password",
"vsphere_datacenter": "Datacenter",
"vsphere_datastore": "datastore",
"vsphere_folder": "Templates",
"vsphere_host": "host.domain.local",
"vsphere_network": "network1",
"vsphere_template_folder": "Templates",
"ssh_root_username": "root",
"ssh_root_password": "password",
"ssh_username": "admin",
"ssh_password": "password"
}

Here is my json file (ubuntu18\_buildtemplate.json)

{
"builders": [
{
"type": "vsphere-iso",

"vcenter_server": "{{user `vsphere_server`}}",
"username": "{{user `vsphere_username`}}",
"password": "{{user `vsphere_password`}}",
"insecure_connection": "true",

"vm_name": "T-ubuntu18",
"datastore": "{{user `vsphere_datastore`}}",
"folder": "{{user `vsphere_folder`}}",
"host": "{{user `vsphere_host`}}",
"convert_to_template": "true",
"network": "{{user `vsphere_network`}}",
"boot_order": "disk,cdrom",

"guest_os_type": "ubuntu64Guest",

"ssh_username": "{{user `ssh_username`}}",
"ssh_password": "{{user `ssh_password`}}",

"CPUs": 1,
"RAM": 1024,
"RAM_reserve_all": false,

"disk_controller_type": "pvscsi",
"disk_size": 32768,
"disk_thin_provisioned": false,

"network_card": "vmxnet3",

"iso_paths": [
"[datastore] ISO/Linux/ubuntu-18.04.4-live-server-amd64.iso"
],


"floppy_files": [
"./ubuntu18_kickstart.cfg"
],
"boot_command": [
"<enter><wait><f6><wait><esc><wait>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs>",
"/install/vmlinuz",
" initrd=/install/initrd.gz",
" priority=critical",
" locale=en_US",
" file=/media/ubuntu18_kickstart.cfg",
"<enter>"
]
}
],

"provisioners": [
{
"type": "shell",
"inline": ["echo 'Template build complete'"]
}
]
}

Here is my kickstart file (ubuntu18\_kickstart.cfg)

&#x200B;

### Base system installation
d-i base-installer/kernel/override-image string linux-server

## Options to set on the command line
d-i debian-installer/locale string en_US.utf8
d-i console-setup/ask_detect boolean false
d-i console-setup/layout string USA

#--------------------------------------------------------------------------------
# ACCOUNTS
#--------------------------------------------------------------------------------
d-i passwd/user-fullname string admin
d-i passwd/username string admin
d-i passwd/user-password password password
d-i passwd/user-password-again password