Ship logs to Elasticsearch over [unstable] internet connection

hi reddit,

I need to ship logs to a single central Elasticsearch over the internet, from a bunch of different locations, in some of them connection can be unstable. I plan to use filebeat installed in every location with reasonably large queue configured, so any connection glitches would not stop logs from eventually being shipped to Elastic.

A question - had you ever worked with scenario like this? Do you know anything better that just putting filebeat with large queue in every remote location? Any thoughts/ideas, please?

Any feedback/suggestion is extremely welcome.

Thank U!!!

https://redd.it/esvi1x
@r_devops

What kind of alerting do you use on your API's?

Trying to setup alerting to detect suspicious behaviour, abuse or anything else...

​

If theres any documentation you guys reccomend on what to alert on that'd be helpful aswell, as I'm not exactly sure what to look for at the moment

https://redd.it/estxbf
@r_devops

Want to learn enough to keep up

Want to learn enough to keep up

If you’re going to be working among the cloud, rather than specifically in the cloud, say in an executive or a marketing capacity, you should consider a Cloud Practitioner certification.

Upon gaining this certification, you’ll be able to keep up with your IT teams and understand cloud concepts sufficiently enough to think about how they can apply to your company or industry.

https://redd.it/est0bn
@r_devops

What is my next step?

Just finished my Comptia Linux+ Cert, and I am graduating with an AAS in CIT with a focus in Cybersecurity this spring; 3.5 GPA. I also have Comptia Network + and A+ certs. I have worked at a paid IT internship in a Windows environment for the past 18 months. I have done some automation but not a ton. I took some online Docker courses and have played around with Kubernetes. Coursework for my AAS included general IT, networking, Cisco, Windows server, Linux server, security management, Powershell, Vsphere, and a lot of other technologies I'm not going to list. I studied CS for 3 semesters at a state university and know my way around C++, Python, Powershell, and scripting, but my GPA was very low. My questions for you are:

​

1. Should I now apply for DevOps internships, a DevOps junior positions, DevOps engineering positions, or Linux Administration positions? What level do I currently qualify for? I feel like I should apply for everything, but I don't want to waste my time or anybody else's. I don't want to get a job I am overqualified for. I live in the Midwest in a big city, and can relocate to expand my search range. I do very well with people in interviews in person, but I look like I'm 19 (I'm 25m).
2. What certification should I go for next? I am finding a lot of AWS recommendations, but the Kubernetes, Ansible, and Docker certs seem pretty relevant and more general. Also I think I would have fun with the red hat certs.
3. I know completing a BA alongside my next job would probably be the right choice for me financially. I am going to wait to see how my next job pays and how heavy the work load is before I decide. I'm sick of college professors at the moment. This isn't really a question but I thought I would get it out there.



Anyway any input or reality checks would be greatly appreciated. I feel like there are a lot of options in front of me, but I don't know enough to accurately judge them.

https://redd.it/et46km
@r_devops

How to purge Cloudflare cache via Jenkins?

I am trying to have it so when a certain Jenkins pipeline is ran that one of the steps in the pipeline is clearing the CloudFlare cache since it's not needed. I found the API though was wondering if someone else had a better solution than a simple bash script and a curl command.

https://redd.it/et3qdv
@r_devops

Running DevOps at a start up, looking for advice

I've been working in DevOps for about 7 years now. I have more Ops experience than Dev, but I'm leaning into the Dev side now. I've recently been approached to run devops at a small early stage startup. This is for sure something above me, and I'll admit I'm a little overwhelmed.

Tech stack is GCP with lots of data flows. Primarily Linux and open source software.

Does anyone have any advice or any materials to read? I've read The Phoenix Project already, but I'd love some good articles or blogs, if anyone has any!

https://redd.it/esxi2m
@r_devops

How do I inject secrets into an instance provisioned by terraform for use in userdata?

Hi all. Please excuse any ignorance, I'm a developer and not super experienced with devops.

I have the following setup currently:
- A custom AMI built using packer
- Terraform provisions an EC2 instance using the AMI and runs some scripts in userdata that requires private git access/credentials and npm credentials.

Problem: I am not sure how to get my credentials into the EC2 instance securely/not in source control so the userdata script can execute successfully.

Only thing I have been able to think of:
- Using packer to configure the credentials. I can look into this if there is no other way but I don't like the idea of baking my credentials into the AMI (even though it's private) and I assume there's a proper way to do this.

Other information:
- This is a windows machine and I'm using powershell (not sure if relevant)

Any help/suggestions would be great. Thanks all.

UPDATE: I found a solution that works for me:

Use `templatefile()` and populate the credentials using TF_VAR_ variables.

https://redd.it/eszypm
@r_devops

Opinion about Windows/Linux DevOps tools and migrating skills

I manage some .NET framework app which will eventually go to .NET core and become cloud-native. Here, PowerShell is my friend. I get the point, I understand the DevOps philosophy, I see the good and bad in company processes. However, I cannot see a lot of resources about Windows DevOps things and I cannot see a solution for some part of the apps and services that have so many restrictions to migrate to cloud. So to speak, it seems like Windows has a lot to do to catch up with Linux regarding every aspect and that would take some time (maybe years).

So I wanted to ask what do you think, will Windows ever catch up with Linux regarding DevOps tools?

The reason for my question is simple. Honestly, there is no so much job opportunities when working with Windows, so I thought taking my skills to Linux, to broaden my area of expertise, learn new skills and create new opportunities. I mean, I am not bad at Linux, I just haven't had a lot of chance to work with it. I am familiar with DevOps processes and same should apply to Linux, it is just different OS.

What are your thoughts and suggestions on this?

https://redd.it/esyx3p
@r_devops

python script to ansible playbook for excel

I need to write a python script that cleans up the output from ansible in to a excel spreadsheet.

i have the ansible written , I need a python script to import information into a spreadsheet.

https://redd.it/esy373
@r_devops

Kubernetes documentation no longer supports most recent GKE version?

Correction to title: "K8s documentation way ahead of GKE (and really not easy to navigate for older versions)"

Noob question here. Just started with k8s a few weeks ago.

As of writing, the default k8s version on GKE is 1.13.The default when using "release channels" is 1.14.Current k8s documentation is at 1.17.

I've been Googling and there aren't many results for this, but I see it as a pretty big red flag. In our case, we've been using k8s via docker-for-mac and just realized we have a versioning difference. Is this just something that is accepted? Is there a strategy for working with GKE which is so far behind?

https://redd.it/etbh6m
@r_devops

DevSecOps (Security in DevOps)

I have done implementation of security tools in DevOps process that helps to make DevOps more secure
Here is the github link
https://github.com/Rishabh-Tamrakar/DevSecOps

For now I have updated only 1st page that will explain process that I followed
Configuration of tools will be updated soon (most Probably this weekend)

Do tell me your thoughts about it and what more can be added, ideas are most welcome.

https://redd.it/etbe2k
@r_devops

Easiest way to deploy and host a running Docker Container?

I'm a bit overwhelmed by everything AWS has to offer, but am trying to find a way to deploy a Docker image to have the container running in AWS. The image is stored and updated in Docker Hub, and built via a CircleCI job.

Should I use Elastic Beanstalk, ECS, or something else?

Also, I was originally in favor of AWS, but if any of you have suggestions to keep the cost down, and improving the ease of maintainability, I'm definitely interested in hearing what's worked best for you!

Thanks

https://redd.it/eteld9
@r_devops

Containers that crash "by design"

Hi Everyone. I'm having a "discussion" with our dev team with regards to a container that was deployed to a non production k8s environment, which immediately went into crash-loop-back-off. I assumed the deployment had failed, but in fact, the pod was just waiting for a certain condition (a queue defined in a rabbitmq cluster) to become "true"

The conditions for the pod to continue running (and not crash) are then triggered by someone doing something somewhere in the UI of another service, which may happen 5 mins later, or 5 months later.

In the meantime, its crash-loop-backoff city for our pod.

The developers are claiming this is Kubernetes best practice to allow kubernetes to restart the container should it fail, but:

1) I can't find mention of this specific use case, and

2) It feels like its not in the spirit of what was intended when the best practices were laid out. E.g. "if your application fails due to some completely unforeseeable event, then kubernetes should restart it"... 100% agree with that, but this is "We're going to run code that we know will likely fail, and just let kubernetes restart it until the conditions exist that make it happy"

My concerns are that having a failing pod being a "normal situation" will reduce in alert spam, and eventually, people ignoring those alerts. Further, unless the devs tells us, how are we to know (and remember that those pods "might be in crash loop back off for some time" and this is ok)

My suggestion to the dev team is to have the container "sleep" (and reflect this state in the liveness probe) so that kubernetes will not direct any traffic to it, but similarly, it wont be stuck in crash-loop-backoff hell.

Thoughts?

https://redd.it/etcx9h
@r_devops

Completely random. Can someone tell me how kubernetes is pronounced?

Every time I see it, I read it as cue-bur-neets , but I overheard someone the other day and they pronounced it like cue-bur-net-ees

Please reply with a 1 for my interpretation or a 2 for the way I heard someone say it. If both wong let a man know lol.

Tia

https://redd.it/etfolj
@r_devops

How do you organize the work?

I’m the first DevOps Engineer at a company working on a newish product. It’s beyond MVP stage and so far the developers have been doing all the infrastructure work. Realizing that’s not sustainable they’ve brought me on.

There’s so much to do, immediate needs and lower priority tasks I want to keep pushing along. These aren’t really Jira tasks inside they product board since they don’t deal with product features or bug fixes. How are you all organizing infrastructure work?

https://redd.it/etfca3
@r_devops

DevOps Stack for Microservices

Hello,

New to DevOps. Do you have a recommendation for a stack to support Microservices and cross cloud deployments? Even a past thread or links will be helpful.

Thanks in advance!

https://redd.it/etblml
@r_devops

Docker future stuff

I've been hearing some rumors about Docker Swarm being end-of-life'd by Mirantis and docker-compose being integrated into 'docker stack'.

It'd be nice to not have to install a separate tool for compose, and Mirantis not wanting to double Kubernetes/Swarm efforts makes sense; however, does anyone have any real facts or source of truth behind any of this?

https://redd.it/etdaue
@r_devops

How much do you value WFH

I’m a relatively Jr DevOps engineer who worked my way up from IT at my current company. I’m making 95K in the Northeast area. My current job is great and I’m still learning and getting good mentorship. I typically WFH 3-4 days a week which is a huge benefit. (Although slightly isolating and depressing in the winter)

I recently got an offer for 135K + 15% bonus doing similar tasks at a more mature tech company. They would like me to be in the office though. My commute is a 10 minute drive to the station and then about an hour of sitting on a train if I was to accept.

40K more seems like a lot and I’m wondering if anyone has had to sacrafice WFH benefits for a large jump in pay and was it worth it? FWIW I don’t have any kids so when I’m home it’s just me and my cat 🐈.

https://redd.it/et6hq2
@r_devops

how to create csv file using ansible data and python?

I have created a yaml file to get ansible facts. I need to create a csv file using python to add all the ansible facts i have. In my yaml file, I have lineinfile command for header and data input and creating a file and deleting a file command. i need a python script for csv file format.

https://redd.it/et5vau
@r_devops

Multiple instances of application with different states and databases?

Hi,

I have an application defined in a docker-compose.yml and now I want to add an orchestrator.

My use case: I'm administering this application for a organization with many different subdivisions. Right now nobody is using this application. I want to be able to run multiple instances of this application on my server, where every subdivision in the organization, that wants to use the application, gets one instance with its own configuration, userbase and database independent from all other instances (maybe each configured instance could even be scaled in the traditional sense, but that's not top priority).

My ideal setup would be:

1. A subdivision wants to also use the new application
2. I run some commands and tweak the configuration for this individual instance
3. The orchestrator starts the application and manages the state (configuration, userbase and database) independent from all the other instances

Now my questions are:

* Which orchestrator would be able to fulfill my use case?
* Are there any resources out there how to achieve something like this?
* EDIT: What would be better in this situation: one database for all with different namespaces, or a database for each

https://redd.it/etp65k
@r_devops

Tech stack agnostic tool for tracking builds/artifacts/tests/releases?

Hi

My situation is that we have a number of different tech stacks, each with their own build/release processes and cycles. I've gotten most of my org to centralise around a handful of domain specific techs, but even so a change that goes to production might be any of a Java app, a Go app, a react front end, or libraries for any of those languages, most deployed in containers on k8s, some on VMs, and some in AWS lambda, native iOS or Android builds, terraform projects, shell scripts in crontabs and probably a few things I can't remember.

Luckily, everything is in a source control repo somewhere, and almost everything has a decent CI process, and many things have pretty solid CD processes. So it's not a complete disaster.

What we don't have is a holistic view across all these areas about what was released, when, by whom, how it was built and tested, and what branch of what repo it came from.

Does anybody know of any tool (open source, SaaS, whatever) that would support ingesting this such data from a range of sources and tools, for developing an audit trail and providing metrics and dashboards around release velocity?

To be specific, I am not looking for a CI/CD tool. Most teams already have their own, and while I would like to consolidate everybody to a single platform, for now I'm just trying to measure what's going on.

Any help or suggestions or help are welcome.

https://redd.it/etq0d3
@r_devops