Need Dedicated Server with Unlimited, unthrottled bandwidth

Hi there, I'm working on a project which requires the streaming of music, and this music streaming could easily exceed the typical bandwidth limits on most regular hosting services. I'm a full-stack JS dev and I had a moderate amount of DevOps experience, but I'm certainly no expert, so I wanted to reach out to the community and ask: Is there a dedicated hosting service where we can have unlimited bandwidth? I know it'll more expensive than a regular hosting service, but that's ok!

Any feedback is much appreciated! Thanks :D

https://redd.it/opkl7l
@r_devops

Rebuilding pipelines for multi-stage Dockerfiles and changes to dev workflow

I’m in the process revising my dev environment and CI/CD pipelines based on a bunch of helpful feedback I recently received.

For instance, I wasn’t using multi-stage Dockerfiles and had some anti-patterns in my CI pipeline that I’m in the process of researching and fixing.

Since I’m implementing multi-stage Dockerfiles, it also affects my dev environment which currently uses Minikube and Skaffold (and a separate Dockerfile.dev for each microservice). This is also causing me to consider better ways of doing some common tasks like running unit tests and managing database migrations.

I’m also considering moving away from Skaffold to something like Devspace, Tilt or Telepresence.

That being said I can’t articulate why I’d need to switch because I have some lingering workflow questions related to unit tests and database migrations. Having answers to those will help me determine if Skaffold will work for me in my new workflow, or if I should look at tools like Devspace, Tilt or Telepresence.

My questions:

First, I know I’d need to target the dev stage in the Dockerfile which, looking at the documentation for Skaffold, would probably involve just doing something like:

build:
docker:
dockerfile: Dockerfile
target: development-stage
buildArgs:
key1: value1
key2: value2

Is this correct?

Second, when it comes to unit tests I know there should be a test-runner stage in the Dockerfile. I’m thinking these should just be run for the microservice that was changed automatically upon save. Thus, there shouldn’t be a need to run something manually, or have a --watch flag, in a separate terminal. Is this correct?

Not sure if this would fall under the test section of Skaffold or Custom Tests. I know that Devspace has Hooks that I would think would allow this.

Third, and lastly, when it comes to running migrations I mostly have an answer for this. At least for DevSpace, it sounds like there would be a “post-sync hook” that would run database model migrations. If not, it looks like you can run something like devspace enter python manage.py migrate to manually do the migrations. Looking at the Skaffold documentation I’m not entirely sure how this would be done or of it would just be using kubectl exec -it on the pod to run migrations after the fact. Can someone elaborate?

Thanks for the help and clarifications on this.

https://redd.it/opm9w2
@r_devops

Starting new role...any advice welcomed.

So after spending the last 12 months taking the jump from traditional Ops guy to a more DevOps orientated role I've been asked to lead the DevOps implementation/provision at my work place.

Although my new title will make alot of people cringe "DevOps Manager" I'm mainly tasked with all things automation, IaC, supporting the dev team and general implementation of the DevOps culture across the business.

I was hoping some good people out there could offer any advice on transitioning into the role, dos and donts and any other experiences they've had with a similar role shift...any advice is much appreciated 🙂

https://redd.it/opm1ex
@r_devops

Filebeat on Ubuntu-18.04

God afternoon family,

I have a filebeat agent running on my Ubuntu-18.04, and I would like to keep it running on the background. Should also start automatically once the server is is up without manual intervention.

Any suggestion?

https://redd.it/opo21t
@r_devops

Microservices architectures

Hello,

I'm a delivery engineer at a relatively small startup. We are trying to position ourselves as a software editor (we do some professional services on the side).

The product is designed based on microservices architecture, and it causing me a huge problems when trying to integrate to clients infrastructure.

Usually the product is installed on the client infrastructure but it can also be delivered as a VM appliance.

The product is composed of 18 docker containers, and we're using docker-compose as an orchestration tool.

Integrating these docker containers is very frustrating because of multiple reasons:

Containers images are not documented, and no one want to take the responsibility to document them.
The containers produce a huge amount of logs which make debugging a very difficult task.
docker-compose sometimes fails with TIMEOUT issues when launching the containers (I have to increase COMPOSE\_HTTP\_TIMEOUT).
Every container image has its own version which gives headaches, because every client uses a different version of microservices.
Updating the product is a pain in the ass, especially when they change/add/delete containers environment variables/volumes.
The system requirement are just insane (16go RAM, 4 CPU ...).
Supervision is very hard.
...

I feel that we're doing so many thing the wrong way and I don't feel that docker-compose is for prodution. The product/dev teams won't change their processes and way of work and I have to face the client an explain/justify all this mess.

Are any of you guys using docker/docker-compose this way ?

Any suggestion on how to improve things are welcomed.

I'm not sure if this is the right sub to ask my questions.

https://redd.it/opkwu4
@r_devops

Hiring managers of /r/devops, how often do you actually look at the candidates github profile if they have one? I feel like if more people actually looked the job search would be breezy.

I can do the work, I've proven I can do the work through code, I have years of experience and a history of proving I can adapt to change, and I like to think I can put on a friendly face and prove I'm not a complete ass to work with. Why do I still have to go through the process of interviewing and getting rejected at hundreds of places still to land a decent gig? Feels like the process hasn't changed since I started as a junior almost 10 years ago.

https://redd.it/opp5wy
@r_devops

Script management/deployment

Hi all,

We have a lot of powershell scripts across our estate. We've not really implemented any source control or management/deployment tools. You can imagine the hours lost when we need to make any changes across 3 separate datacentres.

Does anyone have any suggestions for a source control/deployment tool? There's been discussions about using Chocolatey for 3rd party software/package deployment and we use Octopus for our in-house apps.

https://redd.it/oormd7
@r_devops

What I need for new small software company

Hey dudes and dud-ettes.


So I got hired for a small project where they have an app that they want to possibly grow and scale.

I did some googling and I couldn't find a general list (Super general question, I know) of aspects of a new small software company that I should consider. Here is what I came up with but I would like to see if I can get some input from The Reddit at the risk of being to general and exposing my naivety.


Documentation
wiki?
Secret Manager
aws secrets manager
keybase?
Image Container repo
docker hub image repo
Communication
slack
mattermost
google chat
Monitoring
GAP stack
Code Repo
gitlab (We decided on this already)
CI/CD tool
gitlab's tool set
Ticketing System
jira
monday?

I'm open for suggestions around each bullet point but I'm more worried about the root topics and what I may be missing.


I'm fully aware of the broad ask here so please feel free to be general with your responses.

Thank you for any help in advance

https://redd.it/ontb9j
@r_devops

I made a lot of noise about getting an actual raise... And I got one!

While the title is probably not surprising for dedicated DevOps Engineers or those further along in their careers, it's certainly a well-known trend that we work somewhere for two or three years, and then bounce for a real raise.

I started at a startup in a major city as employee #36, moving out from a role paying half as much in the rural Midwest as a QA Engineer. Because of this incredible opportunity, I was willing to start at 80k, mainly because that already seemed like so much money; asking for anything higher felt a bit ridiculous, and I was coming in as a "Cloud Operations Support Engineer".

As one of the founding members of the DevOps team in a true DevOps organization from the C-Suite and founding engineers down, I learned a ton. I was promoted to a DevOps Engineer, and working directly with seasoned Software Engineers on my team, and being a software delivery team as well, I had to ramp up my skill-set very quickly to keep up. Eventually I became a pretty core employee and the company size has nearly tripled since then and I got a few minor raises that brought me to 100k, but that was still substantially below my market value, which I confirmed through multiple interviews and offers I received while still employed in my current position.

The thing is, I really didn't want to leave. I love my company, the work, the team. Everything. And I've learned such an insane amount that I became very confident in my worth to the company and also realized in the grand scheme of things, 100k is a drop in the bucket for most organizations.

I had multiple conversations with my extremely attentive and caring manager who went to bat for me and got me a raise to 130k! I know that's probably not a lot for some of you guys, but for me, who was making $8.25 as a line cook at Buffalo Wild Wings just five years ago, no degree, no certifications (but a massive bookshelf!) it's really a dream come true to feel like I have "made it", and that I can rise to the occasion and carry my weight and be a leader on my team.

I'm sure for the company it was weird that someone would make such a fuss about their pay and not leave with allegedly multiple offers on the table, but it was a good lesson for me to learn to stand up for myself, and not just bitch about my pay to coworkers and friends. A lesson I will carry forward in my career and encourage other people to do as well.

All in all, I am making 50k more than I was when I started 2.5 years ago, and I have to give props to my org for being attentive to me.

https://redd.it/optq8o
@r_devops

How would you improve the infrastructure of this Wordpress application?

We currently have a Wordpress running cronjob feeding data from a MSSQL db to a MYSQL db every 15 min running on a CENTOS linux with varnish for the caching. The Apache server uses nginx for the certificate. I am not sure if we're using any reverse proxy, I am not sure how to verify, but basically:

Everything goes through varnish, then apache (nginx is only used for the ssl certs) and MYSQL db. We use puppet for the deployment, I am not sure why, but we install npm, I think it might be for the js compilation script with gulp.

Any tip on how to improve this? Also, how do you check what exactly npm is used for?

https://redd.it/optu8t
@r_devops

How Adding Security into DevOps Accelerates the SDLC

A massive challenge that security teams face is the resistance that adding security into the software development lifecycle slows down the process and restricts progress.

Here is a good article that runs through how and why it accelerates the SDLC by Tiexin Guo.

https://blog.gitguardian.com/how-adding-security-into-devops-accelerates-the-sdlc-pt-1/

https://redd.it/opyr1q
@r_devops

ArgoCD UI not showing repository

I created a new repository

apiVersion: v1
kind: Secret
metadata:
name: private-repo
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
stringData:
url: [email protected]:user/project.git
sshPrivateKey: |
-----BEGIN OPENSSH PRIVATE KEY-----
...
-----END OPENSSH PRIVATE KEY-----

The public key of the PKI has been uploaded to Gitlab, and is working. I also tried using url: ssh://[email protected]:user/project.git but it didn't change anything. The UI is completely empty, nothing under Settings/Repositories, so what am I missing here?

https://redd.it/opyggp
@r_devops

article terraform | git commit -m “all the secrets”

not mine. interesting pov. thought it would warrant a discussion from people in the field to give their 2c about it.

https://medium.com/cloudandthings/terraform-git-commit-m-all-the-secrets-5dfea9b111de

https://redd.it/opzm65
@r_devops

5 Must Read DevOps Books

For all the people out there who want to understand DevOps and  CI/CD methodology better and start implementing it in their organization, I have curated a list of the top 5 must-read DevOps books. So let us dive into the world of books. 

https://redd.it/oq18nm
@r_devops

Your Ideal CI workflow and what's stopping you from implementing it?

Recently we integrated webhooks for CI into Devtron. You can currently configure these Webhooks on Devtron to run automatic builds for Pull Requests, Tags. For me, it felt like these are some of the most necessary Use Cases in CI to run automatic builds.

I can think of one or two more such configurations such as 'only run builds if the Pull requests are approved by someone' or 'Don’t run builds if the commit message doesn’t match a regex pattern,' but I am not sure how many would use these.

I am sure there are many more trigger points to run automated or manual builds. I am really curious to know what is your ideal CI workflow and if you haven't implemented it then why?

https://redd.it/oq26ah
@r_devops

Deploy cloud image via ISO

Hi all,

My enterprise does not allow instantiating VMs from images.
The only allowed method is to create a new VM and boot from one or more ISOs.

Is there a common way to create a bootable ISO that copies a cloud image to the VMs virtual disk? (E.g.: this focal img)

Right now, I'm using the autoinstall approach. But it's unnecessarily slow.

Cheers,

https://redd.it/oq12n7
@r_devops

How to incentivize team members to participate in code reviews?

We have a team of about 4-5 developers where we all contribute to various projects. When a change is made we send a PR link to our team and ask the team to review. Typically there is 1-2 people who are constantly reviewing and the rest of the team doesn't seem to want to participate at all.

We had discussed assigning PR's to developers randomly but that didn't seem like a great idea because of the varying level of knowledge/experience per developer as well as the developers current load.

Is there anyway to solve this problem without building any complex solution? How to incentivize the non-participating team members to contribute?

https://redd.it/oq3pkm
@r_devops

Azure DevOps Project

https://youtu.be/cWif2x2y1KM

https://redd.it/oq5wne
@r_devops

What is the problem with tags on DataDog?

Reddit keeps showing me an ad that suggests I should be frustrated with DataDog tags. I have never used DataDog but when I have briefly looked at its docs in the past, its tags seemed like a reasonable concept to me. What is wrong with them in practice such that an ad promising to get rid of them would be an effective pitch?

https://redd.it/oq6q7n
@r_devops

I'm on a quest to make developers in our org self sufficient when it comes to deployments, but unsure about the best approach when it comes to app configuration management. I would really appreciate your input.

I am of firm belief that if devs can't deploy their code to production without ops/admin involvement then something is missing in the process, therefore I'm working on a goal to make developers in the org self sufficient when it comes to deploying code to production.

In the process of implemeting ci/cd with (gitea + drone) I've come across an issue that I need an elegant solution for.The app in question uses config.ini file for configuration that currently is manually updated and contains configuration as well as secrets.


Migrating to a different configuration scheme (like central config server) is not viable at this time, due to time constraints, so the goal is to have the config.ini be constructed during ci/cd process.


I've come up with a plan to start version controlling config for production/dev/etc... in git and leave secrets values out, which will be injected into config file dynamically during ci/cd process.


For example, new config file in git would look like this:

config.ini.prod


DB_HOST = prod.site.com #would be real value
DB_USER = db #would be real value
SECRET_DB_PASS = #would be empty
...

One of the steps in ci/cd would do this:

cp config.ini.prod config.ini
./secrets-fill.sh
./secrets-confirm.sh
...

Where ./secrets-fill.sh would iterate through lines in config.ini and for any rows starting with SECRET_ it would fill value from secrets service like Vault

Then secrets-confirm.sh step would make sure no secrets are empty or else the pipeline would fail

This would make it possible for developers to ask admins to add secret to vault, and any time they would deploy code, ci/cd pipeline would be able to construct, fill and conirm config file was built fully.


Here are my questions:

* If we had to keep relying on config.ini file, is there any way you'd improve this approach?
* If I could convince stake holders to scrap this approach, ditch using config.ini and go with something else altogether, what would you recommend doing?
* In your org, who updates secrets in Vault/secrets service? Is there a bit of a silo situation there?

https://redd.it/oq5479
@r_devops

When automating a deployment involving Terraform, the common pattern is to store the state on a shared storage. What is the reason for not using a git private repository instead?

As the questions says. I'm looking into how terraform works from an automation perspective (deploying dedicated Shared VM for some legacy utility task runners from on-premise).

Everywhere I look it seems that the recommended solution is to store the tfstate files into a shared storage and pull them back in the pipeline workspace before the apply so that you can verify the state of the resources, then push them back.

To me it sounds a lot like a simple pull and push into a repository and I wonder why it's not suggested to use a dedicate repo instead so that we have a full changes history using git as well (yes, some storage solution have versioning as well, I'm aware).

I should add that most of the devops code/configs in our case are already in a dedicated git repository and the secrets in a secret manager. Everything get pulled together by the pipeline when a deployent need to happen.

It would have been natural to me to add a terraform-states directory in there and store the tfstate files but maybe I am missing something important if no one actually suggest that instead.

Links that explain it are more than welcome as well if you don't want to go into a detailed response!

A Little disclaimer: I know some people here are really passionate about what they use and I appreciate the devops community for that passion, I'm serious, but here I'm really asking more about the Why(s) than what 3rd party tool I should use :)

https://redd.it/oq92vo
@r_devops